viiwork: LLM inference across your own GPU machines - Blog

Two RTX 5090 hosts: how much does a faster link help?

2026-10-27T00:00:00+00:00

When a long-running agent's next turn lands on the other GPU host, that host needs the conversation state to avoid reading the whole prompt again. viiwork's kv_share copies a replica to its peer so a host switch can reuse most of the work already done.

I ran the same coding-agent workload on two RTX 5090 machines while changing the connection between them: 1 GbE through a LAN switch, a direct 10 GbE cable carrying Tailscale traffic, and that same cable with kv_share sent directly. Each host had four model slots. The agents built a todo app from a spec, and I measured both 12-session and 16-session runs over 90 minutes.

What is session affinity? It is the preference to send successive turns of one conversation to the same host, where its prompt cache is likely still warm. When that host is busy, the request may move; kv_share makes that move less costly. The earlier session affinity study shows why keeping a session near its cache matters in the first place.

The cable shortened the copy

The chart uses the 16-session runs, where the load stayed busy and the phases are most comparable. A typical replica delta was about 247 MiB. At 1 GbE it took 3.24 seconds to send. Over the 10 GbE cable through Tailscale, it took 0.81 seconds. Bypassing Tailscale cut the send to 0.55 seconds.

The raw cable carried 9.41 Gb/s in a single-stream test; Tailscale carried about 3.0–3.4 Gb/s. That explains why the first jump was roughly fourfold rather than tenfold. The direct socket was quicker again, but by then reading, hashing, and writing the replica at the receiver appeared to set the pace.

Did turns get warmer?

Yes, modestly. In the 16-session comparison, warm host switches rose from 93.8% at direct 1 GbE to 96.8% over 10 GbE with Tailscale. Switches that reused some, but not enough, of the prompt fell from 5.4% to 2.2%. A faster copy was more likely to reach the other host before the next turn arrived.

The direct 10 GbE run sent replicas faster still, but its warm share was 96.2%. That difference from 96.8% is within the variation between these runs. At this load, removing Tailscale improved transfer time without a clear end-to-end cache benefit. Forcing the direct cable down to 1 GbE also reproduced the switched 1 GbE send time, so the LAN switch itself was not the bottleneck.

What remained slow

An import into the receiving engine still took about 2.8 seconds at every link speed. Cold switches stayed around 1–2% in the busier runs. Traces showed a recurring software problem: viiwork remembered that it had imported a replica, but the engine could later evict that conversation. On a return visit, viiwork skipped the import even though the engine no longer held the cache. That import-once bug was fixed in viiwork v2.11.1 by asking the engine what it still holds before skipping an import.

For these two hosts and this workload, 1 GbE was workable: average kv_share traffic was roughly 320–450 Mb/s in each direction, and most switches were warm. A 10 GbE link made transfers more timely and reduced partly warm turns. The larger remaining gains were in engine import and cache correctness, rather than more network bandwidth.

Method note: these are single 90-minute runs per configuration, not repeated statistical trials. The 12-session runs were less evenly loaded, so the comparison above uses the 16-session J, H, and L phases. A “warm” switch reused at least 90% of its prompt; “partly warm” reused 30–90%. The measurements were made on 26 October 2026 with viiwork v2.11.0-beta1 and Strata 0.2.6, before the import-once fix.

Why coding agents need session affinity

2026-10-03T00:00:00+00:00

A coding agent rarely sends a fresh prompt. Each turn carries the conversation so far. If it returns to a GPU process that still holds that prefix, the model can reuse it. If it lands somewhere else, the model may have to read tens of thousands of tokens again before producing its first new token.

That is the reason for session affinity: successive turns of one agent session should prefer the same place, while still being able to move when capacity runs out. In September 2026 I measured the cost of moving during an overnight swarm of up to eight agents building the same todo app from a 437-line spec. The model was Qwen3.8-27B on one A4000-pair host and three Radeon VII-pair hosts. The study was the baseline for viiwork v2.7.0's affinity work.

A miss could cost minutes

On a Radeon VII pair, a cold 30,000-token prefill ran at about 200 tokens per second. That meant roughly 2.5 minutes before the first token, compared with about 1.3 seconds when the turn stayed on a warm cache. In the default-routing window, 38% of turns changed node; moved turns had a median time to first token of 111–128 seconds.

The graph shows the share of turns that reused at least 80% of the previous context, excluding compaction turns. With two sessions on two slots, every turn hit. With eight sessions entering through one node and routing freely across 34 slots, the hit rate fell to 72.9%. Spreading entry points or pinning each run to a node improved the picture, though neither solved movement between GPU processes within a node.

A host is not the whole cache

In the default-routing run, the entry node repeatedly chose among peers with similar free capacity. Of 189 turns executed on the Radeon VII hosts, 129 changed node. Even after sessions were distributed so they stopped crossing nodes, the receiver's round-robin choice still sent most turns to another backend on the same node.

That local move was cheaper because a backend could retain an older prefix in its RAM cache. In the spread-entry run, moving between backends cost a median 26 seconds to first token, versus roughly 120 seconds for a cross-node move. Good affinity therefore has two levels: prefer the same node, then the same backend, while allowing a soft fallback when either is full.

There is a second constraint: a backend can evict a conversation even if routing sends the next turn back to it. On one A4000 backend, raising llama.cpp's RAM prompt cache from 8 GiB to 64 GiB removed the measured eviction misses for three sessions, at the cost of much more host memory. Placement and cache size have to work together.

The key has to identify the session

An agent can send X-Session-Affinity or X-Session-Id with a stable value for its run. That gives the router a dependable key. Hashing only the system prompt and first user message would have failed in this swarm: every agent began with the same two messages, so all eight would have appeared to be one session.

Affinity also has a natural limit. When the agent compacts its history, it rewrites the prompt; the old cached prefix no longer matches. Those turns were separated from the hit-rate chart because no placement rule can recover a prefix the client has replaced.

The practical rule is simple: keep a session close to the cache that served it last, but treat affinity as a preference, not a hard pin. The later two-host kv_share experiment asks the next question: when a session must move, how much does a faster link help move its cache with it?

Method note: the hit threshold was 80% of the previous turn's context. The windows ran in sequence, with one model, one agent, and one task; context sizes and run mix changed over the night. The chart is a comparison of observed windows, not a controlled estimate of an affinity feature's isolated effect. The measurements are from 28–29 September 2026.

How the viiwork mesh routes a request

2026-09-15T00:00:00+00:00

A viiwork node serves the models configured on its own machine and keeps track of what every other node serves. That is what lets a client use one OpenAI-compatible endpoint for the whole fleet, without a list of machines or backends of its own. This post follows a request through the mesh, using the rules in the mesh documentation for viiwork v2.11.0.

Nodes find each other

There are no peer lists. On a Tailscale tailnet, which is the default, a node asks the local tailscaled for the other online machines and contacts them. On a plain LAN it finds them with mDNS. Where neither reaches, a few seed addresses in the config are enough. Discovery repeats every minute, which is also how two halves of a split network merge again.

Every machine uses two fixed ports: 8086/tcp for the API and the dashboards, and 7946 over both tcp and udp for membership gossip. In a secured mesh every machine has the same VIIWORK_MESH_SECRET. Gossip is then encrypted and authenticated, a node without the secret cannot join, and requests forwarded between nodes are signed. An open mesh has no secret and has to be declared with mesh.open: true; a node that has neither refuses to start.

The entry node looks at capacity

Every node is an entry point. Membership travels by gossip, but capacity is polled: each node asks every live member for its /v1/capacity once a second. When a request arrives, the node that received it looks for hosts that serve the requested model and have a free slot. A report older than three seconds is not used for routing, so a machine that has just gone quiet stops getting work before it leaves the membership list.

With performance routing, which is on by default, each node measures its own time to first token for each model and publishes the result. For a request, every host with a free slot gets a predicted time from its score and the size of the prompt, and only hosts within 1.25 times the best prediction are considered. If the entry node is one of them, it runs the request itself. Otherwise faster hosts are picked more often. A host with no score yet is priced at the fleet's median and gets one request in twenty, so it can earn a score of its own. When no host has a score, the request goes to a free local backend, else to the member with the most free slots.

A forward happens once

When the request goes to another machine, the receiving node admits it only into a free local slot. If it has none, it answers 429 at once. It never queues a forwarded request and never forwards it again, so a request cannot bounce around the mesh. A forward that is refused before its first byte (a 429, a 503 or no connection) is retried on another route, and that member is treated as full for the model until a newer capacity report arrives.

When every slot is busy

The entry node puts the request in a first-in, first-out queue for that model and waits up to 20 seconds (routing.queue_timeout) for a slot anywhere in the mesh. If none opens, the client gets HTTP 429 with Retry-After: 2.

Steering a request

Most clients can let the mesh choose. A coding agent should send X-Session-Affinity or X-Session-Id with a value that stays the same for the session: its turns then go to the same host, whichever node they enter by, and reuse that host's prompt cache. ?host=<node name> pins a request to one machine, and ?prefer=a,b names machines to try first when they have room; if none of them does, the request routes as if the list were not there. Both names are only compared with member names and never dialled as addresses.

The mesh page has the remaining details, including aliases, the mesh-wide model names that clients can use instead of real model names.

Bringing a new machine into a viiwork mesh

2026-08-06T00:00:00+00:00

A machine becomes a viiwork node through viiwork init, the setup wizard. It finds the hardware and the model files, proposes a layout, and shows every file it will write before it changes anything. This post walks through a run of the wizard in viiwork v2.11.0; the install guide has the commands.

What the machine needs

The wizard supports three kinds of machine: Linux with Radeon VII, MI50 or MI60 cards (gfx906), Linux with NVIDIA cards, and Apple Silicon Macs. Other AMD cards are found too, but no image is verified for them yet, so on those the wizard writes the config only. On Linux the engine runs in Docker; on a Mac it runs natively under a LaunchAgent. Every machine needs GGUF model files in one directory, because the wizard downloads nothing, and ports 8086 and 7946 open between the machines of the mesh.

The wizard proposes a layout

After its preflight checks, the wizard lists the GPUs and asks for your models directory. It shows each model it finds with its architecture and size, and asks which ones to serve and under what name. Use the same name on every machine, so a request for it can run on any of them.

It then proposes which cards each model gets, the context per slot and the number of slots. You can accept the proposal, edit one model's line or drop a model, and a layout the node would refuse is shown together with the reason. On a Mac the plan is made against the memory that Metal can use, since the GPU shares it with everything else on the machine.

Joining or starting a mesh

Next the wizard looks for nodes nearby, on your tailnet and on the LAN, and offers three choices: join an existing mesh with a join code, start a new secured mesh, or start an open mesh without a secret. A join code carries the address of a node already in the mesh and the mesh secret, so the new machine must be able to reach that node, and the code has to be kept as private as the secret itself.

Before writing, the wizard shows every file with its content and the secret masked. On a yes it writes them, starts the node (on NVIDIA and on a Mac) and waits until every model has loaded, then runs its acceptance checks. On a gfx906 machine it stops after writing the configuration, because there is no published image for those cards; you build the image from the same release and start it with the example compose file.

Checking that it serves

Once the node runs, ask it what the mesh serves:

curl http://localhost:8086/v1/models

The dashboard on port 8086 shows the node, /mesh shows the whole mesh, and viiwork top shows the same in a terminal. When another machine joins, its models appear in that list on every node.

The setup page lists every file the wizard writes and covers updating and uninstalling.

Where viiwork came from

2026-07-16T00:00:00+00:00

viiwork did not start as a project. It started as a problem I kept having at my desk.

A pile of cheap VRAM

The hardware came first. gfx906 cards — Radeon VII, MI50, MI60 — had fallen off the back of the datacenter and onto the secondhand market, and for the price of one current GPU you could buy a stack of them with more VRAM between them than anything I could otherwise afford. They are not fast by today's standards, but a model either fits in VRAM or it does not, and these fit. I put them in whatever machines I had, pointed llama.cpp at a directory of GGUF files, and started serving models on the local network.

That worked for exactly one machine.

viiwork 1, and the wall it built

The first version of viiwork was the thin layer I wrapped around that: a process that launched a model, gave it a port, and answered an OpenAI-shaped API. One model, one process, one port. When I wanted a second model I started a second process on a second port. When I bought a second machine I did it all again over there.

The trouble was never one machine. It was the seams between them.

Every model was its own instance file, with its own port, and the ports followed per-family schemes I had to keep straight in my head.
A model's id came from its GGUF filename, so renaming a file quietly renamed the model.
And every machine carried a hand-written peers.hosts list naming every other machine. Add a node and you edited that list everywhere — on the new box and on all the old ones — and if one copy drifted, requests fell into a hole.

By the time the fleet was three machines — gb1, forge-white, yeti — bringing a fourth online was an afternoon of editing config on four hosts to tell them all about each other. I was maintaining a map of the network by hand, on every machine, in triplicate. The inference worked. The bookkeeping was the whole job.

The idea that became version 2

The rewrite came from one stubborn thought: the machines are already on a network that knows which of them are up. Why am I telling each one what the others are?

Everything the machines were already on — the tailnet, the LAN — could answer "who else is here" without my help. So viiwork 2 threw out the peer lists and let the mesh form itself:

One node per machine. A single process supervises every model on the box, instead of one process per model. Its models: list replaces the pile of instance files.
Two fixed ports, the same everywhere. The API and dashboards on 8086, membership gossip on 7946. No more port schemes to remember.
A mesh with no roster. A node advertises itself and finds the others through tailscaled, or through mDNS on a LAN. Adding a machine means writing that one machine's config and nothing anywhere else. A machine that stops is dropped from routing within seconds.
Explicit model names. A node serves exactly the name you give a model, and the same name on every machine means a request for it can run on any of them.
Routing that knows about slots. A request goes to a local backend with a free slot, or to the member with the most free, or waits briefly in a queue — instead of landing on whichever host a peer list happened to name first.

The config that used to live in four places collapsed into one file per machine that only describes that machine. The map of the network stopped being something I maintained and became something the mesh worked out for itself.

That change is the whole reason there is a v1-to-v2 migration guide: v2 refuses a v1 viiwork.yaml at startup, because almost every assumption underneath it had moved. If you are coming across from that era, migrating to v2 is the map.

Where it is now

Everything since has been built on that spine. Aliases let a stable name like stable-coder point at a real model and be switched mesh-wide from any machine. kv_share carries a session's warm state to the next host so a busy machine no longer means a cold read. Releases are reproducible and signed, so a node will only update itself to a build the publisher actually stood behind. The reference fleet is still those gfx906 cards, now joined by NVIDIA boxes and a couple of Apple Silicon Macs that run the engine natively.

But the shape of it has not changed since the rewrite. Switch a machine on, and within seconds its models are usable from every other node, with nothing edited anywhere else. That is the thing I wanted at my desk, back when I was keeping the network map in my head and losing. If you want to see it from the other end, bringing a new machine into a mesh is a single machine's first viiwork init.