A coding agent rarely sends a fresh prompt. Each turn carries the conversation so far. If it returns to a GPU process that still holds that prefix, the model can reuse it. If it lands somewhere else, the model may have to read tens of thousands of tokens again before producing its first new token.
That is the reason for session affinity: successive turns of one agent session should prefer the same place, while still being able to move when capacity runs out. In September 2026 I measured the cost of moving during an overnight swarm of up to eight agents building the same todo app from a 437-line spec. The model was Qwen3.8-27B on one A4000-pair host and three Radeon VII-pair hosts. The study was the baseline for viiwork v2.7.0's affinity work.
A miss could cost minutes
On a Radeon VII pair, a cold 30,000-token prefill ran at about 200 tokens per second. That meant roughly 2.5 minutes before the first token, compared with about 1.3 seconds when the turn stayed on a warm cache. In the default-routing window, 38% of turns changed node; moved turns had a median time to first token of 111–128 seconds.
The graph shows the share of turns that reused at least 80% of the previous context, excluding compaction turns. With two sessions on two slots, every turn hit. With eight sessions entering through one node and routing freely across 34 slots, the hit rate fell to 72.9%. Spreading entry points or pinning each run to a node improved the picture, though neither solved movement between GPU processes within a node.
A host is not the whole cache
In the default-routing run, the entry node repeatedly chose among peers with similar free capacity. Of 189 turns executed on the Radeon VII hosts, 129 changed node. Even after sessions were distributed so they stopped crossing nodes, the receiver's round-robin choice still sent most turns to another backend on the same node.
That local move was cheaper because a backend could retain an older prefix in its RAM cache. In the spread-entry run, moving between backends cost a median 26 seconds to first token, versus roughly 120 seconds for a cross-node move. Good affinity therefore has two levels: prefer the same node, then the same backend, while allowing a soft fallback when either is full.
There is a second constraint: a backend can evict a conversation even if routing sends the next turn back to it. On one A4000 backend, raising llama.cpp's RAM prompt cache from 8 GiB to 64 GiB removed the measured eviction misses for three sessions, at the cost of much more host memory. Placement and cache size have to work together.
The key has to identify the session
An agent can send X-Session-Affinity or X-Session-Id with a stable value for its run. That gives the router a dependable key. Hashing only the system prompt and first user message would have failed in this swarm: every agent began with the same two messages, so all eight would have appeared to be one session.
Affinity also has a natural limit. When the agent compacts its history, it rewrites the prompt; the old cached prefix no longer matches. Those turns were separated from the hit-rate chart because no placement rule can recover a prefix the client has replaced.
The practical rule is simple: keep a session close to the cache that served it last, but treat affinity as a preference, not a hard pin. The later two-host kv_share experiment asks the next question: when a session must move, how much does a faster link help move its cache with it?
Method note: the hit threshold was 80% of the previous turn's context. The windows ran in sequence, with one model, one agent, and one task; context sizes and run mix changed over the night. The chart is a comparison of observed windows, not a controlled estimate of an affinity feature's isolated effect. The measurements are from 28–29 September 2026.