# Two RTX 5090 hosts sharing a KV cache over Ethernet
URL: https://viiwork.pulu.app/blog/kv-share-two-5090-hosts/
Updated: 2026-10-27

A large language model does not reread your whole conversation from scratch on every turn. As it works through the prompt, each layer produces key and value tensors for every token — the **KV cache** (also called the prompt cache). Keep that cache and the model can skip straight to generating the next token. Lose it and it has to push the whole prompt through the model again first, a step called *prefill*. On a long coding-agent conversation that can mean reading tens of thousands of tokens before the first new one appears — seconds to minutes of waiting.

The catch is that this cache lives in the GPU process that served the turn, on one machine. A [viiwork mesh](/docs/mesh/) normally sends a session's next turn back to the same host for exactly that reason: its cache is still warm. But when that host is busy, the turn has to run somewhere else, and the new host starts cold — a full prefill before it can answer.

**KV-cache sharing is viiwork's way around that.** After a turn, `kv_share` copies a replica of the session's cache across the network to the host the session would spill to next. When the switch happens, the receiving host *imports* that replica instead of recomputing the prompt, and the turn resumes warm. The copy only helps if it reaches the other host before the next turn arrives — so the link between the two hosts matters.

This post asks how much it matters. I ran the same coding-agent workload on two RTX 5090 machines while changing the connection between them: 1 GbE through a LAN switch, a direct 10 GbE cable carrying Tailscale traffic, and that same cable with `kv_share` sent directly. Each host had four model slots. The agents built a todo app from a spec, and I measured both 12-session and 16-session runs over 90 minutes.

> **Why would a session move at all?** Because a host can be busy when the next turn arrives. The preference to keep a conversation on the host with its warm cache is called *session affinity*; when that host is full the turn moves, and `kv_share` is what makes the move cheap. The companion [session affinity study](/blog/session-affinity-coding-agents/index/) shows why keeping a session near its cache matters in the first place.

## Without sharing, a switch starts cold

To see what the copy buys, picture the move without it. When a turn spills to a host that has never seen the conversation, that host has no cache to reuse: it runs the entire prompt through the model before it can emit a single new token. The cost scales with the prompt, and a coding agent's prompt grows all day. The slower the card and the longer the conversation, the worse it gets — a cold prefill is about **41 seconds for 50,000 tokens on an older RTX A4000, and minutes at 200,000 tokens on the gfx906 fleet**. On fast RTX 5090s it is quicker, but it is still pure repeated work: the receiving host re-derives a KV cache that already existed on the host the session just left. Session affinity avoids this most of the time by keeping a session put, but a busy host forces the move, and without a shared cache every forced move pays that cold prefill.

![Bar chart comparing time to first token after a host switch: about 41 seconds for a full cold prefill without a shared cache, against about 2.8 seconds to import a shared replica and resume.](assets/kv-cache-vs-no-cache.svg)

## How the copy works, on Strata

The engine running the model here is [Strata](/docs/models/#strata-one-moe-family-across-several-cards), which serves the Qwen3.8-Flash-Next mixture-of-experts model and holds each conversation in its own cache. `kv_share` is a Strata capability that viiwork drives, and it moves the cache in three steps:

1. **Save.** After a turn long enough to be worth it — 8,192 tokens by default, since a short conversation is cheap to prefill anyway — the host that ran the turn asks Strata to save the session to a file.
2. **Send.** It ships that file to the one member the session would spill to next, not to everyone. And only the *delta* travels: about 240 MB of a roughly 1 GB cache per 50k-token turn, since most of the conversation was already sent on the previous turn.
3. **Import.** Before the session's next turn runs on the receiving host, viiwork imports the replica into Strata's cache. Strata resumes from it only if the prompt still matches, so a stale or missing replica costs the import and nothing worse — a failed import just serves that turn cold, exactly as if there were no sharing at all.

The full mechanism, its config (`group`, `min_tokens`, `max_gib`) and its requirements are in the [kv_share documentation](/docs/models/#sharing-sessions-between-hosts-kv_share). The save and the import happen on each host; the one step whose cost the *network* governs is the send in the middle — which is the part this experiment changes.

## The cable shortened the copy

The chart uses the 16-session runs, where the load stayed busy and the phases are most comparable. A typical replica delta was about 247 MiB. At 1 GbE it took 3.24 seconds to send. Over the 10 GbE cable through Tailscale, it took 0.81 seconds. Bypassing Tailscale cut the send to 0.55 seconds.

![Horizontal bar chart of median replica send time: 3.24 seconds at 1 GbE, 0.81 seconds on 10 GbE through Tailscale, and 0.55 seconds on direct 10 GbE.](assets/kv-share-send-time.svg)

The raw cable carried 9.41 Gb/s in a single-stream test; Tailscale carried about 3.0–3.4 Gb/s. That explains why the first jump was roughly fourfold rather than tenfold. The direct socket was quicker again, but by then reading, hashing, and writing the replica at the receiver appeared to set the pace.

## Did turns get warmer?

Yes, modestly. In the 16-session comparison, warm host switches rose from **93.8% at direct 1 GbE** to **96.8% over 10 GbE with Tailscale**. Switches that reused some, but not enough, of the prompt fell from 5.4% to 2.2%. A faster copy was more likely to reach the other host before the next turn arrived.

The direct 10 GbE run sent replicas faster still, but its warm share was 96.2%. That difference from 96.8% is within the variation between these runs. At this load, removing Tailscale improved transfer time without a clear end-to-end cache benefit. Forcing the direct cable down to 1 GbE also reproduced the switched 1 GbE send time, so the LAN switch itself was not the bottleneck.

![Diagram showing a session switching from one GPU host to another: replica transfer shrinks with a faster link, while the receiving engine's import remains about 2.8 seconds.](assets/kv-share-transfer-and-import.svg)

## What remained slow

An import into the receiving engine still took about **2.8 seconds** at every link speed. Cold switches stayed around 1–2% in the busier runs. Traces showed a recurring software problem: viiwork remembered that it had imported a replica, but the engine could later evict that conversation. On a return visit, viiwork skipped the import even though the engine no longer held the cache. That *import-once* bug was fixed in viiwork v2.11.1 by asking the engine what it still holds before skipping an import.

For these two hosts and this workload, **1 GbE was workable**: average `kv_share` traffic was roughly 320–450 Mb/s in each direction, and most switches were warm. A 10 GbE link made transfers more timely and reduced partly warm turns. The larger remaining gains were in engine import and cache correctness, rather than more network bandwidth.

*Method note: these are single 90-minute runs per configuration, not repeated statistical trials. The 12-session runs were less evenly loaded, so the comparison above uses the 16-session J, H, and L phases. A “warm” switch reused at least 90% of its prompt; “partly warm” reused 30–90%. The measurements were made on 26 October 2026 with viiwork v2.11.0-beta1 and Strata 0.2.6, before the import-once fix.*