# Models and Hardware URL: https://viiwork.pulu.app/docs/models/ Updated: 2026-10-27 What the reference fleet runs, what has been measured on it, and the hardware rules those measurements imply. Numbers here are measured throughput, not estimates. Most of this document is about the gfx906 lane: 10x Radeon VII (16 GB HBM2) on a mining-rig topology, running `llamacpp`. There, one hard constraint shapes every recommendation: **a dense model whose weights plus KV cache do not fit in a single 16 GB card pays a ~3x throughput tax** (validated on Qwen3.5-A3B Q4_K_M vs Q3_K_M on the same GPU). Above that line, tensor-split across 2+ GPUs avoids the tax at the cost of single-stream parallelism — see [configuration.md](/docs/configuration/#tensor-split-mode). **That rule is a property of the engine and the card, not of viiwork.** The fleet's `freetoken` lane breaks it deliberately: a sparse MoE far larger than one card's VRAM runs on a *single* GPU by keeping only the hot experts resident and streaming the rest from host memory. See [Large models on one card](#large-models-on-one-card-freetoken-and-moe-offload). The `strata` engine does the same for one model family on gfx906 itself, with the layers split across a backend's cards: see [Strata](#strata-one-moe-family-across-several-cards). - [What the reference fleet runs today](#what-the-reference-fleet-runs-today) - [Large models on one card: FreeToken and MoE offload](#large-models-on-one-card-freetoken-and-moe-offload) - [Strata: one MoE family across several cards](#strata-one-moe-family-across-several-cards) - [Models from viiwork-parrot](#models-from-viiwork-parrot) - [Tuning rules measured on gfx906](#tuning-rules-measured-on-gfx906) - [Validated production deployments](#validated-production-deployments) - [Bring-ups in progress](#bring-ups-in-progress) - [Single-GPU picks](#single-gpu-picks-16-gb) - [Tensor-split picks](#tensor-split-picks-multi-gpu) ## What the reference fleet runs today Five models across a five-host ROCm mesh, read from `/v1/capacity` on 2026-09-13. The table below this one is the evaluation catalogue — measurements for models that were benchmarked, whether or not they are deployed — so start here for what is actually in service. | Model | Spread | Context per slot | Role | |---|---|---|---| | `Qwen3.8-27B` | 2 hosts | 49152 | general coder and prose; behind `stable-coder` and `stable-prose` | | `granite-4.2-8b` | 2 hosts | 16384 | fast utility model; behind `stable-granite` and `stable-factcheck` | | `translategemma-27b-it` | 3 hosts, 3 backends each | 4096 | the translation lane; behind `stable-translate` | | `gemma-4-31B-it` | 3 hosts | 6144 | prose | | `Ornith-1.5-35B-A3B` | 1 host | 262144 | long context | Two things this shows that the catalogue does not. **A model is usually served by more than one host**, which is what lets a node forward a request rather than queue it — spread is a routing property, not a redundancy nicety. And **context per slot varies by an order of magnitude between models on the same hardware**, because it is traded against slot count and VRAM per backend, not fixed by the checkpoint. Aliases are mesh-wide and resolve on whichever node a request reaches; see [mesh.md](/docs/mesh/#aliases). `viiwork alias list` against any node prints the live table. Those five are the ROCm hosts. The fleet also runs `vllm` on one host and `freetoken` on two — all three engines in one mesh behind one API, which is what v2.2.0 is. `strata` is the fourth engine, added later; it has run on one ten-card Radeon VII host (below). ## Models that read images A chat request may carry a picture as an `image_url` content part. A backend reads it only when it was started with the model's vision part, and from v2.10.0 the mesh knows which backends were. **Turning it on, per engine.** | Engine | How | What the node says | |---|---|---| | `llamacpp` | `llamacpp.mmproj: /models/.gguf` (or `--mmproj` in `args`) | yes with a projector, no without; cannot say for a model fetched with `-hf` | | `strata` | a `"vision"` section in the model's Strata JSON: `exe` (`strata-vision`), `mmproj`, `model`, `gpu`, `max_tokens`, and `threads` for a CPU encoder | what the server's `/health` reports as `images` | | `vllm`, `freetoken` | the engine's own flags | cannot say | Strata's encoder runs on the model's card (`"gpu": true`, about 700 MiB of VRAM held back for it) or on the CPU (`"gpu": false`), which costs no VRAM. **What the mesh does.** Each node publishes `images` per model: `true`, `false`, or absent for "cannot say". A request with an image then goes: 1. to a host that says yes, waiting in the queue for it when it is full; 2. when no host says yes, to hosts that cannot say, as before v2.10.0; 3. when every host that serves the model says no, nowhere: the client gets `400` with `model "" does not read images on any node`, or, with `?host=` naming a host that says no, `... does not read images on host ""`. So one name can be served with a projector on one node and without on another, and a picture still lands where it can be read. A text request is not affected by any of this. **What to expect.** A request body is at most 32 MB, images included. The prompt history keeps a request's text and not its images. How many tokens an image costs and how well a model reads a screenshot depend on the model and its projector. ### Measured: Strata with a CPU encoder on one RTX 5090 2026-10-13, one host: an RTX 5090 (32 GB), 128 CPU threads, 247 GB of RAM, `qwen3.8-flash-next` (IQ3_XXS) under Strata v0.1.41 at 262144 tokens of context and 4 slots. The card was full before the change (31992 of 32607 MiB). The vision section used the CPU encoder: `"gpu": false`, `"threads": 64`, projector `mmproj-Qwen3.8-Flash-Next-BF16.gguf` (908 MB). | | Text only | With the vision section | |---|---|---| | VRAM in use | 31992 MiB | 32000 MiB | | RAM of the node's container | 58.8 GiB | 59.8 GiB | | Load, park to healthy | not measured | 60 s, of which 30 s is the warm-up | | One text request, 400 tokens out | 125 tok/s | 125 tok/s | | Four text requests at once | 182 tok/s together | 185 tok/s together | | Slots | 4 | 4 | So the encoder cost this host no VRAM, no slot and no text speed. A 1440x1000 PNG screenshot (26 KB) cost about 990 prompt tokens with `"max_tokens": 1024` in the vision section, and about 290 with 300, Strata's default for a CPU encoder. One request with the screenshot and a short question took 6.5 s to answer with about 240 tokens; four at once took 7.6 s and produced 142 tok/s together. The screenshot was a web dashboard with a red rectangle drawn at the right end of a titled band, and the question was which panel the rectangle is in. - At 1024 image tokens, 7 of 8 answers placed the rectangle correctly (top right, above the row of cards, beside the intro text), and 5 of those 7 named the band's title. One answer put it in the navigation bar. - At 300 image tokens, 0 of 3 answers were right: all put it in or beside the navigation bar. A screenshot with small text needs the larger budget. This is one image and eleven answers on one model, enough to choose a setting and not enough to call the model reliable at reading interfaces. ## Large models on one card: FreeToken and MoE offload FreeToken's unit of deployment is **one card per process** — it has a `--tensor-parallel-size` flag but rejects more than one GPU, and viiwork fails the config rather than letting it die at load with a Python traceback. So it cannot reach a large model the way llama.cpp does, by spreading layers across cards. It reaches them the other way: **the weights do not all have to be resident.** For a sparse MoE, only the hot experts stay in VRAM and the rest stream from host memory, so a checkpoint far larger than the card runs on one GPU. The strategy is the `moe_backend` key: ```yaml - name: DeepSeek-V4-Flash engine: freetoken path: /models/DSV4-Flash-NVFP4 gpus: [0, 1, 2] gpus_per_backend: 1 # three single-card backends context: 32768 # -> --max-seq-len-override parallel: 4 # -> --max-running-requests startup_timeout: 45m freetoken: memory_ratio: 0.90 moe_backend: auto # auto | fused | offload | cpu | hybrid ``` **Proven, not theoretical.** v2.2.0's release criterion was that a real model run on the hardware it is meant for, and DeepSeek-V4-Flash serving from NVFP4 checkpoints on RTX 5090s is what met it. That lane took part in the 60-minute fleet soak alongside the gfx906 and vLLM hosts — 104 slots, 7 models, all three engines, 0.1x to 4x of capacity. Three things to know before you plan one: - **Loading is slow, and that is the offload working.** FreeToken loads weights, sizes its cache pools and captures CUDA graphs before it serves, and on an offload MoE backend it then fills the GPU expert cache from host memory. On the models this engine exists to run that is minutes to tens of minutes, which is why viiwork's default `startup_timeout` for `freetoken` is **30 minutes** — ten times the llama.cpp default. Size it from your observed cold-load time. - **The advertised context is not the servable context, and viiwork does not correct it.** On a live RTX 5090 capture, DeepSeek-V4-Flash advertises **1,048,576** tokens while the KV pool the engine actually built holds **64,128** — a factor of 16. FreeToken sizes that pool from the VRAM left after weights *and* the expert cache, so the ceiling is a property of the card and the checkpoint, not of the checkpoint alone. The node publishes the `context:` you wrote, so writing the advertised number gets you a mesh advertising it and a backend answering `400 context_length_exceeded`. **Set `context:` to what the card can actually serve**, and check `/v1/cache/status` on the backend to find out what that is. - **Set `kv_reserve_tokens` on every offload model.** viiwork never generates `--moe-cache-auto`, but the engine turns it on itself whenever the strategy is offload, cpu or hybrid and no cache-sizing flag was given — which is what `moe_backend: auto` resolves to on a MoE model. The expert cache is MoE-priority and takes VRAM the KV cache needs, and `0` is not "no reserve": the flag is simply not generated and the engine keeps its own floor of 8192 tokens, so the backend serves 8192 tokens however much context the node publishes. Set it to at least `context`, so one slot can always use the window the node advertises; `context × parallel` is the ceiling, every slot full at full context. Everything else — dtype, attention backend, MoE cache size, KV capacity, page size, CUDA-graph sizes — FreeToken resolves from the checkpoint and the card, and does it better than a config file can. That is why the `freetoken:` block is four keys; anything else belongs in `models[].args`. ## Strata: one MoE family across several cards [Strata](https://github.com/Niko1221/Strata) (MIT) serves one model family, Qwen3.8-Flash-Next (a 125B-parameter sparse MoE) in several sizes and fine-tunes. Like FreeToken it keeps the experts in host memory and the hot ones on the card; unlike FreeToken a backend spans **several cards through a layer split**, and it has a gfx906 build, so it runs on the Radeon VII hosts. Each backend is a Python server that starts the C++ engine from a JSON config. viiwork launches and probes the Python server; **`path` is that JSON file, not the weights**: ```yaml models: - name: qwen3.8-flash-next-coder engine: strata path: /s/run/strata-coder.json # Strata's own JSON config gpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] gpus_per_backend: 5 # two backends of five cards context: 131072 # per slot parallel: 1 startup_timeout: 45m env: {STRATA_ARENA_MMAP: "1"} # needs --pcie-frac 0 in the JSON's args strata: dir: /opt/strata # the Strata checkout; required python: python3 # default warmup: 30s # default ``` - **The paths depend on the image.** The gfx906 image (`viiwork-strata`, built from source) and the CUDA image (`viiwork-strata-cuda`, upstream's ready-made 0.2 image) lay Strata out differently, and the model's block and the JSON must name what is in the image they run on: | | `viiwork-strata` (gfx906) | `viiwork-strata-cuda` (Strata 0.2.3) | |---|---|---| | `strata.dir` | `/opt/strata` | `/opt/strata/terminal` | | `strata.python` | `python3` (default) | `/opt/strata/terminal/python/bin/python3` | | JSON `"exe"` | `/opt/strata/build-906/strata` | `/opt/strata/components/engine/0.1.48-linux-x86_64-cuda-cuda13/strata` | | JSON `"cwd"` | `/opt/strata` | `/opt/strata/terminal` | | JSON `"lib_dirs"` | — | `["/opt/strata/components/engine/0.1.48-linux-x86_64-cuda-cuda13"]` | | expert profile | `/opt/strata/data/expert-profile.bin` | `/opt/strata/terminal/data/expert-profile.bin` | The engine directory is named for the engine package's version (0.1.48 in Strata 0.2.3; the fork's kv_share build is 0.1.48.1). With another image's paths the backend dies at launch. The CUDA image's paths were the `/opt/strata` checkout and `/opt/strata/.venv/bin/python` up to Strata 0.1.45 (viiwork v2.11.0-beta1 and earlier). - **One JSON serves every backend of the model.** The node passes host, port and cards as flags; the file's `host`, `port` and `gpu` are ignored. Leave `layer_split` out and Strata splits the layers itself; the measurements below used explicit split points. - **The file must agree with the entry**, or the backend refuses to start and names the key: `model_name` (or an alias) with `name`, `--max-context` with `context`, and `lazy_load`, `idle_unload_s` and `api_key` off. See [configuration.md](/docs/configuration/#models). - **`parallel` is 1 to 8, and the JSON's `"parallel"` must say the same.** Each slot is a conversation of the full `context`, and every slot takes VRAM that the expert cache would otherwise hold, so a second slot can cost more speed than it gains; upstream's `docs/BATCHING.md` has the numbers. The engine runs fewer slots than asked when they do not fit and says so only in its log; the node publishes the count `/slots` lists, so look at `/v1/status` after a load. A request that is alone decodes on the engine's faster single path, and a second one moves both into batch slots. Needs Strata v0.1.41: an older build lists one slot whatever the file says. - **Preparing the model is upstream's job.** The pack, the GGUF shards and the MTP layer come from Strata's own `setup.py`; viiwork does not fetch or build them. Images and their traps are in [BUILDS.md](https://github.com/janit/viiwork/blob/v2.11.1/BUILDS.md). - **Host RAM is part of the budget.** With `STRATA_ARENA_MMAP=1` the backends of one model share one 24 GB mapped expert file; two backends loaded used 36 GB. Without it one instance held about 27 GB resident. - **`--trim-stage-weights` is what fits the Coder on two cards.** Without it 90% of the experts were resident and decode fell from 56 to 21 tok/s. - **With three cards per backend, leave `--kv-resident` out.** The whole KV cache of a 262,144-token context is about 1 GiB at `--kv int8`, and three Radeon VIIs have the room. On gb3 (2026-10-05, three backends of three cards, three coding sessions for 25 minutes) total output rose from 37 to 60 tok/s and decode at 40 to 80 thousand tokens of context from 31 to 57 tok/s; a 250,000-token cold prompt was read at 676 tok/s and left about 850 MiB free on the fullest card. `--prefill auto` was slower than `--prefill 4096` there: it borrowed expert-cache slots for its larger chunks and prompt reading fell to between a quarter and a half. - **A backend waits 30 seconds after loading** (`strata: {warmup: 30s}`, phase `warming up`). The wait was added for `saving a checkpoint part failed` on Radeon VIIs and does not prevent it; `warmup: 0s` turns it off. - **`saving a checkpoint part failed` on Radeon VIIs is fixed in the `viiwork-strata` image from Strata v0.1.40.1** (the gfx906 patch, [BUILDS.md](https://github.com/janit/viiwork/blob/v2.11.1/BUILDS.md); upstream has the fix itself since v0.1.42). It was the first long prompt after a load, on a backend of more than one card: the engine saves a checkpoint every 16,384 prompt tokens while another thread is still capturing its prompt graphs, and the ROCm runtime refuses that copy. An image built before that still fails that way, and the backend reloads. ### Measured on Radeon VII The Coder (IQ1_M quant, 54 GB), 131,072 context, one slot per backend, on a ten-card Radeon VII host with a 4-core EPYC 3151. Upstream's `benchmark.py`, fresh prompts, 256-token outputs, Strata v0.1.39. | Cards per backend | Prompt read tok/s (4K / 32K / 128K) | Decode tok/s (4K / 32K / 128K) | Time to first token, s | |---|---|---|---| | 2 | 124 / 265 / 254 | 48 / 42 / 37 | 33 / 124 / 505 | | 4 | 282 / 747 / not run | 44 / 37 / not run | 15 / 44 / not run | | 8 | 192 / 750 / 896 | 36 / 32 / 30 | 21 / 44 / 143 | | 5, with a second five-card backend loaded | 212 / ~710 / 640–694 | 34 / 28–30 / 33 | 19 / 46 / 185–200 | The two-card row is three runs per length, the four-card row two, the others one. More cards read a long prompt faster and decode slower. **A cold 128K prompt takes minutes to the first token** on any of these layouts, so size `routing.queue_timeout` and client timeouts for it. Under viiwork, on the same host as two backends of five cards: - Both backends were healthy about 10 minutes after start; loads are serial, 3.5 minutes each, about 6 on the first start that writes the mapped expert file. - Pinning holds: each backend's engine processes sat on its own five cards. - `/health` and `/slots` answered in 1 to 2 ms throughout a 41,409-token cold prompt, so a long prefill does not trip the health ladder. - A repeated prompt reports `cached_tokens` in `usage`. - Thinking: a stream carries the reasoning in `reasoning_content`, apart from the answer; see [thinking-models.md](/docs/thinking-models/#strata). - The CUDA image served under viiwork on three RTX A4000 (2026-10-04) at the model's full 262,144-token context: a 258,663-token cold prompt answered correctly in 88.5 s (about 2,900 tok/s prompt reading, about 60 tok/s decode), `/v1/status` answering within 15 ms throughout, and a prompt over the limit refused with a 400. Known gaps, as of 2026-10-05: - **Untested:** `layer_split` left to auto. - **A stream cut short reads as complete.** Strata's server ends a streamed reply by closing the connection, so when a backend dies mid-reply the node cannot tell the early close from the end: the client's stream ends cleanly without a `[DONE]` line. A client that needs to know checks for `[DONE]`. ### Sharing sessions between hosts (kv_share) A client that sends `X-Session-Affinity` (or `X-Session-Id`) keeps its turns on one host, where Strata's conversation cache holds the conversation. When that host is full, the turn spills to the next one and pays a cold prefill: 41 s for 50k tokens on an RTX A4000, minutes at 200k on gfx906. With `strata.kv_share`, the conversation follows the session: ```yaml strata: dir: /opt/strata/terminal # the kv-export-0.2 image's layout python: /opt/strata/terminal/python/bin/python3 kv_share: group: flash # the members that share; same model files and JSON min_tokens: 8192 # default: a shorter conversation is cheap to prefill max_gib: 32 # default: replicas this node holds # dir: ... # default /kvshare/; new or empty, never shared ``` - After a turn of 8192 tokens or more, the node that ran it asks its backend to save the session (`POST /slots/session?action=save`, about 0.7 s at 50k tokens, during which that backend's slots pause) and sends the file to one member: the first after itself in the session's rendezvous order, which is where the router spills the session, or the session's home when the turn ran on the spill host. Only what changed travels: about 240 MB of a 1 GB file per 50k-token turn (0.6 s at 10 Gbit/s, 2.3 s at 1 Gbit/s). A save waits 2 s after its turn and is dropped when the session's next turn starts first. - Before a session's turn runs on a backend, the node imports the replica it holds into that backend's conversation cache (`POST /slots/0?action=import`, about 1.5 s at 50k tokens, on the turn's clock), once per replica and backend launch. Strata resumes from it only if the prompt matches, so a stale or foreign replica costs the import and nothing else, and a failed import serves the turn cold. - Replicas travel on the API port (`/v1/kvshare/manifest` and `/put`) and are accepted only from members, under the same rule as a forward: a signature in a secured mesh, the member's own address in an open one. A replica is a conversation's prompt. - Routing knows where a session's state is. The entry node remembers which host ran each session's last turn and prices that host, and the member holding its replica, by what resuming costs there rather than by a full prefill: the replica host at its measured import time per 1000 tokens. So when the session's host is full, the turn goes to the replica host even if another host is faster at a cold prefill. This holds for turns that enter through the same node; a session that enters elsewhere routes as without kv_share. - `/v1/capacity` publishes the model's group as `kv_share_group` and the measured import cost as `kv_share_import_ms_per_1k`, and `/v1/status` counts pushes, puts and imports under `kv_share`. Requirements, refused at start when unmet: the janit/Strata `kv-export-0.2` build (upstream 0.2.3.1 plus the session save, import and delta endpoints, which upstream lacks), `"slot_save_path"` in the JSON at a path the node reads and writes (in the Docker image the node and the engine share the container, so it is the same path), a conversation cache (on by default since Strata 0.2; an explicit `--conversation-cache-mib` of 0 in `args` is refused), and one card per backend (Strata saves no layer-split session). Keep `slot_save_path` on disk, not tmpfs: Strata writes it with O_DIRECT, and tmpfs was slower (1.12 s against 0.68 s). Members of a group must load the same model files with the same JSON: Strata checks both fingerprints and refuses anything else on import. **The fork's image has the 0.2 layout** of the CUDA image (the table under [Strata](#strata-one-moe-family-across-several-cards)), with its own engine package: the JSON's `"exe"` is `/opt/strata/components/engine/0.1.48.1-linux-x86_64-cuda-cuda13/strata` and `"lib_dirs"` names that same directory (its `libcublas`, `libcudart` and `libstrata_kq`). forge-white (one RTX 5090) has served this way since v2.11.0-beta1, and forge-black beside it since 2026-10-25. **Measured between those two hosts** (12-16 coding-agent sessions on 8 slots, 90 minutes per run): 93-97% of host switches resumed warm and 1-2% cold, over 1 GbE as over 10 GbE. A copy took 3.2 s at 1 GbE and under a second at 10 GbE, which moved only 2-3 points of switches from partly warm to warm. What mattered was the conversation cache: at 16 GiB / 8 conversations Strata evicted imported copies before their session came back (17% cold); at 64 GiB / 32, under 2%. **An evicted copy is imported again** (v2.11.1). The conversation cache evicts oldest first and does not say which conversation went, so a node cannot assume that a copy it imported earlier is still there. When a session comes back to a backend with the copy that backend was already given, the node asks Strata how much of the conversation it still holds (`POST /slots/session?action=held`, in janit/Strata `v0.2.6-kvx1` and later) and imports again when that is less than the copy. With an older fork build the question gets a 400, and the node imports each copy once as before; in the measured runs that was most of the remaining avoidable cold switches. **Where the files go.** `kv_share.dir` (the replicas a host holds for its peers) and the JSON's `slot_save_path` (the engine's own saves, and the file an import reads) are a cache: a lost or damaged file costs one cold prefill, never a wrong answer, so what matters is write endurance and speed, not durability. Every turn of a long session writes about 240 MB on the host that ran it and about as much on the host that receives it, and an import writes the whole file (about 1 GB at 50k tokens) before the engine reads it; a save pauses the backend's slots while it runs. Put both on a drive of their own, Optane where there is one, rather than on the root disk or the one the weights are read from. Size them at about 1 GB per 50k-token conversation each (`max_gib` bounds `kv_share.dir`, default 32). Mount the drive at the same path on every host of a group, so one JSON, and with it one `slot_save_path`, serves them all. ## Models from viiwork-parrot A model can name a viiwork-parrot catalog id instead of a file: ```yaml models: - name: Qwen3.8-27B engine: llamacpp source: viiwork-parrot:qwen3.8-27b-q4kxl gpus: [0, 1] context: 49152 ``` At start the node asks the host's viiwork-parrot (`POST /ensure` on `viiwork_parrot.api`, default `127.0.0.1:7950`) for the model and runs the engine on the path it returns. viiwork-parrot downloads it if needed, checks every sha256 and seeds it; viiwork keeps no copy of its own. - Set `path` or `source`, never both. - While viiwork-parrot fetches, the model's backends are `starting` with phase `fetching`, and the node logs progress every 10%. A download does not count against `startup_timeout`, and it does not hold up any other model's load. - If viiwork-parrot is not running yet, the node waits for it (retrying up to once a minute) — start order does not matter. - If viiwork-parrot refuses (unknown id, not available on this host, a failed verification, no disk space), the model's backends are dead with its message. Fix the cause and restart the node or change the model and send SIGHUP. - **Paths must match.** viiwork-parrot answers with host paths. In a container, mount its data directory at the same path (for example `/srv/viiwork-parrot:/srv/viiwork-parrot:ro`); the node checks the path exists and says so if it does not. - A folder model (a safetensors directory) needs an engine that takes a directory — vllm or freetoken. llamacpp refuses a directory. - `viiwork-accept config` checks a sourced model read-only through viiwork-parrot's `GET /status`. ## Tuning rules measured on gfx906 Each of these cost a bring-up to learn. They are specific to gfx906 where they say so; the reasoning generalises further than the numbers do. > **Build note.** Hybrid-attention models (Qwen3.5-A3B, Qwen3.6/3.8, Laguna, anything using DeltaNet / linear attention) need an upstream-current `llama.cpp` — build a fresh image from `docker/Dockerfile.rocm`. > > Same-week architectures can need more than "current master" — they can need an *unmerged* one. Check `general.architecture` in the GGUF before planning a bring-up: if `llama-server` answers `unknown model architecture: 'X'`, no flag will fix it and the only path is a build from whichever PR adds `X` (Qwen3.8-Flash-Next needed PR #27742 from `unslothai/llama.cpp`; master rejected it outright). Pin the PR head in a dedicated `docker/test/Dockerfile.-test` rather than tracking `master`, and re-pin to a release tag once it merges. > **Quant choice on gfx906: take the highest quant that fits.** Measured on > Qwen3.8-27B, Q6_K costs **0 tok/s** against Q4 despite reading 28% more bytes > per token. This hardware is kernel-bound, not bandwidth-bound, and higher > quants trade bytes for dequant ALU work — the two cancel. The usual > "drop a quant level for speed" instinct is wrong here; drop one only to fit > VRAM or buy context. One exception: archs whose tensor columns are not > divisible by 256 (e.g. `nemotron_h_moe`) silently fall back to non-K types, > where Q6_K becomes q8_0 and is pure loss — check before assuming. > **Split mode must stay `layer`.** Measured 2026-08-15: `--split-mode row` is > refused outright by the ROCm backend ("does not support split buffers"), and > `--split-mode tensor` loads but runs ~9× slower on prefill while using *more* > VRAM. Layer split runs a group's cards strictly sequentially — one card > computes at a time — so **extra GPUs in a group buy VRAM and context, never > throughput**. For throughput, run more backends, not wider ones. > **Size `startup_timeout` from your cold-load time.** `llama-server` answers > `/health` with 503 *"Loading model"* for the whole time it is reading tensors. > A starting backend waits for that patiently, but only for the model's > `startup_timeout` (10 minutes by default for llama.cpp), after which the node > gives up and respawns it. Measured 2026-08-27: a 104 GB model on USB loaded at > ~30 MB/s (~60 min), and a restart minutes from ready discarded 78 GB already > placed in VRAM. Rule of thumb: `model_bytes / observed_read_bytes_per_sec`, > then double it. > **A single tensor larger than one card is a hard wall, not a tuning problem.** > Layer split assigns whole *layers* to cards and `-ot` assigns a whole *tensor* > to one device — neither can spread one oversized tensor, and `--split-mode row` > (which would) is refused by this ROCm backend. So any model carrying a > monolithic tensor above **16.37 GB** must keep it in host RAM here, no matter > how many GPUs you own. When that tensor is on the per-token path the model > becomes single-core CPU-bound and the GPUs idle. Qwen3.8-Flash-Next is the > worked example below: a 26.82 GB n-gram embedding, unchanged across every > published quant. Check the largest tensor before assuming VRAM total is what > matters — total capacity is necessary, not sufficient. > **Gemma 4 quant: prefer QAT.** Gemma 4 ships [quantization-aware-trained Q4 checkpoints](https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/) — int4 weights at near-bf16 quality and ~3× less memory than fp16. `scripts/download-gemma4-31b.sh` defaults to these. Two gotchas, both verified on gfx906: (1) Google's own day-one GGUFs are broken (garbage detokenization / leaked special tokens) — use Unsloth's clean requants (`unsloth/gemma-4-*-it-qat-GGUF`) and a current `llama.cpp` (`viiwork:latest`, b10437+); (2) Gemma 4 is a *thinking* model — for prose/direct output, disable thinking server-side with `args: ["--jinja", "--chat-template-kwargs", "{\"enable_thinking\": false}"]` (`--reasoning-budget 0` does not take on this template). ## Validated production deployments These layouts have stress-test data behind them. The numbers were measured under viiwork 1.x, one instance per model; the snippets below are the same layouts as `models:` entries for the machine's one `viiwork.yaml` ([migrating-to-v2.md](/docs/migrating-to-v2/) has the rules they follow — above all, `context` is **per slot**). **General all-rounder pick: `gpt-oss-120b` on 2× TS=5.** If you have 10 GPUs and want a single deploy that's both fast (≥40 tok/s single-stream) and high quality across coding, prose, translation, and reasoning, run the entry below. The 117B / 5.1B-active MoE is large enough to be smart and sparse enough to be quick on this hardware. Set `Reasoning: low` for snappy chat, `high` for harder problems. ```yaml models: - name: gpt-oss-120b engine: llamacpp path: /models/openai_gpt-oss-120b-MXFP4_MOE-00001-of-00002.gguf gpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] gpus_per_backend: 5 # two backends, each layer-split across five cards context: 16384 # per slot parallel: 1 ``` | Model | Quant | Mode | Measured | |---|---|---|---| | **gpt-oss-120b (MoE, 5.1B active)** — *all-rounder* | MXFP4_MOE (native) | 2× TS=5 (10 GPUs) | **41 tok/s** single-stream, **73 tok/s** aggregate at conc=4 (5-min sustained, 120/120 success). Per-request decode held flat under load (40.9 → 40.3 tok/s). Latency p50/p95: 4.9 / 6.7 s single, 10.2 / 12.2 s at conc=4. Reasoning-enabled (harmony format) — set `Reasoning: low/medium/high`. | | Gemma-4-26B-A4B-IT (MoE, 4B active) | UD-Q3_K_XL + KV-q4 | replica × 5 | **142 tok/s** aggregate at conc=10 (5.5h KV bench, 0 fail). KV-q4 vs fp16 is +9.2% throughput, -2 GB VRAM, 7/7 functional eval matches baseline. Highest aggregate throughput on this hardware. *Quant note: the QAT Q4 checkpoint (`unsloth/gemma-4-26B-A4B-it-qat-GGUF`, UD-Q4_K_XL, ~14.2 GB) is the quality-first choice but is tight for replica×5 on 16 GB — Q3_K_XL remains the measured throughput config until QAT is benched on this fleet.* | | Qwen3.6-27B (dense hybrid) | Q4_K_M | 5× pair tensor-split (`gpus_per_backend: 2`) | **76 tok/s** aggregate at conc=10 across all 10 GPUs (15-min stress, 0 fail). Single-pair single-stream: 16.9 tok/s. | | Qwen3.8-27B (dense hybrid) — *supersedes 3.6* | Q6_K | TS=2 pair | ~15 tok/s single-stream, **the same as 3.6 at Q4** — this is a quality and VRAM upgrade, not a speed one. 1.4 GB lighter than 3.6 and ships MTP weights embedded. Context ceiling is **98304, not 131072**: MTP allocates a *second* KV cache that also scales with context, and 131072 OOMs at `common_speculative_init_result`. Prefill 176 tok/s at `-ub 512`. | | Qwen3.5-35B-A3B (MoE hybrid, 3B active) | Q3_K_M + KV-q4 | replica per GPU | **40.7 tok/s** sustained at conc=9 (15-min stress, 0 fail). 2.8× faster than Q4_K_M because weights fit fully in VRAM. | | Gemma-4-31B-IT (33B dense) | QAT UD-Q4_K_XL | TS=2 single backend | ~17.3 GB across 2 GPUs (down from ~21.5 GB at the old Q5_K_S, same prose quality); used as the prose generator in the localization pipeline. Run with `--jinja --chat-template-kwargs '{"enable_thinking": false}'` for direct output; the entry is below the table. | | EuroLLM-22B-Instruct-2512 | Q5_K_M | TS=2 single backend | ~16 GB across 2 GPUs; purpose-trained on 24 EU languages + Norwegian / Icelandic / Russian — the translator step in the localization pipeline. | | Laguna-XS-2.1 (MoE, 33B / ~2.8B active) — *evaluated, since displaced* | Q4_K_M | 5× TS=2 (10 GPUs) | 36.3 tok/s single-stream per backend; **113 tok/s aggregate** at conc=5. That is 3.1× single-stream, not 5× — the reference host has 4 CPU cores for 5 backends and viiwork warns about the oversubscription at startup. VRAM 14.6/13.2 GB per pair at 128K. TS=2 is mandatory: 20.3 GB does not fit one 16 GB card. | | Laguna-S-2.1 (MoE, 118B / 8.1B active) — *evaluated, not retained* | unsloth UD-Q6_K (97.9 GB) | TS=10 (whole host) | 20.8 tok/s decode, 176 tok/s prefill. Beat its own projection on decode but **prefill is the weak side**: ~3.4 TFLOPS effective, 2.9× less FLOP-efficient per token than a dense 27B, because top-10-of-256 routing at `-ub 512` leaves each expert ~20 tokens of work. A cold 256K fill costs ~72 min, so the advertised context is real in VRAM and unaffordable in wall-clock. Replaced by the XS fleet above after use. | | Granite-4.2-8B | Q4_K_M | single-GPU replica × N | ~5 GB weights, generous KV headroom for 16k context. Run with `-fa on`. IBM's enterprise/utility model — strong instruction following, function/tool calling, RAG / structured-output workflows, multilingual; well-suited to back-office automation, doc Q&A, and embedding into agentic loops where you want a small, predictable, English-leaning helper next to a heavier reasoning model on the mesh. | Gemma-4-31B-IT as the pipeline's prose generator, with thinking off: ```yaml models: - name: gemma-4-31b-it engine: llamacpp path: /models/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf gpus: [0, 1] gpus_per_backend: 2 # one backend across both cards context: 16384 # per slot parallel: 1 args: ["--jinja", "--chat-template-kwargs", '{"enable_thinking": false}'] ``` ## Bring-ups in progress Not production rows yet — recorded so the next attempt does not rediscover the same walls. | Model | State | What is known | |---|---|---| | Soofi-S-30B-A3B (hybrid Mamba-2 / MoE) | **Blocked** on HuggingFace manual approval | A TS=2 layout on two cards was drafted but never checked in. The GGUF declares `general.architecture = nemotron_h_moe`, **not** "soofi" — it reuses an existing arch, so no llama.cpp bump is needed; do not grep binaries for "soofi". Quant choice inverts the rule above: columns (2688/1856/3712) are not divisible by 256, so every K-quant falls back — Q6_K becomes q8_0 (~32 GB, no quality gain) and Q5_K_M becomes q5_1 (~25 GB, the pick). No community requant exists to route around the gate, and self-converting is blocked because the base repo is gated too. | | Qwen3.8-Flash-Next (125B total / 6B active, GDN + QSA hybrid) | **Runs, but loses to the 27B** — not retained | Loads and serves correctly across all 10 GPUs, and is *slower than Qwen3.8-27B on two*: 8.7 / 16.0 / 20.4 tok/s at conc 1 / 2 / 4 against the 27B's 10.2 / 18.6 / 27.0. The cause is structural, not tuning. `general.architecture = qwen4exp`, which upstream master rejects outright (`unknown model architecture`); support is only in the still-open PR #27742 from `unslothai/llama.cpp` (branch `qwen4exp/qwen3.8-flash-next`). The blocker is `per_layer_token_embd.weight`: **26.82 GB as one indivisible IQ4_NL tensor** (51.2B elements — the n-gram table). A Radeon VII holds 16.37 GB and `-ot` assigns a whole tensor to one device, so it can never be GPU-resident here and stays in host RAM. Decode is then pinned to a **single CPU core** (measured 0.91 of 4 cores busy with GPUs at 0%), which is the real ceiling: 48.5/160 GB VRAM is in use while 30.9 GB sits in RSS. Needs cards ≥27 GB to be worth revisiting. Re-tested at UD-Q4_K_XL (103.7 GB) to rule out the quant: decode was **unchanged at 12.6 tok/s**, confirming the ceiling is CPU, not quantization — on gfx906 the higher quant rides free. Quality at Q4_K_XL was *better* than the 27B on Finnish (correct terminology throughout vs four terminology errors and a case error) and equal on strict-JSON extraction, and it was more token-efficient (5 of 8 eval prompts completed in budget vs the 27B's 2 of 8). But **neither model solves hard reasoning through this stack**: on one bridge-crossing problem the 27B burned 6,000 tokens / 6.4 min and Flash-Next 10,000 tokens / 17.1 min, both still mid-deliberation, and Flash-Next's decode degraded 12.6 -> 9.7 tok/s as context grew (QSA attention cost). Both emit raw chain-of-thought into `content` with no `` delimiters, which looks like a template/integration gap rather than a reasoning limit — worth retrying under vLLM/SGLang before concluding anything about the models. Verdict: not retained; the 27B gives comparable quality at 1.6x the speed on 2 GPUs instead of 10. | | Muse-Glimmer-30B (meta-models) | Ran on GPUs 4+7, since displaced | Needs llama.cpp **b10369+** (`muse_glimmer` landed in PR #26841); the older b9222 pin could not load it. kquant-dynamic is 19.65 GB so TS=2 is required, not preferred. Output needs `reasoning_strength: low` — at the template default the model self-talks and that text leaks into `content`. `--mmproj` and the DFlash drafter are deliberately not wired in (upstream #26873, #26894). | ## Single-GPU picks (≤16 GB) For lightweight / multi-replica setups. Q3_K_M is the practical ceiling on a Radeon VII for the 30B class — anything heavier triggers the VRAM-fit tax. | Model | Quant | Approx VRAM | Notes | |---|---|---|---| | Gemma-4-26B-A4B-IT | QAT UD-Q4_K_XL | ~14.2 GB | Best general-purpose pick on 16 GB; QAT Q4 = near-bf16 quality. Tight on a single card — run with KV-q4 + short context (`-fa on --cache-type-k q4_0 --cache-type-v q4_0`). For replica×N throughput or more KV headroom, drop to non-QAT UD-Q3_K_XL (~12.5 GB). | | Gemma-4-E4B-IT | QAT UD-Q4_K_XL | ~4.2 GB | 8B multimodal; QAT Q4 = near-bf16 quality at half the VRAM of the old Q8_0 (~8.2 GB). | | Granite-4.2-8B | Q4_K_M | ~5 GB | Strong instruction following, tool calling, RAG / structured-output workflows. Use as a fast utility model alongside a heavier reasoner. | ## Tensor-split picks (multi-GPU) For models above the single-GPU ceiling. Layer-mode tensor split costs roughly 2-13% per extra GPU on the reference fleet's PCIe-gen1-x1 mining-rig topology (measured on the gfx906 fork 6h stress). On modern PCIe gen3/4/5 the penalty is smaller. | Model | Quant | Min GPUs | Why tensor-split | |---|---|---|---| | Gemma-4-31B-IT | QAT UD-Q4_K_XL | 2 | 33B dense at near-bf16 QAT Q4 (~17.3 GB); higher prose quality than the 26B MoE. | | EuroLLM-22B | Q5_K_M | 2 | 22B dense translator; doesn't fit comfortably at Q5 on one card. | | Qwen3.6-27B | Q4_K_M | 2 (per pair, `gpus_per_backend: 2`) | Hybrid dense at single-stream tensor-parallel speed; 5-pair layout gives both per-request latency and aggregate throughput. | | Laguna-XS-2.1 | Q4_K_M | 2 (per pair, `gpus_per_backend: 2`) | 20.3 GB will not fit a 16 GB card, so TS=2 is the floor rather than a tuning choice. Five pairs is the throughput layout on a 10-GPU node. | | gpt-oss-120b | MXFP4_MOE | 5 (per group, `gpus_per_backend: 5`) | 117B / 5.1B-active MoE; the all-rounder pick on a 10-GPU node — see the validated row above for measured throughput. | > Other 30-32B models (Qwen3-32B, DeepSeek-R1-Distill, Qwen2.5-Coder, etc.) load on this hardware but aren't currently part of the reference fleet — add a `models:` entry to a node's `viiwork.yaml` and run `scripts/bench-sustained.sh` to add measured numbers.