Run the setup wizard on each machine
The wizard finds the GPUs and the GGUF model files, proposes which cards each model gets and how many slots it has, and shows every file before it writes anything.
Open source LLM inference for several machines
Run LLMs across
all your GPU machines.
Install one viiwork node on each machine. The nodes find each other, and a request sent to any of them runs on a free slot of whichever machine serves that model, through an OpenAI-compatible API on port 8086.
Runs on AMD and NVIDIA GPUs under Linux and on Apple Silicon Macs.
EXAMPLE MESH 4 NODES
ENTRY NODEOpenAI-compatible API
NODE 01AMD GPUs2 free slots
NODE 02NVIDIA GPUs4 free slots
NODE 03Apple Silicon Mac1 free slot
A request runs wherever a slot is free
HOW IT WORKS
Each machine runs a node, and the nodes pool their slots.
A node serves the models configured on its own machine and learns what every other node serves. A client can send its request to any node and it finds a slot somewhere in the mesh.
Nodes find each other
On a Tailscale tailnet a node finds the others through tailscaled, and on a LAN through mDNS, so there is no peer list to keep. A new machine joins a secured mesh with a join code printed by any node.
Call any node
Every node accepts OpenAI-compatible requests on port 8086. A request runs on a free slot for its model, on the same machine or another one, and waits up to 20 seconds in a queue when every slot is busy.
FEATURES
A node routes, aliases and measures on its own.
All of this ships in the one viiwork binary and is described in the documentation for the current release.
Read the documentationSeveral engines on one machine
llama.cpp, vLLM, FreeToken and Strata can run on the same node, each model on its own GPUs. viiwork drives the engines and does no inference itself.
Routing by measured speed
Each node measures its own time to first token per model and publishes it. Requests are sent mostly to the hosts predicted to answer soonest, and a conversation with a session header stays on one host so its prompt cache stays warm.
Model aliases
Clients can ask for a stable name such as stable-coder. You change which model it means once, from any machine, and the whole mesh follows.
Dashboards and viiwork top
Every node serves a page for itself and one for the whole mesh, with models, running requests, backends and prompts. viiwork top shows the mesh in a terminal.
Power and energy
A node can record wattage, energy per model in kWh and, with a spot price source configured, what the electricity cost.
Clients that find the models themselves
OpenCode and Roo Code can read the served models and their context lengths from a node, so nobody keeps a model list by hand.
FROM THE BLOG
Recent posts
ARCHITECTURE · 5 MIN READ
How the viiwork mesh routes a request
Any viiwork node accepts a request for any model in the mesh. This post follows one request from discovery and capacity reports to the slot that runs it.
GETTING STARTED · 4 MIN READBringing a new machine into a viiwork mesh
What the viiwork setup wizard checks, proposes and writes when it turns a machine with GPUs and GGUF files into a node, and how the next machine joins.
QUICK START
Ask a node what the mesh serves.
Once the wizard has started a node, list the models of the whole mesh and send a chat completion to one of them. The install guide covers download, verification and adding more machines, and an AI assistant can follow it with you.
Open the install guideTERMINAL
# Models served anywhere in the mesh
$ curl http://localhost:8086/v1/models
# The mesh, live in a terminal
$ viiwork top