Adding an Engine to viiwork

viiwork supervises inference engines. This is how you add one.

An engine package is a command line, a readiness rule, an occupancy reading, and the options that feed them. Everything else — spawning, GPU pinning, health ladders, respawns, routing, the mesh — belongs to the node and is already written. If you find yourself implementing any of it, stop and read "What the node already does" below.

Status. This documents the engine API as of v2.1.0, and as of 2026-09-13 it is the API that exists: every declaration below was re-verified against the code on the v2.1 branch after it was built. If you are working from a checkout that predates it, read "Writing against this before it ships" at the end.

This document is the contract you write against. The design rationale behind each rule — why it is this way, and what went wrong when it was not — is kept with the project's internal specs (C7) and is not part of the published tree. Where the two disagree, C7 wins and this file is the bug.

The shape of the job

internal/engine/<name>/
    <name>.go        Engine type, Name, Command, DefaultStartupTimeout, init registration
    health.go        Probe, Load, and any optional capability
    options.go       Options struct, defaults, ValidateOptions
    <name>_test.go   enginetest.Run plus your own tests
    testdata/        real engine output, captured verbatim

Plus one line in one file, the list of shipped engines:

// internal/engine/all/all.go
_ "github.com/janit/viiwork/v2/internal/engine/<name>" // <name>

Everything that needs to know the engines imports that package: the node (internal/node), viiwork-accept (which validates a config file before the node that will run it starts, so an engine it cannot see is an engine it reports as unknown) and the acceptance tests. None of them lists engines itself.

That is the entire footprint. Nothing in internal/config, internal/supervisor, internal/node, internal/route or internal/proxy learns your engine's name. If your engine cannot be added without touching one of them, that is a finding about the contract — say so rather than working around it.

The API

Everything you compile against. Copy these declarations into a stub if you are working before v2.1.0 lands.

`internal/engine`

package engine

import (
    "context"
    "time"

    "github.com/janit/viiwork/v2/internal/gpu"
    "gopkg.in/yaml.v3"
)

// Spec is one backend of one model: what the node decided, plus this engine's
// own configuration block.
type Spec struct {
    Name     string      // the model's mesh-wide name; the backend MUST answer to it
    Path     string      // models[].path, uninterpreted by the node
    GPUs     []int       // this backend's cards, in config order; empty = CPU
    Port     int         // the loopback port the node assigned
    Vendor   gpu.Vendor  // informational, never a gate
    Context  int         // tokens PER SLOT, always >= 1
    Parallel int         // sequences this backend admits, always >= 1
    Backends int         // backends this model runs on this node, always >= 1
    Args     []string    // models[].args, the operator's own flags
    Options  yaml.Node   // this model's engine block, or a zero Node
}

// Command is how to start a backend. Env is added on top of the supervisor's
// environment, which already carries the GPU pinning variables.
type Command struct {
    Path string
    Args []string
    Env  []string
}

// Probe is the result of one readiness check.
type Probe struct {
    Ready    bool
    Phase    string  // engine-reported load phase, "" when it reports none
    Progress float64 // 0..1, or -1 when unknown. Never 0 to mean unknown.
    Reason   string  // why a backend is not ready, for the activity log
}

// Load is a backend's current occupancy.
type Load struct {
    Slots      int   // what the backend will actually admit
    Busy       int   // engine-reported running sequences
    Waiting    int   // engine-internal queue; 0 when the engine publishes none
    CtxPerSlot int64 // what the backend will actually serve, per slot
}

// Engine is implemented once per inference server.
type Engine interface {
    Name() string
    Command(Spec) (Command, error)
    Probe(ctx context.Context, addr string) (Probe, error)
    // s is the Spec Command was called with for this backend.
    Load(ctx context.Context, s Spec, addr string) (Load, error)
    DefaultStartupTimeout() time.Duration
}

// Register makes an engine available by name. Call it from init.
// An empty or duplicate name panics at init rather than failing at first use.
func Register(e Engine)

// Lookup returns the engine registered under name.
func Lookup(name string) (Engine, bool)

// Names returns every registered engine name, sorted. It is what config
// validation reports in "must be one of: ...".
func Names() []string

// DecodeOptions decodes this model's engine block into dst, rejecting any key
// dst does not define. Its errors name the key and the line the operator wrote
// it on, never a Go type:
//
//     unknown key gpu_memory_utilisation (line 12)
//     ratio: cannot unmarshal !!str `high` into float64 (line 14)
//
// Your ValidateOptions adds the path, so what the operator finally sees is
//
//     models[0].vllm: unknown key gpu_memory_utilisation (line 12)
//
// A zero Spec.Options leaves dst untouched, so set your defaults on dst first
// and decode over them.
func DecodeOptions(s Spec, dst any) error

Capability interfaces

Optional, found by type assertion. Implement what your engine can answer honestly and nothing else — a method that returns a plausible-looking zero is worse than an absent one, because the mesh renders absent as "unknown" and zero as fact.

// OptionsValidator validates this engine's config block. path is the model's
// position ("models[0]"), so every error can name the field that is wrong.
// Called during config validation, before anything starts.
//
// The Spec it receives is BACKEND 0's, built before any backend exists: Name,
// Path, Context, Parallel, Args, Backends and Options are the model's; GPUs
// are the first backend's cards, so len(GPUs) is gpus_per_backend, which is
// not itself a Spec field; and Port is 0 and Vendor is "" because neither has
// been decided. A rule that reads Port or Vendor validates against nothing.
type OptionsValidator interface {
    ValidateOptions(path string, s Spec) error
}

// CPURunner is implemented by an engine that can serve with no GPUs.
// Not implementing it means `gpus:` is required for your engine.
type CPURunner interface {
    RunsOnCPU() bool
}

// TokenProgressReader is implemented by an engine that can report PER-REQUEST
// token progress. Cumulative counters are NOT this. Called in place of Load on
// every load tick.
type TokenProgressReader interface {
    LoadProgress(ctx context.Context, s Spec, addr string) (load Load, decoded, remain int64, err error)
}

// GPUBindingReader is implemented by an engine that reports which card it
// actually bound. Called once on the transition to healthy. A mismatch against
// the host inventory is REPORTED, never acted on. ok=false means the engine
// said nothing, which is not a mismatch.
type GPUBindingReader interface {
    BoundGPU(ctx context.Context, addr string) (uuid string, ok bool, err error)
}

// ImageReader is implemented by an engine that can say whether a backend
// reads images. Called once per launch, on the transition to healthy, with
// the Spec the backend was started with. ok=false means the engine cannot say
// for this backend, which is not "no": a request with an image goes to a
// backend that says yes while one exists, to one that cannot say when none
// does, and never to one that says no.
type ImageReader interface {
    ReadsImages(ctx context.Context, s Spec, addr string) (reads, ok bool, err error)
}

// DisplayNamer says what the engine is called by people ("llama.cpp" where
// Name is "llamacpp"). It is what status.models[].engine_name carries.
// enginetest.Run requires it: 1 to 40 bytes, no control characters.
type DisplayNamer interface {
    DisplayName() string
}

// Versioner is an engine that can say which version of itself is installed
// and which version this viiwork needs. MinVersion "" means no requirement;
// Version runs the binary the Spec's options name; AtLeast compares in the
// engine's own order. Without it nothing is checked.
type Versioner interface {
    MinVersion() string
    Version(ctx context.Context, s Spec) (string, error)
    AtLeast(installed, min string) (bool, error)
}

// UsageReporter declares how the engine's streamed chat responses report
// token usage, which is what performance routing measures with:
//   Unasked      - a streamed response ends with a usage chunk even when the
//                  client did not set stream_options.include_usage;
//   CachedTokens - usage.prompt_tokens_details.cached_tokens is present
//                  whenever any prompt token came from cache, so its absence
//                  means zero rather than "unknown".
type UsageReporter interface {
    UsageReporting() UsageReporting
}

// ReasoningSeparator is implemented by an engine whose streamed responses
// keep reasoning apart from the answer: reasoning arrives as
// delta.reasoning_content with no <think> tags, and the answer follows as
// delta.content. The proxy then passes the stream through as it is for a
// client that did not ask for thinking, instead of renaming reasoning to
// content.
type ReasoningSeparator interface {
    SeparatesReasoning() bool
}

// PerfKeyer is implemented by an engine whose model speed depends on
// something the node's model entry does not show, such as a config file of
// the engine's own. PerfKey returns a short string that changes when it
// does; the node adds it to the key of the saved performance baseline.
type PerfKeyer interface {
    PerfKey(s Spec) string
}

// WarmUpper is implemented by an engine whose server answers its probe as
// ready before it is safe to send it work. WarmUp is how long a backend
// must have been ready, without a break, before the node calls it healthy;
// until then it stays in starting with phase "warming up" and keeps its
// place in the load gate. Zero is no warm-up.
type WarmUpper interface {
    WarmUp(s Spec) time.Duration
}

// SessionSharer is implemented by an engine that can write a client
// session's conversation to a file and park such a file in its own cache
// (kv_share). The node moves the files between members.
type SessionSharer interface {
    SessionShare(s Spec) (cfg SessionShareConfig, ok bool)
    SaveSession(ctx context.Context, addr, session, name, base string) (SavedSession, error)
    ImportSession(ctx context.Context, addr, name string) error
}

// SessionHolder is implemented by a SessionSharer that can say what a
// backend still holds of a session's conversation.
type SessionHolder interface {
    SessionHeld(ctx context.Context, addr, session string) (int64, error)
}

ReasoningSeparator decides what a client sees when it streams without "think": true. Without it, reasoning that carries no <think> tags is renamed to content, which is right for a llama-server that may put its whole answer in reasoning_content. With it, the stream is passed through untouched: reasoning stays in reasoning_content, where a client either shows it as thinking or ignores it, and the answer is in content. Holding the reasoning back instead leaves an interactive client with nothing on screen for as long as the model thinks. The plain, non-streamed reply still drops the reasoning. Declare the capability only for a server that always writes its answer to content.

SessionSharer lets a session that spills to another member resume warm (kv_share; Strata only today). After a local turn of a session the node asks the backend to save that session's conversation (SaveSession, keyed by the client's raw X-Session-Affinity or X-Session-Id), sends the file as a delta to the member the session would spill to, and deletes it. Before a session's turn runs on a backend the node imports the replica it holds, if any (ImportSession), which must only park it: a later prompt that matches resumes from it, and one that does not is served as if nothing had been imported. A miss must never be a wrong answer. Return engine.ErrSessionGone when the backend no longer holds the session; the node drops that save quietly. SessionShare names the directory both calls use (SaveDir), which the node must be able to read and write at the same path.

SessionHolder is optional. A node imports each replica version into a backend once, but an engine's cache evicts on its own, so when a session comes back with the replica it was already given, the node asks SessionHeld how many tokens of that conversation the backend still holds and imports again if that is fewer than the replica's. An error means "cannot say", and the node then trusts its record of the import, which is all an engine without the method gets.

UsageReporter is what lets a host earn a performance score. A sample needs the prompt's uncached token count, so viiwork asks the engine for usage (injecting include_usage and stripping the chunk again for a client that did not ask) only when it declares CachedTokens. An engine without the capability is never asked for usage: it yields no time-to-first-token samples, so it publishes no score, and with routing.performance on the router treats it as a learning host — priced at the fleet median and given the 5% trickle — for as long as it serves. Declare only what the engine really does; a CachedTokens: true on an engine that omits the field for a cache hit records every cached prompt as fully prefilled and makes the host look faster than it is.

`internal/gpu`

type Vendor string

const (
    VendorNVIDIA Vendor = "nvidia"
    VendorAMD    Vendor = "amd"
    VendorApple  Vendor = "apple" // an Apple Silicon Mac: one GPU, no device variable
    VendorNone   Vendor = "none"
)

Spec.Vendor is informational. Use it if your engine spells a flag differently per vendor. Never refuse to run because of it: engines are named for the engine, not for a GPU vendor, and an engine that gains a second accelerator backend should need no change here.

`internal/engine/enginetest`

package enginetest

// Run asserts the contract every engine must honour whatever it drives.
// Call it from your package's test with cases covering the Spec shapes you
// support. It spins up its own httptest servers for the generic Probe and Load
// assertions; you supply no server.
func Run(t *testing.T, e engine.Engine, cases ...Case)

type Case struct {
    Name string      // subtest name
    Spec engine.Spec // a Spec your engine should accept
    // Options is the block's CONTENTS as an operator would write them — the
    // lines under `vllm:`, not the header — dedented to column zero. When
    // non-empty it is parsed into Spec.Options, so a case reads like the
    // config file it stands for.
    Options string
    // NameInConfig is for an engine whose server takes the model name from
    // a config file. Instead of looking for Spec.Name on the command line,
    // the kit calls Command again under a name the file does not carry and
    // requires an error.
    NameInConfig bool
}

What it checks: Name() is non-empty, lowercase and registered under that name; Command puts Spec.Name on the command line, binds 127.0.0.1 and Spec.Port, and appends Spec.Args last; Command does not panic on an options block it cannot use (returning an error is right, and succeeding is allowed for an engine that ignores options); Probe against a closed port returns an error, and against a server that answered returns no error, whatever it decides about readiness; Load reports no negative field, or an error; DefaultStartupTimeout is positive.

A note on that Probe pair, corrected after writing the kit: the universal rule is only that an answer is not a transport failure. Whether a 200 carrying nonsense means ready is the engine's own rule — llama.cpp reads a 200 as ready, FreeToken reads the document — so the kit does not assert readiness either way, only that answering is never reported as an error.

The kit is itself tested against a deliberately broken engine, in a subprocess, so that "my engine passes" is evidence rather than decoration.

What the node already does

The most common mistake when porting an engine from a standalone supervisor is reimplementing this half. None of it belongs in your package.

The node owns	So you never write
Spawning, the process group, SIGTERM then SIGKILL after a grace	process management, zombie reaping, "kill the children too"
GPU pinning: stripping inherited device variables, then `CUDA_DEVICE_ORDER=PCI_BUS_ID` + `CUDA_VISIBLE_DEVICES`, or `ROCR_VISIBLE_DEVICES`	anything in `Command.Env` about device selection
Port assignment, and a free retry if the engine fails to bind	port scanning or collision handling
The health ladder: starting, healthy, unhealthy after `health.max_failures`, respawn, dead after `health.max_respawns`	retry logic, backoff, failure counting
Serialising loads node-wide so two models do not thrash the disk	load queueing
The on-GPU PID check, which condemns a backend on no card of ours	anything that kills a backend
Draining in-flight requests before a respawn, a removal or shutdown	graceful shutdown
Alias resolution, routing, the origin queue, peer forwarding	anything about where a request goes
`/v1/models`, `/v1/capacity`, the dashboards, the mesh	anything on the wire

When your methods are called

	When	Timeout	On error
`Command`	once per backend launch and per respawn	none	the backend fails to start; the error is logged and counted
`Probe`	every 1 s while `starting`; every `health.interval` (default 5 s) once healthy	`health.timeout`, default 10 s, as the context deadline	counts one failure toward `health.max_failures` (default 3)
`Load`	every 1 s while healthy	1 s, as the context deadline	logged once, last good value kept; after 3 s stale the node falls back to `Slots = Parallel`, `Busy =` its own in-flight count
`DefaultStartupTimeout`	at launch	—	overridden by `models[].startup_timeout` when non-zero
`BoundGPU`	once, on the transition to healthy	short	logged, nothing else

Two consequences worth internalising:

While starting, a failed probe is the expected case. It does not count toward the failure ladder; only the startup timeout ends that patience. Return Ready: false with a Phase and Progress if your engine reports them — an operator watching a large model load then sees movement instead of silence.
A Load error is better than a guess. The node degrades to its own in-flight count within 3 seconds, which is honest. A zero published as fact makes the router send a burst at a saturated backend.

Writing it

Command

func (e *Engine) Command(s engine.Spec) (engine.Command, error) {
    opts := defaultOptions()               // your defaults, first
    if err := engine.DecodeOptions(s, &opts); err != nil {
        return engine.Command{}, err
    }
    args := []string{
        "serve", s.Path,
        "--host", "127.0.0.1",
        "--port", strconv.Itoa(s.Port),
        "--served-model-name", s.Name,     // REQUIRED: see below
        // ... whatever your engine needs, derived from s and opts
    }
    // Operator args LAST, so their flag wins: every engine in the fleet takes
    // the last occurrence of a repeated flag.
    return engine.Command{Path: opts.Binary, Args: append(args, s.Args...)}, nil
}

Three things enginetest enforces and the fleet depends on:

The backend must answer to s.Name. A client asks for Qwen3.8-27B and reaches whichever node has a free slot; a backend advertising its file path instead is unreachable through the mesh.
Bind 127.0.0.1 and s.Port. A backend on a routable interface is an unauthenticated inference server on the tailnet.
s.Args go last.

s.Backends is how many backends the whole model runs on this node, for host-level tuning that divides a shared resource between them: llama.cpp's automatic --threads is a fair share of the CPUs across a model's backends, and without it ten backends each take half the machine.

s.Context is per slot. Translate it to whatever your engine wants — llama.cpp gets --ctx-size s.Context*s.Parallel, an engine with a per-sequence limit gets s.Context directly. Whatever you pass, the backend must actually serve s.Context per slot, because that is the number the node publishes to the mesh.

Probe

The single most common porting bug: an HTTP 200 is not readiness. Engines differ, and they differ in ways that break silently.

func (e *Engine) Probe(ctx context.Context, addr string) (engine.Probe, error) {
    resp, err := e.get(ctx, addr, "/health")
    if err != nil {
        return engine.Probe{}, err   // transport failure: refused, EOF, timeout
    }
    defer resp.Body.Close()
    // ... decide readiness from the status, the body, or both
}

Return an error only for a transport failure. An engine that answers and says "still loading" is a successful probe reporting Ready: false — if you return an error there, the node cannot tell "loading" from "gone".

If your engine answers 200 while still loading (some do, for the whole many minutes a large model takes), readiness is in the body and a 200 alone is a trap: the node would advertise the backend and every routed request would come back 503.

Load

func (e *Engine) Load(ctx context.Context, s engine.Spec, addr string) (engine.Load, error) {
    // scrape whatever your engine publishes
    if !foundRunningGauge {
        return engine.Load{}, fmt.Errorf("...: no running-sequence gauge in this build")
    }
    return engine.Load{
        Slots:      s.Parallel,   // or what the engine reports, if it reports it
        Busy:       running,
        Waiting:    waiting,      // 0 if your engine publishes no queue depth
        CtxPerSlot: int64(s.Context),
    }, nil
}

Slots and CtxPerSlot are what the backend will ACTUALLY serve, which is not always what the node asked for. Report what your engine says when it says it — llama.cpp's /slots gives both, and it returns those rather than the config, because a value the engine silently clamped is only visible if somebody reports the real one. Fall back to s.Parallel and s.Context when your engine reports neither. Never return zero for either: the mesh reads a number as fact, and "0 slots" takes the model out of routing.

Load is handed its Spec, and that is a correction. The signature was Load(ctx, addr) in v2.1.0's first cut, which left these two fields unfillable by any engine that cannot observe them. Two independent clean-room engines reached the same workaround — a map from address to the Spec Command was called with — and it misreports as soon as the node reassigns a port, which it does after a failed bind. One parameter removed the class.

If a gauge is missing from this build of your engine — they get renamed between versions — return an error, and consider probing a list of known names rather than one.

Options

Your engine owns its YAML block, named for the engine:

models:
  - name: granite-4.2-8b
    engine: myengine          # names both the engine and the block below
    path: /models/granite-4.2-8b
    gpus: [0]
    gpus_per_backend: 1
    context: 16384            # per slot
    parallel: 8
    args: ["--some-operator-flag"]
    myengine:                 # captured verbatim, decoded by your package
      binary: myengine
      some_knob: 0.85

type Options struct {
    Binary   string  `yaml:"binary"`
    SomeKnob float64 `yaml:"some_knob"`
}

func defaultOptions() Options { return Options{Binary: "myengine", SomeKnob: 0.9} }

func (e *Engine) ValidateOptions(path string, s engine.Spec) error {
    opts := defaultOptions()
    if err := engine.DecodeOptions(s, &opts); err != nil {
        return err
    }
    if opts.SomeKnob <= 0 || opts.SomeKnob > 1 {
        return fmt.Errorf("%s.myengine.some_knob %v must be in (0, 1]", path, opts.SomeKnob)
    }
    return nil
}

Two rules: every error names the field that is wrong, and an unknown key is an error rather than a shrug — DecodeOptions handles the second for you.

Keep the block small. The right home for an engine's long tail of tuning flags is models[].args, not a mirrored YAML key per flag: a key with a zero value cannot express "let the engine decide", and most engines resolve most of their own settings from the checkpoint and the hardware better than a config file can. Promote a flag to a key when the node is the one deciding it, or when its value is published to the mesh.

Testing

No test may need a GPU, your engine's binary, or the network. Every engine viiwork has ever had was developed on a machine that could not run it.

func TestConformance(t *testing.T) {
    enginetest.Run(t, New(),
        enginetest.Case{
            Name: "single card",
            Spec: engine.Spec{Name: "m", Path: "/models/m", GPUs: []int{0},
                Port: 18080, Context: 8192, Parallel: 4, Vendor: gpu.VendorNVIDIA},
            Options: "binary: myengine\nsome_knob: 0.85\n",
        },
    )
}

Then your own tests, for what is specific to your engine:

Command line: golden tests. The exact argument slice, for each Spec shape you support. This is where a flag rename shows up.
Probe and Load: httptest servers replaying testdata/. Capture real output from the engine version the fleet will run, verbatim, including the awkward cases: a body from an older version with different metric names, a multi-process engine reporting several series, a document with a field absent.
Cover the absent case explicitly. A test that only ever sees a healthy engine will not catch publishing a zero as fact.

The node-level test (one per release, not per engine) runs the real supervisor against a fake binary that asserts the flags it was given and then serves your fixtures. If you want your engine covered there, say so — it lives in internal/node and needs a mode added to the test helper.

Packaging

One image per engine under docker/. The Go binary is built CGO-off and dropped into whatever base image carries your engine's runtime; viiwork compiles against none of it, because it only ever spawns and supervises. Pinning the base image is how the inference stack gets pinned.

Name the file for the engine, not the hardware: docker/Dockerfile.<name>. An engine that gains a second accelerator backend gains a sibling base image, not a fork.

If your engine JIT-compiles kernels, needs a toolchain at run time, or keeps a cache it would otherwise rebuild on every container start, say so in the Dockerfile's header comment and give the compose example the volumes it needs. A reviewer cannot infer it and an operator will not discover it until the first cold start.

Done when

Options with defaults, and ValidateOptions if any value can be wrong
The five methods, DisplayName, and any other capability your engine can answer honestly
engine.Register(New()) from init
enginetest.Run passes
Golden command-line tests, and httptest tests over captured testdata/
One blank import in internal/engine/all/all.go
docker/Dockerfile.<name> and a compose example, if the runtime needs one
A model block in viiwork.yaml.example and a row in README's engine table
gofmt, go vet, go test ./... clean

If any step forced an edit to internal/config, internal/supervisor or internal/proxy, write down what and why — that is a finding about the contract, and the contract is what gets fixed.

What two ports taught

The vLLM and FreeToken engines were written clean-room against this document by someone who did not read internal/engine, then landed in v2.2.0. Neither needed an edit outside its own package — the contract held — but the exercise corrected several things, and these are the ones that cost the most time.

An Engine is a singleton. Keep no per-backend state in it. One instance is registered from init and shared by every model on the node. Both engines initially kept a map from backend address to the Spec Command was called with, because an early draft of Load had no Spec. That map was wrong four ways: nothing removed an entry, so a reload leaked one; the supervisor reassigns a port after a failed bind, so a later model could be handed a port an earlier one used and the map would answer with the wrong model's Parallel and Context; it made Load silently depend on Command having run first in the same process; and it put mutable state on what should be a stateless strategy object. Load now receives its Spec, and none of that is needed. If you find yourself remembering something between calls, check whether the argument you want is already on its way.

Spec.GPUs at config-validation time is ONE BACKEND's cards, not the model's whole gpus: list — config.ModelSpec passes BackendGPUs(0). This matters if your engine has a rule about how many cards it can take. FreeToken's is "exactly one per process", expressed as len(s.GPUs) > 1, and it is correct precisely because of that. Read it as the whole list and the ordinary gpus: [0,1,2] with gpus_per_backend: 1 — three single-card backends, the shape that engine wants — would be rejected on day one.

The status code is often not the readiness rule. vLLM answers /health only when it is ready, so there its code is the rule. FreeToken answers 200 in every lifecycle state, including the minutes a frontier MoE model spends loading and any window a cache rebuild takes it out of service — both of which 503 every generation request. Porting vLLM's probe to it would advertise the backend to the mesh and have every routed request come back 503. Read your engine's readiness endpoint in each state before deciding what "ready" means, and report the phase you find so /v1/status can show it.

Absent is not zero, in Load as on the wire. A vLLM build that publishes no running-sequence gauge makes Load return an error. That is not pedantry: a zero would freeze "idle" into the mesh for a busy backend, while an error degrades visibly to the node's own in-flight count within routing.stale_after.

DecodeOptions error lines: look the key up in the original node. Rebasing with Options.Line - 1 works for a flat block and breaks on a nested or multi-document one. The real implementation does the lookup; do not reinvent the offset.

Your DefaultStartupTimeout is the single-card figure. vLLM's is 20 minutes, which is right for one card and short for a tensor-parallel backend that must also shard the checkpoint. viiwork-accept flags a multi-card backend whose timeout is under 45 minutes, and the fix is models[].startup_timeout in the operator's file rather than a bigger default — the engine cannot see the split from DefaultStartupTimeout(), which takes no Spec.

Check what a flag interacts with, not only that the binary accepts it — and then check who turns that interaction on. This section used to say the opposite of the truth, and it is worth keeping as the example rather than quietly correcting. --kv-reserve-tokens is a floor consulted only by --moe-cache-auto, and viiwork never generates --moe-cache-auto; from that it concluded the key was inert unless an operator added the flag to args:.

The engine turns it on itself. ft serve enables moe_cache_auto for any offload-family strategy — offload, cpu, hybrid — that was given no cache-sizing flag, which is precisely what moe_backend: auto resolves to on a MoE model. So the floor is live on every model this engine exists to run, and its 8192 default silently caps context. The fleet found this the hard way before the source was read: yeti's config carries the comment "experts eat the budget and KV landed at exactly 8192 tokens on a checkpoint declaring 262,144".

Tracing one hop — from the flag viiwork generates to the flag it interacts with — was right. Stopping there, at "we do not generate it", was the mistake: the engine is also a party that can set its own flags.

Blank imports used to be three places. internal/node/node.go, cmd/viiwork-accept/main.go and internal/accept/engines_test.go each kept their own copy of the engine list, and a copy that fell behind made acceptance validate configs for an engine it had not registered as though the engine did not exist. They now all import internal/engine/all, which is the only list.

A third port found one rule the kit stated too narrowly. Strata's server has no flag for the model name: it reads model_name and aliases from its own JSON config. "The backend must answer to s.Name" still holds, but the kit had checked it as "s.Name is on the command line". With Case.NameInConfig the kit checks the same rule the other way round: Command reads the config file, and the kit requires it to refuse a name the file does not carry. The same port expected to need a working directory on engine.Command and did not: PYTHONPATH in Command.Env was enough.

Read what the server does with the environment you gave it. The node pins a backend's cards before the child starts, and FreeToken's lesson was to add nothing on top. Strata's server cannot be left alone the same way: it builds its engine's device variables itself from a gpu list, overwriting CUDA_VISIBLE_DEVICES on NVIDIA and numbering inside the visible set on Radeon. So that engine passes --gpu — the real indexes on NVIDIA, 0..n-1 on Radeon — and this is what Spec.Vendor is for. "Pinning belongs to one layer" is still the rule; the engine's job was to stop the second layer from undoing the first.

An engine's run-time toolchain is your problem, not the node's. FreeToken JIT-compiles CUDA kernels on first use and shells out to ninja and nvcc by name. The node passes models[].env to the backend and nothing else, so on a native install those must be on the child's PATH or the backend starts its API server and then dies — which reads as a crash on first load rather than as a missing dependency. If your engine compiles anything at run time, say so in its package documentation and in the image, because the failure is not self-explanatory.

One Dockerfile trap. An ARG written after the first FROM is scoped to that stage. If your runtime stage's FROM interpolates a version ARG, declare it before any stage or it expands to empty and the build fails with invalid reference format.