Every v* tag on the public repository gets release archives for linux/amd64, linux/arm64 and darwin/arm64:

viiwork_vX.Y.Z_linux_amd64.tar.gz     viiwork, viiwork-accept, LICENSE
viiwork_vX.Y.Z_linux_arm64.tar.gz
viiwork_vX.Y.Z_darwin_arm64.tar.gz
SHA256SUMS                            sha256 of each archive
SHA256SUMS.sig                        ed25519 signature over the version and SHA256SUMS

CI builds the archives and uploads them with an unsigned SHA256SUMS. The publisher then rebuilds the same version from their own checkout, checks that every file matches CI's byte for byte, and only then signs. Nothing downloaded is ever run on the machine that holds the key, so neither a compromised workflow nor anyone who can move the public tag can get other code signed. The signature covers the version, and only a SHA256SUMS listing exactly that version's three archives is signed, so a signature can never be replayed for another release. CI never holds the key, so a compromised workflow can publish archives but cannot get them signed. A release without SHA256SUMS.sig is not installable by viiwork update: a node refuses to stage it.

Builds are reproducible

Every binary is built by scripts/gobuild.sh — make, CI and the local rebuild alike — with CGO_ENABLED=0, -trimpath, -buildvcs=false and only -X main.version in -ldflags. scripts/build-release.sh pins the compiler to go.mod's exact version (GOTOOLCHAIN) and writes deterministic archives. bash scripts/build-release_test.sh builds the committed tree twice from two directories and checks that every byte agrees.

Publishing a release

  1. Cut the release as usual (CHANGELOG.md heading, scripts/publish.sh --tag vX.Y.Z …). The tag push starts the release workflow on the public repository.

  2. When the workflow has finished, on the publisher's machine:

    scripts/release-sign.sh vX.Y.Z [COMMIT]

    Run it in the private repository at the commit that was published (HEAD by default). It downloads the archives, rebuilds that commit, runs viiwork-release compare on every archive, signs SHA256SUMS for the version, checks the signature against that commit's compiled-in keys, and attaches SHA256SUMS.sig. bash scripts/release-sign_test.sh exercises it end to end against a stand-in gh.

    The workflow refuses to re-upload archives once a release carries a signature, and only its publish job, which runs nothing but gh, holds write access.

  3. release-sign.sh then publishes the engine images, from the same local build it just compared: ghcr.io/janit/viiwork-llamacpp-cuda (amd64, arm64), viiwork-llamacpp-vulkan and viiwork-vllm, each tagged with the version. Each is the upstream engine image (pins in docker/pins.env) with viiwork appended as one layer, streamed registry-to-registry by crane. Its gh token needs the write:packages scope (gh auth refresh -s write:packages). A new package on ghcr is private until made public once in its settings. gfx906 (Radeon VII) has no published image: make docker builds it. SKIP_IMAGES=1 signs without publishing. Every image records its base by digest (org.opencontainers.image.base.digest), and a released tag is written once. If a push fails after the signature is attached, IMAGES_ONLY=1 scripts/release-sign.sh vX.Y.Z rebuilds, compares, verifies the existing signature and publishes only the images. The registry login lives in the run's temporary directory and goes with it.

    The base images keep their own HEALTHCHECK (llama.cpp's probes :8080); a compose file sets its own.

viiwork-parrot follows the releases feed and seeds each release as a torrent once its signature is attached.

The signing key

go run ./cmd/viiwork-release keygen makes the pair once, on the publisher's machine: the private key at ~/.config/viiwork/release.key (mode 0600; sign refuses a key others can read) and the public key at internal/release/keys/release.pub, which is committed and compiled into every build. keygen refuses to overwrite an existing key.

Back the private key up offline. Losing it, or suspecting it has leaked, means a rotation: make a new pair under a new file name (--pub internal/release/keys/<year>.pub), ship a release carrying both public keys, sign later releases with the new key, and remove the old public key once no node runs a version that lacks the new one.

Checking a download by hand

The checksum proves the archive is the one listed in SHA256SUMS. The signature proves SHA256SUMS is the publisher's, for exactly that version. Checking the signature needs a checkout and Go 1.27.2:

curl -fLO "$base/SHA256SUMS.sig"                  # beside the archive and SHA256SUMS
sha256sum --check --ignore-missing SHA256SUMS      # on a Mac: shasum -a 256 --check --ignore-missing SHA256SUMS
git clone --branch vX.Y.Z https://github.com/janit/viiwork viiwork-src
(cd viiwork-src && go run ./cmd/viiwork-release verify --version vX.Y.Z ../SHA256SUMS ../SHA256SUMS.sig)
# prints "signature ok"

How a node updates itself

A node takes part only with update.enabled: true. Everything lives under <state_dir>/releases/: state.json (current, last_good, pending, the recorded launcher), previous (the release that was last good before last_good) and one directory per staged version. previous is a file of its own because a launcher from v2.6.0-beta4 refuses any field in state.json it does not know.

  • Stage (POST /v1/update/stage {"version": "vX.Y.Z"}): the node downloads SHA256SUMS, its signature and this host's archive from https://github.com/janit/viiwork/releases/download/vX.Y.Z/ (the only update.source a node accepts; a redirect is followed only to GitHub's asset hosts), verifies the signature for that version against the compiled-in keys, checks the archive, unpacks viiwork and viiwork-accept, runs the new binary with --version (it must run here and report that version) and --engine-requirements (this host's engines must be new enough). Anything that fails leaves nothing behind.

    SHA256SUMS and its signature always come from GitHub when GitHub answers. The archive comes from the node's viiwork-parrot (viiwork_parrot.api, POST /ensure-release with the other mesh members as peer hints), checked against GitHub's sums; when parrot is absent, refuses, shows no progress for 2 minutes, has not finished after 5, or hands over an archive that does not match, the node logs why and downloads the archive from GitHub (a parrot answering 503, which has no feed yet, is waited out within those limits). The host's engine helper on a Docker install stages through the same parrot. Only while GitHub is unreachable — no connection, a 30 s timeout, a connection dropped mid-answer, 5xx, 429 — does the node take parrot's copy of the signed files, verified against the same keys. A definite answer from GitHub (404, 403, a bad signature) is final, and so is any other answer: a redirect off GitHub, or signed files over their size cap.

    On a Mac set up by viiwork init (its install.json names a llama root), staging first reads the new binary's --build-info. When it carries a llama.cpp pin with its macOS digest, the node fetches that build into the llama root from the release's fixed URL, with no GitHub API call, and requires the digest to match; the engine requirement is then checked against that build, which is the one the release will run.

  • The Mac's engine follows the binary. A model whose llamacpp.binary is exactly <llama root>/<tag>/llama-<tag>/llama-server runs the build of the pin the running viiwork carries, when that build is present, and the configured one otherwise. Activating a release therefore moves the engine with it, and a rollback moves it back, with nothing recorded. A binary anywhere else is used as written. When a release confirms, builds under the llama root that no kept release runs (current, last good, previous, the floor, this process, and any tag the config names) are removed; if any of those cannot be read, nothing is.

  • Activate (POST /v1/update/activate): refused while any model is still loading. Otherwise it records the backends healthy right now as the baseline, marks the version pending, answers 202, shuts down in the usual order and execs the launcher, which runs the staged release in the same process.

  • Confirm: the new release becomes last good once every baseline backend is healthy again. A baseline backend going dead, or the window passing (update.confirm_timeout, else the sum of the models' startup timeouts plus 10 minutes, fetching time excluded), rolls it back. A release that crashes before it can confirm is started twice; the third start runs last good. Every start counts, a routine restart during the window included.

  • Rollback (POST /v1/update/rollback): back to last good at once. On a host already on its last good release, back to the release that was last good before it (previous in GET /v1/update), which becomes last good again; refused when there is none, or its binary no longer verifies. It is remembered for that one last good release only, so after a newer image resets the node to its builtin binary there is nothing to go back to.

  • A newer image or binary installed out of band always wins over an older staged release. A floor that is the pending release (the same version, exactly) runs it and keeps it pending, so it is still confirmed, not forgotten. A floor the node installed itself by following a confirmed release (a Mac, below) is not out of band: it is recorded in releases/cli.json with its sha256, and while the launcher is exactly that file the state still decides, so a rollback below it runs.

Writes need a meshauth signature in a secured mesh and come only from loopback in an open one, exactly like alias writes. GET /v1/update reports the state, the staged versions and the installed engine versions.

The host's CLI follows the release

viiwork update moves the node; the viiwork on the host's PATH is a separate file, and an old one lacks newer commands and asks for older confirmation phrases. Since v2.7.2 it follows the release, only ever forward (a CLI at or past the release, or a developer's build, is left alone), and always by one rename in its own directory, with the CLI it replaced kept as <path>.prev (overwritten each time):

  • Docker install made by viiwork init: the engine helper does it (below), as root, from its own verification, to the path install.json records as the install's binary (/usr/local/bin/viiwork) and nowhere else.
  • Mac install made by viiwork init: the CLI ~/.local/bin/viiwork is the LaunchAgent's binary, the launcher that hands over to a staged release. The node runs as the user who owns it, so when a release confirms, the node installs that staged release there itself (the binary it was handed over to, re-checked against the sha256 recorded when it was staged), records it in releases/cli.json and moves the launcher's recorded sha256 with it. Without that record the next start would treat the new launcher as an out-of-band upgrade and forget the staged releases, rollback included.
  • Anything else (a hand-built node, a Docker host not made by viiwork init, an install whose manifest records no binary): viiwork update cli by hand, below.

viiwork update cli (with sudo where the CLI is root's) asks the node on loopback which release it runs (GET /v1/update, running), takes that release's viiwork from <state_dir>/releases/<version>/, and verifies it as staging does: the signed SHA256SUMS from GitHub, the archive from GitHub or the host's viiwork-parrot, staged afresh beside the CLI with this binary's keys, and the node's copy must be byte for byte that release's viiwork. It then replaces the running CLI (links resolved), or --to PATH, keeping <path>.prev. --dry-run says what it would do; --state-dir names the state directory when the config (--config, else this machine's) does not. It refuses when it cannot write the CLI's directory, when the release is not in the state directory, or when anything fails to verify. On a host whose CLI is also the node's own binary (a hand-built systemd node started from it), the next start treats the new file like any newer binary installed out of band.

A CLI must itself be v2.7.2 or newer to have the command. To bring an older one up once, run a v2.7.2-or-newer viiwork from a release archive checked by hand (above) as sudo ./viiwork update cli --to <the old CLI> — never the copy in the state directory, which a container can write and which would run as root. A Mac needs nothing: the node installs its CLI when v2.7.2 itself confirms. A Docker install made by viiwork init with the engine helper needs nothing either.

On a Docker install: the image follows the release

A container cannot replace its own image, so on a Docker install made by viiwork init a small helper on the host does it: viiwork engine-sync, run as root from /usr/local/bin/viiwork by viiwork-engine.path (when the node's state.json or image request changes) and viiwork-engine.timer (every five minutes). Its one rule: the compose file's image tag is the node's current release, or for builtin the tag the install had when the helper first ran, which it records in its own root-only directory, /var/lib/viiwork-engine/sync.json. There the release is the image, engine included:

  • Stage also asks the helper, through releases/engine-request.json (a version and a request id, nothing else), to verify and pull the release's image. The helper answers in releases/engine-result.json, with the engine version the image carries, which the release's engine requirement is checked against. A bad image fails the stage, before any host restarts, and activation never waits on a pull.
  • Activate is refused (409) until the helper has prepared that exact version's image: stage it first. Activate and rollback then save the state and answer 202, and never exec a staged binary: the helper swaps the image and docker compose up -d restarts the node through its ordinary shutdown. The new container counts its starts of the pending release and the confirmer decides it as usual; a rollback, the confirmer's included, sets current back and the helper swaps the previous image in (still local, so nothing is pulled). If the node is still running two minutes after an activate or rollback — no helper, or a broken one — it restores the state it had and logs why.
  • Swap. The helper verifies the release itself, with the keys compiled into the host binary and the same staging code a node uses, into /var/lib/viiwork-engine/releases/; pulls <repo>:<version>; reads the image's /usr/local/bin/viiwork out of a container it creates and never starts; and swaps only when that file is byte for byte the signed archive's. It then rewrites the compose file's image: line (the file as it was is kept as docker-compose.yaml.bak) and runs docker compose up -d. The engine layers under viiwork are trusted as at install time: by registry and tag.
  • Only forward, or back to what ran here. The helper moves to a version older than the running tag only when it is the install's own tag or one of the last eight it swapped to itself (kept in sync.json); anything else is refused and logged, pre-pulls included. So a node that writes an older release into state.json cannot skip /v1/update's consent for a downgrade; a rollback goes back to a release this host has run.
  • Disk. An image already on the host is used, not pulled again. The helper removes every image it pulled itself once nothing needs it, keeping the running tag, the install's own, the last two it ran and the one prepared for the latest stage; an image it did not pull is never removed.
  • Itself, and the host's CLI. Once the node has confirmed the release its image carries, the helper installs its own verified copy of that release's binary at the path install.json records as the install's binary (/usr/local/bin/viiwork, which is also the helper), so the host's CLI and the helper's own rules move with the release. It writes nowhere else, and nothing when the manifest records no binary. It skips a CLI whose --version already reports the release or a newer one, never takes the node's staged copy (the container can write it), replaces the file by rename and keeps the old one as viiwork.prev, and logs what it did. A release that fails to verify leaves the CLI and its .prev as they were. (Before v2.7.2 it wrote /usr/local/bin/viiwork unconditionally, compared its own version rather than the file's, and kept viiwork.bak.)

What it trusts. The helper runs as root, and the node's releases directory is the container's to write. From it the helper takes state.json's version names and an image request's version and id, each validated as exactly that, and nothing else: never a path, a repository or a URL. It reads them through an os.Root on the state directory, refuses a link anywhere there, reads only small regular files without blocking, and replaces its result file by rename, so a planted link can make it neither read nor write elsewhere. The repository comes from the host's compose file (and must be one of the published ghcr.io/janit/viiwork-* images), the download root is GitHub's (the only update.source a config may name), and where things are from the host's install.json. A compromised node can still ask for any signed release, newer than the running one, or one this host has already run.

A node knows it is on such an install from the engine_helper entry in the install.json its container sees in /etc/viiwork. Anything else — a hand-built node, or an install from before v2.6.1 until sudo viiwork init adds the helper — updates its viiwork binary only, as before: the engine stays as installed, and staging refuses a release whose engine requirement the host does not meet. journalctl -u viiwork-engine.service shows what the helper did and why.

Rolling a release across the mesh

viiwork update status                        # every member: running, current, last good, previous, pending, engines
viiwork update                               # the latest signed release, everywhere it is enabled
viiwork update --to v2.6.0 --hosts node-a,node-b   # one version, some hosts
viiwork update --parallel 3                  # activate up to three hosts at a time
viiwork update rollback --host node-b              # one host back a release

After a host confirms, the rollout also waits until the node it talks to sees that host alive on the new release, so a release that breaks mesh membership stops the rollout at the first host.

A rollout lists its plan — every member with what will happen to it, or why it is skipped (not alive, updates off, already there, newer, older than rolling updates) — and asks for YES I WANT TO UPDATE THE NODES ABOVE (or --confirm). It then stages the release on every chosen host — all at once, at most eight together, since staging changes nothing a host runs — so a download, signature or engine problem stops everything before any host restarts, and activates them one at a time, the node it talks to last. Each host must come back on the new release and confirm it before the next starts; a host that rolls itself back, or misses its own confirmation deadline, stops the rollout with the rest untouched.

--parallel N activates in waves of up to N hosts instead, and waits for every host of a wave before the next. The node the CLI talks to is still last and alone. No wave holds every alive host that lists a model in its status, counting hosts outside the rollout, so a model served by two or more machines never goes offline; a model on one machine is offline while it restarts, as it is one at a time. The plan shows the waves before the phrase. If any host of a wave fails, the CLI still follows the rest of that wave, reports where each host ended up, and starts no later wave. In a secured mesh, run it where the mesh secret is loaded. On a Linux node installed by viiwork init, mesh.env is readable by root only: sudo sh -c 'set -a; . /etc/viiwork/mesh.env; /usr/local/bin/viiwork update'. On a Mac installed by viiwork init, the CLI reads the secret from the node's LaunchAgent. Without a secret the CLI says so before asking for anything. An open mesh accepts update writes only from each node's own machine, so there a rollout updates only the node the CLI talks to — run it on that machine against 127.0.0.1 — and lists the others as skipped. A version below a host's installed binary is skipped too: the node's newer-binary rule would undo it at the next start. Names in --hosts that are not members are refused.