working note · confidence: medium · 2026-09-24 · [04]

Anvil is the rack’s training control plane

The rack does not need another place to run models. It needs a place that decides which artifacts may land, which jobs may start, and what counts as evidence before anything moves toward live.

The house already has GPUs, shelves, and workflows. What it lacked was a control plane that treats admission and activation as first-class work, not a pile of one-off kubectl applies.

Anvil is that plane. It admits exact AI artifacts — models, workflows, bundles, runtime images — places them on approved targets, promotes only after evidence, and keeps rollback and an audit trail. It does not serve inference. It does not own the product voice path. Those jobs stay on the rest of the rack.

This is a living note. The UI will deepen. I will update the note, not republish the essay.

Why build it

A homelab that trains models without a control plane drifts into folklore. Someone remembers which Job template to copy. Someone else applies a fat image by tag and hopes the next restart will notice. Quarantine becomes a folder with no card that says signed or not. Promotion becomes a chat thread.

We wanted the opposite shape:

• Artifacts enter through a tray, not a storefront.
• Jobs come from recipes the git tree already owns.
• Digests are pinned after review.
• Unsigned never reaches Ready.
• Live routes stay stock until a separate, deliberate cutover.

Anvil is the bench for that discipline. Marketplace scrapes stay off the tray. Drag-drop is a file onto quarantine, not a shop. Final approval of a training artifact — adapter plus an MLflow receipt when we have one — sits on the same tray as the candidate weights. Training outputs stay quarantined until a separate eval and promotion step.

Experiments and Queue

Two lanes keep the work honest.

Experiments is the governed champion-versus-challenger workbench. You stage a reviewed challenger on its declared target. You run the fixed comparison pack against the active champion. You record outputs and latency. Then you require operator verdicts — and eligibility — before anything is treated as promotion-ready. That is where holdout checks and voice-review evidence belong. A green training Job is not a model score. An experiment that Completes is a receipt, not a flip of the live planner.

Queue is the durable operations lane for operator and agent work: Forge recipe runs, training submissions, evaluation jobs, and the like. Leases and exact bindings — not free-form commands, not remote artifact URLs. The signed-in operator confirms. The controller creates the Job. Serving stays untouched while the Job runs.

That split matters. Experiments answer “is this challenger better on the pack we care about?” Queue answers “run this exact recipe with these exact inputs and put outputs in quarantine.” Mixing the two turns every compare into an accidental deploy.

Recipes instead of one-off kubectl

Reviewed training recipes live as git-owned suspended templates in the isolated anvil-training namespace. The signed-in operator confirms inputs. Anvil’s controller creates a Job from that named template, mounts prepared read-only inputs and a quarantine output path, and never pauses serving Deployments.

That is deliberate. A one-off kubectl apply against a hand-edited Job is easy to lose and hard to audit. A recipe in git can be reviewed, digest-pinned, and re-run the same way next week. Flux still owns many of the underlying Job templates in the AI stack. Where that path exists, Anvil only passes approved pin ids and gate-recipe ids — not a second apply path that bypasses the tree.

Anvil does not get tokens that would let it submit Argo or Beaker work on its own. Fail notifications are enough. Green CI on a runtime image is not a signal to auto-activate a pin.

Digest pins and Reloader

Tags lie. Digests do not.

Deploys pin images by digest so the revision you reviewed is exactly what runs. Reloader restarts Anvil when fleet or recipe config changes so the live catalog and the image stay aligned. A stale pin plus a config reload is how Forge and recipe admission break — so we treat pin and reload as one discipline, not two afterthoughts.

Warm-pool work stays gated on a real cosigned digest: a placeholder digest means the pool stays suspended. Unsigned weights never enter the warm set. Weights that want to be warm live on local NVMe next to the GPUs; network shelf storage is for quarantine and copy-in, not for serving hot.

The rule that keeps us from lying to ourselves: unsigned never reaches Ready, and Ready is not the same as live. The live planner and the live voice aliases stay on stock pins until someone intentionally cutovers. MLflow is a receipt on the tray card — name, size, signed or not, optional run id — not a badge that means “ship it.”

Lab-only by design

Anvil sits behind the same dual-gate principle as the rest of the Homelab apps: identity on the way in, and the product stays on the lab network rather than on the public Notes face. Placeholder host: anvil.lab…. That is a principle, not a how-to. The public site keeps receipts. The control plane stays in the house.

Intentionally not in Anvil yet

We do not apply Flux from the Anvil UI. We do not mutate live serving aliases from the UI. We do not scrape marketplaces or allow arbitrary build-on-serve. We do not own voice or inference. Training outputs stay quarantined until separate eval and promotion. Flux still owns many ai-ml Job templates; Anvil only hands approved pin and gate-recipe ids where that path exists.

Those omissions are features. Every time we were tempted to put Start Training or a CI badge on the tray, the honest move was to leave the tray as final approval only and keep the train path in recipes and Jobs.

What we learned so far

Fail-closed beats clever chrome.

Evidence before activate. Stage the challenger. Run the fixed pack. Record a verdict and eligibility. Keep adapters in quarantine until the house is ready to admit them. A completed train Job with a better val loss than baseline is still an offline candidate until someone says yes on the tray.

Separate the product voice path. The computers listen, then they think, then they speak — that loop is its own note. Anvil admits and activates. It does not become another voice console.

Keep Flux as the writer for cluster desired state. Anvil should not grow an “apply this Flux change” button just because the operator is already looking at a pin.

What’s next

Laya RLCD and other challenge work belong in Experiments first: stage the challenger, fixed pack, operator verdict and eligibility, quarantine until promote. Warm-pool and more Anvil chrome stay behind cosign fail-closed and real digests. Pref-RL and companion candidates keep the same rule — not promotion-eligible until evidence says otherwise.

The product we are building is Anvil-signed, fail-closed experiment loops. The rack stays the rack. Anvil is the place that refuses to pretend a receipt is a cutover.

When a signed pin loads and the admission path is dull enough to trust, we will deepen this note. Until then, stock stays stock, and the tray stays a tray.