ADR-0019: NixOS VM scenario tests for the end-to-end behaviour
Status: accepted · Date: 2026-10-14 · Decision makers: James Hebden
Context and Problem Statement
ADR-0008 set up two tiers of NixOS VM tests: one with the mock driver and one with a real k3d cluster and no deployers. That left the behaviour the Go project’s integration suites covered with nothing behind it: installing deployers, ordering them, a cluster without Traefik, CA re-signing, several environments on one host, and a registry cache. Those suites were removed with the Go tree (ADR-0017). The gap was real: with no test that ran up with a deployer, no deployer manager was registered in production and nothing noticed.
Decision Drivers
- Test the compiled binary, through its real entry point, against a real cluster
- Reproducible with
nix build, with no network in the sandbox - One behaviour per test, so a failure points at a cause
- Shared setup in one place rather than copied into every test
- A test must be able to fail for the right reason
Considered Options
- One NixOS VM test per scenario, sharing a library (
nix/tests/lib.nix) for the VM definition and the Python helpers - One large test that runs every scenario in one VM
- Shell suites like the Go project’s, run in CI against a Docker-in-Docker daemon
- Rust integration tests that start k3d from
cargo test
Decision Outcome
Chosen option: “One VM test per scenario with a shared library”. A single VM (option 2) would run for most of an hour, and a failure early on would hide every later scenario. Shell suites (option 3) need a Docker daemon and a network on the runner, which is what ADR-0008 set out to avoid. Rust tests that start k3d (option 4) can’t be hermetic.
Seven checks now exist (e2e-mock, e2e-k3d, and e2e-deploy, e2e-no-ingress, e2e-ca-drift, e2e-cells, e2e-registry-mirror). The five new ones use nix/tests/lib.nix: the VM (Docker, k3d, kubectl, helm, kustomize, openssl, jq), a prelude that loads the pinned images, and helpers that run xianyun and kubectl and report each command’s exit status, duration, and warnings. e2e-k3d predates the library and keeps its own copy. One more image, zot, is pinned in nix/images/zot.nix.
Workloads are ConfigMaps, because the VM can’t pull an application image. Each scenario checks its result with a tool other than Xianyun (kubectl, helm, openssl, docker inspect), so a bug in Xianyun’s own reporting can’t pass the test.
Consequences
Good:
- Five scenarios from the Go suites are covered again, and each ran to a pass against the final code on 2026-10-14
- Writing them found a default-logging bug and a 120-second discovery wait (GAP-010 and GAP-009)
- Controls are part of the tests:
e2e-no-ingressfirst shows that k3s does deploy Traefik by default, ande2e-registry-mirrorfirst shows that an uncached pull fails
Bad:
- Each test takes several minutes and needs KVM, so CI runs them manually and allowed to fail until a runner provides it
- The
builtin:dev-caandbuiltin:traefik-tlsstoreoverlays aren’t covered, since cert-manager’s CRDs and Traefik can’t be pulled - Two Go scenarios (the mirrord reloader, with and without a project file) have no test, because the edit-mode process supervisor stops after its preflight checks (GAP-002)
- The scenarios differ from the Go suites in places: no application serves a request, and managed zot isn’t exercised
- The shared library is a second place that has to track
e2e-k3d’s setup until that test is moved onto it
Neutral:
- Only
x86_64-linuxwas run. Theaarch64-linuxvariants evaluate but weren’t built.
Implementation Location
nix/tests/lib.nixnix/tests/e2e-deploy.nixnix/tests/e2e-no-ingress.nixnix/tests/e2e-ca-drift.nixnix/tests/e2e-cells.nixnix/tests/e2e-registry-mirror.nixnix/images/zot.nix