Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-0008: Tiered NixOS VM tests with pinned image digests

Status: accepted · Date: 2026-10-02 · Decision makers: James Hebden

Context and Problem Statement

The Rust unit suite mocks the cluster driver throughout, so it never boots the compiled binary, never parses a real config from disk, and never goes through Cli::parse() to dispatch(). Two real bugs hid in that gap: a before_hook whose resolved config was silently discarded, and a kubectl::Client that carried a kubeconfig but never set KUBECONFIG. The project needed end-to-end coverage that’s reproducible and doesn’t depend on a developer’s Docker daemon or on network access during the build.

Decision Drivers

  • Test the real binary through its real entry point
  • Reproducible results from nix build, with no network access in the Nix sandbox
  • A cheap tier that runs fast and a heavier tier that exercises a real cluster
  • Pinned inputs, so a floating image tag can’t change a test result

Considered Options

  1. NixOS VM tests (runNixOSTest) in two tiers, with the k3d images fetched as fixed-output derivations pinned by digest
  2. Docker Compose based integration tests in CI
  3. Shell scripts under integration-tests/ run by hand or in CI, as the Go project did
  4. Rely on the unit tests

Decision Outcome

Chosen option: “Two-tier NixOS VM tests with pinned digests”, because VM tests give a throwaway machine per run and fixed-output derivations let the Nix-sandboxed build load k3d images without reaching a registry. Option 3 exists for the Go binary and needs a Docker daemon and network.

Tier 1 (e2e-mock) boots a VM with only the xianyun binary and a cluster.driver: mock config, and drives version, config validate, config print, doctor, status, down, and destroy. Tier 2 (e2e-k3d) creates a real k3d cluster, checks it with kubectl get nodes, and tears it down. Checks are registered for x86_64-linux and aarch64-linux only.

Consequences

Good:

  • Both bugs above were found by writing these tests
  • Digest pinning makes Tier 2 reproducible

Bad:

  • The first two tiers didn’t run up with a deployer, which is why the unregistered deployer registry went unnoticed (GAP-001 in the spec; measured 2026-10-13). Scenario tests built on nix/tests/lib.nix now cover it (ADR-0019).
  • The repository has no CI job that runs cargo test, the VM checks, or cargo audit: .gitlab-ci.yml contains no cargo or nix invocation (it includes a shared standard-build template whose scanner coverage of Cargo.lock wasn’t checked) (TM-SUP-001)
  • Tier 2 needs KVM and substantial memory, and there’s no aarch64-darwin equivalent
  • The cluster-domain discovery Job still pulls busybox:1.36 by tag (TM-EXEC-004)

Neutral:

  • The macOS build host runs the unit suite under daemonless single-user Nix; see macOS

Implementation Location

  • nix/tests/e2e-mock.nix
  • nix/tests/e2e-k3d.nix
  • book/src/project/testing.md
  • flake.nix

More Information

  • Retroactively documented on 2026-10-13. The date is that of feat(nix): add NixOS VM tests; Tier 2 followed on 2026-10-12.
  • Related: NFR-001, NFR-007, SR-013.