Testing

The unit suite

nix develop -c cargo test --workspace runs every crate’s tests. Measured on Linux on 2026-10-14: 1,119 passed, 0 failed, 23 ignored, out of 1,142. Of the ignored tests, most need a live Docker daemon or Kubernetes cluster, a few need root and the real /etc/hosts, and one (compile_schema_lazy_loading in xianyun-core) is an unimplemented placeholder. Dependency scanning is nix develop -c cargo audit; the last run found no vulnerabilities and four warnings, recorded in the threat model.

NixOS VM tests

The unit-test suite mocks the cluster driver for most of its 1,100+ tests, so it never actually boots the compiled xianyun binary, parses a real xianyun.yaml from disk, or exercises the real Cli::parse() -> dispatch() path. This file documents the NixOS VM tests (pkgs.testers.runNixOSTest) that close that gap: they build the real binary, boot a throwaway NixOS virtual machine with QEMU, and drive the binary through its real command-line interface inside it.

They’re registered as checks.<system> flake outputs, on x86_64-linux and aarch64-linux only. NixOS VM tests require Linux (they need the Linux KVM/QEMU path); there’s no equivalent for aarch64-darwin, so that system has no checks entry at all.

Running them

$ nix build .#checks.x86_64-linux.e2e-mock -L

-L streams the full VM boot log and test-script output, which is worth having the first time something fails. Swap in .#checks.aarch64-linux.e2e-mock on an aarch64 Linux host; this repo’s own development machine is x86_64-linux, so that output is evaluated and built on CI/aarch64 hosts, not exercised here (see “What’s verified vs. evaluated” below).

nix flake check runs every checks.<system>.* output for every system in flake.nix’s systems list, which includes aarch64-linux – on an x86_64-linux machine that means either a slow QEMU-emulated cross-build or skipping it. Prefer building the specific checks.<system>.<name> output for the system you’re actually on.

Tier 1: e2e-mock (nix/tests/e2e-mock.nix)

A NixOS VM containing only the compiled xianyun binary – no Docker, no k3d, no kubectl, nothing beyond what a minimal NixOS closure already has. The test script:

1. Writes a real xianyun.yaml to disk with cluster: { driver: mock }.
2. Runs xianyun version and xianyun config validate / xianyun config print against that file, asserting each exits 0 and that config print actually echoes the configured mock driver back – this exercises the real (non-mocked) YAML parser and config-file-discovery walk, not a hand-built Context the way the unit tests do.
3. Runs xianyun doctor, asserting only that it doesn’t panic (this VM deliberately has none of helm/kustomize/git/k3d/docker installed, so a non-zero exit reporting missing tools is expected and fine).
4. Runs xianyun status, xianyun down, and xianyun destroy against a cluster that was never started, asserting each exits 0 – confirming the mock driver’s idempotent lifecycle calls work when invoked through the real binary’s dispatch path, not just through a run_status/run_stop/run_destroy call built in a Rust test.
5. Greps every command’s combined output for panicked at, failing the test if any command panicked regardless of its exit code.

Every assertion in this test runs in well under a minute once the VM boots (VM boot + test script: under 30 seconds on this machine, plus whatever time nix build spends compiling the binary the first time; rebuilds with a warm cargo/nixpkgs cache are much faster).

Why xianyun up isn’t in Tier 1

cluster.driver: mock only fakes the cluster lifecycle calls (start/stop/destroy/status) – up::run_start (the real up handler) still builds a genuine kubectl client pointed at whatever kubeconfig the driver reports, in order to wait for the cluster API to answer, regardless of which driver is configured. The mock driver’s get_kubeconfig() returns a fake value with no API server behind it, so a real xianyun up invocation needs either a live Kubernetes API (Tier 2) or a kubectl binary that will reliably fail fast rather than hang the test retrying for up’s 180-second cluster-ready timeout. This was confirmed by running the test against the real binary rather than assumed – see “A regression this test already caught” below.

A regression this test already caught

Writing this test immediately surfaced a real bug: commands::dispatch resolved the config file path via hooks::before_hook but then discarded the result, and every subcommand’s public run() wrapper built a fresh, empty Context::default() instead of using it. Every unit test for up/down/destroy/status/ca/config/etc. calls the inner run_start/run_stop/run_destroy/… function directly with a manually constructed Context, bypassing run() entirely, so the whole suite stayed green while the real binary failed every config-reading subcommand with configuration file not found: no global config in context. This is exactly the class of regression a real, disk-and-binary VM test catches and a mocked-driver unit test cannot. Fixed in the same change that added this test: dispatch() now builds the Context from before_hook’s resolved GlobalConfig once and threads it into every subcommand handler that needs it.

Tier 2: e2e-k3d (nix/tests/e2e-k3d.nix) – real k3d-backed lifecycle

A NixOS VM with virtualisation.docker.enable = true plus k3d/kubectl/docker-client installed, running xianyun up against cluster.driver: k3d to create a genuine k3d/k3s cluster inside the VM’s Docker-in-VM, verifying it’s actually ready with a real kubectl get nodes (not through xianyun’s own reporting), then xianyun down / xianyun destroy to tear it down and confirming no k3d- containers or clusters are left behind.

A previous session investigated this and deliberately didn’t ship it (see below for why that call made sense at the time); this session closed the gap it identified.

The sandbox problem, and the fix

k3d cluster create needs container images that aren’t part of the base NixOS closure, and Nix’s build sandbox has no network access by design – a nixosTest VM’s build still runs inside that sandbox, so pulling any of them over the network isn’t possible. Four images turned out to be needed, found empirically (not guessed from memory) by running this test and reading exactly which pull failed next each time:

Each is pre-fetched as its own pkgs.dockerTools.pullImage fixed-output derivation (nix/images/k3s.nix, nix/images/k3d-proxy.nix, nix/images/k3d-tools.nix, nix/images/pause.nix). FODs are allowed network access during their own build because their output is verified against a pinned content hash afterwards – this is the standard, sandbox-compatible way nixpkgs gets any network-sourced content (a tarball, a container image, a source archive with no reproducible build step of its own) into a build without breaking reproducibility. Each derivation’s imageDigest is the image’s manifest digest (from skopeo inspect --raw docker://<image>:<tag>, picking the amd64/ linux platform entry from the manifest list/index); its hash is the hash of the pulled tarball itself, found by building the derivation once with a wrong hash and reading the mismatch Nix reports.

The first three images back containers the VM’s outer Docker daemon creates directly (the k3s node, the serverlb, the tools container), so this test’s build loads them with docker load before xianyun up ever runs k3d cluster create, and pins cluster.k3d.image in the generated xianyun.yaml to the exact k3s tag that was pre-loaded – k3d never has a reason to reach a registry. The pause image is different: it’s pulled by the k3s node’s inner embedded containerd, a separate image store docker load on the outer VM can’t reach. The test script instead runs k3d image import <pause-tarball> -c xianyun once the cluster exists (right after xianyun up returns), which pushes the tarball through the cluster’s shared image volume into the node’s containerd; kubelet’s own retry backoff picks it up and system pods recover shortly after. Nix shares its host store read-only into the VM, so the test script references the FOD store paths directly – the same idiom nixpkgs’ own nixos/tests/docker-tools.nix uses for pullImage examples; no copying needed.

A second bug this test caught: kubectl::Client ignored its own kubeconfig

Pinning every image got k3d cluster create to succeed, but xianyun up then failed with “the connection to the server localhost:8080 was refused” – kubectl’s hardcoded fallback when it can’t find a kubeconfig anywhere. xianyun-cluster’s k3d driver creates clusters with --kubeconfig-update-default=false (so it never writes to ~/.kube/config), and kubectl::Client (crates/xianyun-backends/src/ kubectl/mod.rs) carried a kubeconfig field from with_config but, per its own doc comment, never acted on it – a known, documented gap (see crates/xianyun-backends/PORT_NOTES.md), but one that made up against any real driver (k3d, colima) non-functional whenever the cluster’s kubeconfig differs from the ambient default, which is the common case. Fixed in the same change that shipped this test: kubectl::Client now materializes its kubeconfig into a temp file and points KUBECONFIG at it (mirroring k3d::Driver’s disposable_kubeconfig_opts) on every command it runs – read_stdout, run_manifest_command, and get_version’s retry loop. New unit tests (*_sets_kubeconfig_env_when_with_config / *_sets_no_env_override_when_plain_new in kubectl::mod::tests) cover the behavior with a dedicated EnvCapturingRunner, since xianyun_testutil::MockCommandRunner doesn’t inspect Options::env. This is exactly the class of regression e2e-mock’s own doc comment already called out: a bug only a real binary, driving a real cluster through its real CLI, can catch.

Running it, resource requirements, and timing

$ nix build .#checks.x86_64-linux.e2e-k3d -L

Observed end to end on this machine (default Nix sandbox, no --option sandbox relaxed, no sandbox = false anywhere): VM boot, docker load of the three outer images, xianyun up (k3d cluster create plus the cluster-API-ready wait), k3d image import of the pause image, xianyun status, a real kubectl get nodes showing the node Ready, xianyun down, xianyun destroy, and a final check that no k3d- containers or clusters remain – test script: ~220 seconds (plus whatever nix build spends compiling the binary and building the VM closure the first time; rebuilds with a warm cargo/nixpkgs cache are faster). virtualisation.memorySize = 4096; virtualisation.cores = 4; was sufficient for k3s + containerd plus the four preloaded images on this machine’s hardware; virtualisation. diskSize = 8192 (MiB) gives the preloaded images and k3s’s own container layers headroom. If a future k3d/k3s version bump needs more, bump these first before suspecting anything else.

Updating the pin

When bumping this flake’s nixpkgs (which can move k3d’s version), re-run nix develop -c k3d --version to get the new default k3s tag, then re-derive each image’s digest with skopeo inspect --raw docker://<image>:<tag> (picking the amd64/linux manifest from the list/index) and rebuild each nix/images/*.nix derivation once with a deliberately wrong hash to read the correct one from the mismatch Nix reports – the same process used to pin them the first time, documented in each file’s own doc comment.

Scenario tests

Five more NixOS VM tests run the compiled binary against a real k3d cluster for one behaviour each. They share nix/tests/lib.nix (the VM definition and the Python helpers) and are described in ADR-0019. They replace the Go project’s integration suites; the specification lists which suite each one succeeds and how it differs.

$ nix build .#checks.x86_64-linux.e2e-deploy -L

These need /dev/kvm, a few GiB of RAM each, and several minutes. Run them one at a time. The times above are from one run on a 16-core, 62 GiB Linux host, not benchmarks.

Every image a scenario needs is a fixed-output derivation under nix/images/, because the Nix sandbox can’t pull. That is why the workloads are ConfigMaps: a deployment that needs a new image can’t start. It also means these tests can’t cover the builtin:dev-ca and builtin:traefik-tlsstore overlays, which need cert-manager’s CRDs and a Traefik.

What writing them found

• Default logging was broken. A command run with no logging flags logged nothing, because LoggingArgs::default() held empty strings and the resulting configuration failed to parse, with the error discarded. The e2e-deploy ordering check couldn’t see any log line, which is how it surfaced. Fixed, with unit tests, and an invalid logging flag is now an error (GAP-010).
• Domain discovery waits two minutes on a cluster that can’t pull. up, deploy, and ca inspect each run a busybox:1.36 probe first and wait out its 120-second timeout when it can’t start (GAP-009). Not fixed: the order was ported from the Go tool.
• No deployer was registered in production (GAP-001), which is the reason the deploy scenario exists.

Two mistakes in the tests themselves are worth knowing about. The NixOS test driver lints and type-checks the Python script, so an unused import or a variable named log fails the build before the VM boots. And a control that “fails” proves nothing unless you check why: the first registry-mirror control failed because k3s crictl doesn’t exist, not because the pull failed.

What’s verified vs. evaluated

• checks.x86_64-linux.e2e-mock: built and run on this machine (the development host is x86_64-linux); passes.
• checks.x86_64-linux.e2e-k3d: built and run on this machine; passes under the default Nix sandbox. See the Tier 2 section above for the full run.
• checks.x86_64-linux.e2e-deploy, e2e-no-ingress, e2e-ca-drift, e2e-cells, e2e-registry-mirror: each built and run to a pass on this machine on 2026-10-14 against the final code.
• checks.aarch64-linux.*: the same derivations, parameterized by pkgs.testers.runNixOSTest’s target system. nix flake show and nix eval confirm they evaluate to valid derivations; neither has been built or run here, since that would require emulated cross-compilation of the entire Rust workspace plus an emulated QEMU-in-QEMU VM boot, which is impractically slow on this machine. Build them on a native aarch64-linux host (or a CI runner with one) to actually execute them – e2e-k3d’s four pinned images are amd64-only manifest digests (see the table above), so that host will also need arm64 digests re-derived the same way before its e2e-k3d can pass; e2e-mock has no such dependency.