Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-0016: Containerd mirror wiring for the registry cache, and Colima on containerd

Status: accepted · Date: 2026-10-13 · Decision makers: James Hebden

Context and Problem Statement

For the cluster to pull through the cache, its container runtime has to be told which registries to mirror. The runtime both drivers share is containerd, but they reach it differently: k3d hands a file to k3s, which turns it into containerd configuration, while Colima’s VM already runs its own containerd. The first design configured dockerd’s registry-mirrors, which only ever mirrors Docker Hub, so images from other registries bypassed the cache.

Decision Drivers

  • Mirror every upstream registry, not only Docker Hub
  • Use the one mechanism both drivers share
  • A driver shouldn’t depend on a host name that only some container setups provide
  • Configuration that Xianyun fully owns should be written, not merged

Considered Options

  1. Write containerd’s per-registry hosts.toml for both drivers, delivered the way each driver can accept it, and run Colima on containerd
  2. dockerd registry-mirrors in daemon.json
  3. k3s registries.yaml in Colima
  4. Run two containerd instances in the Colima VM

Decision Outcome

Chosen option: “containerd hosts.toml for both drivers, and Colima on containerd”.

  • k3d passes --registry-config <registries.yaml>, and k3s expands it into containerd configuration at node start.
  • Colima is always started with --runtime containerd. Xianyun writes /etc/containerd/config.toml (with config_path set to /etc/containerd/certs.d) and one /etc/containerd/certs.d/<upstream>/hosts.toml per upstream over SSH, then restarts containerd and k3s so they re-read it. The files are overwritten in full on every up, which is simpler than merging and avoids duplicate entries, because Xianyun owns that directory and the shipped config.toml has nothing else to preserve.
  • The two drivers reach the host cache by different names: Colima uses host.docker.internal and k3d uses host.k3d.internal. The shared code carries only the port. host.docker.internal is a Docker Desktop and Colima contract that rootless Docker doesn’t provide, so a shared name would let k3d pulls silently skip the cache.
  • Nothing is removed on destroy. The configuration lives inside the VM, and deleting the VM deletes it.

Option 2 shipped first and mirrored Docker Hub only. Option 3 doesn’t work on Colima because its k3s uses the VM’s external containerd, so registries.yaml is never rendered into it. Option 4 doubles the footprint for no gain.

Consequences

Good:

  • Every upstream in registry_mirror.upstreams is mirrored
  • Re-running up converges to the same configuration

Bad:

  • A Colima profile no longer exposes a Docker socket, so docker context use against it stops working for docker build and docker run. Use nerdctl or a separate Colima profile.
  • The k3d mirror configuration sets insecureSkipVerify: true for the endpoint (TM-REG-003)
  • The Colima path has unit tests against a mocked runner but no end-to-end test

Neutral:

  • Documented as shipped behaviour: the config reference says Xianyun always starts Colima on containerd

Implementation Location

  • crates/xianyun-cluster/src/driver/colima.rs
  • crates/xianyun-cluster/src/driver/k3d.rs
  • crates/xianyun-cli/src/commands/up.rs

More Information

  • Recorded on 2026-10-13 from decisions 9 through 12 in the Go project’s registry-cache design, checked against the Rust drivers.
  • Related: ADR-0007, ADR-0011.