Documentation review

This is the report from the all-aboard review of 2026-10-13, updated the same day after the documentation moved into this book. The first section says what changed since the first version.

Follow-up: the Xianyun book

Three decisions followed the first report:

1. The documentation names xianyun and leads with MCP. The new pages describe the xianyun binary and put the MCP server first: connecting an agent, how the server works, the tool reference, confirmation, workflows, and security. The terminal comes second.
2. A separate site built with mdBook and rustdoc. Recorded in ADR-0010. The Hugo site and the docs/ directory are left for the Go binary. The ADRs, specification, stories, threat model, and this report moved into book/src/project/, so they no longer go through Hugo and the links between them resolve.
3. The work went ahead. What changed:
   • 48 pages in book/src/ (23 outside project/), including a command reference generated from xianyun --help by scripts/gen-cli-reference.sh, with a --check mode that fails when it drifts.
   • scripts/build-docs.sh, nix build .#docs, and nix build .#xianyun-mcp. The dev shell now has mdbook and mdbook-linkcheck2.
   • rustdoc builds clean under -D warnings after fixing 20 warnings, mostly links from public docs to private items.
   • The repository linter covers book/src.
   • book/STYLE_GUIDE.md records the accessibility, theme, and voice decisions.

The pages were written from behaviour I ran, not from memory. Measured while writing them: the MCP handshake and tool calls over stdio against a mock-driver project; that a gated tool fails on a client at protocol 2025-11-25; the exact CA name constraints for a sample config ([.local .test .svc]); config validation errors; variable expansion with and without an exported variable; and the xianyun version, ca inspect, and --help output. Doing so found three things I’d gotten wrong in the spec, now fixed: the CA’s permitted names (the defaults are .local, .test, .svc, not a subtree of your hostname), a claim that up merges the kubeconfig (it fetches it; set-kube-context merges), and config validate examples that omitted the required cluster.network block. It also added GAP-008, user-facing strings that still say Caproni.

Still unchecked: the book wasn’t read in a browser here (screenshots failed in this sandbox), so layout is verified from the generated HTML only, and I didn’t test a real MCP client end to end beyond a scripted stdio session.

Follow-up: GAP-001, the end-to-end scenarios, and removing telemetry

The headline finding of the first review is closed. Every statement below was measured on 2026-10-14 unless it says otherwise.

GAP-001 is fixed. The three deployer managers now implement the deployer::Manager trait and are registered in xianyun-cli. Helm also had no undeploy, which the trait needs; it was added, test first. The e2e-deploy NixOS VM test runs xianyun up on a real k3d cluster and installs a namespace, a Kustomize overlay, and a Helm chart in dependency order. It also repeats the deploy (the Helm upgrade path), checks that --repo deploys only the named deployer, and checks that a broken chart fails the run and is named.

Five of the seven lost scenarios are back as VM tests (e2e-deploy, e2e-no-ingress, e2e-ca-drift, e2e-cells, e2e-registry-mirror), each run to a pass against the final code. The two that need edit mode are blocked by GAP-002. ADR-0019 records the design, and Testing lists what each checks and how long it takes.

Telemetry is removed (ADR-0018): the module, its environment variables, and 29 tests, with the GitLab-employee heuristic going with it.

Found by writing the tests, not by reading code:

• A command run with no logging flags logged nothing. LoggingArgs::default() held empty strings, the logger failed to build, and dispatch discarded the error. Fixed, with unit tests (GAP-010).
• up, deploy, and ca inspect each wait about two minutes on a cluster that can’t pull busybox:1.36, because discovery tries that probe first (GAP-009, not fixed: the order was ported from the Go tool).
• Two of my own test mistakes, recorded in Testing: a control that failed for the wrong reason, and a test script the driver rejected before the VM booted.

Still open: edit mode (GAP-002, GAP-006), doctor’s missing diagnostics (GAP-003), --no-cache (GAP-005), the /etc/hosts check in up (GAP-007), the Caproni strings (GAP-008), the discovery wait (GAP-009), and the 12 High threats, none of which is mitigated. Not covered by any scenario: the builtin:dev-ca and builtin:traefik-tlsstore overlays, managed zot, and the Colima driver. Several threat entries that I marked “latent until GAP-001 is fixed” are now reachable, and the entries say so; I haven’t re-measured them through a real up.

A flaky test, found by the first CI run. registry_mirror_config_active_test in the Colima driver failed on GitLab’s runner and had never failed locally. Driver code reads SUDO_USER, several tests set it, and only the tests that set it took the environment lock, so a reader could run while it was set and take the wrong code path. Reproduced deterministically (the same panic), then fixed by having every test that builds a driver take the lock, with a regression test. 100 runs of the file passed afterwards. The same first run showed the pages job failing on my own ls | head under pipefail; fixed in the next commit.

Test counts: 1,119 passed, 0 failed, 23 ignored (1,142 in all). It was 1,137 before: 29 tests left with the telemetry code, and 11 were added. Fmt, clippy with warnings denied, and cargo deny pass.

Follow-up: removing the Go tree, and publishing

The project is meant to be public, so the Go implementation and everything that described it were removed (ADR-0017). The sections below this one describe the repository as it was when the review ran, including docs/ and the Hugo site, which no longer exist. Treat them as a record, and the remaining-gaps table as superseded by this section.

Removed: cmd/, internal/ (368 Go files), the Go module and tooling configuration, the Dockerfiles, the Hugo site and docs/, the seven Go integration suites and the one that validated examples, and the GitLab-internal template and automation configuration.

Carried over, each checked against the binary: composition, lifecycle hooks, Linux and Podman setup, troubleshooting and resets, use cases, and the architecture page. The old decision log and registry-cache design became ADR-0011 through ADR-0016.

Added: a GitLab CI pipeline (fmt, clippy, test, deny, example validation, a generated-reference check, Markdown and shell lint, a docs build, and a pages deploy), deny.toml, scripts/validate-examples.sh, and a rewritten README, CONTRIBUTING.md, and AGENTS.md. The pipeline follows the pattern in GitLab’s hardened-builds project: build and check the site in one job, publish it in another.

Found while doing it:

• cargo deny’s default settings hid two soundness advisories that cargo audit reported (the lru crate, through ratatui). deny.toml now reports unsound and unmaintained advisories for every dependency, and records four accepted advisories with reasons. See TM-SUP-001.
• deny also rejected the workspace because no crate declared publish = false; they now do.
• examples/own-infrastructure.yaml passed --default-issuer-name=caproni-ca, an issuer the Rust build doesn’t create. It now says xianyun-ca, and its header no longer carries Go-era migration advice.

Verified locally: cargo fmt --check, cargo clippy -D warnings, cargo test --workspace (1,119 passed, 0 failed, 23 ignored), cargo deny check, scripts/validate-examples.sh (15 configurations), rumdl check, shellcheck, nix build .#docs, and a sub-path serving test of the built site. GitLab accepts the pipeline file (checked with the project’s CI lint API).

Not verified: the pipeline hasn’t run on GitLab. The nixos/nix image, the shared runners’ ability to run nix develop for the whole workspace, cache sizes, job times, and the e2e:mock job’s need for KVM are all untested. The Pages site hasn’t been published, and the weekly advisory schedule doesn’t exist yet.

Original report

Date: 2026-10-13 Scope: the Rust workspace (crates/) and the documentation that describes it Commit reviewed: d5f74650 on main, plus uncommitted changes from this run

Executive summary

The project had no specification, no ADRs, no threat model, and no user stories. What it had was 50-odd user-facing pages that describe the Go caproni binary, a decision log with two stale rows, and per-crate porting notes. This run created all four missing artifact types for the Rust workspace, fixed the clear factual errors in existing docs, and checked the new material against the source with an independent adversarial review.

The headline finding isn’t a documentation defect. xianyun up and xianyun deploy can’t install anything today, because no deployer manager is registered in production code. The NixOS VM tests don’t catch this since their configs list no deployers. Measured, not inferred: see GAP-001 in the spec.

The review also caught two mistakes in my own first drafts (a wrong variable-expansion requirement, and a confirmation-bypass tagged “unproven” when one test settles it). Both are fixed and the second is now a measurement. Everything below uses the corrected numbers.

Assessment results

Work performed

Created

• docs/threat-models/xianyun-threat-model.md: STRIDE per element, DREAD scored, data flow diagram, 7 trust boundaries. Each threat is tagged MEASURED, CODE, or UNPROVEN. 7 have a runtime measurement and 2 more a search or scan measurement.
• docs/adr/0001 to 0009: record-keeping, the rewrite, compatibility aliases, the CommandRunner seam, the MCP server, the dev CA, the registry cache, VM tests, and the config model.
• docs/specs/xianyun-spec.md: retroactive spec with EARS requirements, Gherkin, a traceability matrix, and 7 implementation tasks.
• docs/stories/STORY-001 to 004: bring-up and deploy, agent over MCP, dev CA trust, legacy project compatibility.

Updated

• docs/config/_index.md: mirror.processes to mirror.edit_mode_processes (the schema field), and the full kubernetes_type list.
• docs/edit-mode-workflow.md: kubernetes_type list.
• crates/README.md: replaced the test counts with figures measured today; noted the one unimplemented placeholder test.
• docs/rust-rewrite-macos.md: removed the duplicate heading and the “not yet implemented” and “DONE” contradiction.
• docs/rust-rewrite-vision.md: it said there was no MCP interface; there is.
• docs/contributing/decisions.md: marked the BYO-only row Rejected/Deferred and the containerd row Accepted, with dated notes pointing at ADR-0007; added a pointer to docs/adr/.
• docs/telemetry.md: added a warning that the Rust binary sends nothing.
• docs/contributing/architecture.md, docs/design/registry-cache.md: dead paths (docs/configuration.md, internal/config/v2/config.go).
• integration-tests/validate-examples/README.md: it described a filename filter the script doesn’t apply.
• AGENTS.md: one missing blank line.

Verification run

Measurements were throwaway tests and temporary HOME directories, deleted afterwards. The repository has no leftover files from them. One environment repair happened outside the repo: the extracted cc-1.5.1 crate under ~/.cargo/registry was missing its src/target/ directory (the cached archive was intact), so I re-extracted that directory to get a build.

Key findings

Measured

Not measured, with the test that would settle each

TM-CFG-002 (http:// in a remote fragment’s relative reference), TM-CFG-005 (option-shaped repository), TM-CFG-006 (YAML alias bomb), TM-CFG-007 (config discovery outside home), TM-PKI-003 (hostile cluster zone), TM-PKI-006 (client enforcement of Name Constraints), TM-REG-001 (anonymous write to zot), TM-EXEC-003 (mise loading repository config). The threat model names the command or test for each. I deliberately skipped the PID -1 test (TM-REG-002) because it would signal the developer’s own processes.

Remaining gaps

Cross-reference health

Links were verified by rumdl (target file and anchor must exist) and an ID script. Counts are links in the new artifacts only.

“Missing” counts links that should exist and don’t. The spec’s own matrix lists the known gaps: FR-005 has no CLI-level test, FR-006, FR-012, FR-015, and IR-001 have no ADR, and NFR-005 and NFR-006 have no test.

Artifact status summary

Statuses use the values the iterate skill parses. Stories stay Revision 1 until their scenarios have passing tests. Four of the scenarios fail today (STORY-001’s first two, STORY-002’s fourth, STORY-003’s third).

Story gap map

Story review (story-reviewer rubric, applied once)

The rubric limit is three review cycles; this was cycle one. I didn’t re-score after revising, so a second pass is still owed.

Decisions for you

1. Which binary do the user docs describe? Resolved: the book describes xianyun; docs/ stays as the Go documentation. The guide pages cover the dev CA, registry cache, edit mode, and host setup, but not everything docs/ does (the GitLab guide, the FAQ, and the use cases have no equivalent yet).
2. Should the new artifacts publish to the Hugo site? Resolved: no. They moved to the mdBook (see the follow-up above). The threat model is published with the book, because the ADRs and specification link to it; if you’d rather keep it unpublished, those links need to go first.
3. How to treat the High threats. SR-001 and SR-003 are small changes with a large effect and have measured evidence, so they’re the cheapest place to start.
4. Should I commit this? Nothing is committed. The repository requires Conventional Commits, pre-commit, and the Default MR template, and no-commit-to-branch blocks main, so it needs a branch.

Maintenance recommendations

• Update a threat’s **Status:** line when its mitigation merges, with the SR reference and date, and move the matching SR to Implemented.
• Re-run the measurements in “Not measured” before closing a threat as unreal. Spec Tasks 3, 4, and 7 turn existing throwaway measurements into committed tests, which is the durable fix.
• Add a new ADR rather than editing an accepted one. ADR-0005 and ADR-0007 each name a follow-up.
• Run /iterate to validate artifact statuses against the implementation. The completion script cross-checks every TM entry, story, SR, ADR, and spec task against the codebase.
• Run this skill again after GAP-001 is fixed and after the user docs are rewritten.

What this review did not cover

The Go implementation, the macOS Keychain code (read, not run), the Helm charts users deploy, mirrord’s cluster-side agent, cargo deny, licence policy, and any fuzzing. Test coverage (about 90 percent in crates/README.md) wasn’t re-measured. Line numbers cited in the threat model were spot-checked by the reviewer and found accurate to within a few lines, but they will drift as the code changes.