Testing

The test suite covers unit behavior, CLI workflows, and local git fixtures.

Required Checks

Run these before opening a pull request:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --all-targets
python3 scripts/check_docs.py
python3 scripts/generate_release_docs.py v0.0.0 --output-dir target/release-docs-test
PYTHONDONTWRITEBYTECODE=1 python3 scripts/test_automation.py

Toolchain

The workspace ships a root rust-toolchain.toml pinning the toolchain to the MSRV (Rust 1.96.1, with clippy + rustfmt), so a bare cargo — cargo build, cargo test, cargo clippy, cargo fmt — uses 1.96.1 without typing cargo +1.96.1. The explicit +1.96.1 form still works and remains the escape hatch. Keep the pinned channel in lockstep with rust-version in the root Cargo.toml.

One caveat: rustup only honors the file when nothing higher-precedence overrides it. A RUSTUP_TOOLCHAIN environment variable, an active rustup override, or a version manager that shims cargo (e.g. mise or asdf, which export RUSTUP_TOOLCHAIN) all win over rust-toolchain.toml. If bare cargo --version in the repo does not report 1.96.1, check rustup show (it names the active override) and either unset RUSTUP_TOOLCHAIN / drop the version-manager rust pin, or just use cargo +1.96.1.

CI is deliberately explicit the other way: its "latest stable" jobs run cargo +stable … so they still catch future stable regressions, while the dedicated msrv job proves the 1.96.1 floor with cargo +1.96.1. (This means local bare cargo lints/tests on the MSRV while those CI jobs use latest stable — an intentional split, so an occasional new-stable clippy or rustfmt nit can surface in CI that a local MSRV run did not; reproduce it with cargo +stable clippy … / cargo +stable fmt … if needed.)

The CI policy: what runs, when (AI-54)

The per-commit CI policy is written down, not emergent (story 11-5, codifying the Epic-9 precedent). The full job set — fmt, clippy, the 3-OS test matrix, build, perf-budgets, docs, boundary, semver, msrv, audit (story 12-3: the supply-chain job that scans Cargo.lock with cargo audit and fails on a RustSec finding without a documented waiver), and coverage — runs on:

• Every pull request (opened / synchronize / reopened).
• Every branch push — the push trigger carries no branch filter beyond matching all branches (branches: ["**"]), so pushing commits to a feature branch with no PR yet runs the FULL cross-OS + coverage CI, not a reduced subset (AI-37: the 3-OS matrix and coverage must not wait for the PR). Repeated pushes to the same branch collapse into one run via the workflow's per-ref concurrency group (cancel-in-progress).
• A nightly schedule (cron 17 7 * * * on the default branch) — an always-fresh cross-OS + coverage record even when nothing pushed.
• Manual dispatch (workflow_dispatch) for an on-demand full run of any ref.

A separate, independent workflow — the docs probe (.github/workflows/docs-probe.yml, daily cron 3 6 * * * plus manual dispatch, with no needs: coupling to the CI job graph) — watches the DEPLOYED docs site: it fetches every page registered in docs/meta.json (mirroring the URL mapping in docs/lib/source.ts) and asserts HTTP 200 plus a content marker on the newest page, failing with ::error:: lines naming each dead page. It exists because the Cloudflare Pages deploy once silently died for ~two weeks (August 2026) — four pages 404'd while every repo CI check stayed green; the probe is the watcher CI could never be.

Unit and Integration Tests

cargo test --workspace --all-targets

Agent lifecycle and the fake_agent helper

Integration tests create local temporary git repositories. They do not require network access. The agent-lifecycle tests spawn a small cross-platform helper binary (fake_agent, in ktesio-conformance) as a real child process to prove start, stop, launch-failure, no-survivor, pause/resume, crash-detection, restart, and orphan-adoption behavior end to end; it is a dev/test artifact and never ships. For pause, fake_agent --heartbeat-ms <ms> prints a periodic incrementing line, so a guaranteed (Unix) pause is provable — the heartbeat stops growing under SIGSTOP and resumes under SIGCONT. For survival, fake_agent --crash-after-ms <ms> runs normally past the readiness window and then exits non-zero, simulating an unrequested crash so the reaper detects it and the Restart Policy fires. The crash/restart legs (tests/crash.rs) prove a never-policy crash lands failed with a crashed cause and no restart, and an on-failure crash is automatically restarted by the reaper (the crash-loop-stops-at-5 and count-reset legs run in the supervisor unit tests with an injected fast backoff, so they never sleep for real seconds while production keeps the 1s×2/60s constants). The engine-kill adoption test (tests/adoption.rs) is the NFR-1 proof: it runs a first engine in a subprocess that starts an agent and exits WITHOUT a graceful stop (a kill -9 model — no destructors run, so the agent survives and re-parents to init), then opens a new engine over the same state dir and asserts the live child is adopted (row running, a subsequent stop truly kills it, no orphan remains) while a record whose process is gone reconciles to failed; the AI-7 (paused-live process adopted and resumable) and AI-8 (phantom running row → failed) cases ride the same file. The reboot-durability test in the same file simulates a machine reboot — a true reboot is infeasible in CI, so it registers several instances in different states, leaves one running via the surviving-engine subprocess, then kills every live agent process (their PIDs would not survive a reboot) and reopens the engine over the same state dir — and asserts the reboot invariants: every registration survives with its name/kind/home intact, the previously-running instance reconciles to failed, the cleanly-stopped one stays stopped, each restart policy and count is unchanged, and no orphan process remains. The ≤1s durability bound behind that guarantee is asserted structurally by the store tests (WAL, synchronous=NORMAL, one committed transaction per state mutation, and a reopen that finds every row intact). The Windows-correct counterpart semantics are asserted POSITIVELY on the Windows matrix leg (AI-29, story 11-5): there, the engine subprocess's exit takes every agent process with it (Job-Object kill-on-close — the Windows-correct NFR-1 outcome), and the windows_* adoption tests in the same file assert the killed children's records reconcile to failed (engine-death, whole-fleet-reboot, and paused-row variants), with the Unix-shaped tests' runtime skips carrying pointer comments to their Windows siblings. The Fleet --json shape is covered end-to-end in crates/kt/tests/agent_cli.rs: kt agent list --json and kt agent show <name> --json emit a single parseable document on stdout (nothing else there), carrying a schema_version and per-instance objects whose budget is a real token-budget object when one is configured and an honest null (never 0) when not, while usage is a real token-totals object and metering_source is surfaced (see the metering paragraph below), with the metering note routed to stderr and an empty Fleet rendered as a valid empty array.

Unified layered config (AD-9)

The unified layered config (AD-9) is tested at three levels. The pure precedence resolver is exhaustively unit-tested in crates/ktesio-engine/src/domain/config.rs with no I/O: a key present in exactly one layer (each of the four), a key present in each adjacent precedence pair (the stronger layer wins), a key present in all four (the strongest wins), a nested-table per-leaf merge where shapes agree (a stronger layer's a.b overrides only a.b while a weaker layer's sibling a.c survives — the data-loss guard), the scalar-over-subtree and subtree-over-scalar shape collisions where they disagree (the stronger layer's shape wins and prunes the weaker layer's orphans, with the surviving leaf tagged to the layer that defines it — no self-contradictory tree, no stale provenance), the empty-single-layer and all-empty cases, and a determinism check; every case asserts the recorded source layer, proving the provenance seam. Write-time validation is unit-tested against fixed inputs: the sole known key (model) and an agent.* pass-through key are accepted; an equally-unknown non-agent.* key is rejected; a near-miss (modle) suggests the nearest key (model, with a deterministic candidate-string tie-break) while a far-miss suggests nothing; an empty dotted segment (agent..b) is rejected; and the hand-rolled Levenshtein has its own coverage. The set/get round trip is proven at the registry level (set_config model then effective_config reflects it tagged as the instance layer, and an invocation override beats it; an unknown key is rejected leaving the on-disk config.toml byte-unchanged; nesting a child under an existing scalar fails closed byte-unchanged; an agent.* key round-trips verbatim; the seeded name identity key is filtered from the resolved view; a malformed instance layer surfaces a typed error, not a panic) and end-to-end through the CLI in crates/kt/tests/agent_cli.rs (kt agent config set model then kt agent config get prints the value on stdout; an empty effective config before any set says so; an unknown key exits non-zero with the suggestion on stderr and the config unchanged; a scalar-shape conflict fails non-zero byte-unchanged; an agent.* key sets and gets verbatim; the whole-config table lists keys on stdout).

Unified → native config mapping (FR-12)

The unified → native config mapping (FR-12) is tested at four levels. The mapping model in crates/ktesio-adapter-api/src/config.rs is unit-tested pure and I/O-free: each ConfigTarget renders its native form (an env var name, a two-token --flag value pair, a file path plus native key), a ConfigMapping builds and reads back deterministically, each target kind deserializes from the [config.<key>] TOML shape (a sub-table naming no native mechanism, or a file sub-table with an unknown field, fails to parse), and validate rejects an empty native token or a file path that is absolute or escapes the Agent Home. The manifest [config] section is tested in manifest.rs: an absent section validates with an empty mapping, all three target kinds parse and validate and read back through the accessor, a malformed rule is an InvalidField naming the [config.<key>] sub-section, and a path escaping the home is rejected. Both adapter kinds declare a mapping in shape-parity: the builtin mock and the conformance MockAdapter each declare model → env MODEL, and the cross-boundary parity test in crates/ktesio-engine/tests/registration.rs asserts the two mappings are identical (guarding the fixture against drift). The application transform (adapter::apply_config_mapping) is unit-tested in isolation: a model value maps to each declared target (env → the launch env, flag → two args, file → a rendered TOML file in the Agent Home at the native key), an agent.* leaf is delivered verbatim to an env var named by its key-tail, an unmapped documented key is a no-op, and the resolved-config → launch transform is deterministic. The end-to-end proof at start runs on BOTH adapter kinds in the supervisor tests: the inert builtin mock proves model → env on the mapped launch the mapping produces (a native adapter carries no live process), while a live fake_agent manifest carrying a [config] section proves model → flag (observed in the spawned process's argv via fake_agent --dump) and model → file (the engine renders the native file into the Agent Home) and an agent.* key delivered verbatim into the process environment. The agent.*-unvalidated marker is covered end-to-end through the CLI in crates/kt/tests/agent_cli.rs (kt agent config get shows a Validated column marking an agent.* leaf unvalidated and a known model key validated, on stdout; a known-key-only config shows no unvalidated marker).

Effective-config provenance (FR-13)

Effective-config provenance (FR-13) is tested at three levels. The provenance accessor is unit-tested in crates/ktesio-engine/src/domain/config.rs (EffectiveConfig::source_label reports the winning layer for a leaf resolved from each of the four layers, agrees with SourceLayer::as_str, respects precedence, and is None for a missing key). The persisted-snapshot DTO and writer are unit-tested in crates/ktesio-engine/src/domain/registry.rs: building the snapshot from a multi-layer effective config yields one entry per leaf with the rendered value plus its source label and the schema version; it round-trips through JSON (the source is the kebab-case wire form, the value is the rendered display string); the writer persists it at EnginePaths::effective_config_snapshot and it parses back; a second write overwrites in place; and a write failure (the snapshot path pre-created as a directory) surfaces a typed SnapshotWrite error, never a panic. The start-seam write is proven in the supervisor tests (crates/ktesio-engine/src/domain/supervisor.rs): starting a live fake_agent instance writes effective-config.json into the Agent Home carrying model tagged instance; a re-start (through the same start_inner seam) overwrites the snapshot with the newly resolved value (AC7); and a snapshot-write failure rejects the start with a typed EngineError::Snapshot before the starting transition, leaving the instance in its prior state (no spurious change). The CLI rendering is covered in crates/kt/tests/agent_cli.rs: the human config get gains a Source column showing the instance layer for a set key (on stdout, with the stale Epic 2.3 deferral note retired from both streams); config get --json emits a versioned document whose per-leaf objects carry { key, value, source, unvalidated } as pure JSON on stdout (an instance-sourced validated key and an agent.* unvalidated leaf); the single-key --json form emits just that leaf; and starting an instance persists the snapshot into the Agent Home (read back from the path register printed). The pure config_json serializer is unit-tested in-process in crates/kt/src/cli/agent.rs (the versioned document carries source + unvalidated per leaf, the single-key form emits one leaf, and the --json value matches the human display form — proving the single display path). Every surface renders through that one display path, the single choke point where secret masking hooks (below).

Secrets (FR-14 / NFR-6 / AD-10)

Secrets (FR-14 / NFR-6 / AD-10) are tested at four levels, anchored by a no-leak matrix that proves the "safe by construction" guarantee. The primitives are unit-tested pure in crates/ktesio-engine/src/domain/: the secret:NAME classifier + secret_name extractor (a non-empty NAME after the prefix classifies; a bare secret: does not; classification is on the value regardless of key) and the display() mask (a secret:NAME leaf renders secret:**** at the single choke point while a non-secret leaf is unchanged) in config.rs; and the SecretString newtype in secret.rs (Display and Debug both redact to [REDACTED] and never the cleartext, expose_secret() returns it, and a struct embedding a SecretString and deriving Debug does not leak — the structural guard). The SecretResolver port + resolvers are unit-tested in ports/secret_resolver.rs: the env resolver reads a set var and misses an unset one; the 0600-file resolver reads a NAME = "value" entry, treats a missing file and a non-string entry as a miss, and errors hard on malformed TOML; the composite tries env-then-file (env wins, file resolves when env is absent), a hard error short-circuits (never silently "absent"), and an unresolved reference names the NAME + resolvers tried with no value. The Unix 0600 permission check lives in backends/unix (the allowlisted #[cfg] home): a 0644 (group/other-readable) secrets file is refused with a chmod 600 remediation while a 0600/0400 file passes; the Windows posture is a documented portable skip. At the engine level (supervisor.rs + registry.rs) a model = secret:NAME leaf resolves (env) to a sentinel: the spawned agent's argv carries the cleartext (usable — delivery), while the persisted snapshot and every transition-event payload carry the mask (no leak); an unresolved secret rejects the start with a typed EngineError::Secret (naming the NAME, never a value) before any state change or snapshot write. The end-to-end matrix in crates/kt/tests/agent_cli.rs drives the real kt binary: a secret:MODEL_KEY leaf resolves to a sentinel that (positively) reaches the adapter's native env (env=MODEL=<sentinel> in the fake_agent --dump) and (no-leak) appears in none of the effective-config snapshot, config get --json, the human config get, or any file in the Agent Home (logs + event payloads included) — the mask appears instead; config get --json --reveal (and the single-key form) is the sole surface that carries the sentinel, while default --json masks, a --reveal on a non-secret leaf is a harmless no-op, and an unresolved secret exits non-zero with a diagnostic naming the NAME while the instance stays registered. The --reveal render + overlay paths are also covered in-process in crates/kt/src/cli/agent.rs.

Self-reported metering and the Usage Ledger (FR-19 / AD-6 / AD-7)

Self-reported metering and the Usage Ledger (FR-19 / AD-6 / AD-7) are tested at three levels, with a no-double-count matrix at the heart. The leaf primitives are unit-tested pure and I/O-free: the UsageEvent/RunId/UsageTotals/UsageUpdateEvent shapes in crates/ktesio-engine/src/domain/usage.rs (the AD-7 minimum shape is tokens-only — no dollar/label/budget field — round-trips snake_case; a thousand back-to-back RunId::mint()s are all distinct; the versioned usage-update wire struct carries its schema version), and the KTESIO_USAGE {json} sentinel-line parser in crates/ktesio-engine/src/ports/usage_source.rs (a well-formed line yields the agent-supplied fields; a non-usage line and a malformed usage line are both skipped without panic; a trailing \r is tolerated; the emitter's format_usage_line and the parser round-trip, guarding the shared convention against drift). The append-only ledger write + reads are unit-tested in crates/ktesio-engine/src/store/sqlite.rs: an event inserts (Inserted) and the same (run_id, sequence) replayed returns DuplicateReplay and adds no row (the DB-level no-double-count invariant); the same sequence under a different run_id is a distinct event; per-run vs cumulative totals sum correctly across two Runs; an absent instance totals zero; and the schema-v3 migration (the sequence column + UNIQUE(instance_id, run_id, sequence) index) upgrades a v1 and a v2 DB while preserving existing rows and enabling dedup on the migrated rows.

The robust, cross-OS end-to-end matrix lives in crates/ktesio-engine/tests/metering.rs, deliberately shaped to avoid the cross-OS-fragile _live pattern the Epic-2 retrospective flagged (spawn an agent, sleep, read a dump FILE on a wall clock). Instead it keeps a single in-process Engine alive for the whole test (like tests/lifecycle.rs), so it needs no cross-lifetime process survival and runs identically on all three OSes — there is no OsId-gated skip anywhere in the file. fake_agent --emit-usage <N> emits <N> sentinel lines with FIXED token sentinels after announcing readiness, and each test polls the committed SQLite ledger (row count / totals, via the engine reads and a direct read-only connection) until the known count is reached — asserting on durable committed STATE the engine writes transactionally, never a timing-sensitive side file. The matrix proves: (a) <N> emitted events land as <N> rows under the Run's id and the Fleet-detail totals equal the ledger exactly; (b) fake_agent --emit-usage <N> --replay-usage re-emits sequence 0 (a delayed batch) and the ledger total is unchanged — no double-count; (c) a stop→start opens a fresh Run (two distinct run ids) and the current-Run totals reflect only the second Run (per-run totals do not bleed); (d) a never-metered instance reports an honest all-zero UsageView (a truthful zero, distinct from the budget null seed) with its Metering Source visible. A source-level single-writer audit in the same file greps the engine src/ and asserts record_usage_event is called only from the one ingestion→commit choke point path (the AD-7 invariant story 3.2's enforcement relies on). The Fleet metering surface is also covered end-to-end through the CLI in crates/kt/tests/agent_cli.rs (list --json/show --json carry a real usage object + metering_source, and a real budget object when configured — null when not; the human list/show render the token totals and — in show detail — the Metering Source).

Token-budget enforcement (FR-18 / FR-21 / AD-7)

Token-budget enforcement (FR-18 / FR-21 / AD-7) is tested the same way, with the bulk of the coverage in pure evaluator unit tests (cross-OS by construction) and a robust end-to-end matrix on top. The pure tests in crates/ktesio-engine/src/domain/budget.rs exhaustively exercise the BudgetEvaluator: below / exactly-at / over each ceiling (the ≥ boundary, and its N-1 companion), per-run-before-cumulative precedence, an unset scope never breaching, a zero ceiling, and each Breach Action riding the decision unchanged — plus the BreachAction/TokenBudget parse + wire forms. The config layer (domain/config.rs) unit-tests that the three budget keys validate at write time (a non-numeric budget or an unknown action is rejected, naming the offender), that resolve_token_budget reads the current resolved values (absent → None/pause), reflects a changed value with no caching (AC-B), and defensively degrades a malformed present value to absent. The breach event + BudgetExceeded cause round-trip (schema-versioned, snake_case, tokens only) in domain/event.rs, and the BudgetView shape (limits + remaining saturating at zero + action) in domain/fleet.rs. The robust, cross-OS end-to-end matrix lives in crates/ktesio-engine/tests/budget.rs, mirroring metering.rs exactly — a single in-process Engine kept alive, no cross-lifetime survival, no OsId-gated skip — driving real fake_agent --emit-usage traffic past a budget set through the config path and polling the committed lifecycle STATE (via a direct read-only DB connection) until it reaches the expected state (the evaluator runs synchronously in the ingestion path, so the transition commits as soon as the breaching event is ingested — deterministic, never a wall-clock sleep). The matrix proves: a cumulative breach fires the default pause and records the breach event + the BudgetExceeded transition cause; the ≥ boundary (a total exactly at the budget breaches); breach_action = stop drives the instance to stopped; breach_action = warn records the breach but keeps the instance running with no pause/stop transition; a per-run budget breaches within a Run (reporting the PerRun scope); a budget lowered while running applies on the next event (AC-B); and an un-budgeted instance never breaches (the negative control). A source-level single-evaluator audit in the same file greps the engine src/ and asserts BudgetEvaluator::evaluate is called only from the supervisor's ingestion choke point (the AD-7 companion to the single-writer audit). The budget CLI surface — the config keys set/reject and the Fleet-detail budget cell (real ceiling + remaining + action, —/null when absent) — is covered end-to-end in crates/kt/tests/agent_cli.rs.

Dollar cost derivation and Cost-Cap enforcement (FR-20 / FR-21 / FR-23 / AD-8)

Dollar cost derivation and Cost-Cap enforcement (FR-20 / FR-21 / FR-23 / AD-8) is tested the same way, and — because this is billing money — the exhaustive pure money-math unit tests are the deliverable's spine. The pure tests in crates/ktesio-engine/src/domain/cost.rs cover the whole boundary surface, instant and identical on every OS: cost_micros for zero tokens, a zero rate, input-only / output-only, input ≠ output rate priced independently, the round-half-up boundary (a token count landing on a half-micro rounds up deterministically; its just-below companion does not), and a u64::MAX × large-rate runaway proven to SATURATE at i64::MAX, never wrap (the un-breach guard); the CostEvaluator reusing the token BreachDecision at the ≥ micro-dollar boundary (exactly-at breaches, one micro below does not, per-run-before-cumulative, unset-scope, a $0 cap); the Micros newtype serializing as a bare integer (never a float, never a $ string) with saturating add / saturating-remaining-floored-at-zero; and the single render_dollars currency formatter ($0.00, sub-cent half-up rounding to the rendered cent, a large value, the label always appended). The config layer (domain/config.rs) unit-tests the dollar-string→micros parse ("3.00" → 3_000_000; "0.000003" → 3 micros; sub-micro precision rejected, non-numeric / signed / multi-dot rejected), that validate_write rejects a malformed Rate/cap value at write time (naming the offender), and that resolve_cost requires both Rate directions (a half-Rate is inert — AC-B), reads the current resolved values with no caching (the live-read behind "re-prices future only"), and defensively degrades a malformed value to absent. The store (store/sqlite.rs) unit-tests the no-retroactive-repricing persistence: each row priced at its own stored Rate, a NULL-rate row contributing $0, a Rate change re-pricing future events only (event 1 at $3/1M + event 2 at $6/1M sums to the per-event prices, not the whole history repriced), and the cost sum saturating. The dollar breach event + BudgetExceeded-dollars cause round-trip (schema-versioned, snake_case, integer micros + label, no $ string, no f64, a pre-3-3 token breach still parsing) in domain/event.rs, and the UsageView/BudgetView dollar fields (present + labeled with a Rate, absent without one) in domain/fleet.rs. The robust, cross-OS end-to-end matrix lives in crates/ktesio-engine/tests/cost.rs, mirroring budget.rs exactly — a single in-process Engine, no cross-lifetime survival, no OsId-gated skip — driving real fake_agent --emit-usage traffic at a known Rate past a known dollar cap and polling the committed lifecycle STATE until it reaches the expected state. The matrix proves: a dollar-cap breach fires the default pause and records a dollar breach event (dimension dollars, labeled) + the dollar BudgetExceeded cause; the ≥ micro-dollar boundary (a cost exactly at the cap breaches); stop and warn variants; a Cost Cap with no Rate is inert while token enforcement still fires (the AC-B honesty crux); a token cap and a dollar cap each fire once on the same Run (the dimension-keyed latch); and no-retroactive-repricing end-to-end (a Rate raised mid-run leaves the cumulative cost strictly below the fully-repriced figure). A source-level single-evaluator audit asserts CostEvaluator::evaluate is called only from the supervisor's ingestion choke point (the dollar companion to the token audit). The dollar CLI surface — the Rate/cap config keys set/reject, the labeled $X.XX (estimated) cells in list --json (integer micros + label, no $ string) and the human table / show (through the one currency module), and the honest inert dollar view when no Rate is configured — is covered end-to-end in crates/kt/tests/agent_cli.rs. A currency grep-lint CI gate (below) proves no module outside domain/cost.rs builds a $-formatted money string (AD-8).

Engine-observed metering (FR-19 engine-observed half / AD-7)

Engine-observed metering (FR-19 engine-observed half / AD-7) is tested the same layered way, with the bulk of coverage in pure/local unit tests (cross-OS by construction) and a robust end-to-end matrix on top. The pure tests in crates/ktesio-engine/src/metering/parse.rs exhaustively exercise the OpenAI usage parse: a well-formed completion body yields the mapped (prompt→input, completion→output) counts; a body with no usage, a usage missing a field, a malformed/truncated/empty body, and an SSE stream chunk are all skipped without panic by the JSON-body parse (observation is best-effort — the agent's call still succeeds); a negative/float/string count is rejected (never a wrapped value); zero counts are valid; a large u64 parses faithfully. Since story 12-2 the terminal SSE usage frame has its own exhaustive pure suite (parse_openai_sse_usage): the null-intermediates-then-terminal-frame shape parses to exactly the terminal counts, the LAST usage-bearing frame wins, "usage": null/missing-usage frames and [DONE] sentinels contribute nothing, CRLF terminators and the no-space data: spelling parse, a malformed mid-stream frame does not poison later frames, zero counts are valid, and a negative count discards its frame. The listener bind is unit-tested in crates/ktesio-engine/src/metering/listener.rs — the resolved SocketAddr is loopback and is not 0.0.0.0/routable (AC-B), each listener gets a distinct ephemeral port, an empty/foreign-scheme (ftp://)/scheme-less upstream is refused with a traffic-free reason (no URL echoed), an https:// upstream is accepted (12-3), and dropping the listener aborts its task (teardown). The injection gate is unit-tested directly (POST + JSON + an object already setting stream:true and no stream_options of its own — every other shape forwarded byte-identical), and in-process relay tests prove a streamed completion's forwarded body carries the injected include_usage, the SSE response is relayed faithfully, the terminal frame's counts land in the queue (exactly one event), and a usage-less stream fabricates nothing. The https composition (12-3) is proven in-process with a self-signed root: an rcgen-minted cert drives a blocking rustls TLS server on a thread, the listener's injected ClientConfig trusts that root (the production default is webpki-roots instead — no system trust store), and the full chain composes — injected include_usage rides the encrypted request, the terminal SSE frame comes back encrypted, and its counts land in the queue. The observed ObservedUsageSource (ports/usage_source.rs) unit-tests the engine-minted per-Run sequence: it maps the parsed counts, increments monotonically per observed completion, is unique within a Run, and resets to 0 for a fresh Run. Source selection is unit-tested in the supervisor (domain/supervisor.rs): a self-reported instance runs no listener (its start path unchanged), an engine-observed instance with no configured upstream rejects with a clear metering.upstream_base_url-naming error, and the injected base_url lands at the reserved metering.base_url key for the config-mapping. Story 12-1's detach surface is pinned at four layers: the supervisor refuses start_detached on an observed manifest before any side effect (state untouched, no record) while a plain start is unaffected; a self-reported detached start runs, commits its write-ahead record (with detach=true), and stops in-process; a guaranteed-interaction manifest proves the detached spawn's stdin force-clear end to end (send_input fails with the no-pipe InteractionUnavailable, never a silent dead-pipe write — review round 2's verification gap); and a crashed detached agent's Restart-Policy restart preserves the detached property (the flag is read off the surviving record and re-written — never silently downgraded to attached; review round 2). The record's flag itself is pinned at the store layer (the detach round-trip both ways, and the additive schema v5→v6 migration — a pre-v6 record survives with detached reading false, then round-trips true). The end-to-end survival proof (tests/adoption.rs::detached_start_survives_a_clean_engine_exit_and_the_next_engine_adopts) drives a helper subprocess that starts detached and drops its engine CLEANLY — the child survives on all three OSes (on Unix the Drop skips the killpg; on Windows the spawn-time Job Object is created WITHOUT kill-on-close, so closing the engine's handles kills nothing — the cross-lifetime Windows limitation that forces _unix suffixes elsewhere is removed by design here) — and proves the full N-command durability: a BENIGN intervening command (a second engine adopts, reads the status, exits) must leave the agent alive, and only a third engine's explicit stop terminates it. The backend-level disarm is pinned per-OS in the backends/ test homes (unix: drop-without-kill + fingerprint adoptability + the ADOPTED detached handle's own drop-disarm + in-process stop; windows: a detached spawn survives drop, adoption + stop work, and the job-without-kill-on-close shape keeps the stop escalation's descendant reach). The CLI half is pinned end-to-end in crates/kt/tests/agent_cli.rs: the refusal (start_detach_of_an_engine_observed_instance_is_refused_with_exit_code_5 — exit 5, the listener-lifetime why, the remediation, and the untouched registered state), the survival journey (start_detach_survives_the_command_exit_and_the_next_command_stops_it — the enforcement-window notice on stderr only, the child alive after exit, a benign kt agent list in the middle that must NOT kill it, then a later command adopting and stopping it), and the --help honesty assertion (start_help_carries_the_enforcement_window_honesty — the help text itself names every between-commands window and the refusal, review loop 1's patch bundle). Observed-drain durability (story 12-4) is pinned in the supervisor by the AI-41-mirror suite: a whole-table fault parks the minted events (front sequence + attempt streak asserted) and the repair restores the FULL schema including the UNIQUE dedup index; the retry commits the SAME minted sequences (never re-minted); a partial failure (a trigger failing only the second insert) parks exactly the un-committed tail; a permanently poisoned event (a trigger keyed on its token count) parks 1→2→3 times and is then SKIPPED loudly (named sequence, announced loss) while the healthy events behind it park with a fresh streak and commit after the fault clears; and the terminal drain announces the loss without any retry claim and drops the park. The robust, cross-OS end-to-end matrix lives in crates/ktesio-engine/tests/observed_metering.rs, mirroring metering.rs exactly — a single in-process Engine kept alive, no cross-lifetime survival, no OsId-gated skip — driving a real fake_agent --observed-calls <N> / --observed-stream-calls <N> (a pure-std HTTP client that POSTs to its injected OPENAI_BASE_URL) through the engine's loopback forward listener to a local pure-std upstream stub (a TcpListener returning a FIXED OpenAI usage, extended in 12-2 to read the forwarded request body and answer "stream": true bodies with the SSE terminal-frame shape, asserting the injected include_usage arrived), and polling the committed SQLite ledger until the known observed-row count lands — never a wall-clock sleep. The matrix proves: (a) <N> observed calls land as <N> rows tagged engine-observed and the Fleet-detail totals equal the ledger exactly; (a') <N> STREAMED calls land as exactly <N> rows with the exact terminal-frame totals and the stub confirms the injected include_usage arrived upstream (12-2 end-to-end); (b) a token budget enforces on observed usage (pauses, records a breach carrying the engine-observed source) — the 3-2 path reused unchanged; (c) a dollar cap enforces on observed usage and its derived figure carries EstimateLabel::Estimated — the 3-3 path reused unchanged, an observed count is honestly an estimate; (d) a no-leak sweep — the fake_agent forwards a sentinel API key, the upstream stub confirms it arrived (the relay is faithful), and the sentinel appears in none of ktesio's surfaces (the ledger DB bytes, the per-instance logs dir, the transition events); (e) an engine-observed instance with no upstream fails to start cleanly (no state change); and (f) the AC-C regression guard — a manifest with no [metering] is still rejected, and engine-observed is itself a valid declaration (no contract change beyond the version constant itself). The CLI half of the detach refusal is pinned end-to-end in crates/kt/tests/agent_cli.rs (start_detach_of_an_engine_observed_instance_is_refused_with_exit_code_5 — exit 5, the listener-lifetime why, the remediation, and the untouched registered state) alongside start_detach_survives_the_command_exit_and_the_next_command_stops_it (the cross-OS CLI proof: the enforcement-window notice on stderr only, the child alive after exit, and the next command adopting and stopping it). This is deliberately NOT the _live wall-clock/dump-file pattern the Epic-2 retro flagged (AI-38): loopback HTTP + committed-state polling is OS-uniform, so there is no legitimate reason for an observed-metering test to be OS-gated.

Reading usage & cost, per-instance and Fleet-wide (FR-22 / FR-23 / AD-8)

Reading usage & cost, per-instance and Fleet-wide (FR-22 / FR-23 / AD-8) — the Epic-3 read/report capstone — is tested with the bulk of coverage in pure aggregate unit tests (the honesty rules are the deliverable's spine, and they are pure, so their tests are instant and cross-OS by construction) plus a thin totals == ledger end-to-end proof. The pure tests in crates/ktesio-engine/src/domain/fleet.rs exhaustively exercise FleetTotals::from_entries over in-memory rows: an empty Fleet (all-zero tokens, dollars absent None, no label, not partial); a Fleet of one Rate'd (the aggregate equals that instance, labeled estimated) and no-Rate (tokens only, dollars None); a mixed Fleet (Rate'd + metered-but-no-Rate + never-metered → tokens sum all three, dollars sum only the Rate'd one, dollars_partial = true, labeled estimated); all Rate'd (dollars sum all, not partial); a Rate'd zero-usage instance (a genuine labeled $0, counted, not partial — distinct from an unpriced instance); zero-not-absent (ten idle instances plus one busy → the token total is exactly the busy one's, proving the zeros were summed not dropped, N instances contributing N contributions); token-sum and Micros-dollar-sum saturation (near u64::MAX / i64::MAX saturate, never wrap); the label is always estimated when present (never reconciled in v1); and the wire shape (integer micros + a snake/kebab label, snake_case fields, no $ string, no f64, the partial flag as data, a round-trip). The FleetListing extension is unit-tested too: new(instances) computes totals as exactly FleetTotals::from_entries(&instances) (the document is self-consistent — the aggregate is derived from the rows it carries), an empty listing has a zero/absent-dollars totals, and the Fleet document version is the bumped 2 (additive — totals gained). The robust, cross-OS end-to-end matrix lives in crates/ktesio-engine/tests/fleet_totals.rs, mirroring cost.rs exactly — a single in-process Engine, no cross-lifetime survival, no OsId-gated skip, and — because the aggregate is pure and the read is over durable committed state — no wall-clock-timing-sensitive assertion (retro AI-49): it registers + meters MULTIPLE real fake_agent --emit-usage instances (one with a Rate + accrued usage, one metered without a Rate, one never metered), polls the committed SQLite ledger until the known row counts land, then asserts the Fleet-wide FleetTotals (composed exactly as the CLI composes it, via FleetListing::new over the engine's fleet() rows) equals the exact sum of the per-instance ledger totals — tokens across all three, dollars only the Rate'd one — is labeled estimated, and is flagged dollars_partial because a metered instance had no Rate (totals == ledger at Fleet scope, AC-A); a companion proves the complete (not-partial) arm when every metered instance is Rate'd, and the empty-Fleet arm. The Fleet-total CLI surface — the list --json top-level totals object (integer micros + label, no $ string, the partial flag as data, the schema bumped to 2), the human list Fleet-total footer (a labeled dollar total through the one currency module, an honest lower-bound note when partial, and a — — never $0.00 — when no instance is Rate'd), and the per-instance show view surfacing both token scopes (cumulative + current-Run) — is covered end-to-end in crates/kt/tests/agent_cli.rs, and the pure footer/scope renderers are unit-tested in the kt bin. The currency grep-lint CI gate (below) still covers the new aggregate render site — the Fleet-total dollar figure formats $ only through domain/cost.rs, the allowlist is not widened — which is the automated proof for FR-23 at the aggregate. This read is engine-side over the existing ledger with no ledger write and no new adapter surface, so CONTRACT_VERSION is untouched by Epic 3 (the semver-check CI job stays green with no contract change; the Fleet document FLEET_SCHEMA_VERSION bump is a different versioned surface). Epic 3 (Cost Governance) is complete.

The Conformance Test Kit

The Conformance Test Kit (TCK) is the systematic proof that an adapter honors the Adapter Contract, shipped as a public library API in ktesio-conformance (story 6-4, FR-27) — a compliance report, not a test framework. The harness registers an adapter with a fresh engine over a hermetic temp state root — the SAME supervisor/registry path the engine integration tests drive, no shadow reimplementation — and returns a serializable ConformanceReport whose fixed-order entries each carry a SectionResult: Pass, Fail { reason } (the first failure reason; the suite NEVER aborts mid-run — every section reports), or NotApplicable { reason } (the skip is justified from the adapter's REGISTERED declaration, never hardcoded per adapter).

Using the kit (the third-party shape)

A third-party adapter crate adds ktesio-conformance as a dev-dependency — before the crates publish (story 7-4), the workable pre-publish form is a git dependency:

# Pin a full commit SHA: until the crates publish (story 7-4) a bare `git =`
# dependency floats on this repo's default-branch HEAD, and a breaking
# report-shape change would break your build without you moving. Update the
# pin deliberately.
[dev-dependencies]
ktesio-conformance = { git = "https://github.com/Ktesio/ktesio", rev = "20ddc204403a5c412e0e3249d4609dd47c30854e" }

Then author the adapter.toml and invoke the harness from the crate's own #[test]: run_mock_conformance(&manifest_dir) (the manifest shape) or run_conformance(&TckAdapter::Native(kind.into())) (a native builtin registered by kind). The caller asserts on the report (is_conformant(), section(id), failures()) — the tests/third_party_manifest.rs file in ktesio-conformance is the exact third-party shape.

The eight sections

• capability_edges — the persisted effective projection matches the declaration.
• lifecycle — start → running → stop with the full transition sequence, plus a crash leg on a twin manifest whose process exits past the readiness window — failed with the exit code preserved and a crashed cause under a Never policy.
• pause — honest per the declared level: Guaranteed proves a real suspension via a heartbeat freeze probe plus plain command causes; BestEffort is still APPLICABLE — it must demonstrate the pause-best-effort / resume-best-effort qualifier causes, not skip; Unsupported reads not_applicable naming the declaration.
• config_mapping — the subject's own declared [config] rules are set before start and proven delivered through its --dump <path> artifact, with the agent.* pass-through delivered verbatim; a native subject's code-declared launch offers no such seam, so it reads not_applicable.
• metering_self_reported — a probe emits three sentinel batches; the ledger gains exactly three rows, a replayed sequence does not double-count, and Fleet totals equal the ledger exactly.
• metering_engine_observed — for EngineObserved declarations only: a loopback upstream stub, the operator sets the real upstream, forwarded calls commit exactly — a SelfReported adapter reads not_applicable naming its declaration, so no section ever demands EngineObserved of a SelfReported adapter.
• memory — attach → status reports the attachment and the declared delivery fact → a probe receives the managed dir through its declared env var → detach clears it; not_applicable when the declaration maps no memory.dir key — delivery is offered, not imposed. A manifest declaration that cannot be re-read to decide applicability fails the section naming the cause — it never collapses to a fabricated "maps nothing".
• interaction — Guaranteed/BestEffort: send_input reaches a running agent and the echo lands in its captured log; Unsupported fails fast naming the declaration on a probe.

Who is under test (subject vs probe twins)

To be plain about it: capability_edges, lifecycle, pause, and config_mapping (for manifest subjects) exercise the CALLER'S registered adapter; the two metering sections, memory, and interaction prove the same engine seams through small TCK-authored probe twins in the same engine, so a probe failure never damages the caller's run. Honesty ceilings are explicit: a Guaranteed pause declaration on Windows reads not_applicable naming the engine's Windows limitation (never a silent best-effort pass), and a manifest declaring no [config] rules — or only flag/file targets, whose delivery the dump artifact cannot prove — reads not_applicable for the config section.

What the report can and cannot prove

The report never asserts more than its sections exercised. The contract's duties split into two classes (see the contract's own statement):

• Declaration-checked — the sections above exercise these mechanically; a pass is machine-checked evidence.
• Honor-system — v1 ratifies the {env:VAR} render guarantee, self-update pinning, and the config-layer disclosure, but NO section can exercise them (proving them requires the real agent's behavior, not a declaration). They are enforced by review and per-release re-validation, never by a TCK pass; "conformant" is not a claim that an adapter honors them.

The interaction section is also channel-blind: it never reads the declared [interaction].channel, so an adapter that declares http is exercised through the same stdin echo probe as a stdio adapter — its interaction: pass proves the declaration-consistent fail-fast and the engine's stdin seam, NOT that the agent's real HTTP surface works. Proving an HTTP transport is the adapter author's own validation duty (a channel-aware TCK section needs engine-side HTTP send machinery and is a post-v1 change).

The report schema

The report carries a schema_version (currently 1 — bumped only on a breaking shape change; additive fields keep it), the contract_version the run was governed by (the engine's negotiated Adapter Contract version, so a report can always say which contract major produced the pass), and the adapter kind; it round-trips through serde, so a CI gate can pin the shape before trusting the entries.

The kit's own integration tests run the harness BOTH ways: against the mock/manifest fixtures (tck.rs in-crate suites plus the third-party simulation — lifecycle, crash, config, and both metering sources each demonstrated) and against the shipping hermes builtin (crates/ktesio-engine/tests/hermes_tck.rs, under the same recorded hermes_shim PATH-sim sandbox as hermes.rs — never a live gateway): every section applicable to Hermes' declaration passes, the EngineObserved section reads not_applicable, and a dedicated pass drives the hermes SUBJECT itself through attach → start and proves the managed Memory Backing dir reached the subject's process as HERMES_HOME (the subject's own declared delivery — the memory probe twin alone proves only the mechanism on the probe's own env var).

The library host journey (FR-31, story 7-1)

FR-31's promise — a host drives the full UJ-3 flow (register → configure → cap → start → breach → pause → stop) through the engine library alone — is proven by two tests whose expected outcomes are pinned in ONE shared module, so the library path and the CLI path are provably behaviorally identical. The expectations home is ktesio_conformance::uj3: the fixture manifest the flow registers under (contract v1, pause + interaction guaranteed on all three OSes, self-reported metering, a [config.model] env mapping so the configure leg is meaningful), the flow's config keys/values and budget/rate/cap numbers, the committed-state readers + poller (a read-only SQLite connection over the same state-db layout — opened through the engine-published paths::STATE_DB_FILE constant, never a hand-typed file name — that both a library Engine::open root and a kt state dir use, with a short busy timeout so writer lock contention never silently eats the poll budget), the committed JSON-Lines readers for the breach/transition logs (an absent log reads as empty; ONE torn trailing append is skipped — the engine appends while live — while a malformed interior line hard-errors, because that is the wrong file, not a race), the usage-ledger reader + received-stream projection + raw-receiver drain (committed_usage_rows/usage_from_payload/drain_receiver — the ONE comparison shape for the 7-2 "received usage stream == committed ledger rows" guarantee, shared with the 7-3 collision test), and the assertion helpers over the observed FleetEntry/UsageView/event reads. Since story 10-1 the suite-WIDE fixture plumbing lives beside it in ONE shared home, ktesio_conformance::test_support: the parameterized manifest-TOML builder with named presets (uj3::write_flow_manifest is its primary preset; the event-subscription suite's lingering/crash-once/replay fixtures, the perf-budgets heartbeat fixture, and kt agent_cli.rs's fake_agent manifest shapes are the other consumers — each call site is a one-line preset call; since stories 10-2/10-3 the diagnostic-sink and resync suites consume it too), the ONE lag-accumulating try_recv drain for BOTH receiver forms (drain_raw_receiver, which uj3::drain_receiver delegates to, and drain_subscription over EventSubscription — the only difference is the Closed policy), and the fake_agent_bin locator parameterized over the cargo target subdirectory hop (BinDir::TestDeps for test binaries, BinDir::Examples for the perf-budgets example). Two deliberately standalone pieces remain: the embedding quickstart's inline fixture (the host copy-paste artifact — its independence is the point) and agent_cli.rs's raw-body manifest_dir writer (it must be able to produce intentionally INVALID manifests for the failure-path tests). The module is test infrastructure only — never a driving surface; it is a dev-dependency of both suites and never crosses the shipping boundary gate.

The library host test (crates/ktesio-engine/tests/uj3_library_host.rs) drives the whole flow through Engine::open + the blocking() facade only — register a manifest adapter, set every flow config key, read the effective config with provenance, read the Fleet before the flow, start, and assert the breach-pause against committed state: exactly one breach per dimension (the token ceiling and the dollar cap cross on the same event; the independent latches each fire once, and the token breach wins the pause), the running → paused transition carrying the TOKEN BudgetExceeded cause, honest labeled dollar surfaces throughout, then stop to the terminal state. It runs unmodified on all three OSes (committed-state polling, no wall-clock sleeps, no OsId gate) and its module docs carry the §4.1–§4.8/§4.10 reachability inventory (memory and interaction are facade-proven by the hermes e2e and cited, not re-tested; this test is the §4.8/FR-31 discharge). One honesty note about the pause leg itself: what is deterministic on every OS is the committed paused STATE — that is what the assertions pin. The suspension behind it is per-OS: on Unix the breach's suspension is a real SIGSTOP freeze (the fixture's emitter freezes mid-batch); on Windows the Windows backend is cooperative best-effort only — there is no hard suspension, so the emitter keeps running through the pause (the engine's own honesty ceiling rates a guaranteed pause on Windows not_applicable). That per-OS difference is exactly why the post-stop ledger is asserted as a committed range, not an exact total — the emitter can be frozen at event 3, 4, or 5 of its batch (or, on Windows, the batch can run to completion) — so the shared assertion pins the input/output split and the unit-Rate dollar consistency of whatever landed, never a brittle exact figure.

The CLI behavioral-identity journey (crates/kt/tests/agent_cli.rs, uj3_governance_journey_through_documented_cli_commands_unix) drives the SAME flow through documented kt commands only — register --manifest, config set, config get --json, show --json / usage --json (each document's schema_version asserted against the engine's pinned Fleet constant), stop — feeding the SAME shared assertions from parsed documents. Because a standalone kt agent start supervises only for the command's lifetime (kill-on-drop) and any intervening adopting command kills a live process too, the breach leg runs through the file's established surviving-engine subprocess harness: an explicitly ARMED re-exec helper starts the instance in its own engine session, waits for the committed pause, asserts the shared paused-entry shape from its live Fleet row, then exits crash-style so the suspended process survives for kt agent stop to adopt (with a zero stop window — a suspended process cannot act on a graceful signal); a best-effort orphan guard in the parent stops the instance on any mid-flow failure, so a test failure can never leak the suspended agent. Windows cannot simulate cross-lifetime survival (Job-Object kill-on-close), so the journey runtime-returns there per the file's _unix naming convention — OS cfg is barred from kt tests by the CI OS-cfg gate — while every OS-independent shared assertion is proven on all three legs by the host test.

Event subscription (FR-33, story 7.2)

The event bus — Engine::subscribe() for async consumers (a raw broadcast receiver) and Blocking::subscribe() for sync consumers (an EventSubscription whose recv bridges through the engine runtime) — is tested at two levels, both asserting on the SAME durable records the query APIs read; the bus is a delivery surface over committed truth, never a second source of it. The bus primitives are unit-tested in crates/ktesio-engine/src/domain/bus.rs: a publish with zero receivers is swallowed (the no-subscriber engine pays nothing — a publish can never fail supervision), two receivers fan out the full sequence independently and in FIFO order, a receiver stalled past the named capacity (EVENT_BUS_CAPACITY, 1024 — the documented flat memory bound: the shared per-engine ring whose windows every receiver views) observes Lagged(n) naming exactly the dropped prefix and then RESYNCs at the tail, receiving every subsequent publish — and lags AGAIN with a correct second count after a resync — and the EngineEvent wrapper round-trips serde with its kind tags pinned (transition / budget_breach / usage_update) while each payload keeps its own schema-version stamp (the wrapper adds no wire vocabulary and duplicates no version). The end-to-end acceptance suite (crates/ktesio-engine/tests/events_subscription.rs) proves the contract families through a real engine, subscribed BEFORE any traffic: (a) a subscriber of the UJ-3 mini-flow (the SHARED uj3 fixture — the same manifest, config pairs, and numbers the 7-1 suites pin) receives every transition, both breach dimensions, and the usage updates with each payload round-tripping into its versioned struct, the received streams equal the committed instance.log/breaches.log EXACTLY — publish order == durable commit order, with the crossing commit's tail pinned (usage → token breach → pause → dollar breach) — and the received usage payloads equal the committed usage_events ledger rows field-for-field, in commit order; (b) a crash-then-restart leg delivers the crashed and restarted transitions in commit order; (c) a subscriber that never drains cannot stall supervision — capacity-derived real pause/resume cycles (a bounded loop; well past the capacity) all commit and the instance reaches its expected state, after which the stalled receiver observes Lagged and its retained window corresponds event-for-event to the durable log's tail (the dropped prefix stays readable via the query APIs), and a post-lag publish proves the resync; (d) two subscribers each independently receive the full sequence; (e) two interleaved instances keep per-instance FIFO — the deterministic facade-call order is the exact global bus order for transitions, and a second pair of instances with overlapping usage-emission cadences keeps per-instance usage FIFO against the committed ledger; (f) a FAILED durable append publishes NOTHING — with the instance's log file replaced by a directory (an append error on every OS), the obstructed transition delivers nothing while the next successful commit delivers (the failed event appears on neither the bus nor the durable log), and an obstructed breach log still enforces (the pause transition delivers) while no breach event ever publishes; (g) a replayed usage batch (metering.rs's --replay-usage pattern) publishes no duplicate — the received stream equals the committed rows exactly; and (h) a real #[tokio::test] consumer drives the raw Engine::subscribe() receiver (recv().await, bounded) while a plain thread drives the flow through the facade, proving the async surface. The determinism posture matches the house style: every facade call's publishes complete under the supervisor lock before the call returns, so after a committed-state wait plus ONE supervisor-lock-taking read (the barrier) a try_recv-until-empty drain is exact, never racy — no wall-clock sleeps against side effects (the one cited exception is the replay settle, metering.rs's proven pattern), and no OS gate anywhere.

Embed-clean verification (FR-34, story 7.3)

The embedding claims — headless, prompt-free, no global process state, full API behind the blocking facade — are proven, not assumed, by crates/ktesio-engine/tests/embed_clean.rs, which keeps three durable instruments in CI. The two-engine collision test is the strongest embeddability proof: TWO independent engines open in ONE process (different hermetic roots), each driving the FULL UJ-3 flow (register → configure → cap → start → breach → pause → stop) through the blocking() facade with a story-7.2 subscriber attached before any traffic. Both engines are opened and subscribed on the main thread and the flows run one-per-thread with a barrier-synchronized start — the synchronized start is ENFORCED, but deeper step-by-step interleaving of the flows is scheduler luck and is not claimed (the isolation proof does not depend on it); the main-thread setup also means a pre-barrier failure fails the test directly instead of hanging a sibling on the barrier, and thread::scope joins BOTH threads before any outcome assertion. Both flows are asserted with the SHARED uj3 expectations inside their threads (a thread panic propagates through join as the flow's failure), and isolation is asserted on three axes: each subscriber's received streams equal its OWN engine's committed truth exactly through the shared uj3 projection (committed_usage_rows/usage_from_payload — transitions == instance.log, breaches == breaches.log, usage == the usage_events ledger rows field-for-field with run_id; any cross-engine leak would be an extra or misordered event against the local committed records), each root's database holds exactly ONE instance row, and the two engines' Run-id sets are disjoint on BOTH event families — usage AND breaches carry run_id, and each subscriber's received stream is disjoint from the OTHER engine's committed Run-id sets (the same instance NAME ran twice; only the Run ids tell them apart). The drain is exact by the 7-2 posture (every publish completes under the supervisor lock before its facade call returns; the terminal fleet() read is the barrier); every wait is committed-state polling through the shared bounded poller — never a wall-clock guess against a side effect — and there is no OsId gate. The no-TTY/no-prompt/no-global-handler audit greps the engine's production sources for stdin reads, interactive prompt prints (including writeln!/write! aimed at stdio), TERMINAL DETECTION (branching on is_terminal-class APIs violates no-TTY without a prompt), env MUTATION (reading KTESIO_STATE_DIR is fine; set_var/remove_var at engine runtime is not), process-global handler installs (signal/panic/console hooks, tokio::signal, the raw signal( C entry point, atexit), lazy/global cells, and runtime constructions outside Engine::open's owned per-engine runtime. It is honest in both directions: the scanner blanks string-literal contents and strips comments before token matching (a log message containing "stdin" cannot match), skips test modules as REGIONS so production code declared after a test module cannot escape the scan (an earlier truncate-at-first-marker heuristic had exactly that blind spot), treats not(test) gates as production code, and panics (naming the file) on an unreadable source rather than silently meaning "unscanned"; and since story 10.2 there is exactly ONE named allowlist entry, the RUN_NONCE global static (the RunId uniqueness tie-breaker: behavior-neutral, guaranteeing PER-PROCESS uniqueness, which is what a multi-engine single-process host needs; cross-process uniqueness is not claimed — separate roots keep separate ledgers). The audit's two HISTORICAL print-site entries — the supervisor's stderr diagnostics (the DC-10 memory-delivery notice and the enforcement breadcrumb) — were closed by story 10.2, not allowlisted forever: both now route through the host-provided diagnostic sink (Supervisor::emit_diagnostic, with stderr as the no-sink default), the print-site allowlist is EMPTY so any new raw print fails CI outright, and the sink plumbing itself carries positive count==1 pins (the choke point, both diagnostic routes into it, and the single io::stderr() default arm) so a diagnostic cannot silently disappear, be reworded, or grow a second direct stdio writer. The blocking-coverage inventory audit enumerates pub async fn across the WHOLE production crate (an impl Engine block in another module cannot escape the count) and asserts every entry point has a Blocking counterpart — matched by NAME AND SIGNATURE (whitespace/trailing-comma-normalized parameter lists), with the bridged subscribe and the story-10.2 sink installer with_diagnostics as the only intentional sync-side extras — that bridges via block_on (FR-34's "covers the full async API"); the companion kt facade-only audit asserts kt's production sources never touch an engine async API or a runtime (no .await, no async {, no tokio::, no futures::, no block_on, matched on comment- and string-cleaned code lines), positively drives .blocking() on a code line, and pins the "no tokio dependency" claim structurally against kt's manifest (TOML-comment excluded) — the source-level companion to the AD-2 build-level boundary gate (whose full build-level proof remains story 7.4's).

The diagnostic sink (story 10.2)

The host-provided diagnostic sink is tested at three levels in crates/ktesio-engine/tests/diagnostic_sink.rs, always through the REAL production paths (never a synthetic emission): the DC-10 memory-delivery notice fires via the attach-unmapped-manifest-then-start path, and the enforcement breadcrumb fires via a budget breach with breach_action = pause on an instance the token breach has already paused (the dollar dimension then fails to enforce a second pause — the exact production path the 7-1 suites exercise; the enforcement site evaluates the TOKEN ceilings first and the pause commits synchronously in the same supervisor-lock pass). The sink-installed engine tests prove both diagnostics arrive in the sink as the EXACT expected lines, in emission order, with NOTHING else (the capture equals the two lines byte-for-byte): the expected lines are computed from the run itself — the engine-reported managed directory, and the same transition-gate error the enforcement path received, reproduced by pausing the already-paused instance through the facade — once through Engine::open_with_diagnostics (the airtight open-time install) and once through Blocking::with_diagnostics (the post-open install). The no-sink default is pinned byte-identical by a subprocess re-exec of the test binary (the adoption.rs pattern — std cannot capture a process's own stderr): the child drives the same flow with no sink and the parent asserts the captured stderr carries EXACTLY the two [ktesio] lines and nothing else, while the sink-mode child proves the complementary row — with a sink installed, the child's stderr contains no [ktesio] line at all and the relayed sink bytes equal the expected pair. Determinism follows the house style: committed-state polling (the shared uj3 poller) or polling the sink's own captured bytes, never a wall-clock guess; no OsId gate. Beyond the two acceptance rows, the suite pins the contract corners the docs promise: a mid-flight ROTATION splits the diagnostics across two sinks (sink A holds exactly the first line, sink B exactly the second — installing replaces, which an if diagnostics.is_none()-shaped install would silently break), an always-failing writer is swallowed with supervision continuing, a writer that panics on its first write is caught (the supervisor mutex never poisons) while the same sink receives the next diagnostic, and one shared sink Arc serves two engines in one process. The audit teeth over the same feature live in the embed-clean suite (above): zero print sites in production sources, plus the count==1 positive pins on the sink choke point, both routes, and the single stderr default arm — the stdio-reach count scanned across fully-qualified calls, bare imported-path calls (use std::io::stdout; then stdout()), stdio imports, and hand-written _print internals. The no-sink stderr verification is pinned end-to-end in TWO places: the engine-level subprocess suite in crates/ktesio-engine/tests/diagnostic_sink.rs itself (the child re-exec described above — same wording, same stream, one line each, byte-exact), and the kt CLI end-to-end pin in crates/kt/tests/agent_cli.rs (a_start_with_an_attached_but_unmapped_memory_backing_says_so_and_still_succeeds — an unmapped memory-backed start prints the notice on the real kt process's stderr, names the reserved key, and never touches stdout).

The event-bus resync helper (story 10.3)

The crash-window backfill (Engine::resync_events / Blocking::resync_events, story 10.3's remedy for the bus's documented at-most-once crash window) is tested in crates/ktesio-engine/tests/resync_events.rs, on the same committed truth the 7-2 suite compares the live bus against — and with the same house posture (committed-state polling, the fleet() supervisor-lock barrier, no wall-clock sleeps, no OS gate). The five families: (a) the crash-window simulation — the whole shared-uj3 mini-flow (register → configure → start → breach-pause → stop) commits while NO subscriber exists (the harshest miss: every publish landed unheard), then ONE resync_events call returns exactly those events — the backfilled transitions equal the durable instance.log via the query API exactly, the breaches equal breaches.log, the usage payloads equal the committed ledger rows field-for-field through the shared committed_usage_rows/usage_from_payload projection, each in its family's commit order, every payload round-tripping serde with its schema stamp, the count equal to the union of the three families (nothing fabricated, nothing dropped), and a re-resync from the returned cursor EMPTY (idempotent); (b) backfill-then-subscribe continuity — phase one commits with no subscriber, the host backfills, THEN subscribes live, phase two commits and delivers, and for every family backfilled ++ received equals the committed record EXACTLY: the backfill is precisely the prefix, the live stream precisely the suffix — no duplicate at the seam, no gap anywhere (the resync-first ordering, one of the two documented orders, held end to end); (c) cursor continuation — a resync consumed mid-stream, the returned cursor passed back after further commits, returns exactly the new tail (never the consumed prefix), and a cursor past a family's committed count (a truncated/rotated log) is a TYPED ERROR naming the family and both counts — never a silent clamp that would re-deliver or silently skip; (d) honest edges — a malformed name fails InvalidName, an unregistered name fails NotFound (a mistyped name must not read as a silent empty backfill — the read_agent_log precedent), and a torn trailing append (hand-written into a real instance's log after a real start/stop, the very damage the crash being healed would leave) is skipped WITH the skip surfaced (torn_tail_skipped on the batch — the engine's next append fuses onto the torn fragment, so the host must know a record may be missing): the good prefix backfills, never a failed recovery; and (e) the wire pins — ResyncCursor/ResyncBatch serialize snake_case and round-trip (the documented host-persistence contract), with torn_tail_skipped additive (#[serde(default)]: an archived batch without the field still loads). The helper's pure parts carry their own unit tests in domain/resync.rs: the family-major assembly + per-family cursor skip/advance math, torn-trailing-line tolerance bounded to the torn-append signature (ONE unparseable, non-newline-terminated trailing line skipped — with the skip surfaced), surfaced errors for every other malformed line (an interior line with its PHYSICAL line number — blank lines never shift it, a newline-terminated trailing line — corruption or a fused post-crash line, and a valid-JSON-wrong-shape trailing line — the wrong file), the truncation guard through read_committed (a cursor past a family's committed count errors naming family and counts), and absent-log-as-empty. The store gained the ledger-row read underneath (SqliteStore::usage_events, rowid-ordered — insertion order IS commit order on the append-only ledger; deliberately an inherent method, not a StateStore trait extension, so the published port surface stays frozen), so the resync reads the SAME committed-truth machinery the query APIs serve, never a side channel.

Performance budgets (NFR-4, story 7.5)

The NFR-4 budgets — read commands < 1 s on a 25-instance Fleet, supervision overhead ≤ 2% CPU and ≤ 50 MB RSS per running instance — are measured, not assumed, by the perf-budgets harness (crates/ktesio-engine/examples/perf-budgets.rs). It builds a real 25-instance Fleet fixture (manifest agents on the conformance fake_agent; 10 RUNNING via the facade start, the rest registered-only) on a hermetic temp root, drives the PUBLIC blocking() facade only, and measures in a fixed order that keeps the fixture startup entirely outside the timed windows (setup → settle → steady-state window → re-settle → read latency → a subscriber-overhead addendum, below):

• Steady-state supervision overhead: with the 10 running instances idling (--heartbeat-ms 1000 — heartbeat only, no usage emission, so the engine supervises but does not meter), the ENGINE process's aggregate CPU% and RSS are sampled once per second over a 12 s window after a 2 s settle; the first sample of each window is discarded (it carries the preceding phase's residue), and per-instance = figure / the actually-Running count sampled at measurement time. RSS is the engine process's resident set ONLY — the supervised agents' own memory is their own (separate processes), not supervision overhead; the allocator may retain pages from earlier phases in the figure, which is why the mean is gated while the max is reported and spike-guarded at 2× the budget. CPU% is sysinfo's two-refresh delta for the engine process, which is the harness process itself (the harness owns the Engine and sleeps between samples, so the figure is idle supervision cost, not measurement-loop cost). Window coverage honesty: 12 s validates heartbeat-frequency supervision work (the reaper cadence + per-second log captures); LONGER-period tasks (sweeps, retention) are not captured — a longer local window is available via PERF_BUDGETS_WINDOW_MS (clamped to [10 s, 1 h]).
• Read latency over 200 iterations (tunable via PERF_BUDGETS_READ_ITERATIONS, hard-capped at 100 000 with a loud stderr note — an absurd override must not multiply the run time unbounded) each of the read kinds kt itself uses: fleet() (agent list), instance_status() (round-robin across the Fleet), the effective_config() read (config get, with the configured value asserted), and the per-instance usage read — labeled usage_via_fleet because it mirrors agent usage <name>'s exact facade shape (fleet() + find) and therefore INCLUDES the fleet listing's cost; it is not an independent read kind. p50/p95/p99 are reported per kind.
• Subscriber-active overhead (GATED at the ratified budgets, story 10.3): NFR-4's gated figures are zero-subscriber figures by design, but a host using the subscription surface needs a budget covering fan-out cost — so after the gated measurements the harness attaches ONE active subscriber, re-measures a single steady-state window plus the same read kinds, and the deltas vs the unsubscribed baseline (the report's subscriber_overhead block) feed three hard gates (below). The addendum still runs after the gated phases so it cannot contaminate them, and the harness's startup liveness check counts its window against the fixture's 60 s orphan bound (a schedule that outgrows the bound exits loudly instead of measuring self-exited idlers).

The subscriber-active budgets were ratified from the observed measurements (story 10.3): the 2026-09-10 local release run (row below) measured a −0.015 percentage-point CPU delta, a +0.01 MiB RSS delta, and a +0.7 ms read-p99 delta — one active subscriber's fan-out cost is effectively zero, so the ratified ceilings sit deliberately far above observed: ~16× the CPU-delta magnitude, and two orders of magnitude on RSS (~200×) and read p99 (~143×) — sized for measurement noise (the addendum runs ONE window even on CI; the shared-runner tolerance is its other noise absorber) while any real fan-out regression — a per-publish broadcast storm, a per-event host callback — still trips them. The budgets were ratified on a macOS local (strict) run and are enforced on ubuntu CI with the same factor shape; Windows measures and reports only (the gate-platform scope below). The budget consts are pinned by the example's gate-math tests, so changing a number is a conscious re-ratification that must update this table with the new observations.

The harness prints a machine-readable JSON report (schema v1, with environment metadata — git SHA, OS, CPU model + core count, CI flag, tolerance factor — and per-gate strict-budget margins) to stdout; the measured record of record is the CI perf-budgets job log, whose JSON report is also uploaded as the job's workflow artifact — local runs print the identical report. It then gates:

The CI tolerance policy (named, not hidden): the CPU gate carries a shared-runner tolerance factor of 1.5 (shared_runner_tolerance, printed as ci_tolerance.factor in every report) AND takes the median of three 12 s windows on CI. GitHub hosted runners are noisy neighbors — a co-tenant can steal cycles from under the sampling window — so in CI (detected via the runner's CI env; force any mode with PERF_BUDGETS_CI=1|0, where any other value exits 2) the CPU gate runs at an effective 3% per instance over the median window. The three story-10.3 subscriber-active delta gates use the SAME factor shape but deliberately stay single-window on CI (the addendum measures one window even there, and the deltas are differences taken seconds apart in one process), so for them the tolerance is the sole noise absorber and the ratified budgets are sized for it. This is NOT a budget change: the strict budgets are what local runs gate, reads and RSS gate at budget on every gate platform, both contamination directions of the window sampling are acknowledged in the report, and the factor is printed in the job log on every run. If a budget ever fails on honest local measurement (not CI noise), the story-7.5 remedy is a documented budget-update proposal routed to the maintainer — never a silent gate tweak. Note the budget semantics: the per-instance figures mean MARGINAL supervision overhead — the fixed engine cost amortizes over the running count — so the 25/10 fixture counts are part of the budget's meaning and a future change to them must consciously revisit the budget.

Gate math is pinned, not just printed: the pure evaluation (percentiles, worst-kind selection, per-instance normalization, CI-mode parsing, tolerance selection, gate-bound strictness — the read gate's strict < must fail a p99 exactly at 1000 ms — pass/fail, the read-iteration cap, the orphan-bound liveness schedule, the subscriber-overhead delta math, and the story-10.3 subscriber delta gates over the ratified budgets with their tolerance shape) lives in testable functions covered by the example's own #[cfg(test)] module — cargo test --all-targets executes them, so a budget or evaluation change must consciously edit those assertions and the alarm cannot silently weaken. The 25/10 fixture counts themselves are pinned by a test too (the NFR-4 premise the budgets are written against), as are the three ratified subscriber-budget constants.

The measured record (local runs; CI runs append their own via the perf-budgets job):

(One column per measured read kind, named as the report names them — the earlier "status" header was a mislabel: that column carried instance_status's percentiles, not a pass/fail status. The gate status lives in the report's gates array, not this table.)

Run it locally from the workspace root:

cargo run --release --example perf-budgets -p ktesio-engine

The harness is an EXAMPLE deliberately: cargo test --workspace --all-targets compiles it and runs its gate-math unit tests but never executes the harness — the ≥ 10 s wall-clock measurement stays out of ordinary test runs without any #[ignore] sprawl. The designated CI perf job (perf-budgets, ubuntu-only, blocking) builds the release example plus the fake_agent helper (carrying the same stale-helper rm + explicit-rebuild guard the test and coverage jobs use — the harness spawns agents) and runs it; a reads or RSS regression fails the job.

Process metrics come from sysinfo — a DEV-dependency of ktesio-engine only, the spec-record-sanctioned choice (dev-only, benchmark-gated; NFR-8's runtime-lean policy governs runtime deps and is untouched — cargo tree -p ktesio -e normal,build is unchanged by the harness, and the msrv job builds dev-deps on the pinned floor so a future sysinfo MSRV bump reds CI). Gate platform scope: enforced on ubuntu CI and strict locally on macOS; Windows figures are measured-and-reported only (sysinfo's memory() is the working-set-size analog there) — not a ratified gate platform. Units: the gate is 50 MiB/instance (the report is byte-derived); this reconciles with NFR-4's "50MB" prose intent.

Cross-platform testing (3-OS matrix)

The per-OS process-control code lives only under crates/ktesio-engine/src/backends/{unix,windows} (the sole place OS-conditional compilation is allowed). This code cannot be verified on a single operating system — the Windows Job-Object backend does not even compile on Linux. The CI test job therefore runs on a matrix of ubuntu-latest, macos-latest, and windows-latest:

• Linux and macOS run the Unix backend (process groups, SIGTERM/SIGKILL, and SIGSTOP/SIGCONT for the guaranteed pause) on both Unixes.
• Windows runs the Windows backend (Job Objects, TerminateJobObject), the only place its behavior — real spawn, terminate, no-survivor, and the cooperative best-effort pause — is actually exercised. The best-effort pause is honest by surfacing a qualifier (a pause-best-effort transition cause plus a CLI stderr note), never a silent fake; that path rides this windows-latest leg and is compile-checked only on Unix.

The self-reported metering tests (tests/metering.rs), the budget-enforcement tests (tests/budget.rs), the dollar cost / Cost-Cap tests (tests/cost.rs), the engine-observed metering tests (tests/observed_metering.rs), the library host journey (tests/uj3_library_host.rs, above), the event-subscription suite (tests/events_subscription.rs, above), the embed-clean verification suite (tests/embed_clean.rs, above), and the diagnostic-sink suite (tests/diagnostic_sink.rs, above) are deliberately OS-agnostic and run unmodified on all three legs — they keep the engine alive in-process (no cross-lifetime survival) and assert on committed SQLite state, the committed event logs the bus mirrors, or the sink's own captured bytes (the no-sink subprocess legs capture a child's stderr, which is uniform across OSes), so there is no wall-clock racing and no OsId-gated skip; the sentinel-line emitter (fake_agent, and its --observed-calls HTTP client), the parsers (the sentinel-line parser and the OpenAI usage parser), the loopback forward listener (portable async TCP + HTTP over hyper, in the engine core — not backends/), the local upstream stub, the budget evaluator/config parse, and the money math are all pure std/portable with no OS cfg. This is by design (retro AI-37/AI-38, codified by story 11-5): the 3-OS matrix runs from every commit — full CI fires on every branch push, nightly, and per PR (the CI policy above) — so any cross-OS fragility surfaces immediately rather than after code is written to a Linux-shaped assumption, and loopback HTTP interception is OS-uniform, so it is one story, never an OS split.

The former Linux-only _live tests (AI-35/AI-38, story 11-5) run on all three legs too. Their spawn+observe is a readiness handshake, not a wall-clock race: the manifest passes fake_agent --marker <path>, and the test waits for the marker file (written by the agent at startup, before its --dump observation file) and then for the dump, with generous bounded deadlines — so the old "fragile spawn latency on macOS/Windows CI" gates are gone. The CLI-level secret no-leak matrix starts its agent through the surviving-engine subprocess harness for the same reason, so the positive-delivery observation survives even the Windows kill-on-close that follows the helper's exit (the startup artifacts are on disk before it). Windows-correct survival/adoption semantics are likewise asserted POSITIVELY on the Windows leg (AI-29): engine death kills the whole agent tree via Job-Object kill-on-close, and the orphaned records reconcile to failed — the exact inverse of the Unix "child survives the crash" harness, with each OS's tests carrying pointer comments to their per-OS siblings.

Only the test job matrixes; the other jobs (fmt, clippy, build, docs, boundary, semver, msrv, audit, coverage) stay Linux-only.

Coverage

The coverage gate is a workspace-aggregate line coverage >= 95%, measured with cargo tarpaulin --engine llvm, on Linux only. Locally a single --workspace pass is the simplest way to reproduce it:

cargo install cargo-tarpaulin
cargo tarpaulin --engine llvm --workspace --fail-under 95

CI computes the same aggregate a different way — a per-crate split (#101). Instead of one --workspace run, CI runs cargo tarpaulin -p <crate> once for each of the five workspace crates (ktesio-adapter-api, ktesio-conformance, ktesio-adapters-hermes, ktesio, ktesio-engine), writes one LCOV tracefile per crate, merges them with lcov, and enforces >= 95% on the merged result. This is a workaround for the hosted runner's memory limit: a single --workspace instrumented run — the whole graph (tokio + bundled-SQLite C + a 13-crate hyper stack) plus all ~1075 tests, instrumented — overflowed the 7 GB runner even after every other lever (dedicated cache, serial tests, swap, disk reclaim, line-tables debuginfo); running one crate at a time keeps each instrumented run's live process tree small enough to fit.

The split is coverage-neutral because tarpaulin reports coverage for every workspace source file each built test binary touches, not just the -p crate's own source — so a ktesio-engine line exercised only by a kt CLI test (which drives the kt binary, and ktesio_engine in-process, from crates/kt/tests/agent_cli.rs) is still credited under -p ktesio. The union of the five per-crate tracefiles therefore reproduces the --workspace number exactly: locally the merged aggregate is 95.22% (5718/6005 lines), identical to the single --workspace pass to the line (verified: zero lines lost, zero gained). The merge passes lcov --ignore-errors inconsistent because lcov 2.x rejects a benign function-start-line drift tarpaulin emits across separate runs — that inconsistency is in function metadata only and does not touch the DA/LF/LH line records the gate reads. The crates run lightest → heaviest (ktesio-engine last), each inside its own ::group:: with a free -h/df -h / dump first, so if the heaviest crate ever still exhausts the runner the lighter crates' numbers are already logged and the failure is attributable to a specific crate rather than one opaque --workspace OOM.

Both CI and local use the --engine llvm (source-based) engine. On macOS the default ptrace engine is unavailable outright — tarpaulin errors with missing section: CoverageFunctions — so llvm is the only option on a macOS dev host; running the same engine on CI keeps the two on the same reported percentage. CI adds the llvm-tools-preview rustup component, which provides the llvm-profdata/llvm-cov the engine shells out to.

CI also keeps a dedicated cargo cache key for coverage (<os>-cargo-coverage-<lockhash>), separate from the other jobs. Tarpaulin compiles the whole dependency graph with coverage instrumentation, whose fingerprints differ from the normal-profile artifacts the other jobs cache — so a shared key gave coverage nothing reusable and, because the job runs last, never saved its own instrumented target either. The effect was a full cold instrumented recompile every run, which is what blew the coverage timeout (AI-23) — not the engine, and not the test run. The dedicated key persists the instrumented target, so only the first run pays the cold-build cost.

The coverage step is also tuned to survive the hosted runner's limits, and this is what motivated the per-crate split (#101). The instrumented build (coverage counters + the fleet/adoption suite each spawning several child processes) overflowed the 7 GB runner and it "lost communication" — an OOM that kills the whole job with no log. Four levers, still applied to every per-crate run, reduce the pressure: RUST_TEST_THREADS=1 runs the instrumented tests serially (one subprocess-spawning test's process tree resident at a time); an 8 GB swap file on the /mnt temp disk absorbs any RAM spike; CARGO_PROFILE_DEV_DEBUG=1 builds with line-tables-only debuginfo, which shrinks the instrumented binaries (less RAM) and the target/ on / (less disk) while leaving coverage unchanged (llvm line-mapping needs only line tables); and the step frees ~30 GB of unused preinstalled SDKs so the instrumented target/ — which lives on /, not the /mnt swap disk — has room. Those four were not enough on their own — a single --workspace instrumented run still OOM'd — so the per-crate split (above) is the decisive lever: it never holds the whole workspace's instrumented tests resident at once, only one crate's, while the merge preserves the same aggregate 95% gate. (The dedicated coverage cache still persists the one shared instrumented target/ all five per-crate runs build against, so the split does not multiply the cold-build cost.)

Generate an HTML report:

cargo tarpaulin --out Html

Coverage honesty for per-OS code

cargo tarpaulin runs on Linux and cannot instrument #[cfg(windows)] code, so the 95% gate is measured on Linux against the OS-agnostic core plus the Unix backend (which compiles and runs on the Linux tarpaulin host, so its lines are covered). The Windows backend's lines are cfg-excluded on Linux and never enter the Linux coverage denominator, so the reported percentage is honest for what Linux can see. The Windows backend's correctness is proven instead by the windows-latest matrix test run passing — a real Job-Object spawn, terminate, and no-survivor check — not by a coverage number. The fake_agent helper binary runs only as a spawned subprocess, so it too is excluded from coverage (its behavior is proven by the tests that spawn and kill it). The story 6-2 hermes_shim launcher is excluded the same way: the integration test copies it to a temp dir and launches it via the engine's PATH resolution, so its lines can never be recorded.

Documentation Checks

python3 scripts/check_docs.py

The docs check validates:

• Root and docs/ Markdown links.
• JSON fenced code blocks.
• Documented kt command examples.
• Stale links to old repository names or generated spec quickstarts.

Release Script Checks

python3 scripts/generate_release_docs.py v0.0.0 --output-dir target/release-docs-test

This verifies the release-note generator can handle a first-release style tag when no previous tag exists.

Automation Helper Tests

PYTHONDONTWRITEBYTECODE=1 python3 scripts/test_automation.py

These tests cover release-note and changelog rendering, Homebrew formula generation, installer dry-run decisions, and CI/workflow expectations.