# Zero-downtime deploys: roll notes (plan 6, task 17)

These are the notes for putting zero-downtime deploys into production, gate
by gate. They are written, not run. Nothing in this file has been executed
against a box, a bucket, a console or an AWS account, and every command
below is quoted for Paul to run or approve.

Spec: `2026-10-07-zero-downtime-deploys-design.md` (rev 17), section 15 for
the order. The plans are `2026-10-07-zero-downtime-deploys-0-index.md` and
plans 1 to 6, in `platform-design-zero-downtime-deploys`.

## Ground rules

- **Approval.** Every gate that touches dev or the flagship is production
  infrastructure (AGENTS.md, "Production infrastructure needs scoped
  approval"). Before running one, get Paul's explicit approval for that gate,
  naming the boxes. Approval for one gate is not approval for the next.
- **AWS profiles.** Assumed-role profiles need the Bash sandbox off for that
  command. Dev is profile `kass`. The flagship is profile `keymail`.
- **Order.** Boxes first, then the console's swap-policy settings, then the
  flag: dev, then one flagship box with a fresh probe app (Gate F), then the
  rest of the flagship. The deploy docs and release notes carry the migrations rule
  before the flag goes on across the fleet. Plan 5 keeps the flag off
  outside dev until this plan has merged.
- **The flag.** `CARLOS_SWAP=overlap` in a box's `/etc/carlos/host.env`. Only
  the exact word `overlap` arms it. Anything else, or nothing, leaves the box
  on today's stop-then-start paths.
- **Scratch.** The probe's scratch lives under `$TMPDIR`, never `/tmp`.

## Before Gate E: owed

**Current state, 2026-10-14.** This list is copied from amadan discussion
#272 ("Zero-downtime deploys: owed before CARLOS_SWAP can go on"). Every
item on it that blocked the flag is now closed, or is one of the open items
named below. The subsections after this one are the history of how each
item closed, kept for their commit references. Where they say "pending
merge", read the state here instead.

- **Merged and released.** Readiness rounds 1 to 3 (`zdd-flag-readiness`,
  `zdd-flag-readiness-2` and `zdd-flag-readiness-3`). Rev 19, the generation
  socket names, is merged as `ebddcfc` and released in v0.27.2. #292 is
  merged as `e15ad6b` and released in v0.27.3.
- **The flag.** Gate F passed on annie-welding, and Gate G armed the other
  three flagship boxes. All four flagship boxes are armed as of 2026-10-14.
  Dev is being armed on 2026-10-14.
- **Kernel checks: closed.** On 2026-10-14 a root test runner ran them on
  Debian 13 arm64, kernel 6.12.96, origin/main `2683f16a`, as `User=carlos`
  with `Delegate=yes` and `cpuset cpu io memory pids` in the unit's
  `subtree_control`. All passed and none skipped:
  - `TestClaimContainmentCleanupRespawn`, all three subtests;
  - `TestGenerationStartRefusedBesideADirectParentMember`, with its sibling
    and recovery controls;
  - the `ensureParent` positive control, which is `internal/cgroup`'s
    `TestGenKernelGenerationRefusedBesideAFlatMember`
    (`CARLOS_GEN_KERNEL=1`) and `TestGenKernelShimRespawnAfterKill`
    (`CARLOS_SHIM_KERNEL=1`).

  The details are on #272.
- **Mixed-version retries.** Dev round 4 passed three cases: a new box with
  a new console, a book that had not saved, and an old box with a new
  console. A new box with an old console was not run. That case is covered
  by construction. An old console can only write an unsigned retry, and
  every new box ignores an unsigned retry, so the worst it can do is a retry
  that does nothing.
- **Owed items from settings plan 4** (branch `zdd-owed-items`):
  - Closed: a cold wake's build-state record is now written at the exec,
    after the replica-key gate and the cgroup refusals, so a refused launch
    leaves no record. The exec is decided again after the write, and a
    nested seed writes its obligation before its record (Astra's review of
    the branch, P1 and P2). This was owed before O8 (replica keys) is
    switched on.
  - Closed: `release.ResolveLivePath` now calls `release.ResolveLiveLink`.
    The two had identical bodies.
  - MOVED to #250: leftover `consumed/` files refuse the claims census. See
    "Still owed (code)".
- **Claims-on swaps** remain a future feature. A box with host claims on
  declines the flag with the reason code `host-claims-on`.

The flag stays off on any box that has not passed its gate, and only Paul
closes items on #272.

### Correctness, from Astra's plan 5 review (`CODEX-FINDINGS-ZDD-IMPL-5*.md`)

- N5 to N7: gaps in the hold-ran marker. N7 is a disagreement over how it
  treats an existing failure record. Closed on zdd-flag-readiness (b402bc0e,
  42865b81; N6 with 0d6029b3, 3e875ff2; final review round 1: 1e842f89,
  c327f703, e72e68d2; final review round 2: fb55d199), pending merge. That
  covers markers keeping records, blocking the worker fallback and signed-rollback provenance, with marker =
  installed = good boots (ruling 205). N6 and the retry ordering use a
  persisted book sequence and per-execution markers, so a pre-retry outcome
  never undoes a retry. The round 1 fixes: an execution's identity carries
  the agent process's incarnation, so a run after a restart is never taken
  for the run before it, and a rollback or retry that matched the old run
  clears nothing of the new one (1e842f89); a retry that removes only a
  marker still orders a late failure of an execution that began before it
  (c327f703); and a host retry of a build the instance does not want yet
  keeps its canary for that build (e72e68d2). Round 2 found e72e68d2 kept
  only one entitlement per host, so a retry of a second restricted build
  took away the first build's canary, and the first build could then never
  be retried again (the book already listed the host as retried for it).
  Since fb55d199 entitlements are kept per host and build: each is spent
  only by the adoption onto its own build, comes back if that adoption is
  proven to have run nothing, and is dropped once the book no longer lists
  the host as retried for that build.
- F5: failure-record durability across a failed write followed by a restart.
  Closed on zdd-flag-readiness (8836e521: write-ahead intents, schema-2 book,
  startup reconciliation; f46d3142, 4416872e: a shutdown join so planned
  restarts leave no intent), pending merge. A box that declined the flag
  also reconciles an armed run's intents into records, but its save of the
  reconciled book is an attempt: a failed save is logged and the box carries
  on with the records in memory, and reconciles the intents again on its
  next start.
- Ruling 206: an uncertain nested start is a hold failure. Closed on
  zdd-flag-readiness (c453a798), pending merge.
- Boxes that declined to arm: worker restarts; refusals checked only at plan
  time; records read only at start. Closed on zdd-flag-readiness (fa7d7374,
  e32ac0c6, e327ae0e; final review round 1: 076bd9c0), pending merge. A
  declined box now gates every launch at exec time, including workers,
  legacy sidecars, claims and move launches. A move launch runs on a row the
  move holds (landing, restoring), and until 076bd9c0 such a row read as
  unrestricted; it is now restricted as its ready row is, and a signed retry
  that arrives while the row is held is applied rather than dropped. It
  honours markers, and signed retries and the pointer pass. Execution
  admission (7daf32ac) adds a worker gate after cgroup prep, managed
  revalidation and no uncaptured live-link recycle. Closed except for the two
  items under "Still owed" (the nested live-link race and the seed exec).
- The 15-item list in round 3, section 5. These code defects on the flag-on
  path are closed on zdd-flag-readiness, pending merge: policy cache, bounded reads and acks
  (f46d3142, 4416872e); clean one-shot good (but see "Still owed": its good
  publication does not yet check ownership or fencing), the WakeWorker budget
  and the halt event (5ad55485); GoodVersion bucket retention and the hold build
  published before its incumbent stops (d3d464cd); the stage relay give-up
  (2c5adbe8); transient status (98c1763d, 61b83e27). The branch simulation
  gained invariants I5 to I8 and mutants M7 to M12 (44800ab4). The 3000-seed
  sweep passed and M1 to M12 were all caught; it was run again after the
  final review round 1 fixes, which changed gate logic, on 3ee4f1d1: both
  baselines passed and M1 to M12 were all caught again. It was run once more
  after the final review round 2 fix, which changed queue admission, on
  fb55d199: both baselines passed and M1 to M12 were all caught. It does not
  cover the restart and replay sequences of round 1's findings 1
  and 3, which have unit tests instead. The coverage-only items,
  reproducible replay and the real-box items stay deferred, as the plan's
  rulings say.

### Carried from plans 2 and 3

- With the flag off, a retained wake is proven by its leader's exit only;
  forked descendants are not checked (F6, pre-existing). The failed-wake half
  is closed on zdd-flag-readiness (7c0f2543, 23d42f0f; final review round 1:
  bbb7fc8f), pending merge: a failed wake now proves its host cgroup empty,
  deciding from the layout its launch used rather than the layout the row
  reads at cleanup (bbb7fc8f). The failed-seed and normal-stop halves are
  still open, see "Still owed".
- Leftover executable files are not swept from `/run`. Closed on
  zdd-flag-readiness (7c0f2543, 23d42f0f), pending merge.
- Claims-backed swaps are refused (P2-9). Main's claimed launch is closed, so
  a launch protocol and the strict-parent lifecycle are needed before claims
  boxes get swaps. Handled on zdd-flag-readiness (cd7de723) by declining the
  flag on a claims-on box, see the claims reply below. Claims-on swaps
  themselves are a future feature.
- Strict hosts get no swaps until a claim path exists (P2-8).

### Dev-box checks (need Paul's approval to run)

- D1: `daemon-reload` cost at titogo-scale unit counts. Above about 1 s per
  swap means transient units instead of drop-ins. (Gate E below runs this
  measurement.) Done: well under 1 s, so keep the drop-ins.
- The plan 2 kernel cgroup suite on arm64 with `User=carlos`.
- Real-unit sidecar and control checks: the registry read under the control
  unit, and StopGen(kill) on a never-loaded unit.
- A guard against legacy cgroup parents that still hold processes. Code
  fixed on zdd-flag-readiness (2c5adbe8), pending merge. See "Dev-box flag
  blockers" below for what is still owed on the dev box.

#### Dev-box flag blockers

Two defects blocked the flag on the dev box. Both are fixed, merged and
released, and both kernel tests passed on a real kernel on 2026-10-14 (see
"Kernel checks" at the top of this section).

- The legacy-parent guard (2c5adbe8). Its kernel test is
  `TestGenerationStartRefusedBesideADirectParentMember`.
- The `cgroup.kill` respawn (c76a1d44, 66c92efe). A cgroup that took
  `cgroup.kill` is recreated, and KillHost and KillSidecar respawn. Its
  kernel test is `TestClaimContainmentCleanupRespawn`.

### Surfaces (plan 6)

- Copy review of all event and refusal wording is closed. Paul approved all 149
  user-facing strings on 2026-10-11. The approved text is pinned in
  `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-copy.json` and
  guarded by `approved_copy_zdd_test.go`. That includes the `WakePlan.Refuse`
  texts, "restart refused: rolling restarts are unavailable on this box",
  the refused-wake texts, and main's "sidecar could not be restarted. retrying"
  line, which now fires more often.
- A member-safe wake reason for `ErrBuildNotPublished` (plan 4 R22).
- `carlos sidecars ls` shows `ServingVersion`.
- Roll notes: boxes first, then console policy, then dev, then
  `bab/adopt-probe` under load with zero failed requests, then the fleet.
  (This file.)

### Added by plan 6's final review (not in #272)

This item comes from Astra's final review of this branch, round 1, not from
the discussion. The controller parked it as a flag-on blocker, so it belongs
on #272 too.

- Host retries ride the unsigned restart-scope log; a box that can write the
  bucket can clear another box's failure hold (Astra plan 6 R1 #5, plan Q4).
  Superseded on zdd-flag-readiness-2 (`6e9e84bd`): grants ride
  `host-retry.json`, not the restart-scope log.
  Paul's ruling (2026-10-11): host retries stay on the unsigned restart-scope
  log for now, with the same trust as `carlos restart`. Signing retries in the
  console, as `carlos promote --retry` is signed, is owed before the flag goes
  on anywhere hold mode is used. Paul is adding this to #272. Closed on
  zdd-flag-readiness (d68a1dde; final review round 1: c12c35c3), pending
  merge: the console signs a host retry as a grant bound to the
  restriction's epoch and to the identity of the box's failure book, and
  every box ignores an unsigned retry. The book identity (c12c35c3) stops a
  grant for one book's restriction clearing another book's at the same
  epoch. Every open of the book also advances its sequence by 2^20 (about a
  million), so a run does not hand out again an epoch an earlier run handed
  out but never saved. That is a bound, not a guarantee: an epoch could be
  reused only if an earlier run made more than a million book changes
  without one successful save, and nothing enforces that limit (see "Still
  owed"). An armed agent saves the advance when it opens the book, or
  refuses managed starts. A box that declined the flag keeps the advance in
  memory until its next write of the book, and that write is an attempt: a
  failed save is logged and the box continues. A console with no signing key answers 503, and the text of that refusal is
  new copy awaiting review (`copy-review/strings.json`).
  Superseded on zdd-flag-readiness-2 (`7c7c6141`, pending merge): the 2^20
  advance and its "bound, not a guarantee" caveat are gone, replaced by the
  durable per-app generation rule (see "Still owed (code)").
- `provision.ListRecords` silently skips unreadable instance records (Astra
  plan 6 R3). A relevant fleet can then drop out of the deploy watch's
  completeness check, so a sleeper on it may be URL-polled when its fleet
  snapshot is also unreadable. This is a flag-on blocker. Owed: expose
  record-read completeness to the instance list
  (docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-6-R3.md). Closed on
  zdd-flag-readiness (f9be4718), pending merge.

### Reply: four claims-on gaps

Plan 5 was merged dormant as `d905a67`, after merging main d6f66ef3 (#268
lease margin, region-HA). Astra's merge review
(`CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md`) said ACCEPT WITH FIXES for a dormant
merge, with no flag-off defect.

Main's `RoleLaunch.Root` is wired in production, so claims-on launches can
execute. When this reply was written, the flag could not go on until the box
either declined arming with host claims on, or all four of these were closed:

1. Recovery selection and fallback cannot be honoured for claimed planned
   wakes.
2. A claimed hold launch can lose its execution evidence and outcome fence.
3. Retained claim processes are missing from the generations file.
4. Successful claimed wakes cannot record their good build or serving
   version.

That reply found no claims check in `swapArming`
(`internal/host/rollout.go:145-156`).

**Conclusion (2026-10-12): decline, do not close.** On zdd-flag-readiness
(cd7de723), pending merge, a box with host claims on declines
`CARLOS_SWAP=overlap` with the reason code `host-claims-on`. It runs today's
stop-then-start paths.
The four gaps above are not closed. They become the future feature "claims-on
swaps", which is not scheduled. The source for the four gaps is
`CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md`. After Gate E on any claims-on box,
the operator sees a dash under SERVING and, in REASON, "its box cannot run
two builds side by side, so it restarts instances the old way". The code
`host-claims-on` is in the instance's JSON record (`rollout.unavailable`) and
in the agent log line `REFUSING CARLOS_SWAP=overlap (host-claims-on)`.

### Still owed (code)

**Current state, 2026-10-14.** Readiness rounds 1 to 3 are merged and
released, so every item this section listed after zdd-flag-readiness is
closed on main. Rev 19 (`ebddcfc`, v0.27.2) and #292 (`e15ad6b`, v0.27.3)
are merged and released too. The kernel checks that were owed here are
closed (see "Kernel checks" at the top of "Before Gate E: owed").

Closed, with the commits that closed them:

- The nested live-link race on a declined box (`903ed7ca`, `c910d020`). A
  declined box pins the live link once, and nested starts and seeds exec the
  pinned build.
- Seeds that exec'd the live link ungated, and the failed seed's proof
  (`903ed7ca`; `4c5e43b5`, `5e51c008`, `4771d147`). A seed's processes are
  proven gone before its database goes, and an unproven seed leaves a
  durable obligation.
- The stranded legacy sidecar (`c910d020`). A refused sidecar restarts on the
  instance's own build.
- The one-shot's good build and WakeWorker's duplicate wake (`deb4c811`).
- The flag-off normal-stop half of F6 (`d6996537`, `ac9f3fef`, `097ca930`,
  `79032948`).
- `ensureParent`'s uncapped paths (`98593909`, `8530210e`). A generation
  whose limits cannot be applied does not start (`cgroup.ErrLimitNotApplied`).
  Its kernel positive control passed on 2026-10-14.
- The declined box's hold row and Retry button (`6e6ad000`, `65d4e135`). Paul
  approved the seven strings on 2026-10-13.
- The rev-18 spec amendment for console-signed host retries, approved by
  Paul on 2026-10-13. The design branch's spec still has to be raised to rev
  18 there.
- The reopen epoch, now the hard rule (`7c7c6141`, `191c12b9`, `692d1533`,
  `a6eb8b35`). The proof and its two assumptions are on `FailureBook`
  (`internal/rollout/failures.go`).
- Mixed-version retry sequencing (`6e9e84bd`, `9f27cdfa`, `4d8957c1`,
  `18bde560`). Grants ride their own object. Dev round 4 checked the
  transitions; the one not run is covered by construction (see "Mixed-version
  retries" at the top of "Before Gate E: owed").
- Readiness round 2's dev blockers, closed by readiness round 3: the
  exec-shim for `RestrictNamespaces=yes`, the `cgroup-placement-failed` arming
  check, `cmd/zddprobe` and the `TestStrictKernel` fix.
- Task 10's grant consumed while the row is moving or landing (`076bd9c0`).
- From settings plan 4, on branch `zdd-owed-items`: a cold wake's build-state
  record is written at the exec, after the replica-key gate and the cgroup
  refusals, so a refused launch leaves no record (owed before O8). The exec
  is decided again after the write, and a nested seed writes its obligation
  first.
- Also from settings plan 4, on the same branch: `release.ResolveLivePath`
  calls `release.ResolveLiveLink`.

Still open:

- Claims-on swaps (the four gaps from `CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md`).
  A future feature, not scheduled. A box with host claims on declines the
  flag.
- **MOVED to #250.** Leftover `consumed/` files refuse the claims census.
  Root's request record `/opt/carlos/consumed/<host>` and its fence lock
  `.<host>.lock` outlive the installation, and the empty-installation census
  refuses both. It also refuses another host's record, which is an
  unclassified file. platform-1e owns it now, under Plan 14. Root's
  retirement removes the consumed file, its lock and the record atomically,
  so a re-install passes the empty-installation census. A runner test covers
  install, retire and re-install. It is owed before host claims are switched
  on, and does not block the flag.

### Roll notes from zdd-flag-readiness

- **Retries: boxes before consoles.** Superseded by the next bullet. Kept
  for history: on zdd-flag-readiness a retry rode `restart-scope.json`, and
  an older binary that appended to it stripped `signed` from pending
  retries. That fails closed, but a box from before the branch applied a
  retry with no book or epoch check, so the roll order mattered. Since
  zdd-flag-readiness-2 it does not (see below).
- **Retries: supported transitions** (zdd-flag-readiness-2, `6e9e84bd`,
  `9f27cdfa`, `4d8957c1`, `18bde560`, pending merge). A signed host retry is
  written to `apps/<acct>/<app>/host-retry.json`, not to `restart-scope.json`.
  No released binary reads or writes it, and `restart-scope.json` no longer
  takes a retry or a signature. A retry is released nowhere before this
  branch, so there is no released retry behaviour to preserve. The
  transitions:
  - *Old console, new box.* A released console has no host retry at all, so
    the CLI and the page offer none until the console is updated. Its
    restarts rewrite `restart-scope.json` only, which holds no grant now and
    so cannot lose one.
  - *New console, old box.* The old box's report carries no `retry_v` and the
    old box never reads `host-retry.json`. The console refuses the retry with
    a message (409 on the API, a flash in the browser, no Retry button on the
    page) and writes nothing. Update the box, then ask again.
  - *New console, new box, book not saved.* A held host at epoch 0 is
    refused with "the box has not saved its failure yet". Ask again in a
    minute. The agent retries its own save.
  - *New console, new box.* The console signs a grant for the epoch and book
    the box reported, the box verifies it on the next serial move, and a
    grant gone stale is a no-op.
  - *A grant written before an agent restart* is checked again after it
    (grants are not seed-skipped). A current one applies, a spent or stale
    one does nothing.
  - *A dev box on an intermediate build* (between `5da1cc9b` and this
    branch) loses the retry path until it is updated. It is unreleased,
    which is why it is acceptable.
  - A console with no signing key answers 503 and writes nothing.
  - A box that declined the flag and keeps a failure book honours a grant too
    (no canary: its next wake starts the build as today).
- **Schema 3 failure book.** The failure book file is written as schema 3
  (it was 2 on zdd-flag-readiness). Downgrading an agent with the flag on
  makes that box refuse managed starts, and that includes a round 1 agent
  (on `main` since `8ebd9d8c`), which accepts only schema 1 and 2. Recover
  by upgrading the agent again. Do not
  delete the file to get past the refusal: it holds the failure records and
  hold-ran markers that keep instances off builds that failed or ran on
  them. A file of schema 1 or 2 is adopted on first open and given a new
  book identity, so grants issued against the old identity clear nothing.
- **Copying or cloning a data dir: drop the book key.** The failure book's
  identity is in the file's `book` key. A copy that keeps it gives two boxes
  one identity, and a copy restored from an older snapshot can hand out again
  a generation already published. Before the flag goes on, any path that
  clones, restores or relocates an agent data dir (a rebuilt box from an
  image or snapshot, a hand copy) must delete the `book` key from
  `swap-failures.json` in the copy. The next open then makes a new identity.

  Audit (fix wave 1, item 7, at `929dfc39`). The book is
  `/data/swap-failures.json` (`CARLOS_DATA_DIR`, default `/data`). Every
  path found in this repo and in `platform-infrastructure` (at `841e94b`)
  that copies, snapshots, restores, rsyncs or images agent data:

  | Path | What it carries | Can the book travel? |
  |---|---|---|
  | Box rebuild or replace (`infra/modules/region-host`, `tofu apply -replace`) | Stock Debian AMI (`data.aws_ami`); root volume `delete_on_termination = true`; no other volume | No. The new box's `/data` is empty, and the first open makes a new identity. |
  | Cold-standby box (`platform-infrastructure/dev/standby-ie.tf`) | A fresh box that converges from the bucket like any new box | No. Same as a rebuild. |
  | Box init (`host-init.sh`, `host-init.sh.tftpl`, `scripts/bootstrap-box.sh`) | Creates `/data` and fetches the binary | No. Nothing is copied into `/data`. |
  | Instance move (`internal/host/movespass.go` `FilesDir`, `internal/moves/archive`) | `/data/<host>/` and `/data/<host>.env` only | No. The book is at the `/data` root and is not in the archive. |
  | Litestream replication and `carlos ops restore-verify` | `*.db` files | No. The book is JSON, not a database. |
  | Hibernate wake restore (`Restore`, `internal/lease`), claims cold restore (`internal/hostbox` `ColdRestore`) | One instance's database | No. |
  | Layout migration (`internal/layout`) | Instance directories inside one box's `/data` | No. Same box, same file. |
  | Box volume records (`internal/boxvolumes`) | Records of EBS mappings; nothing is reattached | No. A terminated box leaves no disk (region HA spec 1 §5.9). |
  | Platform update (`scripts/update-platform-binary.sh`, `carlos system update`, `carlos-system-recover.service`) | `/opt/carlos` binaries | No. |

  No path carries the book, so no code change was needed. Not audited, and
  outside these two repos: `carlosframework/carloku-infrastructure` (the
  flagship root since 2026-10-01) and `teamtito/carlos-infrastructure`
  (`RUNBOOK-edge-failover.md`), which have no local checkout here; AWS
  Backup or DLM policies made outside tofu (none appear in either repo);
  and any copy made by hand. Each of those must drop the `book` key as
  above before the flag goes on.
- **An armed box without `CLONE_INTO_CGROUP` refuses every generation
  start** (`98593909`, `8530210e`). Superseded on zdd-flag-readiness-3: the
  Manager no longer uses `clone3` and places through the shim, so the shipped
  unit no longer causes this refusal. See "Roll notes from
  zdd-flag-readiness-3". The rest holds, with "the shim cannot place" in
  place of "clone3 is unavailable". A generation start must be placed in its
  cgroup before exec. If clone3 placement is unavailable (the fallback has
  latched, or the cgroup cannot be opened), the start is refused with
  `cgroup.ErrLimitNotApplied` and is never done by moving the PID after
  exec. Wakes of that box show the maintenance text, and the refusal is the
  box's condition, not a failure recorded against the build; a swap abandons
  its attempt and is retried, as for the populated legacy parent below. The
  flat (flag-off) start is unchanged. Recovery: turn the
  flag off and restart the agent, or fix the kernel or cgroup state. The same
  refusal covers a parent that cannot be made, does not delegate memory, or
  whose backstop or child limit cannot be written. The log line begins
  `cgroup:` and says "generation start refused".
- **The seed obligation file `<db>.seed-unproven`** (`5e51c008`, `4c5e43b5`,
  `4771d147`). Before a seed runs, the agent writes `<db>.seed-unproven`
  beside the instance's database (three lines: `carlos seed obligation v1`,
  `db=owned-by-seed` or `db=there-before-seed`, `seed=flat` or
  `seed=nested`). A write that fails refuses the seed before anything runs.
  The file is removed only when the seed's processes are proven gone. After
  a failed seed, a database the seed created goes with it; a successful
  seed's discharge keeps the database. While the file is there no wake
  treats the database as warm, and a wake that cannot prove the seed gone
  fails with `activator: seed unproven; processes remain` and holds the lease
  to its TTL. The obligation is a file, so it survives an agent restart. It
  does not end in `.db`, so the litestream glob never takes it. A box on a
  kernel without `cgroup.kill` proves a seed by its leader's exit alone
  (logged).

  Normally do nothing by hand. Once the processes are gone, the next wake
  proves it, discharges the obligation itself, removes a seed-owned database
  and seeds again. The agent removes the database (`<db>`, `<db>-wal`,
  `<db>-shm`) first, syncs the directory, and removes the obligation last,
  so a crash or a wake in between never sees a database without its
  obligation. Prefer that path: clear one by hand only for a proof the agent
  cannot make.

  A hand clear must not race a wake. Between your proof and your delete, a
  wake can discharge the obligation itself and start a new seed (Restore,
  `settleSeedObligation`), and you would then delete that new seed's
  database. There is no per-host way for an operator to hold off wakes. The
  activator's maintenance fence (`FenceAndStop` / `LiftFence`,
  `internal/activator/maintenance.go`) is used only by the move worker, the
  layout pass and the claim-stop coordinator, and nothing exposes it to an
  operator. A layout journal (`<data>/layout/<host>.json`) fences wakes too,
  but the layout pass reads and acts on it, so do not plant one. A row
  status other than `ready` is set only by the move worker and does not stop
  a wake already in flight. So clear an obligation by hand only with the
  agent stopped. Stopping `carlos-edge` stops every instance on the box, so
  plan for a short box-wide outage:

  1. Stop the agent: `systemctl stop carlos-edge`. Wait until it is
     inactive. No wake can run until it starts again. Its delegated cgroups
     and the units bound to it (`carlos-instance-seed@`,
     `carlos-instance@`, `carlos-instance-gen@`, `carlos-instance-sidecar@`)
     stop with it.
  2. Prove the seed's processes gone. The host's `carlos-exec@<host>` cgroup
     (and for a nested layout the host slice's seed unit) must show an empty
     `cgroup.procs` and `populated 0` in `cgroup.events`, or be absent, and
     no process may still hold the database open.
  3. Only when the obligation's body says `db=owned-by-seed`, delete `<db>`,
     `<db>-wal` and `<db>-shm`.
  4. Sync the directory (`sync "$(dirname <db>)"`, which fsyncs it) so the
     deletes are on disk before the obligation goes.
  5. Remove `<db>.seed-unproven` last, never first. Deleting it first
     leaves a half-built database that the next wake serves as warm. Sync
     the directory again.
  6. Start the agent: `systemctl start carlos-edge`.

  A body of `db=there-before-seed`, or one the agent does not recognise,
  keeps the database, and the next wake serves it warm, by design. Never
  delete the obligation to get past a refusal while a process may still be
  writing the database. A planted non-empty directory at that name keeps
  that tenant's row down until it is removed by hand.
- **The armed agent writes its failure book on start.** Since c12c35c3 an
  agent with the flag on saves the book when it opens it, even when there
  was no file, and refuses managed starts if it cannot (the same refusal as
  an unreadable book). A box whose data dir is read-only will show that.
- **A populated legacy parent cgroup.** A generation start is refused while
  `carlos-exec@<host>` holds a process directly (2c5adbe8). The agent logs a
  line beginning `cgroup: <host>: generation start refused: the parent
  cgroup holds N process(es) directly`, wakes of that host fail, and swaps
  abandon and are retried every 15 s. Nothing lifts this automatically. Clear it by hand:
  kill the processes listed in that cgroup's own `cgroup.procs` (under the
  agent's delegated cgroup), or restart `carlos-edge`, which kills the
  delegated subtree. The next wake or retried swap then starts normally.
- **`carlos-instance-control` is not restarted by the old script.**
  `platform-infrastructure/scripts/update-platform-binary.sh` does not
  restart it. Use `carlos system update` (main's system apply restarts it,
  0bb57a31), or restart it by hand after the old script.

### Roll notes from zdd-flag-readiness-3

Branch `zdd-flag-readiness-3`, from `main` at `34a4328f`, on amadan
discussion #272. Built and task-reviewed on the branch: not merged, not
released, not rolled, and only Paul closes it on #272. The whole-branch
review and CI come after these notes. Nothing here was run on a real box.

- **The four round-2 blockers, and where each is closed.** Paul's dev check
  on 2026-10-13 ("Dev-box checks, round 2" on #272) found four things that
  stopped the flag. They are closed in code on this branch, pending merge:
  1. `RestrictNamespaces=yes` on `carlos-edge` makes systemd's seccomp filter
     answer `clone3` with ENOSYS, so `CLONE_INTO_CGROUP` never worked under
     the shipped unit and every generation start was refused. Closed by the
     exec-shim (`ff4b814a`, `a34013c6`), by routing every Manager start
     through it (`961aed71`), and by a kernel test under the edge unit's
     sandbox lines (`891c0fb7`).
  2. Arming did not check that placement works, so a box that could not place
     armed anyway and then failed every generation start. Closed by an
     arm-time check (`7fbafe7e`, `1f9d6a83`): see the decline code below.
  3. The roll notes' probe app rejected the `--db` argument the platform
     passes, so it exited at once. Closed by `cmd/zddprobe` (`59b1895a`).
  4. `TestStrictKernel` had a defect of its own and could not pass in its full
     mode. Closed by `496fb6ed`.

  The minor from the same check, failed generation units lingering in
  `systemctl --failed` after the control service stopped them, is closed by
  `496fb6ed` too: the control service now runs `reset-failed` on a generation
  unit after a stop or an abandon-kill that proved it gone. It is best-effort.
  A failure is logged as `instance-control: reset-failed <unit>: ...` and the
  stop still answers 204.
- **The shim.** A spawn starts `/proc/self/exe __cgexec`. The shim places
  itself in the target cgroup, then `execve`s the tenant. The tenant's
  path, argv and environment travel over a pipe, never in the shim's own argv
  or environment, so the PID the agent tracks is the shim's and then the
  tenant's. Paul's ruling (2026-10-13): no line of
  `internal/host/units/carlos-edge.service` is relaxed. `RestrictNamespaces=yes`
  stays, and so does every other sandbox line. The unit file is not edited
  on this branch. The cost is one extra exec of the `carlos` binary per
  spawn, measured on DOCS-T1 at about 4.5 ms (median 4.9 ms against 0.42 ms
  for a direct start).
- **Behaviour change on roll, flag off.** Every exec child on a rolled box
  now starts through the shim, flag on or off, not only generations. Healthy
  looks like today: no new log lines at all. In particular the agent log has
  no `placement before exec failed` line and no PID-move line. Wakes answer
  as before, a few milliseconds slower.

  A failing shim on a flat (flag-off) start logs:

  ```text
  cgroup: <host>: placement before exec failed (<why>) — using the PID-move fallback for this spawn
  ```

  and the spawn goes ahead by starting the process and then moving its PID
  into the cgroup. That is exactly what every flag-off start does on the
  shipped unit today, so the box still serves. Only a placement the shim
  proved happened before exec falls back. A start that may have run is
  reported started and never retried, so a tenant never runs twice. If you see
  the line on a healthy box, stop and look: the shim is not working there and
  the box is on the old path.

  A generation start never falls back. The same words end with
  `— generation start refused`, the start answers `cgroup.ErrLimitNotApplied`,
  and the refusal is the box's condition, not a failure recorded against the
  build. The "armed box without `CLONE_INTO_CGROUP`" note above is replaced by
  this one.

  A start whose own context ended while its shim was being placed (a seed's
  wake that timed out or was stopped) is not a shim fault. It logs:

  ```text
  cgroup: <host>: the start's context ended during placement (<why>); nothing ran
  ```

  and answers the context's own error, as a direct start would. Flat or
  generation, it does not fall back and is not the box's condition. The
  member's wake event reads as such a wake always read (`wake failed: seeding
  <host>: context canceled`).
- **The shim keeps the agent's ignored signals.** A tenant started through
  the shim has the same ignored signals as one started directly: the agent
  sends its current set (`SigIgn` in `/proc/self/status`) with the spawn, and
  the shim ignores those again just before it execs the tenant. Without
  that, the shim's own Go runtime would have reset every ignored signal but
  `SIGHUP` and `SIGINT` to its default. This changes nothing on a box whose
  agent ignores no other signal.
- **The new decline code `cgroup-placement-failed`.** At arm time (the agent's
  start with `CARLOS_SWAP=overlap`) the agent starts a real shim placement
  into a scratch cgroup named `carlos-placement-check`, under its delegated
  root, within one 4 s budget for the whole check. If that fails the box
  declines the flag for every row, nested rows included, and logs:

  ```text
  agent: REFUSING CARLOS_SWAP=overlap (cgroup-placement-failed): a test placement into a scratch cgroup failed (<why>), so generations could not be started inside their limits; this box stays on today's stop-then-wake paths
  ```

  The code is in the instance's JSON record (`rollout.unavailable`) and uses
  the approved declined wording, so there is no new copy. A box whose check
  fails declines in about 4 s, inside the 20 s `carlos system update` gate.
  The 4 s bounds the check's own waits. After them the agent kills and
  reaps what it started, which takes milliseconds when the kernel ends a
  killed process promptly. It does not when that process is stuck in
  uninterruptible kernel I/O: SIGKILL ends it only when the I/O returns, and
  the decline waits for that. So on a box whose disk or cgroupfs is
  stalled, the decline can take longer than 4 s, and the update gate can
  time out. A box that passes logs
  `agent: overlap swaps armed (CARLOS_SWAP=overlap)` as before.

  *Mixed versions.* Consoles and CLIs roll after boxes, so for a while a
  rolled box can decline with a code the others do not know. A console older
  than the box shows the unknown-code phrase for `cgroup-placement-failed`. It
  treats an unknown code as not declined, so it cannot offer or sign a host
  retry for that box until it is updated. An older `carlos` CLI's
  `deploy --watch` does the same: it does not follow the box as declined, so
  it shows that box as pending until the watch times out (`classifyRollout`
  and `follows`, `deploywatch.go`). The deploy itself is unaffected. Update the
  console and the CLI and ask again.
- **Claims-on swaps: a new precondition.** The strict claims path
  (`PrepareStrict`, `PrepareStrictParent`, `(*Prepared).Start`) keeps its
  `clone3` placement, by Paul's ruling. Under the shipped unit `clone3`
  answers ENOSYS, so on a box with host claims on every strict launch answers
  `cgroup.ErrStrictUnsupported`. It is refused, never run uncapped. Claims are
  off everywhere today and a claims-on box declines arming (`host-claims-on`),
  so nothing changes now. But claims-on swaps, a future feature, cannot work
  until the strict path gets the shim too. Count that as a precondition of
  them.
- **Dev re-check for Paul, replacing round 2's blocked steps.** Needs Paul's
  approval, naming the box, as Gate E does. In order:
  1. Roll the build to the dev box with the flag still off. Wake an
     instance. It answers 200, and the agent log (`journalctl -u carlos-edge`)
     has no `placement before exec failed` line.
  2. Set `CARLOS_SWAP=overlap` in `/etc/carlos/host.env` and restart
     `carlos-edge`, as Gate E does. The log shows
     `agent: overlap swaps armed (CARLOS_SWAP=overlap)` and no `REFUSING`
     line. If it shows `(cgroup-placement-failed)`, stop and read the cause in
     the line.
  3. Run the probe (built from `cmd/zddprobe`, see "The probe") under load
     through `zdd-1` to `zdd-3`. The pass is 0 failed requests and Gate E's
     other conditions.
  4. Run the probe's `hold` control. It must show the pause.
  5. After the swaps, `systemctl --failed` on the dev box lists no
     `carlos-instance-gen@` unit that the control service stopped.
  6. Check on the real unit the edge-unit lines the kernel test could not.
     Task 4's test ran in a user unit on DOCS-T1, not the shipped system unit.
     Its report marks these `untested` there: `ProtectProc`, `ProtectClock`,
     `RestrictRealtime` and `SystemCallArchitectures`. `ProtectSystem` was
     observed only as a read-only `/usr`, so `strict` was not told from
     `full`. `NoNewPrivileges` is implied by the seccomp lines. These it
     excluded, because only the real unit has them: `User`, `Group`,
     `AmbientCapabilities`, `CapabilityBoundingSet`, `ReadWritePaths`, `Slice`,
     `OOMPolicy`, and `Delegate=cpu memory io pids` (the user manager gives
     `cpu memory pids`, no `io`). The user unit also adds a user namespace the
     real unit lacks. On dev, read what systemd applied with
     `systemctl show carlos-edge -p ProtectProc -p ProtectClock -p RestrictRealtime -p SystemCallArchitectures -p ProtectSystem -p NoNewPrivileges`,
     and treat steps 2 and 3 passing as the proof that placement works under
     the unit as shipped, as `carlos` with its real `ReadWritePaths` and slice.
  7. Run the two activator kernel tests that now start through the shim and
     could not run on DOCS-T1, because user delegation there has no `io`
     controller: `TestClaimContainmentCleanupRespawn` and
     `TestGenerationStartRefusedBesideADirectParentMember`. They are owed on
     the dev box. Both skip unless `CARLOS_CLAIM_SPAWN_KERNEL=1`, and both
     call `cgroup.Setup`, which needs a delegated cgroup offering `cpu`,
     `memory`, `io` and `pids`; without one they fail with
     `missing delegated cgroup`.

     Build the binary for the dev box from the checkout root on DOCS-T1, and
     copy it to the dev box by the route the roll uses for binaries:

     ```bash
     GOOS=linux GOARCH=arm64 go test -c -o activator.test ./internal/activator
     ```

     On the dev box, in a fresh directory under `/tmp` (the activator's
     socket paths need a short root), first the two shim checks, with the
     shipped unit's `RestrictNamespaces=yes`:

     ```bash
     d=$(mktemp -d /tmp/zddk.XXXXXX)
     install -m 0755 activator.test "$d/"
     sudo systemd-run --wait --pipe --collect -p Delegate=yes -p User="$(id -un)" \
       -p RestrictNamespaces=yes -E TMPDIR="$d" -E CARLOS_CLAIM_SPAWN_KERNEL=1 \
       "$d/activator.test" -test.run '^TestGenerationStartRefusedBesideADirectParentMember$' -test.v -test.count=1
     sudo systemd-run --wait --pipe --collect -p Delegate=yes -p User="$(id -un)" \
       -p RestrictNamespaces=yes -E TMPDIR="$d" -E CARLOS_CLAIM_SPAWN_KERNEL=1 \
       "$d/activator.test" -test.run '^TestClaimContainmentCleanupRespawn$/^Kill' -test.v -test.count=1
     ```

     The pass is `--- PASS` for `TestGenerationStartRefusedBesideADirectParentMember`
     and its `sibling_control` and `recovery_control`, and for
     `KillHost_then_respawn` and `KillSidecar_then_respawn`, with no
     `--- SKIP`. These use a system unit run as you (`sudo`, `-p User=`)
     because systemd delegates `io` to it. Where the user manager offers
     `io` too, the same commands as `systemd-run --user`, without `sudo` and
     `-p User=`, do as well; where it does not, they fail with
     `missing delegated cgroup`.

     Then the strict subtest, on its own and **without**
     `RestrictNamespaces`:

     ```bash
     sudo systemd-run --wait --pipe --collect -p Delegate=yes -p User="$(id -un)" \
       -E TMPDIR="$d" -E CARLOS_CLAIM_SPAWN_KERNEL=1 \
       "$d/activator.test" -test.run '^TestClaimContainmentCleanupRespawn$/^strict_primitive$' -test.v -test.count=1
     rm -rf "$d"
     ```

     `strict_primitive` calls `(*Prepared).Start`, the strict claims path,
     which keeps its `clone3` placement by Ruling 3. Under
     `RestrictNamespaces=yes` `clone3` answers ENOSYS, so that subtest is
     expected to fail there (`cgroup.ErrStrictUnsupported`). That is the
     claims-on precondition above, not a regression, and it is why it runs
     without the line. Run as one `-test.run '^TestClaimContainmentCleanupRespawn$'`
     under `RestrictNamespaces=yes`, the test fails on that subtest alone.
- **Kernel tests on DOCS-T1.** They skip unless asked for, and they need a
  delegated unit. From the checkout root, build the test binary into a
  directory of its own, run it in a transient user unit, and remove that
  directory and nothing else:

  ```bash
  d=$(mktemp -d "$TMPDIR/zddk.XXXXXX")
  go test -c -o "$d/cgroup.test" ./internal/cgroup
  systemd-run --user --wait --pipe --collect -p Delegate=yes \
    -E TMPDIR="$d" -E CARLOS_GEN_KERNEL=1 -E CARLOS_SHIM_KERNEL=1 \
    "$d/cgroup.test" -test.run '<regex>' -test.v -test.count=1
  rm -rf "$d"
  ```

  Run the shim, generation and placement-check tests a second time with
  `-p RestrictNamespaces=yes`, the shipped unit's line, and leave
  `TestGenStrictKernel` out of that run: it is the strict path, which keeps
  `clone3` (Ruling 3). User delegation offers `cpu memory pids`, not `io`.
  Task 4's edge-hardening test is a driver that starts its own units, so it
  runs outside any unit:

  ```bash
  mkdir -p /tmp/k/zddr3 && go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup
  (cd internal/cgroup && CARLOS_EDGE_HARDENING_KERNEL=1 TMPDIR=/tmp/k/zddr3 \
    /tmp/k/zddr3/cgroup.test -test.run '^TestEdgeHardeningPlacement$' -test.v -test.count=1)
  rm -rf /tmp/k/zddr3
  ```

  This test needs a `TMPDIR` under `/tmp`, so it can check `PrivateTmp`. It
  fails with "needs TMPDIR under /tmp" otherwise, which is why this command
  does not use the on-disk `$TMPDIR` of the recipe above.

  `TestStrictKernel` takes a mode in `CARLOS_STRICT_KERNEL` and a proof
  directory in `CARLOS_STRICT_PROOF_DIR`. This is the `full` mode. It runs in a
  delegated unit without `RestrictNamespaces`, because the strict path keeps
  `clone3`:

  ```bash
  mkdir -p /tmp/k/zddr3/strictproof && go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup && \
    systemd-run --user --wait --pipe --collect -p Delegate=yes \
    -E TMPDIR=/tmp/k/zddr3 -E CARLOS_STRICT_KERNEL=full \
    -E CARLOS_STRICT_PROOF_DIR=/tmp/k/zddr3/strictproof \
    /tmp/k/zddr3/cgroup.test -test.run '^TestStrictKernel$' -test.v -test.count=1
  rm -rf /tmp/k/zddr3
  ```

### Real-box checks (need Paul's approval; not run by this branch)

Readiness round 2 adds these to the list. None of them was run. The kernel
suites skip unless `CARLOS_CLAIM_SPAWN_KERNEL=1`, and the temp-dir cgroups of
the unit tests cannot stand in for a real cgroupfs.

- Task 7's kernel tests, in
  `internal/activator/genparent_kernel_linux_test.go`:
  `TestGenerationStartRefusedBesideADirectParentMember`'s "sibling control"
  (g1 starts and stays up, is capped in `g1`, and g2 starts beside it) and
  "recovery control" (kill the legacy process, the parent empties, g1 starts).
  The live-g1 respawn control is `killSidecarRespawn`, in
  `internal/activator/claimspawn_kernel_linux_test.go`, run by
  `TestClaimContainmentCleanupRespawn`: a live g1 survives each sidecar
  respawn and the parent's `cgroup.procs` stays empty. The same checks need
  to hold with the placement the box really has. Since zdd-flag-readiness-3
  that is the shim, not `CLONE_INTO_CGROUP`, and the refusal to watch for is
  `cgroup-placement-failed` at arming or `generation start refused` at a
  start.
- Task 8's seed and normal-stop kernel control: a forked helper killed by
  `cgroup.kill` before the lease goes, for a flat seed, a nested seed unit
  and a flag-off stop. No kernel test of it exists yet. `KillAndEmpty`, the
  wake-time `EmptyHost` and a nested seed unit stopped by root were only
  modelled through seams.
- The real-unit control and the sidecar checks (a unit route and the
  sidecar members on dev, including a refused sidecar restarting on the
  instance's own build).
- Dev rollout, and a retry across a mixed-version roll on dev: each
  transition under "Retries: supported transitions", with an old box and a
  new console, a new box and an old console, and a book that has not saved.
  Include a data-dir copy with the `book` key dropped, and a downgrade onto a
  schema 3 book to see the refusal.
- The flagship probe's gates E, F and G, as before.
- Readiness round 3's dev re-check, replacing round 2's blocked steps: see
  "Roll notes from zdd-flag-readiness-3". It includes the two activator
  kernel tests below, which now place through the shim and need the dev box's
  `io` controller. Its step 7 has the commands, and runs the strict subtest
  of `TestClaimContainmentCleanupRespawn` apart, without
  `RestrictNamespaces=yes`.
- Round 1's kernel checks, still as listed next.

 `TestClaimContainmentCleanupRespawn` (now the
  KillHost and KillSidecar respawn) and
  `TestGenerationStartRefusedBesideADirectParentMember` on the dev box
  (arm64, `CARLOS_CLAIM_SPAWN_KERNEL=1`, a delegated unit with `io`; the
  commands are step 7 of the readiness-3 dev re-check). Also check
  whether children of a killed kept cgroup, and cgroups systemd kills on
  `carlos-edge` stop, are affected.
- The plan 2 kernel suite rerun.
- The flagship probe (Gates E to G).
- D1 is done: well under 1 s, keep drop-ins.

### Reply: wiring caution (settings plan 2, merged as `70f8119`)

Held staging checks authorization inside `stageHeld`, so every entry point is
covered, including main's claim-path `MaterializeHeldUnder`.

Whoever wires zero-downtime plan 5 Task 9: when `movesProbeOwners.Release`
is connected to `MaterializeHeldUnder`, it must pass the parsed
`CARLOS_RELEASE_PUBKEY` and the agent's digest ratchet. A missing key or
ratchet refuses every stage (fail closed). That is not a bug to "fix" by
passing nil.

### Where plan 6 stands against that list (for information)

The discussion text predates plan 6 ("not started"). On this branch the
`carlos sidecars ls` change is built (task 13), and the refusal and wake
wording is approved (task 14b). This file is task 17. Only Paul closes the
discussion's items.

## What plan 6 changed that the roll should check

- **The copy is approved and pinned.** Paul approved every new user-facing
  string on 2026-10-11. The record is
  `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-copy.json`, and
  `approved_copy_zdd_test.go` fails if any string in it drifts from the code.
  Where this file says "the approved line", it means the text in that record.
- **`carlos sidecars ls` changes with the flag off.** It reads
  `ServingVersion` and prints "unknown" for a sidecar that has never started
  anywhere. Plan 4 already changed what the registry `Version` means for a
  sidecar, so the old column would lie. Look at it on dev after Gate B and
  expect different output from the last release. The code is in
  `internal/console/apisidecars.go:121-130` and `sidecars_cli.go:368-378`.
- **The swap-policy door is person-only.** `PUT
  /api/cli/apps/{account}/{app}/swap` is registered with the zero role set
  (`internal/console/authcli.go:863`, reasoning at
  `internal/console/apiswap.go:1-8`). A signed-in member can run
  `carlos apps set`. A CI token or any credential the app itself holds
  cannot. If CI needs to set the policy, that is a one-line role change and
  Paul's call.
- **The first non-default swap policy creates `config.json` but restarts
  nothing (plan Q3, Astra plan 6 R1 #1).** Setting the defaults (`overlap`,
  1) on an app with no `config.json` writes nothing
  (`internal/appconfig/swap.go:47-57`). The first `hold` or concurrency above
  1 creates the file with no vars in it. Config sync leaves an instance's
  config file absent when its bundle renders nothing and it has no instance
  overlay (`internal/appconfig/syncexec.go:1397`), so no instance restarts,
  no config revision reaches the release scheduler and no config ack is
  written. That is the state every never-configured app's instances are in.
  An instance that already has a config file converges on its bytes as
  before. This answers Q3's restart concern; closing Q3 is Paul's call.
- **A host retry rides the unsigned restart-scope log (plan Q4).**
  Superseded on zdd-flag-readiness-2 (`6e9e84bd`): grants ride
  `host-retry.json`, not the restart-scope log; see "Retries: supported
  transitions" above.
  `carlos instances retry` asks the box to retry through the restart-scope
  log. That log is not signed, and the box's `RetryHost`
  (`internal/rollout/manager.go:501-515`) trusts any request that names a host
  on that box. Only the app's record and the rollout report gate it, at the
  console, not at the box. It is on the owed list above as a flag-on
  blocker. Closed on zdd-flag-readiness (d68a1dde), pending merge: retries
  are console-signed root grants bound to the host's restriction epoch, and
  every box ignores an unsigned retry.
- **`FailureBook` is not rendered anywhere.** The box publishes it
  (`internal/status/snapshot.go:268-275`) but no `carlos` surface or console
  page shows it. It is parked in #272. What `carlos` shows for a failed or
  held build comes from the per-host rollout report instead.
- **The deploy watch reads the instance list every 5 s, not every 2 s.** It
  falls back to polling the URL (today's `watchVersion`) only on evidence
  that no targeted instance is on a reporting box: the list answers with a
  4xx other than 429 (a console with no list for the app), or every targeted
  instance is on the list, the console read every box's report (the list
  carries no `rollout_incomplete`), and none of them has a report. A list cut
  short for the whole app neither causes nor blocks the fallback. A 5xx, a
  429 or a network error is retried every 5 s until the deadline, and the
  timeout then says the instance list could not be read. So a flag-off
  deploy against a console whose list is failing times out instead of
  URL-watching. One deadline covers the list reads, the waits between them
  and the fallback. A box that refuses to arm is named in the timeout, and
  its instances are never URL-polled. A box that declined overlap runs
  today's stop-then-start paths, so its instances are URL-watched, as with
  the flag off. An instance counted as done goes back to pending if its box
  later reports a different build installed. The constants are
  `deployWatchListEvery` and `deployWatchPollEvery`
  (`deploywatch.go:140-142`); the fallback rule is `listAbsent` and
  `legacyOnly` (`deploywatch.go:107`, `deploywatch.go:413`).
- **A dash under SERVING means the build is not reported, for either of two
  reasons.** No rollout report (flag off, or an older console), or a box that
  set the flag and declined (`Unavailable`). `servingCell` is at
  `instances_cli.go:450-455`. After Gate E, a dev instance still showing a
  dash means the box did not arm. Look for the box's reason code and the
  agent log line `REFUSING CARLOS_SWAP=overlap`
  (`internal/host/rollout.go:150`).
- **The migrations rule.** The new "Two builds at once" section of the public
  CLI page (`docs/site/cli.md`)
  and the release notes carry it: add in one release and remove in a later
  one, and keep the names of scheduled work the same across the two builds.
  `--swap hold` and `carlos apps set --swap hold` are the way out for a
  release that cannot follow it.

## Gate A: release

Needs Paul's approval, naming the release. This gate touches no box.

Plans 1 to 6 are merged to amadan `main` and `main`'s CI is green. Cut a
platform release from that commit, following README "Releasing the platform"
(steps 1 to 9). Its release notes (step 5, copy-reviewed) lead with the
migrations rule (spec section 9.5) and say the flag ships off. Call the
release `vX.Y.Z` below.

README's "Releasing the platform" section ends with a place for release
specific operator notes. Add one for this release naming the flag-on
preconditions above, so the person rolling it reads them.

## Gate B: boxes, flag off

Needs Paul's approval, naming the boxes: dev (`platform.dev.carlosframework.com`,
channel `canary`) and the flagship (`console.carloku.com`, channel `stable`),
which is `ie`, `us`, `uk` and `au`.

```bash
carlos system update --console https://platform.dev.carlosframework.com --channel canary --version vX.Y.Z --yes
carlos system status --console https://platform.dev.carlosframework.com
carlos system update --console https://console.carloku.com --channel stable --version vX.Y.Z --yes
carlos system status --console https://console.carloku.com
```

Expected: every box reports `vX.Y.Z`. Sydney is slow, so give it the full
gate. With `CARLOS_SWAP` unset nothing changes: a `carlos deploy` still
prints `live https://...` lines, and `carlos instances list` shows a dash
under SERVING.

## Gate C: consoles

Needs Paul's approval, naming the consoles and the app.

Roll each console after its boxes. The script is platform-infrastructure's
`roll-console.sh`; it is not in this repo (not checked here). Or use direct
mode:

```bash
carlos ship --app console --kind binary --version <sha> console-linux-arm64
carlos promote --app console <sha> canary/console
```

with `CARLOS_DEPLOYMENT_BUCKET` set for both. Then check the door:

```bash
carlos apps set adopt-probe --account bab --swap overlap --console https://console.carloku.com
```

Expected: the approved `zdd.apps-set.done` line for `adopt-probe`, overlap,
1:
`adopt-probe swaps in overlap mode, 1 at a time on each box`, then a muted
note about backward-compatible migrations. Setting the default writes
nothing (see the Q3 note above).

A console that predates the door answers that the app is not in this account
or that the console predates `carlos apps set`. Roll the console and run it
again. In direct bucket mode (`CARLOS_DEPLOYMENT_BUCKET` or
`CARLOS_DEPLOYMENT_DIR` set in the shell) `carlos apps set` refuses and says
so, so run the door check from a shell that is logged in to the console and
has neither variable set.

## Gate D: docs and changelog

Needs Paul's approval to publish. This gate touches no box.

Run README step 10: `go run ./cmd/changelog` in a full checkout of `main`
(after `git fetch origin --tags '+refs/amadan/closed/*:refs/amadan/closed/*'`,
with `gh` and `amadan` signed in), map the release's bullets in
`docs/site/changelog-links.json`, and land `docs/site/changelog.json` and the
links file on a branch. Then run `node hack/sync-docs.mjs <checkout>` in
`carlosframework/website` and ship the site the way its AGENTS.md describes.
`hack/sync-docs.mjs` and that AGENTS.md live in the website repo (not checked
here).

Check that carlosframework.com/platform/changelog lists the release and the
public CLI page shows "Two builds at once". The page's source is
`docs/site/cli.md:330` here; where the live page sits in the website is not
checked here. Both must be live before Gate G.

## Gate E: dev, the daemon-reload check, then the flag

Needs Paul's approval, naming the boxes: dev box `i-05498eada6f0afd23` (the
dev console box, eu-west-1) and dev box `i-046c7fe43620944bc` (eu-west-1),
profile `kass`. Both box ids come from the brief; only the first appears in
another repo document (`2026-10-06-console-release-defences-plan-4-d4.md`),
and the second is not checked here. The approval also names the probe's app
actions on dev: creating the `zdd-probe` app, flipping its swap policy to
`hold` and back, and four deploys of it. **Do not start this gate while the
"Before Gate E: owed" list has an open item that Paul has not waived.**

First the index's D1 guard: measure `systemctl daemon-reload` on a dev box
carrying titogo-scale unit counts. **Done 2026-10-11 on dev:** median 309 to
424 ms, so keep the drop-ins. Repeat it only if unit counts grow a lot; the
commands are kept below for that case.

```bash
aws --profile kass --region eu-west-1 ssm send-command --instance-ids i-046c7fe43620944bc \
  --document-name AWS-RunShellScript --query Command.CommandId --output text \
  --parameters 'commands=["systemctl list-units --all --no-legend | wc -l","for i in 1 2 3; do s=$(date +%s.%N); systemctl daemon-reload; e=$(date +%s.%N); echo \"$e - $s\" | bc; done"]'
aws --profile kass --region eu-west-1 ssm get-command-invocation --instance-id i-046c7fe43620944bc --command-id <the id printed> --query StandardOutputContent --output text
```

Expected: a unit count, then three times. If any reload takes more than about
1 s, stop: plan 2's D1 says nested generation units move to transient units in
a follow-up before the flag goes on. Otherwise set the flag on both dev
boxes:

```bash
aws --profile kass --region eu-west-1 ssm send-command \
  --instance-ids i-05498eada6f0afd23 i-046c7fe43620944bc --document-name AWS-RunShellScript \
  --parameters 'commands=["grep -q ^CARLOS_SWAP= /etc/carlos/host.env || echo CARLOS_SWAP=overlap >> /etc/carlos/host.env","systemctl restart carlos-edge"]'
```

If `host.env` already has a `CARLOS_SWAP=` line, even an empty or different
one, the command leaves it alone and the box does not arm. Read the file
first. The unit is `carlos-edge` and it loads `/etc/carlos/host.env`
(`internal/host/units/carlos-edge.service:18`).

Restarting `carlos-edge` stops every instance on the box (spec section 10).
Each wakes on its next request.

To revert a box, remove the `CARLOS_SWAP=` line from `/etc/carlos/host.env`
and restart `carlos-edge`. Then remove the last generation's leftover `g<n>/`
directory under each instance's run directory,
`/run/carlos/instances/<host>/g<n>/`. Dev round 5 (#272) found that the
flag-off restart leaves it behind with a stale `app.sock`. Nothing uses a
`g<n>/` directory with the flag off, so it stays until a reboot. List what is
left with `ls -d /run/carlos/instances/*/g[0-9]*/` after the restart, and
remove each one.

Then run the probe below against a dev app. Create one with:

```bash
carlos apps create --app zdd-probe --console https://platform.dev.carlosframework.com
```

then an operator's `carlos instances enable --app zdd-probe`, and:

```bash
carlos instances create --app zdd-probe --host <label>.<the account's platform domain>
```

Use the host it prints as `HOST`. For this gate: `APP=zdd-probe`,
`CONSOLE=https://platform.dev.carlosframework.com`, and `ACCOUNT` is the dev
account (an empty `--account` uses your only account).

Check the flag took before trusting the probe: `carlos instances list --app
zdd-probe` should show a build under SERVING once the box has reported. A
dash means the box did not arm (see the dash note above).

## Gate F: the flagship probe box

**Status: passed on annie-welding (`ie`) on 2026-10-14 (#272). annie stays
armed.** The first attempt, on 2026-10-13, was reverted. The retry ran the
probe against a fresh app, as below. Its arm restart left one tenant's lease
held for 15 minutes; that is the edge restart, not the flag (#292).

Needs Paul's approval, naming the box and its region. The approval also
names the probe's app actions: creating a throwaway app in `bab` with one
instance on that box, flipping its swap policy to `hold` and back, four
deploys of it, and deleting it afterwards. **Do not start this gate while
the "Before Gate E: owed" list has an open item that Paul has not waived.**

**Use a fresh app with no program history, not `bab/adopt-probe`.**
adopt-probe's recorded program is `adoptprobe`, so root refuses a zddprobe
build for it as the wrong program unless that build is shipped with
`carlos ship --program-change`, which `carlos deploy` cannot do. That
refusal, not a slow root pass, is what stopped zdd-1 in the first Gate F:
root logged `wrong program` and restored the old build, and the watch timed
out waiting for an install that would never come.

The 2026-10-14 retry set the app up like this:

1. Create the app in `bab` through the console's New app form, with no
   region. It was `bab/zdd-probe`.
2. An operator runs `carlos instances enable --app zdd-probe --account bab`,
   then creates one instance:

   ```bash
   carlos instances create --app zdd-probe --account bab --host zddprobe1.bab.oncarlos.com --console https://console.carloku.com
   ```

3. Check it landed on the gate's box: `carlos instances list --app zdd-probe
   --account bab --console https://console.carloku.com` shows that box's
   region (the retry's was "Ireland · Dublin"), and only that box's edges
   file lists the host.
4. A new host may not resolve yet. While DNS lags, point the load at the
   box's public IPv4 with `RESOLVE` (see "The probe"):
   `RESOLVE=zddprobe1.bab.oncarlos.com:443:<box IP>`.

The flagship boxes, profile `keymail`:

| box | instance id | region |
| --- | --- | --- |
| `ie` | i-092c0c1eea75723cb | eu-west-1 |
| `us` | i-0ed0641d8e476f1b8 | us-east-1 |
| `uk` | i-03c310ceaf50a9bab | eu-west-2 |
| `au` | i-022288023f5e7496d | ap-southeast-2 |

`ie`'s id matches the flagship console box id in
`2026-10-05-scanner-host-rollout.md`. The other three are not checked here.

Run Gate E's flag command against that one box with `--profile keymail` and
its region. Then run the probe with:

```bash
APP=zdd-probe ACCOUNT=bab CONSOLE=https://console.carloku.com HOST=zddprobe1.bab.oncarlos.com
RESOLVE=zddprobe1.bab.oncarlos.com:443:<box IP>   # only while DNS lags
```

Afterwards, delete the app: `carlos apps delete zdd-probe --account bab
--console https://console.carloku.com`. On 2026-10-14 the retry then removed
the instance's leftovers on annie by hand (its instance, builds, links, data
and `/data/programs/` file) and its bucket object, and checked that no
`g<n>/` directory was left. An empty `carlos-instances-zddprobe1…slice` stayed.

## Gate G: the rest of the flagship

Needs Paul's approval, naming each box before it is done. **Do not start
this gate while the "Before Gate E: owed" list has an open item that Paul
has not waived.**

One box at a time, with Gate E's flag command. After each box, do one
ordinary deploy of `adopt-probe` and run `carlos instances list --app
adopt-probe --account bab`, which must show INSTALLED equal to SERVING. Tito
is Tito's operator's call; do not touch Tito's boxes from here.

## The probe

Spec section 12: zero failed requests under continuous load, and no request
held longer than a normal response. Run it at Gate E (dev) and Gate F (a
fresh app on the flagship box, never `adopt-probe`; see Gate F for why).

### 1. Build the probe app, in scratch

The probe is `cmd/zddprobe` in this repository. It serves on the unix socket,
sleeps 50 ms per request so some are always in flight, answers `ok <version>`,
and drains on SIGTERM. It takes the platform's own argv, `--socket <sock>` and
`--db <db>` (it ignores `--db`), and vet's `-socket <sock>`. The first dev check
used a probe that parsed only `-socket`, so it exited at once when the
platform launched it with `--db`.

Build it from the root of a platform checkout at the release commit
(`cd` there first: the path `./cmd/zddprobe` is relative). The binaries go to
scratch:

```bash
mkdir -p "$TMPDIR/zdd-probe"
for v in zdd-1 zdd-2 zdd-3 zdd-4; do
  CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags "-X main.version=$v" -o "$TMPDIR/zdd-probe/probe-$v" ./cmd/zddprobe
done
```

### 2. A load phase

Four request loops for the given seconds, cache-busted, logging status and
time. With `RESOLVE` set (`<host>:443:<box IP>`), curl skips DNS and goes
straight to that box:

```bash
load() { # usage: load <phase> <secs>
  mkdir -p "$TMPDIR/zdd-probe/$1"
  for i in 1 2 3 4; do
    ( end=$((SECONDS+$2)); n=0
      while [ $SECONDS -lt $end ]; do n=$((n+1))
        curl -s ${RESOLVE:+--resolve "$RESOLVE"} -o /dev/null -m 35 -w '%{http_code} %{time_total}\n' "https://$HOST/?zdd=$1-$i-$n" >> "$TMPDIR/zdd-probe/$1/load-$i.log"
      done ) &
  done
  wait
}
summary() { cat "$TMPDIR/zdd-probe/$1"/load-*.log | awk -v p="$1" '$1 !~ /^2/ {bad++} {if ($2>max) max=$2; n++} END {printf "%s: %d requests, %d failed, slowest %.2fs\n", p, n, bad+0, max}'; }
```

### 3. Run it

```bash
cd "$TMPDIR/zdd-probe"
carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-1 probe-zdd-1
load baseline 60; summary baseline
( load overlap 240 ) & sleep 20
carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-2 probe-zdd-2
sleep 40
carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-3 probe-zdd-3
wait; summary overlap
# Control: hold mode must show the pause, or the probe proves nothing.
carlos apps set "$APP" --account "$ACCOUNT" --console "$CONSOLE" --swap hold
( load hold 120 ) & sleep 20
carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-4 probe-zdd-4
wait; summary hold
carlos apps set "$APP" --account "$ACCOUNT" --console "$CONSOLE" --swap overlap
```

**Pass.** `overlap` has 0 failed and its slowest is at most the larger of 1 s
and twice the baseline's slowest. Each deploy's watch ends with the approved
`swapped https://$HOST` line and a box line, exit 0. The `hold` control's
slowest is clearly above the baseline. If it is not, the probe cannot see a
pause and the overlap result means nothing.

The watch prints `swapped https://<host>` (the approved word `swapped`,
`deploywatch.go:569`) and `box <label>: ...` (`deploywatch.go:44`). With the
flag off it prints `live https://<host>` instead (`deploywatch.go:564`), so a
`live` line during the overlap phase means that box is not armed.

On the control: the first `--swap hold` on an app creates `config.json`, and
on boxes running this release that restarts nothing (the Q3 note above), so
the hold phase's pause is the zdd-4 swap's. If the numbers look odd anyway,
rerun the hold phase once with the policy already set.

### 4. Clean up

```bash
rm -rf "$TMPDIR/zdd-probe"
```

On dev, run `carlos apps delete zdd-probe` once Paul is done with it. On
the flagship, delete the Gate F app as Gate F says.

## Names checked against this branch

Checked in the code on `zdd-6-surfaces` (HEAD `77d77b3a` plus this file;
the anchors final fix wave 1 moved were re-checked after it):
the verbs and flags `carlos system update|status`, `carlos apps set|create|delete`,
`carlos instances list|create|enable|retry`, `carlos deploy`, `carlos ship`,
`carlos promote`, `carlos promote --retry`, `carlos rollback`; the flag
`CARLOS_SWAP` and its exact value `overlap`; the unit `carlos-edge` and
`/etc/carlos/host.env`; the SERVING, INSTALLED and REGION columns; the
`swapped`, `live` and `box` watch lines; `cmd/changelog`;
`docs/site/changelog.json`, `docs/site/changelog-links.json` and the "Two
builds at once" section.

Added by zdd-flag-readiness-3 (checked on that branch): `__cgexec`
(`cgroup.ShimVerb`), `carlos-placement-check`, the decline code
`cgroup-placement-failed`, `cgroup.ErrLimitNotApplied`,
`cgroup.ErrStrictUnsupported`, `PrepareStrict`, `PrepareStrictParent`, the
agent log lines `placement before exec failed`, `generation start refused`,
`the start's context ended during placement`,
`REFUSING CARLOS_SWAP=overlap (<code>)` and `overlap swaps armed`, the control
service's `reset-failed` log line, `classifyRollout` and `follows` in
`deploywatch.go`, `cmd/zddprobe`, and the kernel-test variables
`CARLOS_GEN_KERNEL`, `CARLOS_SHIM_KERNEL`, `CARLOS_EDGE_HARDENING_KERNEL`,
`CARLOS_STRICT_KERNEL` and `CARLOS_CLAIM_SPAWN_KERNEL`.

Not checked here (not in this repo, or not verifiable without running
something): `roll-console.sh` and `update-platform-binary.sh` (platform-infrastructure);
`hack/sync-docs.mjs` and the website's AGENTS.md (carlosframework/website);
the live URLs; the AWS instance ids except `i-092c0c1eea75723cb` and
`i-05498eada6f0afd23`; the SSM `AWS-RunShellScript` documents; that `bc`
exists on the boxes; the dev account and the flagship's channel names; the
behaviour of the running boxes.