# Console release defences

Status: draft rev 5, 2026-10-06, for Paul's review. Rev 5 applies Astra
round 4 (`…-r4.md`, **accept with fixes**): rev 4's three fixes were judged
sufficient, and two new P2s are fixed. 14 (a non-canary route waiting for a
proof stays in the fast loop) is in §4.5, and 15 (readiness follows the
effective upstream, Unix socket first) is in §2.4. Rev 4 answered Astra round 3 (`…-r3.md`,
reject). Of the six round-2 findings it re-checked, four were fixed. Two
needed more: findings 10 and 11, now in §2.3, where groups key on a non-empty
`Socket` or `Addr` and every group route is stamped to the bound route's
restored version. New finding 13, a `promote --retry`, is in §2.5. All
accepted. Branch `design-console-release-defences`.
Rev 2 answered Astra round 1 (`CODEX-FINDINGS-console-release-defences.md`,
verdict reject, nine findings). All nine were accepted, and each fix is marked
**(codex N)** in the text: 1 §2.3, 2 §2.2, 3 §4.2, 4 §2.3, 5 §4.3, 6 §4.5,
7 §3.2, 8 §5.2 and §6, 9 §6. Rev 3 answers round 2 (`…-r2.md`, reject: 6 of
the 9 fixes sufficient, 3 new findings). All six are accepted. Round 1's 6,
7 and 9 are redone: a separate world-readable policy file instead of
host.env, a per-route program history for every route kind, and
direct-bucket mode for the rehearsal. The new findings 10 and 11 (rollback
scoped to the process group, registry stamps restored, refusal for every host
in the group) are in §2.3, and 12 (a binding mismatch keeps propagating the
restart's error) is in §2.2.

## 0. Why

On 2026-10-06 the flagship console (console.carloku.com, dashboard.carloku.com,
platform.carlosframework.com) answered 502 for 24 minutes, 09:30 to 09:54 UTC.

What happened, confirmed from the box journal and the bucket:

- At 09:30:15 a console release named `v0.24.2` was shipped to
  `apps/bab/console/releases/v0.24.2/` and promoted to `canary/console` at
  09:30:30. Its one artifact was the public `carlos-linux-arm64` from the
  v0.24.2 GitHub release (sha256 `bc53cfc5…`, matching that release's
  `SHA256SUMS.txt`): the CLI, not `./cmd/console`. The manifest says
  `built_by: carlos v0.24.2`. It was a hand step, not `roll-console.sh`, which
  builds `./cmd/console` and names versions by git sha.
- The README's release recipe never mentions the console, and the public
  release carries no console artifact, so whoever rolled v0.24.2 had nothing
  correct to ship.
- The root stage pass staged it, repointed all three live links, and ran
  `systemctl try-restart console.service`. `console serve` is not a `carlos`
  verb, so the new process printed the CLI's usage and exited 2 after 55 ms.
  `try-restart` had already returned 0 (the unit is `Type=simple`), so
  `adoptHost`'s existing rollback branch (`internal/release/adopt.go`, the
  `rerr != nil` arm: poison, repoint back, start) never ran.
- systemd restarted the unit five times in one second, hit its start limit and
  left it `failed`. The live links named the bad build; nothing looked again.
- Nothing alerted. A person noticed.

Recovery was a promote of the previous build `f4478b15` back to
`canary/console`, then `systemctl reset-failed` and `start` over SSM: the
repoint alone does not revive a unit systemd has given up on.

## 1. Rulings

These were decided with Paul on 2026-10-06 and are not reopened here.

1. **Four defences**, in this order of value: (D1) a box rolls back a named
   unit that will not come up; (D2) `carlos ship` and adoption refuse a binary
   whose Go main package differs from the app's previous release; (D3) the
   flagship console adopts only builds proven on dev; (D4) process: the release
   publishes a console artifact and the recipe rolls it dev first.
2. **No new AWS account.** Dev (account 640012953448,
   platform.dev.carlosframework.com) already is the separate test deployment.
   The failure was that it was skipped. D3 makes skipping it impossible for the
   flagship console without root on the console box.
3. **D1 and D2 are on for every deployment** and add no steps. **D3 is off
   unless a deployment turns it on.** An operator who does not want the
   testing ceremony (Tito, a self-hoster) configures nothing and adopts
   exactly as before, still covered by D1 and D2. There is no per-promote
   bypass flag: the bypass is local and explicit, as for bake windows
   (`internal/release/bake.go`).
4. **Enforcement lives on the box that adopts**, not in the `carlos` client.
   Anyone holding a deployment's release key can promote anything, and
   today's promote was a hand step, so a client-side check would not have
   fired.
5. **Uptime alerting from outside the box** (would have paged within a
   minute) belongs to #192, not this spec.
6. **Scope is named-unit routes** (`registry.Route.Unit`, today only
   `console.service`) for D1 and D3. Box binaries keep `carlos system update`'s
   own gate and rollback. Exec children and `carlos-app@` tenants are out of
   scope for D1; D2 applies to every binary app.

**Threat model.** These are defences against mistakes. A compromised release
key or bucket is the signing chain's and the bake window's problem. Where a
mechanism below trusts an unsigned manifest field, that is why.

## 2. D1: roll back a named unit that will not come up

### 2.1 The gap

`makeRestartRoute`'s first arm (`internal/host/agent.go`) returns
`systemctlTryRestartUnit(rt.Unit)`, whose error is only systemctl's. For a
`Type=simple` unit that is "the start job was queued", which says nothing about
whether the new binary runs. `adoptHost` already does the right thing when the
restart callback returns an error: it poisons the version for the pass,
repoints back to `cur`, restores capability drop-ins, and starts the unit on
the restored build (`WithStartUnit`). D1 makes the named-unit arm return an
honest error.

### 2.2 The binding comes first (codex 2)

The console is one unit behind several hostnames, and adoption is per host in
`ORDER BY host` (`internal/registry/registry.go:800`). Each `adoptHost`
repoints its own `live/<host>` and then restarts. The unit runs whichever link
its `ExecStart` names: on flagship that is `live/platform.carlosframework.com`,
which sorts **last** of the three. A restart fired from an earlier host runs
the old build. The console-as-an-app spec already requires that the one
`Unit`-stamped route be the host `ExecStart` names (2026-08-23 design §4c), and
`roll-console.sh`'s final `systemctl restart` is what has hidden it when it
isn't.

So D1 judges a restart only when the binding is verified. Before the
settle check, the named-unit arm reads `ExecStart` (`systemctl show -p
ExecStart <unit>`). Its executable path must be `live/<rt.Host>` for this
route. If it is not, the arm logs `binding: <unit> runs live/<X>, but the unit
stamp is on <rt.Host>; restart not checked`, does today's plain try-restart,
and returns **its error unchanged**, as the arm does today
(`internal/host/agent.go:3107`), so a failed `try-restart` still drives the
existing rollback (codex 12). Only the settle check is skipped. That is no
worse than today. Rollout step 1 (§6) verifies the
binding on every console box and restamps where it is wrong, and
`roll-console.sh` keeps its final restart until the binding check passes on
that box.

### 2.3 The settle check

After `systemctl try-restart <unit>` returns 0 on a verified binding, the
named-unit arm waits for the unit to **settle**:

- Poll every 500 ms, up to **30 s** (`CARLOS_UNIT_SETTLE`, a duration, floor
  5 s), reading `ActiveState`, `InvocationID`, `MainPID` and `NRestarts` from
  `systemctl show`.
- The first poll that finds the unit `active` with an `InvocationID`
  different from the one before the restart fixes **the new invocation**.
  Everything after is judged against it, never against the previous
  invocation's counters: systemd resets `NRestarts` on a manual start, so a
  before-and-after comparison misreads healthy restarts (codex 4).
- **Healthy** when, within the new invocation, `ActiveState=active`, the
  MainPID's `/proc/<pid>/exe` resolves into the new version's directory, and
  the route's readiness check passes (§2.4). Return nil immediately. The
  console takes about 8 s, so a healthy restart costs about 8 s, not 30.
- **Failed** as soon as `ActiveState` is `failed`, or the `InvocationID`
  changes again (systemd restarted it), or `NRestarts` within the new
  invocation goes above zero. Return an error carrying the state and the last
  20 lines of `journalctl -u <unit> -o cat`.
- **Failed** at the deadline if the unit never satisfied "healthy". That
  includes a process that is still running but never became ready.

A unit that was **not running** before the restart (an operator stopped it)
is not checked: `try-restart` started nothing, and that stays a no-op.

The check runs inside the root stage pass and makes it up to 30 s longer
(about 8 s for a healthy console). A stage request written while that pass
runs past the relay's 15 s window waits for the next trigger: the known gap
recorded with the deploy-pickup fix. Only the bound route (§2.2) carries the
`Unit` stamp, so there is one checked restart per unit per pass. Hosts that
sort after it skip a rejected version through `poisoned`, and hosts before it
are repointed by the rollback below.

On a failure the existing `adoptHost` branch then repoints to `cur`. The
rollback must **replace** whatever is running, not merely start (codex 1). A
deadline failure can leave the rejected build running and unready, and
`systemctl start` does nothing to a running unit (`internal/host/agent.go:3208`).
So for a named unit that D1 rejected, the rollback runs `systemctl
reset-failed <unit>`, which clears a start limit, then `systemctl restart
<unit>`. Every other caller of `makeStartRoute` keeps `start`. The rollback
gets the same settle check against `cur`'s directory. Its failure is logged
as "rollback did not come up either", the one case that still needs a person
(§2.6 alerts it).

**The process group (codex 10, 11).** Hosts that sort before the bound host
may already have repointed their links to the rejected version and had their
registry versions stamped (`SetVersions`, `internal/release/adopt.go:1203`),
and the edge's `X-Carlos-Version` comes from that stamp
(`internal/edge/edge.go:752`). The rollback covers exactly the routes that
share the rejected **process**. A route's upstream is its `Socket` if set,
else its `Addr` (`Socket` wins, `internal/registry/registry.go:40`). The group
is the bound route plus every unstamped route of the same app with the **same
non-empty upstream**. A bound route with no upstream at all is a group of one,
and two empty upstreams never match (codex 10, round 3). A route of the same
app with a different upstream is a different process, and the rollback does
not touch it, whatever version it runs.

Every route in the group is served by the one process, so after a rollback
each of them is serving whatever the bound route rolled back to. On rollback,
every group route whose live link names the rejected version is repointed to
the **bound route's restored version**, and its registry versions are set to
match with `SetVersions`. This applies whether the route moved in this pass
or in an earlier, interrupted one (codex 11, round 3). The stamp then names
what is actually answering. The refusal record (§2.5) is written for **every
host in the group**, so no sibling re-adopts the rejected build on the next
pass.

### 2.4 Readiness

In order:

Readiness follows the route's **effective upstream**: `Socket` if set, else
`Addr`. `Socket` wins, as it does at the edge
(`internal/registry/registry.go:40`), and named units may be socket-backed
(`unitRail`, `cli.go:1624`) (codex 15, round 4). In order:

1. The route's own health probe (`registry.Route.Health`, the probe-config
   JSON instances already use), sent as HTTP to the effective upstream: over
   the Unix socket for a socket-backed route, over TCP to `Addr` otherwise.
2. Else, for a route with an upstream: a connect to it succeeds (Unix or TCP).
3. Else: the unit stays `active` within one invocation for 5 s.

The three console routes get `{"path":"/healthz","status":200}` as part of
rollout (§6). The console's `/healthz` (`internal/console/server.go`) is
unauthenticated and answers as soon as the listener is up.

### 2.5 Refusing the build until something changes

`poisoned` lives for one pass. Without more, the next pass, and the agent's
2 s fast loop asking for a root stage pass every 15 s, would re-adopt the same
bad build and bounce the console each time.

When D1 rolls a route back, the root pass writes a **refusal record** to the
box's bake directory (`BakeDir`, under the data dir, which the sandboxed agent
can read):

```json
{"schema":1,"host":"console.carloku.com","app":"console","version":"v0.24.2",
 "sha256":"bc53cfc5…","platform":"linux-arm64",
 "pointer_promoted_at":"2026-10-06T09:30:30Z","refused_at":"…","reason":"…"}
```

`adoptHost`, on every pass, root or agent, before `ensureStaged` and before
`stageBlocked` can relay: if the channel pointer names this sha256 for this
platform **and** the pointer's `PromotedAt` equals the record's, print
`refused: <version> did not come up on this box at <time> (<reason>); retry
with: carlos promote --retry …` and return. A pointer that moves to different
bytes clears it, and so does a pointer re-signed with a new `PromotedAt`.

**The retry needs a new flag (codex 13, round 3).** Promoting the version a
channel already names is a successful no-op that does not rewrite the pointer
(`PromoteOnLadder`, `internal/release/release.go:910`), so a plain re-promote
leaves `PromotedAt` unchanged and the refusal standing.
`carlos promote --retry <version> <channel>` re-signs the pointer with a fresh
`PromotedAt` even when it already names `<version>`. It refuses unless it does
already name `<version>`, so it can only renew a promotion, never change one.
The API promote door takes the same `retry` field. Both reach it through one
`PromoteOnLadder` option, so the ladder, scope and authorization checks run
exactly as for any promote.

The record keys on the artifact sha256 read from the manifest, not on the
version name, because names get reused.

### 2.6 Being seen

- A rollback writes a line naming both versions, the reason and the journal
  tail to the root pass's output (journal).
- The agent raises an alert through its existing sinks (`alertsink.go`: SNS,
  which works on flagship, and `CARLOS_ALERT_WEBHOOK`) the first time it reads
  a new refusal record: subject `console.carloku.com: v0.24.2 did not start,
  rolled back to f4478b15`. "Rollback did not come up either" alerts with a
  louder subject. The plan decides whether this joins the metrics `Event`
  shape or a small sibling; it must not wait on #192.
- `carlos status --live` shows a refused host with the refusal's version and
  reason.

## 3. D2: refuse the wrong program

### 3.1 At ship

For a `binary` artifact, `carlos ship` reads the Go build info with
`debug/buildinfo.ReadFile`. This works on a stripped (`-s -w`) binary built for
another architecture without running it: a stripped arm64 `carlos` read on
amd64 gives `path github.com/carlosframework/platform`, and a console build
gives `github.com/carlosframework/platform/cmd/console`.

- The manifest gains `program` (string, the build info's main package path),
  additive to schema 1 like `capabilities` and `label`: omitted when empty,
  ignored by old binaries. A non-Go binary has no build info and no `program`.
  With several platform artifacts, every one must report the same path, or
  ship refuses.
- Ship reads the app's **latest** release's manifest. If both name a
  `program` and they differ, ship refuses:

  ```
  this artifact is github.com/carlosframework/platform; console's releases
  are github.com/carlosframework/platform/cmd/console. If the app really is
  changing program, ship again with --program-change.
  ```

- `--program-change` ships anyway and records `"program_change": true` in the
  manifest.
- No previous release, a previous release without `program`, or a non-Go
  artifact: ship as today. The check starts with the next release of each app.
- The console's API ship door (`internal/console/apicli.go`, the
  `release.Ship` caller) makes the same check server-side, so API-mode ships
  cannot skip it and direct-bucket ships (`cli.go`) do it client-side. Both
  call one function in `internal/release`.

"Latest" is the newest release by manifest `built_at` under
`apps/<acct>/<app>/releases/`. The plan may use the channel heads instead if
listing every release is too slow; either catches today's case.

### 3.2 At adoption

`program` is an unsigned manifest field, and an older client writes none. So
the box checks the bytes themselves. In `adoptHost`, after `ensureStaged` and
before the capability converge and `PointLive`: read build info from the
staged new binary and from the binary `cur` names. If both are readable and
their main package paths differ, and the new manifest does not carry
`program_change: true`, refuse:

```
console.carloku.com: v0.24.2 is github.com/carlosframework/platform, the live
build is …/cmd/console — not pointed live (ship with --program-change if this
is intended)
```

and write a refusal record (§2.5) so the agent does not relay it again. This
alone would have stopped today's outage on the box, with no service
interruption, even with the bad manifest already promoted.

**Going back across a deliberate change (codex 7).** After an intended A→B
change, promoting an old A release back would be refused, because A's
immutable manifest has no `program_change`. So each route keeps a **program
history** in the bake directory: every main package path the box has pointed
live for that route, recorded at `PointLive` for every route kind (exec,
`carlos-app@`, named unit). The check passes when the staged binary's program
is in the route's history. Going back to A is therefore accepted on any route
that ran A here, and a program the route has never run still needs
`--program-change`. A route with no history yet, such as a box upgraded to
this release, seeds it from the live binary on its first pass.

## 4. D3: the flagship console adopts only builds proven on dev

### 4.1 Shape

- A **proving deployment** (dev) mints a signed **proof** once a build has
  run healthily on its console for a bake period.
- An operator **carries** the proof to the target deployment's bucket. It is
  signed, so carrying needs no trust and no access between the AWS accounts.
- A **requiring deployment** (flagship) adopts a build of a listed app only
  with a proof for those exact bytes from a prover it trusts, or a build it
  has itself run healthily before.

### 4.2 The proof

```json
{"schema":1,"app":"console","platform":"linux-arm64",
 "sha256":"e47fcc1e…","version":"f4478b15",
 "prover":"dev","active_since":"…","proven_at":"…",
 "sig":"<base64 ed25519 over the canonical JSON of every other field>"}
```

It binds app, platform and artifact sha256. `version` is informational. The
target compares sha256, so flagship must run **the same bytes** dev ran
(§5.2), and a version name reused for different bytes has no proof.

**Every sha256 in D3 is hashed from bytes, never copied from a manifest
(codex 3).** A version name can be reused after pruning
(`internal/release/release.go:632`). An already-staged binary skips the
download check (`internal/release/adopt.go:1317`), and `backfillManifest`
fetches whatever manifest the bucket now holds for that name without
comparing it to the staged file (`internal/release/adopt.go:1402`). So a
manifest's sha256 can name bytes the box is not running. Minting hashes the
running process's executable (`/proc/<MainPID>/exe`). The requiring box
hashes the staged file it is about to point live, on a cache hit too, and the
healthy history records the hash of the executable that passed the settle
check.

The proving key is a fresh ed25519 key, separate from both release keys, held
only by the proving console box as a root-only file. `carlos proof keygen`
writes the private half and prints the public half.

### 4.3 Minting on dev

Dev's console box gets a `prove` block in its proof policy file (§4.5):
apps `console`, prover `dev`, `after` 15m, and the private key at
`/etc/carlos/proving.key` (root 0600).

A root pass mints a proof for a named-unit route of a listed app when all of
these hold at once:

- the route's binding is verified (§2.2), and the unit is `active` with an
  `ActiveEnterTimestamp` at least `after` ago and the same
  `InvocationID` throughout;
- S is the sha256 of the bytes of `/proc/<MainPID>/exe`, and the live link
  resolves to that same file;
- the route's readiness check (§2.4) passes now;
- no refusal record names S;
- no proof for S exists yet in dev's bucket.

It writes `apps/<acct>/<app>/proofs/<sha256>/<prover>.json` to dev's own
bucket: the same key shape the target uses (§4.4), so `proof get` and `proof
put` move one object between two buckets under one name.

Minting needs root (the key), and root passes run hourly or when the agent
asks. So the agent asks: on its fast loop, when a listed app's route has been
live and active for `after` with no proof in the bucket, it
requests a root pass through the existing stage-request relay. It **keeps
asking every 2 minutes until it sees the proof in the bucket** (codex 5). The
relay can miss a request written while `carlos-stage.service` is already
running (`internal/host/agent.go:1203`), and a mint write can fail. A proof is
then normally minted within a minute or two of the bake period ending, and at
worst within the hourly pass.

Dev's bucket layout and IAM already let the box write under its own prefix;
the plan confirms the exact prefix the box role may write.

### 4.4 Carrying it

- `carlos proof get --app console --sha256 <S>`, against the proving
  deployment (dev env), prints the proof JSON, or fails "not proven yet:
  <S> has been live on dev since <t>, proven after <15m>".
- `carlos proof put --app console <file>`, against the target (flagship env),
  checks the JSON's shape and writes it to
  `apps/<acct>/<app>/proofs/<sha256>/<prover>.json`. It cannot check the
  signature without the target's trust config, and need not: the box does.

`roll-console.sh` runs both (§5.2).

### 4.5 Requiring on flagship

Flagship's console box gets a `require` block in its proof policy file,
naming app `console`, prover `dev` and dev's public key. The file's shape and
how it is read are below.

In `adoptHost`, for a route whose app is listed, the check runs twice:

1. **A cheap gate, after reading the new manifest and before
   `ensureStaged`,** so an unproven build is neither downloaded nor relayed to
   root. The box takes the manifest's sha256 for its platform and looks for
   `apps/<acct>/<app>/proofs/<sha256>/<prover>.json`. It continues if the
   proof parses, its signature verifies against the configured key, and its
   `app`, `platform` and `sha256` match.
2. **The binding check, after `ensureStaged` and before `PointLive`.** The box
   hashes the staged file and requires that hash to equal the proof's
   sha256, or to be in the healthy history. A mismatch means the staged bytes
   are not the proven ones (a reused name, a stale cache). The box refuses
   the version with a refusal record (§2.5) naming both hashes.

It also proceeds, with no proof, if the sha256 is in the box's own **healthy
history**: a record in the bake directory of builds this box has run well. A
root pass adds a sha256 when D1's settle check passes for it, and also when it
finds any named-unit route's live build meeting §4.3's first three mint
conditions (bound, active in one invocation for the policy's `after`, the
hashed exe, ready; 15m when there is no policy file). The second rule is what puts the build already live when
the requirement is switched on into the history without a restart. So rolling
back to a build flagship ran before, as this morning's recovery did, never
waits on dev.

Otherwise it prints `console.carloku.com: v… (sha256 …) is not proven on dev;
staying on <cur>` and returns. This is not a refusal record: the box looks
again, so the build adopts soon after its proof arrives. The agent's fast
loop runs every 2 s, so a missing proof is cached per (route, sha256) for
30 s: at most two bucket GETs a minute per listed route while waiting, none
once the build is live.

**A waiting route stays in the fast loop (codex 14, round 4).** `fastMembers`
(`internal/host/agent.go:1550`) includes canary routes on every tick, but
drops a non-canary route as soon as it has a live version. So a non-canary
route waiting for a proof would be rechecked only by the next serial bump or
the hourly pass. While a listed route has an outstanding "not proven" verdict
for its channel head, `fastMembers` keeps it as a member, and the 30 s cache
above bounds what that costs. The console is on `canary/console` and is
always a member; this is for any other listed app.

`carlos status --live` shows "waiting for a dev proof" for that host.

**Where the setting lives (codex 6).** The agent gets host.env through
`EnvironmentFile` only when it starts (`internal/host/units/carlos-edge.service:18`),
while each root pass starts fresh (`carlos-stage.service:30`). An edit would
reach one process and not the other, and restarting `carlos-edge` takes the
box's edge down. host.env is also root-only 0600
(`infra/modules/region-host/README.md:23`), so the agent cannot read it
itself.

So the policy is **not in host.env**. It lives in its own file,
`/etc/carlos/proof-policy.json`, root-owned, mode 0644. It holds no secrets:
app names, the prover name, public keys and durations:

```json
{"schema":1,
 "prove":{"apps":["console"],"prover":"dev","after":"15m",
          "key_file":"/etc/carlos/proving.key"},
 "require":[{"app":"console","prover":"dev","key":"<base64 ed25519 public key>"}]}
```

Both the agent and root read it at the start of every pass. An absent file
means D3 is off. An unreadable or unparseable file means D3 is **on and
waiting** for every named-unit route, since the box cannot tell which apps
were listed. It logs this every pass, because a typo must not open the gate. `key_file` names the root-only
private key, and only root opens it.

**Break glass:** delete the `require` entry (or the file) as root,
then `touch` the stage-request path (or run `carlos adopt --stage-only`) so a
root pass adopts at once. The agent's next tick also sees the line gone.
While the requirement is off for an app on a box that has required a proof
for it before (a marker the root pass writes when it reads a `require`
entry), the box logs once an hour that the requirement is off. A box that
never had the requirement, such as Tito's or a self-hoster's, stays quiet. Nothing in the bucket
or the CLI can switch it off.

### 4.6 Who sees none of this

A deployment with no proof policy file: no proof is minted, read or required,
and adoption is unchanged except for D1 and D2. Tito, self-hosters and every
customer app on carloku.com are in this position.

## 5. D4: process

### 5.1 The release carries a console

The release recipe (README "Releasing the platform") builds
`console-linux-arm64` and `console-linux-amd64` from the tag with
`go build -trimpath -ldflags "-s -w -X …version.Build=vX.Y.Z" ./cmd/console`,
lists them in `SHA256SUMS.txt` (signed as today), and publishes them with the
other files. Step 4's "eight files" becomes ten.

### 5.2 The console roll recipe

`carloku-infrastructure/scripts/roll-console.sh` changes:

- **A `dev` target** with the dev coordinates (profile `kass`, bucket
  `carlos-dev-640012953448`, console box `i-05498eada6f0afd23`,
  platform.dev.carlosframework.com). The dev coordinates move here from
  platform-infrastructure, because flagship's rule now depends on dev.
- **`--artifact <file>`**: roll exactly this file instead of building. A
  release's console roll uses the public `console-linux-arm64`, verified
  against the signed `SHA256SUMS.txt`, so dev and flagship run the bytes the
  public got. Without `--artifact` it builds once with `-trimpath` and keeps
  the file for the next target.
- **`flagship` runs `proof get` against dev first** and refuses to ship if
  there is none, naming how long dev has had the build. With a proof it ships,
  promotes, runs `proof put` and verifies as today.
- **The adopt and restart SSM steps go once the box's binding is verified**
  (§2.2): the script reads the binding line from the box first, and keeps its
  final `systemctl restart` on any box where the binding is wrong. Otherwise
  adoption restarts the unit and D1 checks it. The `/proc/<pid>/exe`
  verification stays.
- **A `--bootstrap` flag for the first flagship roll only** skips the proof
  step, for the one roll that installs D3 before dev can mint (§6). It prints
  that it is skipping. It does not exist on the box: flagship's requirement is
  switched on only after that roll.

The recipe's console step reads: roll the release's console to dev, wait for
the proof (15 minutes), roll it to flagship. It says in so many words that
the public `carlos-linux-*` binary is never a console artifact. D2 enforces
that now, but the recipe should not lean on it.

## 6. Rollout

Order matters: the box binary is what checks, so it goes first.

1. **Fix the bindings.** On dev's and flagship's console boxes, check that the
   one `Unit`-stamped route is the host `ExecStart` names, and restamp where
   it isn't (2026-08-23 design §4c). Flagship was stamped on
   `console.carloku.com` while `ExecStart` names
   `live/platform.carlosframework.com`; confirm the current state on the box.
2. Ship D1, D2 and D3 in one platform release. With no config, D3 is inert.
3. Set the health probe on the console routes on dev and flagship.
4. Roll the release's boxes (`carlos system update`) on dev and flagship.
5. **On dev:** run `carlos proof keygen` on the console box, write its proof
   policy file with the `prove` block, and roll the console to dev with the
   new recipe. That
   roll is the first live check of D1: watch the journal for the settle line.
6. **Rehearse on dev** (codex 9):
   - Ship the public `carlos-linux-arm64` as a console release using a
     **v0.24.1 or older client in direct-bucket mode** (`CARLOS_DEPLOYMENT_BUCKET`
     set, no `--console`), which is how today's bad ship went out. That
     client has no ship-side check and writes no `program`. API mode would
     hit the console's server-side check instead (codex 9, round 2). Promote
     it and confirm that the box's D2 check refuses it, with no restart.
   - Ship a console built to exit 2 on start, from a real `./cmd/console` so
     D2 passes. Confirm D1 rolls back within 30 s, writes a refusal, raises
     the alert, stays refused after a plain re-promote, and retries after
     `carlos promote --retry`.
   - Ship a console that runs but never answers `/healthz`. Confirm the
     deadline rollback replaces the running process.
   - Delete all three releases.
7. **Wait for dev's proof** of the build rolled in step 5.
8. **Roll that build to flagship.** The recipe carries the proof (codex 8).
   If the flagship console must move before dev can mint, use `--bootstrap`
   once.
9. **On flagship:** write the proof policy file with the `require` block and
   dev's public key. The live build is unaffected, because the requirement
   gates moves only, and it enters the healthy history 15 minutes later
   (§4.5).
10. Every later console roll goes through dev, with the proof carried.

The proof policy files are not delivered by tofu `user_data`, because a
whole-root apply restarts every box (memory: carloku-infra-split). Write them
over SSM and record their contents in carloku-infrastructure's runbook (for
flagship) and platform-infrastructure's (for dev). The plan checks whether
they have a managed home they should go in instead.

## 7. Testing

- **D1:** `makeRestartRoute`'s named-unit arm with a fake `systemctl show`
  sequence: healthy after N polls; `failed`; `InvocationID` changed after
  the new invocation was fixed; `NRestarts` reset to 0 by the manual restart
  and the unit healthy (must pass); deadline with the process still running;
  not running before (no check); `ExecStart` naming another host's link (no
  check, plain try-restart). A D1 rollback runs `reset-failed` then
  `restart`, and replaces a still-running rejected process. Other start
  callers keep `start`. A binding mismatch whose `try-restart` fails still
  returns the error and rolls back. `adoptHost` with a failing settle:
  every group route whose link names the rejected version is repointed and
  stamped to the bound route's restored version, including one advanced by
  an earlier interrupted pass; groups key on `Socket` before `Addr`; two
  routes with empty upstreams are never grouped; a same-app route on a
  different upstream is left alone; the refusal record is written for every
  host in the group; the next pass, agent or root, skips without relaying.
- **Readiness:** a socket-backed named unit with a configured probe passes
  over its Unix socket; a route with both `Socket` and `Addr` is probed on the
  socket only.
- **Retry:** a plain re-promote of the refused version is still a no-op and
  the refusal stands; `promote --retry` through the CLI (direct mode) and the
  API door re-signs with a fresh `PromotedAt` and clears it; `--retry` against
  a channel naming a different version is refused.
- **D2:** `buildinfo` read from tiny stripped binaries with two different
  main packages, cross-built for linux-arm64 by a test helper at test time
  (no binaries checked in). Ship refuses, accepts with
  `--program-change`, accepts non-Go, accepts a first release. The API door
  and the CLI call the same function. `adoptHost` refuses a mismatched staged
  binary before `PointLive` and writes a refusal. It accepts a program in the
  route's program history (rollback across a deliberate change) on an exec
  route, a `carlos-app@` route and a named-unit route, and seeds the history
  from the live binary on a route that has none.
- **D3:** proof canonical encoding round-trip and signature; mint conditions
  each false in turn, including a live link and a running exe that hash
  differently; `adoptHost` with the requirement set: no proof, bad signature,
  wrong sha256, wrong app, wrong prover, valid proof, healthy history; a
  valid proof for the manifest's sha256 with a staged cache hit whose bytes
  hash differently (refused at the binding check). The policy file changes
  between passes and the agent sees the change without a restart; an
  unparseable policy file keeps the gate closed. The agent re-requests a mint
  until the proof appears. A non-canary listed route with a live version
  stays in `fastMembers` while it waits for a proof, and leaves once it
  adopts. No requirement set: no bucket GET is made (assert on the fake
  store).
- **The dev rehearsal in §6 step 6** is the end-to-end test. Mocks cannot show
  that systemd's start-limit state and `reset-failed` behave as written.

## 8. Not in this spec

- External uptime alerting: #192.
- D1 for exec children and `carlos-app@` units. Those restarts already have
  other failure handling, and a wrong-program ship to them is D2's job.
- Making `carlos system update` require dev proofs for box binaries. Its own
  gate covers box binaries, and the proof shape extends to it if wanted later.
- Deleting the bad `apps/bab/console/releases/v0.24.2` release. That is an
  operator clean-up, independent of this work; D2's box check makes it inert
  once rolled.