# `carlos system update` — the platform rolls itself

**Date:** 2026-09-06 · **Status:** draft, two review rounds (Fable, Codex),
awaiting Paul · **Requested by:** Paul

**Builds on:** discussion #100 (boxes should self-update via a channel
subscription), 2026-08-23-console-as-an-app-design.md (slice 3, deferred
there; the dividing line this spec applies), the root-relay pattern
(`carlos-stage.path` → `carlos adopt --stage-only`), and pinfra's
`update-platform-binary.sh`, whose 85 lines are the procedure this spec
moves into the binary and then retires.

## 0. How this spec was arrived at

One brainstorm with Paul on 2026-09-05/06, after a merge of `main` stalled
on "and deploy": the only way to roll a platform binary today is an
operator with SSM access to every box, running a script per box, in an
order held in operator memory. Paul's framing, which is the requirement:

> Ultimately, a fleet operator should be able to do `carlos system update`
> and the whole platform updates to the latest version based on what we
> just released. […] And I guess each piece should be updatable
> individually, `carlos system router update` etc.

Three questions settled the shape before any design was written:

- **Trigger: promote-driven.** A fleet operator says when. Boxes do not
  poll the public release feed and roll on their own; a new release exists
  in the world for as long as the operator wants before any box sees it.
- **Rollout: two channels, two promotes.** `canary` and `stable`, one
  operator action each, exactly like an app. No auto-promotion in v1.
- **Third-party deployments in v1.** Tito's operator rolls Tito's boxes
  from Tito's console. The artifact comes from the public signed release,
  so the vendor never needs access to a customer box (the trust shape #100
  was filed about).

Three approaches were compared; this document is the first. The other two
are recorded in §12. The draft went through two rounds of review by two
independent readers checking claims against the code (§13); everything
below is the post-review shape, and §13 says what the drafts got wrong.

One ruling stated up front because it changes what "each piece" means:
the edge, the activator and the adoption loop are **one process**
(`carlos agent`, unit `carlos-edge.service`). There is nothing to update
"individually" inside it without splitting the process, which is a
separate project. So the honest component list today is one: the
**router**, which is the substrate binary. The console is already an app
and is updated as one (`carlos deploy --app console`). §3 keys everything
per component so a split later adds a name, not a schema.

---

## 1. What exists, and what is missing

The platform already has most of this machinery, built for apps:

| Have | Where | Reused as |
|---|---|---|
| Signed channel pointer, CAS-written, single parse site | `release.ChannelPointer`, `signPointer`/`verifyPointer` (`release.go:102,350,397`) | The platform pointer's wrapper, signing and verification |
| The one unsigned "something changed" signal, polled at 2s | `release.BumpSerial` on `control/serial` (`serial.go:51`), `fastConvergeLoop` (`agent.go:1412`) | How a box learns there is a new platform pointer |
| Root relay from a sandboxed agent | `/data/stage-requested` → `carlos-stage.path` → `carlos-stage.service` | Same shape, new marker, new oneshot |
| Fleet-box bucket proxy with an explicit grant for the platform binary | `storedoor.go:383` `aclRead`: exact `bootstrap/carlos`, "the sanctioned self-update path (F7)" | Boxes behind the door fetch the artifact with no public egress |
| Verified download of a public release | `update.go` `selfReplace`: tag-pinned URL, `SHA256SUMS.txt.sig` over the sums file, per-asset sha, lie-detector `version` run | The console's ingest of a release |
| A box's running build, published to the bucket | `version.Build` → `BoxMetrics.Release` in `status/boxes/<label>.json` every 5 min (`agent.go:2318`) | `carlos system status` and the fleet-side half of the gate |
| The unit list a box must have armed, and the exact `systemctl` sequence | `enabledUnits`, `systemctlPlan` (`bootstrap.go:72,96`): enable all, start timers and paths, **never start a service** | What apply arms, so apply and bootstrap never disagree and apply never starts itself |
| The apply procedure | `update-platform-binary.sh` (pinfra) | Moved into `carlos system apply`, line for line, then retired |

What is missing, and is the whole of this spec:

1. A **pointer for something that is not an app** — no account, no route,
   no upstream, and a version that is a platform tag, not a release
   serial.
2. A **box-side leg** that notices the pointer, fetches and verifies the
   artifact, and asks root to apply it.
3. A **root apply** that does the script's dance from inside the binary,
   with the three things the script does not have: a **journal** so an
   interrupted apply is undone rather than left half-done, a **health
   gate**, and a **rollback**. Today `carlos-edge.service` is
   `Restart=always`, `RestartSec=1`, with no `StartLimit*` of its own, so
   a binary that dies on start hits systemd's default limit (5 starts in
   10 s) and the unit goes to `failed` with start-limit-hit; nothing puts
   the old binary back, and there is no `.prev` anywhere.
4. A **verb** an operator runs, and a **status** they read.

---

## 2. The model: the platform is a subscriber of its own releases

One sentence: **a platform pointer names a version and its artifacts; a
box whose channel pointer names a version other than the one it is
running fetches, verifies, and asks root to swap and restart; root
journals every step, verifies everything again from a copy it owns, and
puts the old binary back if the new one does not come up and stay up.**

Four things the model deliberately does *not* do:

- **A box never decides to update.** It reacts to a pointer the deployment
  key signed, placed by an operator's promote. No box reads the public
  release feed.
- **A box never publishes the binary.** `aclWrite` already refuses
  `bootstrap/**` for boxes (`storedoor_test.go:81`), and stays that way.
  Only the console, holding the bucket and the signing key, writes the
  artifact and the pointer.
- **Root takes exactly one input from the sandboxed user, and treats it
  as a hint.** `/data` is `chown -R carlos:carlos` (`bootstrap.go:390`):
  any path under it can be a symlink, any file can be swapped between
  check and use. So root reads one file at one fixed path, walking to it
  component by component with no symlinks followed, and everything root
  then trusts — which channel, which version, which sha, which key —
  comes from a pointer root fetched and verified itself and from root's
  own `host.env`. The candidate binary is copied into a directory only
  root can enter, verified there, run there, and installed from there.
  A compromised `carlos` user can make the box install exactly what the
  deployment key signed for the box's configured channel, and nothing
  else — the trust the stage relay already grants, made true by
  construction.
- **Root never leaves the box between two builds.** Every step that
  changes the box is journaled before it runs; a killed oneshot, a
  timeout, or a reboot mid-apply is recovered — always backward, to the
  build that was known good — before anything else happens (§6).

---

## 3. The pointer: `control/system/<component>/<channel>.json`

Placed under `control/`, beside `control/serial`, because that is where the
deployment's platform-wide signals live, and explicitly **not** under
`apps/` — the console-as-an-app spec's line is *anything adoptable through
a route is an app*, and this is not.

```go
// SystemPointer is the platform's channel pointer. It is what a
// ChannelPointer is for an app, for the thing that does the adopting.
type SystemPointer struct {
    Schema     int             `json:"schema"`    // 1
    Component  string          `json:"component"` // "router"
    Channel    string          `json:"channel"`   // "canary" | "stable"
    Current    SystemRelease   `json:"current"`
    History    []SystemRelease `json:"history"`   // newest first; rollback targets
    Spread     string          `json:"spread"`    // "5m": boxes jitter their start over this
    PromotedAt time.Time       `json:"promoted_at"` // of THIS write; the replay guard reads this one
    Audit      []Promotion     `json:"audit"`     // reused from release.go
}

// SystemRelease is everything a box needs to install one version. It
// travels whole into History so a rollback re-installs what was there,
// not a version string with someone else's artifacts.
type SystemRelease struct {
    Version    string              `json:"version"`   // what `carlos version` must print
    Artifacts  map[string]Artifact `json:"artifacts"` // "linux-arm64", "linux-amd64"
    Source     string              `json:"source"`    // "release:v0.18.0" | "file"
    ReleasedAt time.Time           `json:"released_at"` // when first mirrored; audit only
}

type Artifact struct {
    Key    string `json:"key"`    // bootstrap/router/<version>/carlos-linux-arm64
    SHA256 string `json:"sha256"` // hex, of the object at Key
}
```

- **Signed with the deployment's release key**, using the same
  `signedWrapper` (payload, sig, optional grant) and the same root-key
  anchor `CARLOS_RELEASE_PUBKEY` a box already holds. **Root key only in
  v1**: a scoped grant's rungs name app channels, and a grant that could
  roll the platform is a bigger key than any app promoter should hold.
  `verifyPointer` grows a sibling `verifySystemPointer`; both share the
  wrapper parse, which stays the single parse site.
- **CAS-written** with If-Match like `ChannelKey`, so two operators cannot
  race each other into a lost promote.
- **Every write sets `PromotedAt` to now** — promote and rollback alike.
  The box's replay guard (§6) reads this field and only this field. A
  rollback that carried the old release's timestamp would be refused by
  every box as a replay of itself; hence the two differently named
  fields.
- **`Version` is a string the box compares for equality** with
  `version.Build`. For a public release that is the tag (`v0.18.0`, which
  release builds put in `Build`); for a `--file` promote (§5) it is whatever
  the operator names, which `carlos system apply` checks by running the
  staged binary's `version` before installing it. Dev deployments that roll
  git-sha builds keep working; nothing assumes semver.
- **A version is immutable.** The console refuses to mirror different
  bytes under a `bootstrap/router/<version>/` that already exists
  (§5 step 3). Without that, a reused `--file --version` could swap the
  artifact under a pointer stable boxes are still on, and boxes already
  reporting that version would never take the replacement.
- **`Artifacts` is per platform** because BYO boxes are not all Graviton.
  A box picks `runtime.GOOS-runtime.GOARCH`; a pointer with no entry for
  the box's platform is "no update for me", logged once, not an error.
- **`Spread`** is the one fleet-policy knob. A box that sees a new pointer
  waits `rand(0, Spread)` before requesting the apply, so a stable promote
  to nine boxes is nine edge restarts over five minutes, not nine in the
  same two seconds. `--spread 0` exists for a hotfix. Home-before-region
  ordering (#300) is *not* in v1: the canary channel, with home boxes on
  it, is the ordering an operator needs today. **The console's own box
  goes on canary by convention** — it hosts the store door, and a fleet
  box mid-apply that finds the door restarting gets a transient store
  error (§4 says what it does with one).
- **`History`** holds whole releases, `historyMax = 10`, and is what
  `carlos system rollback` swaps with (§5).

The artifact lives at **`bootstrap/router/<version>/carlos-<os>-<arch>`**.
`bootstrap/` because the door already means "the platform's own binary"
by that name; per-version because canary and stable boxes must be able to
run different binaries at the same time, which a single `bootstrap/carlos`
cannot express. The existing exact key stays as it is until §10 retires the
script.

**Store-door changes, both reads, in `aclRead`:**

1. `control/system/<component>/<channel>.json` — exact three-segment shape
   under `control/system/`. Without it, a door-backed box gets 404, which
   `fleetapi.Get` maps to `os.ErrNotExist` (`fleetapi.go:150`), and the
   fleet #100 exists for silently never updates.
2. `bootstrap/router/<version>/<asset>` — a *shape*, three segments, no
   listing (`aclDir` does not change), and still nothing else under
   `bootstrap/`. The existing pin that `bootstrap/carlos.sig` is refused
   stays, and gains siblings: `bootstrap/router/../../x` and
   `control/system/../serial` are refused.

Boxes need no signature file because the sha in the signed pointer *is*
the artifact's signature. `aclWrite` does not change: a box still cannot
write under `control/` or `bootstrap/`.

**A streaming read on the store.** `release.Store.Get` returns `[]byte`
and every implementation buffers (`store.go:43`; `fleetapi.go:133`);
`maxStoreBodyBytes` bounds PUT bodies only (`storedoor.go:180`), the
door's GET is uncapped and in memory. The box must not hold a binary in
memory under `MemoryHigh=70%`, so the interface gains
`Open(ctx, key) (io.ReadCloser, etag string, err error)` with all four
implementations (`S3Store`, `DirStore`, `fleetapi`, `prefixStore`).
The door's own GET stays in memory in v1 (§9, §11).

---

## 4. The box: notice, fetch, verify, ask

A new leg in the agent, **`systemLeg`**, with two entry points so the
spread has something to run it:

- `observe(ctx)` — called from `fastTick` on a serial move and from
  `adoptLoop` on its hourly pass. Reads the pointer, decides, and either
  does nothing or records a *pending* request with a deadline.
- `tick(now)` — called on **every** 2 s fast tick, reads no store. Fires
  a pending request when its deadline passes, and **re-touches the marker
  once a minute while a request is outstanding** — a request root has not
  answered in `result.json`. `PathChanged` does not replay a change made
  before the unit was armed, and drops one that lands while the service
  is mid-run (`carlos-restart.path`'s own comment); the stage relay
  survives that because the unstaged route is re-observed every 2 s, and
  this leg needs the same persistence by hand.

It is sandboxed (`User=carlos`, `ProtectSystem=strict`) and can write only
`/data` and `/run/carlos`, which is exactly enough.

1. **Read** `control/system/router/<channel>.json` through the box's normal
   store (S3, dir, or the fleet door — `openDeployStore` already picks).
   The channel is **`CARLOS_SYSTEM_CHANNEL` in `host.env`**, default
   `stable`; an operator puts a box on canary by editing one line and
   restarting the edge, which is how every other per-box setting works.
   Missing pointer → nothing to do; this is the state of every deployment
   until its first `carlos system update`.
2. **Verify** the pointer with the release pubkey. A pointer that does not
   verify is logged and ignored, like a bad channel pointer.
3. **Decide.** Read root's `seen.json` (§6; root-owned, world-readable)
   for this channel's watermark. Then:
   - pointer older than the watermark → ignore (root would refuse it);
   - `Current.Version == version.Build` and pointer not newer than the
     watermark → nothing to do;
   - `Current.Version == version.Build` and pointer newer → a **no-op
     request** (step 7 with `noop: true`, no artifact fetched), so root
     sees every promote and the watermark moves without a 25 MiB download
     on every box;
   - otherwise → an update. A dev build (`Build == "dev"`) never matches,
     so a hand-built binary on a dev box is never rolled over by accident;
     it *is* rolled when the operator promotes something, which is the
     point.
4. **Refuse to repeat a failure.** If `result.json` names this exact
   `(channel, version, promoted_at)` with outcome `rolled-back`,
   `rollback-failed` or `refused`, do nothing until the pointer changes.
   A pointer that broke one box must not have that box firing a root
   oneshot at every serial bump. `deferred` (a transient store error on
   root's side) is retried with the per-host backoff the unstaged-route
   sweep uses. A terminal outcome is also when the leg **deletes its own
   cache** under `/data/system/router/<version>/` — root never deletes
   anything under `/data`.
5. **Jitter** `rand(0, Spread)` from the first sighting, as the pending
   deadline `tick` watches.
6. **Fetch** the artifact for this box's platform with `Store.Open`,
   reading at most 96 MiB + 1 byte — the extra byte is how an object
   exactly at the limit is told from one over it — with a running sha,
   into `/data/system/router/<version>/carlos.tmp` then `rename`. The
   object is never in memory on the box; an oversized object from a
   bucket writer is cut off at the limit, not at the OOM killer.
   **Verify** the sha against the pointer; mismatch → delete, log, retry
   with backoff.
7. **Write the request** `/data/system/router/requested.json`
   `{version, promoted_at, noop, seen_at}` — **no channel, no key, no
   sha, no path**: root takes the channel from its own `host.env`, the
   key and sha from the pointer, and derives the cache path from the
   version — and **touch `/data/system-update-requested`**, throttled to
   once a minute like `stageRequester`.
8. **A transient store error** anywhere above (the door restarting because
   the console box is rolling, a network blip) is a retry, not a refusal:
   nothing is recorded, the next `observe` starts over.

The marker fires **`carlos-system-update.path`** (`PathChanged`, the same
form as `carlos-stage.path`), whose unit is **`carlos-system-update.service`**:
`Type=oneshot`, root, `StartLimitIntervalSec=0`, `TimeoutStartSec=15min`
(a hung apply is a failed apply), `OnFailure=carlos-system-recover.service`
(a killed or timed-out apply is recovered without waiting for anyone to
touch anything), the same hardening set as `carlos-stage.service` — and
no `ProtectSystem`, because it must write `/opt/carlos` and
`/etc/systemd/system`. A third unit, **`carlos-system-recover.service`**
(`Type=oneshot`, `Before=carlos-edge.service`, `WantedBy=multi-user.target`,
`TimeoutStartSec=10min`), runs once per boot and on the update unit's
failure.

**Both units exec through one line of shell:** when a journal is open and
`carlos.prev` exists, exec `carlos.prev system apply …`; otherwise exec
`carlos system apply …`. After the rename the binary at
`/opt/carlos/carlos` is the *candidate*, and the one thing known about a
candidate that needs recovering is that it did not work; the recovery
must not depend on it. (§11 names the residual: a `.prev` that cannot
run `system apply` either, which is a box that was already broken before
this roll.)

`enabledUnits` gains all three, and `systemctlPlan` does with them what
it does with everything: enable all, `start` only the `.path` — never a
service, which is why apply can call it from inside itself without
starting a second apply.

---

## 5. The verbs

All under `carlos system`, a new top-level dispatch beside `ops`. Every
verb runs in **API mode against the console** (`--console`,
`requireDeployAPISession`), because the console holds the bucket, the
signing key, and the egress; the workstation holds none of them.

**Authorization, stated once:** the endpoints
(`PUT /api/system/upload`, `POST /api/system/update`,
`GET /api/system/jobs/<id>`, `POST /api/system/rollback`,
`GET /api/system/status`) require a **person session** (`apiPerson`,
`apicli.go:298`) whose member is an **owner of the ops account**
(`OpsAccountID`, `store.go:1158`), checked server-side on every call.
Service credentials are refused, and so is every other account: a
deployment has one ops account, and rolling the root binary of every box
is that account's owners' act and no one else's. `requireDeployAPISession`
selects console mode; it grants nothing (`deploy.go:1151`), which is why
this paragraph exists.

**The update endpoint is a job**, not a request: it returns a job id, the
console does the fetch-verify-mirror-sign work in a goroutine, and the
CLI polls the job and prints each step as it completes. Fetching two
25 MiB assets from GitHub inside one HTTP request is how a promote times
out halfway through a mirror. Jobs live in console memory; a console
restart mid-job loses it, the CLI reports that, and re-running is safe
because every step is idempotent (mirror skips what is present and
equal; the pointer write is CAS). `--file` is a separate
`PUT /api/system/upload` (body limit 128 MiB, `--platform` required
because the console cannot run a foreign-arch binary to ask it) that
returns a handle the update job then consumes.

**`carlos system update [--channel stable] [--version vX.Y.Z] [--file path --platform linux-arm64 --version v] [--spread 5m] [--yes]`**

The fleet operator's one command. It is a promote for the platform:

1. **Resolve the version.** Default: `https://apt.carlosframework.com/latest.txt`
   (the `carlos update` discovery, `releaseTag`-shaped). `--version`
   pins one. `--file` skips step 2.
2. **Ingest the release, exactly as `carlos update` does.** The console
   fetches `SHA256SUMS.txt` and `.sig` from the tag-pinned
   `updateDownloadBase`, verifies the signature with the embedded
   `cliUpdatePubKey`, and fetches each asset the deployment needs, checking
   its sha against the sums. The asset set is the distinct platforms in
   every box heartbeat — `status/boxes/*.json` **and**
   `fleets/*/status/boxes/*.json` — via a new `BoxMetrics.Platform` field
   written alongside `Release`; a box that has not reported since the
   field shipped counts as `linux-arm64`. `selfReplace`'s fetch-and-verify
   is factored out into a function both callers share; it does not move.
3. **Mirror** each asset to `bootstrap/router/<version>/<asset>`. Present
   with the same sha → skip (a second promote of the same version to
   another channel copies nothing). Present with a different sha →
   **refuse the whole promote**: versions are immutable (§3).
4. **Write the pointer** for the channel, signed, CAS, `PromotedAt` now.
   `History` gets the previous `Current` whole; `Audit` gets a
   `Promotion{From, To, Actor, At}`.
5. **Bump the serial.** Boxes see it within two seconds and start their
   spread.
6. **Print what will happen:** which boxes are on this channel (from a new
   `BoxMetrics.SystemChannel`, also in the heartbeat — without it the
   console cannot know which boxes a promote reaches), running what, and
   the spread. Then return.

Without `--yes` it confirms once, showing the box list. The channel ladder
is advisory in v1: promoting to `stable` a version that was never on
`canary` prints a warning and proceeds, because a hotfix needs to.

**`carlos system router update …`** is the same verb scoped to one
component. With one component today it is an alias; it exists so the
command Paul named works and so a future split needs no new verb.
`carlos system update` with no component updates every component the
pointer schema knows.

**`carlos system status [--channel …]`**

Reads the pointers and every heartbeat under both prefixes and prints one
row per box: label, platform, channel, running `Release`, channel target,
and the last apply outcome (§6, published in the heartbeat as
`BoxMetrics.System`). This is the fleet-side half of the health gate: a
box whose heartbeat still shows the old `Release` ten minutes after a
promote is the row an operator is looking for. Heartbeats are
five-minutely, so this is also the honest answer to "is it done yet":
*not yet* for up to five minutes after the last box restarts. A `noted`
outcome (a no-op request, §6) is shown as such, not as an apply.

**`carlos system rollback [--channel …]`**

Pops `History[0]` into `Current` — whole releases, so the previous
artifacts and hashes come back with the version — re-signs with
`PromotedAt` now, bumps the serial. The outgoing release is not pushed
back, so a second rollback goes one further back rather than undoing the
first. Nothing GCs `bootstrap/router/` in v1, so the objects are there. Every box rolls back through the same path
it rolled forward on; the box-side `.prev` (§6) is for the box that
cannot come up, not for this.

**`carlos system apply [--recover]`** is the root oneshot, §6. It is not
for operators and says so if run interactively.

What is deliberately **not** a verb: `carlos system console update`. The
console is an app; `carlos deploy --app console` is its verb and already
works. Giving it a second name under `system` is the confusion the
dividing line exists to prevent.

---

## 6. `carlos system apply`: the journal, the dance, the gate, the way back

Runs as root from `carlos-system-update.service` (or, with `--recover`,
from `carlos-system-recover.service`). Every step is one the pinfra script
does today, plus the ones it lacks. Root state lives in
**`/opt/carlos/system/router/`** (0755, root-owned, never writable by
`carlos`):

- `lock` — an `flock` every apply takes first. The path-triggered apply
  takes it non-blocking and exits 0 with "another apply is running" if it
  cannot; `--recover` waits for it. Two processes never touch
  `/opt/carlos/carlos` at once, whatever an operator's hand-run
  `systemctl start` does.
- `staging/` — **0700**. The root-owned copy of the candidate and the two
  staged unit roots. Removed on every exit, success or not.
- `journal.json` — the open transaction, if any: `{phase, target,
  prev_build, units_added, units_removed, started_at}`. Written before
  each box-changing step, removed on completion.
- `seen.json` — `{<channel>: promoted_at}`: the newest pointer root has
  *verified* per channel. The replay guard, and nothing else.
- `applied.json` — `{version, channel, at}`: the last build that **passed
  a gate** on this box, forward or rollback. The `.prev` authority, and
  nothing else. One record, not per channel: it is a fact about the box.
- `result.json` — `{channel, version, promoted_at, outcome, reason, at}`
  of the last run, keyed by the **request's** tuple (two promotes inside
  one spread must not let a refusal of the first suppress the second).
  What the agent publishes and §4 step 4 reads.

Every file is written temp-then-`fsync`-then-`rename`-then-**`fsync` of
the directory**, so the rename itself is durable before the next step
depends on it; a leftover `*.tmp` at start is deleted unread.
`carlos.prev` is made the same way (`carlos.prev.tmp`, fsync, rename,
directory fsync), so a `.prev` is either whole or absent.

`/opt/carlos/carlos` is installed **`root:root 0755`** from here on, not
`carlos:carlos` as the script did. Every root oneshot on the box execs
that file; the sandboxed user has no reason to own it.

**Every run begins with recovery.** If `journal.json` exists, a previous
run was cut off (SIGKILL, OOM, the timeout, a reboot). Which way it goes
depends on one thing: whether the gate had passed.

- **Phase `gated`** (step 10 passed, steps 11–12 not finished): the box
  is on the new build and proved it. Recovery goes **forward**: finish
  steps 11–12. Undoing a roll that passed its gate because the process
  that recorded it was killed would be a rollback with no reason.
- **Any earlier phase**: recovery goes **back** — an apply that did not
  reach its gate is not one to trust, and `.prev` passed one. It is
  **files only**: stop the quiesce set of step 7 (an operator or the
  timer may have started a pass since); restore `carlos.prev` over
  `carlos` (keeping `.prev`); `disable` and delete `units_added`; restore
  `units_removed` and the rest of the previous unit set from
  `staging/prev/` or, if `staging/` is gone, from a fresh `carlos.prev
  bootstrap -root`; sysusers, tmpfiles, `daemon-reload`; `enable` (not
  start) the previous plan's units so they come up at boot. Then, only if
  `carlos-edge` has started at least once this boot
  (`ActiveEnterTimestampMonotonic != 0`): `reset-failed`, `restart
  --no-block carlos-edge`, and `start` the previous plan's timers and
  paths. At boot the edge is ordered after this unit and its own start
  *is* the restart, and the enabled timers and paths start with the
  target; a recovery that restarted and gated the edge from inside a
  unit the edge waits on would wait on itself forever. **Only now** write
  `result.json` `{outcome: "rolled-back", reason: "interrupted at
  <phase>"}` and remove the journal: a kill before this point leaves the
  journal in place and the next run recovers again, which is idempotent;
  a kill after it leaves nothing to do.

`--recover` ends here. A path-triggered run continues below if there is
also a request.

Then the run proper. Failures in steps 1–6 change nothing on the box and
are `refused` (or `deferred`, for a store error); from step 7 on, any
failure goes to step 13.

1. **Read the request.** One fixed path, `/data/system/router/requested.json`,
   reached by opening `/data`, then each component with
   `O_DIRECTORY|O_NOFOLLOW`, then the file with `O_NOFOLLOW`; must be a
   regular file under 4 KiB. It yields `{version, promoted_at, noop}` and
   nothing root will act on without checking. `version` must match
   `^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`, or the run is `refused` — this is
   what makes the cache path derivable and un-traversable.
2. **The channel is root's.** `CARLOS_SYSTEM_CHANNEL` from the unit's own
   `EnvironmentFile`, default `stable`. The request does not carry one and
   would not be believed if it did: a signed canary pointer authorizes
   canary boxes, not a stable box whose sandboxed user would like to be
   one.
3. **Fetch and verify the pointer from the bucket** with
   `CARLOS_RELEASE_PUBKEY`. A store error is `deferred`. The request's
   `version` and `promoted_at` must equal the pointer's `Current.Version`
   and `PromotedAt`, or `refused` (a request from a spread that started
   under an earlier promote). **Replay guard:** the pointer's `PromotedAt`
   must not be *older* than `seen.json[channel]`; equal is allowed, so an
   apply that was interrupted or refused can be retried for the same
   promote. Then advance `seen.json[channel]`. A box with no entry for the
   channel accepts the first pointer it sees — a fresh, hand-rolled or
   channel-moved box has nothing to protect yet (§11).
4. **A no-op request ends here:** `result.json` `{outcome: "noted"}`,
   exit 0. Root has seen the promote; nothing was downloaded, nothing
   changes.
5. **Copy the candidate into `staging/`.** The path is *derived*:
   `/data/system/router/<version>/carlos`, walked like step 1, regular
   file only. Copy to `staging/carlos` as **0600**; **verify the copy's
   sha** against the pointer's artifact for this platform; then `chmod
   0755` and run it: `staging/carlos version`, 30-second deadline, must
   print `carlos <version>`. The lie detector from `update.go:350`, on a
   file only root can reach, made executable only after it is known to
   be what the key signed. Nothing under `/data` is read again.
6. **Decide `.prev`, then stage both unit sets.** Run
   `/opt/carlos/carlos version` (30 s deadline). If it equals
   `applied.json.version`, or `applied.json` is absent and `.prev` is
   absent (a hand-rolled box: the running binary passed the script's own
   post-condition), the on-disk binary is known good: **journal
   `backing-up`** and copy it to `carlos.prev`. Otherwise — after a
   `rollback-failed`, when the disk holds a bad binary and `.prev` the
   good one — leave `.prev` alone. Then stage units from **both
   binaries**: `staging/carlos bootstrap -root staging/new -plan json`
   and `carlos.prev bootstrap -root staging/prev -plan json` (120 s
   deadline each; `-plan json` is new and prints `enabledUnits` and the
   inert list as JSON, so apply arms what *that* binary's bootstrap would
   arm). `units_added = new − prev`, `units_removed = prev − new`, both
   into the journal. Staging from `.prev`, not from the on-disk file,
   because after a `rollback-failed` those are different builds and a
   rollback must restore the units of the binary it restores.
7. **Journal `installing`**, then **quiesce**: `systemctl stop` the root
   timers and paths *and their services* — `carlos-adopt.timer`,
   `carlos-adopt.path`, `carlos-stage.path`, `carlos-restart.path`,
   `carlos-adopt.service`, `carlos-config-sync.service`,
   `carlos-stage.service`, `carlos-restart.service`. Stopping a timer does
   not stop the pass it already started, and a root pass running the new
   binary against the old edge and registry is the window this exists to
   close. Those passes are idempotent; killing one costs a retry. Then
   **install**: `rename` `staging/carlos` over `/opt/carlos/carlos`
   (same filesystem, atomic). **Journal `installed`.**
8. **Install units and configs**: `disable --now` and delete
   `units_removed` (a unit the new version dropped must not stay armed
   and exec a command the new binary no longer has); copy
   `staging/new/etc/systemd/system/*` into place; then, in this order and
   never the other, **sysusers** (`cp` + `systemd-sysusers`), then
   **tmpfiles** (`cp` + `systemd-tmpfiles --create`), then
   `daemon-reload`. The ordering is failure-mode row 6 of the
   console-as-an-app spec, and the reason is in the script's comment:
   tmpfiles silently drops a line whose group does not resolve, and then
   `carlos-edge` (`ReadWritePaths=/run/carlos`, no `-`) can never start
   after the next boot. Never full `carlos bootstrap`: it re-chowns
   `/data` to 0750, which breaks static-user tenants on the Tito box (the
   script's opening comment). And never copy the staging root's `data/` —
   it contains a 0-byte `registry.db` (`bootstrap.go:310`). **Journal
   `units`.**
9. **Restart the edge**: `systemctl reset-failed carlos-edge` first —
   always, because a unit that hit its start limit refuses `restart`
   with "start request repeated too quickly", and the rollback in step 13
   would be refused the same way — then `systemctl restart carlos-edge`.
   The one tenant-visible step; on a healthy box it is a one-second blip.
   **Journal `restarted`.**
10. **The gate**, 20 seconds to pass and then a **15-second soak**:
    - `carlos-edge` is `active`; across the soak `NRestarts` stays 0 (a
      manual restart resets it, so it counts only crash restarts since
      step 9) and `ActiveEnterTimestampMonotonic` does not move. A binary
      that starts, answers, and dies a second later is caught here, not
      recorded as a success.
    - The edge accepts a TCP connection on its listener and answers a
      request on loopback with *any* HTTP status. Not a tenant probe: a
      box with no route yet must still pass.
    - `/opt/carlos/carlos version` prints the target.
    **Journal `gated`.** From here recovery finishes forward.
11. **Arm**, now that the box is on the new build and the gate has said
    so: run `systemctlPlan(new plan's enabledUnits)` verbatim — enable
    all, `start` timers and paths — which re-arms what step 7 stopped and
    arms this spec's own path unit for the next roll. Not before the
    gate: a rolled-back box is armed once, on the previous plan.
12. **Record.** `applied.json` `{version, channel, at}`, `result.json`
    `{outcome: "applied"}`, remove the journal, remove `staging/`. Done.
    The agent deletes its `/data` cache when it sees the outcome.
13. **Roll back.** `rename` `carlos.prev` to `carlos` keeping a copy as
    `.prev`; `disable --now` and delete `units_added`; restore
    `staging/prev/etc/systemd/system/*` (which brings `units_removed`
    back); sysusers, tmpfiles, `daemon-reload`; `reset-failed`, restart
    the edge, run the gate; then `systemctlPlan(prev plan's
    enabledUnits)`. `result.json` `{outcome: "rolled-back", reason}`,
    `applied.json` unchanged (the `.prev` build is what passed a gate,
    and it is what `applied.json` already names), journal removed, exit
    non-zero so systemd's journal shows it. If the rollback fails *its*
    gate: `outcome: "rollback-failed"`, journal removed, exit non-zero —
    `applied.json` still names the `.prev` build, so the next run's step
    6 leaves `.prev` alone. At every exit the installed binary and the
    installed units came from the same build, *including* exits by kill,
    timeout or reboot, because the next run recovers first and
    `OnFailure=` makes sure there is a next run.

The agent, whichever binary it now is, reads `result.json` on its next
heartbeat and publishes it as **`BoxMetrics.System{Channel, Version,
Outcome, Reason, At}`**, which is what `carlos system status` shows. A
rolled-back box is therefore visible fleet-wide within five minutes, with
the reason, and (§4 step 4) will not try that pointer again.

---

## 7. The first roll, and compatibility

The first binary that contains this spec reaches a fleet by the old road:
`update-platform-binary.sh`, one last time per box. That run must **arm
`carlos-system-update.path` and enable `carlos-system-recover.service`**,
or the first promote does nothing anywhere: a staged bootstrap runs no
`systemctl` (`bootstrap.go:327`), and the path unit cannot arm itself
before it works. Two belts:

1. The script (pinfra, and Tito's copy in `tito/carlos-infrastructure`)
   gains the two lines, guarded on the files existing, the way
   `carlos-restart.path` was added.
2. The root hourly pass (`RunAdopt`, from `carlos-adopt.timer`) gains a
   **new, non-fatal first step**, placed before its registry and pubkey
   `log.Fatalf` gates: if the embedded system-update units are missing
   from `/etc/systemd/system`, or the path unit is inactive, or the
   recover unit is not enabled, install and arm them via `systemctlPlan`.
   New code — `RunAdopt`'s existing `capabilityPass` converges tenant
   `carlos-app@<host>.service.d` drop-ins through `caps.Converge` and
   installs no platform unit. So a box the script forgot is armed within
   the hour, and the first promote is not lost: the leg re-touches its
   marker every minute until root answers (§4), and `PathChanged` sees
   the next touch after arming.

From then on the script is never needed, because that binary contains the
leg, the units and the apply. There is no flag day and no dual-running: an
old agent does not read `control/system/`, so writing a pointer into a
deployment whose boxes are old does nothing until they are rolled by hand
once.

Within one deployment, canary and stable boxes run different binaries by
design, for as long as the operator leaves them that way. Nothing in the
bucket schema they share changes with this spec (the pointer is a new key;
`status/boxes` gains three optional fields), so a two-version fleet is
exactly as safe as it is today after a partial script run.

`carlos update` (the workstation CLI) is untouched except for the factored
fetch-and-verify. Its header sentence — *a self-updating box binary would
fight that machinery* — becomes true in the other direction: the box
machinery is now the sanctioned one, and `runUpdate`'s refusal to run on a
box stays.

---

## 8. Third-party deployments

Tito's operator, on a laptop with a Tito console session, as an owner of
Tito's ops account:

```
carlos system update --console https://console.tito.example --channel canary
carlos system status  --console https://console.tito.example
carlos system update --console https://console.tito.example --channel stable
```

The Tito console fetches the public release from GitHub, verifies the
vendor's signature, mirrors it into Tito's bucket, signs the pointer with
Tito's key. Tito's boxes, behind Tito's store door, fetch from Tito's
bucket. At no point does a CARLOS developer hold anything of Tito's, and
at no point does a Tito box reach anything but Tito's console. What the
vendor ships is a signed public release and nothing else, which is what
#100 asked for.

Egress: the **console** needs to reach `apt.carlosframework.com` and
`github.com`. Boxes need nothing new. A deployment whose console cannot
reach GitHub uses `--file`.

---

## 9. What this does NOT do

- **Split the process.** `router` is the one binary. Updating the edge
  without restarting the activator is not possible until they are separate
  units, and that is not this spec.
- **Order boxes within a channel** beyond the spread jitter. Two channels
  are the ordering primitive in v1.
- **Auto-promote canary to stable**, or bake-time enforcement. The
  ladder is advisory.
- **GC `bootstrap/router/`.** Old versions stay; they are ~25 MiB each and
  the rollback verb needs them. A later slice can keep `History`'s depth.
- **A console page.** `carlos system status` is the read surface in v1;
  the fleet page's box rows already show `Release`, and a target column
  there is a small follow-up.
- **Scoped-grant signing** for platform pointers.
- **Update the console** — it is an app.
- **Stream the store door's GET.** It holds the object in memory today
  for `bootstrap/carlos`; it will for `bootstrap/router/…`. Named in §11.
- **Durable update jobs.** A console restart mid-promote loses the job;
  re-running is safe.
- **Any change to `carlos bootstrap` for a fresh box** beyond `-plan
  json` and the three new units in `enabledUnits`. New boxes still
  bootstrap and get their first binary by hand (`infra/remote-host.md`);
  from the first heartbeat on they are on `stable` and roll with the
  fleet.

---

## 10. What this retires

- `update-platform-binary.sh` and the per-deployment roll runbooks, after
  each deployment has been hand-rolled once onto a binary that carries
  this spec. The script's sysusers/tmpfiles guard for *pre-#104* binaries
  is not carried: nothing that old is rolled through this path.
- The exact `bootstrap/carlos` grant in `aclRead`, and the key itself, one
  release after the script is gone. Until then it is harmless.
- The vendor needing SSM on a third-party deployment for a routine roll.
- The "blocked on deploy scope" state this spec was born from: a merged
  `main` becomes a release, and a release becomes a fleet with one
  command per channel.

---

## 11. Proof, and live checks owed

**Unit-level, in this repo:**

- Pointer sign/verify round trip; scoped grant refused; wrong component
  or channel refused; rollback pointer carries a new `PromotedAt`. The
  door: both new shapes granted, both traversal siblings refused,
  `bootstrap/carlos.sig` still refused, listing under either prefix still
  refused, PUT under either still refused. `Store.Open` on all four
  implementations, with the limit+1 read telling exact-limit from over.
- The leg against a `DirStore`: no pointer → no request; pointer for another
  platform → no request; version equal, pointer not newer than `seen` →
  nothing; version equal, pointer newer → no-op request and no fetch;
  pointer older than `seen` → nothing; sha mismatch → no request and file
  deleted; oversized object → cut at the limit, no request; a
  `result.json` naming this tuple as any terminal outcome → no request
  and cache deleted; `deferred` → retried; a good pointer →
  `requested.json` with no channel/key/path and the marker; the marker
  re-touched every minute until a result names the tuple; the spread →
  `tick` fires after the deadline and not before, with no store read in
  between.
- `carlos system apply` against a fake root and a fake `systemctl` that
  records every call in order: `lock` held → path-triggered run exits 0
  untouched, `--recover` waits; request read refuses a symlinked parent,
  a symlinked file, an oversized file, a version that fails the pattern;
  channel taken from the environment, not the request (a request under a
  canary pointer on a stable box → `refused`); a stale tuple → `refused`
  recorded against the *request's* tuple; replay: older → `refused`,
  equal → proceeds, `seen` advanced only after verification; no-op →
  `noted`, no `staging/`; the candidate copied 0600, sha-checked, chmod
  after; a swap of the cache file after the copy → no effect; `version`
  over its deadline → `refused`, `staging/` removed; `.prev` refreshed
  when disk `version` equals `applied.json`, on a fresh box, and **not**
  after `rollback-failed`; `staging/prev` from `.prev`, not from disk;
  `units_added`/`units_removed` computed and journaled; quiesce stops
  the services too, before the rename; `units_removed` disabled and
  deleted forward; sysusers before tmpfiles by *call order*;
  `reset-failed` before every restart; **no `start` of any `.service` in
  any run** (the self-start deadlock); arming after the gate, forward and
  back, from the right plan; a gate that passes and then fails in the
  soak → rolled back; every journal phase before `gated` → recovery rolls
  back files only, restores `units_removed`, restarts the edge only when
  it has started this boot, and removes the journal last; phase `gated`
  → recovery finishes forward and `applied.json` names the candidate; the exec shim picks `.prev` when a journal is open;
  `rollback-failed` leaves `applied.json` and `.prev` alone; every write
  is temp-fsync-rename and a leftover `.tmp` is discarded.
- `carlos system update` against a stub release host: non-owner or service
  credential → 403 before any fetch; bad sums signature refused; asset sha
  mismatch refused; mirror skip when present and equal; mirror refused
  when present and different; pointer CAS conflict retried; serial bumped
  last; upload without `--platform` refused; job polled to completion;
  job lost on restart reported, re-run idempotent.

**Live, on the dev deployment first, then flagship canary:**

1. Hand-roll the dev box with the script onto a build carrying this spec,
   and confirm afterwards: `carlos-system-update.path` active,
   `carlos-system-recover.service` enabled. That is the last script run
   for that box.
2. `carlos system update --file <next build> --version <sha> --channel
   canary` with the dev box on canary. Watch `status` show the new
   `Release`. Re-promote the same version: `status` shows `noted`, no
   download in the edge log.
3. Promote a deliberately broken binary (one that exits at start) the same
   way. Confirm the edge went start-limit-hit and came back on `.prev`
   regardless, `status` shows `rolled-back` with the reason, and a second
   serial bump does not re-attempt it.
4. Promote a binary that starts and exits after five seconds. Confirm the
   soak catches it.
5. Kill the oneshot between `installed` and `restarted` (a `sleep` build
   with a `kill -9`). Confirm `OnFailure=` recovers to `.prev` without a
   touch. Then reboot the box in the same state and confirm the boot unit
   does, and that the edge starts after it without a hang.
6. Move the box from canary to stable in `host.env`. Confirm a stable
   pointer older than the canary one is applied, not refused.
7. `carlos system rollback --channel canary`. Confirm a forward roll to the
   previous version through the normal path, artifacts and all.
8. On flagship: the console box on canary; promote a real tag; then stable
   with the default spread; watch the tenant hostnames answer 200
   throughout (the plan doc's rule at `console-as-an-app.md:1910`, now
   observed rather than performed).
9. Tito: Tito's operator runs §8 against a Tito console; the CARLOS side
   watches nothing and touches nothing.

**Known gaps to name so they are not discovered:**

- The gate's loopback probe proves the process serves HTTP, not that a
  tenant's route works. A binary whose routing is broken but whose
  listener is fine passes the gate; `status` and the tenant's own
  monitoring are the backstop, as they are today.
- The replay guard has no baseline on a box that has never seen a pointer
  for its channel. A bucket writer without the key can put an old signed
  pointer in front of such a box once. That is the exposure app channel
  pointers already have on every box, and the same actor could point
  `control/serial` at nothing at all; the guard narrows it, it does not
  close it.
- Recovery depends on `carlos.prev` being able to run `system apply
  --recover`. `.prev` passed a gate on this box, so the case is a `.prev`
  that has since been damaged on disk — a box that was broken before this
  roll, and is now a hand job.
- The store door's GET holds the artifact in memory inside the console
  box's edge (`MemoryHigh=70%`). Nine fleet boxes fetching 25 MiB inside
  the same spread is not a problem; ninety would be, and streaming the
  door's GET is the fix when that day comes.
- `Spread` jitters starts, it does not serialize them; two boxes can still
  restart in the same second. With nine boxes and a five-minute spread the
  odds are small and the cost is two one-second blips.
- A box that is asleep or dark during the promote rolls when it next
  ticks. `status` shows it on the old version until then, which is the
  truth.
- A permanently failing recovery re-runs every minute (the agent re-touches
  the marker while its request is unanswered) and never writes
  `result.json`, so the fleet sees a stale result; that is the needs-a-human
  state, and it is fleet-invisible.

---

## 12. Approaches not taken

**Boxes fetch from the public release host directly.** No mirroring, no
console egress. But every box then needs egress to GitHub, which a fleet
box behind the store door may not have — it excludes exactly the
deployment shape #100 exists for, and leaves the door's `bootstrap/` grant
unused.

**The platform as a reserved app (`ops/platform`).** `carlos promote
--app platform` would just work and the console would show it. It is the
shape the console-as-an-app spec rejected: the substrate has no route, no
upstream, no instance, and every app assumption — health probe,
hibernation, the adoption pass moving a `live/<host>` symlink — has to be
special-cased for the one "app" that is not one. Cheap on day one and
expensive on every day after.

---

## 13. Review findings, and where each landed

Two rounds, two independent readers each, checking claims against the
code. The table is so a reader can see what the drafts got wrong and how,
rather than only that the text is now different.

**Round one** (both readers unless noted):

| Finding | Landed in |
|---|---|
| Root hashed and exec'd a file the sandboxed user owns — TOCTOU to root | §2, §6 steps 1, 5 |
| No authorization rule; `requireDeployAPISession` grants nothing | §5 |
| Door-backed boxes cannot read `control/system/` — silent never-update on the Tito shape | §3 |
| Interrupted apply strands the box; on-disk `Build` equality then says "done" forever | §6 journal, recovery-first, boot unit, `OnFailure=` |
| Rollback restored the binary but not the unit set | §6 steps 6, 8, 13 |
| Edge start limit (default 5/10 s) refuses the rollback's restart | §1, §6 step 9 |
| Gate had a deadline but no observation window | §6 step 10 |
| `History` of version strings cannot reconstruct artifacts for rollback | §3 `SystemRelease` |
| Root units exec the new binary against the old edge during the swap | §6 steps 7, 11 |
| Generic "arm every timer" contradicts `enabledUnits` | §6 `-plan json`, `systemctlPlan` |
| First-roll arming was circular | §7 |
| `.prev` overwritten by a later run after `rollback-failed` | §6 step 6 |
| Replay watermark missed no-op promotes; blocked canary→stable moves | §4 step 3, §6 `seen.json` |
| Spread had nothing to run it between hourly passes | §4 `observe`/`tick` |
| No exec deadlines; oneshot could hang before the gate | §4 units, §6 steps 5–6 |
| 32 MiB cap is on PUT, not GET | §3 `Store.Open`, §4 step 6 |
| Versions not immutable (Codex) | §3, §5 step 3 |
| Console cannot know channel membership or fleet-prefixed heartbeats (Codex) | §5 |
| Suppression covered only `rolled-back` (Fable) | §4 step 4 |
| Store error during the console box's own roll read as a refusal (Fable) | §4 step 8, §6 `deferred` |
| Binary owned `carlos:carlos` for no reason (Fable) | §6 |

**Round two**, on the fixes above:

| Finding | Landed in |
|---|---|
| `enable --now` on the plan starts the update oneshot from inside itself, and the recover unit mid-apply (both) | §4, §6 step 11: `systemctlPlan` verbatim, never starts a service; `lock` |
| Boot recovery ordered before the edge restarts and gates the edge, waiting on itself (both) | §6 recovery: files only, restart only if the edge has started this boot |
| One watermark doing replay guard *and* `.prev` authority breaks both (both) | §6 `seen.json` vs `applied.json` |
| Recovery execs the candidate after the rename (both) | §4 exec shim prefers `.prev` under an open journal |
| Stopping timers does not stop running passes; re-arm before restart reopened the window (both) | §6 steps 7, 11 |
| `enable --now` would start a start-limited edge before `reset-failed` (Codex) | §6 step 11: no service is ever started by the plan |
| `staging/prev` built from the on-disk binary, which after `rollback-failed` is not `.prev` (Codex) | §6 step 6 |
| Crash durability of journal/`.prev` unspecified (Codex) | §6 temp-fsync-rename |
| Request could name another channel's pointer (Codex) | §6 step 2 |
| Request `path` still a root input; `O_NOFOLLOW` covers the last component only; copy before check exposed secrets in a world-readable dir (Fable) | §6 steps 1, 5: derived path, per-component walk, 0700 staging, 0600 until verified |
| `LimitReader` cannot bound a `[]byte` `Get` (both) | §3 `Store.Open` |
| Recover unit had no timeout (both) | §4 |
| Inactive oneshots read as broken by the hourly belt and §11 (Codex) | §7, §11: path active, recover enabled |
| Nothing re-touched the marker; `carlos adopt` does not converge platform units (both) | §4 `tick`, §7 belt 2 |
| Rollback pointer's `PromotedAt` refused as a replay (both) | §3, §5 |
| Forward install left removed units armed (Codex) | §6 steps 6, 8 |
| No-op promote downloaded 25 MiB on every box (Fable) | §4 step 3, §6 step 4 |
| `applied.json` shape undecided; canary→stable move skipped the `.prev` refresh (Fable) | §6: one record, compared to disk `version` |
| `refused` recorded against the wrong tuple (Fable) | §6 `result.json` |
| Update job lifecycle undecided (Fable) | §5, §9 |

**Round three** (Codex closure check; 16 of 20 closed, four residuals):

| Finding | Landed in |
|---|---|
| Interruption between `applied.json` and journal removal undid a roll that passed its gate | §6 step 10 `gated`; recovery goes forward from it |
| Recovery removed its journal before issuing the restart; a kill in the gap left the candidate running with no journal | §6 recovery: result and journal removal last |
| Recovery did not quiesce passes or restore the previous plan's armed state | §6 recovery: stops the quiesce set, `enable` at boot, `start` at runtime |
| Directory fsync unspecified after renames | §6 durability |