# 🤖 CARLOS platform

> 🤖 This document was written by an LLM (Claude), on the ideas,
> instruction, and editing of humans, per
> [factor X](https://11factor.org/#x).

A black-box deployment platform for CARLOS apps: `carlos ship` publishes an
immutable, content-addressed release; the platform places it on instances,
routes to it through a shared edge, and surrounds it with platform-level
glue. One deployment is **one object-store bucket + N hosts running the
`carlos` binary + DNS** — the whole self-hosting story in one sentence. See
the `2026-08-01-carlos-platform-design.md` design spec (private `platform-infrastructure` repo)
for the full design.

## Build & test

    make build   # CGO_ENABLED=0 build of the carlos binary
    make test    # go vet, gofmt -l, go test ./...

One static binary plays every role, selected by subcommand: `carlos edge`,
`carlos agent` (edge + activator + release adoption + fleet status, one
process per host), `carlos ship`/`promote`/`rollback`, `carlos vet`. Run
`carlos` with no arguments for the full command list.

### Platforms

Linux and macOS run every role — box and workstation alike — and install
from apt (`sudo apt install carlos`) or Homebrew
(`brew install carlosframework/tap/carlos`).

**Windows is a client-only target.** A Windows workstation can `carlos
login`, `carlos ship`, `carlos deploy`, `carlos logs` — everything that
talks to a console or a bucket over the network — against a Linux fleet.
It cannot BE a box: `carlos agent` and the root-side passes need unix
sockets, `flock` and systemd, and the packages that provide them compile
on Windows (the CLI is the same binary) but refuse at their per-OS
boundary rather than pretending. See `internal/edge/purge_listen_windows.go`
and `internal/host/{statfs,flock}_windows.go` for what that refusal means
in each case.

Two consequences worth knowing before you install it:

- **`carlos update` does not self-replace on Windows.** It still checks the
  release feed and tells you when you are behind, then hands you the URL of
  the zip to unpack over your existing binary. Windows cannot rename a new
  binary over a running `.exe`, and a self-replace nobody here can test is
  worse than an honest manual step.
- **Your credentials file is not permission-protected on Windows.** On Linux
  and macOS `~/.carlos/credentials` is written `0600`; the equivalent mode
  on `%USERPROFILE%\.carlos\credentials` means nothing, so any other user
  of that machine can read your session. Do not `carlos login` on a shared
  Windows machine.

Install by downloading `carlos-windows-amd64.zip` (or `-arm64`) from
[carlosframework/releases](https://github.com/carlosframework/releases/releases),
unpacking `carlos.exe` somewhere on your `PATH`, and verifying its checksum
against the release's signed `SHA256SUMS.txt`.

## Accounts

An **account** is the tenancy primitive: a project or team that owns apps and
has members. Kass, Keymail and Woodstar each get their own rather than sharing
one flat list, and a self-hosted single-app deployment can give team A and
team B separate accounts that deploy independently.

An account is `{id, sqid, name}`. The **id** is the next free integer,
allocated at creation by a create-only write of `accounts/<sqid>/account.json`
in the deployment bucket; the registry only ever *materializes* what the
bucket says, never assigns. That is what makes account identity survive a
registry rebuild and be identical on every box: minted once, recorded in the
bucket, never reassigned. Sequential ids keep sqids short enough to live in a
hostname you can read out (`kass.bac.oncarlos.com`); they are not secret, and
never were — every platform hostname's certificate is published in
Certificate Transparency. The **sqid** is the frozen encoder over that id and
is the account's public identity, in every platform domain and every bucket
key. The **name** is free text and deliberately not unique: two accounts may
both be called "Blog", because identity is the sqid and there is nothing to
squat.

`ops` is the built-in operator account — id 1, sqid `bab` — created by
migration on every deployment. CLI-added apps live there by default, so a
console-less self-hoster is unaffected by any of this.

**Roles.** Exactly two:

| Role | Can |
|---|---|
| **owner** | everything a member can, plus invite, remove members, rename |
| **member** | deploy, promote, rollback, create/retire canaries, view everything |

Every account has at least one owner; the last one can be neither removed nor
demoted (transfer first).

**In the console.** The account switcher sits top left on every account page,
where the wordmark used to be: a dropdown whose trigger names the account you
are in, listing the accounts you are a member of and linking to the
**New account** screen at `/accounts/new`. It is a `<details>` disclosure, so
it is keyboard-operable and works with scripting off.
URLs are explicit and shareable — `/a/<sqid>/apps/<app>` — so a link a
teammate opens lands where you meant; `/` redirects to the account you were
last in (or your first membership, or one minted on the spot). A signed-in
non-member asking for an account URL gets the **same 404 a nonexistent page
gets**: account existence is not probeable.

**Invites** go into the addressee's Keymail Apps inbox — the platform never
emails anyone — as a single-use link that expires in 7 days and is revocable.
Clicking it while signed in as that address creates the membership.

**From the CLI.** Every app verb takes `--account`, a sqid or a name,
defaulting to `ops`:

    carlos auth login --console https://platform.carlosframework.com
    carlos accounts create --name kass        # you own it
    carlos accounts create --name kass --owner paul@keymail.dev   # headless
    carlos accounts list                     # sqid, name, app count
    carlos apps create --account kass --app blog   # claim the name (rate limited;
                                             # the console's New app form is not)
    carlos ship --account kass --app blog ...
    carlos add --account kass --app blog blog.example.com

**Colour.** The CLI colours its output when — and only when — it is writing
to a terminal: a write that happened reads green, a dry run or a bypassed
rail yellow, a refusal red, and context you did not ask for (the `using
<console> as <you>` banner, "nothing to do" verdicts, follow-up hints) is
dimmed. Redirect it, pipe it, or run it in CI and every byte is what it
always was, so parsing scripts and test assertions are unaffected. Override
with `CARLOS_COLOR=always|never`; `NO_COLOR=1` and `TERM=dumb` are honoured.

**Who the CLI is.** `carlos auth login` gives the command line an identity,
and that identity owns the accounts it creates. It authenticates against the
**deployment's own console** — already a registered Keymail OAuth client and
already the authority on who belongs to what — rather than minting a second
OAuth client, so there is one consent screen, one place to revoke, and a
self-hoster's CLI trusts their own box.

It prints a code rather than opening a browser or listening on a local port,
because the machine that needs the credential is usually an SSH session where
neither works. You open `/auth/cli` in whatever browser you are already signed
in on and **type the code** the terminal printed.

Typing it is the point, not a nicety: a link that arrives one click from an
Approve button is the device-code phishing attack (the one run at scale against
Microsoft tenants) — an attacker starts the flow, sends you their link, and one
click hands them a credential that acts as you. So the console never approves
from a prefilled code, and the CLI never prints the ready-made link. When a
token is issued, a note goes to your Keymail inbox naming the terminal, where
it connected from, and how to revoke it.

The token lands in `~/.carlos/credentials` (mode 0600, one entry per console
origin), expires in 90 days, and is listed with a revoke button at
`/auth/cli/tokens`.

    carlos auth whoami                       # who this terminal is, per the console
    carlos auth logout                       # revokes the token, then forgets it

**Shipping from a logged-in laptop.** Once you're logged in, `ship`,
`promote`, `rollback`, `channels`, and `env`/`secrets` (issue #43) speak
through the console instead of the bucket directly — no
`CARLOS_DEPLOYMENT_BUCKET`/`_DIR` on a developer's machine at all:

    carlos ship --app blog --version $(git rev-parse --short HEAD) --target linux-arm64 blog-linux-arm64
    carlos promote --app blog abc1234 canary/my-branch
    carlos channels --app blog

Canary and edge promote exactly as before; **promoting onto `stable` is
refused** — the console names itself in the refusal, because moving `stable`
needs a passkey step-up only a browser can do. Finish it at the URL the
refusal prints.

**CI uses the console too.** Mint a deploy credential for the app, limited
to the rungs CI may move, and give CI the secret and the console's address:

    carlos services create --role deploy --app blog --rungs canary,edge ci
    # in CI:
    export CARLOS_SERVICE_TOKEN=... CARLOS_CONSOLE=https://console.example.com
    carlos deploy --app blog --version "$GITHUB_SHA" --channel edge blog-linux-arm64

`carlos deploy` ships, promotes and watches the app's hosts, the same as
from a laptop. The first deploy of a new app still needs a person: a deploy
credential cannot create the app or set up its first host. Every promote is
audited in the console and reaches the boxes within seconds. A deploy
credential can ship, and promote or roll back only its rungs. It cannot
reach stable, hotfix, delete the app, or change retention or domains.

**Direct bucket mode is for operators.** With `CARLOS_DEPLOYMENT_BUCKET`
(or `_DIR`) set, every verb goes straight to the bucket and the console
never sees it. Use it to set up a deployment before its console exists, to
run one that has no console, to recover when the console is down, and for
the platform's own releases. A scoped `CARLOS_RELEASE_KEY` limits that path
only. The CLI refuses to run with both `CARLOS_SERVICE_TOKEN` and a bucket
variable set.

**On a box, or in CI, don't log in.** There is no browser and no human, and a
long-lived token sitting on a box is a standing credential nobody is watching.
Name the owner outright instead — it needs no identity of its own and takes
precedence over any logged-in one:

    carlos accounts create --name kass --owner paul@keymail.dev

Either way an owner is named. `carlos accounts create` **refuses** with no
login and no `--owner`: an account with no owner materializes into the console
with no members, which means nobody can open it, invite to it, or see the apps
migrated into it.

An exact sqid wins; otherwise a unique name match; an ambiguous name lists the
candidates and refuses rather than guessing which tenant you meant.

`--owner` is required and is recorded in the account manifest, which the console
materializes into the account's first owner membership. Membership *is*
authorization here, so an account created without one exists, holds its apps,
and cannot be opened by anybody — a state the flagship reached for real during
the multi-account migration.

**Bucket layout, and the migration.** App objects live under
`apps/<account-sqid>/<app>/…`, and that key path *is* the app→account linkage
— there is no separate manifest. **Every read path resolves those qualified
keys and nothing else** (the legacy read-fallback was retired on 2026-08-03,
once both deployments were migrated). Moving an app is a verb:

    carlos accounts migrate --app kass --to <sqid> --dry-run   # prints the plan
    carlos accounts migrate --app kass --to <sqid>
    console migrate-app --app kass --to <sqid>                # on the console's box

It copies the objects and re-stamps the app's routes. The console keeps its
own app→account row — console state is the console's authority, and the
carlos binary deliberately never opens its database — so `console
migrate-app` completes the move; `carlos accounts migrate` prints that exact
command when it finishes. Both halves are idempotent.

**Upgrading from a pre-accounts binary?** Your objects are all at the old
account-less `apps/<app>/…` keys, which nothing reads any more — so the apps
will go quiet rather than serve something stale, and every miss tells you so
by name:

    carlos accounts migrate --app <app> --from legacy --to bab   # --to <sqid> for a tenant

That `--from legacy` source is permanent. However long a deployment sits on an
old build, this is the way off those keys.

**It never deletes a legacy object.** Deleting them is a separate operator
step, and an unhurried one now that nothing resolves through them.

## Releasing the platform

A release is a `vX.Y.Z` tag on this repo plus signed binaries on [carlosframework/releases](https://github.com/carlosframework/releases). Everything downstream reads that one set of files: `carlos update`, the Homebrew tap, apt, and every deployment's `carlos system update`. Cut it from a green `main` and verify it from the public side, never from your build directory.

1. **Tag.** `git tag -a vX.Y.Z <sha> -m "carlos vX.Y.Z"` on the `main` commit you are releasing, then `git push origin vX.Y.Z` to the forge.
2. **Build.** Six targets, darwin, linux and windows on amd64 and arm64, each with `make build GOOS=<os> GOARCH=<arch> SHA=vX.Y.Z` from a clean checkout of the tag. Build with `TMPDIR` and `GOCACHE` on a real disk: `/tmp` is a shared tmpfs, and a full one fails the link with "no space left on device". Zip each Windows binary as a lone `carlos.exe`.
3. **Debs.** One per architecture, built with `dpkg-deb --root-owner-group`, holding `usr/bin/carlos`. Copy the last release's `control` and change the version and the description's last sentence. Wrap the description without breaking on hyphens, or apt shows "step- by-step".
4. **Sums and signature.** Write `SHA256SUMS.txt` over the eight files. Sign it with the ed25519 seed in SSM `/carlos/cli-update-key`, passed on stdin and never in argv, and write the signature base64-encoded to `SHA256SUMS.txt.sig`. Before signing, check that the key's public half equals `cliUpdatePubKey` in `update.go`. A mismatch publishes a release nobody can update to.
5. **Release notes.** Draft them, and put them through copy review before anything is published. Lead with what changed for people using the CLI. Add a short section for operators, taken from the release notes below this recipe, about anything to check before rolling.
6. **Publish.** `gh release create vX.Y.Z -R carlosframework/releases --latest --title vX.Y.Z --notes-file NOTES.md` with all ten files. Confirm it is the latest with `gh api repos/carlosframework/releases/releases/latest -q .tag_name`.
7. **Homebrew.** In `carlosframework/homebrew-tap`, bump `Formula/carlos.rb`: the version, the four url and sha256 pairs from the published `SHA256SUMS.txt`, and the `carlos vX.Y.Z` test string. Open a PR, check each hash against the sums file, and squash-merge it.
8. **apt.** Download the live tree, `apps/bab/apt-carlosframework/releases/<last>/site.tar.gz` from the flagship bucket, and unpack it. Add the two debs under `pool/main/c/carlos/`, and regenerate each architecture's `Packages` with `dpkg-scanpackages --multiversion`. Delete `Release`, `InRelease` and `Release.gpg`, then rebuild `Release` with `apt-ftparchive release`, passing Origin and Label `CARLOS`, Suite and Codename `stable`, the architectures and the component. Write it outside the tree first, or it lists itself. Sign `InRelease` and `Release.gpg` with the SSM key `/carlos/apt-signing-key` in a throwaway `GNUPGHOME`, set `latest.txt` to the tag, and check both signatures with `gpgv` against the tree's own keyring.
9. **Ship apt.** Ship the directory and let `carlos ship` build the tarball. A hand-made tar with a leading `./` entry took apt down once. With `AWS_PROFILE=keymail` and `CARLOS_DEPLOYMENT_BUCKET` set to the flagship bucket, run `carlos ship -app apt-carlosframework -account-id bab -kind static -version vX.Y.Z-1 site/`, then `carlos promote -app apt-carlosframework -account-id bab vX.Y.Z-1 canary/apt` with `CARLOS_RELEASE_KEY` from SSM `/carlos/release-key` set for that one command. The channel is the literal `canary/apt`.
10. **Changelog.** In a full checkout of `main`, with `gh` and `amadan` signed in, run `git fetch origin --tags '+refs/amadan/closed/*:refs/amadan/closed/*'`, then `go run ./cmd/changelog`. Map each of the new release's bullets to the branches behind it in `docs/site/changelog-links.json`, keyed by the bullet's bold lead, and run it again. Land `docs/site/changelog.json` and the links file on a branch. Then, in `carlosframework/website`, run `node hack/sync-docs.mjs <checkout>` and ship the site the way its AGENTS.md describes. The page is carlosframework.com/platform/changelog.

Then verify from the public side:

```sh
v=vX.Y.Z; rel=https://github.com/carlosframework/releases/releases/download/$v
curl -sSLO $rel/carlos-linux-arm64; curl -sSLO $rel/SHA256SUMS.txt; curl -sSLO $rel/SHA256SUMS.txt.sig
printf '\x30\x2a\x30\x05\x06\x03\x2b\x65\x70\x03\x21\x00' > pub.der
echo 'R+EO3qQ47VsuKl9Qq/pkDXBucPYvzp5lU2ut4zpnJPc=' | base64 -d >> pub.der
base64 -d SHA256SUMS.txt.sig > sig.bin
openssl pkeyutl -verify -pubin -inkey pub.der -keyform DER -rawin -in SHA256SUMS.txt -sigfile sig.bin
sha256sum -c --ignore-missing SHA256SUMS.txt
curl -sS https://apt.carlosframework.com/latest.txt          # prints $v
carlos system update --console <console> --version $v         # then: carlos system status
```

Last, roll it, and roll dev before flagship. The order is fixed: dev boxes, then the dev console, then check dev end to end, then flagship boxes, then the flagship console. Do not start on flagship until dev has been exercised on the new build.

`carlos system update --console <console> --version vX.Y.Z`, then `carlos system status`. Every box should read `current` within the spread. A box that reads `rolled-back` or `refused` says why on the same line; fix that box, and do not re-cut the release.

**Checking dev.** The check is `wakecheck`, a standing hibernating app on dev. Each request writes one SQLite row and replies with its build, the box's host and the row count (`hits=`). It has no region, so both dev boxes serve it: `carlos-dev-ie`, which is the console box, and `carlos-dev-ie-b`. Once dev is rolled:

1. Confirm it is asleep. `carlos logs --app wakecheck --console https://platform.dev.carlosframework.com --since 30m` should end in "instance exited". Do not use `instances list --health` for this, because it always reads Asleep (#287). Idle is 15 minutes and bots can keep it awake, so wait it out.
2. Wake it on `carlos-dev-ie` through DNS: `curl -sS https://wakecheck.dev.carlosframework.com/`. A pass is a 200 whose `host=` is that box, in about 0.6 s to first byte.
3. Wake it on `carlos-dev-ie-b` by pinning its address: `curl -sS --resolve wakecheck.dev.carlosframework.com:443:18.200.150.210 https://wakecheck.dev.carlosframework.com/`. A pass is a 200 whose `host=` is that box. It takes about 1.6 s, because it restores from S3.
4. `carlos status --layout` shows `wakecheck` as `nested` on both boxes.

On dev, `hits=` going backwards from one box to the other is not a failure. With the claim off, a wake that switches box can serve a stale local DB. That is the known hazard in #286, and it stays until the claim ruling lands. Judge the check on the 200s and the `host=` values, not on the count.

Three things catch people out on the roll:

- **`system update` rolls a whole channel.** It rolls every box on the channel within the spread. It cannot canary one box. To put a release on one box first, roll that box by hand with `scripts/update-platform-binary.sh` in platform-infrastructure.
- **Old updaters need `carlos-instance-control` restarted.** A box whose updater predates v0.26.0 does not restart `carlos-instance-control` when it rolls. The moment the new edge serves, restart it by hand. Until then the new agent refuses isolated wakes as "running an older build". Use a one-off watcher to do the restart, not memory.
- **The dev console needs its release key first.** Before dev can be rolled with `system update`, its console must hold the release key. That is `/etc/carlos/release.env` (root:root, 0600) and a `console.service.d/release-key.conf` drop-in that loads it.

- Refresh the embedded timezone tables when a new IANA tzdata release is out:
  `scripts/update-tzgeo.sh <version>`, then `go test ./internal/geo`.

A release-specific note goes here when a release changes something an
operator needs to check or act on before rolling it. Read the note for
the release you are about to roll.

**Binary size (#275).** The linux/arm64 platform binary doubled to 77.7 MB in earlier builds because SDK clients were kept whole by the linker. This release is 45.7 MB again, so boxes download about 32 MB less. CI now fails a build over 52 MB.

**Pointer digests (#253): writers before readers.** Once a box has verified a channel pointer that carries manifest digests, it refuses pointers for that channel without them. So digests are turned on in order. First upgrade every writer: every console, and every CLI or deploy script that promotes, rolls back or fast-tracks. Then run `carlos channels digests on`. Then watch `carlos channels digests`: a channel marked as written by an old writer means one is still in use. Rolling a writer back below this release stops it writing pointers boxes accept; `carlos channels digests off` suspends digests while it is fixed, but does not clear the boxes that have already seen them. When `carlos channels digests` lists nothing, set `CARLOS_REQUIRE_POINTER_DIGEST=on` in every box's host.env, so a new box refuses old pointers too.

**Instance update settings: the roll order.** Roll every box to the release that applies update settings before anyone turns on manual updates, a major hold or a pin: the console refuses those until every box reports it can protect the builds they name from pruning, and says how many boxes it is waiting on. Before rolling any box back below that release, turn retention off for the deployment (`carlos releases retention --off` for each app that has one), because an older housekeeper prunes builds a pin or a held instance still needs. After such a rollback the console shows *settings not enforced* for instances on the older boxes: they follow their channel until the box is rolled forward again.

**v0.25.0: instances move to the nested layout on their own.** This is
the first release on which an isolated (nested) instance wakes, and nested
is the default. On each box, migration starts unattended on the first
serial move after the release lands and the box's isolation probe passes:
every activator-managed instance that is asleep is moved into its own
directory and namespace at once; one that is awake gets an hour and is
then stopped for its move. An instance whose move is refused for a reason
a tenant could have planted (a file in its new directory, a second name
for its database) is held down automatically, and stays down across agent
restarts until the reason is cleared or the layout is reverted. An
operator who wants control over any of that acts BEFORE rolling: set
`CARLOS_LAYOUT_DEFAULT=flat` in each box's host.env to keep the fleet flat,
or record `carlos system layout migrate --box <label>` to move one box at
a time, or `carlos system layout revert` to opt the fleet out; a recorded
intent always wins over the default. Watch with `carlos status --layout`,
which marks a box `(nested by default)` and says why a box the default
has not reached is still flat. Cut v0.25.0 from a main that contains
7b7d3026 (plan 4's edge openat2 fix); without it nested instances cannot
wake.

**App regions rollback floor.** Once any app holds a region other than the
home, do not roll the console binary below the first release that
understands app regions: an older console places a no-region create at the
home. That release's version is named here, in its release notes, and in the
console roll recipe in carloku-infrastructure; check all three name the same
version before turning `CONSOLE_APP_REGIONS` on. If a rollback below the
floor happens anyway, write down when it started and ended: on its next
start the console logs "possible placement discrepancy" lines, and the
interval tells you which to review. Turning the switch off is the feature's
own rollback: it stops new non-home regions and keeps existing ones in force.

The first release that understands app regions is v0.23.0.

**Turning app regions on.** In order:

1. Stage the deployment's `CARLOS_SECRETS_KEY` on every box in every region
   that should take placed instances, and set `CARLOS_SECRETS_PUBKEY` on the
   console. A box without the key refuses to mint or wake placed exec instances
   whose config contains sealed secrets.
2. Roll the box release that publishes `serves_shared_pool`,
   `accepts_inbound` and `secrets_key_fp`. Check every region's
   `control/edges/*.json` carries them, and that each `secrets_key_fp` equals
   the console's.
3. Roll the console with the switch off.
4. On dev, turn the switch on. Create an app and an instance, read their
   region, and run the service gate below for every region.
5. On production, with the switch still off, create one probe instance per
   eligible region with an explicit `--region`, deploy a known build, and run
   `scripts/region-service-gate.sh <probe-host> <build> <every other edge EIP>`
   for each. Every region must pass. Delete the probes.
6. Turn the switch on. Re-run step 5 after any period with the switch off,
   and whenever a new region becomes eligible.

**#292: instance-control changes.** The release carrying #292 changes
`carlos-instance-control`: it orders a host's legacy run starts against
its stops. So the service must restart on roll. A box whose updater is
v0.26.0 or later does that itself, so no manual step is needed; an older
updater needs the hand restart described under "Old updaters" above.

### Tenant email senders, plan 2 of 5 (2026-09-21)

This release turns the one implicit per-app email sender into an explicit
sender record (design spec `2026-09-15-tenant-email-senders-design.md`).

- SES event counts now attribute to the sender that sent them. an app
  sharing a domain with a sibling app stops being charged for the
  sibling's sends. a tenant watching their own numbers will see them move.
- the app's email record moved to
  `apps/<acct>/<app>/email/senders/app.json`, with a read fallback to the
  old `email.json`. a console rolled back after the first sender write
  loses any pause or rotation made in between. that has to be redone by
  hand.
- two prerequisites from the design spec (§4.2) before any release sends
  with a tenant header.
    - every domain provisioned before this release has no SES tenant
      association, and nothing will create one on its own. a backfill, or
      a re-run of `carlos email domains add` against every existing
      domain, is needed before any release that sends with a tenant
      header.
    - the control-plane role needs `ses:CreateTenantResourceAssociation`
      and `ses:DeleteTenantResourceAssociation` before this rolls.
- `CARLOS_MAIL_LEGACY_SWEEP=1` is off by default. it does two separate
  things. it deletes the bucket's legacy `email.json` objects, and it
  drops the database's `mail_counts` table. do not set it until the
  release after this one has been live for a while.

### Tenant email senders, plan 4 of 5 (2026-09-22)

This release adds the per-instance config layer (design spec
`2026-09-15-tenant-email-senders-design.md` §6.2): a third config bundle,
layered over an app's default and named-environment config, scoped to one
route's host rather than the whole app.

- a new failure mode comes with this layer. every config sweep now also
  lists `apps/<acct>/<app>/instances-config/` before it converges that
  app's routes. s3 throttling, an iam policy change that adds a prefix
  condition to `s3:ListBucket`, or a transient 503 on that call all look
  the same from a box: the listing fails, and every route of that app
  freezes on its last-good config file until the listing succeeds again.
  this holds for the whole app, not just routes with an overlay of their
  own, because the listing runs once per app regardless of who uses the
  layer.
- grep the box's log for `instance config unresolved` to find it. it
  prints once per app, naming the account, the app and one route the pass
  reached. every other route of that app is covered by the same line and
  the same cause.
- it clears itself. once the listing succeeds again, the next pass
  converges normally, with nothing to clean up by hand.
- the in-repo iam policies grant `s3:ListBucket` on the whole deployment
  bucket with no prefix condition, so a platform box is unaffected by this
  on its own account's policy. check a hand-edited or self-hosted
  deployment's policy before this rolls there, since a narrower grant is
  exactly what would trip it.

### Availability, and why the edge had no answer (2026-09-24)

This release moves every 502, 503 and 504 the edge sees from the Errors
tab to a new Availability tab, grouped by cause (errors spec
`2026-09-07-errors-and-analytics-design.md` §11), and adds replica
placements (#409).

- roll the boxes and the console together. the boxes record causes and the
  console shows them. the console alone gives the Availability tab, and
  502s already stored regroup there under "cause not recorded"; until the
  boxes roll, new 502s land there too. boxes alone, with an old console,
  file new records the old console lists on Errors, one row per cause,
  named like "502 · the instance refused the connection".
- rolling the console back is safe. nothing in the bucket is rewritten;
  the regrouping happens on read.
- the Errors tab and `carlos errors` show fewer rows. edge 502/503/504 and
  every platform-stream line (failed wakes, exits, sidecar restarts) are
  on Availability now. anyone watching Errors for wake failures should
  watch Availability instead.
- a request whose visitor gives up after waiting five seconds or more,
  before the instance has sent response headers, is now filed as an
  outage, as "the visitor gave up waiting for the instance". the five
  seconds count from when the edge received the request, so on a
  hibernating route they include the wake. an endpoint that holds its
  response headers longer than that by design, like a long poll, will
  show a standing row for it. that row is expected; it is not a
  regression. a slow export that sends its headers straight away does
  not.
- a visitor who leaves within five seconds is no longer filed at all.
  previously those were filed as a 502 on whatever path they were
  loading, so expect fewer single-count 502s on static assets.
- on a route this box proxies over a tunnel or to a fleet, the front box
  now files only its own hop failures. the box serving the app files the
  rest, so a tunnelled 500 is filed once, not twice. two edges follow from
  that: a visitor who leaves after five seconds can be filed as
  "gave up waiting" on both boxes, and a far agent whose status store
  failed to open files nothing, which the front box no longer covers.
- a new CLI against a console that has not been updated:
    - `carlos availability` fails with `not found: check --account and
      --app, and that your token's address is a member`. until the
      console rolls, that message means the console predates the
      command, not that the app or your membership is missing.
    - `carlos instances create --role replica` is silently ignored by an
      old console, which creates an ordinary home instance in that region
      instead. update the console before creating a replica.
- replica placements add two registry columns (`routes.replicas`,
  `routes.replica`) on each box's start, by an idempotent `ALTER TABLE`.
  there is no migration to run, and a box rolled back keeps two unused
  columns harmlessly. do not create a replica until every box is on this
  release: only this release's DNS steering leaves replicas out of
  latency routing.

### Seven days of history for sleeping instances (2026-10-08)

This release keeps seven days of database snapshots for instances that
sleep, the same as instances that stay awake (#263). `carlos db pull --at`
(#262) can then reach back a week on them, not a day.

- boxes only. the console is unaffected.
- the window fills in over the week after the roll. snapshots older than a
  day were already deleted under the old setting, so on the day of the roll a
  sleeping instance still reaches back about a day, and a week only seven
  days later.
- an instance's history moves only while it is awake. one that sleeps for
  several days takes no snapshot in that time, and loses none either:
  retention also runs only while it is awake.
- storage grows by up to six more daily snapshots per sleeping instance,
  plus the hourly files back to the oldest one. there are no extra s3
  requests.

### Bucket reads only when something changed (2026-10-22)

This release stops every box and the console from reading the whole deployment bucket on every pass (#336). Writers now record which app changed in `control/changes.json`, and readers re-read only those apps. A full read of the bucket still runs every 30 minutes as a safety net.

- roll every box and the console before you arm any reader. writers update the index as soon as they run the new build, and readers stay off until you arm them.
- arm a box by adding `CARLOS_CHANGE_INDEX=read` to its `host.env` and restarting it, and the console the same way in its environment. do dev first, and boxes before the console.
- to turn it off, remove the variable and restart. readers go back to reading the whole bucket, and writers keep updating the index, which does no harm.
- anyone using an older carlos CLI that writes straight to the bucket (promote, env set, trash) should update it. until they do, their changes reach boxes at the next 30-minute read, not within seconds.
- customer fleet boxes keep reading the way they do today.
- to see the effect, look at daily S3 ListBucket and GetObject requests for the deployment bucket in Cost Explorer. they should fall within a day of arming.

### Email sending users get a permissions boundary (2026-10-23)

This release creates every email sending user (IAM path `/carlos-mail/`) under a permissions boundary, the way store users already are. Without one, anything that can act as the console's role can create a user on that path, give it an administrator policy and mint it a key.

A console with `CARLOS_MAIL_REGION` set refuses to start on this release without `CARLOS_MAIL_USER_BOUNDARY`. Its value is the ARN of a customer managed policy in the same account, `arn:aws:iam::<12-digit account>:policy/<name>`, that allows `ses:SendRawEmail` and nothing else. A console without `CARLOS_MAIL_REGION` needs nothing.

Roll it in this order:

1. create the boundary policy. set `CARLOS_MAIL_USER_BOUNDARY` to its ARN in every console's environment before any console gets this release. an older console ignores the variable. this release refuses to start without it.
2. apply IAM step 1 to the console role. add `iam:PutUserPermissionsBoundary` on `user/carlos-mail/*`, conditioned on `iam:PermissionsBoundary` equal to the boundary ARN. without that condition the role can set any boundary it likes. add `iam:ListUsers` for the sweep in step 4. AWS does not scope `ListUsers` to a path, so it is granted on `Resource "*"`. it only reveals user names and paths, and the sweep asks only for `/carlos-mail/`. the role already has `iam:GetUser` on the path. never grant `iam:DeleteUserPermissionsBoundary`. nothing calls it.
3. roll the console. from now on every new sending user gets the boundary. an existing user gets it the next time the console touches it: enable, a standalone credential, a domain change, and key rotation too, since no key is minted for a user without the boundary.
4. run the sweep, which also reaches users nothing will touch again. set `CARLOS_MAIL_BOUNDARY_SWEEP=1` on one console and restart it. it logs `set the boundary on <user>` for each user it fixes. its last line is either `all N mail users bounded` or the users that are not, with why. fix any failure and run it again. then remove the variable.
5. check it yourself. for every user in `aws iam list-users --path-prefix /carlos-mail/`, `aws iam get-user --user-name <user>` must show `PermissionsBoundary` with the boundary ARN.
6. apply IAM step 2. condition the role's `iam:CreateUser` and `iam:PutUserPolicy` on `iam:PermissionsBoundary` equal to the boundary ARN. do this only after every console runs this release, because an older console creates users without the boundary and step 2 refuses them.
7. wait a few minutes for IAM to propagate, then run the check in step 5 again with your own operator credentials, not the console role. `iam:CreateAccessKey` cannot be conditioned on the boundary, so anyone holding the console role could have made an unbounded user between step 5 and step 2. delete any user without the boundary, or set the boundary on it. the gap is closed only when this second check is clean.

## App config & secrets

Per-app config lives in the deployment bucket
(`apps/<acct>/<app>/config.json`): plain vars via `carlos env`, secrets
sealed client-side via `carlos secrets` — the bucket never sees plaintext.

    carlos secrets genkey                            # mint CARLOS_SECRETS_KEY / CARLOS_SECRETS_PUBKEY
    carlos env set --app keymail LOG_LEVEL=debug
    carlos secrets set --app keymail SES_SECRET=...   # sealed with CARLOS_SECRETS_PUBKEY before upload
    carlos secrets list --app keymail                 # key names only — a value is never readable again

Like `ship`/`promote` (see "From the CLI" above), `env` and `secrets` work
through a logged-in console with no `CARLOS_DEPLOYMENT_BUCKET`/`_DIR` on your
machine at all (issue #43) — `carlos auth login` alone is enough, including
for `secrets set`: with no `CARLOS_SECRETS_PUBKEY` in your own shell, it
seals against the console's own served public key instead (set via
`CARLOS_SECRETS_PUBKEY` in *the console's* environment; the private half
never leaves the hosts that hold it either way). Direct-bucket mode
(`CARLOS_DEPLOYMENT_BUCKET`/`_DIR` set) is unchanged.

The keypair is the mirror image of the release signing pair:
`CARLOS_SECRETS_PUBKEY` only encrypts and can be held widely (CI, laptops
— write-only credentials); `CARLOS_SECRETS_KEY` decrypts and belongs in
exactly two places, your secret manager and every host's
`/etc/carlos/host.env`.

Hosts converge during the hourly root `carlos adopt` pass: with
`CARLOS_SECRETS_KEY` set, each pass materializes the manifest into
`/etc/carlos/apps/<route.Host>.env` (the file `carlos-app@.service`
already reads) and restarts routes whose file changed. With it unset,
config sync is inert — the same convention as every other optional loop.
A write also reaches exec-backed instances in seconds rather than waiting
for that hourly pass (issue #44) via the agent's own fast poll loop; run
`carlos env sync --app <a>` to request convergence on demand.

Operational notes:

- Delivery is split by backing: unit-backed routes (`Backing: "unit"`)
  materialize to `/etc/carlos/apps/<host>.env` via the hourly root
  `carlos adopt`; exec-backed instance routes (socket-upstream)
  materialize to `/data/<host>.config.env` via the agent's in-process
  hourly loop, which also bounces a running instance when its config
  changes. At spawn the instance's own `/data/<host>.env` is layered on
  top — it wins every conflict, so `CARLOS_ADMIN_TOKEN` and box-local
  overrides are never clobbered from the bucket. Addr-upstream
  (proxy-only) routes get no delivery — the process behind them is
  externally managed.
- Migrate an app's config into the bucket *before* enabling
  `CARLOS_SECRETS_KEY` on its hosts: once a manifest object exists for an
  app, the bucket is authoritative for its env files (fully unset ⇒
  converges to empty). An app with no manifest at all is left alone.
- Values are write-only. Rotating the keypair means re-entering every
  secret of every app from its source of record — there is no rewrap yet.

## Domains

Every app has domains as a first-class concept: a **platform domain** it is
born with, and any number of **custom domains** layered on top — each with
DNS and TLS health visible in the console, and point-your-DNS-here guidance
when a custom domain isn't connected yet. The scheme:

    <app>.<account-sqid>.<apps-domain>          kass.bab.oncarlos.com
    <canary>.<app>.<account-sqid>.<apps-domain> pricing.kass.bab.oncarlos.com

The account carries the sqid (base-23 over a frozen no-lookalike alphabet,
encoding the account's minted id — deliberately not configurable, so nobody
can squat a vanity label; custom domains are the vanity mechanism), the app
carries its name — app names are unique per account, not globally. Account #1
is `ops`, the deployment operator's built-in account, created by migration;
CLI-added apps live there, and its sqid (`bab`) is identical on every
deployment. The
console writes `apps/<app>/account.json` at claim time; the agent's
reconcile pass (riding the 5-minute tick) reads those manifests and mints
platform routes, and removes only routes it minted itself — **custom
domains are never touched by reconciliation**. DNS needs one wildcard A
record (`*.<apps-domain>` → edge; RFC 4592 wildcards match multiple
labels); certs come per-host from the edge's existing ACME machinery.

A platform domain is an **alias**: a second name for the app's canonical
route, pointing at the same backend. It is not a lifecycle host — release
adoption and the fleet status counts both skip minted routes, so a
deployment that turns this feature on does not double-stage artifacts or
publish an inflated instance count.

**Three honest limits**, none of them papered over:

- **Minting is additive — an app is never re-homed.** Once an app has a
  platform apex, changing its account does *not* mint a new one or remove
  the old: one app, one apex, and moving an app between accounts is a
  deliberate operator migration (delete the old apex route by hand, let the
  next pass mint the new one), never a silent side effect of editing a
  manifest.
- **A hibernating app's platform domain 502s while the app sleeps.** The
  activator arms on the canonical route only; minted routes deliberately do
  not copy the `hibernate` flag, because a minted route carrying it would
  make the activator spawn a *second* process for the same app against an
  empty `DBPath` — meaningfully worse than a 502. Wake the app through its
  canonical host; the platform domain serves normally once it's up.
- **Canary hosts are minted for static apps only.** For a static app the
  channel genuinely is the backend (the edge resolves the channel pointer
  to find that release's tar), so a canary host serves exactly the canary.
  An instance app's minted canary would only be able to alias the app's
  *stable* process — a URL claiming to serve the canary while proxying
  stable — so none is minted; spawning a real per-host canary instance is
  future work (`carlos canary`), and until it exists the pass logs the gap
  rather than faking it.

**Config surface:**

- `PLATFORM_APPS_DOMAIN` — the deployment's apps domain (flagship:
  `oncarlos.com`), read by `carlos agent`, the edge, and the console (the
  member-facing `carlos instances enable` pins manifests to
  `<sqid>.<this domain>`; the console refuses a malformed value at boot).
  **Unset = the whole feature is inert**: no minting, no manifests read,
  the enable door answers 503 — the same litestream convention as
  everything else here.
- `CONSOLE_RESERVE_PLATFORM_DOMAINS` — set to `true` on a deployment that
  hosts apps for other people (the flagship, carloku). `carlos domains
  attach` then refuses every name under `PLATFORM_APPS_DOMAIN` and under the
  console's own hostname. **Unset = false**: a self-hosted deployment can
  attach any name except the console's own hostname and
  `CONSOLE_DENY_DOMAINS`. A value that is not a boolean stops the console at
  boot.
- `CONSOLE_DENY_DOMAINS`: comma-separated hostnames in the deployment's own domains (flagship: `carloku.com`). `carlos domains attach` refuses each one, and every name under it, to members. An owner of the ops account, or an address on `CARLOS_SYSTEM_OPERATORS`, can still attach one to an app with `carlos domains attach --platform`, and after that only they can change or detach it. The console's own hostnames, the steering host, the tunnel host and, with `CONSOLE_RESERVE_PLATFORM_DOMAINS`, everything under the apps domain stay refused to everyone.
- `CONSOLE_DOMAIN_PROOF` — set to `true` to require proof of control before
  an attached hostname is served: a TXT record at `_carlos-challenge.<host>`
  (`carlos domains verify`). A deployment that hosts apps for other people
  must set it, after the fleet runs a build that knows pending claims.
  **Unset = false**: attach is first-writer-wins, as before. A value that is
  not a boolean stops the console at boot.
- `PLATFORM_EDGE_IPS` — comma-separated edge IPs the DNS health checks
  compare against, read by both the agent (checks) and the console
  (guidance text), so the two can never name different addresses. Default:
  the box's public IP via IMDS, best-effort; if neither is knowable the
  checks stand down and health reads unknown, never a guess.
- `CONSOLE_APP_REGIONS`: `on` lets new apps be given a region other than the
  home. It needs `CONSOLE_HOME_REGION` set too. With the home unnamed, the
  console logs once and leaves app regions off. Default off. Automatic
  placement outside the home also needs `CARLOS_SECRETS_PUBKEY`, so the
  console can check that a box holds the deployment's secrets key.
- `CONSOLE_GEOIP_DB`: the path to an operator-supplied MaxMind-format `.mmdb`.
  DB-IP Lite is recommended, and it requires the attribution the new-app page
  shows. Unset or unusable turns the IP step of the region guess off, and the
  console starts either way. Replace the file by atomic rename, then restart.
  For the platform-wide GeoIP that every edge serves to apps, use
  `carlos system geoip publish` instead (docs/site/cli.md, "Platform GeoIP").
  Uploads are staged in `geoip-staging/` next to `CONSOLE_DB`, which the
  console clears at startup.
- `CARLOS_SIDECAR_ACCOUNTS`: comma-separated account sqids (not names) that
  may still enable and create native sidecars. Set, every other account gets
  403 `sidecars-paused` from `carlos sidecars enable` and `carlos sidecars
  create`. List, get, wake and delete still work, and existing sidecars keep
  running. Native sidecars run on the flat tier, outside the nested tenants'
  isolation, until nested sidecars land (discussion #370). **Unset or empty =
  no gate.** An entry that is not a sqid stops the console at boot.

**Canonical-host caveat**, stated honestly: an app that pins its own host
(e.g. keymail's `--host keymail.dev`) will redirect or 404 on its platform
domain — correct behavior. Rastrillo apps accept any granted host from day
one; for legacy apps the platform domain is the pre-custom-domain bootstrap
URL.

## Fleet status

Every deployment computes a fleet status view: how many instances are
running, how busy the boxes are, whether replication is healthy — the
CARLOS analogue of Eleven's own status pages. Each box's agent (`carlos
agent`) writes its own snapshot to the deployment bucket every 5 minutes;
whichever box holds the status lease that tick merges every snapshot into
`status/fleet.json` plus the embedded viewer (`status/index.html`,
`status/status.css`) and republishes all three. The edge serves them back
out through a `kind=status` route, bucket-backed with a short cache — no
release machinery involved, no CAS churn, just "read the key, cache it
briefly, serve it."

Three commitments hold end to end, enforced in the writer rather than the
page: the page publishes only fleet-wide aggregates and band words
(`comfortable`/`busy`/`critical`, `plenty`/`getting-full`/`at-capacity`) —
never raw ratios, hostnames, or machine ids. An unknown reading publishes
loudly as `null`, and a box that has stopped reporting renders `stale`,
never silently dropped. App names are opt-in only, via `carlos add
--status-named`, and everything else folds into one honest `apps_unlisted`
count, so the aggregate never lies by omission.

A route added this way is described by no bucket record, so nothing converges
it: its channel and environment are box-local until an instance record adopts
it. A provisioned route's channel is repointed through the record, not here.

**Config surface** — every env var this feature reads, gathered here since
no single reader owns the whole contract:

- `CARLOS_STATUS_BUCKET` / `CARLOS_STATUS_PREFIX` / `CARLOS_STATUS_REGION`
  — read by `carlos agent`'s status loop: the bucket each box writes its
  snapshot to and, when it wins the lease, publishes the merged feed to;
  region also feeds the cross-box status-lease election. **The serving side
  (the edge's `kind=status` route and the console's operator view) must be
  pointed at this same bucket** — nothing enforces that agreement
  automatically, publishing and serving are separately configured. To make
  the common case foolproof, an unset `CARLOS_STATUS_BUCKET` defaults to
  `CARLOS_DEPLOYMENT_BUCKET` (an explicit `CARLOS_STATUS_BUCKET` still
  always wins), so a single-bucket deployment needs only one env var for
  both release adoption and status. Both unset disables the whole loop —
  inert until configured, the same convention litestream replication uses.
  **Honest limitation**: that default only ever resolves to an S3 bucket —
  a deployment adopting from `CARLOS_DEPLOYMENT_DIR` (no bucket at all)
  does not publish fleet status yet; give it its own `CARLOS_STATUS_BUCKET`
  if it wants the page. `CARLOS_STATUS_PREFIX` does double duty: it
  namespaces the status-lease key, AND (customer-fleets plan, Task 5)
  rebases every key the status loop's AND the metrics loop's store
  touches — a fleet box set to `CARLOS_STATUS_PREFIX=fleets/<name>/` writes
  its own status snapshot at `fleets/<name>/status/boxes/<label>.json` and
  its raw/rollup metrics and alert state under `fleets/<name>/metrics/boxes/…`
  and `fleets/<name>/alerts/boxes/…`, instead of the shared paths a platform
  box uses — so neither leaks into the platform's own `fleet.json`,
  monitoring page, or alert webhook. Empty (every platform box today) is a
  pure passthrough. A **publishing** box with a prefix set would relocate
  `status/fleet.json` itself under that prefix too — moot today, since
  nothing sets a prefix on a publishing (non-fleet) box, and a fleet box
  never publishes at all (see `CARLOS_FLEET`, below).
- A box with `CARLOS_HIBERNATE_BUCKET` set but no deployment store
  (`CARLOS_DEPLOYMENT_BUCKET` or `CARLOS_DEPLOYMENT_DIR`) refuses placed exec
  instances, because it cannot tell whether their config is sealed. Set
  `CARLOS_NO_SEALED_CONFIG=1` to say this deployment has none.
- `CARLOS_FLEET` — read by `carlos agent`'s status loop (customer-fleets
  plan, Task 5). Set (to the fleet's name) marks this agent as a fleet box:
  it still self-reports its own snapshot every tick, but never contends for
  the cross-box publish lease and never merges/publishes `status/fleet.json`
  — that feed belongs to the platform's own boxes only. Unset (every
  platform box today) is unaffected.
- `CARLOS_FLEET_BOX` — this box's operator-chosen `fleet_boxes.label`
  (customer-fleets plan, Task 6 fix round). This — NOT the public status
  page's hashed machine-id pseudonym — is the box's real identity: the
  channel name the edge dials is `<fleet>/<label>`, so both the fleet
  binding a box publishes on wake and its own heartbeat filename
  (`fleets/<fleet>/status/boxes/<label>.json`) use it. Optional: defaults to
  `CARLOS_TUNNEL_NAME` with the `<fleet>/` prefix stripped, since a fleet
  box's tunnel name already has this shape. Both the fleet name and the
  resolved label are validated with `registry.ValidTunnelName`; a fleet box
  whose identity fails to resolve refuses (loudly) to arm binding-publish at
  all, and its heartbeat falls back to the hashed pseudonym instead of going
  dark.
- `STATUS_BOX_REGION` — also read by `carlos agent`'s status loop: an
  operator-set, coarse region word (e.g. `"EU · Ireland"`) carried onto
  this box's snapshot verbatim; unset omits the field. Free text the
  operator chooses, never derived from cloud metadata.
  Keep it geographically consistent with the region's place name in the
  console's region picker (for example "EU · Ireland" beside
  "Ireland · Dublin"); it may stay coarser.
- `STATUS_HOST` / `STATUS_PUBLIC` — not read by the agent process itself;
  consumed by ops/tofu to decide whether the edge fronts the bucket's
  `status/` prefix with a public hostname. `STATUS_PUBLIC=1` publishes the
  page at `STATUS_HOST`; leaving it unset keeps the feed private, visible
  only to console operators (`console grant --operator <address>`).

**Dry run.** Before pointing a binary at a real bucket, `carlos status
--dry-run <dir>` runs the whole publish pipeline — `CollectBox` →
`WriteBox` → `BuildFleet` → `PublishFleet` — against a scratch directory
stuffed with fixture secrets (hostnames, a machine id, mid-band fractions),
then reads every object it just published back out and fails loudly if any
of them leaked. There is no live-publish CLI verb; the 5-minute loop inside
`carlos agent` is the only publisher.

**Incident fallback, and its actual limits.** The page footer links "if this
page is unreachable" straight at the bucket's own copy of `fleet.json`, but
that link is built client-side from `location.origin` (see
`internal/status/assets/index.html`) — it is only meaningful once you are
*already* viewing the page served from the bucket's own origin (the public
`STATUS_HOST`, or the raw bucket URL). It is not a link that rescues you from
a *down* deployment; if the edge/console are unreachable you won't be looking
at this page's footer in the first place. The real operational fallback for
a down deployment is fetching the objects directly, either the raw bucket
URL in a browser or:

    aws s3 cp s3://<bucket>/status/fleet.json -
    aws s3 cp s3://<bucket>/status/index.html -

One current wrinkle: `index.html` fetched straight from S3 currently
**downloads instead of rendering** in a browser, because the store's `Put`
sets no `Content-Type` and S3 falls back to `application/octet-stream`. It
still opens fine as plain text in an editor, and `aws s3 cp ... - | less`
works, but "open the bucket URL in a browser and read it" doesn't work yet
for the HTML — see the design spec's rollout list (§5) for the queued fix
(setting content-type on publish). `fleet.json` is unaffected (any client
reading raw JSON already expects to view its own headers).

A fully-private deployment (`STATUS_PUBLIC` unset) whose console is down has
no page at all — the `aws s3 cp` fallback above is the entire story there;
nothing papers over it.

## Host & process monitoring

Operator-grade monitoring layered on top of fleet status: real CPU, memory,
and disk numbers per host and per process, with history, alerting, and a
console dashboard — needles where the public status page shows only band
words. See
the `2026-08-02-host-process-monitoring-design.md` design spec (private `platform-infrastructure` repo)
for the full design.

**Bucket layout:**

- `metrics/boxes/<box>/raw/<YYYY-MM-DD-HH>.json` — 60-second host + per-unit
  samples, one object per UTC hour, retained 48 hours.
- `metrics/boxes/<box>/days/<YYYY-MM-DD>.json` — hourly min/avg/max rollups,
  kept forever (the capacity/trend history).
- `alerts/boxes/<box>.json` — on-box alert state (fire/clear, hysteresis).

**Config surface** — read by the box agent from `/etc/carlos/host.env`:

- `CARLOS_ALERT_WEBHOOK` — optional; the agent POSTs the alert JSON here on
  fire and on clear. Unset simply skips the webhook channel (keymail
  delivery and the console UI channel are unaffected).
- `CARLOS_ALERT_SNS_TOPIC_ARN` — optional; the agent publishes each fire and
  clear to this SNS topic as a plain-text email-sized message, signed with
  the box's instance role (which needs `sns:Publish` on the topic). The
  region is read from the ARN, not from `AWS_REGION`. Set it to the box's
  own region's alerts topic and anyone subscribed to that topic gets the
  email. Either sink works alone and both may be set. A failed send is
  logged and counted, never fatal. The SDK tries a retryable failure up to
  three times within ten seconds. After that, a later tick does not send
  that alert again, unless the same tick's bucket write also failed, which
  repeats the whole transition on the next tick. A value that is not an SNS
  topic ARN (a subscription ARN, for example) is logged and the sink stays
  off. So does a client that cannot be built, which `carlos status --live`
  shows as `alerts=sns-unavailable`.
- With neither sink set, the box says so: the agent logs `NO SINK
  CONFIGURED` at start, and the box's private status snapshot carries
  `metrics.alerts` with both sinks false, which `carlos status --live` prints
  as `alerts=not-delivered`.
- `CARLOS_ALERT_DISK_PCT` / `CARLOS_ALERT_MEM_PCT` / `CARLOS_ALERT_CPU_PCT` —
  optional threshold overrides for the built-in disk / memory / host-CPU
  rules, as percent integers 0-100 (e.g. `95`). Anything else is ignored
  with a log line, and that rule keeps its built-in default.

**Console surfaces:**

- `GET /fleet/metrics` — operator-only dashboard: per-box needles and
  48-hour sparklines, a per-box per-unit table (ordered by CPU, busiest
  first), and active alerts at the top.
- A "Resources" section on each app's own page — owner-facing, scoped to
  that app's own instances only (current CPU and memory plus 48-hour
  sparklines); never host-level totals or other tenants' units.

The `days/` rollups are written from the first flush onward but nothing
renders them yet — they are the history a later trend view will read.

## App capabilities

A handful of apps need more from the platform than the `carlos-app@<host>`
template gives every instance for free — a subcommand other than `serve`, a
privileged port, the box's cloud instance role, or the resource headroom a
document-conversion engine needs. Rather than hand-provision a bespoke unit
per app, a release **declares** what it needs and the platform decides how —
and whether — to provide it, from a closed vocabulary of eight names. See
the `2026-08-03-app-capabilities-design.md` design spec (private `platform-infrastructure` repo)
for the full design; this section is the operator-facing summary.

**The manifest shape** — an optional `capabilities` object on the release
manifest, additive to schema 1 (a manifest with no `capabilities` key parses
as "none", today's behaviour byte for byte):

    "capabilities": {
      "v": 1,
      "command": "relay",
      "daemon": true,
      "children": true,
      "runtime_dir": "keymail",
      "ports": [ { "port": 25, "name": "mx" } ],
      "data": { "replicate": "app" },
      "host_role": true
    }

A document-conversion engine (Gotenberg, an OCR service) declares a different
shape — `host_role` and `doc_engine` are mutually exclusive and a manifest
carrying both is refused at ship, because `host_role` re-opens the cloud
metadata address and `doc_engine` exists to fence the engine's egress away
from private networks:

    "capabilities": {
      "v": 1,
      "daemon": true,
      "children": true,
      "doc_engine": true
    }

`command`, `daemon`, `children`, `runtime_dir` and `data` are self-service —
they shape only the app's own sandbox and materialize from the manifest
alone. `ports`, `host_role` and `doc_engine` are operator-consented: declaring
one asks for it, but nothing materializes until an operator has granted it
**on that specific box**, because the resource (a privileged port, the box's
IAM role, the private-network egress fence that pays for a conversion
engine's JIT and memory headroom) is box-local, not fleet-wide.

**Declaring a capability** rides `carlos ship --capabilities <file.json>`,
where the file holds exactly the `capabilities` object above (not a whole
manifest). The same strict decoder and validation gate ship and adopt both
run: an unknown capability name or field, a bad charset, a reserved
`runtime_dir`, or an out-of-range/reserved port is refused at the ship
terminal, where a human is standing to read it — no per-capability flags,
because a flag vocabulary would drift from `internal/caps`'s own.

**Granting a capability** is `carlos route`, per route, per box:

    carlos route --host keymail.dev --backing unit      # first: grants need it
    carlos route --host keymail.dev --grant port:25
    carlos route --host keymail.dev --grant host-role
    carlos route --host gotenberg.docs.example.com --grant doc-engine
    carlos route --host keymail.dev --revoke port:25
    carlos route --host keymail.dev --hibernate=false

Each of `--backing`, `--grant`/`--revoke` and `--hibernate` is its own operation:
they require `--host` (never `--app` — a bulk grant is precisely the operation
nobody should be able to type by accident), and none of them combines with
another, or with `--channel`/`--environment`, in one invocation.

The order above is not cosmetic. **A grant requires unit backing**: a route
that is not a `kind=instance`, `backing=unit` row has nothing for a grant to
materialize into, so `--grant`/`--revoke` are refused there, and grants cannot
be staged ahead of a migration. `--backing unit` is the switch that moves a
route from an agent-run exec child onto a `carlos-app@<host>` systemd
instance; an exec-backed route that declares capabilities materializes
nothing and is advised once per pass until it is switched over (see the
design's §5 migration steps). Stamping it arms the **hourly**
`carlos-adopt.timer`, and `carlos adopt` for a `daemon` route enables and
STARTS the service — so the data copy comes before the stamp, not after it.

A declared-but-ungranted capability does not fail the app or fail adoption:
the drop-in simply is not written, and the hourly adopt pass logs one line
naming the route, the capability, and the exact grant command. The mirror
line is there too — a token this box has granted that the live release no
longer declares materializes nothing, and is named with the revoke command
rather than left sitting in the registry looking like live authority.

**`doc-engine` carries two box preconditions** the platform cannot check for
you, which is what makes granting it an operator decision rather than a
checkbox. The grant buys the app JIT memory and a raised resource tier, and
in the same drop-in fences its egress away from every private network range —
RFC1918, link-local, CGNAT, IPv6 ULA and the NAT64 prefix — because a
conversion engine fetches whatever URLs the documents handed to it name, and
an unfenced one is an SSRF proxy into the VPC. Two consequences:

- **The box must hand the tenant a loopback DNS resolver.** The fence
  deliberately leaves 127.0.0.0/8 open — the engine drives its own Chromium
  and LibreOffice over loopback — but it closes 10.0.0.0/8, and on AWS the
  VPC resolver lives inside that range (VPC base + 2). A box running
  systemd-resolved on 127.0.0.53 resolves through loopback and the fence is
  transparent; a box pointing tenants straight at the VPC resolver gives them
  an engine that cannot resolve a hostname.
- **The box should be dedicated.** Loopback staying open means an SSRF'd
  engine can still reach anything listening on 127.0.0.1 on its own box — a
  co-tenant console, a co-tenant's loopback-bound upstream. The fence closes
  the network half of that risk; not co-locating tenants closes the rest.
  Run `doc-engine` routes on a box of their own.

**Deploy order is pinned**, same rule and reason as scoped signing keys: the
platform binaries that understand a capability name have to be on the box
*before* a release declaring it, never after. An older binary meeting a
manifest it does not understand refuses the version on that box — not
staged, not pointed at — rather than materialize a unit missing something
the app says it needs.

**"Platform binaries" includes the console binary**, and on the flagship that
is the one that matters: `carlos ship` in API mode posts through the console,
so the console is what either understands the `capabilities` key or silently
drops it as an unknown JSON field — storing a manifest that declares nothing,
which on a box is the instruction to REMOVE the drop-ins. The ship response
therefore echoes the capability block it stored, and `carlos ship` fails hard
when it sent a block and gets no echo back ("this console does not understand
capability declarations — update the console binary first, or ship without
`--capabilities`"). Update the console before shipping a capability-bearing
release.

**What refuses vs what leaves alone** — the distinction that matters most:

- An unreadable or unresolved staged manifest **leaves every drop-in exactly
  as it is** and logs once per pass. A bucket hiccup must never strip `:25`
  off a running mail server.
- A manifest that actually resolves and no longer declares a capability (or
  a grant that gets revoked) **removes** the generated drop-in.
- A hand-edited drop-in — anything at the generated path without the
  `# GENERATED by carlos adopt — do not hand-edit` marker on its first
  line — is never overwritten and never deleted; it is reported and that
  unit's half is skipped while the other half still converges.
- An unknown capability name or an unsupported `"v"` refuses the *version on
  that box*, not the whole fleet — the box keeps serving what it already
  runs.

Console gets, at most, a read-only line beside each route's other facts —
"granted: host-role, port:25" for a route with grants, nothing at all
otherwise, and only on routes belonging to the account whose page it is.
There is no console UI for granting or revoking; grants stay a `carlos route`
operation on the box, and nothing about the console's
sign-in/session/CSRF/owner gates changes.