# Console release defences, Plan 4 of 4: D4 (the release carries a console; the roll goes dev first) and the rollout > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Each public release publishes a signed console artifact. The flagship console is rolled with exactly those bytes, through dev first, carrying dev's proof. The four defences are switched on in the order spec §6 requires. **Architecture:** Three repos. The platform README's release recipe gains the console build and a console roll step. `carloku-infrastructure/scripts/roll-console.sh` gains: - a `dev` target; - `--artifact` and a single `-trimpath` build; - a binding check; - the dev-proof carry, with a one-time `--bootstrap`. The rollout is a runbook section in carloku-infrastructure's README, run by an operator over SSM. It is not code. **Tech Stack:** bash, the `carlos` CLI (with Plans 1–3 built), AWS CLI with SSM, `jq`. **Spec:** `docs/superpowers/specs/2026-10-06-console-release-defences-design.md` (rev 5): §5 (D4), §6 (rollout). **Depends on Plans 1–3 being merged and in a release.** The script calls `carlos proof get|put` (Plan 3) and relies on the box's settle check (Plan 1). ## Global Constraints - The release builds `console-linux-arm64` and `console-linux-amd64` with `go build -trimpath -ldflags "-s -w -X github.com/carlosframework/platform/internal/version.Build=vX.Y.Z" ./cmd/console` and lists them in the signed `SHA256SUMS.txt` (spec §5.1). - Dev coordinates: profile `kass`, bucket `carlos-dev-640012953448`, console box `i-05498eada6f0afd23`, origin `platform.dev.carlosframework.com`, region eu-west-1 (spec §5.2). - Flagship coordinates: profile `keymail`, bucket `carlos-flagship-271376211898`, console box `i-092c0c1eea75723cb`, origin `platform.carlosframework.com`. - The script refuses a flagship roll without a dev proof, except with `--bootstrap`, which prints that it is skipping (spec §5.2). - The script keeps its final `systemctl restart` on any box whose binding is wrong (spec §5.2, §2.2). - The public `carlos-linux-*` binary is never a console artifact, and the recipe says so (spec §5.2). - README prose in both repos goes through the copy-review skill before it is committed (Paul's standing rule for READMEs). Plans 1 and 2's README notes need the same gate. The copy rule: no em dashes in new prose (`[[copy-voice-no-em-dashes]]`). - Tito is a third party: the script's `tito` target is left as it is and never run by this plan. - **Plan decision:** the console routes' readiness uses the TCP-connect fallback (spec §2.4 rule 2) on `127.0.0.1:8090`, not a configured probe. The console's `/healthz` answers as soon as its listener is up, so a probe would check nothing more than the connect does. No box-local CLI sets `registry.Route.Health` on a console row today; adding one is a follow-up, needed only if that equivalence ever stops holding. ## Review Focus 1. **The public release's `SHA256SUMS.txt` lacks a console line** (an older release, or a recipe step skipped). Expect: `--artifact` refuses an unverifiable file, never ships it. Covered in Task 2 (`verify_artifact`). 2. **Dev has the build but its proof isn't minted yet** (inside the 15 minutes). Expect: the flagship target exits non-zero naming how long dev has had it, with nothing shipped to flagship. Task 2's dry-run check. 3. **The binding on a box is wrong.** Expect: the script says so, keeps its final restart, and still verifies `/proc//exe`. Task 2's dry-run check against the flagship box today (stamped on `console.carloku.com`, while `ExecStart` names `platform.carlosframework.com`; confirm on the box). 4. **An operator rolls a hotfix sha, not a release.** Expect: one `-trimpath` build reused for dev and flagship, so the proof's hash matches. Task 2's build-once rule. 5. **Someone runs the old two-repo habit** (platform-infrastructure's copy of the script for flagship). Expect: that copy says it is retired and points here. Task 3. --- ### Task 1: The release recipe publishes a console and rolls it dev first **Files:** - Modify: `README.md` (platform repo): "Releasing the platform" steps 2, 4 and 6, and a new console step after "Last, roll it" - [ ] **Step 1: Draft the recipe changes** Step 2, append: ```markdown Build the console twice, from the same clean checkout: `CGO_ENABLED=0 GOOS=linux GOARCH= go build -trimpath -ldflags "-s -w -X github.com/carlosframework/platform/internal/version.Build=vX.Y.Z" -o console-linux- ./cmd/console` for arm64 and amd64. The console is `./cmd/console`. The `carlos` binary is never a console artifact; shipping it as one took the flagship console down on 2026-10-06. ``` Step 4: "over the eight files" becomes "over the ten files". Step 6: "with all ten files" becomes "with all twelve files". After the "Last, roll it" paragraph, add: ```markdown **Roll the console.** After the boxes are current, roll the release's own console bytes, dev first: curl -sSLO $rel/console-linux-arm64 # verified against SHA256SUMS.txt like the CLI above carloku-infrastructure/scripts/roll-console.sh --artifact console-linux-arm64 --version vX.Y.Z - dev # wait for dev's proof: 15 minutes of the build running healthily carloku-infrastructure/scripts/roll-console.sh --artifact console-linux-arm64 --version vX.Y.Z - flagship The flagship roll refuses until dev has proven those exact bytes. ``` - [ ] **Step 2: Put the new prose through copy review** Invoke the `copy-review` skill on the three new passages above. Apply Paul's edits verbatim. - [ ] **Step 3: Commit (platform repo, implementation branch)** ```bash git add README.md git commit -m "README: the release builds and rolls its console, dev first (console defences D4)" ``` --- ### Task 2: `roll-console.sh`: dev target, one artifact, binding check, proof carry **Files:** - Modify: `~/github.com/carlosframework/carloku-infrastructure/scripts/roll-console.sh` (whole file below) - Create the work on a branch in that repo, `console-release-defences`. It is a GitHub repo: commit on a branch and open a PR for Paul. Do not merge. **Interfaces:** - Consumes: `ship-app.sh ` (same directory, unchanged); `carlos proof get --app console --sha256 ` and `carlos proof put --app console ` (Plan 3); `carlos version` (prints `carlos `). - Produces: `roll-console.sh [--artifact FILE --version V] [--bootstrap] dev|flagship|tito`. - [ ] **Step 1: Replace the script** ```bash #!/bin/bash # 🤖 Roll the console: build (or take) ONE artifact, ship + promote it, # verify the running binary. Console release defences D4 (platform spec # 2026-10-06-console-release-defences-design.md §5.2). # # roll-console.sh [--artifact FILE --version V] [--bootstrap] # # --artifact rolls exactly FILE (a release's public console-linux-arm64, # checked against that release's signed SHA256SUMS.txt first), so dev and # flagship run the bytes the public got. Without it the script builds # ./cmd/console ONCE at the checkout's HEAD with -trimpath and keeps the file # in $CONSOLE_BUILD_DIR (default ~/.cache/roll-console/) for the next # target: a second build would be different bytes, and flagship's dev proof # names bytes. # # flagship refuses unless dev has proven those bytes (`carlos proof get` # against dev). --bootstrap skips that once, for the roll that installs D3. # # The box adopts and restarts the console itself; its settle check rolls back # a build that does not come up. The old adopt + restart SSM steps run only on # a box whose console.service ExecStart does not name the unit-stamped route's # live link, where the settle check cannot judge the restart. set -euo pipefail artifact= version= bootstrap=0 while [ $# -gt 2 ]; do case $1 in --artifact) artifact=$2; shift 2;; --version) version=$2; shift 2;; --bootstrap) bootstrap=1; shift;; *) echo "unknown flag $1" >&2; exit 2;; esac done repo=${1:?usage: roll-console.sh [--artifact FILE --version V] [--bootstrap] } target=${2:?target: dev, flagship or tito} scripts=$(cd "$(dirname "$0")" && pwd) # name -> "aws-profile bucket instance-id console-origin" coords() { case $1 in dev) echo "kass carlos-dev-640012953448 i-05498eada6f0afd23 platform.dev.carlosframework.com";; flagship) echo "keymail carlos-flagship-271376211898 i-092c0c1eea75723cb platform.carlosframework.com";; tito) echo "titogo carlos-tito-platform-665106592142 i-025a0b70f3d93304d carlos.tito.io";; *) echo "unknown deployment $1: dev, flagship or tito" >&2; exit 2;; esac } read -r profile bucket instance origin <<<"$(coords "$target")" # A native carlos to drive ship/promote/proof. From the checkout when one is # given, else the one on PATH (it must have `proof`, i.e. Plans 1-3). tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT if [ "$repo" != - ]; then if [ -n "$(git -C "$repo" status --porcelain)" ]; then echo "refusing to build: $repo has uncommitted changes" >&2; exit 1 fi (cd "$repo" && CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X github.com/carlosframework/platform/internal/version.Build=$(git -C "$repo" rev-parse --short HEAD)" -o "$tmp/carlos" .) carlos=$tmp/carlos else carlos=$(command -v carlos) fi "$carlos" proof -h >/dev/null 2>&1 || { echo "$carlos has no 'proof' verb: build from a checkout with the console release defences" >&2; exit 1; } if [ -n "$artifact" ]; then [ -n "$version" ] || { echo "--artifact needs --version (the release tag)" >&2; exit 2; } sums=$(dirname "$artifact")/SHA256SUMS.txt [ -f "$sums" ] || { echo "no $sums next to $artifact: download the release's SHA256SUMS.txt and .sig and verify them first (README, Releasing the platform)" >&2; exit 1; } want=$(awk -v f="$(basename "$artifact")" '$2==f {print $1}' "$sums") [ -n "$want" ] || { echo "$sums has no line for $(basename "$artifact"): this release did not publish a console" >&2; exit 1; } got=$(sha256sum "$artifact" | cut -d' ' -f1) [ "$got" = "$want" ] || { echo "$artifact sha256 $got does not match $sums ($want)" >&2; exit 1; } file=$artifact else [ "$repo" != - ] || { echo "no --artifact and no checkout to build from" >&2; exit 2; } version=$(git -C "$repo" rev-parse --short HEAD) dir=${CONSOLE_BUILD_DIR:-$HOME/.cache/roll-console/$version} file=$dir/console-linux-arm64 if [ ! -f "$file" ]; then mkdir -p "$dir" (cd "$repo" && CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath \ -ldflags "-s -w -X github.com/carlosframework/platform/internal/version.Build=$version" -o "$file" ./cmd/console) echo "built $file (kept for the next target)" fi fi sha=$(sha256sum "$file" | cut -d' ' -f1) echo "== console $version sha256 $sha -> $target" # ssm_run : one command on this target's console box (jq-built # parameters; see the 2026-08-09 note in git history for why). ssm_run() { local params id status params=$(jq -n --arg c "$1" '{commands: [$c]}') id=$(aws --profile "$profile" --region eu-west-1 ssm send-command --instance-ids "$instance" \ --document-name AWS-RunShellScript --parameters "$params" --query Command.CommandId --output text) aws --profile "$profile" --region eu-west-1 ssm wait command-executed --command-id "$id" --instance-id "$instance" || true status=$(aws --profile "$profile" --region eu-west-1 ssm list-command-invocations --command-id "$id" --details \ --query 'CommandInvocations[0].CommandPlugins[0].Status' --output text) aws --profile "$profile" --region eu-west-1 ssm list-command-invocations --command-id "$id" --details \ --query 'CommandInvocations[0].CommandPlugins[0].Output' --output text [ "$status" = Success ] } # The binding: which live link console.service runs, and which route carries # the unit stamp. Equal means the box's settle check judges the restart. binding=$(ssm_run 'exec=$(systemctl show -p ExecStart console.service | sed -n "s/.*path=\([^ ;]*\).*/\1/p"); \ stamped=$(sqlite3 -readonly /data/registry.db "select host from routes where unit=\"console.service\"" | tr "\n" " "); \ echo "$exec|$stamped"') exec_link=${binding%%|*}; stamped=${binding#*|} bound=0 [ "$exec_link" = "/opt/carlos/live/${stamped% }" ] && bound=1 if [ $bound = 1 ]; then echo " binding ok: console.service runs $exec_link, stamped on ${stamped% }" else echo " binding WRONG: console.service runs ${exec_link:-?}, unit stamp on '${stamped% }' — keeping the final restart (fix: platform spec §6 step 1)" fi if [ "$target" = flagship ]; then proof=$tmp/proof.json if AWS_PROFILE=kass AWS_REGION=eu-west-1 CARLOS_DEPLOYMENT_BUCKET=carlos-dev-640012953448 \ "$carlos" proof get --app console --sha256 "$sha" >"$proof"; then echo " dev proof found" elif [ $bootstrap = 1 ]; then echo " --bootstrap: NOT checking for a dev proof (only for the roll that installs D3)" proof= else echo "flagship: dev has not proven $sha (above); roll it to dev and wait, or --bootstrap once" >&2 exit 1 fi fi CARLOS_BIN=$carlos "$scripts/ship-app.sh" "$profile" "$bucket" console "$version" canary/console binary "$file" if [ "$target" = flagship ] && [ -n "${proof:-}" ]; then AWS_PROFILE=keymail AWS_REGION=eu-west-1 CARLOS_DEPLOYMENT_BUCKET=carlos-flagship-271376211898 \ "$carlos" proof put --app console "$proof" fi if [ $bound = 0 ]; then ssm_run "bash -c 'set -a; source /etc/carlos/host.env; /opt/carlos/carlos adopt' 2>&1 | tail -8" ssm_run "systemctl restart console && sleep 2 && systemctl is-active console" fi echo "== $target: waiting for the running binary to be this build" for i in $(seq 1 24); do exe=$(ssm_run 'readlink /proc/$(systemctl show -p MainPID --value console)/exe' || true) case $exe in */versions/$version/console*) break;; esac sleep 5 done case $exe in */versions/$version/console*) ;; *) echo "$target: console is running ${exe:-nothing}, not versions/$version/console — check the box's adopt journal for a settle rollback" >&2; exit 1;; esac curl -fsS -o /dev/null "https://$origin/" echo "== $target: console $version live at https://$origin" ``` Checked on the flagship console box on 2026-10-06: `/usr/bin/sqlite3` is present, and `ExecStart` is `/opt/carlos/live/platform.carlosframework.com`, while `carlos instances` shows the `console.service` stamp on `console.carloku.com`. That is the wrong binding Review Focus 3 expects. Run `ssm_run 'command -v sqlite3'` against dev once as well. If dev lacks it, install it there (`apt-get install -y sqlite3`) rather than branch the script. - [ ] **Step 2: Static checks** Run: `bash -n scripts/roll-console.sh && shellcheck -S warning scripts/roll-console.sh` Expected: no output. Fix every warning except SC2016 on the single-quoted SSM command strings, which are expanded on the box on purpose. - [ ] **Step 3: Read-only check against dev and flagship** Temporarily add `[ "${ROLL_DRY:-0}" = 1 ] && exit 0` immediately after the binding report. Run `ROLL_DRY=1 scripts/roll-console.sh - dev` and `ROLL_DRY=1 scripts/roll-console.sh - flagship` with the sandbox off (assumed-role profiles: `[[aws-from-the-sandbox]]`). Expected: - dev prints its binding line; - flagship prints `binding WRONG`, unless someone already restamped it, which rollout step 1 does; - nothing is shipped. Then remove the line. - [ ] **Step 4: Commit and open the PR (carloku-infrastructure)** ```bash cd ~/github.com/carlosframework/carloku-infrastructure git checkout -b console-release-defences git add scripts/roll-console.sh git commit -m "roll-console: dev target, one artifact for every target, binding check, dev proof before flagship" git push -u origin console-release-defences gh pr create --title "roll-console: dev first, one artifact, dev proof (console release defences D4)" \ --body "Implements platform spec 2026-10-06-console-release-defences-design.md §5.2. Not to be run until the platform release carrying Plans 1-3 is on the boxes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)" ``` This repo lives on GitHub, not amadan, so a PR is the right door here. Leave it for Paul to merge. --- ### Task 3: Retire platform-infrastructure's copy of the script **Files:** - Modify: `~/amadan.net/carlos/platform-infrastructure/scripts/roll-console.sh`: replace the body with a pointer - [ ] **Step 1: Replace the body** ```bash #!/bin/bash # Retired 2026-10: the console roll lives in carloku-infrastructure # (scripts/roll-console.sh), which rolls dev first and carries dev's proof # to flagship. Platform spec 2026-10-06-console-release-defences-design.md §5.2. echo "roll-console.sh moved to carloku-infrastructure/scripts/roll-console.sh (dev, flagship, tito)" >&2 exit 2 ``` - [ ] **Step 2: Commit on a branch in that repo** Read that repo's own instructions first (its README or AGENTS.md) for where and how changes land. Then commit: `git commit -m "roll-console: retired, see carloku-infrastructure"`. --- ### Task 4: The rollout runbook **Files:** - Modify: `~/github.com/carlosframework/carloku-infrastructure/README.md`: add a section "Console release defences: rollout", on the Task 2 branch - [ ] **Step 1: Write the section** The commands below are the spec §6 steps, made concrete. Each SSM command is run with `.claude/ssm.sh ''` or the script's `ssm_run` pattern. ````markdown ## Console release defences: rollout Run once, after the platform release that carries the defences is published. In order: 1. **Fix the bindings.** On each console box (dev `i-05498eada6f0afd23`, flagship `i-092c0c1eea75723cb`), the one route stamped with `console.service` must be the host its `ExecStart` names: systemctl show -p ExecStart console.service # path=/opt/carlos/live/ /opt/carlos/carlos route --host --unit console.service /opt/carlos/carlos route --host --unit "" On flagship today that means stamping `platform.carlosframework.com` and clearing `console.carloku.com`. 2. **Roll the boxes** with `carlos system update --console --version vX.Y.Z` on dev and flagship, then `carlos system status`. 3. **Dev proving.** On dev's console box, as root: /opt/carlos/carlos proof keygen --out /etc/carlos/proving.key # prints the public key cat > /etc/carlos/proof-policy.json <<'EOF' {"schema":1,"prove":{"apps":["console"],"prover":"dev","after":"15m","key_file":"/etc/carlos/proving.key"}} EOF chmod 0644 /etc/carlos/proof-policy.json Record the public key here: `dev proving key: `. 4. **Roll the release's console to dev:** `scripts/roll-console.sh --artifact console-linux-arm64 --version vX.Y.Z - dev`. Watch the box journal for `settled on`. 5. **Rehearse on dev**, then delete each rehearsal release from dev's bucket (`apps//console/releases//`) and its staged copy on the box: - ship the public `carlos-linux-arm64` as console `rehearse-cli` with a **v0.24.1** `carlos`, direct-bucket mode (`CARLOS_DEPLOYMENT_BUCKET` set). Promote it, and expect the box journal to show `not pointed live` with no restart; - ship a console built from `./cmd/console` with `os.Exit(2)` as the first line of `main`, as `rehearse-exit`. Expect a rollback within 30 s, a refusal, and an alert email. A plain re-promote stays refused, and `carlos promote --retry rehearse-exit canary/console` retries; - ship a console whose `serve` sleeps forever before listening, as `rehearse-hang`. Expect a deadline rollback that replaces the process. 6. **Wait for dev's proof** of step 4's build: `carlos proof get --app console --sha256 ` against dev. 7. **Roll it to flagship:** `scripts/roll-console.sh --artifact console-linux-arm64 --version vX.Y.Z - flagship`. 8. **Switch flagship's requirement on.** On flagship's console box, as root: cat > /etc/carlos/proof-policy.json <<'EOF' {"schema":1,"require":[{"app":"console","prover":"dev","key":""}]} EOF chmod 0644 /etc/carlos/proof-policy.json The live build is unaffected. It enters the box's healthy history after 15 minutes. **Break glass:** on flagship, `rm /etc/carlos/proof-policy.json`, then `/opt/carlos/carlos adopt --stage-only`. The box logs hourly that the requirement is off until the file is back. The policy files are not in tofu `user_data` on purpose: a whole-root apply restarts every box. ```` `--out` is Plan 3 Task 9's flag; its default is `/etc/carlos/proving.key`. - [ ] **Step 2: Put the section through copy review** Invoke the `copy-review` skill on the section's prose (not the commands). Apply Paul's edits verbatim. - [ ] **Step 3: Commit on the Task 2 branch and push** ```bash git add README.md git commit -m "README: console release defences rollout runbook" git push ``` --- ### Task 5: Run the rollout This task is operational and outward-facing: it changes production boxes. **Confirm with Paul before each of steps 1, 7 and 8** of the runbook. Steps 2–6 touch only dev or are read-only on flagship, so they need his go-ahead once, at the start. - [ ] **Step 1:** Run runbook steps 1–8 in order, pasting each command's output into the branch's amadan discussion (`amadan discuss new carlos/platform -branch -title "Console release defences rollout"`). - [ ] **Step 2:** After step 8, check `carlos status --live` against flagship shows no refusals and `curl -sI https://console.carloku.com/ | grep -i x-carlos-version` names vX.Y.Z. - [ ] **Step 3:** Update the `[[console-release-defences]]` memory with what was rolled and when.