# Zero-downtime deploys: rev 18 amendment (signed host retries) Status: approved by Paul, 2026-10-13. Nothing here is merged, released or rolled. This file amends `2026-10-07-zero-downtime-deploys-design.md` (rev 17), which lives on the design branch (`platform-design-zero-downtime-deploys`) and is not edited from this branch. It quotes each line it replaces and gives the replacement. After sign-off, one copy step puts the text into the spec and raises the status to rev 18. Why a revision at all. Rev 14 dropped a separate signed retry object and let `carlos promote --retry` be the retry signal. That holds for the app-wide retry and is unchanged. It never covered `carlos instances retry `, the one-host retry, which rev 17 describes only as "clears it for that host only" and which §11 wrongly lists under the re-signed pointer. The built feature is a console-signed grant, bound to one failure book and to one restriction state of that book, and the spec has to say so before the flag goes on. Built on `zdd-flag-readiness-2`, on top of the grant's first form on `zdd-flag-readiness` (`d68a1dde`, then `c12c35c3` for the book identity; the branch is pending merge): - the restriction generation and the book rules: `7c7c6141`, `191c12b9` - the host-retry log, the box's read of it, and the console's report check: `6e9e84bd`, `9f27cdfa`, `4d8957c1`, `18bde560` - the console page for a declined box: `6e6ad000`, `65d4e135` ## 1. Status line (spec line 3) Replace the opening words of the line, "Status: rev 17, 2026-10-07. Rev 17 adds", with: > Status: rev 18, 2026-10-12. Rev 18 makes `carlos instances retry ` a > root-signed grant bound to one failure book and one restriction state, and > says where it rides and how a box checks it (§8.4, §11). Rev 17 adds Replace this sentence (line 3): > Rev 14 adopts `carlos promote --retry` from `main` (9e628024) as the retry > signal, in place of a separate signed retry object. That is a simplification > found while planning. with: > Rev 14 adopted `carlos promote --retry` from `main` (9e628024) as the > app-wide retry signal, in place of a separate signed retry object. That > stands. Rev 18 adds a signed object back for the one-host retry only, > because a one-host retry cannot ride a channel pointer (§11). The rest of line 3 ("**Accepted by Astra in round 13**" and "Awaiting Paul's review") stays, with one sentence added after "Awaiting Paul's review": "Rev 18 has not been through a numbered spec round. The code that builds it was reviewed task by task on `zdd-flag-readiness-2`, and Astra's whole-branch review of that code is separate." ## 2. §8.4, the one-host retry (spec lines 704 and 705) Replace: > - **`carlos instances retry `** clears it for that host only, and > the next attempt runs as a canary. with: > - **`carlos instances retry `** clears it for that host only, on a > grant the console signs (§11). On a box that arms the scheduler, the next > attempt runs as a canary. A box that declined the overlap flag (for > example with `host-claims-on`) still honours the grant: the failure hold > is cleared and its next wake starts the build as it does today, with no > canary, because that box runs none. The console offers the retry only for > a host whose box reports that it can take the grant, and refuses it with > a message otherwise (§11). Lines 706 onward (`carlos promote --retry `) are unchanged. ## 3. §11, the retry sentence (spec lines 836 and 837) Replace: > - A retry is a re-signed channel pointer (`carlos promote --retry`), so one > box can't forge a retry that clears another box's failure records (§8.4). with: > - An app-wide retry is a re-signed channel pointer (`carlos promote > --retry`), so one box can't forge a retry that clears another box's > failure records (§8.4). A one-host retry (`carlos instances retry > `) is a grant the console signs with the deployment root key. A box > clears a failure hold only on such a grant, checked as described below. ## 4. §11, new paragraph: the host retry grant Add after the sentence above: > **The host retry grant.** `carlos instances retry ` makes the console > sign a `HostRetryGrant` and write it to the app's host-retry log. The grant > is `{v: 1, account, app, host, build, epoch, book, issued_at}`. `build` is > the build whose restriction the grant lifts, which is not the build the > host should move to. `epoch` and `book` are what the host's box last > reported for that restriction (`held_epoch` and `held_book` in its status), > so the grant names the exact state the member saw. > > *Signature.* The envelope is signed with the deployment root key. A scoped > release key is refused at signing: a channel delegate has no say over > another instance's failure hold. A box verifies against the root public key > it already holds for channel pointers, and a box without one drops every > grant. The log is writable by every box, so an unsigned entry is a request > anyone with a box could have written, and it clears nothing. > > *Time.* A box accepts a grant for 24 hours after `issued_at` and tolerates > one minute of clock skew into the future. That window is a backstop. What > binds a grant to one failure is the next two items, not the clock. > > *Book and generation.* A box's failure book has a random identity, and it > keeps one restriction generation per app, a counter that moves on every > change to any restriction of that app (a record made, changed or removed, a > host retried, a hold-ran marker set, renewed, replaced or removed). The box > publishes the generation only once it is on disk, and never hands the same > value out twice. A grant clears a host's restriction only if the box's book > is the one the grant names, the book is saved, the app's generation still > equals the grant's epoch, and the grant's build still restricts the host. > So a grant clears the state the member saw and nothing that changed after > it, not even after an agent restart: a reopened book starts past every > value it ever saved, and a book written by an earlier agent that may have > published unsaved values is given a new identity when it is adopted, so > every grant issued before names the old one. Epochs restart per book, which > is why the grant needs both. A retry spends the grant: it moves the > generation, so the same grant clears nothing a second time. > > The guarantee rests on two assumptions about the book's file that the box > cannot check from inside. Only an agent that follows this rule writes it > (the file is written as schema 3, which an earlier agent refuses to open), > and it is never restored from an older copy. Restoring or cloning an agent > data dir must give the copy a new identity, by dropping the file's `book` > key before the agent starts with the flag on. The proof sketch and the > assumptions are in the doc comment on `FailureBook` > (`internal/rollout/failures.go`). > > *Where it rides.* Grants live in `apps///host-retry.json`, a > CAS-guarded request log of the same shape as the restart-scope log > (`apps///restart-scope.json`, which carries `carlos restart` > narrowed to a host or an environment), kept to the last 32 entries. They do not ride > the restart-scope log. A console built before the grant existed > re-marshals that whole log whenever it appends a restart, and in doing so > strips the signature from every pending retry, so the member's retry > silently clears nothing. No released binary reads or writes the host-retry > log, so no older console can rewrite it and no older box ever sees a grant. > The restart-scope log no longer accepts a retry kind or a signature when it > is written. A box still honours a signed grant that an unreleased build left > in the restart-scope log, and checks it the same way. A box reads the > host-retry log on the serial move, at the cost of one extra read per app, > and does not skip the entries it finds at agent start: a grant written > before an agent restart is checked and, if still current, applied, and a > spent or stale one is a no-op under the generation rule. > > *What a box drops.* An unsigned entry, a forged signature, a grant for > another account, app or host than its entry names, a grant outside its > window, one with no book or a zero epoch, one for another book, and one for > a generation that has moved are each dropped with a log line and clear > nothing. A box that declined the overlap flag and still keeps a failure > book honours grants too, with no canary. > > *The console.* A console with no signing key answers 503 and writes > nothing. Before it signs, it reads the box's report for the host and > refuses, with a message and a 409 on the API, a box whose report carries no > `retry_v` (a build older than the host-retry log), no `held_book`, or an > epoch of zero (a book that owes a save). It offers no Retry button for > those hosts. Nothing is written for a refused retry, so the sequence of box > and console rolls cannot make a grant apply unsafely: a Retry pressed > before the box has updated is refused with a message, never lost silently. > `retry_v` is additive on the wire, and older readers ignore it. ## 5. Unchanged `carlos promote --retry` stays the app-wide re-signed pointer. The §9.1 list (spec lines 761 and 762: "`carlos promote --retry` (existing) retries a build that failed (§8.4)." and "`carlos instances retry ` clears one instance's failure hold.") and the line-998 table row ("§8.4: `carlos deploy --retry`, and host-scoped clearing") still hold as written. ## 6. Rulings this amendment records - **Paul, 2026-10-11 (plan 6, Q4).** Host retries would stay on the unsigned restart-scope log for now, with the trust of `carlos restart`, and signing them in the console was owed before the flag goes on anywhere hold mode is used. This amendment is that signing, as built: the console signs, and a box ignores an unsigned retry. - **Paul, 2026-10-12 (the crash-epoch hard limit).** A stale grant must never clear a later failure, including across an agent crash, so a generation is made durable before it is used and never repeats. The earlier bound (the sequence advanced by 2^20 on every open, c12c35c3) is replaced by that rule (`7c7c6141`). The cost accepted: a box whose disk is failing shows a hold with no Retry until a save lands. ## 7. What Paul is asked to sign off 1. The §8.4 and §11 text above, including the declined-box sentence (a declined box clears the hold and runs no canary). 2. That the one-host retry is a signed object while the app-wide retry stays a re-signed pointer. 3. That the hard rule, and the data-dir copy rule that goes with it (drop the `book` key), are the supported way to keep the guarantee. New member-facing strings from the console's retry pages are separate: they are in `copy-review/strings.json` on the branch and wait for the copy review.