# `carlos db` for instances whose data lives in another region **Status:** rev 7, 2026-10-23. Rev 7 (plan 3, 2026-10-23): the pull's cap and chunk size, from a measurement on dev (§5.7); the request bound is unchanged. Rev 6, 2026-10-22, changes no mechanism: it states four things the plan review (`CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md`) found the plans had to decide, each marked **(plan review N)**: what a result names when a chunk cannot be confirmed (§5.4, 9), a box's `cancel` read before it elects (§5.5 step 4, §5.8, 11), a retry's reservation (§5.7, 10), and the order of the audit row, the creates and the serial bump, with nothing but `cancel` once the request has ended, checked before each write (§5.5 step 3, §9.3, 11 and 28). Round 2 of the plan review added DBF-32 (§10) for SQL that JSON would rewrite, and the console's refusal of it (§5.5 step 3, 36). Rev 5 answers Astra's round 4 (`CODEX-FINDINGS-DB-REGION-FORWARD-R4.md`), which found J1 to J5 fixed, with four direct fixes and no new mechanism: flushed progress lines and the backhaul header bound (codex 19), the result held until its chunks are out (codex 20), restart reconciliation by refusal and a 5 s worker exit (codex 8), and the retention size wording (codex 7). Design only, nothing built. Written for Paul, who approved the direction (§2). Astra accepted rev 5 in round 5, and Paul approved building it on 2026-10-22, with the rulings in §16 (ruling 1: ship through the deployment bucket, per-region bucket as plan 5). Review brief: `CODEX-REVIEW-BRIEF-DB-REGION-FORWARD.md`. Rev 1 was `b35b3ab1`, rev 2 `2cf4a88f`, rev 3 `944a61fd`, rev 4 `c07629e4`. Rev 4 answered Astra's round 3 (`CODEX-FINDINGS-DB-REGION-FORWARD-R3.md`) by making a forwarded read one synchronous HTTP request (§5.2): no follow, no cancel route, nothing the console writes after the request ends, and the store's lifecycle rule as the only deletion. Five invariants (§5.3) replace rev 3's seven, and every finding is mapped to the invariant that closes it or to the synchronous design that removes it. Each change is marked **(codex N)** where it lands, and §18 lists what was declined. The coordinator's provisional answers to the open questions are now Paul's rulings (§16, 2026-10-22). **Companion specs:** `2026-09-02-carlos-db-design.md` (the db door, cited below as "db spec"), `2026-10-08-carlos-run-design.md` (the run transport, "run spec"), `2026-09-11-isolated-exec-design.md` (the tenant unit sandbox). ## 1. The problem `carlos db query`, `carlos db pull` and their `--at` forms restore a copy of an instance's database on the member's own machine. The CLI fetches the replica's LTX files by coordinate through the console's db door (`internal/console/apidb.go`). The console reads them from the data bucket of its own region. The console refuses every instance whose record `Region` differs from its own `CARLOS_REGION` (`resolveReplica`, `ErrResidency`, `dbResidencyRefusal`). That refusal is correct for the console: each region's box role is granted its own region's data bucket and no other (`infra/modules/region-host/main.tf`, `host_buckets`), and the console runs on the ie box. On 2026-10-21 all twelve instances of one customer account moved from ie to uk with the offline operator move. The customer can no longer run `carlos db query` against any of them. Every instance in a region other than the console's has the same gap, and self-serve moves and nearest-region placement will make that the common case. ## 2. The direction, and what it changes Paul approved this direction: the console forwards a db read to a box in the region where the data lives. That box reads its own region's replica, runs the read-only operation there, and returns only the result. The console never reads another region's bucket. There is no console per region. Two earlier rulings bend under it, and this spec says so rather than working around them: - **db spec §9, "no path that runs anything on the box".** A forwarded query runs member SQL on a box. §8 is how that stays safe. - **db spec §6 rejected "tenant database bytes in the deployment bucket".** A forwarded read has to come back to the console through the deployment bucket (§5.1). This spec puts only encrypted bytes there, sealed to a key that exists only in the member's CLI (§5.6), until the store's lifecycle rule expires them (§5.7). In phase 1 that is only a query's rows. Phase 2's pull would send the planned replica files, which are a pull's result, under ruling 1 (§16, Paul, 2026-10-22). A per-region bucket that keeps this ciphertext in the data's region is a follow-up (plan 5, §14). One earlier ruling holds unchanged: the run spec's "the console still never connects to a box". The transport is the run's. ## 3. What exists today (facts this design rests on) 1. **The CLI restores locally.** `db.go` restores the replica through `dbdoor.RestoreAsOf` over a read-only client (`internal/dbdoor/client.go`) into a private 0700 directory, checks it with `lease.CheckDB`, then either copies it to `--out` or runs one query. The query runs with `mode=ro`, `query_only`, `temp_store(2)`, `SQLITE_LIMIT_ATTACHED` 0 and the one-statement and pragma guard `checkQuerySQL`. The console never executes tenant SQL. 2. **`--at` depends on listing times.** litestream's timestamp plan keeps a file only if its `CreatedAt` is before the target. The legacy door lists S3 `LastModified`, which has second resolution, so the target is truncated to its second. The claim-mode door lists no times, and `--at` refuses there (`dbdoor.ErrNoTimes`). 3. **`carlos run` already reaches any region's boxes.** Its transport is objects in the deployment bucket under the app's subtree (`internal/runs`): `request.json`, `out/` chunks of 256 KiB, `result.json`, written create-only where it matters. The console bumps `control/serial`; boxes poll it every 2 s. The console's follow door pages chunks to the CLI, and its sweep (`internal/console/runsweep.go`) expires, closes lost runs, and deletes old ones. A run is admitted with a CAS on the instance record, because a run needs the instance to itself. 4. **The deployment bucket is writable by every platform box**, so an unsigned control object is a request any box could have written. The host-retry grant (`internal/release/hostretry.go`) answers that with a root-signed envelope: `release.SignRootEnvelope` on the console, `VerifyRootEnvelope` against `CARLOS_RELEASE_PUBKEY` on the box. Scoped release keys are refused. 5. **Sealing to a public key exists.** `internal/appconfig/crypto.go` seals with ECDH P-256, HKDF-SHA256 and AES-256-GCM, with bound associated data (the v2 envelope), using only the standard library and `golang.org/x/crypto/hkdf`, which `go.mod` already carries. 6. **Boxes publish capabilities.** `control/edges/.json` carries `capabilities` (`region-ha-1`, `offline-move-1`, `offline-stage-1`). `edgesfile.ReadLiveSnapshot` joins each file with the box's status snapshot for liveness. The offline move's doors refuse a region holding a box without the capability, and each box publishes the capability only when its armed worker can act on it. 7. **The box role can read its region's replicas.** In carlosframework/carloku-infrastructure, `region-uk.tf` passes `data_bucket_arn = aws_s3_bucket.data_uk.arn` to the `region-host` module (ref v0.21.0; ie is at v0.28.8). At both refs `host_buckets` grants `s3:GetObject` on `data_bucket_arn/*` and `s3:ListBucket` on the bucket, plus read and write on the deployment bucket, which stays in eu-west-1 and is shared by every region. Sibling regions get push-only `moves/*`. So a uk box can read uk replicas, and nothing else can. No IAM change is needed for this design. 8. **Tenant processes already run sandboxed.** The claim-bound unit template (`internal/host/claimunit_template.go`) and the `carlos-app@` units use `ProtectSystem=strict`, `TemporaryFileSystem=/data:ro`, `InaccessiblePaths=/etc/carlos`, `PrivateTmp`, `NoNewPrivileges`, an empty capability set and an `IPAddressDeny` list that covers the metadata service. `carlos-app@` sockets are bound by root systemd (`SocketUser=carlos`), so the agent can start a sandboxed unit with no privilege of its own by connecting to its socket. 9. **The agent runs as `carlos`** (`carlos-edge.service`, `User=carlos`). The console runs as `carlos-console`. ## 4. Decisions | # | Decision | Section | |---|---|---| | D1 | Phase 1 forwards `query` and `query --at`. `pull` and `pull --at` forward in phase 2, under ruling 1 (§16). The claim-mode door and the seeds `--from` door stay refused. | §6 | | D2 | Any live box in the record's region that publishes `db-read-1` and is not draining may serve. The first to create the read's `picked` marker serves it. The instance's lease holder gets no preference. | §7 | | D3 | Transport is the run's objects in the deployment bucket and a serial bump, but the read is one synchronous HTTP request: the console waits on the box's objects inside it and streams them to the CLI. The request is a root-signed envelope. No record write. Every object is a create-only write under `db-reads/`, never replaced or deleted by the protocol; the store's lifecycle rule is the only deletion (§5.2, §5.3). | §5 | | D4 | Results are sealed to an ephemeral key the CLI makes for each read. The console and the deployment bucket only ever hold the result as ciphertext. The console does see the SQL, as the envelope's signer. | §5.6 | | D5 | The box splits the work. The agent (`carlos`) verifies, resolves, plans and streams objects. Everything that decodes tenant bytes (LTX, SQLite, the member's SQL) runs in a new sandboxed unit, `carlos-dbread@.service`, as a `DynamicUser` with no network. | §8 | | D6 | The worker gets the planned LTX files and nothing else. Its scratch is its own private `/var/tmp`, which systemd removes when the unit stops. | §8.3 | | D7 | Rollout: boxes publish `db-read-1`; the console forwards only with `CARLOS_DB_FORWARD=1` and a capable region. Dormant by default. | §11 | | D8 | Three plans here. Two follow-ups are listed, not written: plan 4 moves in-region reads to the box path after phase 3 (ruling 5); plan 5 moves this traffic to a per-region bucket (ruling 1). | §14 | ## 5. Transport and authentication ### 5.1 Why this transport Three candidates exist in the tree: - **Deployment-bucket objects** (`carlos run`, restart scope, the offline move's stager and sealer reports). Established, provider-neutral, works for a box with no inbound path, and already crosses regions. - **Edge backhaul** (`internal/edge/homemap.go`, `mesh.go`). A box-to-box HTTPS hop for tenant traffic. Its HMAC key proves only "a box in this deployment" and is deliberately allowed to buy nothing but `X-Forwarded-For` belief. Using it for db reads would give that key a new authority across the IAM region boundary, and the console would start connecting to boxes, which the run spec ruled out. - **The fleet store door** (`internal/edge/storedoor.go`). Serves bucket objects to fleet boxes on the tunnel hostname. Wrong direction, wrong principal. Decision: deployment-bucket objects. The cost is latency (a pickup is seconds, not milliseconds) and a LIST per serial move on capable boxes (§5.8). Both are acceptable for a command a person runs by hand. ### 5.2 One read, one request Rounds 1 to 3 kept finding defects in one place: rev 1 to rev 3 made a read a durable object with a life of its own. It could be followed, cancelled, expired or declared lost by the console after the fact, and its admission slots outlived the request that took them. Every rule added to manage that life made a new edge (codex 3, 4, 5, 7, 13, 14, 16, 17). Rev 4 removes the life: **a forwarded read is one synchronous HTTP request.** The CLI's request to the console stays open for the whole read, the console waits on the box's objects inside that request, and when the request ends the read is over for everyone. Nothing resumes it. **Why this is possible (the numbers).** Nothing between the CLI and the console cuts a long request: - **The edge.** The public listeners set `ReadHeaderTimeout` only, and say why: "No ReadTimeout/WriteTimeout, those would cut long-lived streams" (`internal/edge/edge.go:3354` to `3358`). The reverse proxies to a unix or TCP upstream, which is how the console is reached, set no `ResponseHeaderTimeout` and no overall timeout, only a 90 s idle bound on pooled connections (`unixProxy`, `tcpProxy`, `edge.go:2377` to `2421`). - **The console.** `cmd/console/main.go:586` to `590` sets `ReadHeaderTimeout` 10 s and `ReadTimeout` 5 minutes, and no `WriteTimeout`. `ReadTimeout` bounds reading the request; the read door reads a body of a few kilobytes and then, as the import door does (`internal/console/apiinstanceimport.go:479` to `483`), sets its own read deadline through `http.ResponseController` so the connection is held for the read's whole bound and no longer. - **The CLI.** The ordinary console client has a 30 s timeout (`auth.go:403`), which is why `carlos ship` already uses its own context of 15 minutes (`apimode.go:4425`). The forwarded read does the same: its own client with no overall timeout and a context of 23 minutes. - **Streaming precedent.** The db door already streams a long response with a per-chunk write deadline and an idle watchdog (`handleAPIDBGet`, `dbStream`, `internal/console/apidb.go`). The longest a read can hold the request is the pickup window (2 minutes with skew) plus the owner's deadline (20 minutes), so 22 minutes, and the CLI's 23-minute context is the outer bound. To keep an idle connection alive through home routers and NATs that drop silent TCP, the console writes and flushes a progress line at least every 15 s (§5.5, codex 19). ### 5.3 The invariants | # | Invariant | |---|---| | **J1 One request** | A read exists for the console and the CLI only inside one open HTTP request. When the request ends, by a result, a timeout, a cancel or a dropped connection, the read is over: no route can follow, resume, cancel or end it later, and the console writes nothing about it afterwards except one best-effort `cancel`. | | **J2 Write once** | Every object is a conditional create (`If-None-Match: *`, `release.Store.Put` with an empty `ifMatch`) and is never replaced or deleted by the protocol. The console writes `request.json`, the queue entry and `cancel`; boxes write `picked`, chunks and `result.json`. | | **J3 One owner** | The first create of `picked` owns the read. A box whose create answers an error or a conflict reads `picked` back and owns the read only if it carries this box's own attempt nonce. Only the owner runs the read. | | **J4 The store's lifecycle rule is the only deletion** | A rule on the `db-reads/` prefix expires every object. It acts long after J1 has ended every reader, so it never races one. | | **J5 Admission is held by what does the work** | The console's per-person and per-console counts are held by the open request and released when its handler returns. A box's read slot and disk reservation are held until its worker unit has stopped. | What closes each open or earlier finding: | Finding | Closed by | |---|---| | codex 3 (progress ack read as the end) | removed by the synchronous design: there are no acks | | codex 4 (late writes after `lost`) | removed by the synchronous design and J1: the console never writes `lost`; a box's late chunks or result are objects nobody reads, and J4 removes them | | codex 5 (cleanup reopens the election) | J2, J4: `picked` is never deleted by the protocol, and the request is refused on age long before the lifecycle rule acts | | codex 7 (stored ciphertext) | J4, with the honest timing in §5.7 | | codex 8 (disk) | J5 | | codex 13 (completion deletes the evidence a read ended) | removed by the synchronous design: nothing needs that evidence after the request ends | | codex 14 (cleanup removes the identity to follow) | removed by the synchronous design: there is no follow | | codex 15 (the db gate on follow routes) | removed by the synchronous design: there are no follow routes. The one route runs the full gate once, at the start, and a `record-changed` result comes back in the same response | | codex 16 (lazy expiry stranded between two writes) | removed by the synchronous design (the console never writes `picked` or a result), and J3 for the box's own ambiguous `picked` | | codex 17 (abandoned reads hold admission) | J5: an abandoned read is a closed connection, and the handler's return releases its counts | | codex 18 (the forgery claim contradicts sealing) | withdrawn: §5.6 states the real trust boundary | ### 5.4 Objects All objects live under `db-reads/`, so one lifecycle rule covers them (J4). `` is `dbr_` and 26 lowercase base32 characters (128 random bits), minted the way `runs.MintID` mints run ids, with its own `ValidID`. Every object is create-only (J2). | Key | Writer | Purpose | |---|---|---| | `db-reads//request.json` | console | The request: a root envelope (§5.5). | | `db-reads/queue///-` | console | An empty pointer, written after the request. Boxes of `` list the current and previous hour's prefix and skip any entry whose `` (the request's `issued_at`) is outside the pickup window without reading it. | | `db-reads//picked` | a box | The owner (J3): box name, boot id, a random attempt nonce and the owner's deadline. | | `db-reads//out/` | the owner | One sealed chunk of output (§5.6). | | `db-reads//result.json` | the owner | How the read ended. Written after every chunk is confirmed. | | `db-reads//cancel` | console | Best effort, after the request ended without a result, so the owner stops early. Never read by the console. | Rev 2 kept these under `apps///` for app purge; one prefix is what lets one rule expire everything, and the rule is what removes them. There is no write to the instance record. A run takes a record CAS because it needs the instance to itself. A db read touches neither the instance nor its lease, needs no exclusivity, and is frequent. **Ambiguous creates on the box.** A box whose `picked` create answers an error or a conflict reads `picked` back: its own nonce means it owns the read, anything else means it steps away, and a read that still fails is treated as not owned (J3, codex 16). The box keeps each sealed chunk's bytes until its create is confirmed; a retry that answers "already exists" reads the object back and compares bytes, equal meaning its earlier attempt landed. It writes `result.json` only when every chunk is confirmed. A chunk it cannot confirm before its deadline is never named: the box stops the read and writes a `failed` result with `interrupted` whose `last_seq` is the last confirmed chunk, so the result names only objects that exist **(plan review 9)**. ### 5.5 The request and its response The envelope is unchanged from rev 3: ```json { "purpose": "carlos-db-read-v1", "v": 1, "id": "dbr_...", "account": "", "app": "shop", "host": "shop.example.com", "incarnation": "", "region": "uk", "op": "query", "sql": "SELECT count(*) FROM orders", "at": "2026-10-07T14:00:00Z", "reply_key": "", "by": "member@example.com", "issued_at": "2026-10-22T09:00:00Z" } ``` - Signed with `release.SignRootEnvelope` under the console's root key. A console that holds only a scoped key cannot sign, and so cannot forward: it answers with today's residency sentence. - `purpose` is a fixed string so the envelope cannot be confused with any other root-signed payload. The box decodes it into the exact type with `VerifyRootEnvelope`, which requires canonical JSON. - The box accepts an envelope only when `id` matches the key it was read from and the queue entry that named it, `region` matches the queue key and the box's own `CARLOS_REGION`, and `issued_at` is no more than 60 s old and no more than 60 s in the future. - `at` is empty for the newest copy. `op` is `query` in phase 1; phase 2 adds `pull`. - The envelope carries no bucket, no prefix and no key. The box derives the replica prefix itself (§8.2). **The sequence.** 1. The CLI calls the existing list door, as today. A new console with forwarding on answers a capable cross-region instance with 409 and an additive body: `{"error": DBF-14, "forward": 1, "region": "uk", "forward_ops": ["query"]}`. An old CLI prints the sentence. A new CLI switches to the forwarded form. 2. The CLI runs `checkQuerySQL` locally, makes an ephemeral P-256 key, and sends one request, `POST /api/cli/apps/{account}/{app}/db/{host}/reads`, with `op`, `sql`, `at` and `reply_key`. 3. The console runs `dbGate`'s order (person only, membership with a byte-identical 404, the db budget, configuration, the record, the class rail, pooled, fleet), refuses a moving record, checks `checkQuerySQL` again, checks capability (§7.1), and takes its admission counts (J5). Any refusal so far is an ordinary status and JSON error, and so is SQL holding U+FFFD, which is what JSON decoding makes of bytes that are not UTF-8 (the CLI refuses such SQL first, with DBF-32) **(plan review 36)**. Then it mints the id, writes the audit row (§9.3), creates the request and the queue entry, bumps the serial, and starts a `200` response of newline-delimited JSON. From here every outcome is a line in that body. The console checks that the request is still open immediately before each of those four writes, and the serial bump runs on the request's own context. A request that ends before `request.json` leaves at most the audit row; one that ends after it leaves `request.json`, perhaps the queue entry, and `cancel`, and never a serial bump or a `200` **(plan review 11, 28, 33)**. 4. Each capable box lists its region's queue for this hour and the last when the serial moves, and every 30 s if it does not. For an in-window entry it has not decided, it reads and verifies the request, skips it if `cancel` exists **(plan review 11)**, skips it while the box is draining (§7.1), checks its own admission (§5.7), waits a random 0 to 500 ms, checks draining again, and creates `picked`. 5. The owner runs §8.2's identity bracket around planning and download, runs the query in a worker, and writes sealed chunks as the output arrives. It checks for `cancel` every 5 s. When the worker ends, it confirms every chunk and creates `result.json`. 6. The console, inside the same request, reads `picked` until it appears. Then, about once a second, it reads `result.json` **first** and lists `out/` **second**, the order the run's follow door uses for the same reason (`internal/console/apirunsfollow.go:101` to `106`): a listing taken before the result could miss a chunk the box confirmed just before writing it **(codex 20)**. It writes to the response, in order: `{"admitted": }` at once, `{"picked": }` once, `{"chunk": , "data": }` for each chunk in sequence order with no gaps, `{"wait": }` whenever 15 s pass with nothing else to send, and finally `{"result": }`. - **The result line comes last, and only when complete (codex 20).** It is written only after every chunk from 1 to the result's open `last_seq` has been written to the response. A result whose chunks are not all listed yet is held, and the console keeps listing once a second until they are, or until the request's own deadline ends it with DBF-12. The console cannot open the sealed summary; the CLI still checks the bytes against it (step 8). - **Every line is flushed (codex 19).** Each line is written under a fresh write deadline and then pushed out with `http.ResponseController.Flush`, under the same deadline; a write or flush error ends the request as a stalled client (step 7). Go buffers small writes in the response writer, and `dbStream` flushes only at the end, so without an explicit flush the wait lines would sit in the console for minutes. The response's first line, `admitted`, is flushed with the status and headers, so the headers leave at once. - **Nothing downstream buffers it.** The response is `Content-Type: application/x-ndjson` with `Cache-Control: no-store` and no `Content-Length`. The edge's compressor does not gzip that type (`compressibleType`, `internal/edge/compress.go:128`), and its gzip writer passes `Flush` through when it does compress (`compress.go:319`). The edge's reverse proxy flushes a response with no `Content-Length` immediately (`httputil.ReverseProxy`), and `no-store` keeps the cache recorder out. - **Backhaul.** A request that reaches the console through another region's edge travels over backhaul, whose transport waits at most 30 s for response headers (`backhaulHeaderTimeout`, `internal/edge/edge.go:2827`, used at `2958`). So the console bounds everything before the `200` (the gate, the record read, the capability read and the two creates) with a 20 s context, refusing with today's "could not read the instance. Try again." on overrun, and flushes the `200` and `admitted` the moment the creates land. After the headers, backhaul sets no further timeout. 7. The console ends the response **without** a result line, and the CLI says DBF-3 or DBF-12, when: - no `picked` has appeared 2 minutes after `issued_at` (DBF-3); - the owner's deadline plus 1 minute passes with no result (DBF-12); - the CLI's connection closes, or a write to it stalls for 30 s. In each case the console then creates `cancel`, best effort, and returns. Its admission counts are released by the return (J5). 8. The CLI buffers every chunk (**codex 6**). On a `done` result it opens the sealed summary, checks the buffered bytes against its count and SHA-256, and only then writes the rows to stdout and today's status line to stderr. Any other result maps to its sentence (§10). A response that ends without a result prints nothing to stdout. **Ctrl-C** closes the connection. The console sees its context end, creates `cancel` and returns; the owner, if any, stops within 5 s. There is no cancel route. **A console restart** ends every read in flight: their connections close, and each CLI says DBF-12. Nothing on the console side needs recovering, because nothing outlived the request (J1). A box keeps running a read whose console went away until it sees `cancel` or its own deadline; its output is read by nobody and expires (J4). ### 5.6 Sealing, the result and the trust boundary - One ephemeral P-256 key pair per read, made by the CLI and held only in its memory. The public half travels in the request. - Each chunk is sealed with the appconfig v2 construction (ECDH P-256, HKDF-SHA256, AES-256-GCM, a fresh ephemeral sender key per seal), with its own HKDF info string, `carlos-db-read-v1`, and associated data binding the read id, the chunk's sequence number, its stream and whether it is the last chunk. A chunk moved to another read, reordered, or relabelled as last fails its tag. The result's sealed summary carries the plaintext byte count and SHA-256, so a dropped or added chunk fails the check. - **What this buys.** The deployment bucket and the console only ever hold the rows as ciphertext. Nobody without the CLI's private key can read them, including whoever holds the bucket. A replayed envelope produces output only the original CLI could open, and that CLI is gone (J1). - **What it does not buy, plainly (codex 18).** Sealing proves who can *read* an answer, not who *wrote* it: the recipient's key is public, and each seal uses a fresh sender key, so no seal carries an author. Any party that can write the deployment bucket, which is every platform box (`infra/modules/region-host/main.tf`, `host_buckets`), could create `picked`, chunks and a matching `result.json` for a read before the real box does, and the CLI would accept that answer as consistent. The CLI's checks prove the answer is complete and unaltered since it was sealed; they do not prove which box sealed it. Rev 3's claim that a non-owner "cannot change an answer the CLI shows" was wrong and is withdrawn. This is the same trust every box already has over `carlos run` output and over everything else in the deployment bucket, and the in-region door trusts the bucket the same way. Authenticated authorship would need per-box signing keys, which exist only in claim mode; §6 leaves claim mode alone. ```json { "state": "done", "code": "", "box": "carlos-flagship-uk", "last_seq": 12, "started": "...", "ended": "...", "sealed": "" } ``` `state` is `done`, `failed`, `refused` or `cancelled`. `code` is a fixed vocabulary the console may read: `nothing-replicated`, `history-gone`, `check-failed`, `changed-twice`, `too-large`, `no-room`, `sql-error`, `timeout`, `output-cap`, `record-changed`, `moving`, `interrupted`. The sealed summary carries the transaction id and time of the copy, the oldest time for `history-gone`, the restored size and cap for `too-large`, SQLite's own error text for `sql-error`, and the byte count and SHA-256 of the output. The CLI maps codes to today's sentences where one exists (`dbRestoreError`), and to §10's rows otherwise. ### 5.7 Limits and storage | Limit | Value | Enforced by | |---|---|---| | Pickup window | 60 s from `issued_at`, plus 60 s of skew | box (refuses older envelopes); console stops waiting after 2 minutes | | Planning and restore | 10 min | worker unit and agent context | | Query | 5 min wall clock | worker | | Owner deadline | 20 min from `picked`, stated in it | agent context, worker `RuntimeMaxSec`; console stops waiting 1 minute after it | | The request | at most 22 minutes; the CLI's own context is 23 | console, CLI | | Restored size | 2 GiB by default, `CARLOS_DB_READ_MAX_BYTES` on the box | agent, from the plan, before any byte moves | | Disk | a reservation per read (below) | agent | | Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` | agent; the CLI's buffer is bounded by the same number | | Chunk | 256 KiB or 1 s, whichever comes first (`runs.ChunkBytes`, `runs.ChunkEvery`) | agent | | Pull | at most 2 GiB of planned LTX files (measured on dev, 2026-10-23: relay rate 3.94 MB/s, the slowest of three runs; the rule gives 3.25 GiB, above the 2 GiB ceiling); `CARLOS_DB_PULL_MAX_BYTES` on a box can only lower it; the read ends `too-large` | agent, from the plan, before any byte moves; the CLI's spool is bounded by the same number | | Pull chunk | 1 MiB or 1 s, whichever comes first | agent | | Reads per box | 2, plus one connection kept for the readiness probe (`MaxConnections=3`) | agent, systemd | | Open reads per person | 2 | console, held by the open request (J5) | | Open reads per console | 16 | console, held by the open request, 429 with `Retry-After` | | Request budget | the db door's existing 600 per token per minute | console | **Admission (codex 17).** The console's counts are taken in step 3 and released by a `defer` in the same handler, so they are exactly the reads whose requests are open. An abandoned read is a closed connection; its handler returns within 30 s (the write stall bound) or at its next wait line, and the slot is free. No timer, sweep or restart is involved. **Stored ciphertext (codex 7).** The protocol deletes nothing. Every object of a read stays in the deployment bucket until the lifecycle rule removes it, and that timing is the store's, not ours. On S3 the rule (current versions expire 1 day after creation, noncurrent versions 1 day after they become noncurrent, and expired delete markers removed) makes an object eligible for expiry about a day after creation, with eligibility rounded to the next midnight UTC, and the noncurrent version eligible a day after that. S3 then removes eligible objects asynchronously, usually within a day or two, with no hard deadline. So the honest statement is: **a read's ciphertext lives until lifecycle expiry, typically two to four days after the read, with no guaranteed upper bound.** The size per read is the plaintext limit plus sealing overhead **(codex 7)**: the output cap limits the rows to 64 MiB of plaintext in phase 1; each chunk object stores its sealed envelope as raw bytes (not base64), which adds under 100 bytes per 256 KiB chunk (a 65-byte sender key, a 12-byte nonce, a 16-byte tag and a short prefix), so a full answer stores about 64.03 MiB of ciphertext. The request, `picked`, `result.json`, the queue entry and `cancel` add a few kilobytes. Base64 appears only on the wire to the CLI. Plaintext rows are never in the bucket. A deployment that needs a hard deadline needs a different reclamation mechanism (version-aware deletes), which this design does not have. **Disk reservations (codex 8).** The agent is the only process that starts workers, so it keeps one reservation table under one mutex. Before `picked` it reserves the floor (5 GiB); after planning it grows that reservation to the plan's LTX bytes plus its restored size. Each step succeeds only if the filesystem holding `/var/tmp` keeps at least 5 GiB and 10% free after subtracting every other open reservation; otherwise the read ends `no-room`. A reservation is released only when its worker **unit** has stopped, since systemd removes the unit's private `/var/tmp` at stop. The agent learns that from `systemctl list-units 'carlos-dbread@*'` (readable without privilege). It admits a new read, or a probe, only when the number of active, activating or deactivating `carlos-dbread@` units is no more than the number of workers it is itself running. Any other unit is an orphan whose scratch is not in the table, so the agent refuses work until it is gone. There is no timeout escape: an orphan worker stops within 5 s of losing its connection (§8.3), and `RuntimeMaxSec` stops it in the end regardless. A read's one retry (a planned file that vanished, §8.2) is a second worker. Its first attempt's reservation stays in the table, under its own key, until that attempt's unit has stopped, and the retry dials only once the unit count says so; the retry takes a fresh reservation from the floor. Closing the first connection is not the unit's stop **(plan review 10)**. **Restart reconciliation (codex 8).** A restarted agent's table is empty, and it cannot know the sizes of workers it did not start. So it reconciles by refusal, against the units actually running: at start, and before every admission, it lists `carlos-dbread@` units, and while any is active, activating or deactivating that it is not itself running, it takes no read and no probe. It never guesses a size for an orphan and never admits beside one. Its first admission therefore happens only after every unit from before the restart has stopped and systemd has removed its scratch. ### 5.8 Cost Each capable box does one LIST of its region's current and previous hour of queue entries per serial move and every 30 s, and two GETs per new in-window entry (the request, and `cancel` before electing, plan review 11). A read adds two console PUTs, one `picked`, one PUT per chunk and one result, and the console's own GETs and LISTs about once a second while the request is open. There are no deletes. The October bill was LIST and GET polling, so plan 1 measures the LIST rate on dev before the flagship roll and reports it. ### 5.9 Replay and forgery | Attack | What stops it | |---|---| | A box writes its own request | No root signature: refused and logged once. | | A box writes its own queue entry for a real request | The box reads the request and requires the queue entry's region, hour and time to match the envelope. | | An envelope copied to another id or region | The id and region are inside the signature and must match the keys. | | The same envelope picked again | `picked` exists until lifecycle expiry, days after the 60 s window has closed (J2, J4); a second create fails on every box, before or after a restart **(codex 5)**. Past the window the envelope is refused on age. | | A box serves a read for another region | It refuses any envelope whose region is not its own `CARLOS_REGION`, and IAM refuses the bucket anyway. | | A request that names a bucket or prefix | There is no such field. The box derives the prefix (§8.2). | | A compromised box forges an answer | Not prevented. §5.6 states the boundary **(codex 18)**. | ## 6. Which operations forward | Operation | Phase | Reason | |---|---|---| | `query` | 1 | This is the regression the customer hit. The result is rows, usually small, and only rows leave the region. | | `query --at` | 1 | Same path with a target time. The box lists the replica the way the console's door does today (`listReplicaLevel`, `LastModified` as `CreatedAt`), so the copy stops at the same transaction an in-region read would. Plan 1 tests that parity. | | `pull` | 2 | The result is the whole database. It rests on ruling 1 (§16, Paul, 2026-10-22: a whole copy's ciphertext rests in the deployment bucket's region until lifecycle expiry, typically days, until plan 5's per-region bucket), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed, in the same single request, and the CLI restores and checks them locally with today's code. The box never restores for a pull. | | `pull --at` | 2 | As `pull`, with the target time. | | Claim-mode door (`apidbclaim.go`) | not planned | Claims are not active anywhere. The claim door reads exact object versions through `hostops`'s versioned boundary and signed file handles, and lists no times. A box-side version of that belongs to claim activation (#250), which must add times anyway. A host-claim console keeps today's residency refusal, as it refuses `carlos run`. | | Seeds `--from` (`resolveDBSource`) | not planned | It shares the source checks but restores into a new instance, a different flow. Cross-region seeding is its own design. | | Fleet and pooled instances | refused, as today | Their data is not in a platform region bucket. | Phase 1 reaches every cross-region instance a member can read today in its own region, for queries. Until phase 2, `pull` answers DBF-8 for a cross-region instance. ## 7. Who serves ### 7.1 Capability and selection A box publishes `db-read-1` in its edges file only when the agent's armed db-reads loop says so, the way `offline-move-1` comes from the armed moves worker (`armedOfflineView`). The loop answers yes only when all of these hold: - hibernation is configured (`CARLOS_HIBERNATE_BUCKET` set); - `CARLOS_REGION` is set; - `CARLOS_RELEASE_PUBKEY` is set, so it can verify envelopes; - `CARLOS_DB_READS` is not `0` (an operator off switch for one box); - `CARLOS_HOST_CLAIM` is not set; - the loop is running, and its last **readiness probe** passed **(codex 9)**. The readiness probe connects to the worker socket and runs the worker protocol's `hello` exchange, which starts a real `carlos-dbread@` instance and gets back the worker's protocol version and its view of its own sandbox (no network interface but loopback, `/data` empty, `/etc/carlos` unreadable). The loop probes at start and every 5 minutes, and after any read whose worker failed to start. A failed probe withdraws the capability on the next edges-file pass. A socket file whose service is missing, masked, broken or the wrong version therefore never advertises anything. **A probe never competes with reads (codex 9).** The socket allows three connections (`MaxConnections=3`): the agent runs at most two reads and at most one probe, so a probe always has a connection and a busy box is never mistaken for a broken one. The agent counts its own connections, so it never opens a fourth. A probe is skipped, not failed, while orphan workers block admission (§5.7); the last result stands until it can run. When a read's worker fails to start, the agent probes as soon as that connection is closed. Rev 2 shared two connections between reads and probes, so two reads at once made the probe fail and withdrew the capability. The console reads `edgesfile.ReadLiveSnapshot` before admitting a read and refuses up front only on complete evidence **(codex 10)**. That function returns a nil error when status snapshots cannot be loaded and reports the gap through `Complete` and `BoxesComplete` instead, so the rule reads those fields, never the error alone. Incomplete evidence comes first and wins over both refusals: 1. `Complete` is false, or `BoxesComplete` is false (an unreadable edges file, an unreadable or missing status snapshot, a fresh status snapshot with no edges file): the request is written and the pickup window decides. A person is waiting, and the console stops waiting after 2 minutes (DBF-3). 2. Both are true, and no file in the record's region publishes `db-read-1` on a non-draining box: DBF-1, no request written. 3. Both are true, capable non-draining boxes exist, and none is `Fresh`: DBF-2, no request written. Rev 2 checked `Complete` alone before DBF-1, so a fresh box whose edges file was missing could be refused as "no server can read databases". Any capable, live, non-draining box in the region may serve. There is no assignment by the console. Region HA (plan 2b, dormant) can put several boxes in a region, and they all read the same region bucket, so the create-only `picked` marker is the whole election. The 0 to 500 ms jitter spreads the race, and a box that fails admission (§5.7: no free read, the disk floor, or an orphan worker) does not enter it. ### 7.2 Why not the lease holder A sleeping instance holds no lease, so there is no holder to ask. An awake instance's holder is busy serving it, and the read does not touch the instance, so there is nothing to gain by sending it there. Choosing by lease would also tie the read to the instance's lifecycle, which is exactly what "never the live database" rules out. ### 7.3 When no box serves | Situation | Answer | |---|---| | No capable box in the region | DBF-1 at once | | Capable boxes, none live | DBF-2 at once | | Live but nobody picks in the window (all busy, all failing admission, agent stuck) | DBF-3, when the console stops waiting 2 minutes after the request | | The owning box dies or restarts mid-read | DBF-12, when the console stops waiting 1 minute after the owner's deadline, up to 22 minutes after the request (§5.5) | ## 8. Read-only and isolation on the box ### 8.1 The split | Half | Runs as | Does | Never does | |---|---|---|---| | Agent (`carlos agent`, `carlos-edge.service`) | `carlos` | lists requests; verifies envelopes; creates `picked`; reads the record; derives the prefix; lists the replica and plans the restore from the listing; reads the last planned file's 100-byte header for the size check; streams planned files from the region bucket into the worker; re-checks the record before the query runs; seals and writes output; writes the result; keeps the disk reservation table | opens SQLite, decodes LTX pages, runs SQL, writes files on disk, reads anything under `/data/instances`, takes or reads the lease | | Worker (`carlos system db-worker`, `carlos-dbread@.service`) | a `DynamicUser` | receives the plan and the files over its socket; writes them into its private `/var/tmp` as a litestream file replica; restores to the agent's target transaction; runs `lease.CheckDBContext`; runs the one statement with today's guards; streams rows back | anything with a network, the box's files, the bucket, other tenants' files | The rule is: everything that decodes page data or SQL runs in the worker. The member's SQL, the replica's pages and the restored database all do. The agent does object input and output and planning from listings, which it already does to wake instances. Its one decode is the last planned file's fixed 100-byte LTX header, for the size check, with the bounds `dbdoor.PlanRestore` already applies (a valid page size, a non-zero commit, TXIDs that match the listing). ### 8.2 The identity bracket The replica prefix is keyed by host alone (`activator.HibernatePrefix`), so a host deleted and recreated, in the same account or another, reuses it. Checking the record once before the read would let a request admitted for the old instance read the new one's replica **(codex 1)**. So the agent brackets every byte it reads with two identical checks, and no output exists until the second one passes. **The check.** The agent reads the record fresh from the bucket and refuses (result `refused`) unless: - the record exists and its incarnation is the envelope's (`record-changed`); - the class rail passes (`dbClassRefusal`, moved where both console and box can call it); - it is not pooled and not a fleet placement; - `rec.Region` equals the box's `CARLOS_REGION` (`record-changed`); - `rec.Move` is nil (`moving`); - `rec.Host` is the envelope's host and passes `provision.ValidHostShape`. **The order.** 1. Check A, then plan from the listing. 2. Stream every planned file into the worker. The worker restores and runs `lease.CheckDBContext`, then answers `ready` with the transaction it holds. It runs no SQL yet. 3. Check B, the same check again, after the last byte has been read from the bucket. 4. Only if check B passes does the agent tell the worker `run`. The first output chunk can exist only after this point. A failed check B ends the read `record-changed`, stops the worker and writes no chunk. A retried plan (today's "changed while it was being read" retry) runs both checks again around its own download. **Why two checks are enough.** The incarnation is `hostclaim.Incarnation(account, app, host, CreatedAt)`. A deleted and recreated host, including one recreated in another account, gets a new incarnation, and an incarnation never comes back once its record is gone. So a record that shows the same incarnation at A and at B was the same instance for the whole download, and the files read in between were that instance's. Pinning object versions would add nothing that this does not already give in non-claim mode (§18). The replica prefix is `activator.HibernatePrefix(, rec.Host)` in the box's own `CARLOS_HIBERNATE_BUCKET`, read through `lease.ReadReplicaClient` and held only as the narrow `replicaReader` interface (`LTXFiles`, `OpenLTXFile`), so no write method is reachable. Nothing in the request chooses a bucket, a prefix, a key or a level beyond what the plan picks. The agent lists only level directories. It never reads `lock.json`, never calls the leaser, and never wakes the instance. ### 8.3 The worker unit `carlos-dbread.socket`, bound by root systemd: - `ListenStream=/run/carlos-dbread/worker.sock`, `SocketUser=carlos`, `SocketMode=0600`, `Accept=yes`, `MaxConnections=3` (two reads and one probe, §7.1, codex 9). `carlos-dbread@.service`, one instance per connection, starting from the claim-bound template's hardening (§3 fact 8) with these differences: - `StandardInput=socket` and `StandardOutput=socket`, so the accepted connection is the worker's stdin and stdout, and `StandardError=journal` for its own diagnostics **(codex 2)**. With `Accept=yes` systemd passes the connection as a file descriptor and does not put it on stdin unless told to; the claim-bound template's `StandardOutput=file:` line is not copied. All protocol traffic, both ways, is on that one connection; - `DynamicUser=yes` instead of `User=carlos`, so the worker owns no file on the box and can read nothing `carlos` owns; - `PrivateNetwork=yes` and `RestrictAddressFamilies=AF_UNIX`, so no route to the metadata service, the bucket or anything else exists, rather than a deny list; - `TemporaryFileSystem=/data:ro /var/lib:ro /opt/carlos:ro /run/carlos:ro` with `BindReadOnlyPaths=/opt/carlos/carlos`, and `InaccessiblePaths=/etc/carlos /run/carlos-control`; - `PrivateTmp=yes`: its `/tmp` and `/var/tmp` are its own. The worker writes only under `/var/tmp`, which is on disk on Debian 13, where `/tmp` is a tmpfs. systemd deletes both when the unit stops, however it stops; - `MemoryMax=512M`, `CPUQuota=100%`, `TasksMax=16`, `LimitFSIZE` set to the restored-size cap (it bounds each file; the agent's reservation in §5.7 bounds the total, codex 8), `RuntimeMaxSec=20min`; - `SystemCallFilter=@system-service`, `NoNewPrivileges=yes`, empty `CapabilityBoundingSet`, `ProtectProc=invisible`. `carlos system db-worker` refuses to run unless its stdin is a connected unix stream socket (checked with `getsockopt(SO_TYPE)` and `SO_PEERCRED`, whose peer must be the `carlos` user), so it cannot be started by hand or pointed at a file. Plan 1's acceptance test drives it only through the installed units. The worker protocol is length-framed messages with a fixed maximum frame size: `hello` (the readiness probe, §7.1), a header naming the plan's files, their sizes, the target transaction and the byte limits, then the files, then `ready` from the worker, then `run` or `stop` from the agent, then framed output and a final status. The worker refuses a file larger than its header said and any write past the header's totals. **The worker stops when its agent goes (codex 8).** From the moment it starts, the worker keeps one goroutine reading its connection. EOF or an error there cancels the one context that restore, `lease.CheckDBContext` (`internal/lease/check.go:44`, never `CheckDB`, which supplies a background context) and the query all run under, and the worker exits. Not every step honours a context promptly: litestream's file replica client opens a plain file without checking it (`litestream@v0.5.14/file/replica_client.go:135`), and the LTX compactor's page reads and writes ignore it (`ltx@v0.5.1/compactor.go:177`, `199`). So the watcher does not rely on cancellation alone: after cancelling, it waits 5 s for the worker to finish, and if it has not, it ends the process itself with `os.Exit` **(codex 8)**. A process exit stops a compaction wherever it is. A worker blocked in a write to a dead connection gets an error at once. So an agent restart ends every worker it had within seconds, systemd stops each unit and removes its private `/var/tmp`, and only then does the restarted agent's orphan count (§5.7) reach zero and admit work. While the agent reads the output, it writes nothing to the worker, so the watcher sees only EOF or the agent's own `stop`. The socket and service files are written and enabled by `ensureSystemUnits`, the same root path that installs the other units, on the next adopt pass or `update-platform-binary.sh`. Within the worker, the query keeps every guard the CLI has: `mode=ro`, `query_only`, `temp_store(2)`, `SQLITE_LIMIT_ATTACHED` 0 and `checkQuerySQL`. The database is opened with `immutable=1` as well, since nothing else ever writes the file. The guards and the query function move from `db.go` into a shared package (`internal/dbquery`), so the CLI and the worker run the same code and the output format cannot drift. ### 8.4 Why a tenant cannot reach another tenant's data 1. The console admits only a member of the account, for an instance of that account's app, and signs exactly that host into the envelope. 2. The box derives the prefix from that host, the fresh record and its own config, and nothing else. 3. The worker receives only the files the agent sent. It has no network, so no credentials from the metadata service and no bucket. Its filesystem view hides `/data`, `/etc/carlos` and `/run/carlos`, and its user owns nothing. ATTACH is off. If a SQLite flaw let the statement open a path, the sandbox leaves it nothing to open but its own scratch. 4. The output is sealed to the requesting CLI's key. ### 8.5 Why the live database is never touched The agent reads bucket objects only. The worker cannot see `/data`. Nothing calls the activator, the leaser or `lock.json`. A read of a busy instance's replica can race litestream's compaction, which is today's "changed while it was being read, twice" case, and is retried the same way. ## 9. Console and CLI behaviour ### 9.1 The one new console route ``` POST /api/cli/apps/{account}/{app}/db/{host}/reads ``` It is `svcrole.Set{}` (people only) and runs the db door's full gate once, at the start (§5.5 step 3). A host-claim console answers it with today's residency sentence. Rev 3's follow and cancel routes are gone (J1), and so is rev 2's acknowledgement pair. Rev 3 needed a second, smaller gate for follow calls so that a `record-changed` result could still be fetched after the instance was deleted or reclassified (codex 15). With one request there is nothing to fetch later: the full gate ran when the read began, and the box's `record-changed` result arrives as the last line of the same response. ### 9.2 Mixed versions | CLI | Console | What happens | |---|---|---| | old | old | today's residency sentence | | old | new, forwarding off or region not capable | today's sentence, or DBF-1 or DBF-2 | | old | new, forwarding on and region capable | DBF-14: update carlos | | new | old | today's sentence; no forward marker, so the CLI prints it as is | | new | new | the forwarded read | A console with `CARLOS_DB_FORWARD` unset answers byte-identically to main. ### 9.3 Audit One `db-read` activity row at admission, in the run's manner, with the detail DBF-21 or DBF-22. It is written before the request and the queue entry, while the request is open, so a read a box can see always has its row; a row the console cannot write refuses the read. A request that ends after the row and before the `200` leaves the row of an admission, and at most the request, the queue entry and `cancel`; never a serial bump **(plan review 11)**. It records the SQL's length in bytes and the first 16 hex digits of its SHA-256, not its text (ruling 3). A member who kept the statement can match it to the row; the row itself holds none of the data a query's literals can carry. That differs from `carlos run`, which records argv, and keeps the in-region rule that SQL never reaches the audit log. The SQL itself stays in the request object, in the deployment bucket, until the lifecycle rule expires it with the rest of the read, on the timing §5.7 states: typically two to four days, with no hard bound. ## 10. Errors and copy Every new user-facing sentence is below. Error and status strings follow Paul's voice rule (ruling 4): lower case at the start and after a full stop, no em dashes. Docs prose keeps the CLI page's own sentence case. None of this is approved yet; the copy-review gate runs before any of it is written into code. | ID | Where | Text | |---|---|---| | DBF-1 | console 409, no capable box in the region | `%s keeps its data in region %s. no server in that region can read databases yet, so carlos db cannot read it for now.` | | DBF-2 | console 409, capable boxes but none live | `%s keeps its data in region %s. no server in that region is answering right now. try again in a few minutes.` | | DBF-3 | CLI, nobody picked the read in 60 s | `no server in region %s picked up the read within 60 seconds. try again in a few minutes.` | | DBF-4 | CLI, `too-large` | `%s's database is %s once restored. carlos db can query at most %s in region %s.` | | DBF-5 | CLI, `no-room` | `the server in region %s does not have room to restore %s's database right now. try again later.` | | DBF-6 | CLI stderr, `output-cap` | `the result was cut off at 64 MB. narrow the query with WHERE or LIMIT and run it again.` | | DBF-7 | CLI, `timeout` | `the query ran for 5 minutes and was stopped. narrow it and run it again.` | | DBF-8 | CLI, `pull` before phase 2 | `%s keeps its data in region %s. carlos db pull cannot fetch a copy from there yet. carlos db query can read it.` | | DBF-9 | console 429, open-read bounds | `too many database reads are open. wait a minute and try again.` | | DBF-10 | console 409, record moving | `%s is moving between regions. read its database once the move has finished.` | | DBF-11 | CLI stderr while waiting | `reading a copy of %s's database in region %s.` | | DBF-12 | CLI, the response ended without a result (the box stopped or never finished, the console restarted, the connection dropped), or `interrupted` | `the read of %s's database in region %s did not finish, so nothing was printed. try again.` | | DBF-13 | CLI, a chunk or summary fails to open or add up | `the answer from region %s did not check out, so carlos db showed none of it. try again.` | | DBF-14 | console 409 with the forward marker, read by an old CLI | `%s keeps its data in region %s. this version of carlos cannot read it from there. run carlos update, then try again.` | | DBF-15 | CLI, `record-changed` | `%s changed while the read was waiting. try again.` | | DBF-16 | CLI, `moving` from the box | as DBF-10 | | DBF-21 | audit detail, query | `forwarded a database query to region %s. the query was %d bytes, sha256 %s (cli)` | | DBF-22 | audit detail, query with `--at` | `forwarded a database query to region %s, as of %s. the query was %d bytes, sha256 %s (cli)` | | DBF-30 | `docs/site/cli.md`, `carlos db`, replacing the paragraph that starts "If the app has more than one instance" | see below | | DBF-32 | CLI, a query that is not UTF-8 text or holds U+FFFD, refused before it is sent: JSON would rewrite it, and the box must run the bytes typed or none **(plan review 36)** | `this query cannot be sent to region %s, because it holds bytes that are not UTF-8 text or the character U+FFFD. fix the query and run it again.` | | DBF-31 | `docs/site/cli.md`, `carlos system`, the worker | `carlos system db-worker runs on platform boxes only. systemd starts it for each forwarded database query. you never run it yourself.` | DBF-13 is true as written only because the CLI buffers the whole answer and checks it before writing anything (§5.5, codex 6). DBF-30: > If the app has more than one instance, name one with `--instance`. Only > instances that sleep have a replica. An instance on your own fleet or in a > pool cannot be read this way, and the command says which. > > When an instance keeps its data in another region than the console's, > `carlos db query` runs in that region. A server there restores the copy, > runs your query and sends back only the rows. The rows are encrypted for > your terminal on the way, so nothing in between can read them, and > nothing is printed until all of them have arrived and checked out. There, > the copy can be at most 2 GB once restored, the query runs for at most 5 > minutes, and the rows can come to at most 64 MB. `carlos db pull` cannot > fetch a copy from another region yet. Unchanged and reused: today's status line (`dbQueryStatus`), the `dbRestoreError` sentences for `nothing-replicated`, `history-gone`, `check-failed` and `changed-twice`, SQLite's own error text for `sql-error`, `stopped` for Ctrl-C, the guard refusals of `checkQuerySQL`, and the residency sentence when forwarding is off. DBF-17 to DBF-20 and DBF-23 to DBF-29 are left free for phase 2's pull rows. ## 11. Rollout ### 11.1 Switches and gates - **Box:** `db-read-1` (§7.1). Published only while the armed loop's readiness probe passes. `CARLOS_DB_READS=0` withdraws one box. - **Console:** `CARLOS_DB_FORWARD=1` turns forwarding on. Unset, every answer is byte-identical to main. The console also needs the root key. - **Store:** the `db-reads/` lifecycle rule (J4) is in place on the deployment bucket before any console turns forwarding on. On the flagship that is a change to the `deployment-buckets` module, a release tag, a ref bump in carloku-infrastructure and a targeted apply. That is production infrastructure, so it needs Paul's scoped approval under AGENTS.md's rule before it is applied; plan 2 prepares the plan for that approval. A self-hosted deployment sets the same rule (MinIO calls it ILM); without it, `db-reads/` objects are never removed. - **CLI:** none. The forward marker negotiates (§9.2). - **Version floor:** none beyond the capability. A box that predates it publishes no capability and is never asked. ### 11.2 Mixed fleet A region with one capable box and one old box works: only the capable box lists the queue. A region with no capable box gets DBF-1. A box that loses the capability mid-read finishes the read it holds, within its deadline, and takes no new one. ### 11.3 Roll order Paul's order (dev boxes, dev console, check, flagship boxes, flagship console), with the lifecycle rule first on each deployment's bucket. Boxes publish an idle capability. The dev console turns `CARLOS_DB_FORWARD=1` and the check is a forwarded query from a dev console to a dev box in another region, plus the LIST-rate measurement (§5.8) and a note of real database sizes (ruling 2). ### 11.4 Tests Acceptance is production-wired. Stand-ins are allowed in unit tests and labelled so; they are not acceptance. | # | Test | Wiring | |---|---|---| | T1 | A forwarded query end to end: console in region ie, agent in region uk, two MinIO buckets on one server with separate credentials. The console's credentials cannot read the uk bucket (a MinIO policy), so any console read of it fails the test. | real console server, real agent loop, real store; CI runs it with the worker reached through a test socket, labelled non-acceptance | | T2 | The same on the root runner with the real `carlos-dbread.socket` and `carlos-dbread@.service` installed by the real `ensureSystemUnits`, so the socket is the worker's stdin and stdout as §8.3 says (codex 2) | acceptance gate for plan 1 | | T3 | Sandbox witness on the root runner, from inside the worker: open `/data/registry.db`, `/etc/carlos/host.env` and another instance's directory; connect to `169.254.169.254:80` and to any TCP address; write outside `/var/tmp`; `ATTACH` and `VACUUM INTO` with the attach limit removed (a mutation); `load_extension`. Every one fails. The private tmp is gone after a normal end, a cancel and `kill -9` of the agent mid-read. | real units | | T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, a forged queue entry, the same envelope offered again before and after the agent restarts (`picked` holds, codex 5), a chunk sealed to another key, a chunk reordered, the last chunk dropped. And the boundary §5.6 states, as a documented negative: a writer that creates `picked`, chunks and a matching result first is accepted by the CLI (codex 18). | real verify and seal code, real store | | T5 | Two capable boxes in one region: exactly one runs each read, across 1,000 reads with random jitter. A box whose `picked` create commits on the server but answers an error reads it back and runs the read; a box whose create answers an error when another box owns it steps away (codex 16). | real store conditional create, a store wrapper that loses answers | | T6 | Parity: the forwarded query and the in-region query print byte-identical stdout for a fixture database (NULL, tabs, newlines, blobs, times). `--at` stops at the same transaction as the in-region door for the same replica and time, including a write in the same second as the target. | real restore, both paths | | T7 | The long request through the real path: a read that holds the request for 21 minutes passes through the real edge and the real console server (with its 5-minute `ReadTimeout`) and returns its result; a CLI context of 23 minutes is the outer bound. During a quiet restore the real CLI receives the `admitted` line within a second and a wait line at least every 15 s, both through the local edge and through backhaul from another edge, whose 30 s header timeout the request survives (codex 19). | real edges, real console, real CLI binary | | T8 | Mixed versions: the five rows of §9.2, with the forwarding-off 409 compared byte for byte against main. | real console | | T9 | Record changes between request and pick, during planning, during download and between download and `run`: moved, recreated in the same account, recreated in another account, class changed. Each ends with its code, the code reaches the CLI in the same response (codex 15), and no chunk is ever written (codex 1). | real records, a store wrapper that pauses the download | | T10 | Approved copy digests for every row in §10, in the `approved_copy_test.go` pattern. | | | T11 | CLI buffering: a dropped or bad last chunk, a summary hash mismatch, and a response that ends without a result each leave stdout empty (codex 6). | real CLI binary through `carlosTestEnv` | | T12 | Disk: two concurrent reads on one box whose free space fits one of them: one runs, the other ends `no-room`. An agent killed mid-read: the restarted agent takes no read and no probe while the old worker unit is active, its worker stops on EOF, and the restarted agent admits work only once the unit and its scratch are gone, without guessing a size for it. A disconnect during local compaction, after the download, still ends the worker within 5 s (codex 8). | real agent, real units, root runner | | T13 | Readiness: a present socket with the service missing, masked, failing to start and speaking another protocol version each withdraws `db-read-1` within one probe and edges pass. A box running two reads keeps the capability, and its probe succeeds on the third connection (codex 9). | real units, root runner | | T14 | Console evidence: status snapshots unreadable, one snapshot missing, an edges file unreadable, an empty edges listing beside a fresh status snapshot. Each admits the read; only complete evidence (both flags) gives DBF-1 or DBF-2 (codex 10). | real `ReadLiveSnapshot` over a real store | | T15 | Admission (codex 17): two reads whose CLIs disconnect before pickup, and two whose CLIs disconnect after it, each free their person's and the console's count when the handler returns, without a console restart; a third read from the same person is then admitted. A console restart mid-read leaves nothing to recover. | real console | | T16 | Late writes are harmless (codex 4, 13, 14): a box that keeps writing chunks and its result after the console's request has ended (by timeout and by disconnect) changes nothing any reader sees, and no protocol path issues a delete or a replacing write (a store wrapper fails the test on any `Delete` or non-create `Put` under `db-reads/`). | real console, real agent, a versioned MinIO bucket | | T17 | The lifecycle rule: the module's rule covers `db-reads/` with current and noncurrent expiry and expired delete markers (a module test). | module test | | T18 | The result overtaking the last chunk: a store wrapper that lets the box commit its last chunk and `result.json` between the console's listing and its result read, in both orders. The console never writes the result line before every chunk through `last_seq`, and the read succeeds (codex 20). | real console, real store | Mutation checks that must turn a test red: drop envelope verification (T4); take the host or prefix from anywhere but the record and config (T1 with a second tenant); remove `PrivateNetwork` (T3); skip the summary hash check (T11); skip check B (T9); add any protocol delete or replacing write (T16); release the console's counts anywhere but the handler's return (T15); skip the read-back of an ambiguous `picked` (T5); drop the orphan count (T12); give probes no connection of their own (T13); remove the read-deadline extension or any flush (T7); list chunks before reading the result, or write the result line before its last chunk (T18); drop the worker's 5 s exit (T12). ## 12. Provider boundary - **Core:** the transport is `release.Store` objects, so it runs on MinIO and any S3-compatible store. Every protocol write is a conditional create, which `release.Store.Put` already supports with an empty `ifMatch`, on S3 and on MinIO. The box reads each planned file with a GET and writes chunks with PUTs; it never uses a server-side copy. Signing is ed25519, sealing is the standard library plus `x/crypto/hkdf`. The sandbox is systemd, which every box already needs. - **One store endpoint per box (codex 11).** Every S3 client in a box process takes its endpoint from the one `CARLOS_S3_ENDPOINT` (`internal/s3compat/s3compat.go`), the deployment store through `s3compat.Options` and the replica client through `lease`'s `s3Endpoint`. So a box reaches its region's data bucket and the deployment bucket on the same server, as it already must to wake and to move instances. This design supports exactly that shape: separate buckets per region, with separate credentials, on one endpoint (§18). - **Expiry.** The lifecycle rule is S3 lifecycle configuration, which MinIO supports as ILM. It is a deployment requirement (§11.1), not something the code can detect, and there is no fallback deleter: the store's rule is the only deletion (J4). - **AWS adapter:** IAM is what makes the console unable to read another region's bucket on the flagship. A self-hosted deployment gets the same split from per-bucket credentials on its store. - **What a non-AWS deployment loses:** nothing it has today. The capability, the liveness join and the box's identity use no AWS API. Multi-box regions are AWS-only today (H26), but nothing here depends on them; one box per region is enough. - **Build:** the worker and its socket code are Linux-only, with `//go:build linux` and a `!linux` stub, so the CLI still cross-compiles for darwin and windows. ## 13. Census **Plan 14 root-read census (codex 12).** The scanner reads every production package and matches `Open`, `QueryContext` and the rest whatever user runs them (`internal/provision/plan14_root_reads_test.go:33`). Its only classes are `authority`, `hint`, `diagnostic` and `legacy` (line 478), and `legacy` needs `accepted-legacy;rev3=` as its basis. "Runs as a `DynamicUser`" is a fact for the ruling's footing, not a class. So each plan proposes, for a ruling on #280: | Declaration | Callers and identity | Proposed class and basis | |---|---|---| | `db.go` `runQueryContext` (main: `legacy`, `rev3=J4`, `basis=root-input`, `plan14-root-reads.tsv:202`) | moved out of `db.go` | retired through the overlay's removal row, the form the H3 carlos-run removal used | | `internal/dbquery` `RunContext` (the moved body, unchanged) | the CLI on a member's machine, as the member; the worker on a box, as its `DynamicUser`. Neither is root, and neither decides anything for root | `legacy`, `accepted-legacy;rev3=;basis=root-input`, the J4 footing main gave the same body | | agent db-reads loop: the queue listing, the request read, the `picked` read-back, the record reads of checks A and B, the replica listing and GETs | the agent, `carlos`, `carlos-edge.service` | `legacy` with J1 (record and route reads) and J4 (object input), as the offline move's agent units took | | worker: the scratch writes and reads, the file replica restore, `CheckDBContext`, the query, the connection watcher | `carlos-dbread@`, `DynamicUser` | `legacy` with J4 | | agent readiness probe and admission (`Statfs` on `/var/tmp`, `systemctl list-units`) | the agent, `carlos` | `diagnostic` for the probe; `legacy` J4 for `Statfs` and the unit listing | (The census's J1 and J4 are its own dispositions, not this spec's invariants of the same names.) The overlay re-pins the source envelopes of every changed package (`.`, `internal/dbquery`, `internal/dbdoor`, `internal/host`, `internal/dbreads`) and updates the legacy seal under the same ruling. The census owner may rule other classes; the spec's requirement is only that every new and moved declaration has one of the four, with both callers of the shared runner named. - **Root writes.** `ensureSystemUnits` writes two new unit files. That is a root-written artifact and joins the Plan 14 write census as an overlay, as every new unit has. - **9B8 record-write census.** No record writer is added. The new `Put` sites (request, queue entry, `picked`, chunks, result, cancel) are `nonrecord`, all conditional creates. There is no `Delete` site. `registerCLIRoutes` changes as a `key-site` for the one route. - **Claim routes.** The route joins `internal/console/testdata/claim-routes.txt`, refused on a host-claim console like the run's. - **go.mod.** No new module, so no root-envelope overlay for it. ## 14. Plans 1. **Box side and wire, dormant.** `internal/dbreads` (ids, keys, the envelope, sealing, the result, limits); `internal/dbquery` (the query runner and guards moved out of `db.go`, with the CLI switched to it and no behaviour change); `dbdoor` planning over any `replicaReader`; the agent's db-reads loop with the identity bracket, conditional creates and the `picked` read-back, admission (reservations and the orphan count) and the readiness probe; `carlos system db-worker`, its protocol and its connection watcher; the two units in `ensureSystemUnits`; the `db-read-1` capability. Gates: T2, T3, T4, T5, T9, T12, T13, the parity half of T6 that needs no console. Census overlay. Nothing reaches a box unless a console asks. 2. **Console and CLI for `query` and `query --at`.** The one route and its streamed response, the read-deadline extension, the forward marker, the evidence rule for capability and liveness, admission held by the request, audit, the CLI's buffered forwarded path, DBF copy, `docs/site/cli.md`, and the `deployment-buckets` lifecycle rule with the approval plan for applying it. Gates: T1, T6, T7, T8, T10, T11, T14, T15, T16, T17, T18, copy review, census overlay. Roll per §11.3. 3. **`pull` and `pull --at`.** On ruling 1 (§16), which Paul confirmed with the storage statement on 2026-10-22. Sealed LTX files in the same single request, local restore and `CheckDBContext` in the CLI, and the pull copy rows. A 2 GiB pull in one request is the case most likely to need a longer request bound; plan 3 measures it before choosing. Two follow-ups are listed here and not written: 4. **Plan 4 (follow-up): in-region reads onto the box path** (ruling 5). Written once phase 3 has run on the flagship. It is not designed here beyond one constraint this spec already keeps: the box path is region-agnostic, so the console can send an in-region read through it unchanged. 5. **Plan 5 (follow-up): per-region db-read bucket** (ruling 1). After the feature works, `db-reads/` traffic moves from the shared deployment bucket to a bucket in each data region, so that ciphertext and request SQL never rest outside the data's region. It needs its own design for the console's write grant into each region's bucket, the boxes' grant on their own, the queue and serial signal, and the lifecycle rule per bucket. Plans 1 to 3 keep the protocol's keys relative to one store (`db-reads/...`) so that plan 5 changes where the store is, not what is in it. ## 15. Review focus: the five likeliest failure modes 1. **The worker reaches more than its copy.** A unit option missed or overridden (network, `/data`, `/etc/carlos`, the metadata service, the socket on stdin and stdout), or a SQLite path (ATTACH, `VACUUM INTO`, an extension, a temp file) that opens something. Any one of these turns a member's SQL into the box role, which reads every tenant in the region. 2. **The prefix or host comes from the request, or the identity bracket leaks.** Any path where an envelope field, rather than the fresh record and the box's config, picks the bucket, the prefix or the host is a cross-tenant read. So is any byte read outside checks A and B, or any output written before check B. 3. **Forgery and replay.** An unsigned or wrongly scoped request accepted, domain separation from other root envelopes missing, the region not bound, or a queue entry trusted without its request. 4. **Something outliving the request.** Any state, route, timer or write on the console side that survives the request's end, any protocol delete or replacing write, or an admission count released anywhere but the handler's return. Each one reopens a finding J1 to J5 closed. 5. **The long request.** A timeout on the real path that the spec missed (a proxy, a client, a keep-alive), wait lines that stall behind a slow reader, or a 2 GiB pull that does not fit the request bound. ## 16. Rulings The coordinator answered rev 1's five questions provisionally on 2026-10-22. Paul ruled on 2026-10-22, after Astra accepted rev 5: ruling 1 as below, and rulings 2 to 5 confirmed as written (he raised no objection). All five are now Paul's. 1. **Pull through the deployment bucket (Paul, 2026-10-22).** Ship the feature through the deployment bucket as designed, for `query`, `query --at`, `pull` and `pull --at`. The true storage statement stands and is what Paul ruled on: **a read's whole ciphertext, up to its size cap, lives in the deployment bucket (eu-west-1 on the flagship) until lifecycle expiry, typically two to four days, with no guaranteed upper bound** (§5.7). For a query that is at most 64 MiB of rows; for a pull it is the planned replica files, up to the pull's cap. The SQL rests in the request object on the same timing (§9.3). The earlier provisional bound ("at most 64 MiB at a time for at most 1 hour") does not hold and is withdrawn. A short or hard bound would need version-aware deletes, with `s3:DeleteObjectVersion` on the deployment bucket for the console's role, which this design does not add. **Follow-up, after the feature works: a per-region bucket for this traffic** (plan 5, §14), so that ciphertext and request SQL never rest outside the data's region. Plans 1 to 3 do not build it and do not wait for it; they keep every `db-reads/` object key relative to one store so that plan 5 can move the store without changing the protocol. 2. **Limits stand.** 2 GiB restored, 5 minutes, 64 MB of rows, two reads per box. Real database sizes on the flagship are still unmeasured; plan 2's dev check reports them. The pull's cap, 2 GiB of planned LTX files (§5.7, rev 7, measured on dev 2026-10-23), is put to Paul as a limit: **pending his confirmation.** 3. **Audit.** The row records the SQL's length and a SHA-256 prefix, not its text (§9.3). The SQL itself rests in the request object until expiry. 4. **Copy casing.** New strings use Paul's voice rule: lower case after a full stop (§10). The existing db door strings that capitalise are left as they are; changing them is a separate copy review. 5. **One path.** In-region reads move to the box path after phase 3 (§14). ## 17. What this does not do - No console per region and no console-to-box connection. - No resumable reads: a read that loses its connection is run again. - No change to in-region reads (until the fourth plan), the claim-mode door, seeds `--from`, fleet or pooled instances. - No IAM change. No new read authority for any role. The one infrastructure change is a lifecycle rule on the deployment bucket. - No per-store S3 endpoints. - No authenticated authorship of answers (§5.6). - No `carlos db shell` and no browser view. ## 18. Declined review findings **Round 1** (`CODEX-FINDINGS-DB-REGION-FORWARD.md`). Every citation was checked against `main` at `5f341f33` and holds. Findings 2 to 10 and 12 are accepted in full. Two are accepted in part, and rounds 2 and 3 agreed with both declines: - **Codex 1, the "stronger guarantee" (pin object versions to the checked incarnation): declined.** The identity bracket (§8.2) closes the finding. Pinning versions needs `s3:GetObjectVersion` and `s3:ListBucketVersions` on the box's data bucket, which the flagship's uk, us and au boxes do not have at module ref v0.21.0, and a versioned reader that today lives only behind the claim boundary. Claim activation can revisit it. - **Codex 11, independently configured deployment and regional-data endpoints: declined.** No box path supports regional data on a separate store server today, because every S3 client in the process reads one `CARLOS_S3_ENDPOINT`. Per-store endpoints, if wanted, are a platform-wide change with its own design. **Round 2** (`CODEX-FINDINGS-DB-REGION-FORWARD-R2.md`). Nothing declined. **Round 3** (`CODEX-FINDINGS-DB-REGION-FORWARD-R3.md`). Every citation was checked and holds, including that the worker must call `lease.CheckDBContext` (`internal/lease/check.go:44`), not `CheckDB`; §8.3 and §13 now say so. Nothing is declined. Rev 4 does not patch findings 13, 14, 16 and 17 or the partly-holding invariants one by one: it removes the durable read they all came from (§5.2), and §5.3 maps every finding to the invariant that closes it or to the synchronous design that removes it. Finding 7's fix is the honest timing statement, not a new mechanism; a hard bound would need version-aware deletes, which is noted for Paul in §16 rather than built. Finding 18 is fixed by withdrawing the claim and stating the trust boundary (§5.6), not by adding per-box signing. **Round 4** (`CODEX-FINDINGS-DB-REGION-FORWARD-R4.md`). Every citation was checked and holds, including litestream's file client and the LTX compactor ignoring their contexts, and the backhaul transport's 30 s header timeout. Nothing is declined. All four fixes reuse what the spec already had: the run's result-before-list order, `http.ResponseController`, the orphan refusal, and a process exit.