# S3 Change Index Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Stop every box and the console from re-walking the whole deployment bucket: writers bump a per-scope generation in `control/changes.json`, readers re-read only what moved, a 30-minute backstop walk bounds staleness. **Architecture:** The bump lives inside the two concrete stores (`S3Store`, `DirStore`), enabled by deployment openers. A process-wide `release.ChangeWatch` polls the index with a conditional GET. `provision.RecordView` keeps the record families the walk reads and exposes immutable snapshots as a read-through `release.Store`, so `Reconcile`, `ReconcileSidecars`, the moves pass, steering and the console sweeps run unchanged over a snapshot. A `release.ReadGate` gates the per-app GET loops on generations plus a maximum age. **Tech Stack:** Go, AWS SDK v2 S3 (existing client in `internal/release`), `DirStore` for tests, MinIO in the delivered-contract test. **Spec:** `docs/superpowers/specs/2026-10-21-s3-change-index-design.md` (rev 3, Astra-accepted). Section numbers below (§N) refer to it. ## Global Constraints - Index key `control/changes.json`, body `{"schema":1,"entries":{"":}}`; generations only increase; entries never removed in v1 (§3.1). - Scopes exactly per §3.2: `apps//` (except data prefixes `logs`, `errors`, `analytics`, `runs`, `metrics`), `accounts/`, `control/certs`, `control/edges`; nothing else. - Bump after a successful mutation only; at most 4 attempts with jitter inside a 2s budget detached from the caller's cancellation; a failed bump is logged and never fails the write (§3.3). - Backstop interval 30 minutes, jittered ±10% (R3). Gated reads have a maximum successful-read age equal to the backstop (§5). - Readers armed only by `CARLOS_CHANGE_INDEX=read` (box `host.env`, console environment). Unset means exactly today's behavior. Writers always bump once shipped (§7). - A store without `ConditionalGetter` + `ETagLister`, a denied index read, or a malformed index means "unknown": walk as today, never "empty and unchanged" (§3.4). - Core only: no AWS SDK outside `internal/release`'s existing S3 client (§6). - Lock order: `w.pass` → `reconcileMu` → `RecordView.mu`; `RecordView.mu` is a leaf (§4.3). - Tests: run only touched packages locally with `-run`; CI runs the full suite. Leave `GOCACHE` at its default. The simulation runs niced, one 3000-seed sweep per revision. - No tenant names in code, tests or comments (use `shop`, `example.com`). - Every new top-level test stands alone: `make test` is about to split `internal/console` and the root package across separate `go test` processes, so no test may depend on another having run first. Root-package e2e tests call `buildCarlos(t)` and never write to the shared read-only binary it returns. - `internal/host` tests call `t.Parallel()` unless they set env (`t.Setenv`), cwd, umask, log output or package vars, or reach `edge.SecureUnixListen`. Tasks 8 and 9 set the package vars `armedChangeWatch`/`armedRecordView` (and `CARLOS_CHANGE_INDEX`), so those tests stay serial; tests that only build their own `fastDeps`, `genCache` or gates call `t.Parallel()`. - Any user-facing text (release notes for R4) goes through the copy-review skill before it is written. ## Review Focus 1. **A deployment with no `control/changes.json` yet** (first arm, fresh bucket): the first `Refresh` must rebuild by walking, and the view must never publish as empty-and-complete. Test in Task 6. 2. **A malformed or future-schema index** (hand edit, newer writer): readers treat it as unusable and walk; writers do not overwrite an unknown schema. Tests in Tasks 1 and 4. 3. **A mixed fleet**: an armed reader while an old binary writes a record without bumping. The record appears after the next `Rebuild`, and no route is retired in between. Tests in Tasks 6 and 7. 4. **An app purge** deleting hundreds of keys under one scope: bumps stay bounded (one per mutation, each a single CAS), the view drops the app once, nothing is retired before the purge's own records are gone. Test in Task 2 (bump count) and Task 6 (view). 5. **Index read denied by IAM** (role lacks `GetObject` on the key): `ChangeWatch.Usable()` stays false, `ReadGate.Due` is always true, every loop behaves as today. Tests in Tasks 4 and 9. --- ### Task 1: Change scope and the index document **Files:** - Create: `internal/release/changeindex.go` - Test: `internal/release/changeindex_test.go` **Interfaces:** - Produces: - `const ChangeIndexKey = "control/changes.json"` - `type ChangeIndex struct { Schema int \`json:"schema"\`; Entries map[string]uint64 \`json:"entries"\` }` - `func ChangeScope(key string) (scope string, ok bool)` - `func ReadChangeIndex(ctx context.Context, s Store) (ChangeIndex, string, error)` — absent object returns `ChangeIndex{Schema: 1, Entries: {}}`, etag `""`, nil; schema ≠ 1 returns `ErrChangeIndexSchema`. - `var ErrChangeIndexSchema = errors.New("release: change index schema unsupported")` - `func bumpChangeIndex(ctx context.Context, get func(context.Context, string) ([]byte, string, error), put func(context.Context, string, []byte, string) (string, error), scopes ...string) error` — unexported; takes raw get/put so the store hook (Task 2) can call it without recursion. - [ ] **Step 1: Write the failing tests** ```go func TestChangeScope(t *testing.T) { cases := map[string]string{ "apps/bab/shop/instances/a.example.com.json": "apps/bab/shop", "apps/bab/shop/config-gen.json": "apps/bab/shop", "apps/bab/shop/channels/canary/x": "apps/bab/shop", "apps/bab/shop/sidecars.json": "apps/bab/shop", "accounts/bab/account.json": "accounts/bab", "accounts/bab/telemetry.json": "accounts/bab", "control/certs/example.com": "control/certs", "control/edges/box-a.json": "control/edges", } for k, want := range cases { got, ok := ChangeScope(k) if !ok || got != want { t.Errorf("ChangeScope(%q) = %q,%v want %q", k, got, ok, want) } } for _, k := range []string{ "apps/bab/shop/logs/box/2026-10-21-10/x", "apps/bab/shop/errors/x", "apps/bab/shop/analytics/x", "apps/bab/shop/runs/r1/request.json", "apps/bab/shop/metrics/x", "apps/bab/shop", "apps/bab/", "apps/", ChangeIndexKey, "control/serial", "control/edge-mesh-key", "control/edge-mesh-key.next", "releases/x", "status/boxes/a.json", "fleets/f/apps/bab/shop/config.json", } { if s, ok := ChangeScope(k); ok { t.Errorf("ChangeScope(%q) = %q, want none", k, s) } } } func TestBumpChangeIndexCreatesThenIncrements(t *testing.T) // DirStore: two bumps of "apps/bab/shop" -> 2; other entry untouched func TestBumpChangeIndexRetriesOnConflict(t *testing.T) // put fails with ErrCASConflict twice, then succeeds -> nil, final gen 1 func TestBumpChangeIndexGivesUpAfterFourAttempts(t *testing.T) // 4 conflicts -> non-nil error, exactly 4 puts func TestBumpChangeIndexRefusesUnknownSchema(t *testing.T) // existing {"schema":2,...} -> ErrChangeIndexSchema, no put func TestReadChangeIndexAbsentIsEmpty(t *testing.T) // Entries non-nil, len 0, etag "" ``` - [ ] **Step 2: Run to verify they fail** Run: `go test ./internal/release/ -run 'TestChangeScope|TestBumpChangeIndex|TestReadChangeIndex' -count=1` Expected: FAIL (undefined: ChangeScope) - [ ] **Step 3: Implement** `ChangeScope`, `ReadChangeIndex`, `bumpChangeIndex` in `internal/release/changeindex.go`. `ChangeScope` requires `apps///` with all three non-empty segments and ``'s first segment not in the data set; `accounts//` with both non-empty. `bumpChangeIndex` applies all scopes in one read-modify-write; conflict is `errors.Is(err, ErrCASConflict)`; jitter 20–120ms between attempts; it does not create its own context (Task 2 passes the detached one). - [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS - [ ] **Step 5: Commit** `git commit -m "release: change scopes and the change index document (#336)"` --- ### Task 2: Bump inside the concrete stores **Files:** - Modify: `internal/release/store.go` (`S3Store` struct `:681`, `DirStore` struct `:342`, `Put` `:414/:787`, `PutFrom` `:456/:820`, `Delete` `:660/:970`) - Modify: `internal/release/conditional_delete.go` (`DeleteIfMatch` `:63/:86`) - Test: `internal/release/changeindex_store_test.go` **Interfaces:** - Consumes: Task 1's `ChangeScope`, `bumpChangeIndex`. - Produces: - `func (s *S3Store) EnableChangeIndex()`, `func (d *DirStore) EnableChangeIndex()` - `func NewDeploymentS3Store(ctx context.Context, bucket string) (*S3Store, error)` and `func NewDeploymentS3StoreIn(ctx context.Context, bucket, region string) (*S3Store, error)` — the S3 constructors plus `EnableChangeIndex()`. - `var changeIndexBumpBudget = 2 * time.Second` (var for tests). - [ ] **Step 1: Write the failing tests** (all on `DirStore` with the index enabled) ```go func TestStoreBumpsAfterEveryMutation(t *testing.T) { // Put, PutFrom, Delete, DeleteIfMatch on "apps/bab/shop/instances/h.json" // each raise entries["apps/bab/shop"] by exactly 1. } func TestStoreDoesNotBumpFailedMutation(t *testing.T) // Put with stale ifMatch -> ErrCASConflict, gen unchanged func TestStoreDoesNotBumpUnscopedKeys(t *testing.T) // logs/, control/serial, the index key itself -> no index object func TestStoreBumpFailureDoesNotFailWrite(t *testing.T) // index object is a directory (unwritable) -> Put returns nil, record written func TestStoreBumpDetachedFromCallerCancel(t *testing.T)// ctx cancelled right after Put returns from the backend -> bump still lands func TestStoreWithoutEnableNeverBumps(t *testing.T) func TestStoreCapabilitiesPreserved(t *testing.T) { d := NewDirStore(t.TempDir()); d.EnableChangeIndex() var s Store = d _, ok1 := s.(ConditionalDeleter); _, ok2 := s.(PutStreamer); _, ok3 := s.(Opener) _, ok4 := s.(SizeLister); _, ok5 := s.(PageLister); _, ok6 := s.(ModLister) if !(ok1 && ok2 && ok3 && ok4 && ok5 && ok6) { t.Fatal("index-enabled DirStore lost an extension") } } func TestPurgeBumpsBounded(t *testing.T) // 300 deletes under one app -> gen 300, each bump one CAS (count puts on a counting wrapper of the raw backend) ``` - [ ] **Step 2: Run to verify they fail** Run: `go test ./internal/release/ -run 'TestStore(Bumps|DoesNot|BumpFailure|BumpDetached|WithoutEnable|Capabilities)|TestPurgeBumps' -count=1` Expected: FAIL (undefined: EnableChangeIndex) - [ ] **Step 3: Implement.** Add an unexported `changeIndex bool` field to both structs. Split each mutation into the existing body (renamed `putRaw`, `putFromRaw`, `deleteRaw`, `deleteIfMatchRaw`) and a public method that calls the raw one and, on success and when enabled and `ChangeScope` matches, calls `bumpChangeIndex` with `getRaw`/`putRaw` under `context.WithTimeout(context.WithoutCancel(ctx), changeIndexBumpBudget)`, logging a failure as `release: change index bump %s: %v`. The struct literal `&release.S3Store{Client:..., Bucket:...}` elsewhere must still compile (exported fields unchanged). - [ ] **Step 4: Run to verify they pass** (same command, plus `go test ./internal/release/ -count=1` for regressions). Expected: PASS - [ ] **Step 5: Commit** `git commit -m "release: concrete stores bump the change index after each mutation (#336)"` --- ### Task 3: Read extensions and their forwarding **Files:** - Modify: `internal/release/store.go` (new interfaces beside `ModLister` `:186-207`; methods on both stores; `prefixStore` forwarding `:113-217`) - Modify: `internal/hostops/mutation_boundary.go` (forwarding surface `:108-130`) - Modify: the MinIO delivered-contract test (`grep -rn TestDeliveredContractMinio`) - Test: `internal/release/readext_test.go`, `internal/hostops/mutation_boundary_test.go` **Interfaces:** - Produces: - `type ConditionalGetter interface { GetIfChanged(ctx context.Context, key, etag string) (body []byte, newETag string, changed bool, err error) }` — `etag == ""` behaves as `Get`; unchanged returns `nil, etag, false, nil`; absent returns an `os.ErrNotExist`-wrapped error. - `type KeyETag struct { Key, ETag string }` - `type ETagLister interface { ListETags(ctx context.Context, prefix string) ([]KeyETag, error) }` — sorted by key; ETags in the same form `Get` returns. - Both on `*S3Store` and `*DirStore`; `prefixStore` and `mutationBoundary` forward both (boundary returns `ErrContract` when the raw store lacks one, matching `ListMod`). - [ ] **Step 1: Write the failing tests** ```go func TestGetIfChangedDirStore(t *testing.T) // put k; GetIfChanged(k,"") -> changed, etag e; GetIfChanged(k,e) -> !changed, body nil; put again; GetIfChanged(k,e) -> changed func TestGetIfChangedAbsent(t *testing.T) // errors.Is(err, os.ErrNotExist) func TestListETagsMatchesGet(t *testing.T) // for each listed key, ETag == etag from Get func TestPrefixStoreForwardsReadExtensions(t *testing.T) // NewPrefixStore(dir,"fleets/f/") answers both, keys stripped func TestBoundaryForwardsReadExtensions(t *testing.T) // in hostops ``` In the MinIO contract test add: `GetIfChanged` returns `changed=false` for the current ETag (S3 304), and `ListETags` ETags equal `Get` ETags. - [ ] **Step 2: Run to verify they fail** Run: `go test ./internal/release/ ./internal/hostops/ -run 'GetIfChanged|ListETags|ForwardsReadExtensions' -count=1` Expected: FAIL - [ ] **Step 3: Implement.** S3: `GetObject` with `IfNoneMatch: etag`; a 304 surfaces from the SDK as a smithy API error with HTTP status 304 — map it to `changed=false`. `ListETags` pages `ListObjectsV2` and keeps `Key`/`ETag` exactly as `Get` reports them (same quoting). DirStore: compute the ETag the way its `Get` does. - [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS. MinIO contract runs in CI. - [ ] **Step 5: Commit** `git commit -m "release: conditional GET and ETag listings, forwarded through prefix stores and the hostops boundary (#336)"` --- ### Task 4: ChangeWatch and ReadGate **Files:** - Create: `internal/release/changewatch.go` - Test: `internal/release/changewatch_test.go` **Interfaces:** - Consumes: Tasks 1 and 3. - Produces: - `const ChangeIndexBackstop = 30 * time.Minute` (R3; here so `provision`, `host`, `edge` and `console` share it without import cycles) - `type ChangeWatch struct` (safe for concurrent use) - `func NewChangeWatch(s Store) *ChangeWatch` - `func (w *ChangeWatch) Poll(ctx context.Context) (moved []string, err error)` — conditional GET of `ChangeIndexKey`; returns scopes whose generation rose (or appeared) since the last successful poll; on 304 returns `nil, nil`; any error, an unknown schema, or a store lacking `ConditionalGetter` marks the watch unusable and returns the error. - `func (w *ChangeWatch) Gen(scope string) (gen uint64, ok bool)` — `ok` false while unusable. - `func (w *ChangeWatch) Observed() (map[string]uint64, bool)` — copy of the last good entries. - `func (w *ChangeWatch) Usable() bool` - `type ReadGate struct`; `func NewReadGate(w *ChangeWatch, maxAge time.Duration) *ReadGate` - `func (g *ReadGate) Due(scope string, now time.Time) (gen uint64, due bool)` — due when the watch is nil or unusable, the scope was never marked, its last mark failed, its generation moved past the marked one, or `now` is at least `maxAge` past the last successful mark. - `func (g *ReadGate) Mark(scope string, gen uint64, now time.Time, ok bool)` — records the generation captured by `Due` before the read. - [ ] **Step 1: Write the failing tests** ```go func TestChangeWatchFirstPollReportsAll(t *testing.T) // index with 2 entries -> moved has both func TestChangeWatchUnchangedIs304(t *testing.T) // second poll, no writes -> moved nil, err nil; counting store shows GetIfChanged with etag func TestChangeWatchReportsOnlyMoved(t *testing.T) func TestChangeWatchAbsentIndexIsUsableEmpty(t *testing.T) // absent -> Usable true, Observed empty func TestChangeWatchDeniedIsUnusable(t *testing.T) // store whose Get of the key fails with access denied -> Usable false func TestChangeWatchUnknownSchemaIsUnusable(t *testing.T) func TestChangeWatchWithoutExtensionIsUnusable(t *testing.T) func TestReadGateDue(t *testing.T) // table: unusable->due; never marked->due; marked ok same gen within age->!due; gen moved->due; age>=maxAge->due; last mark failed->due ``` - [ ] **Step 2: Run to verify they fail** Run: `go test ./internal/release/ -run 'TestChangeWatch|TestReadGate' -count=1` Expected: FAIL - [ ] **Step 3: Implement** in `changewatch.go`. Absent index is usable with empty entries (Review Focus 1 is handled by `RecordView`, not here). - [ ] **Step 4: Run to verify they pass.** Expected: PASS - [ ] **Step 5: Commit** `git commit -m "release: ChangeWatch polls the change index; ReadGate gates reads on generation and age (#336)"` --- ### Task 5: Every deployment opener enables the index **Files:** - Modify: deployment-bucket openers among the `release.NewS3Store`/`NewS3StoreIn` call sites: `cli.go:1957,2067,3591`, `cmd/console/main.go:633` (the deployment bucket; `:333` is the database bucket), `internal/host/agent.go:4368,4462`, `internal/edge/edge.go:3407`, `internal/host/relocatepass.go:251,680`, `internal/host/activation_command_linux.go:67,92,206,283`, `internal/ops/ops.go:339`, `internal/vet/vet.go:226`, `internal/host/seedwiring.go:102`. Classify each by reading it; switch deployment ones to `NewDeploymentS3Store[In]`. - Modify: `internal/host/boxpublish.go` only if its test shows the concrete assertion `:259` failing (it should not: the type is still `*release.S3Store`). - Create: `internal/release/openers_test.go` **Interfaces:** - Consumes: Task 2's `NewDeploymentS3Store[In]`. - [ ] **Step 1: Write the failing test** `TestS3StoreOpenersClassified`: parse every non-test `.go` file in the module with `go/parser`, find calls to `release.NewS3Store`, `release.NewS3StoreIn` (and the package-internal `NewS3Store` outside `store.go`), and require each `file:func` to appear in a table `otherBucketOpeners map[string]string` (value: the reason, for example `"database bucket"`, `"hibernate data bucket"`). Deployment openers must call `NewDeploymentS3Store[In]` instead, so they never appear. - [ ] **Step 2: Run to verify it fails** Run: `go test ./internal/release/ -run TestS3StoreOpenersClassified -count=1` Expected: FAIL listing the unclassified call sites - [ ] **Step 3: Switch** each deployment opener to `NewDeploymentS3Store[In]` and list the rest in `otherBucketOpeners` with a reason. - [ ] **Step 4: Box publication witness** `TestBoxPublishThroughDeploymentOpener` in `internal/host`: `openDeployStoreEnv` against a `DirStore`-backed fake is out of reach, so assert at the type level that `NewDeploymentS3Store`'s result satisfies the `*release.S3Store` assertion used at `boxpublish.go:259` (compile-time `var _ *release.S3Store = mustDeploy()` plus a runtime check through the same `opened.(*release.S3Store)` expression). - [ ] **Step 5: Run** `go test ./internal/release/ ./internal/host/ -run 'TestS3StoreOpenersClassified|TestBoxPublishThroughDeploymentOpener' -count=1` and `go build ./...`. Expected: PASS, build clean. - [ ] **Step 6: IAM check.** Read `infra/modules/deployment-buckets/main.tf` and `infra/modules/region-host` policies: confirm every writing role (box, console) is allowed `s3:GetObject` and `s3:PutObject` on `control/changes.json` and no bucket-policy Deny covers it. If anything denies it, add an allow and a `tests/*.tftest.hcl` assertion in that module. Record the result (and the same check owed on the Tito infrastructure repo) in the branch's #336 thread. - [ ] **Step 7: Commit** `git commit -m "Deployment openers enable the change index; opener classification test (#336)"` --- ### Task 6: RecordView and its snapshot store **Files:** - Create: `internal/provision/recordview.go`, `internal/provision/recordview_store.go` - Test: `internal/provision/recordview_test.go` **Interfaces:** - Consumes: Tasks 1, 3, 4 (including `release.ChangeIndexBackstop`); existing `knownAccounts` (`reconcile.go:1760`), `listAppSubtree` (`:2022`), `ManifestKey`/`RecordsPrefix` (`provision.go:321`), `SidecarsPrefix` (`sidecar.go:122`), `release.ReadAppTombstone`'s key, `domains.AccountManifestSqid`. - Produces: - `type RecordView struct` with `mu sync.Mutex` (leaf lock) - `func NewRecordView(live release.Store, watch *release.ChangeWatch) *RecordView` - `func (v *RecordView) Refresh(ctx context.Context) error` — §4.2 under `mu`: `watch.Poll`, then re-read every scope with `applied < observed` or new; an unusable watch, or a view never built, runs `Rebuild` instead. - `func (v *RecordView) Rebuild(ctx context.Context) error` — §4.4 under `mu`: read `watch.Observed()` (after a `Poll`) first, then walk everything, publish, set `applied` = the pre-walk generations. - `func (v *RecordView) Snapshot() *ViewSnapshot` — atomic load; nil before the first publication. - `type ViewSnapshot struct { Epoch uint64; Complete bool /* plus unexported cached families */ }` - `func (sn *ViewSnapshot) Store(live release.Store) release.Store` — read-through store, below. - `func ScopeApp(scope string) (acct, app string, ok bool)` — parses `apps//`. **Cached families** (what a snapshot answers; everything else goes to `live`): - `List("accounts/")` and `List("accounts//")`: complete key listings per account, kept from `ListETags("accounts//")` on each account-scope re-read. - `Get("accounts//account.json")`: cached body or not-found. - `ListDirs("apps/")`, `ListDirs("apps//")`: accounts and apps that have at least one in-scope app scope; `BucketShapedListDirs()` returns true so `listAppSubtree` walks. - `List` of `apps///instances/` and `apps///sidecars/`: complete, from `ListETags`. - `Get` of each record key under those prefixes, of `apps///deleted.json` (trash marker), `instances.json` and `sidecars.json`: cached body, cached not-found (an `os.ErrNotExist`-wrapped error), or the cached read error. - Every `Put`/`PutFrom`/`Delete`/`DeleteIfMatch` goes to `live`, and the key, and every listing prefix it falls under, is answered from `live` for the rest of that snapshot store's life (read-your-writes and CAS retries). - Optional extensions not listed above: forwarded to `live`. - [ ] **Step 1: Write the failing tests** (DirStore with the index enabled, `ChangeWatch` over it) ```go func TestViewMatchesWalk(t *testing.T) // 3 accounts, 5 apps, records/sidecars/manifests/trash: ListAllRecordsStrict over sn.Store(live) == over live func TestViewRefreshRereadsOnlyMovedApp(t *testing.T) // counting store: after one record Put in shop, Refresh lists only apps/bab/shop/{instances,sidecars}/ and GETs only the changed key + that app's three manifests/marker func TestViewRefreshNoChangeIsOneGet(t *testing.T) // counting store: exactly one GetIfChanged func TestViewRetriesFailedScopeOn304(t *testing.T) // inject one List failure for shop after its bump; next Refresh (index 304) re-reads shop; Complete false until then (codex 4) func TestViewFirstRefreshWithoutIndexRebuilds(t *testing.T) // no control/changes.json, records exist -> snapshot has them, Complete true (Review Focus 1) func TestViewUnusableWatchRebuilds(t *testing.T) // malformed index -> Refresh walks (Review Focus 2) func TestViewBypassWriteAppearsAfterRebuild(t *testing.T) // record written with index disabled -> absent after Refresh, present after Rebuild (Review Focus 3) func TestViewPurgeDropsApp(t *testing.T) // delete every key under shop -> ListDirs("apps/bab/") no longer has shop (Review Focus 4) func TestViewEpochsIncrease(t *testing.T) func TestViewBackstopCannotOverwriteNewer(t *testing.T) // codex 2 interleaving: Rebuild blocked mid-walk by a hook; concurrent Refresh must wait on mu; final snapshot has the record func TestSnapshotStoreReadYourWrites(t *testing.T) // Put via sn.Store, then Get/List return live state func TestSnapshotStoreErrorsMatchLive(t *testing.T) // not-found Get of a cached-family key: errors.Is(err, os.ErrNotExist) func TestSnapshotStoreOtherKeysGoLive(t *testing.T) // Get of config.json / List("apps/bab/shop/channels/") hit live ``` - [ ] **Step 2: Run to verify they fail** Run: `go test ./internal/provision/ -run 'TestView|TestSnapshotStore' -count=1` Expected: FAIL - [ ] **Step 3: Implement** `recordview.go` (state, `Refresh`, `Rebuild`, publication with `atomic.Pointer[ViewSnapshot]`) and `recordview_store.go` (the read-through store). `Rebuild` reuses `knownAccounts` and `listAppSubtree` over `live` (and `ListETags` to skip unchanged bodies already cached); an app-scope re-read uses `ListETags` on both prefixes and `GetIfChanged` for the three per-app objects. Incomplete stays incomplete until every scope with `applied < observed` succeeds. - [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS - [ ] **Step 5: Commit** `git commit -m "provision: RecordView keeps the record families and serves snapshots as a read-through store (#336)"` --- ### Task 7: Invariant simulation and mutation checks **Files:** - Create: `internal/provision/recordview_sim_test.go` **Interfaces:** - Consumes: Task 6. - [ ] **Step 1: Write the simulation** `TestRecordViewSimulation` (seeded by `-seed` and `-seeds` flags, default 200 in CI, skipped under `-short`): random actors on a `DirStore` — writers that bump, writers that bypass (`EnableChangeIndex` off), writers whose bump fails (injected), incremental `Refresh`, `Rebuild`, and a consumer that takes snapshots under a fake `w.pass` and records epochs. Check after every step: (a) after a `Refresh` that observed generation g for a scope, every bumped write to that scope at ≤ g is in the snapshot; (b) after `Rebuild`, `ListAllRecordsStrict(sn.Store(live))` equals `ListAllRecordsStrict(live)`; (c) snapshot epochs only increase and no scope's contents revert; (d) a snapshot with `Complete == false` never returns a nil error from `ListAllRecordsStrict`; (e) a failed scope is re-read on the next `Refresh` with the index unchanged. The seed set must include the three named interleavings in §8 (codex 1, 2, 4) as fixed seeds or scripted cases. - [ ] **Step 2: Run** `nice -n 19 go test ./internal/provision/ -run TestRecordViewSimulation -seeds 3000 -count=1`. Expected: PASS - [ ] **Step 3: Mutation checks** in a scratch copy of the tree under `$TMPDIR` (never in the worktree): apply each mutation, run the sweep, confirm it FAILS, discard the copy. Mutations: bump before write (Task 2); record `applied` for failed scopes (Task 6); skip the pre-walk index read in `Rebuild`; publish the rebuild outside `mu`; consume a handed-over snapshot instead of taking one under `w.pass`. Record the five results in the commit message. - [ ] **Step 4: Commit** `git commit -m "provision: RecordView invariant simulation; five mutation checks fail as they must (#336)"` --- ### Task 8: Box agent reads through the view **Files:** - Modify: `internal/host/agent.go` (`reconcileSerializedWith` `:1569`, fast-converge deps and `fastTick` `:2537`, `fullPass` `:2787`, `instancesLoop` `:3358`, agent start-up wiring) - Modify: `internal/host/movespass.go` (`movesPass` `:1060`), `internal/host/movesoffline.go` (`refreshOfflineView` `:241` callers), `internal/host/steering_loop.go:100` - Modify: `internal/appconfig/syncexec.go` (new option beside `:751-810`) - Create: `internal/host/changeindex.go` - Test: `internal/host/changeindex_test.go`, `internal/appconfig/syncexec_test.go` **Interfaces:** - Consumes: Tasks 4, 6. - Produces: - `func changeIndexArmed(getenv func(string) string) bool` — `getenv("CARLOS_CHANGE_INDEX") == "read"` - `var armedChangeWatch atomic.Pointer[release.ChangeWatch]`, `var armedRecordView atomic.Pointer[provision.RecordView]` (set once at start-up when armed; nil means today's behavior everywhere) - `func viewStore(ctx context.Context, live release.Store) (s release.Store, epoch uint64, ok bool)` — `Refresh` then `Snapshot().Store(live)`; `ok` false when disarmed, so callers use `live`. - `func recordViewBackstopLoop(ctx context.Context, v *provision.RecordView)` — `Rebuild` at start and every `release.ChangeIndexBackstop` ±10%, then runs `fullPass` (unrestricted). - `func appconfig.WithRouteFilter(keep func(registry.Route) bool) SyncOption` - `fastDeps.scopedPass func(ctx context.Context, moved []string)` - `movesWorker.lastEpoch uint64` - [ ] **Step 1: Write the failing tests** ```go func TestReconcileUsesViewWhenArmed(t *testing.T) // armed, counting live store: two reconcileSerializedWith calls with no writes -> zero List calls on live func TestReconcileWalksWhenDisarmed(t *testing.T) // disarmed -> today's List calls (regression guard) func TestMovesPassTakesSnapshotUnderPass(t *testing.T) // codex 1: hook pauses a pass after Reconcile; a second pass runs moves at epoch N+1; first pass's moves then sees epoch >= N+1 (never older) func TestMovesPassRefusesOlderEpoch(t *testing.T) // forced older snapshot -> pass skipped with a log line, no move step taken func TestFastTickIndexOnlyRunsScopedPass(t *testing.T) // serial unchanged, index moved for shop -> scopedPass called with ["apps/bab/shop"], fullPass not called func TestScopedPassTouchesOnlyMovedApp(t *testing.T) // codex 10 witness: registry hosts shop and cafe; bump shop; counting store sees no key under apps/bab/cafe/ func TestSerialChangeStillRunsFullPass(t *testing.T) func TestInstancesLoopFloorIsRefreshWhenArmed(t *testing.T) // nothing changed -> one GetIfChanged on live, no List func TestSyncExecWithRouteFilter(t *testing.T) // in appconfig: filter excluding cafe -> no read of cafe's bundle ``` - [ ] **Step 2: Run to verify they fail** Run: `go test ./internal/host/ ./internal/appconfig/ -run 'TestReconcileUsesView|TestReconcileWalksWhenDisarmed|TestMovesPass(Takes|Refuses)|TestFastTickIndexOnly|TestScopedPass|TestSerialChangeStill|TestInstancesLoopFloor|TestSyncExecWithRouteFilter' -count=1` Expected: FAIL - [ ] **Step 3: Implement.** - Start-up: when armed, open the watch on the deployment store, build the view, store both pointers, start `recordViewBackstopLoop`. A fleet box (store door) leaves itself disarmed: its watch would be unusable and every `Refresh` would fall back to a full walk anyway. - `reconcileSerializedWith`: inside `reconcileMu`, `s2, _, ok := viewStore(ctx, s)`; pass `s2` (or `s`) to `refreshOfflineView`, `provision.Reconcile` and `provision.ReconcileSidecars`. - `movesPass`: inside `w.pass`, take `viewStore`; refuse and log `agent: moves pass: snapshot epoch %d older than %d consumed` if the epoch is lower than `w.lastEpoch`; set `w.lastEpoch`. - `steeredRoutesFromRecords` callers pass `viewStore`'s store. - `fastTick`: when the watch is armed, `moved, err := watch.Poll(ctx)` every tick; a serial change keeps `fullPass`; otherwise non-empty `moved` (after the 5s clamp) runs `scopedPass(ctx, moved)`. - `scopedPass`: `reconcileSerialized`, `telemetryPass`, `runMovesPass`, `runRunsPass`, aliases as in `fullPass`, then `release.PassFilter` with `extra` keeping routes whose `(Account, App)` is in `ScopeApp(moved)`, then `syncExecSerialized` with `appconfig.WithRouteFilter` on the same set. - `instancesLoop`: unchanged structure; its reconcile now goes through the view, so a quiet floor costs one conditional GET. - [ ] **Step 4: Run to verify they pass** (same command), then `go test ./internal/host/ -count=1 -run 'Moves|Reconcile|FastTick|Steer'` for regressions. Expected: PASS - [ ] **Step 5: Commit** `git commit -m "host: reconcile, sidecars, moves and steering read through the record view; index-only changes run a scoped pass (#336)"` --- ### Task 9: Box GET loops gated by generation and age **Files:** - Modify: `internal/host/agent.go` (`genPollPass` `:1720`, `configGenLoop` `:1979`, the `PassFilter` call `:2628` and `fastMembers` `:2214`) - Modify: `internal/host/telemetry_pass.go:58` - Test: `internal/host/readgate_test.go` **Interfaces:** - Consumes: Task 4's `ReadGate`, Task 8's `armedChangeWatch`. - Produces: `genCache` gains a `gate *release.ReadGate`; `fastDeps` gains `canaryGate *release.ReadGate`; `telemetryPass` uses a package `telemetryGate` plus a per-pass account cache. All gates are `NewReadGate(armedChangeWatch.Load(), release.ChangeIndexBackstop)`; with a nil watch every read is due (today's behavior). - [ ] **Step 1: Write the failing tests** ```go func TestConfigGenSkipsUnmovedApp(t *testing.T) // armed, two ticks, no bump -> one GET of config-gen.json (the first) func TestConfigGenReadsMovedApp(t *testing.T) func TestConfigGenReadsAtMaxAge(t *testing.T) // fake clock past 30m -> GET (codex 9) func TestConfigGenRetriesAfterError(t *testing.T) func TestCanaryPointerGated(t *testing.T) // canary route stays a member; pointer GET only on move or age func TestTelemetryAccountReadOncePerPass(t *testing.T) // 3 apps in one account -> one GET of accounts//telemetry.json func TestTelemetryGatedWithAgeBound(t *testing.T) // codex 9 func TestGatesDueWhenWatchUnusable(t *testing.T) // Review Focus 5: watch denied -> every read due on every tick ``` - [ ] **Step 2: Run to verify they fail** Run: `go test ./internal/host/ -run 'TestConfigGen|TestCanaryPointerGated|TestTelemetry(Account|Gated)|TestGatesDueWhenWatchUnusable' -count=1` Expected: FAIL - [ ] **Step 3: Implement.** Each read calls `gen, due := gate.Due(scope, now)`; skip when not due; after the read `gate.Mark(scope, gen, now, err == nil)`. For canaries, gate the pointer read inside the `PassFilter` membership for `canary/*` routes only; stable routes keep today's path. - [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS - [ ] **Step 5: Commit** `git commit -m "host: config-gen, canary pointers and telemetry read only on change or at the backstop age (#336)"` --- ### Task 10: Edge cert mirror and home map **Files:** - Modify: `internal/edge/certsync.go` (`certMirror` struct, `Refresh` `:622`, loop `:411`) - Modify: `internal/edge/homemap.go` (`Refresh` `:320`, `refreshHosts` `:264`) - Modify: the edge's wiring in `internal/edge/edge.go` to pass the process's watch (nil when disarmed) - Test: `internal/edge/certsync_test.go`, `internal/edge/homemap_test.go` **Interfaces:** - Consumes: Tasks 3, 4. - Produces: `certMirror.watch *release.ChangeWatch`, `homeMap.watch *release.ChangeWatch`; each keeps the last successful full rebuild time. - [ ] **Step 1: Write the failing tests** ```go func TestCertMirrorSkipsWhenUnmoved(t *testing.T) // armed, two polls, no bump -> second poll: no List, no Get func TestCertMirrorFetchesOnlyChanged(t *testing.T) // one cert rewritten -> ListETags + exactly one Get func TestCertMirrorDropsRemovedKey(t *testing.T) // deleted challenge token disappears from the snapshot func TestCertMirrorFullRebuildAtBackstop(t *testing.T) func TestCertMirrorBootRefreshUnchanged(t *testing.T) // boot path still a full read func TestHomeMapKeysPolledIndependently(t *testing.T) // codex 8: rotate control/edge-mesh-key, control/edges unchanged -> new key loaded within one poll func TestHomeMapHeartbeatOnUnchangedPoll(t *testing.T) // RefreshAttempted advances when hosts are skipped ``` - [ ] **Step 2: Run to verify they fail** Run: `go test ./internal/edge/ -run 'TestCertMirror|TestHomeMap(Keys|Heartbeat)' -count=1` Expected: FAIL - [ ] **Step 3: Implement.** Cert mirror: when the watch is usable and `control/certs` has not moved and the last full rebuild is younger than `release.ChangeIndexBackstop`, the poll does nothing; when it moved, `ListETags(certPrefix)`, GET new or changed keys, drop keys absent from the listing; otherwise today's full `Refresh`. Home map: same rule for `refreshHosts` on `control/edges`; `refreshKeys` and `health.RefreshAttempted` run on every poll as today. - [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS - [ ] **Step 5: Commit** `git commit -m "edge: cert mirror and home map re-read only what moved; mesh keys keep their own poll (#336)"` --- ### Task 11: Console sweeps **Files:** - Modify: `cmd/console/main.go` (arm watch and view when `CARLOS_CHANGE_INDEX=read`; start the backstop loop) - Modify: `internal/console/server.go` (Server fields), `internal/console/runsweep.go` (`:62-114`), `internal/console/seedsweep.go` (`:140-165`), `internal/console/store.go` (new query beside `RecordActivity` `:2591`) - Test: `internal/console/runsweep_test.go`, `internal/console/seedsweep_test.go`, `internal/console/store_test.go` **Interfaces:** - Consumes: Tasks 4, 6. - Produces: - `Server.recordView *provision.RecordView` (nil when disarmed) - `func (s *Store) AppsWithActionSince(action string, since time.Time) ([]AppRef, error)` with `type AppRef struct{ Sqid, App string }` — distinct `(account sqid, app)` from the `audit` table where `action = ?` and the time column ≥ `since`. - `const orphanCandidateWindow = 48 * time.Hour`, `const orphanFullSweepEvery = 24 * time.Hour`, `Server.lastOrphanFull time.Time` - [ ] **Step 1: Write the failing tests** ```go func TestRunSweepReadsViewWhenArmed(t *testing.T) // counting releases store: no List of apps/ across two sweeps with no writes func TestOrphanHousekeepingListsOnlyRecentRunApps(t *testing.T) // shop has run-start 1h ago, cafe none -> ListDirs only under apps/bab/shop/runs/ (codex R2-1) func TestOrphanHousekeepingFullSweepDaily(t *testing.T) // fake clock past 24h -> every app with records is swept func TestAppsWithActionSince(t *testing.T) // rows older than since excluded; distinct pairs func TestSeedSweepSkipsWalkWithoutSnapshots(t *testing.T)// no snapshots -> no strict live walk; abandonment still sees a Seed record via the view func TestSeedSweepWalksStrictWithSnapshots(t *testing.T) // snapshots present -> live ListAllRecordsStrict after the snapshot listing, as today ``` - [ ] **Step 2: Run to verify they fail** Run: `go test ./internal/console/ -run 'TestRunSweepReadsView|TestOrphanHousekeeping|TestAppsWithActionSince|TestSeedSweep(Skips|Walks)' -count=1` Expected: FAIL - [ ] **Step 3: Implement.** `runSweepPass` takes its record store from `recordView.Refresh` + `Snapshot().Store(live)` when armed. Housekeeping builds its app set from `AppsWithActionSince("run-start", now-48h)` unless `now-lastOrphanFull >= 24h`, in which case it uses every app with records and sets `lastOrphanFull`. Seed sweep: when the snapshot listing is empty and the view is armed, read records from the view snapshot; otherwise keep the strict live walk exactly where it is today (after `seedsPassBetweenListings`). - [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS - [ ] **Step 5: Commit** `git commit -m "console: run sweep reads the record view; orphan housekeeping narrows to recent runs; seed sweep skips the walk with nothing to delete (#336)"` --- ### Task 12: Census overlay, release note, rollout notes **Files:** - Create: the plan14 root-reads census overlay for every touched covered package (recipe in the `litestream-284` memory and #280), `census-ruling-s3-change-index` - Modify: `README.md` rollout/arming section (where release recipes live), `CHANGELOG` or release-notes location the repo uses - Test: the census test package (`internal/provision`'s root-reads census test) - [ ] **Step 1: Run the census test** `go test ./internal/provision/ -run RootRead -count=1`. Expected: FAIL naming changed packages/units. - [ ] **Step 2: Merge main first, then generate the overlay** so it loads after the latest one on main (as of 2026-10-21 the overlays from `ci-speed-activator` and `securelisten-337`; check main's latest when you get there), with the scratch-test recipe (`rootReadProduction`, `loadRootReadInventory`, `rootReadLegacyFingerprint`, `rootReadLegacySeal`), add it, and post the census-ruling request on #280. - [ ] **Step 3: Run the census test again.** Expected: PASS - [ ] **Step 4: Release note for R4** (old CLIs in direct-bucket mode converge at the 30-minute backstop until updated) and the arming steps from §7: draft the strings, run them through the copy-review skill, apply Paul's edits verbatim. - [ ] **Step 5: Full local gate on touched packages** `go test ./internal/release/ ./internal/provision/ ./internal/host/ ./internal/edge/ ./internal/console/ ./internal/appconfig/ ./internal/hostops/ -count=1` (host and console with `-run` filters per the targeted-tests rule if the full packages are too heavy), `go vet ./...`, and the cross-compile check (`GOOS=darwin go build ./...`, `GOOS=windows go build ./...`). Expected: PASS - [ ] **Step 5b: Claim invariant.** If any console test added here swaps process-wide state (a package var, log output, a client transport) or uses `t.Setenv` inside a `TestClaimConsoleMutationInvariant` fixture, list it serial (`claimSerialDoors` or the probe's `serial` field) and call `claimMustRunAlone(t, ...)` before the swap. Task 11 adds no mutation doors, so this is a check, not expected work. - [ ] **Step 6: Shard weights.** If any new console or root-package test takes several seconds, refresh `.amadan/test-weights.tsv` with `.amadan/test-weights.sh` (CI runs `.amadan/test-gate.sh`, which shards console 8 ways and root 3 ways). Put no CI-only script under `scripts/`: the root-read census counts every `.sh` outside dot directories. - [ ] **Step 7: Merge main, commit and push**; wait for CI green on the tip (one `=== ` section per job; failures listed under FAILED: at the end).