# Region HA 2a: The Provider-Neutral Core Implementation Plan (rev 4) > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Build region HA spec 2's provider-neutral core, dormant behind `CARLOS_REGION_HA`: box and group roles, the placement policy, H27's no-room refusal on every door that chooses a box, the load balancer health answer, private-address forwarding with a validated address source, and the eligible-capacity metric. **Architecture:** Every new rule sits on spec 1's code without changing what it proves. The placement policy runs over `boxchoice.Eligible`'s identity-filtered candidates inside `hostops.ExecutorChooser`, which creates and relocation targets share, and which the create and move doors ask whether a region has room; roles live in the signed roster. A create door chooses its box before its first write, so a refusal leaves nothing behind. The health answer is an in-memory state (`internal/lbhealth`) that the agent's own loops feed and the edge reads on `:80`. Forwarding changes only the address an edge dials. Each AWS part is an adapter in a file named for AWS (`internal/host/awsaddr.go`, `internal/console/awsregionmetric.go`), and a test holds the core to that. **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `internal/hostactors`, `internal/boxchoice`, `internal/edgesfile`, `internal/hostops`, `internal/host`, `internal/edge`, `internal/activator`, `internal/console`, the `carlos` CLI (repo root); AWS SDK v2 CloudWatch in one adapter. **Spec:** `docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md`, **rev 15** (approved), which records Paul's ruling **H27** (no pending creates; a refused request writes nothing and alerts; choose before registering; the four choosing doors; the alert limit is per console process) and settles §6's certificate readiness as the mirror's first load. Spec 1, for context: `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`. ## Rev 2: Astra round 1 and census ruling region-ha-2a Rev 2 answers Astra's review of rev 1 (`docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2A-1.md`, seven P2s, all accepted, with the coordinator's rulings now in spec rev 15's H27) and platform-1e's census ruling region-ha-2a (approved in principle with conditions, amadan #236). | # | Finding or condition | Rev 2 | Where | |---|---|---|---| | A1 | The no-capacity path left a registration behind; the test called `StartCreate` directly | Every create door chooses before `RegisterStart`: `beginClaimCreate` calls `chooseExecutorFirst`, which refuses before anything is registered and otherwise carries the choice to the create (`hostops.WithChosenExecutor`, honoured by `SelectExecutor`). Tested through the HTTP door and a recovery pass: no registration, no claim, nothing written by recovery | Task 11 | | A2 | H27 was missing from the other choosing doors | One refusal (`noCapacityError`, answered by `claimResponseRefusal`) on instance create, domain attach's first route (API and browser, through `attachClaimFirstRoute`, unchanged), standalone sidecar create, and the move door's destination. Each has an HTTP or door-level test. Sidecars attached to an instance choose nothing and are unchanged | Tasks 11, 12 | | A3 | Instance create's earlier region check refused before the chooser | With region HA on, an app-derived region is judged by the chooser's eligible set (`PlaceableBoxes`); an empty set is the no-room refusal. HTTP tests: every box stale (503 and an alert, not 400), and a misconfigured sibling beside an eligible box (the create goes ahead; 400 with region HA off, as on main) | Task 11 | | A4 | Move role filtering ran with region HA off | The move destination is filtered only through `placementChooser`, which exists only with region HA on; no roster is read otherwise. Test: host claims on, region HA off, a legacy destination and an unreadable roster, both main's answer | Task 12 | | A5 | `dialAddress` used private addresses with the reader's region HA off | The home map dials private only with the reader's own switch on (`homeMap.privateDial`). Tested both ways | Task 8 | | A6 | The alert limit did not survive a restart | H27 (rev 15) makes the limit per console process and accepts one extra alert after a restart. The limiter stays in memory; the wording says so | Task 11 | | A7 | Pending copy reached CLI help and docs | The copy gate is real: an id in `regionHACopyBuilt` must be an approved row. No pending row is ever marked approved for checking. Tasks 10 and 11 wait for their rows' approval | Tasks 1, 10, 11 | | B1 | `private_ip`: root-configured source; validated (IPv4, private, inside the region's VPC CIDR, not loopback or link-local); omitted and refused otherwise; routing data only; tests | The source is IMDS on AWS (the VPC's CIDR blocks from `network/interfaces/macs//vpc-ipv4-cidr-blocks`) or `host.env` (`CARLOS_PRIVATE_IP` with a required `CARLOS_PRIVATE_CIDR`). `edgesfile.PrivateAddress` validates on both sides: the writer omits what fails, and a reader dials a private address only when it passes against the reader's own published network. Tests: a malformed, a metadata-range and an out-of-CIDR address, each omitted and refused | Tasks 7, 8 | | B2 | `Server.boxRoster`: operator-only, read-only, the same roster source, no role feeds root's claim authority | `boxRoster` is called only by the two role doors, past `apiSystemRoller`; the move door no longer reads roles through it (it uses the chooser) | Task 10 | | B3 | The overlay loads last; values recomputed on the actual build base, after Plan 14 Parts 2 and 3 | Stated in the census section below; the recompute tooling is kept | Census | Task numbers moved: the move door is its own task (12), and the metric is now Task 13. ## Rev 3: Astra round 2 and census ruling region-ha-2a rev 2 Rev 3 answers Astra's round 2 (`docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2A-2.md`: rev 2's findings closed but one, and two new P2s, all accepted) and platform-1e's ruling region-ha-2a rev 2, which approved items 3 and 4 below with conditions. | # | Finding or condition | Rev 3 | Where | |---|---|---|---| | R1 | The move door's earlier placement checks (a destination with no edges file, or only blank public addresses) still answered 400 with no alert | With region HA on, `roomIn` asks the chooser before those checks: a destination the roster places boxes in, with no box that has room, is H27's refusal (503, RHA-C17, the alert). The instance create door's armed-placement gate gets the same treatment, the same defect. HTTP tests through the whole move door (both shapes, region HA on and off, and with room) and the create door | Tasks 11, 12 | | R2 | `handleAPIInstanceCreate` bound the environment (`bindEnvironmentCreate`, which can clear a seed's mark on a store) before `beginClaimCreate`'s early choice | The create door chooses before its first write (`chooseCreateFirst`, before the binding) and passes the choice through; `beginClaimCreate` keeps a choice the context carries and does not choose again. Tests: a refused create leaves the store's bytes unchanged and writes nothing, both when every box is stale and when the last box goes between the region check and the choice; with room, the same create takes the store | Task 11 | | R3 | The capacity metric took each region's AWS region from edges files | `ExecutorChooser.Capacity` also answers the AWS region the signed roster names for the placement, the mapping the chooser reads, and the metric goes there. Tests: a conflicting file (another AWS region, sorting last) does not move the count, unit and end to end | Task 13 | | C3 | Ruling item 3: `chooseExecutorFirst` takes no lease and makes no write; its executor is passed through and never chosen again; a refusal leaves nothing behind | `TestChooseFirstWritesNothingAndChoosesOnce`: no bucket or backend event while choosing, the choice is the authority's own choice for the same record, a carried choice is kept without calling the build, and `SelectExecutor` returns the carried choice. Every refusal test also checks that nothing was written (`wroteNothing`) | Task 11 | | C4 | Ruling item 4: the move room check only refuses, never picks or changes the destination | `moveRoomCheck` replaces rev 2's `placeableMoveTargets`: it refuses when no destination candidate is placeable and otherwise leaves the candidates as main's. `TestMoveRoomCheckOnlyRefuses`: with room the selection equals region HA off's; with none, the refusal; a legacy box beside a default one gives main's several-boxes refusal, not a pick | Task 12 | | C5 | Record-write re-digests stand only if no write site changes; a new write site needs its own line | No write site changes and none is added: the record-write census reports digest changes only (`handleAPIInstanceCreate`, `handleAPIInstanceMoveRequest`, `registerCLIRoutes`, `Directory.Replace`) | Census | One new census item: `Server.chooseCreateFirst` (Task 11), a new accepted-legacy unit whose one root read is the claim read `beginClaimCreate` makes, made before the binding. The census tooling's `unitdigest` build line is corrected (rev 2 built it over its own source directory). ## Rev 4: census ruling region-ha-2a item 5 Astra's round 3 (`docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2A-3.md`) found nothing and calls the plan ready to build. platform-1e approved item 5, `Server.chooseCreateFirst`, in the current structure (not the hook alternative), with one condition (amadan #236), which rev 4 meets: | # | Condition | Rev 4 | Where | |---|---|---|---| | C6 | `chooseCreateFirst` returns its choice with the claim observation it decided on; `beginClaimCreate` requires its own claim read to be that observation, and refuses or re-chooses otherwise, never proceeding on the old choice | `chooseCreateFirst` carries the choice and the observation (the claim's ETag, or its absence) in the context. `beginClaimCreate` passes its own read to `chooseExecutorFirst`, which keeps a carried choice only when that read is the same observation. Otherwise it **refuses**: `errChoiceStale`, which is `hostclaim.ErrConflict`, so the door answers main's retryable busy refusal (409, `Retry-After: 2`) before anything is registered or written. Refusing is the smaller change: no new copy, and no second choice to prove. `TestAStaleEarlyChoiceIsNeverUsed`: a claim write landing between the choice and begin is refused, with nothing chosen again, registered or written; without the check the test fails (begin goes on past `RegisterStart` with the old choice) | Task 11 | Census values move with it: new digests for `Server.chooseCreateFirst` and `Server.beginClaimCreate` (kinds and read sites unchanged), their `@legacy` fingerprints, `internal/console`'s `@coverage` row in Tasks 11 to 13, and the seal in Tasks 11 and 12. Open question 2 is closed as decided. ## Scope Spec 2 is four plans. This is **2a**. The others, and the seams 2a leaves them: | Plan | Builds | 2a's seam | |---|---|---| | 2b | Joining mode, the membership-confirmation worker, the console's enrolment pass | `lbhealth.Default.SetHosting` (Task 9): 2b's mode selection owns it, and joining mode leaves it false. `hostactors.Directory.EnrollAutomatic` already gives an admitted box its group's role in the enrolling write (Task 3) | | 2c | The `region-host` module's group mode, the start-up script, SSM | The health path `/.carlos/lb-health` on port 80 (Task 9). The metric `Carlos/Region EligibleBoxes`, dimension `Region` = the placement code, in each region's own CloudWatch, the AWS region the signed roster names for it (Task 13): 2c grants the platform box's role `cloudwatch:PutMetricData` conditioned on `cloudwatch:namespace = Carlos/Region`, and sets `CARLOS_REGION_METRICS=cloudwatch` on the console. The box role's IMDS access must cover `network/interfaces/macs/*` (Task 7), and the security group opens 443 from the region's VPC CIDR | | 2d | The §8 single-box fixes | With region HA on the move door already refuses a destination with no placeable box (Task 12), without narrowing its candidates; 2d's rework of the door (its multi-box regions, the source from the claim) builds on it | **Not in 2a:** forwarding of platform routes from a group box to the platform box (§6 "Platform routes"), the region ingress list in the console (§6), and every row of §8; those are 2d's or 2c's. ## Global Constraints - Everything new is dormant with `CARLOS_REGION_HA` off: with it off, every door, file, edge and box behaves exactly as on main. The health answer is not served, the role doors answer 404, the edges file publishes no `private_ip` and keeps its EIP hold, the home map dials public addresses only, the create and move doors choose as on main (no early choice, no room check, no roster read), and the capacity pass does not start. - Multi-box regions stay AWS-only until #299 (H26). Spec 2's new core files import no `github.com/aws/...` package; AWS parts are adapters in files named for AWS (§25). `region_ha_boundary_test.go` holds both, plus `s3compat.Options()` on every S3 client in the packages 2a touches. - Wire shapes are additive: `private_ip`, `private_cidr`, `mem_total_mib`, `disk_total_mib` (edges file), `group` and `role` (roster box), `role` (roster region) are all `omitempty`, and a reader that predates them is unaffected. - `private_ip` is routing data only: an edge dials it, and nothing grants anything on it. It comes only from IMDS or root-owned `host.env`, never from a file the agent can write, and it is published and dialled only when `edgesfile.PrivateAddress` passes it. - The roster's default role is stored as `""`. Role values: box `""`, `legacy`, `platform`; group `""`, `legacy`. Roles are read by the placement chooser alone and never feed root's claim authority. - §7's thresholds: free memory at least 15% of total (`boxchoice.MinMemPercent`), free disk at least 10% of the volume (`MinDiskPercent`), an edges file no older than 15 minutes (`MaxEdgesAge`). Choice: spread, then most `mem_free_mib`, then the P5-D2 hash. - H27: when no box qualifies, every door that chooses a box (instance create, domain attach's first route, standalone sidecar create, the move door's destination) answers 503 with RHA-C17, and alerts at most once per region an hour per console process. A door's own earlier placement checks answer a region with no room as H27 too. A create door chooses before its first write (registration, environment binding or any other) and passes that choice through; nothing chooses again. - §6's health answer: `GET` or `HEAD` `/.carlos/lb-health`, any host, port 80. 200 only when hosting, forwarding armed (map and key each loaded once, latched), the certificate mirror loaded once (latched), the edge's and the activator's beats within 60 seconds, and a home map refresh attempt completed within 5 minutes. - User-facing copy is in `docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md`, gated by `approved_region_ha_copy_test.go`. No pending row reaches code, CLI help or docs. No em dashes. - Go commands: `export GOFLAGS=-buildvcs=false GOCACHE=$TMPDIR/gocache`. Run tests with `nice -n 19` and a `-run` filter; never the whole `internal/host` suite (CI's `make test` is the full run). Tests that bind unix sockets fail inside the sandbox ("socket: operation not permitted"); that is the sandbox, not the change. - Census: every task that changes a non-test Go file under a package directory re-pins that directory's `@coverage` row in `internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv`. ## Census items for platform-1e (census ruling region-ha-2a) Approved in principle with conditions (amadan #236). The overlay asks for exactly these: | Task | Unit | Class | Change | Seal | |---|---|---|---|---| | 7 | `internal/host/edgesfile.go:edgesFileWriteClaims` | authority, F,J1 (kept) | Re-digest, class, task, basis and sites unchanged: with region HA on it reads the address source before every publication | stands | | 10 | `internal/console/apiboxroles.go:Server.boxRoster` | accepted-legacy, J4 root-input | New: the role doors' read of the signed roster (`Directory.Read`), operator-only, read-only | moves | | 11 | `internal/console/claim_create.go:Server.beginClaimCreate` | accepted-legacy, J4 root-input | Re-freeze: one call (`chooseExecutorFirst`) before `RegisterStart`, given this unit's own claim read, which keeps a carried choice only when that read is the observation the choice was decided on; its one read site unchanged (approved, ruling rev 2 item 3) | moves | | 11 | `internal/console/nocapacity.go:Server.chooseCreateFirst` | accepted-legacy, J4 root-input | New in rev 3, approved (ruling item 5) with its condition met in rev 4: the instance create door's choice before its first write. Its one root read is the host's claim (`hostclaim.Read`), the read `beginClaimCreate` makes, feeding only the placement choice; it takes no lease and writes nothing, and returns its choice with the claim observation it decided on | moves | | 12 | `internal/console/apiinstancemove.go:Server.handleAPIInstanceMoveRequest` | accepted-legacy | Re-freeze: two refusals added before any write, both no room in the destination (`roomIn` before the placement checks, `moveRoomCheck` through `selectMoveBoxes`); both only refuse; read sites unchanged, offsets moved (approved, ruling rev 2 item 4; the basis text names both refusals) | moves | `Server.attachClaimFirstRoute` is not changed: the first route's refusal comes through `beginClaimCreate` and `claimResponseRefusal`, which is not a root-read unit. Record-write census (B8) digests change for `Directory.Replace` (Task 3), `registerCLIRoutes` (Task 10), `handleAPIInstanceCreate` (Task 11) and `handleAPIInstanceMoveRequest` (Task 12). **No write site changes and none is added** (ruling rev 2): each re-digest is a change of the unit's other lines; the census reports no new or changed site. **Build base (condition 3).** `plan14-root-reads-region-ha-2a.tsv` loads last at build time. The values in this plan were computed at this branch's base, main `70e57449`; the build base will be later (after Plan 14 Parts 2 and 3 land), so recompute every value there with the tooling below: each `@coverage` row, each B8 digest, each unit's kinds and digest, each `@legacy` fingerprint, and each seal. The rows to write stay the same. ## Census tooling Create these in `$TMPDIR/census` (never in the repo), and delete the directory when the plan is done. `coverage.sh` prints a directory's `@coverage` row exactly as `rootReadProduction` computes it (run from the repo root): ````bash #!/bin/bash # usage: coverage.sh ... ; run from the repo root. Prints the Plan 14 # @coverage line of each directory, as rootReadProduction computes it. for dir in "$@"; do hash=$(for f in "$dir"/*; do [ -f "$f" ] || continue case "$f" in *_test.go) continue;; *.go|*.s|*.sh|*.service|*.path|*.socket|*.timer|*.conf|*.tf|*.tftpl|*.tmpl|*/go.mod|*/go.sum) ;; *) continue;; esac printf '%s\0%s\n' "${f#./}" "$(sha256sum "$f" | cut -d' ' -f1)" done | LC_ALL=C sort | head -c -1 | sha256sum | cut -d' ' -f1) printf '@coverage\t%s\t%s\n' "$dir" "$hash" done ```` `unitdigest/main.go` prints a top-level unit's digest (the sha256 of its declaration's source), the value both censuses pin. Build it with `cd $TMPDIR/census/unitdigest && GOFLAGS= go build -o ../unitdigest.bin main.go` (not `-o ../unitdigest`, which is the source directory: Go would write the binary inside it), then run `$TMPDIR/census/unitdigest.bin ` from the repo root: ````go // unitdigest prints the census digest (sha256 of the declaration's source) // of each named top-level unit: unitdigest [ ...]. // Unit names are the censuses': Func, Recv.Method, "type T", "var v". package main import ( "crypto/sha256" "encoding/hex" "fmt" "go/ast" "go/parser" "go/token" "os" ) func recv(e ast.Expr) string { for { switch x := e.(type) { case *ast.StarExpr: e = x.X case *ast.ParenExpr: e = x.X case *ast.IndexExpr: e = x.X case *ast.IndexListExpr: e = x.X case *ast.Ident: return x.Name default: return "?" } } } func main() { src, err := os.ReadFile(os.Args[1]) if err != nil { panic(err) } fset := token.NewFileSet() f, err := parser.ParseFile(fset, os.Args[1], src, 0) if err != nil { panic(err) } want := map[string]bool{} for _, u := range os.Args[2:] { want[u] = true } for _, d := range f.Decls { name := "" switch d := d.(type) { case *ast.FuncDecl: name = d.Name.Name if d.Recv != nil { name = recv(d.Recv.List[0].Type) + "." + name } case *ast.GenDecl: if d.Tok == token.IMPORT || d.Lparen.IsValid() { continue } switch s := d.Specs[0].(type) { case *ast.ValueSpec: name = d.Tok.String() + " " + s.Names[0].Name case *ast.TypeSpec: name = "type " + s.Name.Name } } if want[name] { sum := sha256.Sum256(src[fset.Position(d.Pos()).Offset:fset.Position(d.End()).Offset]) fmt.Printf("%s\t%s\t%s\n", os.Args[1], name, hex.EncodeToString(sum[:])) } } } ```` `legacyrow.py` prints the `@legacy` row and the entry row for a unit (`python3 legacyrow.py `): ````python import hashlib, sys # legacyrow.py : prints the @legacy row and the entry row. f,u,k,d,b = sys.argv[1:6] fields=[f,u,"legacy","accepted-legacy",b,k,d] fp=hashlib.sha256("\t".join(fields).encode()).hexdigest() print("\t".join(["@legacy",fp]+fields)) print("\t".join(fields)) ```` A unit's kinds string (for an unclassified or changed root read) is printed by a scratch test. Create `internal/provision/zz_census_scratch_test.go` with this content, run it, and delete it by exact path before committing: ````go package provision import ( "os" "strings" "testing" ) // Scratch: prints the census scan of the named units (RHA2A_UNITS, comma // separated "file:unit"). Never committed. func TestZZCensusScratchUnits(t *testing.T) { want := os.Getenv("RHA2A_UNITS") if want == "" { t.Skip() } got, err := rootReadProduction(censusRoot(t)) if err != nil { t.Fatal(err) } for _, e := range got.Entries { for _, w := range strings.Split(want, ",") { if e.File+":"+e.Unit == w { t.Logf("ENTRY\t%s\t%s\t%s\t%s", e.File, e.Unit, e.Kinds, e.Digest) } } } } ```` ````bash RHA2A_UNITS="internal/console/apiboxroles.go:Server.boxRoster" nice -n 19 go test ./internal/provision/ -run TestZZCensusScratchUnits -count=1 -v | grep ENTRY ```` The seal after a task's legacy rows is `rootReadLegacySeal` over every overlay in the test's load order. Add this to the scratch file to print it (the list is the load order at `70e57449`; on a newer base, copy the order from `TestPlan14RootReadCensus`): ````go func TestZZCensusScratchSeal(t *testing.T) { if os.Getenv("RHA2A_SEAL") == "" { t.Skip() } root := censusRoot(t) inv := rootReadInventory{Entries: map[string]classifiedRootRead{}, Coverage: map[string]string{}} files := []string{"plan14-root-reads.tsv"} if _, err := os.Stat(root + "/internal/host/movesheldrelease_production.go"); err == nil { files = append(files, "plan14-root-reads-moves.tsv") } files = append(files, "plan14-root-reads-merge.tsv", "plan14-root-reads-fix1.tsv", "plan14-root-reads-fix2.tsv", "plan14-root-reads-merge3.tsv", "plan14-root-reads-fix4.tsv", "plan14-root-reads-fix5.tsv", "plan14-root-reads-fix6.tsv", "plan14-root-reads-zddr3.tsv", "plan14-root-reads-hotfix281.tsv", "plan14-root-reads-zddr4.tsv", "plan14-root-reads-reserved285.tsv", "plan14-root-reads-292.tsv", "plan14-root-reads-census-ruling-275.tsv", "plan14-root-reads-census-ruling-owed.tsv", "plan14-root-reads-carlos-run.tsv", "plan14-root-reads-seeded-instances.tsv", "plan14-root-reads-hostfiles-cap-236.tsv", "plan14-root-reads-moves-overcap-cut-236.tsv", "plan14-root-reads-region-ha-2a.tsv") for i, f := range files { if err := loadRootReadInventory(root+"/internal/provision/testdata/"+f, &inv, i > 0); err != nil { t.Fatal(f, err) } } t.Logf("SEAL\t%s", rootReadLegacySeal(inv)) } ```` ````bash RHA2A_SEAL=1 nice -n 19 go test ./internal/provision/ -run TestZZCensusScratchSeal -count=1 -v | grep SEAL ```` The overlay is tab-separated: keep the tabs in every row this plan gives, or the loader refuses the row. The census checks every task runs: ````bash nice -n 19 go test ./internal/provision/ -run 'TestPlan9B8RecordWriteCensus|TestPlan14RootReadCensus$' -count=1 ```` ## Review Focus 1. **A roster write by another path must keep a box's role** (a quarantine, an activation, an enrolment): a box that silently drops back to `default` takes new instances the operator excluded. Task 3's `TestOtherRosterWritesKeepTheRole` pins the quarantine path; `Replace` compares every other field, so no write may change `Group`. 2. **IMDS answering blank on one pass** must publish no public address rather than an old one, and the box then drops out of `Eligible` until the next pass, never forwarding to an address it no longer holds. Task 7's `TestEdgesFilePublishesOnlyWhatTheSourceReports`. 3. **An edges file stamped by a box whose clock runs ahead** is believed for a minute of skew and no more; a stale or unreadable generation is never trusted. Task 4's `TestPlaceableAppliesEachRule` (the future and unreadable cases). 4. **A bucket outage after a box has armed** must leave its health answer at 200 (no termination for a shared outage), while a refresh stuck inside one attempt turns it 503. Task 2's `TestBucketOutageAfterArmingStaysHealthy` and Task 9's `TestHomeMapFeedsTheHealthState`. 5. **A client retrying a refused create in a loop** must not page the operator on every retry, and must leave nothing for recovery each time. Task 11's `TestCreateDoorsRefuseWithNoRoomBeforeRegistering` (three refusals, one page, no registration, no claim after a recovery pass). 6. **A create refused for want of room must change nothing, even when the fleet changes under it.** The choice comes before the environment binding, and the create names the box that was chosen. Task 11's `TestRefusedCreateLeavesTheEnvironmentsStoreAlone` (the last box goes between the region check and the choice) and `TestChooseFirstWritesNothingAndChoosesOnce`. Nor may a create go on with a choice whose claim changed: `TestAStaleEarlyChoiceIsNeverUsed`. 7. **The move door's room check only refuses.** A regression that narrows the candidates would quietly pick a destination main's selection refuses (a region with several boxes). Task 12's `TestMoveRoomCheckOnlyRefuses`. ## Dry run Every code block below was written into this branch on main `70e57449` (Tasks 1 to 10 at `aa5b6b07` for rev 2, replayed unchanged for rev 3; Tasks 11 to 13 rewritten at `0a31a455` for rev 3, and again at `e5a3a3de` for rev 4), task by task in order, built, vetted with `GOOS=linux` and `GOOS=darwin` (and built with `GOOS=windows`), and its tests run as each task's steps say; the census values are those runs'. The copy gate was checked without approving anything: with every row pending, `TestRegionHABuiltCopyIsApplied`, `TestRegionHACopyRecordIsComplete` and `TestRegionHACopyHasNoEmDash` pass, and `TestRegionHABuiltCopyIsApproved` fails naming exactly the built ids (RHA-C1 to RHA-C16 after Task 10, RHA-C1 to RHA-C19 after Task 11). That failure is what an implementer sees until Paul approves the rows. Each rev 3 regression test was first run against rev 2's code and failed (the store test's refused create wrote the store; the move door answered 400; a legacy box beside a default one was picked; the metric went to the edges file's region). Rev 4's `TestAStaleEarlyChoiceIsNeverUsed` was run with the observation check removed and failed. The whole `TestClaimConsoleMutationInvariant` ran once on the final tree. The plan text was then replayed onto a clean export, task by task, and every intermediate state built and passed both censuses. The code was then removed. --- ### Task 1: The copy record and its gate **Files:** - Create: `docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md` - Create: `approved_region_ha_copy_test.go` **Interfaces:** - Consumes: the run and moves gates' helpers at the repo root (`parseCopyTable`, `checkMovesInventory`, `checkMovesNoEmDash`, `checkRunBuilt`, `checkRunApplied`, `builtRows`, `reportAll`, `copyRow`). - Produces: `regionHACopyBuilt []string`, which Tasks 10 and 11 extend with the ids whose text they write; the record's ids `RHA-C1` to `RHA-C19`. Every user-facing string 2a writes is drafted here first, all `pending`. The gate is real: an id in `regionHACopyBuilt` must be an `approved` row, and its text must be in the file the row names. Nothing marks a pending row approved for checking. - [ ] **Step 1: Write the record** `docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md`: ````markdown # Region HA plan 2a: copy record (pending Paul's review) Every user-facing string plan 2a (`2026-10-16-region-ha-2a-core.md`) writes: console refusals, CLI output and usage lines, the operator alert, and the docs prose. The gate is `approved_region_ha_copy_test.go` at the repo root, which reuses the moves gate's parser, so the format is the moves record's: one table per heading, columns id, file, text, status. The rules: - A row is `pending` until Paul approves it in the copy review (the `copy-review` skill). His edits are applied to the text here verbatim and the row becomes `approved`. - No pending string reaches code, CLI help or docs. The task that writes a row's text adds its id to `regionHACopyBuilt` in the gate in the same commit, and from then on the gate fails unless the row is `approved` and its text is in the file it names. Tasks 10 and 11 therefore wait for their rows' approval. - In a Go file the text is one string literal, or one `fmt.Sprintf` format, equal to the row with `%s` where the row has a ``. A row whose file cell says "a fragment" is a piece of a larger literal (the `carlos system` usage text). - No em dashes. A full stop and a new sentence instead. The role doors' refusals start lower case, as the console's other CLI refusals do; the no-capacity refusal is shared with the browser's domain door, so it is a capitalised sentence, as `claimResponseUnavailable` is. Placeholders: `` a box's roster reference or label; `` a role name (default, legacy or platform); `` a placement code or the home region's name. ## Box and group roles (Task 10) | id | file | text | status | |---|---|---|---| | RHA-C1 | internal/console/apiboxroles.go | role must be default, legacy or platform | pending | | RHA-C2 | internal/console/apiboxroles.go | a group's role must be default or legacy | pending | | RHA-C3 | internal/console/apiboxroles.go | could not read the box roster. Try again. | pending | | RHA-C4 | internal/console/apiboxroles.go | the box roster changed while the role was being saved, and nothing was changed. Try again. | pending | | RHA-C5 | internal/console/apiboxroles.go | could not save the role. Try again. | pending | | RHA-C6 | internal/console/apiboxroles.go | no enrolled box is named | pending | | RHA-C7 | internal/console/apiboxroles.go | has no group | pending | | RHA-C8 | internal/console/apiboxroles.go | is retired, so it takes no role | pending | | RHA-C9 | system_roles.go | now has the role | pending | | RHA-C10 | system_roles.go | the group in now has the role | pending | | RHA-C11 | system.go (a fragment of the usage) | whether new instances may be placed on a box. A legacy or | pending | | RHA-C12 | system.go (a fragment of the usage) | platform box keeps the instances it has and takes no new ones | pending | | RHA-C13 | system.go (a fragment of the usage) | the same for every box a region's group has enrolled, and for | pending | | RHA-C14 | system.go (a fragment of the usage) | every box it enrolls later | pending | | RHA-C15 | docs/site/cli.md | Box and group roles | pending | | RHA-C16 | docs/site/cli.md | carlos system box role says whether new instances may be placed on a box. A box with the default role can take them. A legacy or platform box keeps the instances it has and takes no new ones: no new instance, relocation or move lands on it. carlos system group role sets the role of every box a region's group has enrolled, and of every box it enrolls later. Both work only on a console with region HA on. | pending | ## No box has room (Task 11) One refusal for every door that chooses a box (H27): instance create, domain attach, standalone sidecar create and the move door's destination. | id | file | text | status | |---|---|---|---| | RHA-C17 | internal/console/nocapacity.go | No box in has room right now. Try again in a few minutes. | pending | | RHA-C18 | internal/console/nocapacity.go (the operator alert's subject) | No box in has room | pending | | RHA-C19 | internal/console/nocapacity.go (the operator alert's body) | A request in was refused because no box there can take it. Every box is full, stale or excluded by its role. Add a box, or free memory or disk on one, and check the eligible-capacity alarm for . | pending | ```` - [ ] **Step 2: Write the gate** `approved_region_ha_copy_test.go`: ````go // approved_region_ha_copy_test.go package main import ( "os" "regexp" "strconv" "testing" ) // The copy gate of region HA plan 2a (Task 1). The record is a markdown // table in the moves record's format, read with the moves gate's parser // and checked with the run gate's helpers, so "approved" and "applied" // mean what they mean there: // // - TestRegionHACopyRecordIsComplete: the record names exactly RHA-C1 to // RHA-C19; // - TestRegionHACopyHasNoEmDash; // - TestRegionHABuiltCopyIsApproved: an id in regionHACopyBuilt is an // approved row. No pending string reaches code, CLI help or docs; // - TestRegionHABuiltCopyIsApplied: a built row's text is in the file it // names. const regionHACopyRecordPath = "docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md" var regionHACopyIDPattern = regexp.MustCompile(`^RHA-C[0-9]+$`) // regionHACopyIDs is the record's inventory: RHA-C1 to RHA-C19, pinned. var regionHACopyIDs = func() []string { ids := make([]string, 0, 19) for i := 1; i <= 19; i++ { ids = append(ids, "RHA-C"+strconv.Itoa(i)) } return ids }() // regionHACopyBuilt is every id whose text the code or docs carry. The // task that writes a row's text adds its id here in the same commit, once // Paul has approved the row. var regionHACopyBuilt = []string{} func mustParseRegionHARecord(t *testing.T) []copyRow { t.Helper() raw, err := os.ReadFile(regionHACopyRecordPath) if err != nil { t.Fatal(err) } rows, err := parseCopyTable(string(raw), regionHACopyRecordPath, regionHACopyIDPattern, "an RHA-C") if err != nil { t.Fatal(err) } return rows } func TestRegionHACopyRecordIsComplete(t *testing.T) { reportAll(t, checkMovesInventory(mustParseRegionHARecord(t), regionHACopyIDs)) } func TestRegionHACopyHasNoEmDash(t *testing.T) { reportAll(t, checkMovesNoEmDash(mustParseRegionHARecord(t))) } func TestRegionHABuiltCopyIsApproved(t *testing.T) { reportAll(t, checkRunBuilt(mustParseRegionHARecord(t), regionHACopyBuilt)) } func TestRegionHABuiltCopyIsApplied(t *testing.T) { reportAll(t, checkRunApplied(builtRows(mustParseRegionHARecord(t), regionHACopyBuilt), ".")) } ```` - [ ] **Step 3: Run it** Run: `nice -n 19 go test . -run 'RegionHACopy|RegionHABuilt' -count=1` Expected: PASS (nothing is built yet). - [ ] **Step 4: Put the record in front of Paul** Run the `copy-review` skill over the record's rows. Apply his edits verbatim to the record, mark each row he approves `approved`, and commit that on its own. Tasks 10 and 11 wire their rows only once they are approved, with Paul's text: where his text differs from a draft below, use his in the code too. Tasks 2 to 9 do not wait. - [ ] **Step 5: Commit** ```bash git add docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md approved_region_ha_copy_test.go git commit -m "region HA 2a: the copy record and its gate" ``` No package envelope changes: neither file is a non-test Go file. --- ### Task 2: The health state (`internal/lbhealth`), the provider-boundary tests and the census overlay **Files:** - Create: `internal/lbhealth/lbhealth.go`, `internal/lbhealth/lbhealth_test.go` - Create: `region_ha_boundary_test.go` - Create: `internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv` - Modify: `internal/provision/plan14_root_reads_test.go` (load the new overlay last) **Interfaces:** - Produces (`internal/lbhealth`): `const Path = "/.carlos/lb-health"`; `HeartbeatWindow = 60 * time.Second`; `RefreshWindow = 5 * time.Minute`; the condition names `CondHosting`, `CondForwarding`, `CondCerts`, `CondEdge`, `CondActivator`, `CondRefresh`; `type State`; `var Default = &State{}`; methods (all safe on a nil `*State`) `SetHosting(bool)`, `Hosting() bool`, `MapLoaded()`, `KeyLoaded()`, `CertsLoaded()`, `EdgeBeat(time.Time)`, `RefreshAttempted(time.Time)`, `SetActivator(func() time.Time)`, `Failing(now time.Time) []string`; `func Serve(w http.ResponseWriter, r *http.Request, s *State, now time.Time) bool`. - Produces (repo root): `regionHACoreFiles`, `regionHAAdapterFiles`, `regionHAPackages`, `regionHAS3Exceptions` in `region_ha_boundary_test.go`, which later tasks extend. - Produces: the overlay file every later task's census step edits, and the `seal = ...` line after it, which Tasks 10, 11 and 12 change. - [ ] **Step 1: Write the failing tests** `internal/lbhealth/lbhealth_test.go`: ````go package lbhealth import ( "net/http" "net/http/httptest" "reflect" "strings" "testing" "time" ) var t0 = time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) // healthy is a state with every condition met at t0. func healthy() *State { s := &State{} s.SetHosting(true) s.MapLoaded() s.KeyLoaded() s.CertsLoaded() s.EdgeBeat(t0) s.RefreshAttempted(t0) s.SetActivator(func() time.Time { return t0 }) return s } func TestFailingNamesEachCondition(t *testing.T) { if got := healthy().Failing(t0); len(got) != 0 { t.Fatalf("healthy box fails %v", got) } if got := (&State{}).Failing(t0); !reflect.DeepEqual(got, []string{CondHosting, CondForwarding, CondCerts, CondEdge, CondActivator, CondRefresh}) { t.Fatalf("zero state fails %v", got) } var nilState *State if got := nilState.Failing(t0); len(got) != 6 { t.Fatalf("nil state fails %v", got) } cases := []struct { name string spoil func(*State) now time.Time want string }{ {"joining", func(s *State) { s.SetHosting(false) }, t0, CondHosting}, {"map never loaded", func(s *State) { s.mapLoaded.Store(false) }, t0, CondForwarding}, {"key never loaded", func(s *State) { s.keyLoaded.Store(false) }, t0, CondForwarding}, {"no certificate cache", func(s *State) { s.certs.Store(false) }, t0, CondCerts}, {"edge beat stale", func(s *State) { s.EdgeBeat(t0.Add(-61 * time.Second)) }, t0, CondEdge}, {"activator beat stale", func(s *State) { s.SetActivator(func() time.Time { return t0.Add(-61 * time.Second) }) }, t0, CondActivator}, {"no activator", func(s *State) { s.SetActivator(nil) }, t0, CondActivator}, {"refresh stuck", func(s *State) { s.RefreshAttempted(t0.Add(-5*time.Minute - time.Second)) }, t0, CondRefresh}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { s := healthy() c.spoil(s) if got := s.Failing(c.now); !reflect.DeepEqual(got, []string{c.want}) { t.Fatalf("Failing = %v, want [%s]", got, c.want) } }) } // The windows are inclusive: a beat exactly 60s old and a refresh // exactly 5 minutes old still count. s := healthy() s.EdgeBeat(t0.Add(-HeartbeatWindow)) s.RefreshAttempted(t0.Add(-RefreshWindow)) if got := s.Failing(t0); len(got) != 0 { t.Fatalf("boundary fails %v", got) } } // A bucket outage after arming: refreshes keep failing, but each failed // attempt is still a completed attempt, and the latches hold. The box // stays healthy (A2-3, A3-3). func TestBucketOutageAfterArmingStaysHealthy(t *testing.T) { s := healthy() for i := 1; i <= 20; i++ { now := t0.Add(time.Duration(i) * 30 * time.Second) s.RefreshAttempted(now) // a failed attempt: no latch is re-set s.EdgeBeat(now) s.SetActivator(func() time.Time { return now }) if got := s.Failing(now); len(got) != 0 { t.Fatalf("after %d failed refreshes: %v", i, got) } } } func TestServe(t *testing.T) { get := func(s *State, method, target, host string) (*httptest.ResponseRecorder, bool) { w := httptest.NewRecorder() r := httptest.NewRequest(method, target, nil) r.Host = host return w, Serve(w, r, s, t0) } w, ok := get(healthy(), http.MethodGet, Path, "10.0.1.7") if !ok || w.Code != http.StatusOK || w.Body.String() != "ok\n" || w.Header().Get("Cache-Control") != "no-store" { t.Fatalf("healthy GET: ok=%v %d %q %v", ok, w.Code, w.Body.String(), w.Header()) } w, ok = get(healthy(), http.MethodHead, Path, "any.example.com") if !ok || w.Code != http.StatusOK || w.Body.Len() != 0 { t.Fatalf("healthy HEAD: ok=%v %d %q", ok, w.Code, w.Body.String()) } joining := healthy() joining.SetHosting(false) w, ok = get(joining, http.MethodGet, Path, "10.0.1.7") if !ok || w.Code != http.StatusServiceUnavailable || !strings.Contains(w.Body.String(), CondHosting) { t.Fatalf("joining GET: ok=%v %d %q", ok, w.Code, w.Body.String()) } for _, c := range []struct{ method, target string }{ {http.MethodPost, Path}, {http.MethodGet, Path + "/"}, {http.MethodGet, "/.carlos/lb-healthz"}, {http.MethodGet, "/"}, } { if _, ok := get(healthy(), c.method, c.target, "10.0.1.7"); ok { t.Fatalf("%s %s answered", c.method, c.target) } } } func TestNilStateIgnoresUpdates(t *testing.T) { var s *State s.SetHosting(true) s.MapLoaded() s.KeyLoaded() s.CertsLoaded() s.EdgeBeat(t0) s.RefreshAttempted(t0) s.SetActivator(func() time.Time { return t0 }) if s.Hosting() { t.Fatal("nil state hosting") } } ```` `region_ha_boundary_test.go`: ````go package main import ( "go/ast" "go/parser" "go/token" "os" "path/filepath" "sort" "strconv" "strings" "testing" ) // Region HA spec 2 §25: the provider boundary. Spec 2's core names no // cloud provider; its AWS parts are adapters in files named for AWS. Each // plan task that adds a core file lists it here in the same commit. // regionHACoreFiles are spec 2's provider-neutral files. None may import // an AWS package. var regionHACoreFiles = []string{ "internal/lbhealth/lbhealth.go", } // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so // a reader can see where the provider lives. var regionHAAdapterFiles = []string{} // regionHAPackages are the packages spec 2 touches. Every S3 client built // in them goes through s3compat.Options(), so CARLOS_S3_ENDPOINT applies. var regionHAPackages = []string{ "internal/activator", "internal/boxchoice", "internal/console", "internal/edge", "internal/edgesfile", "internal/host", "internal/hostactors", "internal/hostops", "internal/lbhealth", } // regionHAS3Exceptions are S3 clients on main that skip s3compat, each // with the discussion that fixes it outside spec 2. A listed file that has // since been fixed passes. var regionHAS3Exceptions = map[string]string{ "internal/host/claimproduction_marker.go": "#298", } func TestRegionHACoreImportsNoAWS(t *testing.T) { for _, f := range regionHACoreFiles { file, err := parser.ParseFile(token.NewFileSet(), f, nil, parser.ImportsOnly) if err != nil { t.Fatalf("%s: %v", f, err) } for _, imp := range file.Imports { path, _ := strconv.Unquote(imp.Path.Value) if strings.HasPrefix(path, "github.com/aws/") { t.Errorf("%s imports %s: spec 2's core names no provider (§25); move it to an AWS adapter", f, path) } } } } func TestRegionHAAdaptersAreNamedForAWS(t *testing.T) { for _, f := range regionHAAdapterFiles { if _, err := os.Stat(f); err != nil { t.Errorf("%s: %v", f, err) } if !strings.Contains(filepath.Base(f), "aws") && !strings.Contains(filepath.Dir(f), "aws") { t.Errorf("%s is an AWS adapter but its name does not say so", f) } } } // s3NewFromConfigCalls returns the line of each s3.NewFromConfig call in // src that does not pass s3compat.Options() as an argument. func s3NewFromConfigCalls(t *testing.T, path string) []int { t.Helper() fset := token.NewFileSet() f, err := parser.ParseFile(fset, path, nil, 0) if err != nil { t.Fatalf("%s: %v", path, err) } s3Names, compatNames := map[string]bool{}, map[string]bool{} for _, imp := range f.Imports { p, _ := strconv.Unquote(imp.Path.Value) name := filepath.Base(p) if imp.Name != nil { name = imp.Name.Name } switch p { case "github.com/aws/aws-sdk-go-v2/service/s3": s3Names[name] = true case "github.com/carlosframework/platform/internal/s3compat": compatNames[name] = true } } var bad []int ast.Inspect(f, func(n ast.Node) bool { call, ok := n.(*ast.CallExpr) if !ok { return true } sel, ok := call.Fun.(*ast.SelectorExpr) if !ok || sel.Sel.Name != "NewFromConfig" { return true } pkg, ok := sel.X.(*ast.Ident) if !ok || !s3Names[pkg.Name] { return true } for _, arg := range call.Args { inner, ok := arg.(*ast.CallExpr) if !ok { continue } isel, ok := inner.Fun.(*ast.SelectorExpr) if !ok || isel.Sel.Name != "Options" { continue } if id, ok := isel.X.(*ast.Ident); ok && compatNames[id.Name] { return true } } bad = append(bad, fset.Position(call.Pos()).Line) return true }) return bad } func TestRegionHAS3ClientsUseS3Compat(t *testing.T) { for _, dir := range regionHAPackages { paths, err := filepath.Glob(filepath.Join(dir, "*.go")) if err != nil { t.Fatal(err) } sort.Strings(paths) for _, p := range paths { if strings.HasSuffix(p, "_test.go") { continue } if _, ok := regionHAS3Exceptions[filepath.ToSlash(p)]; ok { continue } for _, line := range s3NewFromConfigCalls(t, p) { t.Errorf("%s:%d builds an S3 client without s3compat.Options() (§25): CARLOS_S3_ENDPOINT would not apply", p, line) } } } } // The checker itself: a bare client is caught, one with Options() is not, // whatever the import names. func TestRegionHAS3CheckerCatchesABareClient(t *testing.T) { dir := t.TempDir() write := func(name, body string) string { p := filepath.Join(dir, name) if err := os.WriteFile(p, []byte(body), 0o644); err != nil { t.Fatal(err) } return p } bare := write("bare.go", `package x import awss3 "github.com/aws/aws-sdk-go-v2/service/s3" func f(cfg any) { _ = awss3.NewFromConfig(cfg) } `) good := write("good.go", `package x import ( "github.com/aws/aws-sdk-go-v2/service/s3" "github.com/carlosframework/platform/internal/s3compat" ) func f(cfg any) { _ = s3.NewFromConfig(cfg, s3compat.Options()) } `) if got := s3NewFromConfigCalls(t, bare); len(got) != 1 { t.Fatalf("bare client: %v", got) } if got := s3NewFromConfigCalls(t, good); len(got) != 0 { t.Fatalf("compat client flagged: %v", got) } } ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/lbhealth/ -count=1` Expected: FAIL, the package does not build (`undefined: State`). `go test . -run 'TestRegionHA(Core|Adapters|S3)'` fails the same way on `internal/lbhealth/lbhealth.go` not existing. - [ ] **Step 3: Write the package** `internal/lbhealth/lbhealth.go`: ````go // Package lbhealth is a box's own liveness, as the load balancer's health // check reads it (region HA spec 2 §6). Every fact here is this box's own: // a flag or a timestamp that one of its own loops sets. Nothing here reads // a disk, a bucket or the network, so an outage of something the boxes // share (the deployment bucket) never turns a working box unhealthy and // never gets it terminated. // // The edge answers GET or HEAD /.carlos/lb-health on port 80 from State. // The loops that feed it are the agent's: the claim agent sets hosting, // the home map latches forwarding and stamps its refresh attempts, the // certificate mirror latches its first load, and the edge's registry watch // and the activator's sweeper each beat. package lbhealth import ( "io" "net/http" "strings" "sync" "sync/atomic" "time" ) // Path is the health check's path, matched exactly. const Path = "/.carlos/lb-health" // HeartbeatWindow is how recent the edge's and the activator's beats must // be. Both loops beat every 30 seconds or faster. const HeartbeatWindow = 60 * time.Second // RefreshWindow is how recent the home map's last completed refresh // attempt must be: its 30-second interval plus its two-minute request // timeout, with room to spare. A failed attempt counts, so a bucket outage // where every attempt fails fast keeps this satisfied; a worker stuck // inside one attempt does not. const RefreshWindow = 5 * time.Minute // The condition names a 503 lists, in this order. const ( CondHosting = "hosting" CondForwarding = "forwarding" CondCerts = "certificates" CondEdge = "edge" CondActivator = "activator" CondRefresh = "refresh" ) // State is one box's liveness facts. The zero value is a box that has // proven nothing, and answers 503. A nil *State ignores every update. type State struct { hosting atomic.Bool mapLoaded atomic.Bool keyLoaded atomic.Bool certs atomic.Bool edgeBeat atomic.Int64 // unix nanoseconds; 0 is never refreshed atomic.Int64 // unix nanoseconds; 0 is never mu sync.RWMutex activator func() time.Time } // Default is the process's state. The agent runs the edge and the // activator in one process, so the claim wiring, the edge and the // activator share it. var Default = &State{} // SetHosting records whether the agent is in hosting mode: enrolled, with // its guard armed (§5). Region HA spec 2's plan 2b adds joining mode, whose // mode selection is what sets this false; until then the claim agent sets // it true once its claim wiring is running (ClaimWiring.startRegional). func (s *State) SetHosting(on bool) { if s != nil { s.hosting.Store(on) } } // Hosting reports what SetHosting last recorded. func (s *State) Hosting() bool { return s != nil && s.hosting.Load() } // MapLoaded latches the first successful load of the forwarding map. func (s *State) MapLoaded() { if s != nil { s.mapLoaded.Store(true) } } // KeyLoaded latches the first successful load of a mesh key. func (s *State) KeyLoaded() { if s != nil { s.keyLoaded.Store(true) } } // CertsLoaded latches the first successful load of the certificate mirror // into the local cache the TLS handshake reads. func (s *State) CertsLoaded() { if s != nil { s.certs.Store(true) } } // EdgeBeat records that the edge's registry watch completed a poll at t. func (s *State) EdgeBeat(t time.Time) { if s != nil { s.edgeBeat.Store(t.UnixNano()) } } // RefreshAttempted records that the home map's refresh finished an // attempt at t, whether it succeeded or failed. func (s *State) RefreshAttempted(t time.Time) { if s != nil { s.refreshed.Store(t.UnixNano()) } } // SetActivator gives the state the activator's own beat: the time its // sweeper last finished a sweep. nil means there is no activator, which // fails the activator condition. func (s *State) SetActivator(last func() time.Time) { if s == nil { return } s.mu.Lock() s.activator = last s.mu.Unlock() } func (s *State) activatorBeat() time.Time { s.mu.RLock() last := s.activator s.mu.RUnlock() if last == nil { return time.Time{} } return last() } // within reports whether at is set and no older than window at now. func within(at, now time.Time, window time.Duration) bool { return !at.IsZero() && now.Sub(at) <= window } func stamp(n int64) time.Time { if n == 0 { return time.Time{} } return time.Unix(0, n) } // Failing lists the conditions that do not hold at now, in the order of // the Cond names. An empty list is a healthy box. A nil *State fails // every condition. func (s *State) Failing(now time.Time) []string { if s == nil { return []string{CondHosting, CondForwarding, CondCerts, CondEdge, CondActivator, CondRefresh} } var out []string if !s.hosting.Load() { out = append(out, CondHosting) } if !s.mapLoaded.Load() || !s.keyLoaded.Load() { out = append(out, CondForwarding) } if !s.certs.Load() { out = append(out, CondCerts) } if !within(stamp(s.edgeBeat.Load()), now, HeartbeatWindow) { out = append(out, CondEdge) } if !within(s.activatorBeat(), now, HeartbeatWindow) { out = append(out, CondActivator) } if !within(stamp(s.refreshed.Load()), now, RefreshWindow) { out = append(out, CondRefresh) } return out } // Serve answers the health check when r is one, and reports whether it // did. It matches as narrowly as the steering probe: the exact path, GET // or HEAD, any host. It answers 200 "ok" when every condition holds and // 503 naming the failing conditions otherwise, never cached. It only reads // s. func Serve(w http.ResponseWriter, r *http.Request, s *State, now time.Time) bool { if r.URL.Path != Path || (r.Method != http.MethodGet && r.Method != http.MethodHead) { return false } body, code := "ok\n", http.StatusOK if failing := s.Failing(now); len(failing) > 0 { body, code = "unavailable: "+strings.Join(failing, ",")+"\n", http.StatusServiceUnavailable } w.Header().Set("Content-Type", "text/plain; charset=utf-8") w.Header().Set("Cache-Control", "no-store") w.WriteHeader(code) if r.Method == http.MethodGet { _, _ = io.WriteString(w, body) } return true } ```` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/lbhealth/ -count=1 && nice -n 19 go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1` Expected: PASS. The S3 call-site test passes with `internal/host/claimproduction_marker.go` excepted for #298. - [ ] **Step 5: Census** Create the overlay: `internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv`: ````text # Plan 14 census overlay region-ha-2a (region HA spec 2, plan 2a: docs/superpowers/plans/2026-10-16-region-ha-2a-core.md), loaded last. # Task 2: the new internal/lbhealth package (the load balancer health state). Envelope-only: no root-read unit changed or was added, nothing runs as root that did not before, and no registry, /data, release or claim read was added, so the legacy seal stands. @coverage internal/lbhealth ad1211a01a4eb3d02cd8ed95cbff0f8b17b3976ce9c0279ebd2cae2f20c07124 ```` Load it last in `TestPlan14RootReadCensus`: ````diff --- a/internal/provision/plan14_root_reads_test.go +++ b/internal/provision/plan14_root_reads_test.go @@ -759,6 +759,17 @@ t.Fatal(err) } seal = "fc38db001ebd8825709a8ca9c41cc0e8f165c1b1967e5967347dc9bd3556afec" + if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 { + t.Fatal(errs) + } + // Census overlay region-ha-2a (region HA spec 2, plan 2a), loaded last. + // Its header says what each plan task added. The accepted-legacy rows + // it adds are census ruling region-ha-2a's (platform-1e), and move the + // seal. + if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv"), &inv, true); err != nil { + t.Fatal(err) + } + seal = "fc38db001ebd8825709a8ca9c41cc0e8f165c1b1967e5967347dc9bd3556afec" errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) errs = append(errs, checkRootReadDeclarations(got, inv)...) if len(errs) > 0 { ```` Run the census checks. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/lbhealth region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv internal/provision/plan14_root_reads_test.go git commit -m "region HA 2a: the load balancer health state and the provider boundary tests" ``` --- ### Task 3: Box and group roles in the roster **Files:** - Create: `internal/hostactors/roles.go`, `internal/hostactors/roles_test.go` - Modify: `internal/hostactors/roster.go` (`Box.Group`, `Box.Role`; `Validate`; `Replace` lets the role change) - Modify: `internal/hostactors/enrollment_confirmation.go` (`Region.Role`) - Modify: `internal/hostactors/enroll.go` (automatic enrollment inherits the group's role) - Modify: `region_ha_boundary_test.go`, `internal/provision/plan9_b8_census_test.go`, the overlay **Interfaces:** - Consumes: `Directory.Replace`, `Directory.Read`, `Clone`, `ErrRoster`. - Produces: `RoleDefault = ""`, `RoleLegacy = "legacy"`, `RolePlatform = "platform"`; `var ErrRole`; `ParseBoxRole(string) (string, error)` (default, legacy, platform); `ParseGroupRole(string) (string, error)` (default, legacy); `RoleName(string) string`; `func (d Directory) SetBoxRole(ctx context.Context, sg release.Signer, o Observed, ref, role string) (Observed, error)`; `func GroupMembers(r Roster, region string) []string`; `func (d Directory) SetGroupRole(ctx context.Context, sg release.Signer, o Observed, region, role string) (Observed, []string, error)`; fields `Box.Group`, `Box.Role` (json `group`, `role`, omitempty), `Region.Role` (json `role`, omitempty). A role is written only by the console (the only root signer). `Replace` keeps comparing every other field of an enrolled box, so `Group` never changes once set. Automatic enrollment (2b's caller) sets `Group` and `Role` from the region's group in the same write that admits the box (A11-1). - [ ] **Step 1: Write the failing tests** `internal/hostactors/roles_test.go`: ````go package hostactors import ( "context" "crypto" "crypto/rand" "crypto/rsa" "crypto/sha256" "crypto/x509" "encoding/base64" "encoding/json" "errors" "reflect" "strings" "testing" "time" "github.com/carlosframework/platform/internal/boxid" "github.com/carlosframework/platform/internal/release" ) func TestParseRoles(t *testing.T) { for in, want := range map[string]string{"default": RoleDefault, "legacy": RoleLegacy, "platform": RolePlatform, " legacy ": RoleLegacy} { if got, err := ParseBoxRole(in); err != nil || got != want { t.Fatalf("ParseBoxRole(%q) = %q, %v", in, got, err) } } for _, in := range []string{"", "Legacy", "draining", "none"} { if _, err := ParseBoxRole(in); !errors.Is(err, ErrRole) { t.Fatalf("ParseBoxRole(%q) accepted", in) } } if _, err := ParseGroupRole("platform"); !errors.Is(err, ErrRole) { t.Fatal("a group took the platform role") } if got, err := ParseGroupRole("legacy"); err != nil || got != RoleLegacy { t.Fatalf("ParseGroupRole(legacy) = %q, %v", got, err) } if RoleName(RoleDefault) != "default" || RoleName(RolePlatform) != "platform" { t.Fatal("RoleName") } } func TestBoxRoleIsSignedAndPersists(t *testing.T) { ctx := context.Background() d, sg, o := fixture(t) next, err := d.SetBoxRole(ctx, sg, o, "box", RoleLegacy) if err != nil { t.Fatal(err) } read, err := d.Read(ctx) if err != nil || read.Roster.Boxes["box"].Role != RoleLegacy || read.ETag != next.ETag { t.Fatalf("read back: %+v %v", read.Roster.Boxes["box"], err) } // The same role again writes nothing. same, err := d.SetBoxRole(ctx, sg, next, "box", RoleLegacy) if err != nil || same.ETag != next.ETag { t.Fatalf("idempotent set wrote: %v", err) } // A stale observation loses the CAS. if _, err = d.SetBoxRole(ctx, sg, o, "box", RolePlatform); err == nil { t.Fatal("stale roster accepted") } back, err := d.SetBoxRole(ctx, sg, next, "box", RoleDefault) if err != nil || back.Roster.Boxes["box"].Role != RoleDefault { t.Fatalf("back to default: %+v %v", back.Roster.Boxes["box"], err) } if _, err = d.SetBoxRole(ctx, sg, back, "nobody", RoleLegacy); !errors.Is(err, ErrRoster) { t.Fatalf("unknown box: %v", err) } if _, err = d.SetBoxRole(ctx, sg, back, "box", "boss"); !errors.Is(err, ErrRole) { t.Fatalf("unknown role: %v", err) } // A retired box keeps no role. retired := Clone(back.Roster) b := retired.Boxes["box"] b.State = "retired" retired.Boxes["box"] = b gone, err := d.Replace(ctx, sg, back, retired) if err != nil { t.Fatal(err) } if _, err = d.SetBoxRole(ctx, sg, gone, "box", RoleLegacy); !errors.Is(err, ErrRoster) { t.Fatalf("retired box: %v", err) } } func TestRosterRefusesUnknownRolesAndAGroupChange(t *testing.T) { ctx := context.Background() d, sg, o := fixture(t) bad := Clone(o.Roster) b := bad.Boxes["box"] b.Role = "boss" bad.Boxes["box"] = b if _, err := d.Replace(ctx, sg, o, bad); !errors.Is(err, ErrRoster) { t.Fatalf("unknown box role written: %v", err) } moved := Clone(o.Roster) b = moved.Boxes["box"] b.Group = "routers" moved.Boxes["box"] = b if _, err := d.Replace(ctx, sg, o, moved); !errors.Is(err, ErrRoster) { t.Fatalf("group rewritten: %v", err) } noGroup := Clone(o.Roster) noGroup.Regions = map[string]Region{"ie": {Account: "123", AWSRegion: "eu-west-1", Role: RoleLegacy}} if _, err := d.Replace(ctx, sg, o, noGroup); !errors.Is(err, ErrRoster) { t.Fatalf("a role on a region with no group: %v", err) } platformGroup := Clone(o.Roster) platformGroup.Regions = map[string]Region{"ie": {Account: "123", AWSRegion: "eu-west-1", GroupName: "routers", Role: RolePlatform}} if _, err := d.Replace(ctx, sg, o, platformGroup); !errors.Is(err, ErrRoster) { t.Fatalf("a platform group: %v", err) } } // groupFixture is fixture plus a group in ie and two boxes it enrolled. func groupFixture(t *testing.T) (Directory, release.Signer, Observed) { t.Helper() ctx := context.Background() d, sg, o := fixture(t) next := Clone(o.Roster) next.Regions = map[string]Region{ "ie": {Account: "123", AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "group-arn", Namespaces: []string{"platform"}, Pools: []string{"default"}}, "uk": {Account: "123", AWSRegion: "eu-west-2"}, } for i, ref := range []string{"g1", "g2"} { next.Boxes[ref] = Box{Enrollment: strings.Repeat(string(rune('c'+i)), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default", Cloud: Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("b", 64), State: "active", LossIDs: []string{}, Group: "routers"} } o, err := d.Replace(ctx, sg, o, next) if err != nil { t.Fatal(err) } return d, sg, o } func TestGroupRoleSetsEveryMemberInOneWrite(t *testing.T) { ctx := context.Background() d, sg, o := groupFixture(t) next, set, err := d.SetGroupRole(ctx, sg, o, "ie", RoleLegacy) if err != nil { t.Fatal(err) } if !reflect.DeepEqual(set, []string{"g1", "g2"}) { t.Fatalf("set %v", set) } if next.Roster.Revision != o.Roster.Revision+1 { t.Fatalf("revision %d, want one write", next.Roster.Revision) } r := next.Roster if r.Regions["ie"].Role != RoleLegacy || r.Boxes["g1"].Role != RoleLegacy || r.Boxes["g2"].Role != RoleLegacy { t.Fatalf("roles: %+v %+v %+v", r.Regions["ie"], r.Boxes["g1"], r.Boxes["g2"]) } // The operator-enrolled box in the same region is not the group's. if r.Boxes["box"].Role != RoleDefault { t.Fatalf("operator box took the group's role: %+v", r.Boxes["box"]) } again, _, err := d.SetGroupRole(ctx, sg, next, "ie", RoleLegacy) if err != nil || again.ETag != next.ETag { t.Fatalf("idempotent group set wrote: %v", err) } back, _, err := d.SetGroupRole(ctx, sg, next, "ie", RoleDefault) if err != nil || back.Roster.Boxes["g1"].Role != RoleDefault || back.Roster.Regions["ie"].Role != RoleDefault { t.Fatalf("back to default: %v", err) } if _, _, err = d.SetGroupRole(ctx, sg, back, "uk", RoleLegacy); !errors.Is(err, ErrRoster) { t.Fatalf("a region with no group: %v", err) } if _, _, err = d.SetGroupRole(ctx, sg, back, "ie", RolePlatform); !errors.Is(err, ErrRole) { t.Fatalf("a platform group: %v", err) } } // Automatic enrollment gives the new box the group's role in the same // roster write (A11-1): a replacement for an excluded group is never // placeable, even for a moment. func TestAutomaticEnrollmentInheritsTheGroupRole(t *testing.T) { for _, role := range []string{RoleDefault, RoleLegacy} { t.Run(RoleName(role), func(t *testing.T) { ctx := context.Background() d, sg, o := fixture(t) key, err := rsa.GenerateKey(rand.Reader, 2048) if err != nil { t.Fatal(err) } doc := []byte(`{"accountId":"123","region":"eu-west-1","instanceId":"i-new"}`) sum := sha256.Sum256(doc) sig, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, sum[:]) if err != nil { t.Fatal(err) } raw, _ := json.Marshal(boxid.Record{V: 1, Box: "new", IdentityDocument: base64.StdEncoding.EncodeToString(doc), IdentitySignature: base64.StdEncoding.EncodeToString(sig)}) v := boxid.Verifier{Account: "123", Certs: map[string]*x509.Certificate{"eu-west-1": {PublicKey: &key.PublicKey, PublicKeyAlgorithm: x509.RSA}}} in := EnrollmentRequest{Reference: "new", Namespace: "platform", Label: "new", Region: "ie", Pool: "default", AWSRegion: "eu-west-1"} next := Clone(o.Roster) next.Regions = map[string]Region{"ie": {Account: "123", AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "group-arn", Namespaces: []string{"platform"}, Pools: []string{"default"}, Role: role}} if o, err = d.Replace(ctx, sg, o, next); err != nil { t.Fatal(err) } if _, err = d.Invite(ctx, sg, in.Reference, raw); err != nil { t.Fatal(err) } o, a, err := d.BeginEnrollmentAttempt(ctx, sg, o, in, raw, "i-new") if err != nil { t.Fatal(err) } now := time.Now().UTC() c := MembershipConfirmation{PassID: "pass-1", AttemptID: a.AttemptID, ConfirmationID: a.ConfirmationID, EvidenceSHA256: a.EvidenceSHA256, TargetPlacement: "ie", AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "group-arn", InstanceID: "i-new", Account: "123", AvailabilityZoneRegion: "eu-west-1", LifecycleState: "InService", InstanceState: "running", ObservedAt: now, RequestIDs: []string{"asg", "ec2"}} c.Confirmer.Box, c.Confirmer.Enrollment, c.Confirmer.Region = "box", o.Roster.Boxes["box"].Enrollment, "ie" wire, _ := json.Marshal(c) if err := release.CreateExact(ctx, d.Store, ConfirmationKey(in.Reference, a.AttemptID, a.ConfirmationID), wire); err != nil { t.Fatal(err) } in.AttemptID, in.ConfirmationID = a.AttemptID, a.ConfirmationID admitted, err := d.EnrollAutomatic(ctx, sg, o, v, in, raw, "i-new", now) if err != nil { t.Fatal(err) } b := admitted.Roster.Boxes["new"] if b.Group != "routers" || b.Role != role { t.Fatalf("admitted box group %q role %q, want routers %q", b.Group, b.Role, role) } if admitted.Roster.Revision != o.Roster.Revision+1 { t.Fatal("the role was not set in the enrolling write") } }) } } // Every other roster write keeps the role: a quarantine, for one, rewrites // the box's entry and must not reopen placement on it. func TestOtherRosterWritesKeepTheRole(t *testing.T) { ctx := context.Background() d, sg, o := fixture(t) o, err := d.SetBoxRole(ctx, sg, o, "box", RoleLegacy) if err != nil { t.Fatal(err) } q, err := d.Quarantine(ctx, sg, o, "box", o.Roster.Boxes["box"].Enrollment, "loss-1") if err != nil { t.Fatal(err) } if b := q.Roster.Boxes["box"]; b.Role != RoleLegacy || b.State != "quarantined" { t.Fatalf("after a quarantine: %+v", b) } read, err := d.Read(ctx) if err != nil || read.Roster.Boxes["box"].Role != RoleLegacy { t.Fatalf("read back: %+v %v", read.Roster.Boxes["box"], err) } } ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostactors/ -run 'Role|Roster|Group|Inherits' -count=1` Expected: FAIL to build (`undefined: ParseBoxRole`, `b.Role undefined`). - [ ] **Step 3: Write the roles** `internal/hostactors/roles.go`: ````go package hostactors import ( "context" "errors" "sort" "strings" "github.com/carlosframework/platform/internal/release" ) // Box and group roles (region HA spec 2 §7, A11-1). A box's role says // whether placement may choose it: the default role may be chosen, and // legacy and platform never are, for a create, a relocation target or a // move target. The box keeps the instances it has (H25). // // The region's group entry carries a role too. Every box automatic // enrollment admits from the group takes the group's role in the same // roster write, and changing the group's role changes the role of every // box enrolled from it, so a replacement launched after the group was // excluded is never placeable, even for a moment. // // The roster stores the default role as "" so a roster written before // roles existed reads as all-default. const ( RoleDefault = "" RoleLegacy = "legacy" RolePlatform = "platform" ) // ErrRole is a role name a box or group cannot take. var ErrRole = errors.New("hostactors: not a role this box or group can take") // ParseBoxRole reads an operator's box role: default, legacy or platform. func ParseBoxRole(s string) (string, error) { switch strings.TrimSpace(s) { case "default": return RoleDefault, nil case RoleLegacy: return RoleLegacy, nil case RolePlatform: return RolePlatform, nil } return "", ErrRole } // ParseGroupRole reads an operator's group role: default or legacy. A // group is never the platform box. func ParseGroupRole(s string) (string, error) { switch strings.TrimSpace(s) { case "default": return RoleDefault, nil case RoleLegacy: return RoleLegacy, nil } return "", ErrRole } // RoleName is the operator's word for a stored role. func RoleName(role string) string { if role == RoleDefault { return "default" } return role } func validBoxRole(role string) bool { return role == RoleDefault || role == RoleLegacy || role == RolePlatform } func validGroupRole(role string) bool { return role == RoleDefault || role == RoleLegacy } // SetBoxRole sets one box's role in a fresh roster write over o. The box // must be enrolled and not retired. Setting the role it already has // changes nothing and returns o. func (d Directory) SetBoxRole(ctx context.Context, sg release.Signer, o Observed, ref, role string) (Observed, error) { if !validBoxRole(role) { return Observed{}, ErrRole } b, ok := o.Roster.Boxes[ref] if !ok || b.State == "retired" { return Observed{}, ErrRoster } if b.Role == role { return o, nil } r := Clone(o.Roster) b = r.Boxes[ref] b.Role = role r.Boxes[ref] = b return d.Replace(ctx, sg, o, r) } // GroupMembers returns, sorted, the boxes enrolled from region's group // that are not retired: the boxes a group role change applies to. func GroupMembers(r Roster, region string) []string { g, ok := r.Regions[region] if !ok || g.GroupName == "" { return nil } var out []string for ref, b := range r.Boxes { if b.Region == region && b.Group == g.GroupName && b.State != "retired" { out = append(out, ref) } } sort.Strings(out) return out } // SetGroupRole sets region's group role and, in the same roster write, the // role of every box enrolled from that group (GroupMembers). The region // must have a group. It returns the roster and the boxes it set. func (d Directory) SetGroupRole(ctx context.Context, sg release.Signer, o Observed, region, role string) (Observed, []string, error) { if !validGroupRole(role) { return Observed{}, nil, ErrRole } g, ok := o.Roster.Regions[region] if !ok || g.GroupName == "" { return Observed{}, nil, ErrRoster } members := GroupMembers(o.Roster, region) changed := g.Role != role for _, ref := range members { changed = changed || o.Roster.Boxes[ref].Role != role } if !changed { return o, members, nil } r := Clone(o.Roster) g.Role = role r.Regions[region] = g for _, ref := range members { b := r.Boxes[ref] b.Role = role r.Boxes[ref] = b } next, err := d.Replace(ctx, sg, o, r) if err != nil { return Observed{}, nil, err } return next, members, nil } ```` ````diff --- a/internal/hostactors/roster.go +++ b/internal/hostactors/roster.go @@ -39,6 +39,14 @@ State string `json:"state"` QuarantineRevision uint64 `json:"quarantine_revision"` LossIDs []string `json:"loss_ids"` + // Group is the Auto Scaling group automatic enrollment admitted the box + // from (the region's GroupName then); "" for a box an operator + // enrolled. It never changes. + Group string `json:"group,omitempty"` + // Role is the box's placement role (roles.go): "" (default), legacy or + // platform. It is the one field besides the state fields a roster + // write may change on an enrolled box. + Role string `json:"role,omitempty"` } type Console struct { EnrolledAt time.Time `json:"enrolled_at,omitzero"` @@ -114,11 +122,19 @@ if b.State != "active" && b.State != "quarantined" && b.State != "retired" { return ErrRoster } + if !validBoxRole(b.Role) { + return ErrRoster + } if b.State == "quarantined" && (b.QuarantineRevision == 0 || len(b.LossIDs) == 0) { return ErrRoster } ids[b.Enrollment] = true clouds[b.Cloud] = true + } + for _, g := range r.Regions { + if !validGroupRole(g.Role) || (g.Role != RoleDefault && g.GroupName == "") { + return ErrRoster + } } for id, c := range r.Consoles { if !segment(id) || !token.MatchString(c.Enrollment) || ids[c.Enrollment] || c.InstanceID != id || c.DeploymentBucket != bucket || len(c.Namespaces) == 0 { @@ -246,6 +262,8 @@ z.QuarantineRevision = 0 a.LossIDs = nil z.LossIDs = nil + a.Role = "" + z.Role = "" if !reflect.DeepEqual(a, z) || b.QuarantineRevision < old.QuarantineRevision { return Observed{}, ErrRoster } ```` ````diff --- a/internal/hostactors/enrollment_confirmation.go +++ b/internal/hostactors/enrollment_confirmation.go @@ -18,6 +18,9 @@ AWSRegion string `json:"aws_region"` GroupName string `json:"group_name"` GroupARN string `json:"group_arn"` + // Role is the role every box automatic enrollment admits from the + // group takes (roles.go): "" (default) or legacy. + Role string `json:"role,omitempty"` } type TerminatedEnrollment struct { PassID string `json:"pass_id"` ```` ````diff --- a/internal/hostactors/enroll.go +++ b/internal/hostactors/enroll.go @@ -68,7 +68,15 @@ a.Consumed = true r.EnrollmentAttempts[in.Reference] = a } - r.Boxes[in.Reference] = Box{Enrollment: enrollment, Namespace: in.Namespace, BoxID: in.Reference, Label: in.Label, Region: in.Region, Pool: in.Pool, Cloud: cloud, EvidenceSHA256: evidence.RecordSHA256, State: "active", LossIDs: []string{}} + box := Box{Enrollment: enrollment, Namespace: in.Namespace, BoxID: in.Reference, Label: in.Label, Region: in.Region, Pool: in.Pool, Cloud: cloud, EvidenceSHA256: evidence.RecordSHA256, State: "active", LossIDs: []string{}} + if automatic { + // A box the group launched takes the group's role in this same + // write (region HA spec 2 §7, A11-1): a replacement for an + // excluded group is never placeable, even for a moment. + g := r.Regions[in.Region] + box.Group, box.Role = g.GroupName, g.Role + } + r.Boxes[in.Reference] = box return d.Replace(ctx, sg, o, r) } ```` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/hostactors/ -count=1` Expected: PASS (the package's whole suite, about 15 seconds). - [ ] **Step 5: Boundary list and census** ````diff --- a/region_ha_boundary_test.go +++ b/region_ha_boundary_test.go @@ -20,6 +20,7 @@ // an AWS package. var regionHACoreFiles = []string{ "internal/lbhealth/lbhealth.go", + "internal/hostactors/roles.go", } // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so ```` `Directory.Replace` is in the record-write census (it names `Put`). Its digest changes: ````diff --- a/internal/provision/plan9_b8_census_test.go +++ b/internal/provision/plan9_b8_census_test.go @@ -791,7 +791,7 @@ {"internal/host/snapshot_loop.go", "putReplacing", "nonrecord", "sel:Put", "11e8639effce142eab707b67cbdd425d0d7cea510ddf95d292bdba8f79c9b510"}, {"internal/hostactors/censuskey.go", "Directory.BindCensusKey", "nonrecord", "sel:Put", "7d9d673435e8c8a6aac3105e3a754bde2adcbc5d603945dd599cef703f334a5b"}, {"internal/hostactors/invitation.go", "Directory.Invite", "nonrecord", "sel:Put", "356618a75a23d0157d8e0402e20187a42956f67a4332d698bb22bf05133ac489"}, - {"internal/hostactors/roster.go", "Directory.Replace", "nonrecord", "sel:Put", "677bd94746cc5d3a5cd966d221f80b509ae096083a8ff4a0ee817d4c12c4a75f"}, + {"internal/hostactors/roster.go", "Directory.Replace", "nonrecord", "sel:Put", "e6e8ac0070650dfacc4a438b080cb69738530b3d9013de55c1bcf06431b0a847"}, {"internal/hostbox/regionalworker.go", "RegionalWorker.Census", "key-site", "lit:instances", "477bcb896b5ae8ec80af592d68eceb7992ad8a23cd6cff6043ad03b3ef2a8085"}, {"internal/hostbox/requeststore.go", "RequestStore.CopyVersion", "versioned-transport", "def:CopyVersion,sel:CopyVersion", "3a50049ce7ca7f2cbc770bef46218944cdc7000205a86c20e4932f0b510da28f"}, {"internal/hostbox/requeststore.go", "RequestStore.PutIfAbsent", "versioned-transport", "def:PutIfAbsent,sel:PutIfAbsent", "3666cbe5d2b91c1c652ffb4459c10a374fbd92e325e4e5c574fa3525d0ce2ffe"}, ```` Add this comment line after the overlay's last comment line: ````text # Task 3: box and group roles in the signed roster (internal/hostactors roles.go; Box.Role and Box.Group, Region.Role, Replace lets the role change, automatic enrollment inherits the group's role). Envelope-only: Directory.Replace, Roster.Validate and Directory.enroll are not root-read units, no root-read unit changed or was added, and the legacy seal stands. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/hostactors 6736698380a247764ada687f2476f89382fc32a1183a50c06afe75967d716eab ```` Run the census checks and `go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1`. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/hostactors region_ha_boundary_test.go internal/provision git commit -m "region HA 2a: box and group roles in the signed roster" ``` --- ### Task 4: The placement policy **Files:** - Create: `internal/boxchoice/policy.go`, `internal/boxchoice/policy_test.go` - Modify: `internal/boxchoice/boxchoice.go` (headroom totals and the generation carried through `Eligible`) - Modify: `internal/edgesfile/snapshot.go` (the reader's half: totals, generation, `BoxChoice`) - Create: `internal/edgesfile/snapshot_headroom_test.go` - Modify: `region_ha_boundary_test.go`, the overlay **Interfaces:** - Consumes: `boxchoice.Candidate`, `Load`, `Eligible`, `Choose` (its hash rule). - Produces: `Headroom.MemTotalMiB`, `Headroom.DiskTotalMiB` (json `mem_total_mib`, `disk_total_mib`); `Box.Generation string` (json `generation`); `Candidate.MemTotalMiB`, `Candidate.DiskTotalMiB`, `Candidate.Generation`; `MinMemPercent = 15`, `MinDiskPercent = 10`, `MaxEdgesAge = 15 * time.Minute`; `func ExcludeRoles(cands []Candidate, roles map[string]string) []Candidate`; `func Placeable(cands []Candidate, roles map[string]string, now time.Time) []Candidate`; `func ChooseRoomiest(host string, cands []Candidate, running map[string]Load, app string) (Candidate, bool)`; `edgesfile.PlacementView.Generation` and its headroom totals. `Eligible` only carries the new fields through; the new rules are `Placeable` and `ChooseRoomiest`, over candidates `Eligible` has already filtered for identity (§25). Any role other than the default excludes a box, so a role this build does not know fails closed. - [ ] **Step 1: Write the failing tests** `internal/boxchoice/policy_test.go`: ````go package boxchoice import ( "reflect" "testing" "time" ) var policyNow = time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) // roomy is a candidate every §7 rule passes at policyNow. func roomy(id string, memFree int64) Candidate { return Candidate{BoxID: id, Region: "eu-west-2", MemFreeMiB: memFree, MemTotalMiB: 8000, DiskFreeMiB: 50000, DiskTotalMiB: 100000, Generation: policyNow.Add(-5 * time.Minute).Format(time.RFC3339)} } func TestPlaceableAppliesEachRule(t *testing.T) { gen := func(d time.Duration) string { return policyNow.Add(d).Format(time.RFC3339) } cases := []struct { name string mut func(*Candidate) role string want bool }{ {"every rule holds", nil, "", true}, {"memory exactly 15%", func(c *Candidate) { c.MemFreeMiB = 1200 }, "", true}, {"memory under 15%", func(c *Candidate) { c.MemFreeMiB = 1199 }, "", false}, {"no memory total", func(c *Candidate) { c.MemTotalMiB = 0 }, "", false}, {"disk exactly 10%", func(c *Candidate) { c.DiskFreeMiB = 10000 }, "", true}, {"disk under 10%", func(c *Candidate) { c.DiskFreeMiB = 9999 }, "", false}, {"no disk total", func(c *Candidate) { c.DiskTotalMiB = 0 }, "", false}, {"edges file exactly 15 minutes old", func(c *Candidate) { c.Generation = gen(-15 * time.Minute) }, "", true}, {"edges file older than 15 minutes", func(c *Candidate) { c.Generation = gen(-15*time.Minute - time.Second) }, "", false}, {"edges file a minute in the future", func(c *Candidate) { c.Generation = gen(time.Minute) }, "", true}, {"edges file further in the future", func(c *Candidate) { c.Generation = gen(time.Minute + time.Second) }, "", false}, {"no generation", func(c *Candidate) { c.Generation = "" }, "", false}, {"unreadable generation", func(c *Candidate) { c.Generation = "yesterday" }, "", false}, {"legacy role", nil, "legacy", false}, {"platform role", nil, "platform", false}, {"a role this build does not know", nil, "draining", false}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { c := roomy("i-0aaa", 4000) if tc.mut != nil { tc.mut(&c) } got := Placeable([]Candidate{c}, map[string]string{"i-0aaa": tc.role}, policyNow) if (len(got) == 1) != tc.want { t.Fatalf("placeable = %v, want %v", ids(got), tc.want) } }) } } func TestExcludeRolesKeepsOrderAndDefaults(t *testing.T) { cands := []Candidate{roomy("c", 1), roomy("a", 1), roomy("b", 1)} got := ExcludeRoles(cands, map[string]string{"a": "legacy", "b": ""}) if !reflect.DeepEqual(ids(got), []string{"c", "b"}) { t.Fatalf("got %v", ids(got)) } if got := ExcludeRoles(cands, nil); len(got) != 3 { t.Fatalf("no roles excluded %v", ids(got)) } } func TestChooseRoomiestSpreadsThenMostMemoryThenHash(t *testing.T) { running := map[string]Load{"a": {Hosts: 1, Apps: map[string]int{"bab/jam": 1}}} // a has the most memory but already runs the app: spread wins. got, ok := ChooseRoomiest("h.example", []Candidate{roomy("a", 7000), roomy("b", 2000), roomy("c", 3000)}, running, "bab/jam") if !ok || got.BoxID != "c" { t.Fatalf("spread then memory: got %q", got.BoxID) } // Spread never refuses: every box runs the app, so memory decides. all := map[string]Load{"a": running["a"], "b": running["a"]} got, _ = ChooseRoomiest("h.example", []Candidate{roomy("a", 1000), roomy("b", 2000)}, all, "bab/jam") if got.BoxID != "b" { t.Fatalf("all run the app: got %q", got.BoxID) } // Fewer hosts is no longer a rule: memory beats it. busy := map[string]Load{"a": {Hosts: 50}, "b": {Hosts: 1}} got, _ = ChooseRoomiest("h.example", []Candidate{roomy("a", 5000), roomy("b", 4000)}, busy, "bab/jam") if got.BoxID != "a" { t.Fatalf("memory over host count: got %q", got.BoxID) } // Equal memory: the P5-D2 hash, whatever the order. one, _ := ChooseRoomiest("h.example", []Candidate{roomy("a", 1000), roomy("b", 1000)}, nil, "bab/jam") two, _ := ChooseRoomiest("h.example", []Candidate{roomy("b", 1000), roomy("a", 1000)}, nil, "bab/jam") if one.BoxID != two.BoxID { t.Fatalf("order changed the answer: %q %q", one.BoxID, two.BoxID) } want, _ := Choose("h.example", []Candidate{roomy("a", 1000), roomy("b", 1000)}, nil, "bab/jam") if one.BoxID != want.BoxID { t.Fatalf("hash tie-break differs from spec 1's: %q, want %q", one.BoxID, want.BoxID) } if _, ok := ChooseRoomiest("h.example", nil, nil, "bab/jam"); ok { t.Fatal("chose from nothing") } } func TestEligibleCarriesTotalsAndGeneration(t *testing.T) { body := haBox("i-0aaa", func(m map[string]any) { m["headroom"] = map[string]any{"mem_free_mib": 1000, "disk_free_mib": 2000, "mem_total_mib": 8000, "disk_total_mib": 40000, "awake": 1} m["generation"] = "2026-10-16T11:55:00Z" }) got := Eligible(views(t, body), rosterFor("i-0aaa"), "uk", "eu-west-2", "ie", testKeyFP, nil) want := Candidate{BoxID: "i-0aaa", Region: "eu-west-2", MemFreeMiB: 1000, DiskFreeMiB: 2000, MemTotalMiB: 8000, DiskTotalMiB: 40000, Awake: 1, Generation: "2026-10-16T11:55:00Z"} if len(got) != 1 || got[0] != want { t.Fatalf("got %+v, want %+v", got, want) } } ```` `internal/edgesfile/snapshot_headroom_test.go`: ````go package edgesfile import ( "encoding/json" "testing" ) // Region HA spec 2 §7: the placement policy reads the headroom totals and // the file's generation, so BoxChoice carries both. func TestBoxChoiceCarriesHeadroomTotalsAndGeneration(t *testing.T) { var f PlacementView body := `{"v":1,"box":"box-a","box_id":"i-a","generation":"2026-10-16T11:55:00Z","headroom":{"mem_free_mib":512,"disk_free_mib":2048,"mem_total_mib":4096,"disk_total_mib":20480,"awake":3},"hosts":{}}` if err := json.Unmarshal([]byte(body), &f); err != nil { t.Fatal(err) } f.Fresh = true got := Snapshot{Files: []PlacementView{f}, Complete: true, BoxesComplete: true}.BoxChoice() if len(got.Boxes) != 1 { t.Fatalf("boxes %+v", got.Boxes) } b := got.Boxes[0] if b.Generation != "2026-10-16T11:55:00Z" || b.Headroom == nil || b.Headroom.MemTotalMiB != 4096 || b.Headroom.DiskTotalMiB != 20480 || b.Headroom.MemFreeMiB != 512 || b.Headroom.DiskFreeMiB != 2048 || b.Headroom.Awake != 3 { t.Fatalf("box lost a field: %+v %+v", b, b.Headroom) } } ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/boxchoice/ ./internal/edgesfile/ -count=1` Expected: FAIL to build (`undefined: Placeable`, `unknown field MemTotalMiB`). - [ ] **Step 3: Write the policy** `internal/boxchoice/policy.go`: ````go package boxchoice import ( "crypto/sha256" "encoding/hex" "sort" "time" ) // Region HA spec 2's placement policy (§7). Its rules run over candidates // Eligible has already filtered for identity, placement and configuration // (§25): a provider-neutral box identity (#299) changes only Eligible, // never these. // The thresholds of §7. const ( // MinMemPercent: a box is placeable only while its free memory is at // least this share of its total, the memory alarm's own line. MinMemPercent = 15 // MinDiskPercent: and its data volume's free space at least this // share of the volume. MinDiskPercent = 10 // MaxEdgesAge: and its edges file, which carries the headroom, is no // older than this. Files republish every 5 minutes plus up to a // minute of jitter, so a healthy box always has one within two // intervals and a margin (A1-8). MaxEdgesAge = 15 * time.Minute // maxEdgesSkew is how far in the future a file's generation may be // before it is not believed: box clocks are synchronised, not equal. maxEdgesSkew = time.Minute ) // ExcludeRoles drops every candidate whose roster role is not the default // one (roles is box id -> role, "" for the default). Legacy and platform // boxes, and any role this build does not know, take no new instances. // The order of cands is kept. func ExcludeRoles(cands []Candidate, roles map[string]string) []Candidate { var out []Candidate for _, c := range cands { if roles[c.BoxID] == "" { out = append(out, c) } } return out } // headroomFresh reports whether a candidate's edges file is recent enough // at now for its headroom to be trusted. func headroomFresh(generation string, now time.Time) bool { at, err := time.Parse(time.RFC3339, generation) if err != nil { return false } age := now.Sub(at) return age <= MaxEdgesAge && age >= -maxEdgesSkew } // Placeable is §7's eligibility on top of Eligible's: the default role, // free memory at least MinMemPercent of the total, free disk at least // MinDiskPercent of the volume, and an edges file no older than // MaxEdgesAge at now. A box that publishes no totals is not placeable: its // headroom cannot be judged. The order of cands is kept. func Placeable(cands []Candidate, roles map[string]string, now time.Time) []Candidate { var out []Candidate for _, c := range ExcludeRoles(cands, roles) { switch { case c.MemTotalMiB <= 0 || c.MemFreeMiB*100 < c.MemTotalMiB*MinMemPercent: case c.DiskTotalMiB <= 0 || c.DiskFreeMiB*100 < c.DiskTotalMiB*MinDiskPercent: case !headroomFresh(c.Generation, now): default: out = append(out, c) } } return out } // ChooseRoomiest is §7's choice among placeable candidates: prefer a box // running no instance of app (soft spread: it never refuses), then the // most free memory, then the greatest hexadecimal SHA-256 of host + "\x00" // + box id (claim P5-D2), then the smallest box id. The answer does not // depend on the order of cands. The bool is false only when cands is // empty. func ChooseRoomiest(host string, cands []Candidate, running map[string]Load, app string) (Candidate, bool) { if len(cands) == 0 { return Candidate{}, false } tie := func(ref string) string { sum := sha256.Sum256([]byte(host + "\x00" + ref)) return hex.EncodeToString(sum[:]) } sorted := append([]Candidate(nil), cands...) sort.SliceStable(sorted, func(i, j int) bool { a, b := sorted[i], sorted[j] aRuns, bRuns := running[a.BoxID].Apps[app] > 0, running[b.BoxID].Apps[app] > 0 switch { case aRuns != bRuns: return !aRuns case a.MemFreeMiB != b.MemFreeMiB: return a.MemFreeMiB > b.MemFreeMiB case tie(a.BoxID) != tie(b.BoxID): return tie(a.BoxID) > tie(b.BoxID) } return a.BoxID < b.BoxID }) return sorted[0], true } ```` ````diff --- a/internal/boxchoice/boxchoice.go +++ b/internal/boxchoice/boxchoice.go @@ -21,11 +21,15 @@ // region HA executor. const Capability = "region-ha-1" -// Headroom is what a box reports it has free. +// Headroom is what a box reports it has free, and (region HA spec 2 §7) +// the totals the free figures are fractions of. A total of 0 is a writer +// that predates it, or a reading that failed. type Headroom struct { - MemFreeMiB int64 `json:"mem_free_mib"` - DiskFreeMiB int64 `json:"disk_free_mib"` - Awake int `json:"awake"` + MemFreeMiB int64 `json:"mem_free_mib"` + DiskFreeMiB int64 `json:"disk_free_mib"` + MemTotalMiB int64 `json:"mem_total_mib"` + DiskTotalMiB int64 `json:"disk_total_mib"` + Awake int `json:"awake"` } // Box is one box's edges file (control/edges/.json, written by @@ -48,6 +52,9 @@ Capabilities []string `json:"capabilities"` Draining bool `json:"draining"` Headroom *Headroom `json:"headroom"` + // Generation is when the box last wrote the file (RFC 3339). Spec 2's + // policy trusts headroom only from a recent file (Placeable). + Generation string `json:"generation"` // Fresh is the caller's verdict that the box's own status snapshot is // still fresh. An edges file outlives its box, so the file alone is // not liveness. @@ -76,10 +83,13 @@ // Candidate is one box that may be named as a host's executor (region HA // spec 1 §2, "eligible box"). type Candidate struct { - BoxID string - Region string - MemFreeMiB, DiskFreeMiB int64 - Awake int + BoxID string + Region string + MemFreeMiB, DiskFreeMiB int64 + MemTotalMiB, DiskTotalMiB int64 + Awake int + // Generation is the box's edges file's (Box.Generation). + Generation string } // Load is what claims already place on a box. @@ -132,9 +142,10 @@ case !slices.Contains(f.Capabilities, Capability): case f.Draining, lost[f.BoxID]: default: - c := Candidate{BoxID: f.BoxID, Region: rb.Cloud.Region} + c := Candidate{BoxID: f.BoxID, Region: rb.Cloud.Region, Generation: f.Generation} if f.Headroom != nil { c.MemFreeMiB, c.DiskFreeMiB, c.Awake = f.Headroom.MemFreeMiB, f.Headroom.DiskFreeMiB, f.Headroom.Awake + c.MemTotalMiB, c.DiskTotalMiB = f.Headroom.MemTotalMiB, f.Headroom.DiskTotalMiB } out = append(out, c) } ```` ````diff --- a/internal/edgesfile/snapshot.go +++ b/internal/edgesfile/snapshot.go @@ -58,11 +58,18 @@ Capabilities []string `json:"capabilities"` Draining bool `json:"draining"` Headroom *struct { - MemFreeMiB int64 `json:"mem_free_mib"` - DiskFreeMiB int64 `json:"disk_free_mib"` - Awake int `json:"awake"` + MemFreeMiB int64 `json:"mem_free_mib"` + DiskFreeMiB int64 `json:"disk_free_mib"` + MemTotalMiB int64 `json:"mem_total_mib"` + DiskTotalMiB int64 `json:"disk_total_mib"` + Awake int `json:"awake"` } `json:"headroom"` Hosts map[string]json.RawMessage `json:"hosts"` + + // Generation is when the box last wrote the file (RFC 3339), which + // region HA spec 2's placement policy reads as headroom's age. + Generation string `json:"generation"` + // The moves protocol (moves spec §11), read only by the Phase 1 move // door's floor check (selectMoveBoxes); Task 13 adds the writer's side. // MovesProtocol is the protocol the box speaks (absent: 0). @@ -246,10 +253,11 @@ Box: f.Box, BoxID: f.BoxID, EIP: f.EIP, Region: f.Region, Placement: f.Placement, HibernationConfigured: f.HibernationConfigured, ServesSharedPool: f.ServesSharedPool, AcceptsInbound: f.AcceptsInbound, SecretsKeyFP: f.SecretsKeyFP, Capabilities: f.Capabilities, - Draining: f.Draining, Fresh: f.Fresh, + Draining: f.Draining, Generation: f.Generation, Fresh: f.Fresh, } if f.Headroom != nil { - b.Headroom = &boxchoice.Headroom{MemFreeMiB: f.Headroom.MemFreeMiB, DiskFreeMiB: f.Headroom.DiskFreeMiB, Awake: f.Headroom.Awake} + b.Headroom = &boxchoice.Headroom{MemFreeMiB: f.Headroom.MemFreeMiB, DiskFreeMiB: f.Headroom.DiskFreeMiB, + MemTotalMiB: f.Headroom.MemTotalMiB, DiskTotalMiB: f.Headroom.DiskTotalMiB, Awake: f.Headroom.Awake} } out.Boxes = append(out.Boxes, b) } ```` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/boxchoice/ ./internal/edgesfile/ -count=1 && nice -n 19 go test ./internal/console/ -run 'Box|Eligib|Executor|RegionCand' -count=1` Expected: PASS. - [ ] **Step 5: Boundary list and census** ````diff --- a/region_ha_boundary_test.go +++ b/region_ha_boundary_test.go @@ -21,6 +21,7 @@ var regionHACoreFiles = []string{ "internal/lbhealth/lbhealth.go", "internal/hostactors/roles.go", + "internal/boxchoice/policy.go", } // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so ```` Add this comment line after the overlay's last comment line: ````text # Task 4: the placement policy (internal/boxchoice policy.go; headroom totals and the edges file's generation carried through Eligible and edgesfile.Snapshot.BoxChoice). Envelope-only: no root-read unit changed or was added, and the legacy seal stands. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/boxchoice 1bfd561bcb48b51e60e63f99e086fd44d3b1be2160e28b3ed0253a14bbb9ff66 @coverage internal/edgesfile da42c799d62e065817b5cce4f9a3ead1308dbd379615a85a7b14533c16d7e804 ```` Run the census checks and `go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1`. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/boxchoice internal/edgesfile region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv git commit -m "region HA 2a: the placement policy over the eligible set" ``` --- ### Task 5: Boxes publish their total memory and data volume size **Files:** - Modify: `internal/host/headroom.go` (`headroomFor` fills the totals; `capacityMiB`) - Modify: `internal/host/edgesfile.go` (`edgeHeadroom`'s two fields) - Modify: `internal/host/headroom_test.go` - Modify: the overlay **Interfaces:** - Consumes: `readHeadroom`, `readMemBytes`, `diskBytes` (unchanged). - Produces: `edgeHeadroom.MemTotalMiB`, `edgeHeadroom.DiskTotalMiB` (json `mem_total_mib`, `disk_total_mib`, omitempty), the fields Task 4's reader takes; `func capacityMiB(mem func() (total, avail *int64), disk func() (total, avail uint64, err error)) (memMiB, diskMiB int64)`. Headroom is published only with the region HA capability (`headroomFor` returns nil without it), so with region HA off the edges file is byte-identical to main's. `readHeadroom` and `readMemBytes` are accepted-legacy units and stay as they are: `headroomFor` calls the existing readers. - [ ] **Step 1: Write the failing tests** ````diff --- a/internal/host/headroom_test.go +++ b/internal/host/headroom_test.go @@ -1,6 +1,13 @@ package host -import "testing" +import ( + "context" + "encoding/json" + "errors" + "testing" + + "github.com/carlosframework/platform/internal/release" +) func TestParseMemAvailable(t *testing.T) { const meminfo = "MemTotal: 16000000 kB\nMemFree: 500000 kB\nMemAvailable: 2263040 kB\n" @@ -24,3 +31,47 @@ t.Fatal("flag on: headroom is nil") } } + +// Region HA spec 2 §7: the totals ride beside the free figures, and a +// failed reading is 0, never a guess. +func TestCapacityMiBReadsTotalsAndZeroesFailures(t *testing.T) { + gib := int64(8 << 30) + mem := func() (*int64, *int64) { return &gib, nil } + disk := func() (uint64, uint64, error) { return 100 << 30, 1 << 30, nil } + if m, d := capacityMiB(mem, disk); m != 8192 || d != 102400 { + t.Fatalf("capacity = %d, %d; want 8192, 102400", m, d) + } + noMem := func() (*int64, *int64) { return nil, nil } + noDisk := func() (uint64, uint64, error) { return 0, 0, errors.New("statfs") } + if m, d := capacityMiB(noMem, noDisk); m != 0 || d != 0 { + t.Fatalf("failed readings = %d, %d; want 0, 0", m, d) + } +} + +func TestHeadroomCarriesTheDataVolumeSize(t *testing.T) { + h := headroomFor([]string{capRegionHA1}, t.TempDir(), edgesTestRegistry(t)) + if h == nil || h.DiskTotalMiB <= 0 { + t.Fatalf("headroom %+v: want the data volume's size", h) + } +} + +func TestEdgesFileWireCarriesHeadroomTotals(t *testing.T) { + s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t) + var last []byte + ident := edgesFile{Box: "annie", BoxID: "i-0abc", Capabilities: []string{"region-ha-1"}, + Headroom: &edgeHeadroom{MemFreeMiB: 2210, DiskFreeMiB: 31200, Awake: 14, MemTotalMiB: 7900, DiskTotalMiB: 61000}} + if err := edgesFilePass(context.Background(), s, reg, ident, "1.2.3.4", &last); err != nil { + t.Fatal(err) + } + body, _, err := s.Get(context.Background(), "control/edges/annie.json") + if err != nil { + t.Fatal(err) + } + var raw map[string]json.RawMessage + if err := json.Unmarshal(body, &raw); err != nil { + t.Fatal(err) + } + if want := `{"mem_free_mib":2210,"disk_free_mib":31200,"awake":14,"mem_total_mib":7900,"disk_total_mib":61000}`; string(raw["headroom"]) != want { + t.Fatalf("headroom = %s, want %s", raw["headroom"], want) + } +} ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/host/ -run 'Headroom|CapacityMiB' -count=1` Expected: FAIL to build (`undefined: capacityMiB`, `unknown field MemTotalMiB`). - [ ] **Step 3: Write the totals** ````diff --- a/internal/host/headroom.go +++ b/internal/host/headroom.go @@ -14,7 +14,23 @@ if len(caps) == 0 { return nil } - return readHeadroom(dataDir, awakeCount(reg)) + h := readHeadroom(dataDir, awakeCount(reg)) + h.MemTotalMiB, h.DiskTotalMiB = capacityMiB(readMemBytes, func() (uint64, uint64, error) { return diskBytes(dataDir) }) + return h +} + +// capacityMiB is the box's total memory and its data volume's size, in +// MiB: the totals region HA spec 2's placement policy judges the free +// figures against (§7). A reading that fails is 0, which the policy reads +// as a box it cannot judge, and so does not place on. +func capacityMiB(mem func() (total, avail *int64), disk func() (total, avail uint64, err error)) (memMiB, diskMiB int64) { + if total, _ := mem(); total != nil && *total > 0 { + memMiB = *total / (1 << 20) + } + if total, _, err := disk(); err == nil { + diskMiB = int64(total / (1 << 20)) + } + return memMiB, diskMiB } // parseMemAvailableMiB returns the MemAvailable line in MiB, or 0. ```` ````diff --- a/internal/host/edgesfile.go +++ b/internal/host/edgesfile.go @@ -151,6 +151,11 @@ MemFreeMiB int64 `json:"mem_free_mib"` DiskFreeMiB int64 `json:"disk_free_mib"` Awake int `json:"awake"` + // MemTotalMiB and DiskTotalMiB are what the free figures are shares + // of (region HA spec 2 §7). Additive and omitted when 0, a reading + // that failed: a reader takes absent and 0 alike, as unknown. + MemTotalMiB int64 `json:"mem_total_mib,omitempty"` + DiskTotalMiB int64 `json:"disk_total_mib,omitempty"` } const capRegionHA1 = "region-ha-1" ```` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/host/ -run 'Edges|Headroom|CapacityMiB|ParseMemAvailable' -count=1` Expected: PASS, including main's `TestEdgesFileRawWireBodyCarriesBoxEvidence` (the totals are omitted when zero). - [ ] **Step 5: Census** Add this comment line after the overlay's last comment line: ````text # Task 5: the edges file's headroom carries the box's total memory and data volume size (headroomFor and capacityMiB, internal/host/headroom.go; edgeHeadroom's two additive fields). headroomFor calls the existing readMemBytes and diskBytes and adds no read of its own. Envelope-only: readHeadroom and readMemBytes are unchanged, no root-read unit changed or was added, and the legacy seal stands. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/host c1cbcb478b2d64b3114a3c33be7d847cf8b266694f42f0fab4e9b28a51bc7f9c ```` Run the census checks. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/host/headroom.go internal/host/headroom_test.go internal/host/edgesfile.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv git commit -m "region HA 2a: edges files carry total memory and disk" ``` --- ### Task 6: Creates and relocation targets choose by the policy **Files:** - Modify: `internal/hostops/executor.go` (`Choose`, `Permits`, `PlaceableBoxes`, `Capacity`, `placeable`) - Modify: `internal/hostops/executor_test.go`, `internal/hostrelocate/chooser_test.go` (fixtures publish roomy, fresh headroom) - Modify: the overlay **Interfaces:** - Consumes: `boxchoice.Placeable`, `boxchoice.ChooseRoomiest` (Task 4); `hostactors.Box.Role` (Task 3). - Produces: `func (x ExecutorChooser) PlaceableBoxes(ctx context.Context, placement string) ([]string, error)` (Tasks 11 and 12 read it) and `func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error)` (Task 13 publishes it, and extends it to answer the roster's AWS region); `Choose` and `Permits` now apply the roles and thresholds. `ExecutorChooser` exists only with `CARLOS_REGION_HA` on (the console binds it only then, and the box only in its region HA relocation runtime), so the policy is dormant with it off. `ExecutorChooser.eligible` is an accepted-legacy unit (merge ruling 2) and does not change: the policy runs over its result. Relocation's `RegionChooser` uses `Choose` and `Permits`, so relocation targets follow the same policy. - [ ] **Step 1: Write the failing tests** ````diff --- a/internal/hostops/executor_test.go +++ b/internal/hostops/executor_test.go @@ -43,7 +43,12 @@ roster.Boxes[ref] = hostactors.Box{Enrollment: strings.Repeat(string("abcdef"[i]), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default", Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"} snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true, - HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}}) + HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}, + // Equal, roomy headroom from a file a minute old: spec 2's + // policy (§7) admits every box, and the choice falls through + // to the hash unless a test changes it. + Headroom: &boxchoice.Headroom{MemFreeMiB: 4000, MemTotalMiB: 8000, DiskFreeMiB: 50000, DiskTotalMiB: 100000}, + Generation: "2026-10-07T08:59:00Z"}) } if edit != nil { edit(&roster) @@ -68,9 +73,9 @@ Body: []byte(`{"host":"shop.bab.oncarlos.com","account":"bab","app":"jam","region":"ie"}`)} } -func TestExecutorChooserSpreadsThenBalancesThenHashes(t *testing.T) { - ctx := context.Background() - x, _, _, loads := chooserFixture(t, "box-a", "box-b") +func TestExecutorChooserSpreadsThenPrefersMemoryThenHashes(t *testing.T) { + ctx := context.Background() + x, _, snap, loads := chooserFixture(t, "box-a", "box-b") absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} first, err := x.Choose(ctx, absent, chooserRecord()) if err != nil { @@ -85,10 +90,80 @@ if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != other { t.Fatalf("soft spread: got %s, want %s", got.Box, other) } - // Both run the app: the fewest hosts wins. + // Both run the app: the most free memory wins (spec 2 §7), whatever + // the host counts say. loads[other] = boxchoice.Load{Hosts: 5, Apps: map[string]int{"bab/jam": 1}} - if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != first.Box { - t.Fatalf("headroom: got %s, want %s", got.Box, first.Box) + for i := range snap.Boxes { + if snap.Boxes[i].BoxID == other { + snap.Boxes[i].Headroom.MemFreeMiB = 6000 + } + } + if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != other { + t.Fatalf("headroom: got %s, want %s", got.Box, other) + } +} + +// withRole sets one box's roster role before the roster is signed. +func withRole(ref, role string) func(*hostactors.Roster) { + return func(r *hostactors.Roster) { + b := r.Boxes[ref] + b.Role = role + r.Boxes[ref] = b + } +} + +// Region HA spec 2 §7: the chooser, and so creates and relocation +// targets, applies the roster's roles and the policy's thresholds after +// Eligible. Permits answers the same set, and Capacity counts it. +func TestExecutorChooserAppliesRolesAndHeadroom(t *testing.T) { + ctx := context.Background() + absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} + cases := []struct { + name string + roster func(*hostactors.Roster) + edges func(*boxchoice.Box) + }{ + {"legacy role", withRole("box-a", hostactors.RoleLegacy), nil}, + {"platform role", withRole("box-a", hostactors.RolePlatform), nil}, + {"memory under 15%", nil, func(b *boxchoice.Box) { b.Headroom.MemFreeMiB = 1199 }}, + {"disk under 10%", nil, func(b *boxchoice.Box) { b.Headroom.DiskFreeMiB = 9999 }}, + {"edges file 16 minutes old", nil, func(b *boxchoice.Box) { b.Generation = "2026-10-07T08:44:00Z" }}, + {"no headroom", nil, func(b *boxchoice.Box) { b.Headroom = nil }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + x, _, snap, _ := chooserFixtureWith(t, tc.roster, "box-a", "box-b") + for i := range snap.Boxes { + if snap.Boxes[i].BoxID == "box-a" && tc.edges != nil { + tc.edges(&snap.Boxes[i]) + } + } + for range 4 { + got, err := x.Choose(ctx, absent, chooserRecord()) + if err != nil || got.Box != "box-b" { + t.Fatalf("chose %+v %v, want box-b", got, err) + } + } + if err := x.Permits(ctx, absent, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) { + t.Fatalf("Permits(box-a) = %v, want ErrNoEligibleBox", err) + } + if err := x.Permits(ctx, absent, chooserRecord(), "box-b"); err != nil { + t.Fatalf("Permits(box-b) = %v", err) + } + if n, err := x.Capacity(ctx, "ie"); err != nil || n != 1 { + t.Fatalf("Capacity = %d, %v; want 1", n, err) + } + if boxes, err := x.PlaceableBoxes(ctx, "ie"); err != nil || len(boxes) != 1 || boxes[0] != "box-b" { + t.Fatalf("PlaceableBoxes = %v, %v; want [box-b]", boxes, err) + } + }) + } + x, _, _, _ := chooserFixture(t, "box-a", "box-b") + if n, err := x.Capacity(ctx, "ie"); err != nil || n != 2 { + t.Fatalf("Capacity with both placeable = %d, %v; want 2", n, err) + } + if _, err := x.Capacity(ctx, "us"); !errors.Is(err, ErrContract) { + t.Fatalf("Capacity of a region the roster lacks = %v, want ErrContract", err) } } ```` ````diff --- a/internal/hostrelocate/chooser_test.go +++ b/internal/hostrelocate/chooser_test.go @@ -5,6 +5,7 @@ "errors" "slices" "testing" + "time" "github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostclaim" @@ -13,13 +14,16 @@ // scopedChooser plugs create's chooser into the fixture's worker, scoped // to namespace, with every fixture box fresh and eligible on its edges -// file and spa the most loaded. +// file, equally roomy under spec 2's policy (§7), and spa the most loaded. func scopedChooser(f *fixture, namespace string) RegionChooser { yes := true snap := boxchoice.Snapshot{Complete: true} + generation := f.w.Now().Add(-time.Minute).UTC().Format(time.RFC3339) for _, ref := range []string{"src", "dst", "spa", "sp2"} { snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true, - HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}}) + HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}, + Headroom: &boxchoice.Headroom{MemFreeMiB: 4000, MemTotalMiB: 8000, DiskFreeMiB: 50000, DiskTotalMiB: 100000}, + Generation: generation}) } loads := map[string]boxchoice.Load{"spa": {Hosts: 9, Apps: map[string]int{"bab/jam": 3}}} return RegionChooser{Records: f.s, Namespace: namespace, Region: "ie", ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser' -count=1` Expected: FAIL to build (`x.Capacity undefined`, `x.PlaceableBoxes undefined`). - [ ] **Step 3: Apply the policy** ````diff --- a/internal/hostops/executor.go +++ b/internal/hostops/executor.go @@ -48,17 +48,19 @@ // r's placement is created on. Identity, placement and AWS region come from // the roster; a box with a valid termination fence, or not active, is never // chosen; a box on the claim's lost list, or owing a park-local chore on -// it, is never chosen. +// it, is never chosen. Among the rest, region HA spec 2's policy (§7) +// decides: placeable boxes only (placeable), then boxchoice.ChooseRoomiest. func (x ExecutorChooser) Choose(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, error) { o, cands, err := x.eligible(ctx, obs, r) if err != nil { return hostclaim.ExecutorSelection{}, err } + cands = x.placeable(o, cands) loads, err := x.Loads(ctx) if err != nil { return hostclaim.ExecutorSelection{}, err } - best, ok := boxchoice.Choose(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App) + best, ok := boxchoice.ChooseRoomiest(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App) if !ok { return hostclaim.ExecutorSelection{}, ErrNoEligibleBox } @@ -69,18 +71,56 @@ // in r's placement now (spec 1 §2, §5.8 "The start CAS ... refused ... // when `to` is not eligible for the host"): the same read Choose makes, // without choosing. A stale edges file, a missing capability or secrets -// key, a draining box, a fence, or an incomplete snapshot refuses. +// key, a draining box, a fence, or an incomplete snapshot refuses, and so +// does a box spec 2's policy does not place on (placeable): a legacy or +// platform role, too little memory or disk, or stale headroom. func (x ExecutorChooser) Permits(ctx context.Context, obs hostclaim.Observed, r Record, box string) error { - _, cands, err := x.eligible(ctx, obs, r) + o, cands, err := x.eligible(ctx, obs, r) if err != nil { return err } - for _, c := range cands { + for _, c := range x.placeable(o, cands) { if c.BoxID == box { return nil } } return fmt.Errorf("%w: %s", ErrNoEligibleBox, box) +} + +// PlaceableBoxes is every box in placement that could take a new +// instance in the chooser's namespace and pool now, sorted: Choose's set, +// before the choice. The console's move door takes a move's destination +// from it (region HA spec 2 §7, H27). +func (x ExecutorChooser) PlaceableBoxes(ctx context.Context, placement string) ([]string, error) { + r := Record{Holder: hostclaim.NewHolder{Region: placement}} + o, cands, err := x.eligible(ctx, hostclaim.Observed{Absent: true}, r) + if err != nil { + return nil, err + } + boxes := []string{} + for _, c := range x.placeable(o, cands) { + boxes = append(boxes, c.BoxID) + } + return boxes, nil +} + +// Capacity is how many boxes PlaceableBoxes names. The console publishes +// it per region as the eligible-capacity metric (region HA spec 2 §4, +// A1-11). +func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error) { + boxes, err := x.PlaceableBoxes(ctx, placement) + return len(boxes), err +} + +// placeable is region HA spec 2's policy over the eligible set (§7): +// each box's roster role, then boxchoice.Placeable's thresholds at +// x.Now(). Identity was Eligible's; nothing here reads a provider. +func (x ExecutorChooser) placeable(o hostactors.Observed, cands []boxchoice.Candidate) []boxchoice.Candidate { + roles := make(map[string]string, len(cands)) + for _, c := range cands { + roles[c.BoxID] = o.Roster.Boxes[c.BoxID].Role + } + return boxchoice.Placeable(cands, roles, x.Now()) } // eligible is one read of the roster, the fences and the edges files, ```` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser|SelectExecutor|ProductionCreate' -count=1 && nice -n 19 go test ./internal/hostrelocate/ -count=1` Expected: PASS. - [ ] **Step 5: Census** Add this comment line after the overlay's last comment line: ````text # Task 6: hostops.ExecutorChooser applies spec 2's policy in Choose and Permits (creates and relocation targets) and names the placeable set in PlaceableBoxes and Capacity. ExecutorChooser.eligible, the accepted-legacy unit (merge ruling 2), is unchanged: the policy runs over its result. Envelope-only, and the legacy seal stands. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/hostops 3dc23aaff7bdae0341a8c9a6fbac076a2f57ca1e4676d83543433ff9ad7d1c79 ```` Run the census checks. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/hostops/executor.go internal/hostops/executor_test.go internal/hostrelocate/chooser_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv git commit -m "region HA 2a: creates and relocations choose by the placement policy" ``` --- ### Task 7: The box's own addresses, validated, and `private_ip` in the edges file **Files:** - Create: `internal/edgesfile/private.go`, `internal/edgesfile/private_test.go` (`PrivateAddress`, the one validator the writer and the reader share) - Create: `internal/host/boxaddress.go` (core: `BoxAddresses`, `AddressSource`, the `host.env` source) - Create: `internal/host/awsaddr.go` (the AWS adapter: IMDSv2 `local-ipv4`, `public-ipv4`, `mac` and the VPC's `vpc-ipv4-cidr-blocks`) - Create: `internal/host/boxaddress_test.go` - Modify: `internal/host/edgesclaims.go` (`edgesClaims.addrs`) - Modify: `internal/host/edgesfile.go` (`edgesFile.PrivateIP`, `PrivateCIDR`; `edgesFileWriteClaims` reads the source with region HA on) - Modify: `internal/host/edgesclaims_test.go`, `region_ha_boundary_test.go`, the overlay **Interfaces:** - Consumes: `imdsToken`, `imdsLeaf` (agent.go, unchanged). - Produces: `func edgesfile.PrivateAddress(addr string, cidrs []string) (string, string)`; `type BoxAddresses struct{ Private, PrivateCIDR, Public string }`; `type AddressSource interface{ BoxAddresses() BoxAddresses }`; `envAddresses{getenv, provider}`; `imdsAddresses`; `func boxAddressSource(getenv func(string) string) AddressSource`; `edgesFile.PrivateIP`, `edgesFile.PrivateCIDR` (json `private_ip`, `private_cidr`, omitempty), which Task 8 reads. Census ruling region-ha-2a's first condition, in full: - **Source.** The address comes from the same root-configured source as the rest of the box's entry: IMDS on AWS, or root-owned `host.env` otherwise (`CARLOS_PRIVATE_IP` with `CARLOS_PRIVATE_CIDR`, which is required beside it; `CARLOS_PUBLIC_IP`). Never from a file the agent can write. - **Validation before writing** (`PrivateAddress`): a well-formed IPv4 address, in a private range, inside the region's VPC CIDR (from IMDS `network/interfaces/macs//vpc-ipv4-cidr-blocks` on AWS, `CARLOS_PRIVATE_CIDR` otherwise), and not loopback, link-local or 169.254.0.0/16. Otherwise it is omitted, and forwarding to the box stays public. - **Routing data only.** It is dialled, and nothing grants anything on it. With region HA on (the edges claims cache exists exactly then), every publication reads the address source afresh and publishes only what it reports: a blank read publishes no address, instead of holding the last EIP, which may since have moved to the region's load balancer (A1-12). With region HA off the publisher is main's, EIP hold included. A host.env setting, when present, is the answer for its field, and one that does not validate publishes nothing rather than fall back to IMDS. **Census (platform-1e):** `edgesFileWriteClaims` is a kept authority row (F,J1). Its sites do not move; its digest does. The overlay re-digests it with its class, task and basis unchanged. No seal moves. - [ ] **Step 1: Write the failing tests** `internal/edgesfile/private_test.go`: ````go package edgesfile import "testing" // Census ruling region-ha-2a: a private address passes only when it is a // well-formed private IPv4 address inside the region network's CIDR, and // never loopback or link-local (the metadata range included). func TestPrivateAddress(t *testing.T) { vpc := []string{"10.0.0.0/16"} cases := []struct { name, addr string cidrs []string want, cidr string }{ {"inside the region network", "10.0.1.5", vpc, "10.0.1.5", "10.0.0.0/16"}, {"inside the second block", "172.31.4.9", []string{"10.0.0.0/16", " 172.31.0.0/16 "}, "172.31.4.9", "172.31.0.0/16"}, {"malformed", "10.0.1", vpc, "", ""}, {"a hostname", "box.internal", vpc, "", ""}, {"the metadata range", "169.254.169.254", []string{"169.254.0.0/16"}, "", ""}, {"loopback", "127.0.0.1", []string{"127.0.0.0/8"}, "", ""}, {"outside the region network", "10.1.1.5", vpc, "", ""}, {"a public address", "203.0.113.4", []string{"203.0.113.0/24"}, "", ""}, {"IPv6", "fd00::5", []string{"fd00::/8"}, "", ""}, {"no network to check it against", "10.0.1.5", nil, "", ""}, {"a malformed network", "10.0.1.5", []string{"10.0.0.0/33"}, "", ""}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { got, cidr := PrivateAddress(c.addr, c.cidrs) if got != c.want || cidr != c.cidr { t.Fatalf("PrivateAddress(%q, %v) = %q, %q; want %q, %q", c.addr, c.cidrs, got, cidr, c.want, c.cidr) } }) } } ```` `internal/host/boxaddress_test.go`: ````go package host import ( "context" "encoding/json" "net/http" "testing" "github.com/carlosframework/platform/internal/release" ) // staticAddresses is a fixed AddressSource. type staticAddresses BoxAddresses func (a staticAddresses) BoxAddresses() BoxAddresses { return BoxAddresses(a) } // countingAddresses counts reads, answering each from its list in turn. type countingAddresses struct { answers []BoxAddresses reads int } func (c *countingAddresses) BoxAddresses() BoxAddresses { a := c.answers[min(c.reads, len(c.answers)-1)] c.reads++ return a } func envOf(m map[string]string) func(string) string { return func(k string) string { return m[k] } } // The core source works with no provider adapter at all (§25). func TestEnvAddressesWithoutAProvider(t *testing.T) { got := envAddresses{getenv: envOf(map[string]string{"CARLOS_PRIVATE_IP": " 10.0.1.5 ", "CARLOS_PRIVATE_CIDR": "10.0.0.0/16", "CARLOS_PUBLIC_IP": "198.51.100.7"})}.BoxAddresses() if got != (BoxAddresses{Private: "10.0.1.5", PrivateCIDR: "10.0.0.0/16", Public: "198.51.100.7"}) { t.Fatalf("got %+v", got) } if got := (envAddresses{getenv: envOf(nil)}).BoxAddresses(); got != (BoxAddresses{}) { t.Fatalf("nothing set: %+v", got) } // CARLOS_PRIVATE_IP needs CARLOS_PRIVATE_CIDR beside it. if got := (envAddresses{getenv: envOf(map[string]string{"CARLOS_PRIVATE_IP": "10.0.1.5"})}).BoxAddresses(); got.Private != "" { t.Fatalf("a private address with no network: %+v", got) } got = envAddresses{getenv: envOf(map[string]string{"CARLOS_PRIVATE_IP": "169.254.169.254", "CARLOS_PRIVATE_CIDR": "169.254.0.0/16", "CARLOS_PUBLIC_IP": "box.example.com"})}.BoxAddresses() if got != (BoxAddresses{}) { t.Fatalf("invalid settings: %+v", got) } } func TestEnvAddressesOverrideTheProviderFieldByField(t *testing.T) { provider := &countingAddresses{answers: []BoxAddresses{{Private: "10.0.9.9", PrivateCIDR: "10.0.0.0/16", Public: "203.0.113.9"}}} got := envAddresses{getenv: envOf(map[string]string{"CARLOS_PUBLIC_IP": "198.51.100.7"}), provider: provider}.BoxAddresses() if got != (BoxAddresses{Private: "10.0.9.9", PrivateCIDR: "10.0.0.0/16", Public: "198.51.100.7"}) { t.Fatalf("got %+v", got) } // Both set: the provider is not asked. provider.reads = 0 envAddresses{getenv: envOf(map[string]string{"CARLOS_PRIVATE_IP": "10.0.1.5", "CARLOS_PRIVATE_CIDR": "10.0.0.0/16", "CARLOS_PUBLIC_IP": "198.51.100.7"}), provider: provider}.BoxAddresses() if provider.reads != 0 { t.Fatalf("provider asked %d times with both settings present", provider.reads) } // A bad private setting publishes no private address; the provider's // answer does not replace it. got = envAddresses{getenv: envOf(map[string]string{"CARLOS_PRIVATE_IP": "not-an-ip", "CARLOS_PRIVATE_CIDR": "10.0.0.0/16"}), provider: provider}.BoxAddresses() if got != (BoxAddresses{Public: "203.0.113.9"}) { t.Fatalf("bad setting: %+v", got) } // The provider's answer is validated too. bad := &countingAddresses{answers: []BoxAddresses{{Private: "10.9.9.9", PrivateCIDR: "10.0.0.0/16"}}} if got := (envAddresses{getenv: envOf(nil), provider: bad}).BoxAddresses(); got.Private != "" { t.Fatalf("an out-of-network provider address: %+v", got) } } func TestIMDSAddressesReadsTheLeavesAndTheVPCNetwork(t *testing.T) { leaves := map[string]string{ "/latest/meta-data/local-ipv4": "10.0.1.5", "/latest/meta-data/public-ipv4": "203.0.113.4", "/latest/meta-data/mac": "0e:12:34:56:78:9a", "/latest/meta-data/network/interfaces/macs/0e:12:34:56:78:9a/vpc-ipv4-cidr-blocks": "172.31.0.0/16\n10.0.0.0/16", } s := imdsAddresses{token: func() string { return "tok" }, leaf: func(token, path string) (string, int) { if token != "tok" { t.Fatalf("token %q", token) } v, ok := leaves[path] if !ok { return "", http.StatusNotFound } return v, http.StatusOK }} if got := s.BoxAddresses(); got != (BoxAddresses{Private: "10.0.1.5", PrivateCIDR: "10.0.0.0/16", Public: "203.0.113.4"}) { t.Fatalf("got %+v", got) } // An address outside the VPC's blocks is not published. leaves["/latest/meta-data/local-ipv4"] = "10.9.1.5" if got := s.BoxAddresses(); got.Private != "" || got.PrivateCIDR != "" { t.Fatalf("out of network: %+v", got) } // No VPC network answer: no private address. leaves["/latest/meta-data/local-ipv4"] = "10.0.1.5" delete(leaves, "/latest/meta-data/network/interfaces/macs/0e:12:34:56:78:9a/vpc-ipv4-cidr-blocks") if got := s.BoxAddresses(); got.Private != "" { t.Fatalf("no network: %+v", got) } // No public address (a box without one, or the EIP gone): blank. delete(leaves, "/latest/meta-data/public-ipv4") if got := s.BoxAddresses(); got.Public != "" { t.Fatalf("no public leaf: %+v", got) } noToken := imdsAddresses{token: func() string { return "" }, leaf: s.leaf} if got := noToken.BoxAddresses(); got != (BoxAddresses{}) { t.Fatalf("no token: %+v", got) } } func readEdgesFile(t *testing.T, s release.Store, box string) edgesFile { t.Helper() body, _, err := s.Get(context.Background(), edgesFileKey(box)) if err != nil { t.Fatal(err) } var f edgesFile if err := json.Unmarshal(body, &f); err != nil { t.Fatal(err) } return f } // Region HA spec 2 §6 (A1-12): with region HA on, every publication reads // the address source afresh, and a blank read publishes no address. The // last published EIP is never held: it may be the region's, which the // load balancer has taken. func TestEdgesFilePublishesOnlyWhatTheSourceReports(t *testing.T) { s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t) seedRoute(t, reg, "app.example.com") claims := newEdgesClaims("annie", (&claimsBook{}).read) src := &countingAddresses{answers: []BoxAddresses{ {Private: "10.0.1.5", PrivateCIDR: "10.0.0.0/16", Public: "203.0.113.9"}, {Private: "10.0.1.5", PrivateCIDR: "10.0.0.0/16"}, }} claims.addrs = src var last []byte ident := edgesFile{Box: "annie", BoxID: "annie"} // The loop's own reading (the eip argument) is not what is published. if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "198.51.100.1", &last, false, claims); err != nil { t.Fatal(err) } if f := readEdgesFile(t, s, "annie"); f.EIP != "203.0.113.9" || f.PrivateIP != "10.0.1.5" || f.PrivateCIDR != "10.0.0.0/16" { t.Fatalf("first publication: %+v", f) } if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "", &last, false, claims); err != nil { t.Fatal(err) } if f := readEdgesFile(t, s, "annie"); f.EIP != "" || f.PrivateIP != "10.0.1.5" { t.Fatalf("after a blank public read: eip %q private %q, want no public address", f.EIP, f.PrivateIP) } if src.reads != 2 { t.Fatalf("source read %d times, want once per publication", src.reads) } } // With region HA off there is no claims cache, and the publisher is // main's: no private address, and a blank probe holds the last EIP. func TestEdgesFileWithRegionHAOffKeepsTheEIPHold(t *testing.T) { s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t) seedRoute(t, reg, "app.example.com") var last []byte ident := edgesFile{Box: "annie"} if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "203.0.113.9", &last, false, nil); err != nil { t.Fatal(err) } if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "", &last, true, nil); err != nil { t.Fatal(err) } if f := readEdgesFile(t, s, "annie"); f.EIP != "203.0.113.9" || f.PrivateIP != "" || f.PrivateCIDR != "" { t.Fatalf("%+v", f) } } ```` The existing claims tests get a fixed address source, never IMDS: ````diff --- a/internal/host/edgesclaims_test.go +++ b/internal/host/edgesclaims_test.go @@ -50,6 +50,11 @@ func publishWithClaims(t *testing.T, reg *registry.Registry, claims *edgesClaims) map[string]edgeHost { t.Helper() + // These tests are about the hosts; the box's addresses come from a + // fixed source, never from IMDS. nil claims is region HA off. + if claims != nil { + claims.addrs = staticAddresses{Public: "1.2.3.4"} + } st := release.NewDirStore(t.TempDir()) var last []byte if err := edgesFileWriteClaims(context.Background(), st, reg, edgesFile{Box: "annie", BoxID: "annie"}, "1.2.3.4", &last, false, claims); err != nil { ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/edgesfile/ -run PrivateAddress -count=1; nice -n 19 go test ./internal/host/ -run 'Address|IMDS|EdgesFile' -count=1` Expected: FAIL to build (`undefined: PrivateAddress`, `undefined: BoxAddresses`, `claims.addrs undefined`). - [ ] **Step 3: Write the validator, the source and the publisher's change** `internal/edgesfile/private.go`: ````go package edgesfile import ( "net" "strings" ) // PrivateAddress validates a box's private address against its region // network's CIDR blocks (region HA spec 2 §6, census ruling region-ha-2a): // a well-formed IPv4 address, in a private range, neither loopback nor // link-local (169.254.0.0/16, the metadata range, included), and inside // one of cidrs. It returns the address and the block that holds it, both // canonical, or "", "". The writer publishes only what it passes, and a // reader dials only what it passes against its own region's network. func PrivateAddress(addr string, cidrs []string) (string, string) { ip := net.ParseIP(strings.TrimSpace(addr)) if ip == nil || ip.To4() == nil { return "", "" } ip = ip.To4() if !ip.IsPrivate() || ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsUnspecified() { return "", "" } for _, c := range cidrs { _, block, err := net.ParseCIDR(strings.TrimSpace(c)) if err != nil || block.IP.To4() == nil { continue } if block.Contains(ip) { return ip.String(), block.String() } } return "", "" } ```` `internal/host/boxaddress.go`: ````go package host import ( "net" "strings" "github.com/carlosframework/platform/internal/edgesfile" ) // BoxAddresses are the addresses this box publishes in its edges file for // other edges to dial (region HA spec 2 §6). Public is for an edge in // another region. Private, with PrivateCIDR (the region network it lies // in), is for an edge in the same region. They are routing data only: an // edge dials them, and nothing grants anything on them. "" is an address // the source could not report, or one that failed validation, and nothing // is published for it. type BoxAddresses struct { Private string PrivateCIDR string Public string } // AddressSource reads this box's addresses afresh on every call. The edges // file publisher asks it before every publication and publishes only what // it reports, so an address the box no longer holds is never republished. // Every source is root-configured: the cloud's metadata service, or // host.env. None reads a file the agent can write. type AddressSource interface { BoxAddresses() BoxAddresses } // envAddresses is the provider-neutral source (§25), from host.env: // CARLOS_PRIVATE_IP with CARLOS_PRIVATE_CIDR, and CARLOS_PUBLIC_IP. A field // whose setting is present is the answer for that field, and a value that // does not validate publishes nothing: an operator's setting is never // silently replaced. CARLOS_PRIVATE_IP without CARLOS_PRIVATE_CIDR is // invalid (census ruling region-ha-2a). A field left unset comes from // provider, the cloud's adapter, when there is one. type envAddresses struct { getenv func(string) string provider AddressSource } func (s envAddresses) BoxAddresses() BoxAddresses { ip := strings.TrimSpace(s.getenv("CARLOS_PRIVATE_IP")) cidr := strings.TrimSpace(s.getenv("CARLOS_PRIVATE_CIDR")) public := strings.TrimSpace(s.getenv("CARLOS_PUBLIC_IP")) privateSet, publicSet := ip != "" || cidr != "", public != "" out := BoxAddresses{} if privateSet { out.Private, out.PrivateCIDR = edgesfile.PrivateAddress(ip, []string{cidr}) } if publicSet { out.Public = ipv4Only(public) } if (!privateSet || !publicSet) && s.provider != nil { p := s.provider.BoxAddresses() if !privateSet { out.Private, out.PrivateCIDR = edgesfile.PrivateAddress(p.Private, []string{p.PrivateCIDR}) } if !publicSet { out.Public = ipv4Only(p.Public) } } return out } // ipv4Only is s when it is an IPv4 address, in its canonical form, and "" // otherwise. func ipv4Only(s string) string { ip := net.ParseIP(strings.TrimSpace(s)) if ip == nil || ip.To4() == nil { return "" } return ip.To4().String() } ```` `internal/host/awsaddr.go`: ````go package host import ( "net/http" "strings" "time" "github.com/carlosframework/platform/internal/edgesfile" ) // imdsAddresses is the AWS adapter of AddressSource (region HA spec 2 // §25): IMDSv2's local-ipv4, public-ipv4, and the VPC's CIDR blocks for // the primary interface (network/interfaces/macs//vpc-ipv4-cidr-blocks), // read on every call. IMDS reports only addresses this instance holds, so // once the region's EIP has moved to the load balancer no box reports it // (A1-12). A leaf that does not answer 200 is blank. type imdsAddresses struct { token func() string leaf func(token, path string) (string, int) } func newIMDSAddresses() imdsAddresses { client := &http.Client{Timeout: 2 * time.Second} return imdsAddresses{ token: func() string { return imdsToken(client) }, leaf: func(token, path string) (string, int) { return imdsLeaf(client, token, path) }, } } func (s imdsAddresses) BoxAddresses() BoxAddresses { token := s.token() if token == "" { return BoxAddresses{} } read := func(path string) string { v, code := s.leaf(token, path) if code != http.StatusOK { return "" } return v } out := BoxAddresses{Public: read("/latest/meta-data/public-ipv4")} ip := read("/latest/meta-data/local-ipv4") mac := read("/latest/meta-data/mac") if ip == "" || mac == "" || strings.ContainsAny(mac, "/\\ ") { return out } blocks := strings.Fields(read("/latest/meta-data/network/interfaces/macs/" + mac + "/vpc-ipv4-cidr-blocks")) out.Private, out.PrivateCIDR = edgesfile.PrivateAddress(ip, blocks) return out } // boxAddressSource is what the edges file publisher reads with region HA // on: host.env's settings, then IMDS on EC2. func boxAddressSource(getenv func(string) string) AddressSource { return envAddresses{getenv: getenv, provider: newIMDSAddresses()} } ```` ````diff --- a/internal/host/edgesclaims.go +++ b/internal/host/edgesclaims.go @@ -4,6 +4,7 @@ "context" "log" "math" + "os" "sort" "time" @@ -71,11 +72,15 @@ passDeadline time.Duration logf func(format string, a ...any) failingRun int // consecutive passes with a failed claim read + // addrs is the box's own addresses, read before every publication + // (region HA spec 2 §6). It rides here because this cache exists + // exactly when region HA is on. + addrs AddressSource } func newEdgesClaims(self string, read func(context.Context, string) (hostclaim.Observed, error)) *edgesClaims { return &edgesClaims{self: self, read: read, now: time.Now, entries: map[string]edgesClaimEntry{}, - passDeadline: edgesClaimPassDeadline, logf: log.Printf} + passDeadline: edgesClaimPassDeadline, logf: log.Printf, addrs: boxAddressSource(os.Getenv)} } // relocatableRow is spec 1 §2's relocatable host as a registry row shows ```` ````diff --- a/internal/host/edgesfile.go +++ b/internal/host/edgesfile.go @@ -205,6 +205,13 @@ Box string `json:"box"` EIP string `json:"eip"` Region string `json:"region,omitempty"` + // PrivateIP is the address an edge in this box's region dials (region + // HA spec 2 §6), and PrivateCIDR the region network that holds it, + // both validated by privateAddress before they are written. Routing + // data only. Additive and omitempty: a reader that does not know them, + // or finds them absent, dials EIP, as before. + PrivateIP string `json:"private_ip,omitempty"` + PrivateCIDR string `json:"private_cidr,omitempty"` // Placement is this box's CARLOS placement code — CARLOS_REGION verbatim // ("ie", "uk"), the OTHER region vocabulary (see Region above, issue // #329). Additive and omitempty like Region: a pre-placement body parses @@ -591,7 +598,15 @@ hosts[key] = edgeHost{Role: role, Wildcard: true, Alias: alias, Account: acct, App: app, Stamp: stamp} } - if eip == "" && *last != nil { + var private BoxAddresses + if claims != nil && claims.addrs != nil { + // Region HA spec 2 §6 (A1-12): publish only what the box's address + // source reports now. A blank read publishes no address, never the + // last one: that may be the region's EIP, which the load balancer + // has since taken. + private = claims.addrs.BoxAddresses() + eip = private.Public + } else if eip == "" && *last != nil { var prior edgesFile if err := json.Unmarshal(*last, &prior); err == nil { eip = prior.EIP @@ -600,6 +615,7 @@ f := ident f.V, f.EIP, f.Hosts, f.Generation = 1, eip, hosts, "" + f.PrivateIP, f.PrivateCIDR = private.Private, private.PrivateCIDR gateCopy := f gateCopy.Headroom = nil // a changing reading is not, by itself, a reason to write gate, err := json.Marshal(gateCopy) ```` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/edgesfile/ -count=1 && nice -n 19 go test ./internal/host/ -run 'Address|IMDS|EdgesFile|Edges|Headroom' -count=1` Expected: PASS: a malformed, a metadata-range, a loopback, a public, an IPv6 and an out-of-network address are each omitted. - [ ] **Step 5: Boundary list and census** ````diff --- a/region_ha_boundary_test.go +++ b/region_ha_boundary_test.go @@ -22,11 +22,15 @@ "internal/lbhealth/lbhealth.go", "internal/hostactors/roles.go", "internal/boxchoice/policy.go", + "internal/host/boxaddress.go", + "internal/edgesfile/private.go", } // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so // a reader can see where the provider lives. -var regionHAAdapterFiles = []string{} +var regionHAAdapterFiles = []string{ + "internal/host/awsaddr.go", +} // regionHAPackages are the packages spec 2 touches. Every S3 client built // in them goes through s3compat.Options(), so CARLOS_S3_ENDPOINT applies. ```` Add this comment line after the overlay's last comment line: ````text # Task 7: the box's own addresses (internal/host boxaddress.go, the provider-neutral source over CARLOS_PRIVATE_IP with CARLOS_PRIVATE_CIDR, and CARLOS_PUBLIC_IP, all from root-owned host.env; awsaddr.go, the IMDS adapter, which reads the VPC's CIDR blocks too) and the edges file's private_ip and private_cidr, validated before they are written by edgesfile.PrivateAddress (census ruling region-ha-2a, condition 1). edgesFileWriteClaims (authority, F,J1) is re-digested with its class, task, basis and sites unchanged: with region HA on it reads the address source before every publication and no longer holds a blank EIP. It adds no registry, /data, release or claim read, and nothing runs as root that did not before; the legacy seal stands. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/edgesfile f62c3973bb48fa36f561d26f496c0ba9a14fe882fe5843867e9f7a9eaf0abbdf @coverage internal/host 02e98726846daefb57c11e2731344f4f7ae7eb4d4a6ec32388290bf1369091a0 internal/host/edgesfile.go edgesFileWriteClaims authority F,J1 root-route;kept=F;scope=protected-decision-only registry:Account@10835,registry:Alias@5297,registry:Alias@5332,registry:Alias@5754,registry:Alias@5911,registry:Alias@5981,registry:Alias@6013,registry:Alias@6482,registry:Alias@8772,registry:App@10847,registry:DNSStatus@5429,registry:DNSStatus@5920,registry:Host@10480,registry:Host@2878,registry:Host@5389,registry:Host@6234,registry:Host@6567,registry:Host@8819,registry:Host@8866,registry:Host@8916,registry:Host@8951,registry:Host@9000,registry:Platform@2921,registry:Provisioned@2935,registry:Replica@2959,registry:Status@877,whole:Route@3212,whole:Route@323,whole:Route@3505,whole:Route@351,whole:RouteAlias@5282,whole:RouteAlias@8725,whole:RoutePath@3520,whole:WildcardDomain@8905 48bad9a5f49c5fca53fcf7bd6a241576fd2cd332974114e463b2cb3832e3598d ```` The appended row is `edgesFileWriteClaims`'s, re-digested: its kinds are main's row's, unchanged (check with the scratch test). Run the census checks and `go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1`. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/edgesfile/private.go internal/edgesfile/private_test.go internal/host/boxaddress.go internal/host/awsaddr.go internal/host/boxaddress_test.go internal/host/edgesclaims.go internal/host/edgesclaims_test.go internal/host/edgesfile.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv git commit -m "region HA 2a: publish the box's validated private address and only what its source reports" ``` --- ### Task 8: Forwarding dials the private address within a region, with the reader's region HA on **Files:** - Modify: `internal/edge/homemap.go` (`edgesFileView` reads `private_ip`, `private_cidr`, `region`, `placement`; `dialAddress`; `regionHAFromEnv`; `homeMap.privateDial`; `foldHomes`) - Modify: `internal/edge/homemap_test.go` (`foldHomes`'s new argument) - Create: `internal/edge/homemap_private_test.go` - Modify: the overlay **Interfaces:** - Consumes: `edgesfile.PrivateAddress` and `private_ip`/`private_cidr` from Task 7. - Produces: `func dialAddress(f, self edgesFileView, selfKnown, privateDial bool) string`; `func regionHAFromEnv(getenv func(string) string) bool` (Task 9 reuses it); `foldHomes(files, self, privateDial)`. `homeEntry.eip` is now the address to dial, so `Home`, `ExactHome` and `Owner` answer it, and the backhaul transport, keyed by that address, dials it. The hop is unchanged: one hop, today's credential and mesh key, the TLS name the host's. Only the address dialled changes, and only with **this edge's own** `CARLOS_REGION_HA` on (Astra finding 5): a box in this box's own AWS region and placement, publishing a `private_ip` that passes `PrivateAddress` against the network **this box's own file** publishes, is dialled there. Every other box, and every box with this edge's switch off, is dialled at its EIP, as before; so is every box when this edge cannot find its own file. - [ ] **Step 1: Write the failing tests** `internal/edge/homemap_private_test.go`: ````go package edge import "testing" // privateFleet publishes this edge's own file (annie, in ie, network // 10.0.0.0/16) and one file per case. func privateFleet(t *testing.T, m *homeMap) { t.Helper() putJSON(t, m.s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","private_ip":"10.0.1.1","private_cidr":"10.0.0.0/16","region":"eu-west-1","placement":"ie","hosts":{}}`) putJSON(t, m.s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","private_ip":"10.0.1.2","private_cidr":"10.0.0.0/16","region":"eu-west-1","placement":"ie","hosts":{"b.test":{"role":"home"}}}`) putJSON(t, m.s, "control/edges/carla.json", `{"v":1,"box":"carla","eip":"3.3.3.3","private_ip":"10.1.1.3","private_cidr":"10.1.0.0/16","region":"eu-west-2","placement":"uk","hosts":{"c.test":{"role":"home"}}}`) putJSON(t, m.s, "control/edges/dora.json", `{"v":1,"box":"dora","eip":"4.4.4.4","region":"eu-west-1","placement":"ie","hosts":{"d.test":{"role":"home"}}}`) putJSON(t, m.s, "control/edges/edna.json", `{"v":1,"box":"edna","eip":"5.5.5.5","private_ip":"10.0.1.5","region":"eu-west-1","placement":"fleet/shop","hosts":{"e.test":{"role":"home"}}}`) putJSON(t, m.s, "control/edges/fern.json", `{"v":1,"box":"fern","eip":"","private_ip":"10.0.1.6","region":"eu-west-1","placement":"ie","hosts":{"f.test":{"role":"home"}}}`) // Addresses a reader refuses: each falls back to the public address. putJSON(t, m.s, "control/edges/gail.json", `{"v":1,"box":"gail","eip":"7.7.7.7","private_ip":"10.0.1","region":"eu-west-1","placement":"ie","hosts":{"g.test":{"role":"home"}}}`) putJSON(t, m.s, "control/edges/hana.json", `{"v":1,"box":"hana","eip":"8.8.8.8","private_ip":"169.254.169.254","private_cidr":"169.254.0.0/16","region":"eu-west-1","placement":"ie","hosts":{"h.test":{"role":"home"}}}`) putJSON(t, m.s, "control/edges/iris.json", `{"v":1,"box":"iris","eip":"9.9.9.9","private_ip":"10.9.1.9","private_cidr":"10.9.0.0/16","region":"eu-west-1","placement":"ie","hosts":{"i.test":{"role":"home"}}}`) refreshHomeMap(t, m) } // Region HA spec 2 §6: with this edge's region HA on, a box in the same // region is dialled at its private address and a box in another region at // its own public address. An address that fails validation against this // edge's own network is refused, and the dial is public. func TestHomeMapDialsPrivateWithinTheRegionAndPublicBetween(t *testing.T) { m := newHomeMap(meshTestStore(t), "annie") m.privateDial = true privateFleet(t, m) for host, want := range map[string]string{ "b.test": "10.0.1.2", // same region and placement: private "c.test": "3.3.3.3", // another region: its own public address "d.test": "4.4.4.4", // a writer with no private_ip: public, as before "e.test": "5.5.5.5", // same AWS region, another placement: public "f.test": "10.0.1.6", // no public address, but private within the region "g.test": "7.7.7.7", // a malformed private address: refused "h.test": "8.8.8.8", // the metadata range: refused "i.test": "9.9.9.9", // outside this region's network: refused } { if addr, _, ok := m.Home(host); !ok || addr != want { t.Errorf("%s dials %q (ok=%v), want %q", host, addr, ok, want) } } // An edge whose own file is not in the map knows no region or network // of its own: every dial is public, and a box with no public address // is no target. ghost := newHomeMap(m.s, "ghost") ghost.privateDial = true refreshHomeMap(t, ghost) if addr, _, _ := ghost.Home("b.test"); addr != "2.2.2.2" { t.Errorf("without its own file: b.test dials %q, want 2.2.2.2", addr) } if _, _, ok := ghost.Home("f.test"); ok { t.Error("without its own file, a box with no public address became a target") } } // With this edge's region HA off, a peer publishing private_ip changes // nothing: every dial is public, as on main, during a mixed roll or a // rollback. func TestHomeMapWithRegionHAOffDialsPublicOnly(t *testing.T) { m := newHomeMap(meshTestStore(t), "annie") privateFleet(t, m) for host, want := range map[string]string{"b.test": "2.2.2.2", "c.test": "3.3.3.3", "d.test": "4.4.4.4"} { if addr, _, ok := m.Home(host); !ok || addr != want { t.Errorf("%s dials %q (ok=%v), want %q", host, addr, ok, want) } } if _, _, ok := m.Home("f.test"); ok { t.Error("a box with only a private address became a target with region HA off") } } func TestEdgeHomeMapFollowsRegionHA(t *testing.T) { e := &Edge{staticStore: meshTestStore(t), box: "annie"} t.Setenv("CARLOS_REGION_HA", "") if e.newHomeMap().privateDial { t.Fatal("region HA unset dials private") } t.Setenv("CARLOS_REGION_HA", "1") if !e.newHomeMap().privateDial { t.Fatal("region HA on does not dial private") } } ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/edge/ -run 'DialsPrivate|PublicOnly|FollowsRegionHA' -count=1` Expected: FAIL to build (`m.privateDial undefined`). - [ ] **Step 3: Choose the dial address** ````diff --- a/internal/edge/homemap.go +++ b/internal/edge/homemap.go @@ -80,10 +80,12 @@ "os" "slices" "sort" + "strconv" "strings" "sync" "time" + "github.com/carlosframework/platform/internal/edgesfile" "github.com/carlosframework/platform/internal/release" ) @@ -132,6 +134,39 @@ Box string `json:"box"` EIP string `json:"eip"` Hosts map[string]edgesHostView `json:"hosts"` + // PrivateIP, PrivateCIDR, Region and Placement choose the dial address + // (region HA spec 2 §6, dialAddress). All are absent from a writer that + // predates them, and the dial is then EIP, as before. + PrivateIP string `json:"private_ip"` + PrivateCIDR string `json:"private_cidr"` + Region string `json:"region"` + Placement string `json:"placement"` +} + +// dialAddress is where this edge dials f's box (region HA spec 2 §6). With +// privateDial (this edge's own region HA switch) on, a box that publishes +// the same AWS region and placement as this one is dialled at its private +// address, so the dial stays inside the region's network, but only when +// that address passes edgesfile.PrivateAddress against the network this +// box's own file publishes (census ruling region-ha-2a). Every other box, +// and every box with privateDial off, is dialled at its public address, +// as before. self is this box's own file, which selfKnown says was found. +// "" is nothing to dial. +func dialAddress(f, self edgesFileView, selfKnown, privateDial bool) string { + if privateDial && selfKnown && f.Region != "" && f.Placement != "" && + f.Region == self.Region && f.Placement == self.Placement { + if addr, _ := edgesfile.PrivateAddress(f.PrivateIP, []string{self.PrivateCIDR}); addr != "" { + return addr + } + } + return f.EIP +} + +// regionHAFromEnv is CARLOS_REGION_HA as strconv.ParseBool reads it, the +// reading every region HA switch makes. +func regionHAFromEnv(getenv func(string) string) bool { + on, err := strconv.ParseBool(strings.TrimSpace(getenv("CARLOS_REGION_HA"))) + return err == nil && on } // edgesHostView is the per-host object. An object rather than a bare string is @@ -161,6 +196,8 @@ // homeEntry is one folded answer: where to dial, and which box that is. type homeEntry struct { + // eip is the address to dial: dialAddress's answer, the home box's + // private address within its region and its EIP from anywhere else. eip string box string // platform carries the publisher's class mark through the fold. @@ -238,6 +275,10 @@ keyMu sync.RWMutex keyCurrent []byte keyNext []byte + + // privateDial is this edge's region HA switch: only with it on does + // the fold dial a same-region box's private address (dialAddress). + privateDial bool } func newHomeMap(s release.Store, self string) *homeMap { @@ -253,7 +294,9 @@ if e.staticStore == nil { return nil } - return newHomeMap(e.staticStore, e.box) + m := newHomeMap(e.staticStore, e.box) + m.privateDial = regionHAFromEnv(os.Getenv) + return m } // Refresh rebuilds the host fold and re-reads the mesh keys. The two halves @@ -330,7 +373,7 @@ if failed > 0 { m.logPartialPoll(failed, tried, lastErr) } - hosts, tied := foldHomes(next, m.self) + hosts, tied := foldHomes(next, m.self, m.privateDial) if len(tied) > 0 { // One claim revision names one executor, so two boxes publishing // one host at the same stamp is a fault upstream. The fold stays @@ -532,9 +575,11 @@ // target would have the edge dial itself. Excluded by the file's own box // field rather than by its object name, because the field is what the // publisher stamps and what self is derived to match. -// - NO EIP. Task 1 publishes a box with no discoverable public IP as -// eip:"" so the DR runbook still gets its hosts list. There is nothing to -// dial, so it is not a target. +// - NO DIAL ADDRESS. Task 1 publishes a box with no discoverable public IP +// as eip:"" so the DR runbook still gets its hosts list. There is +// nothing to dial, so it is not a target. The address is dialAddress's: +// a box in this box's own region is dialled at its private address when +// it publishes one (region HA spec 2 §6), every other at its EIP. // - A ROLE THIS BUILD DOES NOT KNOW. Only "home" routes. The role field // exists so read-local roles can be added later; a reader that treated // every role as a home would send traffic to a box that had declared it @@ -546,28 +591,36 @@ // split-and-wrong: the whole fleet aiming at one box is a condition that heals // on the next publish, where half the fleet aiming at each is a condition that // looks fine from any single box. -func foldHomes(files map[string]edgesFileView, self string) (map[string]homeEntry, []string) { +func foldHomes(files map[string]edgesFileView, self string, privateDial bool) (map[string]homeEntry, []string) { keys := make([]string, 0, len(files)) for k := range files { keys = append(keys, k) } slices.Sort(keys) + var own edgesFileView + ownKnown := false + for _, k := range keys { + if f := files[k]; self != "" && f.Box == self { + own, ownKnown = f, true + } + } hosts := make(map[string]homeEntry) for _, k := range keys { f := files[k] if f.Box == "" || f.Box == self { continue } + addr := dialAddress(f, own, ownKnown, privateDial) for host, h := range f.Hosts { if host == "" { continue } switch h.Role { case edgeRoleHome: - // A dial target needs an address. A file with no EIP (a box + // A dial target needs an address. A file with none (a box // with no discoverable public IP) still carries its cert // halves below; its dial halves are simply not dialable. - if f.EIP == "" { + if addr == "" { if !h.Wildcard || !strings.HasPrefix(host, "*.") { continue } @@ -583,7 +636,7 @@ default: continue } - in := homeEntry{eip: f.EIP, box: f.Box, platform: h.Platform && !h.Alias && !h.Wildcard, alias: h.Alias, wildcard: h.Wildcard, dial: h.Role == edgeRoleHome, + in := homeEntry{eip: addr, box: f.Box, platform: h.Platform && !h.Alias && !h.Wildcard, alias: h.Alias, wildcard: h.Wildcard, dial: h.Role == edgeRoleHome, account: h.Account, app: h.App, stamp: h.Stamp} prev, dup := hosts[host] if !dup { ```` ````diff --- a/internal/edge/homemap_test.go +++ b/internal/edge/homemap_test.go @@ -1715,7 +1715,7 @@ "control/edges/betty.json": file("betty", "2.2.2.2", map[string]edgesHostView{ "tied.test": {Role: "home", Stamp: 7}, "fine.test": {Role: "home", Stamp: 9}, "plain.test": {Role: "home"}}), } - hosts, tied := foldHomes(files, "carol") + hosts, tied := foldHomes(files, "carol", false) if len(tied) != 1 || tied[0] != "tied.test" { t.Fatalf("tied = %v, want [tied.test]: unstamped duplicates are normal (static sites run on every box) and are not reported", tied) } @@ -1735,7 +1735,7 @@ "control/edges/betty.json": file("betty", "2.2.2.2", map[string]edgesHostView{"h.test": {Role: "home", Stamp: 7}}), "control/edges/carol.json": file("carol", "3.3.3.3", map[string]edgesHostView{"h.test": {Role: "home", Stamp: 7}}), } - hosts, tied := foldHomes(files, "dave") + hosts, tied := foldHomes(files, "dave", false) if len(tied) != 1 || tied[0] != "h.test" { t.Fatalf("tied = %v, want [h.test]", tied) } @@ -1751,7 +1751,7 @@ "control/edges/annie.json": file("annie", "1.1.1.1", map[string]edgesHostView{"h.test": {Role: "home", Stamp: 7}}), "control/edges/betty.json": file("betty", "2.2.2.2", map[string]edgesHostView{"h.test": {Role: "home", Stamp: 7}}), } - hosts, tied := foldHomes(files, "annie") + hosts, tied := foldHomes(files, "annie", false) if len(tied) != 1 || tied[0] != "h.test" { t.Fatalf("tied = %v, want [h.test]", tied) } ```` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/edge/ -run 'HomeMap|Backhaul|Hop|KnownHost|DialsPrivate|PublicOnly|FollowsRegionHA' -count=1` Expected: PASS. - [ ] **Step 5: Census** Add this comment line after the overlay's last comment line: ````text # Task 8: with the reader's own region HA on, forwarding dials a same-region box's private address when it passes edgesfile.PrivateAddress against the reader's own published network, and every other box at its public address (internal/edge homemap.go: dialAddress, regionHAFromEnv, edgesFileView's private_ip, private_cidr, region and placement). The home map is not a root-read unit. Envelope-only, and the legacy seal stands. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/edge 2c474d685471295daf20f061e595becab78175c5754dbef0d2187f0f8519fbe3 ```` Run the census checks. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/edge/homemap.go internal/edge/homemap_test.go internal/edge/homemap_private_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv git commit -m "region HA 2a: dial a box's private address within its region" ``` --- ### Task 9: The load balancer health answer **Files:** - Create: `internal/edge/lbhealth.go`, `internal/edge/lbhealth_test.go` - Modify: `internal/edge/edge.go` (`Edge.lbHealth`, set in `New`; `Watch` beats; `httpFallbackHandler` answers first) - Modify: `internal/edge/homemap.go` (`homeMap.health`; `Refresh` latches and stamps) - Modify: `internal/edge/certsync.go` (`certMirror.health`; a landed `Refresh` latches) - Modify: `internal/activator/activator.go` (`lastSweep`, `LastSweep`) - Create: `internal/activator/lastsweep_test.go` - Modify: `internal/host/claimwiring.go` (`startRegional` marks the box hosting) - Create: `internal/host/claimwiring_hosting_test.go` - Modify: `region_ha_boundary_test.go`, the overlay **Interfaces:** - Consumes: Task 2's `lbhealth.State` and `lbhealth.Serve`; Task 8's `regionHAFromEnv`. - Produces: `func lbHealthFromEnv(getenv func(string) string, act Activator) *lbhealth.State` (nil with region HA off); `func (e *Edge) lbHealthAnswer(w http.ResponseWriter, r *http.Request) bool`; `func (a *activator.Service) LastSweep() time.Time`; the hosting hook `lbhealth.Default.SetHosting(true)` in `ClaimWiring.startRegional`. **Seam for 2b:** 2b's mode selection takes `SetHosting` over; joining mode never sets it. The answer is served on port 80 only (the `:80` handler, `httpFallbackHandler`), ahead of every host-based branch, and reads only in-memory state. Each condition is fed by the loop it describes: | Condition | Fed by | |---|---| | hosting | `ClaimWiring.startRegional`: the claim agent only starts with its host-actor enrollment and a matching roster row | | forwarding | `homeMap.Refresh`: the first successful map load and the first successful mesh key load, latched | | certificates | `certMirror.Refresh`: the first refresh that lands `control/certs/` in the local cache the handshake reads, latched (spec rev 15: certificates are per host, so there is no single default certificate) | | edge | `Edge.Watch`: every completed registry poll (every 2 seconds) | | activator | `activator.Service.LastSweep`: every finished sweep (at most 30 seconds apart) | | refresh | `homeMap.Refresh`: every completed attempt, failed or not | `RunWithActivatorReady`, `Edge.ServeHTTP` and `runAgent` are accepted-legacy or kept authority units; none changes. The edge's state is `lbhealth.Default` (process-wide) because the agent runs the edge and the activator in one process; tests give each edge its own `State`. - [ ] **Step 1: Write the failing tests** `internal/edge/lbhealth_test.go`: ````go package edge import ( "context" "errors" "net/http" "net/http/httptest" "strings" "testing" "time" "golang.org/x/crypto/acme/autocert" "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/release" ) // sweeper is an Activator whose only working method is the sweep beat. type sweeper struct { Activator at time.Time } func (s sweeper) LastSweep() time.Time { return s.at } func lbEnv(v string) func(string) string { return func(k string) string { if k == "CARLOS_REGION_HA" { return v } return "" } } func TestLBHealthStateFollowsRegionHA(t *testing.T) { if s := lbHealthFromEnv(lbEnv(""), nil); s != nil { t.Fatal("region HA unset armed the health answer") } if s := lbHealthFromEnv(lbEnv("0"), nil); s != nil { t.Fatal("region HA off armed the health answer") } at := time.Now() s := lbHealthFromEnv(lbEnv("1"), nil) if s != lbhealth.Default { t.Fatal("region HA on: not the process's state") } t.Cleanup(func() { lbhealth.Default.SetActivator(nil) }) lbHealthFromEnv(lbEnv("true"), sweeper{at: at}) if got := lbhealth.Default.Failing(at); strings.Contains(strings.Join(got, ","), lbhealth.CondActivator) { t.Fatalf("the activator's sweep was not wired: %v", got) } } // fallbackGet drives the :80 handler with one request. func fallbackGet(e *Edge, method, path, host string) *httptest.ResponseRecorder { w := httptest.NewRecorder() r := httptest.NewRequest(method, path, nil) r.Host = host e.httpFallbackHandler().ServeHTTP(w, r) return w } // healthyState meets every condition now. func healthyState() *lbhealth.State { s := &lbhealth.State{} now := time.Now() s.SetHosting(true) s.MapLoaded() s.KeyLoaded() s.CertsLoaded() s.EdgeBeat(now) s.RefreshAttempted(now) s.SetActivator(func() time.Time { return now }) return s } // The answer is on :80, for any host (the load balancer sends the // target's address), GET or HEAD, ahead of the redirect and unknown-host // branches. With region HA off the path is ordinary :80 traffic. func TestLBHealthAnswersOnPort80ForAnyHost(t *testing.T) { e, err := New(openReg(t), nil) if err != nil { t.Fatal(err) } t.Cleanup(e.probers.Stop) e.lbHealth = healthyState() for _, host := range []string{"10.0.1.7", "10.0.1.7:80", "unknown.example.com"} { w := fallbackGet(e, http.MethodGet, lbhealth.Path, host) if w.Code != http.StatusOK || w.Body.String() != "ok\n" { t.Fatalf("host %s: %d %q", host, w.Code, w.Body.String()) } } if w := fallbackGet(e, http.MethodHead, lbhealth.Path, "10.0.1.7"); w.Code != http.StatusOK { t.Fatalf("HEAD: %d", w.Code) } joining := healthyState() joining.SetHosting(false) e.lbHealth = joining if w := fallbackGet(e, http.MethodGet, lbhealth.Path, "10.0.1.7"); w.Code != http.StatusServiceUnavailable || !strings.Contains(w.Body.String(), lbhealth.CondHosting) { t.Fatalf("joining: %d %q", w.Code, w.Body.String()) } // A POST, or another path, is ordinary :80 traffic. if w := fallbackGet(e, http.MethodPost, lbhealth.Path, "10.0.1.7"); w.Code == http.StatusOK || w.Code == http.StatusServiceUnavailable { t.Fatalf("POST answered by the health check: %d", w.Code) } e.lbHealth = nil if w := fallbackGet(e, http.MethodGet, lbhealth.Path, "10.0.1.7"); w.Code == http.StatusOK || strings.Contains(w.Body.String(), "unavailable:") { t.Fatalf("region HA off answered the health check: %d %q", w.Code, w.Body.String()) } } // blockingStore makes every List wait until release is closed, and fail // while down is set: a refresh stuck inside one attempt, and a bucket // outage. type blockingStore struct { release.Store block chan struct{} down bool entered chan struct{} } func (s *blockingStore) List(ctx context.Context, prefix string) ([]string, error) { if s.block != nil { s.entered <- struct{}{} <-s.block } if s.down { return nil, errors.New("bucket unreachable") } return s.Store.List(ctx, prefix) } func (s *blockingStore) Get(ctx context.Context, key string) ([]byte, string, error) { if s.down { return nil, "", errors.New("bucket unreachable") } return s.Store.Get(ctx, key) } // The home map latches its first loaded map and key, stamps every attempt // it finishes, failed or not, and stamps nothing while one is stuck. func TestHomeMapFeedsTheHealthState(t *testing.T) { inner := meshTestStore(t) if err := mintMeshKey(context.Background(), inner); err != nil { t.Fatal(err) } putJSON(t, inner, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`) s := &blockingStore{Store: inner} h := &lbhealth.State{} m := newHomeMap(s, "annie") m.health = h refreshHomeMap(t, m) now := time.Now() failing := strings.Join(h.Failing(now), ",") if strings.Contains(failing, lbhealth.CondForwarding) || strings.Contains(failing, lbhealth.CondRefresh) { t.Fatalf("after a good refresh: %s", failing) } // A bucket outage: the attempt fails but finishes, and the latches hold. s.down = true before := time.Now() if err := m.Refresh(context.Background()); err == nil { t.Fatal("refresh against a down bucket succeeded") } if got := strings.Join(h.Failing(before.Add(lbhealth.RefreshWindow)), ","); strings.Contains(got, lbhealth.CondForwarding) || strings.Contains(got, lbhealth.CondRefresh) { t.Fatalf("a failed attempt dropped forwarding or went unstamped: %s", got) } // A stuck attempt stamps nothing. s.down = false s.block, s.entered = make(chan struct{}), make(chan struct{}, 1) done := make(chan struct{}) go func() { _ = m.Refresh(context.Background()); close(done) }() <-s.entered stuckAt := time.Now() if got := strings.Join(h.Failing(stuckAt.Add(lbhealth.RefreshWindow+time.Second)), ","); !strings.Contains(got, lbhealth.CondRefresh) { t.Fatalf("a refresh stuck for longer than the window still counts: %s", got) } close(s.block) <-done } // A key that never loads leaves forwarding unarmed, whatever the map did. func TestHomeMapWithNoMeshKeyIsNotArmed(t *testing.T) { s := meshTestStore(t) putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{}}`) h := &lbhealth.State{} m := newHomeMap(s, "annie") m.health = h refreshHomeMap(t, m) if got := strings.Join(h.Failing(time.Now()), ","); !strings.Contains(got, lbhealth.CondForwarding) { t.Fatalf("no mesh key, yet forwarding armed: %s", got) } } func TestCertMirrorLatchesItsFirstLoad(t *testing.T) { s := meshTestStore(t) h := &lbhealth.State{} m := newCertMirror(s, autocert.DirCache(t.TempDir())) m.health = h if !strings.Contains(strings.Join(h.Failing(time.Now()), ","), lbhealth.CondCerts) { t.Fatal("certificates counted before any load") } if err := m.Refresh(context.Background()); err != nil { t.Fatal(err) } if strings.Contains(strings.Join(h.Failing(time.Now()), ","), lbhealth.CondCerts) { t.Fatal("a loaded mirror did not latch the certificate condition") } } // The edge's registry watch beats on every poll it completes. func TestRegistryWatchBeats(t *testing.T) { e, err := New(openReg(t), nil) if err != nil { t.Fatal(err) } t.Cleanup(e.probers.Stop) h := &lbhealth.State{} e.lbHealth = h stop := make(chan struct{}) t.Cleanup(func() { close(stop) }) go e.Watch(stop) deadline := time.Now().Add(5 * time.Second) for strings.Contains(strings.Join(h.Failing(time.Now()), ","), lbhealth.CondEdge) { if time.Now().After(deadline) { t.Fatal("the registry watch never beat") } time.Sleep(50 * time.Millisecond) } } ```` `internal/activator/lastsweep_test.go`: ````go package activator import ( "context" "testing" "time" ) // Region HA spec 2 §6: the sweeper's finished sweeps are the hosting // loop's beat. Nothing has swept before Start. func TestLastSweepBeatsAfterEachSweep(t *testing.T) { a, _, _, _ := testActivator(t, 400*time.Millisecond) // sweeps every 100ms if !a.LastSweep().IsZero() { t.Fatal("swept before Start") } var none *Service if !none.LastSweep().IsZero() { t.Fatal("a nil service swept") } a.Start() t.Cleanup(func() { a.Shutdown(context.Background()) }) waitFor(t, "a sweep", func() bool { return !a.LastSweep().IsZero() }) first := a.LastSweep() if time.Since(first) > 5*time.Second { t.Fatalf("beat %v is not wall-clock time", first) } waitFor(t, "a later sweep", func() bool { return a.LastSweep().After(first) }) } ```` `internal/host/claimwiring_hosting_test.go`: ````go package host import ( "testing" "github.com/carlosframework/platform/internal/hostwrite" "github.com/carlosframework/platform/internal/lbhealth" ) // Region HA spec 2 §5, §6: a claim agent that is running is enrolled, so // it is hosting; an agent with claim mode off never says so. Plan 2b's // joining mode is what leaves this false. func TestClaimWiringStartMarksTheBoxHosting(t *testing.T) { lbhealth.Default.SetHosting(false) t.Cleanup(func() { lbhealth.Default.SetHosting(false) }) off := &ClaimWiring{} if err := off.startRegional(t.Context(), nil, nil); err != nil { t.Fatal(err) } if lbhealth.Default.Hosting() { t.Fatal("claim mode off marked the box hosting") } loop := testRegionalLoop{started: make(chan struct{})} w := &ClaimWiring{Writes: hostwrite.Config{Enabled: true}, Providers: ClaimProviders{Regional: loop}} if err := w.startRegional(t.Context(), nil, nil); err != nil { t.Fatal(err) } <-loop.started if !lbhealth.Default.Hosting() { t.Fatal("a running claim agent is not hosting") } if err := w.close(t.Context()); err != nil { t.Fatal(err) } } ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/edge/ -run 'LBHealth|FeedsTheHealth|NoMeshKeyIsNotArmed|CertMirrorLatches|RegistryWatchBeats' -count=1` Expected: FAIL to build (`undefined: lbHealthFromEnv`, `e.lbHealth undefined`). The activator test fails to build on `a.LastSweep undefined`; the host test builds and fails (`a running claim agent is not hosting`). - [ ] **Step 3: Write the answer and its feeds** `internal/edge/lbhealth.go`: ````go package edge import ( "net/http" "time" "github.com/carlosframework/platform/internal/lbhealth" ) // The load balancer health answer (region HA spec 2 §6): GET or HEAD // /.carlos/lb-health on port 80, from any host. The network load // balancer's HTTP check sends the target's address as the Host header, so // the steering probe, which answers only for its own hostname, cannot // serve it. Every condition is this box's own; lbhealth has the list. // lbHealthFromEnv is the state the edge answers from: the process's // (lbhealth.Default) with CARLOS_REGION_HA on, and nil with it off, which // leaves the path to the ordinary :80 handling exactly as before. The // activator's sweep is the hosting loop's beat; an edge with no activator // (carlos edge) has none, and its answer is never 200. func lbHealthFromEnv(getenv func(string) string, act Activator) *lbhealth.State { if !regionHAFromEnv(getenv) { return nil } s := lbhealth.Default if a, ok := act.(interface{ LastSweep() time.Time }); ok { s.SetActivator(a.LastSweep) } return s } // lbHealthAnswer answers the health check when r is one and region HA is // on, reporting whether it did. func (e *Edge) lbHealthAnswer(w http.ResponseWriter, r *http.Request) bool { if e.lbHealth == nil { return false } return lbhealth.Serve(w, r, e.lbHealth, time.Now()) } ```` ````diff --- a/internal/edge/edge.go +++ b/internal/edge/edge.go @@ -56,6 +56,7 @@ "github.com/carlosframework/platform/internal/gate" "github.com/carlosframework/platform/internal/geo" "github.com/carlosframework/platform/internal/geoip" + "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/provision" "github.com/carlosframework/platform/internal/registry" "github.com/carlosframework/platform/internal/release" @@ -131,6 +132,10 @@ // environment; see steeringprobe.go for why it is a serving concern at // all. steeringHost string + // lbHealth is the load balancer health state (region HA spec 2 §6): + // lbhealth.Default with CARLOS_REGION_HA on, nil and inert with it + // off. Set once in New. + lbHealth *lbhealth.State // appsDomain is the platform apps domain (PLATFORM_APPS_DOMAIN, // normalized), read once in New; cookieLogged is the set of route hosts // whose cookie rewrite has been logged. Both serve cookiescope.go. "" @@ -368,7 +373,8 @@ // New builds an Edge over reg and performs the startup reload. act may be nil // (no hibernation). func New(reg *registry.Registry, act Activator) (*Edge, error) { - e := &Edge{reg: reg, act: act, gate: gate.New(), probers: newProberSet(), gzipOn: gzipEnabled(), steeringHost: steeringProbeHostFromEnv(), appsDomain: appsDomainFromEnv()} + e := &Edge{reg: reg, act: act, gate: gate.New(), probers: newProberSet(), gzipOn: gzipEnabled(), steeringHost: steeringProbeHostFromEnv(), appsDomain: appsDomainFromEnv(), + lbHealth: lbHealthFromEnv(os.Getenv, act)} e.geo = geo.NewSwappable() e.geoLookup = e.geo.Lookup // Generations (zero-downtime spec §7.2), before the startup Reload so @@ -2226,6 +2232,12 @@ if h, _, err := net.SplitHostPort(host); err == nil { host = h } + // The load balancer's health check (region HA spec 2 §6, lbhealth.go): + // any host, so it comes before every host-based branch. Inert with + // region HA off. + if e.lbHealthAnswer(w, r) { + return + } // The steering hostname answers here too (ServeHTTP's own first // branch, and the same reasoning): the health check is HTTPS today, // but a :80 probe — or an operator's curl — must not be handed this @@ -2305,6 +2317,9 @@ continue } errs = 0 + // The edge's own loop is alive and its routing table readable + // (region HA spec 2 §6). A wedged poll stops beating. + e.lbHealth.EdgeBeat(time.Now()) if fp == last { continue } ```` ````diff --- a/internal/edge/homemap.go +++ b/internal/edge/homemap.go @@ -86,6 +86,7 @@ "time" "github.com/carlosframework/platform/internal/edgesfile" + "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/release" ) @@ -279,6 +280,11 @@ // privateDial is this edge's region HA switch: only with it on does // the fold dial a same-region box's private address (dialAddress). privateDial bool + + // health is the load balancer health state (region HA spec 2 §6), + // nil without region HA: Refresh latches the first loaded map and key + // and stamps every attempt it completes. + health *lbhealth.State } func newHomeMap(s release.Store, self string) *homeMap { @@ -296,6 +302,7 @@ } m := newHomeMap(e.staticStore, e.box) m.privateDial = regionHAFromEnv(os.Getenv) + m.health = e.lbHealth return m } @@ -303,10 +310,23 @@ // fail INDEPENDENTLY — a bucket that answered the edges list but not the key // object should still land the fold — so both run and their errors join. // Whatever fails leaves its own last-good in place; the caller logs. +// +// For the load balancer's health answer (region HA spec 2 §6) each half +// latches its first success, and every call that returns stamps an +// attempt, failed or not: a bucket outage after arming leaves the box +// healthy, and only a refresh stuck inside one attempt goes stale. func (m *homeMap) Refresh(ctx context.Context) error { m.refreshMu.Lock() defer m.refreshMu.Unlock() - return errors.Join(m.refreshHosts(ctx), m.refreshKeys(ctx)) + hostsErr, keysErr := m.refreshHosts(ctx), m.refreshKeys(ctx) + if hostsErr == nil { + m.health.MapLoaded() + } + if current, _ := m.meshKeys(); keysErr == nil && len(current) > 0 { + m.health.KeyLoaded() + } + m.health.RefreshAttempted(time.Now()) + return errors.Join(hostsErr, keysErr) } // refreshHosts rebuilds the whole map from one list plus a get per key — no ```` ````diff --- a/internal/edge/certsync.go +++ b/internal/edge/certsync.go @@ -50,6 +50,7 @@ "golang.org/x/crypto/acme/autocert" "golang.org/x/net/idna" + "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/release" ) @@ -516,6 +517,10 @@ // first-and-every-tenth log cadence. partialPolls int + // health is the load balancer health state the first landed refresh + // latches (region HA spec 2 §6); nil without region HA. + health *lbhealth.State + // synced remembers, per name, the mirrored bytes last reconciled into the // local cache, so a steady state costs no local I/O at all. syncMu sync.Mutex @@ -546,6 +551,7 @@ return nil } m := newCertMirror(e.staticStore, local) + m.health = e.lbHealth m.gated = func() bool { _, held := e.issuerGate() return !held @@ -752,6 +758,10 @@ m.snap = snap m.mu.Unlock() m.syncLocal(ctx, snap) + // The local cache the handshake reads now holds what the bucket does: + // the certificate condition is met for good (§6). A later failed + // refresh keeps serving this copy and leaves the latch set. + m.health.CertsLoaded() if m.wake != nil && wildcardMaterialChanged(prev, snap) { // A takeover (or a first wildcard for a new account) reaches the // serving path in one poll instead of up to twelve hours. ```` ````diff --- a/internal/activator/activator.go +++ b/internal/activator/activator.go @@ -1237,6 +1237,10 @@ sweepStop chan struct{} sweepDone chan struct{} + // lastSweep is when the sweeper last finished a sweep, in unix + // nanoseconds (0: never): the hosting loop's own beat, which the load + // balancer's health answer reads (region HA spec 2 §6, LastSweep). + lastSweep atomic.Int64 // gens is the generations switch (SetGenerations) as read at New. Off: // every spawn binds the canonical socket and no retire hook fires. @@ -4439,9 +4443,25 @@ return case <-t.C: a.sweepOnce() + a.lastSweep.Store(time.Now().UnixNano()) } } }() +} + +// LastSweep is when the sweeper last finished a sweep, or the zero time +// before its first. The sweep runs at least every 30 seconds, so a box +// whose last sweep is older than a minute has a hosting loop that is stuck +// (region HA spec 2 §6). A nil service has never swept. +func (a *Service) LastSweep() time.Time { + if a == nil { + return time.Time{} + } + n := a.lastSweep.Load() + if n == 0 { + return time.Time{} + } + return time.Unix(0, n) } // markCleanParkLocked records that a clean-exit park is stopping e's current ```` ````diff --- a/internal/host/claimwiring.go +++ b/internal/host/claimwiring.go @@ -10,6 +10,7 @@ "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostguard" "github.com/carlosframework/platform/internal/hostwrite" + "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/provision" "github.com/carlosframework/platform/internal/schedule" ) @@ -94,6 +95,12 @@ w.cancel = cancel w.joined = make(chan struct{}) go func() { defer close(w.joined); w.Providers.Regional.Loop(workerCtx, serial, report) }() + // The claim agent is running: it refused to start without its + // host-actor enrollment and a matching roster row, so this box is + // hosting (region HA spec 2 §5, §6). Plan 2b's mode selection owns + // this flag from then on: joining mode leaves it false, and only an + // active roster entry with an armed guard sets it. + lbhealth.Default.SetHosting(true) return nil } func (w *ClaimWiring) close(ctx context.Context) error { ```` - [ ] **Step 4: Run the tests** ```bash nice -n 19 go test ./internal/edge/ -run 'LBHealth|FeedsTheHealth|NoMeshKeyIsNotArmed|CertMirrorLatches|RegistryWatchBeats|HomeMap|Backhaul|Hop|HTTPFallback|FollowsRegionHA|PublicOnly' -count=1 nice -n 19 go test ./internal/activator/ -run LastSweep -count=1 nice -n 19 go test ./internal/host/ -run ClaimWiring -count=1 ``` Expected: PASS. - [ ] **Step 5: Boundary list and census** ````diff --- a/region_ha_boundary_test.go +++ b/region_ha_boundary_test.go @@ -24,6 +24,7 @@ "internal/boxchoice/policy.go", "internal/host/boxaddress.go", "internal/edgesfile/private.go", + "internal/edge/lbhealth.go", } // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so ```` Add this comment line after the overlay's last comment line: ````text # Task 9: the load balancer health answer (internal/edge lbhealth.go, answered in httpFallbackHandler; the home map and the certificate mirror latch their first loads, Edge.Watch beats, the activator's sweeper beats through Service.LastSweep, and ClaimWiring.startRegional marks the box hosting). None of the changed units is a root-read unit, and the answer reads only in-memory state. Envelope-only, and the legacy seal stands. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/host 87d13f40c52d39585cd9579eafea61b772577c6d3bf23e9e3951af778711e19a @coverage internal/edge 8e47251c54e3c5ee9cb6e8b9b1ad3ae87da528436c6a870cadb07ca3a8ac3146 @coverage internal/activator c2eb11024375a706d70652dee68d446202a264927f819deab6321b652e82c2bb ```` Run the census checks and `go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1`. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/edge internal/activator internal/host/claimwiring.go internal/host/claimwiring_hosting_test.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv git commit -m "region HA 2a: the load balancer health answer" ``` --- ### Task 10: The role doors and `carlos system box role` and `group role` **Files:** - Create: `internal/console/apiboxroles.go`, `internal/console/apiboxroles_test.go` - Modify: `internal/console/authcli.go` (two routes, with host claims on) - Modify: `internal/console/svcroutes_test.go`, `internal/console/claim_invariant_fixtures_test.go`, `internal/console/claim_invariant_probes_nonapp_test.go`, `internal/console/testdata/claim-routes.txt` - Create: `system_roles.go`, `system_roles_test.go` - Modify: `system.go` (the `box` and `group` verbs and their usage) - Modify: `docs/site/cli.md` (a "Box and group roles" section) - Modify: `approved_region_ha_copy_test.go` (wire RHA-C1 to RHA-C16), `region_ha_boundary_test.go` - Modify: `internal/provision/plan9_b8_census_test.go`, `internal/provision/plan14_root_reads_test.go` (the seal), the overlay **Interfaces:** - Consumes: Task 3's `ParseBoxRole`, `ParseGroupRole`, `RoleName`, `SetBoxRole`, `SetGroupRole`; `apiSystemRoller` (ops account owners and `CARLOS_SYSTEM_OPERATORS`). - Produces: `POST /api/cli/system/boxes/{box}/role` and `POST /api/cli/system/groups/{region}/role`, body `{"role": ""}`, answering `{"box", "role"}` and `{"region", "group", "role", "boxes"}`; `func (s *Server) boxRoster(ctx) (hostactors.Directory, release.Signer, hostactors.Observed, error)`; CLI `setBoxRole`, `setGroupRole`, `runSystemBoxVerb`, `runSystemGroupVerb`. **Wait for the copy.** This task writes RHA-C1 to RHA-C16 into console replies, CLI help and `docs/site/cli.md`. Start it only once Paul has approved those rows (Task 1, Step 4), and use his text where it differs from the drafts below. Until then `TestRegionHABuiltCopyIsApproved` fails, by design; do not mark a row approved to get past it. The doors are registered only with host claims on (the roster exists only then), closed to every service credential, and answer 404 unless `CARLOS_REGION_HA` is on too. A box is named by its roster reference or its label. **Census ruling region-ha-2a, condition 2.** `Server.boxRoster` reads the signed roster (`Directory.Read`, kind `file:Read`) through the same source and verification as every other roster read. It is operator-only (it is reached only past `apiSystemRoller`, from the two role doors), it is read-only (the doors write through Task 3's `SetBoxRole` and `SetGroupRole`, which verify and sign as `Directory.Replace` always has), and the roles it carries are read by the placement chooser alone and never feed root's claim authority. The move door does not call it (Task 12 reads roles through the chooser). It is a new accepted-legacy unit (J4 root input), and the seal moves. - [ ] **Step 1: Write the failing tests** `internal/console/apiboxroles_test.go`: ````go package console import ( "bytes" "context" "encoding/json" "errors" "net/http" "net/http/httptest" "reflect" "testing" "github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/release" ) // roleFixture is a console with region HA on and a signed roster: in ie, // an operator-enrolled box ie-old and two boxes the group "routers" // enrolled; uk has no group. func roleFixture(t *testing.T) *consoleFixture { t.Helper() fx := newConsoleFixture(t) fx.srv.cfg.RegionHA = true old := moveRosterBox(ieBox("ie-old")) g1, g2 := moveRosterBox(ieBox("ie-g1")), moveRosterBox(ieBox("ie-g2")) g1.Group, g2.Group = "routers", "routers" putMoveRoster(t, fx, hostactors.Roster{ Boxes: map[string]hostactors.Box{"ie-old": old, "ie-g1": g1, "ie-g2": g2}, Regions: map[string]hostactors.Region{ "ie": {Account: moveTestAccount, AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "arn:group"}, "uk": {Account: moveTestAccount, AWSRegion: "eu-west-2"}, }, }) return fx } // roleCall drives one role door as addr, past cliRoute. func roleCall(t *testing.T, h http.HandlerFunc, addr, pathKey, pathValue, body string) (int, map[string]any) { t.Helper() r := httptest.NewRequest(http.MethodPost, "/api/cli/system/x/y/role", bytes.NewBufferString(body)) r.Header.Set("Content-Type", "application/json") r.SetPathValue(pathKey, pathValue) r = r.WithContext(context.WithValue(r.Context(), cliRouteKey{}, Principal{Kind: PrincipalUser, Address: addr})) w := httptest.NewRecorder() h(w, r) var out map[string]any _ = json.Unmarshal(w.Body.Bytes(), &out) return w.Code, out } func rosterNow(t *testing.T, fx *consoleFixture) hostactors.Roster { t.Helper() _, _, o, err := fx.srv.boxRoster(context.Background()) if err != nil { t.Fatal(err) } return o.Roster } func TestBoxRoleDoor(t *testing.T) { fx := roleFixture(t) h := fx.srv.handleAPISystemBoxRole if st, _ := roleCall(t, h, "someone@keymail.dev", "box", "ie-old", `{"role":"legacy"}`); st != http.StatusNotFound { t.Fatalf("a non-operator = %d, want 404", st) } fx.srv.cfg.RegionHA = false if st, _ := roleCall(t, h, "paul@keymail.dev", "box", "ie-old", `{"role":"legacy"}`); st != http.StatusNotFound { t.Fatalf("region HA off = %d, want 404", st) } fx.srv.cfg.RegionHA = true if st, out := roleCall(t, h, "paul@keymail.dev", "box", "ie-old", `{"role":"draining"}`); st != http.StatusBadRequest || out["error"] != boxRoleRefusal { t.Fatalf("unknown role = %d %v", st, out) } if st, out := roleCall(t, h, "paul@keymail.dev", "box", "nobody", `{"role":"legacy"}`); st != http.StatusNotFound || out["error"] != noEnrolledBox("nobody") { t.Fatalf("unknown box = %d %v", st, out) } // By label, as an operator names it. st, out := roleCall(t, h, "paul@keymail.dev", "box", "ie-old-label", `{"role":"legacy"}`) if st != http.StatusOK || out["box"] != "ie-old" || out["role"] != "legacy" { t.Fatalf("set = %d %v", st, out) } if got := rosterNow(t, fx).Boxes["ie-old"].Role; got != hostactors.RoleLegacy { t.Fatalf("the roster says %q", got) } st, out = roleCall(t, h, "paul@keymail.dev", "box", "ie-old", `{"role":"default"}`) if st != http.StatusOK || out["role"] != "default" || rosterNow(t, fx).Boxes["ie-old"].Role != hostactors.RoleDefault { t.Fatalf("back to default = %d %v", st, out) } } func TestGroupRoleDoor(t *testing.T) { fx := roleFixture(t) h := fx.srv.handleAPISystemGroupRole if st, out := roleCall(t, h, "paul@keymail.dev", "region", "ie", `{"role":"platform"}`); st != http.StatusBadRequest || out["error"] != groupRoleRefusal { t.Fatalf("platform group = %d %v", st, out) } if st, out := roleCall(t, h, "paul@keymail.dev", "region", "uk", `{"role":"legacy"}`); st != http.StatusNotFound || out["error"] != noGroupIn("uk") { t.Fatalf("no group = %d %v", st, out) } st, out := roleCall(t, h, "paul@keymail.dev", "region", "ie", `{"role":"legacy"}`) if st != http.StatusOK || out["group"] != "routers" || out["role"] != "legacy" || !reflect.DeepEqual(out["boxes"], []any{"ie-g1", "ie-g2"}) { t.Fatalf("set = %d %v", st, out) } r := rosterNow(t, fx) if r.Regions["ie"].Role != hostactors.RoleLegacy || r.Boxes["ie-g1"].Role != hostactors.RoleLegacy || r.Boxes["ie-g2"].Role != hostactors.RoleLegacy || r.Boxes["ie-old"].Role != hostactors.RoleDefault { t.Fatalf("roster after the group role: %+v", r.Boxes) } } func TestRoleWriteErrors(t *testing.T) { for _, c := range []struct { err error code int msg string }{ {release.ErrCASConflict, http.StatusConflict, boxRosterChanged}, {hostactors.ErrRoster, http.StatusConflict, boxRosterChanged}, {errors.New("SlowDown"), http.StatusServiceUnavailable, boxRoleWriteFailure}, } { w := httptest.NewRecorder() roleWriteError(w, "test", c.err) var out map[string]string _ = json.Unmarshal(w.Body.Bytes(), &out) if w.Code != c.code || out["error"] != c.msg { t.Fatalf("%v = %d %v", c.err, w.Code, out) } } } // The doors exist only on a host-claim console, operator-only and 404 // with region HA off. func TestRoleDoorsAreRegisteredWithHostClaims(t *testing.T) { st := openGatedStore(t) rel := release.NewDirStore(t.TempDir()) path := "/api/cli/system/boxes/ie-old/role" off, err := NewServer(Config{Origin: "https://console.example", Keymail: "https://keymail.example", Releases: rel}, st) if err != nil { t.Fatal(err) } ghostStatus, ghostBody := serveHostClaim(t, off, "/unknown-role-route", "") if status, body := serveHostClaim(t, off, path, ""); status != ghostStatus || !bytes.Equal(body, ghostBody) { t.Fatalf("claims off = %d %q, want the unknown route's %d %q", status, body, ghostStatus, ghostBody) } token := mintHostClaimToken(t, st, "paul@keymail.dev") on, err := NewServer(Config{Origin: "https://console.example", Keymail: "https://keymail.example", Releases: rel, HostClaim: true, SystemOperators: []string{"paul@keymail.dev"}}, st) if err != nil { t.Fatal(err) } if status, _ := serveHostClaim(t, on, path, ""); status != http.StatusUnauthorized { t.Fatalf("unsigned = %d", status) } if status, _ := serveHostClaim(t, on, path, token); status != http.StatusNotFound { t.Fatalf("region HA off = %d, want 404", status) } } ```` `system_roles_test.go`: ````go package main import ( "context" "errors" "net/http" "reflect" "testing" ) func TestSystemVerbKnowsBoxAndGroup(t *testing.T) { for _, verb := range []string{"box", "group"} { got, rest, _ := systemVerb([]string{verb, "role", "x", "legacy"}) if got != verb || len(rest) != 3 || rest[0] != "role" { t.Fatalf("systemVerb(%s) = %q %q", verb, got, rest) } } } // fakeRolePost answers one role call with status and body, recording what // was asked. type fakeRolePost struct { status int body string path string in any } func (f *fakeRolePost) post(_ context.Context, path string, in any) (int, []byte, error) { f.path, f.in = path, in return f.status, []byte(f.body), nil } func TestSetBoxRole(t *testing.T) { f := &fakeRolePost{status: http.StatusOK, body: `{"box":"i-0abc","role":"legacy"}`} line, err := setBoxRole(context.Background(), f.post, "old box", "legacy") if err != nil || line != "i-0abc now has the legacy role" { t.Fatalf("line %q, %v", line, err) } if f.path != "/api/cli/system/boxes/old%20box/role" || !reflect.DeepEqual(f.in, map[string]string{"role": "legacy"}) { t.Fatalf("asked %s %v", f.path, f.in) } f = &fakeRolePost{status: http.StatusBadRequest, body: `{"error":"role must be default, legacy or platform"}`} if _, err := setBoxRole(context.Background(), f.post, "x", "boss"); err == nil || err.Error() != "role must be default, legacy or platform" { t.Fatalf("refusal: %v", err) } failing := func(context.Context, string, any) (int, []byte, error) { return 0, nil, errors.New("offline") } if _, err := setBoxRole(context.Background(), failing, "x", "legacy"); err == nil { t.Fatal("a failed call printed a line") } } func TestSetGroupRole(t *testing.T) { f := &fakeRolePost{status: http.StatusOK, body: `{"region":"ie","group":"routers","role":"legacy","boxes":["i-1","i-2"]}`} lines, err := setGroupRole(context.Background(), f.post, "ie", "legacy") if err != nil { t.Fatal(err) } want := []string{"the group in ie now has the legacy role", "i-1 now has the legacy role", "i-2 now has the legacy role"} if !reflect.DeepEqual(lines, want) || f.path != "/api/cli/system/groups/ie/role" { t.Fatalf("lines %q path %s", lines, f.path) } f = &fakeRolePost{status: http.StatusOK, body: `{"region":"ie","group":"routers","role":"default","boxes":[]}`} if lines, _ := setGroupRole(context.Background(), f.post, "ie", "default"); len(lines) != 1 { t.Fatalf("an empty group: %q", lines) } } ```` The route censuses learn the two doors: ````diff --- a/internal/console/svcroutes_test.go +++ b/internal/console/svcroutes_test.go @@ -286,6 +286,10 @@ hostClaimDoors := []string{ "POST /api/cli/ops/hosts/{host}/resolve/inspect", "POST /api/cli/ops/hosts/{host}/resolve", + // Box and group roles (region HA spec 2 §7): operator-only, and 404 + // unless CARLOS_REGION_HA is on as well. + "POST /api/cli/system/boxes/{box}/role", + "POST /api/cli/system/groups/{region}/role", } movesDoors := []string{ "POST /api/cli/apps/{account}/{app}/instances/{host}/move/resolve", ```` ````diff --- a/internal/console/claim_invariant_fixtures_test.go +++ b/internal/console/claim_invariant_fixtures_test.go @@ -1129,6 +1129,8 @@ }}, "route POST /api/cli/ops/hosts/{host}/resolve/inspect": claimRead("operator host-loss inspection: reads the claim, the record and the roster and signs a challenge; writes nothing"), "route POST /api/cli/system/antivirus": claimNonApp("the deployment scanning switch (system/antivirus.json)"), + "route POST /api/cli/system/boxes/{box}/role": claimNonApp("operator box role (region HA spec 2 §7): one signed roster write to control/host-actors.json; names no app"), + "route POST /api/cli/system/groups/{region}/role": claimNonApp("operator group role (region HA spec 2 §7): one signed roster write to control/host-actors.json, the group and the boxes it enrolled; names no app"), "route POST /api/cli/system/layout": claimNonApp("the deployment layout intent (control/layout.json)"), "route POST /api/cli/system/activation/cancel": claimNonApp("cancels an unpublished system activation"), "route POST /api/cli/system/pointer-digests": claimNonApp("operator switch: writes or removes each account's root-signed pointer digest marker (releases//pointer-digests), classified exactly by the mutation boundary"), ```` ````diff --- a/internal/console/claim_invariant_probes_nonapp_test.go +++ b/internal/console/claim_invariant_probes_nonapp_test.go @@ -674,6 +674,54 @@ return "" }, }, + // Box and group roles (region HA spec 2 §7): one signed roster write, + // only with region HA on. + "route POST /api/cli/system/boxes/{box}/role": { + setup: func(t *testing.T, cx *claimCase) { cx.srv.cfg.RegionHA = true }, + do: claimAPIReply(http.MethodPost, "/api/cli/system/boxes/box-ie-default/role", `{"role":"legacy"}`), + body: []string{`"role":"legacy"`}, + effect: func(t *testing.T, cx *claimCase, r claimReply) string { + o, err := cx.claimFixtureDirectory().Read(context.Background()) + if err != nil || o.Roster.Boxes["box-ie-default"].Role != hostactors.RoleLegacy { + return "box-ie-default does not have the legacy role in the signed roster" + } + return "" + }, + }, + "route POST /api/cli/system/groups/{region}/role": { + setup: func(t *testing.T, cx *claimCase) { + cx.srv.cfg.RegionHA = true + ctx := context.Background() + o, err := cx.claimFixtureDirectory().Read(ctx) + if err != nil { + t.Fatal(err) + } + next := hostactors.Clone(o.Roster) + g := next.Regions["ie"] + g.GroupName, g.GroupARN = "routers", "arn:aws:autoscaling:eu-west-1:123456789012:autoScalingGroup:routers" + next.Regions["ie"] = g + b := next.Boxes["box-ie-default"] + b.Group = "routers" + next.Boxes["box-ie-default"] = b + next.Revision++ + raw, err := release.SignRootEnvelope(release.RootSigner(cx.priv), next) + if err != nil { + t.Fatal(err) + } + if _, err = cx.rel.Put(ctx, hostactors.Key, raw, o.ETag); err != nil { + t.Fatal(err) + } + }, + do: claimAPIReply(http.MethodPost, "/api/cli/system/groups/ie/role", `{"role":"legacy"}`), + body: []string{`"boxes":["box-ie-default"]`}, + effect: func(t *testing.T, cx *claimCase, r claimReply) string { + o, err := cx.claimFixtureDirectory().Read(context.Background()) + if err != nil || o.Roster.Regions["ie"].Role != hostactors.RoleLegacy || o.Roster.Boxes["box-ie-default"].Role != hostactors.RoleLegacy { + return "the group in ie and its box do not have the legacy role in the signed roster" + } + return "" + }, + }, "route POST /api/cli/system/seeds-floor": { do: claimAPIReply(http.MethodPost, "/api/cli/system/seeds-floor", `{"min_release":"v0.27.0"}`), body: []string{`"min_release":"v0.27.0"`}, ```` ````diff --- a/internal/console/testdata/claim-routes.txt +++ b/internal/console/testdata/claim-routes.txt @@ -237,6 +237,8 @@ route POST /api/cli/services/{account}/{name}/rotate route POST /api/cli/system/activation/cancel route POST /api/cli/system/antivirus +route POST /api/cli/system/boxes/{box}/role +route POST /api/cli/system/groups/{region}/role route POST /api/cli/system/layout route POST /api/cli/system/legacy-routes route POST /api/cli/system/pointer-digests ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/console/ -run 'BoxRoleDoor|GroupRoleDoor|RoleWriteErrors|RoleDoorsAreRegistered' -count=1` Expected: FAIL to build (`undefined: handleAPISystemBoxRole`, `undefined: boxRoster`). `go test . -run 'SystemVerbKnowsBoxAndGroup|SetBoxRole|SetGroupRole'` fails to build on `undefined: setBoxRole`. - [ ] **Step 3: Write the doors and the CLI** `internal/console/apiboxroles.go`: ````go package console import ( "context" "errors" "fmt" "log" "net/http" "github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/release" ) // Box and group roles (region HA spec 2 §7, A11-1): `carlos system box role // ` and `carlos system group role `. A legacy or // platform box takes no new instance, relocation or move; it keeps the // ones it has (H25). A group's role is every box it enrolled, now and // later. The roles live in the signed roster, so the console, the only // holder of the root key, is the only writer. // // Operator-only (apiSystemRoller, the update doors' authority), registered // only with host claims on, and answering 404 unless CARLOS_REGION_HA is on // too: with region HA off nothing reads a role. const ( boxRoleRefusal = "role must be default, legacy or platform" groupRoleRefusal = "a group's role must be default or legacy" boxRosterUnread = "could not read the box roster. Try again." boxRosterChanged = "the box roster changed while the role was being saved, and nothing was changed. Try again." boxRoleWriteFailure = "could not save the role. Try again." ) func noEnrolledBox(name string) string { return fmt.Sprintf("no enrolled box is named %s", name) } func noGroupIn(region string) string { return fmt.Sprintf("%s has no group", region) } func retiredBoxRole(box string) string { return fmt.Sprintf("%s is retired, so it takes no role", box) } // boxRoster reads the signed roster with the root signer that writes it, // for the role doors alone, which have already let only an operator in // (roleOperator). It is a read, of the one roster every reader reads // (control/host-actors.json through hostactors.Directory). Census ruling // region-ha-2a's conditions hold it to that: operator-only, read-only, the // same roster, and no role it reads feeds root's claim authority (roles // are read by the placement chooser alone). func (s *Server) boxRoster(ctx context.Context) (hostactors.Directory, release.Signer, hostactors.Observed, error) { if s.cfg.Releases == nil || s.cfg.Bucket == "" || len(s.cfg.Key) == 0 { return hostactors.Directory{}, release.Signer{}, hostactors.Observed{}, errors.New("console: no deployment bucket or root key") } signer := release.RootSigner(s.cfg.Key) dir := hostactors.Directory{Store: s.cfg.Releases, PublicKey: signer.RootPublicKey(), Bucket: s.cfg.Bucket} o, err := dir.Read(ctx) return dir, signer, o, err } // rosterRef finds a box by reference, or by the one label no retired box // shares (hostactors.Directory.Resolve's rule, over a roster in hand). func rosterRef(r hostactors.Roster, name string) (string, bool) { if _, ok := r.Boxes[name]; ok { return name, true } found := "" for ref, b := range r.Boxes { if b.Label == name && b.State != "retired" { if found != "" { return "", false } found = ref } } return found, found != "" } // roleOperator is the role doors' gate: an operator, with region HA on. func (s *Server) roleOperator(w http.ResponseWriter, r *http.Request) (string, bool) { addr, _, ok := s.apiSystemRoller(w, r) if !ok { return "", false } if !s.cfg.RegionHA { http.NotFound(w, r) return "", false } return addr, true } // roleWriteError answers a roster write that did not land. func roleWriteError(w http.ResponseWriter, what string, err error) { if errors.Is(err, release.ErrCASConflict) || errors.Is(err, hostactors.ErrRoster) { jsonError(w, http.StatusConflict, boxRosterChanged) return } log.Printf("console: %s: %v", what, err) jsonError(w, http.StatusServiceUnavailable, boxRoleWriteFailure) } func (s *Server) handleAPISystemBoxRole(w http.ResponseWriter, r *http.Request) { by, ok := s.roleOperator(w, r) if !ok { return } var req struct { Role string `json:"role"` } if !readJSON(w, r, &req) { return } role, err := hostactors.ParseBoxRole(req.Role) if err != nil { jsonError(w, http.StatusBadRequest, boxRoleRefusal) return } dir, signer, o, err := s.boxRoster(r.Context()) if err != nil { log.Printf("console: box role: roster: %v", err) jsonError(w, http.StatusServiceUnavailable, boxRosterUnread) return } name := r.PathValue("box") ref, ok := rosterRef(o.Roster, name) if !ok { jsonError(w, http.StatusNotFound, noEnrolledBox(name)) return } if o.Roster.Boxes[ref].State == "retired" { jsonError(w, http.StatusConflict, retiredBoxRole(ref)) return } next, err := dir.SetBoxRole(r.Context(), signer, o, ref, role) if err != nil { roleWriteError(w, "box role "+ref, err) return } log.Printf("console: box %s has the %s role (by %s, roster revision %d)", ref, hostactors.RoleName(role), by, next.Roster.Revision) writeJSON(w, http.StatusOK, map[string]string{"box": ref, "role": hostactors.RoleName(role)}) } func (s *Server) handleAPISystemGroupRole(w http.ResponseWriter, r *http.Request) { by, ok := s.roleOperator(w, r) if !ok { return } var req struct { Role string `json:"role"` } if !readJSON(w, r, &req) { return } role, err := hostactors.ParseGroupRole(req.Role) if err != nil { jsonError(w, http.StatusBadRequest, groupRoleRefusal) return } dir, signer, o, err := s.boxRoster(r.Context()) if err != nil { log.Printf("console: group role: roster: %v", err) jsonError(w, http.StatusServiceUnavailable, boxRosterUnread) return } region := r.PathValue("region") g, ok := o.Roster.Regions[region] if !ok || g.GroupName == "" { jsonError(w, http.StatusNotFound, noGroupIn(region)) return } next, boxes, err := dir.SetGroupRole(r.Context(), signer, o, region, role) if err != nil { roleWriteError(w, "group role "+region, err) return } log.Printf("console: the group in %s (%s) and its boxes %v have the %s role (by %s, roster revision %d)", region, g.GroupName, boxes, hostactors.RoleName(role), by, next.Roster.Revision) if boxes == nil { boxes = []string{} } writeJSON(w, http.StatusOK, map[string]any{"region": region, "group": g.GroupName, "role": hostactors.RoleName(role), "boxes": boxes}) } ```` ````diff --- a/internal/console/authcli.go +++ b/internal/console/authcli.go @@ -512,6 +512,10 @@ // ActorDirectory both routes stop at the same inert refusal. s.cliRoute("POST /api/cli/ops/hosts/{host}/resolve/inspect", svcrole.Set{}, s.handleAPIHostResolveInspect) s.cliRoute("POST /api/cli/ops/hosts/{host}/resolve", svcrole.Set{}, s.handleAPIHostResolve) + // Box and group roles (region HA spec 2 §7, apiboxroles.go): + // operator-only, and 404 unless CARLOS_REGION_HA is on too. + s.cliRoute("POST /api/cli/system/boxes/{box}/role", svcrole.Set{}, s.handleAPISystemBoxRole) + s.cliRoute("POST /api/cli/system/groups/{region}/role", svcrole.Set{}, s.handleAPISystemGroupRole) } // carlos status --layout (plan 4, task 5): which layout each instance is // on, box by box. Same posture as the door above — an empty role set, the ```` `system_roles.go`: ````go package main import ( "context" "encoding/json" "fmt" "log" "net/http" "net/url" "os" ) // carlos system box role and carlos system group role // (region HA spec 2 §7, A11-1). A box's role says whether // placement may choose it: legacy and platform boxes take no new // instances and keep the ones they have. A group's role is every box it // has enrolled and every box it enrolls later. Operators only, and only on // a console with region HA on. // rolePost is the one console call a role change makes. type rolePost func(ctx context.Context, path string, in any) (int, []byte, error) // boxRoleLine is what the CLI says once a box has its role. func boxRoleLine(box, role string) string { return fmt.Sprintf("%s now has the %s role", box, role) } // groupRoleLine is what it says once a group has its role. func groupRoleLine(region, role string) string { return fmt.Sprintf("the group in %s now has the %s role", region, role) } // setBoxRole asks the console to give box the role, and returns the line // to print. func setBoxRole(ctx context.Context, post rolePost, box, role string) (string, error) { status, body, err := post(ctx, "/api/cli/system/boxes/"+url.PathEscape(box)+"/role", map[string]string{"role": role}) if err != nil { return "", err } if status != http.StatusOK { return "", apiErrorFromBody(status, body) } var out struct { Box string `json:"box"` Role string `json:"role"` } if err := json.Unmarshal(body, &out); err != nil { return "", fmt.Errorf("unreadable response: %w", err) } return boxRoleLine(out.Box, out.Role), nil } // setGroupRole asks the console to give region's group the role, and // returns the lines to print: the group's, then one per box it set. func setGroupRole(ctx context.Context, post rolePost, region, role string) ([]string, error) { status, body, err := post(ctx, "/api/cli/system/groups/"+url.PathEscape(region)+"/role", map[string]string{"role": role}) if err != nil { return nil, err } if status != http.StatusOK { return nil, apiErrorFromBody(status, body) } var out struct { Region string `json:"region"` Role string `json:"role"` Boxes []string `json:"boxes"` } if err := json.Unmarshal(body, &out); err != nil { return nil, fmt.Errorf("unreadable response: %w", err) } lines := []string{groupRoleLine(out.Region, out.Role)} for _, b := range out.Boxes { lines = append(lines, boxRoleLine(b, out.Role)) } return lines, nil } // roleArgs reads `role ` with an optional --console flag // after them, and exits with the usage on anything else. func roleArgs(rest []string) (name, role, console string) { if len(rest) < 3 || rest[0] != "role" { systemUsage() os.Exit(2) } name, role = rest[1], rest[2] tail := rest[3:] switch { case len(tail) == 0: case len(tail) == 2 && (tail[0] == "--console" || tail[0] == "-console"): console = tail[1] default: systemUsage() os.Exit(2) } return name, role, console } func runSystemBoxVerb(rest []string) { box, role, consoleFlag := roleArgs(rest) sess := requireDeployAPISession(consoleFlag) printUsing(stdout, sess) line, err := setBoxRole(context.Background(), sess.postJSON, box, role) if err != nil { log.Fatalf("carlos system box role: %v", err) } fmt.Fprintln(stdout, line) } func runSystemGroupVerb(rest []string) { region, role, consoleFlag := roleArgs(rest) sess := requireDeployAPISession(consoleFlag) printUsing(stdout, sess) lines, err := setGroupRole(context.Background(), sess.postJSON, region, role) if err != nil { log.Fatalf("carlos system group role: %v", err) } for _, l := range lines { fmt.Fprintln(stdout, l) } } ```` ````diff --- a/system.go +++ b/system.go @@ -49,7 +49,7 @@ // untouched in rest for runSystemAntivirusVerb to read. // layout is a family too: migrate|revert|status stay in rest for // runSystemLayout. - case "guard-arm", "guard-exec", "journal-migrate", "floor-allows-prev", "update", "status", "rollback", "cancel-activation", "apply", "antivirus", "layout", "geoip", "seeds-floor", "legacy-routes", "help", "-h", "--help": + case "guard-arm", "guard-exec", "journal-migrate", "floor-allows-prev", "update", "status", "rollback", "cancel-activation", "apply", "antivirus", "layout", "geoip", "seeds-floor", "legacy-routes", "box", "group", "help", "-h", "--help": return args[0], args[1:], component } return "", args, component @@ -88,6 +88,10 @@ runSystemSeedsFloorVerb(rest) case "legacy-routes": runSystemLegacyRoutesVerb(rest) + case "box": + runSystemBoxVerb(rest) + case "group": + runSystemGroupVerb(rest) case "help", "-h", "--help": systemUsage() default: @@ -122,6 +126,8 @@ carlos system geoip [flags] carlos system seeds-floor publish --min-release carlos system legacy-routes assign --name --to / + carlos system box role + carlos system group role The platform's own binary (ops account owners and CARLOS_SYSTEM_OPERATORS): @@ -138,6 +144,12 @@ the first release whose boxes can place a seed, such as v0.27.0 legacy-routes assign --name --to / `+systemLegacyRoutesVerb+` + box role + whether new instances may be placed on a box. A legacy or + platform box keeps the instances it has and takes no new ones + group role + the same for every box a region's group has enrolled, and for + every box it enrolls later Deployment-wide settings (CARLOS_STORE_OPERATORS): ```` ````diff --- a/docs/site/cli.md +++ b/docs/site/cli.md @@ -2200,6 +2200,17 @@ carlos system legacy-routes assign --name shop --to bac/shop ``` +#### Box and group roles + +carlos system box role says whether new instances may be placed on a box. A box with the default role can take them. A legacy or platform box keeps the instances it has and takes no new ones: no new instance, relocation or move lands on it. carlos system group role sets the role of every box a region's group has enrolled, and of every box it enrolls later. Both work only on a console with region HA on. + +```sh +carlos system box role i-0123456789abcdef0 legacy +carlos system group role ie default +``` + +Ops account owners and `CARLOS_SYSTEM_OPERATORS`. + ### carlos move Moves one instance to another region. The instance keeps serving while its ```` - [ ] **Step 4: Run the tests** ```bash nice -n 19 go test ./internal/console/ -run 'BoxRoleDoor|GroupRoleDoor|RoleWriteErrors|RoleDoorsAreRegistered|EveryCLIDoor|ClaimConsoleRouteCensus|NoRoleReaches' -count=1 nice -n 19 go test ./internal/console/ -run 'TestClaimConsoleMutationInvariant$/probes/route_POST_/api/cli/system/' -count=1 -v | grep -E -- '--- (PASS|FAIL).*role' nice -n 19 go test . -run 'SystemVerb|SetBoxRole|SetGroupRole' -count=1 nice -n 19 go test ./internal/docsite/ -count=1 ``` Expected: PASS, and both role probes PASS. The second command runs only the probes, so its parent test reports app-scoped writers "reached by no door's open run": that is the filter, not the change. The whole `TestClaimConsoleMutationInvariant` takes longer than the default ten-minute test timeout; run it once with `-timeout 90m` before handing off (it is part of CI's `make test`). - [ ] **Step 5: Copy gate, boundary list and census** ````diff --- a/approved_region_ha_copy_test.go +++ b/approved_region_ha_copy_test.go @@ -36,7 +36,7 @@ // regionHACopyBuilt is every id whose text the code or docs carry. The // task that writes a row's text adds its id here in the same commit, once // Paul has approved the row. -var regionHACopyBuilt = []string{} +var regionHACopyBuilt = []string{"RHA-C1", "RHA-C2", "RHA-C3", "RHA-C4", "RHA-C5", "RHA-C6", "RHA-C7", "RHA-C8", "RHA-C9", "RHA-C10", "RHA-C11", "RHA-C12", "RHA-C13", "RHA-C14", "RHA-C15", "RHA-C16"} func mustParseRegionHARecord(t *testing.T) []copyRow { t.Helper() ```` Run: `nice -n 19 go test . -run 'RegionHACopy|RegionHABuilt' -count=1` Expected: PASS once RHA-C1 to RHA-C16 are approved. While any is pending, `TestRegionHABuiltCopyIsApproved` fails naming it, and the task is not done. ````diff --- a/region_ha_boundary_test.go +++ b/region_ha_boundary_test.go @@ -25,6 +25,8 @@ "internal/host/boxaddress.go", "internal/edgesfile/private.go", "internal/edge/lbhealth.go", + "internal/console/apiboxroles.go", + "system_roles.go", } // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so ```` `registerCLIRoutes` is in the record-write census (a key site). Its digest changes: ````diff --- a/internal/provision/plan9_b8_census_test.go +++ b/internal/provision/plan9_b8_census_test.go @@ -680,7 +680,7 @@ // Registers instance and sidecar HTTP paths; record handlers use the reviewed provision writers. // Re-reviewed at the merge of main into carlos-run (2026-10-13): main's routes (the retry door among them) with the five run doors registered beside them; record writes stay in the reviewed handlers. // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's routes and the run doors with the four import doors, the seeds-floor door and the legacy-routes door registered beside them; record writes stay in the reviewed handlers. - {"internal/console/authcli.go", "Server.registerCLIRoutes", "key-site", "lit:instances,lit:sidecars", "bb32a85461c3253b2b20f4191ae5fb1f2aa26e5e6c038bc338b6d16a14e7301a"}, + {"internal/console/authcli.go", "Server.registerCLIRoutes", "key-site", "lit:instances,lit:sidecars", "b55987735c833b952a7ef835302dba728c803379ecb474c3d465bb268cf4c781"}, {"internal/console/browserrestart.go", "Server.handleBrowserRestart", "key-site", "lit:instances", "955d71813504f2cfec1e1f64817d5aa5a921aa38e1044c0e903d2cfbe4d073f4"}, // Builds the instance page path and picks the confirmation line (a declined box's own line); the only write is the reviewed retry door (requestHostRetry), no record write here. {"internal/console/browserretry.go", "Server.handleInstanceRetry", "key-site", "lit:instances", "4d85d392f3660373f3c282c6391f58d8e25b6df738257ccc067b816e24dfe7ff"}, ```` The `Server.boxRoster` rows are census ruling region-ha-2a's (platform-1e). If the ruling's own basis text differs from the row's, use it, then recompute the fingerprint with `legacyrow.py` and the seal. Add this comment line after the overlay's last comment line: ````text # Task 10: the operator's box and group role doors (internal/console apiboxroles.go, registered with host claims on in registerCLIRoutes) and the CLI (system_roles.go). Census ruling region-ha-2a (platform-1e, approved with conditions) covers one accepted-legacy unit, Server.boxRoster: the role doors' read of the signed roster (Directory.Read), operator-only (it is reached only past apiSystemRoller), read-only, the same roster every reader reads, and no role it reads feeds root's claim authority. Its basis is main's for a console root input (J4). The seal moves. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/console bffe4b4c505d98b077d2e5e7906c32615ed62efb369da70eb18b4cc7084a1882 @coverage . 011b3548e5615fa1bb046e5a6d0b0e4345206ac7774c571a8320cfd124ba9af3 @legacy b02b534b6617f57381ba12e9d9a2bf1bd63f737dc68005de926ff18b7db93dd3 internal/console/apiboxroles.go Server.boxRoster legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@484 9be9925de14dbbd7d86123bf9faf1535c7b116575cadcb8ba983a665af0513f4 internal/console/apiboxroles.go Server.boxRoster legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@484 9be9925de14dbbd7d86123bf9faf1535c7b116575cadcb8ba983a665af0513f4 ```` ````diff --- a/internal/provision/plan14_root_reads_test.go +++ b/internal/provision/plan14_root_reads_test.go @@ -769,7 +769,7 @@ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv"), &inv, true); err != nil { t.Fatal(err) } - seal = "fc38db001ebd8825709a8ca9c41cc0e8f165c1b1967e5967347dc9bd3556afec" + seal = "a57250c350c20e4116e5c87b559cfda71f3e0c65d59a2858d83c5f78bea4e9b5" errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) errs = append(errs, checkRootReadDeclarations(got, inv)...) if len(errs) > 0 { ```` Run the census checks and `go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1`. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/console/apiboxroles.go internal/console/apiboxroles_test.go internal/console/authcli.go internal/console/svcroutes_test.go internal/console/claim_invariant_fixtures_test.go internal/console/claim_invariant_probes_nonapp_test.go internal/console/testdata/claim-routes.txt system.go system_roles.go system_roles_test.go docs/site/cli.md approved_region_ha_copy_test.go region_ha_boundary_test.go internal/provision git commit -m "region HA 2a: box and group role doors and the carlos system verbs" ``` --- ### Task 11: The create doors choose first, and refuse when no box has room (H27) **Files:** - Create: `internal/hostops/chosen.go` (`WithChosenExecutor`, `ChosenExecutor`) - Modify: `internal/hostops/backfill.go` (`ProductionAuthority.SelectExecutor` honours the door's choice) - Modify: `internal/hostops/executor_test.go` - Create: `internal/console/nocapacity.go`, `internal/console/nocapacity_test.go` - Modify: `internal/console/claim_create.go` (`beginClaimCreate` calls `chooseExecutorFirst` before `RegisterStart`) - Modify: `internal/console/claim_response.go` (`claimResponseRefusal` answers `noCapacityError` with a 503) - Modify: `internal/console/apiinstances.go` (`handleAPIInstanceCreate`: the app-derived region check uses the chooser, the armed-placement gate asks `roomIn` first, and `chooseCreateFirst` runs before the environment binding) - Modify: `internal/console/server.go` (`Server.noCapacity`) - Modify: `approved_region_ha_copy_test.go` (wire RHA-C17 to RHA-C19), `region_ha_boundary_test.go`, `internal/provision/plan9_b8_census_test.go`, `internal/provision/plan14_root_reads_test.go` (the seal), the overlay **Interfaces:** - Consumes: Task 6's `ExecutorChooser.PlaceableBoxes` and `Choose` (through `ProductionAuthority.SelectExecutor`); `hostops.ErrNoEligibleBox`; `alerting.Page`; `Server.homeRegionOut`. - Produces: `func hostops.WithChosenExecutor(ctx, host string, sel hostclaim.ExecutorSelection) context.Context`; `func hostops.ChosenExecutor(ctx, host string) (hostclaim.ExecutorSelection, bool)`; `type noCapacityError struct{ Region string }` (`errors.Is(err, hostops.ErrNoEligibleBox)`); `type noCapacityAlerts` (`due(region string, now time.Time) bool`); `func (s *Server) noRoom(ctx, region string) error`; `func (s *Server) placementChooser(pool string) *hostops.ExecutorChooser` (Tasks 12 and 13 use it); `func (s *Server) roomIn(ctx, region, pool string) error` (Task 12 uses it); `func (s *Server) chooseExecutorFirst(ctx, host string, before hostclaim.Observed, build func(time.Time, string) (hostops.Record, error)) (context.Context, error)`; `func (s *Server) chooseCreateFirst(ctx, host string, build ...) (context.Context, error)` (the returned context carries the choice and the claim observation it was decided on); `var errChoiceStale` (`errors.Is(err, hostclaim.ErrConflict)`); `var pageNoCapacity = alerting.Page` (the test seam). **Wait for the copy.** RHA-C17 to RHA-C19 are the refusal and the alert. Start once Paul has approved them, and use his text. H27 (spec rev 15), as Astra's findings ask (round 1: 1, 2, 3 and 6; round 2: 1 and 2): - **Choose before the first write.** Every create through the claim starts in `beginClaimCreate`. With region HA on, `chooseExecutorFirst` builds the record the create would write, asks `SelectExecutor` for its executor, and returns before `RegisterStart` when there is none. Otherwise it puts the choice in the context, and `SelectExecutor` returns that choice to `StartCreate` (claim Plan 5's checks still run on it), so the create names exactly the box the door checked. The instance create door has a write before `beginClaimCreate`: `bindEnvironmentCreate` can clear a seed's mark on the environment's store (`takeSeedStore`). So that door calls `chooseCreateFirst` before the binding; it reads the host's claim and chooses the same way, and `beginClaimCreate` finds the choice in the context and keeps it. The domain attach and standalone sidecar doors write nothing before `beginClaimCreate`, and the seed doors refuse on a claim console (`seedRegionReady`). - **The observation travels with the choice (ruling item 5).** `chooseCreateFirst` returns the choice together with the claim observation it decided on (the claim's ETag, or its absence). `beginClaimCreate` hands its own claim read to `chooseExecutorFirst`, which keeps the carried choice only when that read is the same observation. If the claim changed in between, the create is **refused** with `errChoiceStale` (`hostclaim.ErrConflict`: main's retryable busy answer) before anything is registered or written; the old choice is never used, and nothing is chosen again. Refusing is the smaller of the two options the ruling allows: it adds no copy and no second choice. - **No lease, no write, chosen once (ruling rev 2, item 3).** The choice is `SelectExecutor`'s read of the roster, the fences and the edges files. It takes no lease and writes nothing; a choice the context carries for the host is never chosen again; a refusal leaves no registration, no claim, no store change and nothing for a recovery pass. - **Every choosing door.** One error, `noCapacityError`, answered in one place, `claimResponseRefusal`: instance create, domain attach's first route (the API's `handleAPIDomainAttach` and the browser's `handleAppDomainAttach`, both through `attachClaimFirstRoute`, which is unchanged), and standalone sidecar create. The move door's destination is Task 12. A sidecar attached to an instance chooses nothing (it lives on its instance's box) and is unchanged. - **The earlier region checks.** `handleAPIInstanceCreate` judges an app-derived region (`regionCandidates`) by the chooser's eligible set instead, and asks `roomIn` before the armed-placement gate for an explicit region. `roomIn` only refuses: a region the roster places boxes in, with no placeable box (no edges file, no public address, stale, full or excluded), is the no-room refusal rather than a 400; a region the roster does not know is left to the gate. - **The alert limit.** At most one alert per region an hour, remembered in the console process; a restart may send one more, which H27 accepts. With region HA off nothing here runs: `roomIn`, `chooseCreateFirst` and `chooseExecutorFirst` return at once, and every create is main's. **Census (platform-1e):** `Server.beginClaimCreate` is an accepted-legacy unit (J4 root input). It gains one call before `RegisterStart`; its one read site is unchanged. The overlay re-freezes it with the ruling's tag. `Server.chooseCreateFirst` is new: its one root read is the host's claim, the read `beginClaimCreate` makes, made before the binding and feeding only the placement choice. It returns its choice with that observation, and `beginClaimCreate` refuses when its own read differs (ruling item 5). The overlay asks for it as accepted-legacy (J4 root input), as approved. The seal moves. `Server.attachClaimFirstRoute` does not change. The record-write census digest of `handleAPIInstanceCreate` changes; no write site is added or changed. - [ ] **Step 1: Write the failing tests** `internal/console/nocapacity_test.go`: ````go package console import ( "context" "encoding/json" "errors" "fmt" "net/http" "strings" "testing" "time" "github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostops" "github.com/carlosframework/platform/internal/objstore" "github.com/carlosframework/platform/internal/provision" "github.com/carlosframework/platform/internal/status" ) func TestNoCapacityAlertsRepeatAfterAnHour(t *testing.T) { var a noCapacityAlerts t0 := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) if !a.due("uk", t0) || a.due("uk", t0.Add(59*time.Minute)) || !a.due("ie", t0) || !a.due("uk", t0.Add(time.Hour)) { t.Fatal("once per region an hour") } } func TestClaimRefusalAnswersNoRoom(t *testing.T) { ref := claimResponseRefusal(fmt.Errorf("begin: %w", &noCapacityError{Region: "uk"})) if ref.Status != http.StatusServiceUnavailable || ref.Msg != noCapacityRefusal("uk") { t.Fatalf("refusal %+v", ref) } var err error = &noCapacityError{Region: "uk"} if !errors.Is(err, hostops.ErrNoEligibleBox) { t.Fatal("a no-room refusal is not ErrNoEligibleBox") } } // pagesTo replaces the alert sender for the test and returns what it sent. func pagesTo(t *testing.T) chan string { t.Helper() pages := make(chan string, 16) old := pageNoCapacity pageNoCapacity = func(_ context.Context, level, subject, _ string) error { pages <- level + " " + subject return nil } t.Cleanup(func() { pageNoCapacity = old }) return pages } func expectPages(t *testing.T, pages chan string, want ...string) { t.Helper() for _, w := range want { select { case got := <-pages: if got != w { t.Fatalf("page %q, want %q", got, w) } case <-time.After(5 * time.Second): t.Fatalf("no page %q", w) } } select { case got := <-pages: t.Fatalf("an extra page: %q", got) case <-time.After(100 * time.Millisecond): } } func noRoomSubject(region string) string { subject, _ := noCapacityAlert(region) return "warning " + subject } // haClaimCase is a host-claim console with region HA on: its create doors // choose through the region HA chooser over the edges files the test // publishes (publishHABox). home names the console's home region ("" for // none): a first route is placed in the home. func haClaimCase(t *testing.T, home string) *claimCase { t.Helper() cx := newClaimCaseWith(t, func(c *Config) { c.HomeRegion = home }) cx.srv.cfg.RegionHA = true a := cx.srv.cfg.ClaimAuthority a.Executors = cx.srv.executorChooser(a.Directory) return cx } // publishHABox publishes (or republishes) a roster box's edges file and // status snapshot: placeable in ie unless mut changes it, and stale when // stale is set. func publishHABox(t *testing.T, cx *claimCase, box string, stale bool, mut func(map[string]any)) { t.Helper() ctx := context.Background() m := map[string]any{ "v": 1, "box": box, "box_id": box, "eip": "203.0.113.7", "region": "eu-west-1", "placement": "ie", "hibernation_configured": true, "serves_shared_pool": true, "accepts_inbound": true, "secrets_key_fp": cx.srv.expectedSecretsFP(), "capabilities": []string{"region-ha-1"}, "headroom": map[string]any{"mem_free_mib": 4000, "mem_total_mib": 8000, "disk_free_mib": 50000, "disk_total_mib": 100000, "awake": 0}, "generation": time.Now().UTC().Format(time.RFC3339), "hosts": map[string]any{}, } if mut != nil { mut(m) } body, err := json.Marshal(m) if err != nil { t.Fatal(err) } key := "control/edges/" + box + ".json" _, tag, err := cx.rel.Get(ctx, key) if err != nil { tag = "" } if _, err := cx.rel.Put(ctx, key, body, tag); err != nil { t.Fatal(err) } at := time.Now() if stale { at = at.Add(-time.Hour) } if err := status.WriteBox(ctx, cx.rel, status.Box{Label: box, GeneratedAt: at}); err != nil { t.Fatal(err) } } // registrations counts account/app's registered operations. func registrations(t *testing.T, cx *claimCase, account, app string) int { t.Helper() a, err := hostops.ReadCanonicalAdmission(context.Background(), cx.rel, account, app) if err != nil { t.Fatal(err) } return len(a.Value.Registrations) } // refusedBeforeRegistering checks what H27 promises of a refused create: // nothing registered, no claim, and nothing recovery writes afterwards. func refusedBeforeRegistering(t *testing.T, cx *claimCase, host string, registered int) { t.Helper() ctx := context.Background() if n := registrations(t, cx, "bab", "titogo"); n != registered { t.Fatalf("registrations %d after the refusal, want %d: the door registered before choosing", n, registered) } if obs, err := hostclaim.Read(ctx, cx.rel, host); err != nil || !obs.Absent { t.Fatalf("claim after the refusal: %+v %v", obs, err) } if err := cx.srv.claimRecoveryPass(ctx); err != nil { t.Fatalf("recovery: %v", err) } if obs, err := hostclaim.Read(ctx, cx.rel, host); err != nil || !obs.Absent { t.Fatalf("claim after recovery: %+v %v", obs, err) } if n := registrations(t, cx, "bab", "titogo"); n != registered { t.Fatalf("registrations %d after recovery, want %d", n, registered) } } // wroteNothing checks that nothing reached the bucket or a backend since // mark: no lease, no registration, no record and no store write. func wroteNothing(t *testing.T, cx *claimCase, mark int) { t.Helper() if events := cx.rec.since(mark); len(events) != 0 { t.Fatalf("a refused request wrote %v", events) } } // Region HA spec 2 §7 (H27): every create door that chooses a box refuses // with a 503 when none is eligible, before it registers anything, and the // operator hears of it once per region an hour. Every box here is stale. func TestCreateDoorsRefuseWithNoRoomBeforeRegistering(t *testing.T) { t.Run("instance create", func(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", true, nil) before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() for range 3 { // a retry loop: refused every time, paged once r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", `{"host":"acme.go.tito.io","region":"ie"}`) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("create = %d %s", r.Status, r.Body) } } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) expectPages(t, pages, noRoomSubject("ie")) }) t.Run("standalone sidecar create", func(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") cx.sidecarWorld(t) before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/sidecars", `{"name":"bot","region":"ie"}`) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("sidecar create = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, provision.SidecarHost("bot", "titogo", "bab"), before) expectPages(t, pages, noRoomSubject("ie")) }) t.Run("domain attach, a first route", func(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "ie") shipVersion(t, cx.consoleFixture, "titogo", "v1") seedChannel(t, cx.consoleFixture, "titogo", "edge", "v1") before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/domains", `{"host":"first.titogo-claim.example","kind":"static"}`) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("attach = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, "first.titogo-claim.example", before) expectPages(t, pages, noRoomSubject("ie")) }) t.Run("browser domain attach, a first route", func(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "ie") shipVersion(t, cx.consoleFixture, "titogo", "v1") seedChannel(t, cx.consoleFixture, "titogo", "edge", "v1") before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.formReply(t, "/a/bab/apps/titogo/domains/attach", "host", "first.titogo-claim.example", "kind", "static") if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("browser attach = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, "first.titogo-claim.example", before) expectPages(t, pages, noRoomSubject("ie")) }) } // appDerivedWorld is titogo with no pool and no region allow-list, whose // app region is ie: a create that names no region takes the app's. func appDerivedWorld(t *testing.T, cx *claimCase) { t.Helper() m := titoManifest() m.Regions = nil m.Pool = "" enableInstances(t, cx.consoleFixture, "titogo", m) cx.srv.appRegionOf = func(int64, string) (string, string, error) { return "ie", "app", nil } } // Region HA spec 2 §7 (H27, Astra 2a round 1 finding 3): with region HA // on, an app-derived region is judged by the chooser's eligible set. Every // box stale is the no-room refusal, alert included, never the region // refusal's 400. func TestAppRegionWithEveryBoxStaleIsNoRoom(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") appDerivedWorld(t, cx) publishHABox(t, cx, "box-ie-default", true, nil) before := registrations(t, cx, "bab", "titogo") r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", `{"host":"acme.go.tito.io"}`) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("create = %d %s", r.Status, r.Body) } refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) expectPages(t, pages, noRoomSubject("ie")) } // ... and a misconfigured sibling (a box in ie holding another secrets // key, outside the roster) no longer refuses a region whose eligible set // holds another box: the create goes ahead on that box. With region HA off // the sibling still refuses, as on main. func TestAppRegionWithAMisconfiguredSiblingStillCreates(t *testing.T) { cx := haClaimCase(t, "") appDerivedWorld(t, cx) publishHABox(t, cx, "box-ie-default", false, nil) publishHABox(t, cx, "box-ie-stray", false, func(m map[string]any) { m["secrets_key_fp"] = "sha256:another-key" }) off := cx.srv.placementChooser("") cx.srv.cfg.RegionHA = false r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", `{"host":"other.go.tito.io"}`) if r.Status != http.StatusBadRequest { t.Fatalf("region HA off: create = %d %s, want main's region refusal", r.Status, r.Body) } cx.srv.cfg.RegionHA = true if off == nil { t.Fatal("no chooser with region HA on") } r = cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", `{"host":"acme.go.tito.io"}`) if r.Status != http.StatusCreated && r.Status != http.StatusAccepted { t.Fatalf("create = %d %s", r.Status, r.Body) } obs, err := hostclaim.Read(context.Background(), cx.rel, "acme.go.tito.io") if err != nil || obs.Absent { t.Fatalf("claim: %+v %v", obs, err) } if exec, chosen := obs.Claim.Op.Exec, obs.Claim.Op.Intent["selected_executor"]; exec != "box-ie-default" && chosen != "box-ie-default" { t.Fatalf("executor %q / %q, want box-ie-default", exec, chosen) } } // putSeedStore gives titogo's canary environment a store a seed made, and // returns the store record's bytes. func putSeedStore(t *testing.T, cx *claimCase) (key string, body []byte) { t.Helper() ctx := context.Background() at := time.Now().UTC() if _, err := objstore.PutIn(ctx, cx.rel, "bab", "titogo", "canary", &objstore.Record{ Declared: &objstore.Declared{EnvPrefix: "S3", At: at, By: "bob@keymail.dev", Seed: newSeedID(t)}, Grant: &objstore.GrantBlock{Backend: "mem", Bucket: "carlos-bab-titogo-canary", Source: objstore.GrantSourceAccountApproval, At: at, By: "operator@keymail.dev"}, }, ""); err != nil { t.Fatal(err) } key = objstore.KeyIn("bab", "titogo", "canary") body, _, err := cx.rel.Get(ctx, key) if err != nil { t.Fatal(err) } return key, body } // Astra 2a round 2 finding 2: the instance create door chooses before its // first write. A create naming an environment whose store a seed made would // take that store (bindEnvironmentCreate). Refused for want of room, it // leaves the store's bytes as they were and writes nothing at all: when // every box is stale (the region check refuses), and when the region's last // placeable box goes between the region check and the choice (the early // choice refuses, before the binding). With room, the same create takes the // store, so the binding is on the path. func TestRefusedCreateLeavesTheEnvironmentsStoreAlone(t *testing.T) { const body = `{"host":"acme.go.tito.io","region":"ie","environment":"canary"}` refused := func(t *testing.T, cx *claimCase, pages chan string) { t.Helper() key, stored := putSeedStore(t, cx) before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("create = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) if now, _, err := cx.rel.Get(context.Background(), key); err != nil || string(now) != string(stored) { t.Fatalf("the store after the refusal:\n%s\nwant\n%s (%v)", now, stored, err) } refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) expectPages(t, pages, noRoomSubject("ie")) } t.Run("every box stale", func(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", true, nil) refused(t, cx, pages) publishHABox(t, cx, "box-ie-tito", false, nil) r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body) if r.Status != http.StatusCreated && r.Status != http.StatusAccepted { t.Fatalf("create with room = %d %s", r.Status, r.Body) } store, _, err := objstore.GetIn(context.Background(), cx.rel, "bab", "titogo", "canary") if err != nil || store == nil || store.Declared == nil || store.Declared.Seed != "" { t.Fatalf("the store after a create with room: %+v %v, want the seed's mark cleared", store, err) } }) t.Run("the last box goes before the choice", func(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", false, nil) x := cx.srv.cfg.ClaimAuthority.Executors edges, reads := x.Edges, 0 x.Edges = func(ctx context.Context) (boxchoice.Snapshot, error) { if reads++; reads > 1 { return boxchoice.Snapshot{Complete: true}, nil } return edges(ctx) } refused(t, cx, pages) }) } // Astra 2a round 2 finding 1, at the create door: with region HA on, a // region whose boxes publish no public address (or no edges file) is a // region with no room, not the armed-placement gate's 400. With region HA // off the gate answers, as on main. func TestCreateRegionWithNoReachableBoxIsNoRoom(t *testing.T) { const body = `{"host":"acme.go.tito.io","region":"ie"}` pages := pagesTo(t) cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", false, func(m map[string]any) { m["eip"] = "" }) cx.srv.cfg.RegionHA = false if r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body); r.Status != http.StatusBadRequest { t.Fatalf("region HA off: create = %d %s, want main's 400", r.Status, r.Body) } cx.srv.cfg.RegionHA = true before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("create = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) expectPages(t, pages, noRoomSubject("ie")) } // Census ruling region-ha-2a rev 2, item 3: the early choice takes no lease // and writes nothing, and its executor is the one the authority chooses on // its own (the create's choice before the early choice existed). A choice // the context already carries is kept: nothing is chosen again. func TestChooseFirstWritesNothingAndChoosesOnce(t *testing.T) { cx := haClaimCase(t, "") appDerivedWorld(t, cx) publishHABox(t, cx, "box-ie-default", false, nil) ctx := context.Background() host := "acme.go.tito.io" build := func(now time.Time, acquisition string) (hostops.Record, error) { rec := provision.Record{Schema: 1, Host: host, App: "titogo", Account: "bab", Channel: "stable", Kind: "static", CreatedBy: "bob@keymail.dev", Acquisition: acquisition} return hostops.PrepareInstance(rec, now, cx.srv.cfg.claimPlacementRegion("ie")) } probe, err := build(time.Now().UTC(), "") if err != nil { t.Fatal(err) } before, err := hostclaim.Read(ctx, cx.rel, host) if err != nil { t.Fatal(err) } want, chosen, err := cx.srv.cfg.ClaimAuthority.SelectExecutor(ctx, before, probe) if err != nil || !chosen || want.Box != "box-ie-default" { t.Fatalf("the authority's own choice: %+v %v %v", want, chosen, err) } mark := cx.rec.mark() got, err := cx.srv.chooseCreateFirst(ctx, host, build) if err != nil { t.Fatal(err) } wroteNothing(t, cx, mark) if sel, ok := hostops.ChosenExecutor(got, host); !ok || sel != want { t.Fatalf("early choice %+v (%v), want the authority's %+v", sel, ok, want) } again, err := cx.srv.chooseExecutorFirst(got, host, before, func(time.Time, string) (hostops.Record, error) { t.Fatal("chose again") return hostops.Record{}, nil }) if err != nil || again != got { t.Fatalf("a carried choice was not kept: %v", err) } if sel, chosen, err := cx.srv.cfg.ClaimAuthority.SelectExecutor(got, before, probe); err != nil || !chosen || sel != want { t.Fatalf("the create's selection %+v %v %v, want the door's %+v", sel, chosen, err, want) } } // Census ruling region-ha-2a item 5 (amadan #236): the early choice // carries the claim observation it was decided on, and beginClaimCreate // keeps it only if its own read of the claim is that observation. A claim // write landing between the choice and begin refuses the create with // main's retryable busy answer: the old choice is never used, nothing is // chosen again, and nothing is registered or written. func TestAStaleEarlyChoiceIsNeverUsed(t *testing.T) { cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", false, nil) ctx := context.Background() const host = "acme.go.tito.io" acct, err := cx.st.AccountBySqid("bab") if err != nil || acct == nil { t.Fatalf("account: %+v %v", acct, err) } build := func(now time.Time, acquisition string) (hostops.Record, error) { rec := provision.Record{Schema: 1, Host: host, App: "titogo", Account: "bab", Channel: "stable", Kind: "static", Pool: "tito", Region: "ie", CreatedBy: "bob@keymail.dev", Acquisition: acquisition} return hostops.PrepareInstance(rec, now, cx.srv.cfg.claimPlacementRegion("ie")) } chosen, err := cx.srv.chooseCreateFirst(ctx, host, build) if _, ok := hostops.ChosenExecutor(chosen, host); err != nil || !ok { t.Fatalf("early choice: %v (chosen %v)", err, ok) } // A claim write lands between the choice and begin. created := time.Now().UTC().Truncate(time.Second) inc, err := hostclaim.Incarnation("bab", "titogo", host, created) if err != nil { t.Fatal(err) } writeServingClaim(t, cx.rel, host, inc, created, "box-ie-tito", "ie") written, _, err := cx.rel.Get(ctx, "control/hosts/"+host+".json") if err != nil { t.Fatal(err) } before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() // The build runs only to choose again or, past RegisterStart, to write // the record: either is the old choice's create going on. req, err := cx.srv.beginClaimCreate(chosen, acct, "titogo", host, "bob@keymail.dev", func(time.Time, string) (hostops.Record, error) { t.Fatal("begin went on after the claim changed") return hostops.Record{}, nil }) if req != nil { req.Close() } if !errors.Is(err, errChoiceStale) || !errors.Is(err, hostclaim.ErrConflict) { t.Fatalf("begin on a changed claim = %v, want the stale-choice refusal", err) } if ref := claimResponseRefusal(err); ref.Status != http.StatusConflict || ref.Msg != claimRetireBusyMsg { t.Fatalf("refusal %+v, want main's retryable busy answer", ref) } wroteNothing(t, cx, mark) if n := registrations(t, cx, "bab", "titogo"); n != before { t.Fatalf("registrations %d, want %d", n, before) } if now, _, err := cx.rel.Get(ctx, "control/hosts/"+host+".json"); err != nil || string(now) != string(written) { t.Fatalf("the claim changed under the refusal: %v", err) } } ```` ````diff --- a/internal/hostops/executor_test.go +++ b/internal/hostops/executor_test.go @@ -491,3 +491,31 @@ t.Fatalf("unbound: %v", err) } } + +// Region HA spec 2 §7 (H27): a door that chose before registering holds +// the create to its choice, for that host only; Plan 5's scope still +// binds it (TestProductionCreateSelectsThroughTheChooser). +func TestSelectExecutorHonoursTheDoorsChoice(t *testing.T) { + ctx := context.Background() + x, _, snap, _ := chooserFixture(t, "box-a", "box-b") + a := ProductionAuthority{Namespace: "platform", Executors: &x} + r := chooserRecord() + absent := hostclaim.Observed{Host: r.Holder.Host, Absent: true} + chosen := hostclaim.ExecutorSelection{Box: "box-b", Enrollment: strings.Repeat("b", 32)} + // Every box is now ineligible, yet the door's earlier choice stands. + for i := range snap.Boxes { + snap.Boxes[i].Draining = true + } + sel, ok, err := a.SelectExecutor(WithChosenExecutor(ctx, r.Holder.Host, chosen), absent, r) + if err != nil || !ok || sel != chosen { + t.Fatalf("with the door's choice: %+v %v %v", sel, ok, err) + } + // Another host's choice is not this create's. + if _, _, err := a.SelectExecutor(WithChosenExecutor(ctx, "other.bab.oncarlos.com", chosen), absent, r); !errors.Is(err, ErrNoEligibleBox) { + t.Fatalf("another host's choice was used: %v", err) + } + // No chooser (region HA off): no selection, whatever the context says. + if _, ok, err := (ProductionAuthority{}).SelectExecutor(WithChosenExecutor(ctx, r.Holder.Host, chosen), absent, r); ok || err != nil { + t.Fatalf("region HA off selected: %v %v", ok, err) + } +} ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/console/ -run 'NoCapacity|ClaimRefusalAnswersNoRoom|CreateDoorsRefuse|AppRegionWith|RefusedCreateLeaves|ChooseFirstWrites|CreateRegionWithNoReachable|AStaleEarlyChoice' -count=1` Expected: FAIL to build (`undefined: pageNoCapacity`, `undefined: noCapacityError`). The hostops test fails to build on `undefined: WithChosenExecutor`. - [ ] **Step 3: Write the early choice and the refusal** `internal/hostops/chosen.go`: ````go package hostops import ( "context" "github.com/carlosframework/platform/internal/hostclaim" ) // A door that chooses its executor before it registers anything (region HA // spec 2 §7, H27) carries the choice to the create through the context, so // the create names exactly the box the door checked, and a refusal leaves // nothing registered, nothing written to the claim and nothing for // recovery to settle. selectExecutor still checks the choice against // claim Plan 5's scope (namespace, region, pool, active, unfenced). type chosenExecutorKey struct{} type chosenExecutor struct { host string sel hostclaim.ExecutorSelection } // WithChosenExecutor records sel as host's executor for the create that // ctx carries. func WithChosenExecutor(ctx context.Context, host string, sel hostclaim.ExecutorSelection) context.Context { return context.WithValue(ctx, chosenExecutorKey{}, chosenExecutor{host: host, sel: sel}) } // ChosenExecutor is the executor ctx carries for host, if any: a door that // already chose finds its choice here and does not choose again. func ChosenExecutor(ctx context.Context, host string) (hostclaim.ExecutorSelection, bool) { c, ok := ctx.Value(chosenExecutorKey{}).(chosenExecutor) if !ok || c.host == "" || c.host != host || c.sel.Box == "" || c.sel.Enrollment == "" { return hostclaim.ExecutorSelection{}, false } return c.sel, true } ```` ````diff --- a/internal/hostops/backfill.go +++ b/internal/hostops/backfill.go @@ -107,9 +107,15 @@ // before the create CAS, when a chooser is configured; otherwise no // selection, and the create is today's. The chooser chooses within claim // Plan 5's scope: this authority's namespace and the record's pool. +// +// A door that already chose for this host (WithChosenExecutor) is held to +// that choice, so the create names the box the door checked (H27). func (a ProductionAuthority) SelectExecutor(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, bool, error) { if a.Executors == nil { return hostclaim.ExecutorSelection{}, false, nil + } + if sel, ok := ChosenExecutor(ctx, r.Holder.Host); ok { + return sel, true, nil } var shape struct { Pool string `json:"pool"` ```` `internal/console/nocapacity.go`: ````go package console import ( "context" "errors" "fmt" "log" "sync" "time" "github.com/carlosframework/platform/internal/alerting" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostops" ) // No box has room (region HA spec 2 §7, ruling H27). Every door that // chooses a box (instance create, domain attach's first route, standalone // sidecar create, and the move door's destination) refuses with a 503 when // the placement policy finds no eligible box, and the operator hears of it // at most once per region an hour per console process: a restart may send // one extra alert, which H27 accepts rather than persist the limit. A // create door chooses before it registers anything (chooseExecutorFirst), // so a refusal leaves nothing registered, nothing written to the claim and // nothing for recovery to settle. // noCapacityAlertEvery is how often one region's alert may repeat. const noCapacityAlertEvery = time.Hour func noCapacityRefusal(region string) string { return fmt.Sprintf("No box in %s has room right now. Try again in a few minutes.", region) } func noCapacityAlert(region string) (subject, body string) { return fmt.Sprintf("No box in %s has room", region), fmt.Sprintf("A request in %s was refused because no box there can take it. Every box is full, stale or excluded by its role. Add a box, or free memory or disk on one, and check the eligible-capacity alarm for %s.", region, region) } // noCapacityError is a request refused because no box in Region (its // display name) can take it. It is hostops.ErrNoEligibleBox, and // claimResponseRefusal answers it with noCapacityRefusal. type noCapacityError struct{ Region string } func (e *noCapacityError) Error() string { return "no eligible box in " + e.Region + ": " + hostops.ErrNoEligibleBox.Error() } func (e *noCapacityError) Is(target error) bool { return target == hostops.ErrNoEligibleBox } // pageNoCapacity sends the alert; tests replace it. var pageNoCapacity = alerting.Page // noCapacityAlerts remembers when each region's alert was last sent, in // this console process. The zero value is ready. type noCapacityAlerts struct { mu sync.Mutex last map[string]time.Time } // due reports whether region's alert should be sent at now, and if so // records it as sent. func (a *noCapacityAlerts) due(region string, now time.Time) bool { a.mu.Lock() defer a.mu.Unlock() if at, ok := a.last[region]; ok && now.Sub(at) < noCapacityAlertEvery { return false } if a.last == nil { a.last = map[string]time.Time{} } a.last[region] = now return true } // noRoom is the refusal for region (record vocabulary, "" for the home): // it logs, sends the operator alert when one is due, and returns the // error the door answers with. func (s *Server) noRoom(ctx context.Context, region string) error { name := s.homeRegionOut(region) log.Printf("console: refused: no eligible box in %s", name) if s.noCapacity.due(name, time.Now()) { subject, body := noCapacityAlert(name) go func() { ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second) defer cancel() if err := pageNoCapacity(ctx, alerting.LevelWarning, subject, body); err != nil { log.Printf("console: no-capacity alert for %s: %v", name, err) } }() } return &noCapacityError{Region: name} } // placementChooser is the create door's chooser scoped as // ProductionAuthority.SelectExecutor scopes it, for pool ("" the shared // pool), or nil with region HA off or no claim runtime. func (s *Server) placementChooser(pool string) *hostops.ExecutorChooser { a := s.cfg.ClaimAuthority if !s.cfg.RegionHA || a == nil || a.Executors == nil { return nil } x := *a.Executors x.Namespace, x.Pool = a.Namespace, pool return &x } // roomIn is H27's question about region (record vocabulary) for pool, asked // with region HA on before a door's own placement checks (Astra 2a round 2 // finding 1): a region the roster places boxes in, whose placeable set is // empty (no edges file, no public address, stale, full or excluded by // role), is the no-room refusal, never a 400. It only refuses: it chooses // nothing and changes nothing the door goes on to do. It answers nil with // region HA off, for a region the roster does not place (the door's own // checks answer that), and when a box has room. func (s *Server) roomIn(ctx context.Context, region, pool string) error { x := s.placementChooser(pool) if x == nil { return nil } boxes, err := x.PlaceableBoxes(ctx, s.cfg.claimPlacementRegion(region)) switch { case errors.Is(err, hostops.ErrContract): return nil case err != nil: return err case len(boxes) == 0: return s.noRoom(ctx, region) } return nil } // errChoiceStale is an early choice whose claim observation is no longer // the claim: a write landed between the choice and beginClaimCreate's own // read. It is hostclaim.ErrConflict, so the door answers main's retryable // busy refusal, and nothing was registered or written. var errChoiceStale = errors.New("console: the host's claim changed after its executor was chosen") // chosenOnKey carries the claim observation an early choice was decided // on, beside the choice itself (hostops.WithChosenExecutor). type chosenOnKey struct{} type chosenOn struct { host, etag string absent bool } // chooseExecutorFirst is beginClaimCreate's first step with region HA on, // given beginClaimCreate's own read of the claim (before). It chooses the // executor for the record build would make, before anything is // registered, and carries the choice to the create // (hostops.WithChosenExecutor). It takes no lease and writes nothing: the // choice is SelectExecutor's read of the roster and the edges files. No // eligible box is noRoom's refusal. A choice ctx already carries for host // (chooseCreateFirst's) is kept only when before is the claim observation // it was decided on (census ruling region-ha-2a, item 5): otherwise the // create is refused with errChoiceStale, and the old choice is never used // and nothing is chosen again. With region HA off it returns ctx // unchanged, and the create is main's. func (s *Server) chooseExecutorFirst(ctx context.Context, host string, before hostclaim.Observed, build func(time.Time, string) (hostops.Record, error)) (context.Context, error) { a := s.cfg.ClaimAuthority if !s.cfg.RegionHA || a == nil || a.Executors == nil { return ctx, nil } if _, ok := hostops.ChosenExecutor(ctx, host); ok { on, ok := ctx.Value(chosenOnKey{}).(chosenOn) if !ok || on.host != host || on.etag != before.ETag || on.absent != before.Absent { return ctx, fmt.Errorf("%w: %w: %s", hostclaim.ErrConflict, errChoiceStale, host) } return ctx, nil } probe, err := build(time.Now().UTC(), "") if err != nil { return ctx, err } sel, chosen, err := a.SelectExecutor(ctx, before, probe) if errors.Is(err, hostops.ErrNoEligibleBox) { return ctx, s.noRoom(ctx, probe.Holder.RecordRegion) } if err != nil || !chosen { return ctx, err } return hostops.WithChosenExecutor(ctx, host, sel), nil } // chooseCreateFirst is the instance create door's choice, made before the // door's first write (Astra 2a round 2 finding 2): the environment binding // (bindEnvironmentCreate) can move a seed's store to the new instance, so // a create refused for want of room must be refused before it. It reads // host's claim and chooses as chooseExecutorFirst does, and returns the // choice together with the claim observation it decided on (its ETag, or // its absence). beginClaimCreate keeps the choice only if its own read is // that same observation. With region HA off it returns ctx unchanged and // reads nothing. func (s *Server) chooseCreateFirst(ctx context.Context, host string, build func(time.Time, string) (hostops.Record, error)) (context.Context, error) { a := s.cfg.ClaimAuthority if !s.cfg.RegionHA || a == nil || a.Executors == nil { return ctx, nil } before, err := hostclaim.Read(ctx, s.cfg.Releases, host) if err != nil { return ctx, err } chosen, err := s.chooseExecutorFirst(ctx, host, before, build) if err != nil { return ctx, err } if _, ok := hostops.ChosenExecutor(chosen, host); !ok { return chosen, nil } return context.WithValue(chosen, chosenOnKey{}, chosenOn{host: host, etag: before.ETag, absent: before.Absent}), nil } ```` ````diff --- a/internal/console/server.go +++ b/internal/console/server.go @@ -837,6 +837,10 @@ type Server struct { claimLossPageMu sync.Mutex claimLossPageDone chan struct{} + + // noCapacity limits the no-capacity alert to once per region an hour + // in this process (region HA spec 2 §7, H27, nocapacity.go). + noCapacity noCapacityAlerts // Enabled claim runtime (ConfigureClaimRuntime). Unset flag-off. claimRuntimeConfigured bool claimOwner hostops.RegistrationOwner ```` ````diff --- a/internal/console/claim_create.go +++ b/internal/console/claim_create.go @@ -37,6 +37,12 @@ } before, err := hostclaim.Read(ctx, s.cfg.Releases, host) if err != nil { + return nil, err + } + // Region HA (spec 2 §7, H27): choose before registering anything. A + // door's earlier choice is kept only if this read is the claim it was + // decided on (census ruling region-ha-2a, item 5). + if ctx, err = s.chooseExecutorFirst(ctx, host, before, build); err != nil { return nil, err } appID, err := s.store.ClaimAppID(acct.ID, app) ```` ````diff --- a/internal/console/claim_response.go +++ b/internal/console/claim_response.go @@ -15,6 +15,11 @@ func claimResponseRefusal(err error) *doorRefusal { log.Printf("console: claim request refused: %v", err) + // No box has room (region HA spec 2 §7, H27): every create door's 503. + var full *noCapacityError + if errors.As(err, &full) { + return &doorRefusal{http.StatusServiceUnavailable, noCapacityRefusal(full.Region)} + } if errors.Is(err, hostops.ErrPending) || errors.Is(err, hostclaim.ErrConflict) || errors.Is(err, hostclaim.ErrWrongState) || errors.Is(err, release.ErrCASConflict) { return &doorRefusal{http.StatusConflict, claimRetireBusyMsg} } ```` ````diff --- a/internal/console/apiinstances.go +++ b/internal/console/apiinstances.go @@ -1259,6 +1259,25 @@ if c, ok := findCandidate(cands, region); ok { reason = c.unavailableReason() } + // Region HA (spec 2 §7, H27): the region is judged by the eligible + // set the create door's chooser will choose from, never by every + // sibling's own answer, and an empty set is the door's no-room + // refusal, alert included. + if x := s.placementChooser(""); x != nil { + switch boxes, err := x.PlaceableBoxes(ctx, s.cfg.claimPlacementRegion(region)); { + case errors.Is(err, hostops.ErrContract): + reason = "absent" + case err != nil: + log.Printf("console: api instances create %s/%s: region %s placeable boxes: %v", acct.Sqid, app, region, err) + jsonError(w, http.StatusServiceUnavailable, msgAppRegionUnconfirmed(s.homeRegionOut(region))) + return + case len(boxes) == 0: + writeClaimResponseError(w, s.noRoom(ctx, region)) + return + default: + reason = "" + } + } // Each list names only what an explicit request for this same // instance would pass (codex plan-R4 2): instanceChoices. if reason != "" { @@ -1331,6 +1350,12 @@ // provision.Record's own (DerivesPaths), asked of a probe record rather // than restated here, so it cannot drift from what reconcile will mint. if placed == "" && region != "" { + // Region HA (spec 2 §7, H27): a region with no box that has room is + // the no-room refusal, before this gate answers it as a 400. + if err := s.roomIn(ctx, region, m.Pool); err != nil { + writeClaimResponseError(w, err) + return + } probe := provision.Record{Kind: strings.TrimSpace(req.Kind), Addrs: req.Addrs, Role: probeRec.Role} caps, published := placements(ctx, s.cfg.Releases) // The replica placement rides the same gate as the record's own @@ -1568,25 +1593,36 @@ cancel() ctx, cancel = context.WithTimeout(r.Context(), instanceStoreTimeout) defer cancel() + buildClaim := func(now time.Time, acquisition string) (hostops.Record, error) { + rec.CreatedAt = time.Time{} + rec.Acquisition = acquisition + region := s.cfg.claimPlacementRegion(rec.Region) + prepared, err := hostops.PrepareInstance(rec, now, region) + if err == nil { + rec.CreatedAt = now + } + return prepared, err + } + // Region HA (spec 2 §7, H27): the box is chosen before the door's first + // write, the environment binding below, so a create refused for want of + // room changes nothing. beginClaimCreate keeps this choice. + createContext := r.Context() + if s.cfg.HostClaim { + var chooseErr error + if createContext, chooseErr = s.chooseCreateFirst(createContext, host, buildClaim); chooseErr != nil { + writeClaimResponseError(w, chooseErr) + return + } + } // An environment named without a store walk of its own takes the store // from a seed that made it (final review F3, seedenvbind.go). if !m.PrivateStorage && !s.bindEnvironmentCreate(ctx, w, rec) { return } var claimRequest *claimCreateRequest - createContext := r.Context() if s.cfg.HostClaim { var createErr error - claimRequest, createErr = s.beginClaimCreate(r.Context(), acct, app, host, principal, func(now time.Time, acquisition string) (hostops.Record, error) { - rec.CreatedAt = time.Time{} - rec.Acquisition = acquisition - region := s.cfg.claimPlacementRegion(rec.Region) - prepared, err := hostops.PrepareInstance(rec, now, region) - if err == nil { - rec.CreatedAt = now - } - return prepared, err - }) + claimRequest, createErr = s.beginClaimCreate(createContext, acct, app, host, principal, buildClaim) if createErr != nil { writeClaimResponseError(w, createErr) return ```` - [ ] **Step 4: Run the tests** ```bash nice -n 19 go test ./internal/console/ -run 'NoCapacity|ClaimRefusalAnswersNoRoom|CreateDoorsRefuse|AppRegionWith|RefusedCreateLeaves|ChooseFirstWrites|CreateRegionWithNoReachable|AStaleEarlyChoice|ClaimCreate|InstanceCreate|DomainAttach|Sidecar|EnvironmentCreate|EnvironmentBinding' -count=1 nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser|SelectExecutor|StartCreate|ProductionCreate' -count=1 ``` Expected: PASS. - [ ] **Step 5: Copy gate, boundary list and census** ````diff --- a/approved_region_ha_copy_test.go +++ b/approved_region_ha_copy_test.go @@ -36,7 +36,7 @@ // regionHACopyBuilt is every id whose text the code or docs carry. The // task that writes a row's text adds its id here in the same commit, once // Paul has approved the row. -var regionHACopyBuilt = []string{"RHA-C1", "RHA-C2", "RHA-C3", "RHA-C4", "RHA-C5", "RHA-C6", "RHA-C7", "RHA-C8", "RHA-C9", "RHA-C10", "RHA-C11", "RHA-C12", "RHA-C13", "RHA-C14", "RHA-C15", "RHA-C16"} +var regionHACopyBuilt = []string{"RHA-C1", "RHA-C2", "RHA-C3", "RHA-C4", "RHA-C5", "RHA-C6", "RHA-C7", "RHA-C8", "RHA-C9", "RHA-C10", "RHA-C11", "RHA-C12", "RHA-C13", "RHA-C14", "RHA-C15", "RHA-C16", "RHA-C17", "RHA-C18", "RHA-C19"} func mustParseRegionHARecord(t *testing.T) []copyRow { t.Helper() ```` Run: `nice -n 19 go test . -run 'RegionHACopy|RegionHABuilt' -count=1` Expected: PASS once RHA-C17 to RHA-C19 are approved. ````diff --- a/region_ha_boundary_test.go +++ b/region_ha_boundary_test.go @@ -27,6 +27,8 @@ "internal/edge/lbhealth.go", "internal/console/apiboxroles.go", "system_roles.go", + "internal/console/nocapacity.go", + "internal/hostops/chosen.go", } // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so ```` `handleAPIInstanceCreate` is a record writer in the record-write census (`key:CreateRecord`). Its digest changes; its write site does not: ````diff --- a/internal/provision/plan9_b8_census_test.go +++ b/internal/provision/plan9_b8_census_test.go @@ -662,7 +662,7 @@ // Region HA stage B: the edges read moved whole into internal/edgesfile (EvidenceComplete); the record is still created through CreateRecord. // Re-reviewed 2026-10-12 (merge of main's home-region work): main's body (claimPlacementRegion) with stage B's one-line EvidenceComplete call; no new write. // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's door with the two seed hand-offs: once every ordinary check has passed, a create that names `from` or `import` leaves for createFrom or createImport, which write their own record (beginFrom, beginImport, below) and refuse a claim console first (seedRegionReady); the ordinary create asks bindEnvironmentCreate before its write. The door's own CreateRecord and its claim create are as main wrote them. - {"internal/console/apiinstances.go", "Server.handleAPIInstanceCreate", "record-writer", "key:CreateRecord", "6763eab45774b9e624a9b820ce1585ef3910355bc3ce589b82298e1d12469a2e"}, + {"internal/console/apiinstances.go", "Server.handleAPIInstanceCreate", "record-writer", "key:CreateRecord", "55b802e0cdbae1a970cc8c8a8d1f79b137f82b60cd2ce76af8a2f408f10380d5"}, // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's door under the environment-binding lock (lockEnvironmentUpdate); the UpdateRecord call is unchanged, and UpdateRecord refuses any change to the record's seed. {"internal/console/apiinstances.go", "Server.handleAPIInstanceUpdate", "record-writer", "key:UpdateRecord", "43a8bead675a4aa33777c6c68431882d55d79a9c1d27e1a54c8f103dda087de0"}, {"internal/console/apiscan.go", "Server.writeScanRecord", "nonrecord", "sel:Put", "c3c092f4b7be9bf6bf09d0fe1e2c9fe52892ee97e57f6c7ab964b85fe36fb5c6"}, ```` The `Server.chooseCreateFirst` rows are census ruling region-ha-2a's (platform-1e). If the ruling's own basis text differs from the row's, use it, then recompute the fingerprint with `legacyrow.py` and the seal. Add this comment line after the overlay's last comment line: ````text # Task 11: H27 on the create doors (internal/console nocapacity.go; hostops chosen.go): beginClaimCreate chooses the executor before it registers anything (chooseExecutorFirst) and carries the choice to the create (hostops.WithChosenExecutor, honoured by ProductionAuthority.SelectExecutor), claimResponseRefusal answers no room with a 503 on instance create, domain attach's first route (attachClaimFirstRoute, unchanged) and standalone sidecar create, and handleAPIInstanceCreate judges an app-derived region by the chooser's eligible set. handleAPIInstanceCreate chooses before its first write, the environment binding (chooseCreateFirst), and beginClaimCreate keeps that choice; with region HA on, an explicit region with no box that has room is the no-room refusal before the armed-placement gate (roomIn). Census ruling region-ha-2a covers two accepted-legacy units: Server.beginClaimCreate re-frozen (one call added before RegisterStart, its one root-read site, the claim read file:Read@470, unchanged), and Server.chooseCreateFirst, new: the same claim read, made before the binding, feeding only the placement choice; it returns its choice with the claim observation it decided on, and beginClaimCreate keeps the choice only if its own read is that observation (ruling item 5). The seal moves. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/hostops 6e494cc9e53bf4fa4433fad798ecd3493c972f65083b0b2153b253f0bd50dcba @coverage internal/console 24ef1664e0251ac26a5cfe4cb2f7f4efd56a3881dc1eee413f5d172f0f72690d @legacy 546fc7978037e417ac9df597a95f21add19b025e2d569f8fef7ed4695f0b8bfd internal/console/claim_create.go Server.beginClaimCreate legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@470 ff9b246aa3dcfe180ea1901ecd9fc1b3b18561dc309e08c6a922870d89179705 internal/console/claim_create.go Server.beginClaimCreate legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@470 ff9b246aa3dcfe180ea1901ecd9fc1b3b18561dc309e08c6a922870d89179705 @legacy 5aa99a1b92eb73601585a25265c5285bcfd0c6b6ae7f402248eb1a299bc79831 internal/console/nocapacity.go Server.chooseCreateFirst legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@271 7cc51e7812cc59f39ea23d5285c621d912557f4890d6e76d566f6182b0fe3ffd internal/console/nocapacity.go Server.chooseCreateFirst legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@271 7cc51e7812cc59f39ea23d5285c621d912557f4890d6e76d566f6182b0fe3ffd ```` ````diff --- a/internal/provision/plan14_root_reads_test.go +++ b/internal/provision/plan14_root_reads_test.go @@ -769,7 +769,7 @@ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv"), &inv, true); err != nil { t.Fatal(err) } - seal = "a57250c350c20e4116e5c87b559cfda71f3e0c65d59a2858d83c5f78bea4e9b5" + seal = "2170924f126b744e720ee9e7c9c4c7d7f916bbe89509267219e10a1b1a5ae986" errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) errs = append(errs, checkRootReadDeclarations(got, inv)...) if len(errs) > 0 { ```` Run the census checks and `go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1`. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/hostops/chosen.go internal/hostops/backfill.go internal/hostops/executor_test.go internal/console/nocapacity.go internal/console/nocapacity_test.go internal/console/server.go internal/console/claim_create.go internal/console/claim_response.go internal/console/apiinstances.go approved_region_ha_copy_test.go region_ha_boundary_test.go internal/provision git commit -m "region HA 2a: create doors choose first and refuse when no box has room" ``` --- ### Task 12: The move door refuses a destination with no room **Files:** - Modify: `internal/console/apiinstancemove.go` (`handleAPIInstanceMoveRequest` asks `roomIn` before its placement checks, and answers the no-room refusal from `selectMoveBoxes`) - Modify: `internal/console/boxselect.go` (`moveRoomCheck`; `selectMoveBoxes` asks it of the destination) - Create: `internal/console/boxselect_ha_test.go` - Modify: `internal/provision/plan9_b8_census_test.go`, `internal/provision/plan14_root_reads_test.go` (the seal), the overlay **Interfaces:** - Consumes: Task 11's `placementChooser`, `roomIn`, `noRoom`, `noCapacityError`, `writeClaimResponseError`; Task 6's `PlaceableBoxes`. - Produces: `func (s *Server) moveRoomCheck(ctx, to string, cands []boxCandidate) error`. The move door chooses a box in the destination, so it is H27's fourth door. With region HA on, two checks ask the chooser, and both only refuse (ruling rev 2, item 4): - **Before the placement checks** (Astra round 2 finding 1), `roomIn(ctx, to, "")`: a destination the roster places boxes in, with no placeable box (no edges file, only blank public addresses, stale, full or excluded by role), is the no-room refusal, alert included, instead of the placement checks' 400. - **In `selectMoveBoxes`**, `moveRoomCheck`: when none of the destination's candidates (main's `eligibleBoxes`) is placeable, the no-room refusal. Otherwise the candidates stay exactly main's, so the destination is the box main's selection picks, and a region with several candidates still gets main's several-boxes refusal: the check never narrows the set to pick one. The source is never checked, so an instance on an excluded or full box can still move out (A10-3). With region HA off (Astra round 1 finding 4), `placementChooser` is nil, so both checks answer nil and no roster is read: host claims on and region HA off behave exactly as main, an unreadable roster included. **Census (platform-1e):** `Server.handleAPIInstanceMoveRequest` is an accepted-legacy unit. It gains two refusals before any write; its read sites are unchanged and their offsets move. The overlay re-freezes it, its basis naming both refusals, and the seal moves. Its record-write census digest changes; its write sites do not. - [ ] **Step 1: Write the failing tests** `internal/console/boxselect_ha_test.go`: ````go package console import ( "context" "encoding/json" "errors" "net/http" "strings" "testing" "time" "github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostops" "github.com/carlosframework/platform/internal/provision" ) // roomyEdges gives each box's edges file fresh, roomy headroom, then edit // (when set) changes uk-box's. func roomyEdges(t *testing.T, fx *consoleFixture, edit func(map[string]any), boxes ...string) { t.Helper() ctx := context.Background() for _, box := range boxes { key := edgesFilePrefix + box + ".json" raw, etag, err := fx.rel.Get(ctx, key) if err != nil { t.Fatal(err) } var m map[string]any if err := json.Unmarshal(raw, &m); err != nil { t.Fatal(err) } m["headroom"] = map[string]any{"mem_free_mib": 4000, "mem_total_mib": 8000, "disk_free_mib": 50000, "disk_total_mib": 100000, "awake": 0} m["generation"] = time.Now().UTC().Format(time.RFC3339) if box == "uk-box" && edit != nil { edit(m) } body, _ := json.Marshal(m) if _, err := fx.rel.Put(ctx, key, body, etag); err != nil { t.Fatal(err) } } } // haRoster signs a roster naming both regions, every box of fleet, and // each box's role from roles. func haRoster(t *testing.T, fx *consoleFixture, roles map[string]string, fleet ...fleetBox) { t.Helper() boxes := map[string]hostactors.Box{} for _, b := range fleet { rb := moveRosterBox(b) rb.Role = roles[b.Box] boxes[b.Box] = rb } putMoveRoster(t, fx, hostactors.Roster{Boxes: boxes, Regions: map[string]hostactors.Region{ "ie": {Account: moveTestAccount, AWSRegion: "eu-west-1"}, "uk": {Account: moveTestAccount, AWSRegion: "eu-west-2"}, }}) } // bindRegionHA turns region HA on for fx's claim runtime, with the region // HA chooser bound as the create door binds it. func bindRegionHA(fx *consoleFixture) { fx.srv.cfg.RegionHA = true a := fx.srv.cfg.ClaimAuthority a.Namespace = "platform" a.Executors = fx.srv.executorChooser(a.Directory) } // haMoveFleet is armMoveFleet's ie-box and uk-box (and any more boxes), // each with fresh, roomy headroom, a signed roster naming both regions // with each box's role from roles, and, with ha, a region HA chooser. func haMoveFleet(t *testing.T, ha bool, roles map[string]string, more ...fleetBox) *consoleFixture { t.Helper() fx := newConsoleFixture(t) fleet := append([]fleetBox{ieBox("ie-box"), ukBox("uk-box")}, more...) armMoveFleet(t, fx, fleet...) names := []string{} for _, b := range fleet { names = append(names, b.Box) } roomyEdges(t, fx, nil, names...) haRoster(t, fx, roles, fleet...) fx.srv.cfg.HostClaim = true if ha { d := hostactors.Directory{Store: fx.rel, PublicKey: fx.pub, Bucket: moveTestBucket} fx.srv.cfg.ClaimAuthority = &hostops.ProductionAuthority{Directory: d} bindRegionHA(fx) } return fx } // Region HA spec 2 §7 (H27), census ruling region-ha-2a rev 2 item 4: the // move door's room check only refuses. With room, the selection is the // one region HA off makes; with none, it is the no-room refusal with its // alert; and it never narrows the candidates to pick a destination main's // selection would not. func TestMoveRoomCheckOnlyRefuses(t *testing.T) { const host, app = "acme.go.tito.io", "bab/titogo" ctx := context.Background() t.Run("room: the selection region HA off makes", func(t *testing.T) { off, err := haMoveFleet(t, false, nil).srv.selectMoveBoxes(ctx, host, "uk", "ie", app) if err != nil { t.Fatal(err) } on, err := haMoveFleet(t, true, nil).srv.selectMoveBoxes(ctx, host, "uk", "ie", app) if err != nil || on != off || on.Target != "uk-box" || on.Source != "ie-box" { t.Fatalf("region HA on %+v %v, off %+v", on, err, off) } }) t.Run("no room: the only destination is legacy", func(t *testing.T) { pages := pagesTo(t) fx := haMoveFleet(t, true, map[string]string{"uk-box": hostactors.RoleLegacy}) _, err := fx.srv.selectMoveBoxes(ctx, host, "uk", "ie", app) var full *noCapacityError if !errors.As(err, &full) || full.Region != "uk" { t.Fatalf("err = %v, want no room in uk", err) } if ref := claimResponseRefusal(err); ref.Status != http.StatusServiceUnavailable || ref.Msg != noCapacityRefusal("uk") { t.Fatalf("refusal %+v", ref) } expectPages(t, pages, noRoomSubject("uk")) }) t.Run("a legacy box beside a default one is not filtered out", func(t *testing.T) { roles := map[string]string{"uk-box": hostactors.RoleLegacy} off, err := haMoveFleet(t, false, roles, ukBox("uk-box-2")).srv.selectMoveBoxes(ctx, host, "uk", "ie", app) if err != nil || !off.Many { t.Fatalf("region HA off: %+v %v, want main's several-boxes refusal", off, err) } on, err := haMoveFleet(t, true, roles, ukBox("uk-box-2")).srv.selectMoveBoxes(ctx, host, "uk", "ie", app) if err != nil || on != off { t.Fatalf("region HA on %+v %v, want region HA off's %+v", on, err, off) } }) t.Run("a platform source still moves out", func(t *testing.T) { fx := haMoveFleet(t, true, map[string]string{"ie-box": hostactors.RolePlatform}) got, err := fx.srv.selectMoveBoxes(ctx, host, "uk", "ie", app) if err != nil || got.Source != "ie-box" || got.Target != "uk-box" { t.Fatalf("selection %+v %v", got, err) } }) } // Astra 2a round 1 finding 4: with host claims on and region HA off, the // move door is main's. A persisted role excludes nothing, and the door // reads no roster, so an unreadable one cannot refuse the move. func TestMoveDestinationWithRegionHAOffIsMains(t *testing.T) { const host, app = "acme.go.tito.io", "bab/titogo" fx := haMoveFleet(t, false, map[string]string{"uk-box": hostactors.RoleLegacy}) got, err := fx.srv.selectMoveBoxes(context.Background(), host, "uk", "ie", app) if err != nil || got.Target != "uk-box" { t.Fatalf("selection %+v %v", got, err) } fx.srv.cfg.Releases = failingGet{Store: fx.rel, key: hostactors.Key} if got, err := fx.srv.selectMoveBoxes(context.Background(), host, "uk", "ie", app); err != nil || got.Target != "uk-box" { t.Fatalf("with the roster unreadable: %+v %v", got, err) } } // haMoveDoor is moveDoorWithClaim's world on a host-claim console with the // claim runtime armed: roomy edges files (uk-box's changed by edit), a // roster naming both regions and, with ha, region HA on. func haMoveDoor(t *testing.T, code string, ha bool, edit func(map[string]any)) (*consoleFixture, string, string) { t.Helper() fx, token, moveURL := moveDoorWithClaim(t, code) roomyEdges(t, fx, edit, "ie-box", "uk-box") haRoster(t, fx, nil, ieBox("ie-box"), ukBox("uk-box")) armClaimMoves(t, fx, true, true, nil) if ha { bindRegionHA(fx) } return fx, token, moveURL } // Astra 2a round 2 finding 1: through the whole HTTP door, a destination // with no box that has room (no edges file, or only blank public // addresses) is the no-room refusal, alert included, and writes nothing: // not the placement checks' 400. With region HA off it is main's 400; with // room the move is admitted to uk-box either way. func TestMoveDoorWithNoRoomInTheDestinationIsNoRoom(t *testing.T) { for _, c := range []struct { name string edit func(map[string]any) }{ {"only a blank public address", func(m map[string]any) { m["eip"] = "" }}, {"no edges file", nil}, } { t.Run(c.name, func(t *testing.T) { noFile := c.edit == nil world := func(t *testing.T, code string, ha bool) (*consoleFixture, string, string) { fx, token, moveURL := haMoveDoor(t, code, ha, c.edit) if noFile { if err := fx.rel.Delete(context.Background(), edgesFilePrefix+"uk-box.json"); err != nil { t.Fatal(err) } } return fx, token, moveURL } _, token, moveURL := world(t, "code-ha-move-off", false) if code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`); code != http.StatusBadRequest { t.Fatalf("region HA off: move = %d %s, want main's 400", code, out) } pages := pagesTo(t) fx, token, moveURL := world(t, "code-ha-move-on", true) claim := claimBytes(t, fx, "acme.go.tito.io") code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`) if code != http.StatusServiceUnavailable || !strings.Contains(out, noCapacityRefusal("uk")) { t.Fatalf("move = %d %s", code, out) } expectPages(t, pages, noRoomSubject("uk")) if claimBytes(t, fx, "acme.go.tito.io") != claim { t.Fatal("a refused move wrote the claim") } a, err := hostops.ReadAdmission(context.Background(), fx.rel, "bab", "titogo", claimMoveAppID) if err != nil || len(a.Value.Registrations) != 0 { t.Fatalf("a refused move registered: %+v %v", a.Value.Registrations, err) } if rec, err := provision.GetRecord(context.Background(), fx.rel, "bab", "titogo", "acme.go.tito.io"); err != nil || rec.Move != nil { t.Fatalf("a refused move journaled: %+v %v", rec.Move, err) } }) } t.Run("room", func(t *testing.T) { fx, token, moveURL := haMoveDoor(t, "code-ha-move-room", true, nil) if code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`); code != http.StatusAccepted { t.Fatalf("move = %d %s", code, out) } o, err := hostclaim.Read(context.Background(), fx.rel, "acme.go.tito.io") if err != nil || o.Claim == nil || o.Claim.Op.Intent["target"] != "uk-box" { t.Fatalf("claim %+v %v, want a move to uk-box", o.Claim, err) } }) } ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/console/ -run 'MoveRoomCheckOnlyRefuses|MoveDestinationWithRegionHAOff|MoveDoorWithNoRoom' -count=1` Expected: FAIL: the legacy-only destination is chosen (`err = , want no room in uk`), and both no-room shapes answer the placement checks' `400` (`region uk cannot take a move in this deployment...`). `TestMoveDestinationWithRegionHAOffIsMains` and the region HA off halves pass already: they pin main's behaviour. - [ ] **Step 3: Ask about room, and answer the refusal** ````diff --- a/internal/console/boxselect.go +++ b/internal/console/boxselect.go @@ -6,8 +6,10 @@ "fmt" "log" "os" + "slices" "github.com/carlosframework/platform/internal/boxid" + "github.com/carlosframework/platform/internal/hostops" "github.com/carlosframework/platform/internal/moves" ) @@ -113,6 +115,11 @@ fp := s.expectedSecretsFP() for _, code := range []string{to, from} { cands := eligibleBoxes(snap, roster, code, aws[code], s.cfg.HomeRegion, fp, nil) + if code == to { + if err := s.moveRoomCheck(ctx, to, cands); err != nil { + return moveSelection{}, err + } + } if len(cands) > 1 { sel.Many, sel.ManyRegion = true, s.homeRegionOut(code) return sel, nil @@ -162,3 +169,29 @@ } return out, nil } + +// moveRoomCheck is a move's destination under region HA (spec 2 §7, +// H27): the destination is a door that chooses a box, so when none of its +// candidates is a box the create door's chooser would place on (roles, +// headroom, fresh edges files), the move is the no-room refusal. It only +// refuses (census ruling region-ha-2a rev 2): the candidates, and so the +// destination main's selection picks, are never narrowed or changed. The +// source is never checked, so an instance on an excluded box can still +// move out (A10-3). With region HA off, or no claim runtime, it answers nil +// and no roster is read. +func (s *Server) moveRoomCheck(ctx context.Context, to string, cands []boxCandidate) error { + x := s.placementChooser("") + if x == nil { + return nil + } + boxes, err := x.PlaceableBoxes(ctx, s.cfg.claimPlacementRegion(to)) + if err != nil && !errors.Is(err, hostops.ErrContract) { + return err + } + for _, c := range cands { + if slices.Contains(boxes, c.BoxID) { + return nil + } + } + return s.noRoom(ctx, to) +} ```` ````diff --- a/internal/console/apiinstancemove.go +++ b/internal/console/apiinstancemove.go @@ -591,6 +591,13 @@ return } } + // Region HA (spec 2 §7, H27): a destination with no box that has room + // is the no-room refusal, before the placement checks below answer it + // as a 400. + if err := s.roomIn(ctx, to, ""); err != nil { + writeClaimResponseError(w, err) + return + } caps, published := placements(ctx, s.cfg.Releases) if !published { jsonError(w, http.StatusBadRequest, "this deployment publishes no armed regions. nothing to move to") @@ -602,6 +609,12 @@ } from := s.homeRegionOut(rec.Region) sel, err := s.selectMoveBoxes(ctx, host, to, rec.Region, acct.Sqid+"/"+app) + // No box in the destination has room (region HA spec 2 §7, H27). + var full *noCapacityError + if errors.As(err, &full) { + writeClaimResponseError(w, err) + return + } if errors.Is(err, errEdgesIncomplete) { jsonError(w, http.StatusConflict, "the fleet's edges files could not all be read. try again") return ```` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/console/ -run 'MoveRoom|MoveDestination|MoveDoor|SelectMoveBoxes|InstanceMove' -count=1` Expected: PASS. - [ ] **Step 5: Census** ````diff --- a/internal/provision/plan9_b8_census_test.go +++ b/internal/provision/plan9_b8_census_test.go @@ -655,7 +655,7 @@ {"internal/console/apidoorkeys.go", "Server.publishDoorKeysLocked", "foreign-recordkey", "key:RecordKey,sel:Put", "0fc498cd4af50570a0b7ed3ff71994213cab40040f6701a303bdf85d94f958da"}, {"internal/console/apiinstancemove.go", "Server.moveTransitionSaying", "record-writer", "key:SetRecordMove", "d1f7162eecdc234607c708a83e9633cdf77cc6931bac6754fb51042dd8409377"}, // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's door with the seed rule (refuseSeedMove) after its gate and before the transition; the transition is unchanged and refuses the same seed again under its CAS. - {"internal/console/apiinstancemove.go", "Server.handleAPIInstanceMoveRequest", "record-writer", "key:SetRecordMove,lit:instances", "d4350f8b160076e4e69a0e6fdb5da9a042f4445a7c768dee49be0f6281fc20f3"}, + {"internal/console/apiinstancemove.go", "Server.handleAPIInstanceMoveRequest", "record-writer", "key:SetRecordMove,lit:instances", "867ed0feabca3d5c72990474887d5f88820b9b0e7ef7f72e48703ac4dfee10a8"}, {"internal/console/apiinstancemove.go", "Server.registerMovesPhase1Routes", "key-site", "lit:instances", "5af589d288ba22856f631481c40cc001aac342ea74cee6ee0464ec77d0320bbc"}, {"internal/console/apiinstances.go", "Server.instanceManifest", "key-site", "lit:instances", "e6160ffe84abda9163527ccb33e2ffff2109b35908adb2f3a754534325658fdf"}, {"internal/console/apiinstances.go", "Server.handleAPIInstanceList", "key-site", "lit:instances", "558d0b3e7b0f06faa8e5067ff6f8c131eb899f13d08b71218dfb2454c3e319e1"}, ```` Add this comment line after the overlay's last comment line: ````text # Task 12: H27 on the move door's destination, through hostops.ExecutorChooser.PlaceableBoxes and only with region HA on: roomIn refuses a destination with no box that has room before the placement checks (no edges file, blank public addresses), and moveRoomCheck (internal/console boxselect.go) refuses when none of selectMoveBoxes' destination candidates is placeable. Both only refuse: the candidates and the destination are main's. Census ruling region-ha-2a is asked to re-freeze one accepted-legacy unit, Server.handleAPIInstanceMoveRequest: two refusals added before any write; its read sites unchanged, offsets moved. The seal moves. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/console e9e63edfe287fef5599d0c40a4e7538fbe8462cefaa208875ffb786720dde6f9 @legacy 88952543956a1b090e6b819c05be0504a899945d5c039dee8373fbaa805881a9 internal/console/apiinstancemove.go Server.handleAPIInstanceMoveRequest legacy accepted-legacy accepted-legacy;rev3=census-ruling-region-ha-2a;basis=the console (cmd/console serve under console.service, user carlos-console): main's body with two refusals added before any write, both no room in the destination (H27): roomIn before the placement checks and moveRoomCheck's through selectMoveBoxes; read sites unchanged, offsets moved file:Exec@7439,file:Exec@7527,file:Exec@8518,file:Exec@8695,file:Exec@8846,file:Exec@9846,file:Read@5925,file:Read@8237 867ed0feabca3d5c72990474887d5f88820b9b0e7ef7f72e48703ac4dfee10a8 internal/console/apiinstancemove.go Server.handleAPIInstanceMoveRequest legacy accepted-legacy accepted-legacy;rev3=census-ruling-region-ha-2a;basis=the console (cmd/console serve under console.service, user carlos-console): main's body with two refusals added before any write, both no room in the destination (H27): roomIn before the placement checks and moveRoomCheck's through selectMoveBoxes; read sites unchanged, offsets moved file:Exec@7439,file:Exec@7527,file:Exec@8518,file:Exec@8695,file:Exec@8846,file:Exec@9846,file:Read@5925,file:Read@8237 867ed0feabca3d5c72990474887d5f88820b9b0e7ef7f72e48703ac4dfee10a8 ```` ````diff --- a/internal/provision/plan14_root_reads_test.go +++ b/internal/provision/plan14_root_reads_test.go @@ -769,7 +769,7 @@ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv"), &inv, true); err != nil { t.Fatal(err) } - seal = "2170924f126b744e720ee9e7c9c4c7d7f916bbe89509267219e10a1b1a5ae986" + seal = "8bb6e5f26df334676dddf16387e78206db872cc46ebaaa5565cabf1dc50f40c6" errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) errs = append(errs, checkRootReadDeclarations(got, inv)...) if len(errs) > 0 { ```` Run the census checks. Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/console/boxselect.go internal/console/boxselect_ha_test.go internal/console/apiinstancemove.go internal/provision git commit -m "region HA 2a: the move door refuses a destination with no room" ``` --- ### Task 13: The eligible-capacity metric **Files:** - Modify: `internal/hostops/executor.go` (`Capacity` also answers the roster's AWS region; `PlaceableBoxes`' comment), `internal/hostops/executor_test.go` - Create: `internal/console/regioncapacity.go` (core: the count and the pass) - Create: `internal/console/awsregionmetric.go` (the AWS adapter: CloudWatch) - Create: `internal/console/regioncapacity_test.go` - Modify: `internal/console/hostclaimops.go` (`StartClaimRecoveryPass` starts the pass) - Modify: `region_ha_boundary_test.go`, the overlay **Interfaces:** - Consumes: Task 6's `ExecutorChooser`; Task 11's `placementChooser`; `Server.liveEdges`; `rootaws.LoadDefaultConfig`. - Produces: `func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, string, error)` (the count and the AWS region the signed roster names for the placement); `type regionCapacity struct{ Placement, AWSRegion string; Eligible int }`; `type capacityPublisher interface{ PublishCapacity(ctx, []regionCapacity) error }`; `func regionCapacities(ctx, snap boxchoice.Snapshot, capacity func(context.Context, string) (int, string, error)) ([]regionCapacity, error)`; `(s *Server) publishCapacity`, `runCapacityPass`; the adapter's `capacityData`, `cloudWatchCapacity`, `capacityPublisherFromEnv(ctx, getenv)`. **For 2c:** the metric is `Carlos/Region` / `EligibleBoxes`, dimension `Region` = the placement code, unit Count, every minute, in the region's own CloudWatch; the console needs `CARLOS_REGION_METRICS=cloudwatch` and the platform box role's `cloudwatch:PutMetricData` on that namespace. The pass runs beside claim recovery, so only on a host-claim console, and only with region HA on and a publisher armed. It counts through the same chooser the create doors use (`placementChooser("")`), so the alarm and the refusal agree. Regions are the placements the boxes' edges files name, so a region whose every box has stopped publishing gets no data point, which the alarm reads as breaching. Each region's count goes to the AWS region **the signed roster** names for it (Astra round 2 finding 3): `Capacity` reads it from the roster `eligible` has just read, with `eligible`'s own home mapping, so a stale or misconfigured edges file naming another AWS region cannot send the count elsewhere. `ExecutorChooser.eligible` is unchanged. A count that fails stops the whole pass rather than publish a partial set. - [ ] **Step 1: Write the failing tests** ````diff --- a/internal/hostops/executor_test.go +++ b/internal/hostops/executor_test.go @@ -150,8 +150,8 @@ if err := x.Permits(ctx, absent, chooserRecord(), "box-b"); err != nil { t.Fatalf("Permits(box-b) = %v", err) } - if n, err := x.Capacity(ctx, "ie"); err != nil || n != 1 { - t.Fatalf("Capacity = %d, %v; want 1", n, err) + if n, aws, err := x.Capacity(ctx, "ie"); err != nil || n != 1 || aws != "eu-west-1" { + t.Fatalf("Capacity = %d in %q, %v; want 1 in eu-west-1", n, aws, err) } if boxes, err := x.PlaceableBoxes(ctx, "ie"); err != nil || len(boxes) != 1 || boxes[0] != "box-b" { t.Fatalf("PlaceableBoxes = %v, %v; want [box-b]", boxes, err) @@ -159,10 +159,10 @@ }) } x, _, _, _ := chooserFixture(t, "box-a", "box-b") - if n, err := x.Capacity(ctx, "ie"); err != nil || n != 2 { - t.Fatalf("Capacity with both placeable = %d, %v; want 2", n, err) - } - if _, err := x.Capacity(ctx, "us"); !errors.Is(err, ErrContract) { + if n, aws, err := x.Capacity(ctx, "ie"); err != nil || n != 2 || aws != "eu-west-1" { + t.Fatalf("Capacity with both placeable = %d in %q, %v; want 2 in eu-west-1", n, aws, err) + } + if _, _, err := x.Capacity(ctx, "us"); !errors.Is(err, ErrContract) { t.Fatalf("Capacity of a region the roster lacks = %v, want ErrContract", err) } } ```` `internal/console/regioncapacity_test.go`: ````go package console import ( "context" "errors" "reflect" "testing" "time" "github.com/aws/aws-sdk-go-v2/service/cloudwatch" "github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostops" ) // The count runs without any cloud adapter (§25): it reads only the // edges snapshot and the chooser. Each region's AWS region is the // chooser's (the signed roster's), whatever a file claims: box z's file // names uk in eu-west-1. func TestRegionCapacitiesCountsEachPlacement(t *testing.T) { snap := boxchoice.Snapshot{Complete: true, Boxes: []boxchoice.Box{ {BoxID: "a", Placement: "uk", Region: "eu-west-2"}, {BoxID: "b", Placement: "uk", Region: "eu-west-2"}, {BoxID: "c", Placement: "ie", Region: "eu-west-1"}, {BoxID: "d", Placement: "fleet/pi", Region: "eu-west-1"}, {BoxID: "e", Placement: ""}, {BoxID: "z", Placement: "uk", Region: "eu-west-1"}, }} asked := []string{} capacity := func(_ context.Context, p string) (int, string, error) { asked = append(asked, p) switch p { case "uk": return 2, "eu-west-2", nil case "ie": return 0, "eu-west-1", nil } return 0, "", hostops.ErrContract // no roster region: left out } got, err := regionCapacities(context.Background(), snap, capacity) if err != nil { t.Fatal(err) } want := []regionCapacity{{Placement: "ie", AWSRegion: "eu-west-1", Eligible: 0}, {Placement: "uk", AWSRegion: "eu-west-2", Eligible: 2}} if !reflect.DeepEqual(got, want) || !reflect.DeepEqual(asked, []string{"fleet/pi", "ie", "uk"}) { t.Fatalf("got %+v asked %v", got, asked) } // Any other error stops the count: a partial set would read as no // capacity. failing := func(context.Context, string) (int, string, error) { return 0, "", errors.New("SlowDown") } if _, err := regionCapacities(context.Background(), snap, failing); err == nil { t.Fatal("a failed count published") } } // capacityRecorder keeps what one pass would publish. type capacityRecorder struct{ got []regionCapacity } func (c *capacityRecorder) PublishCapacity(_ context.Context, counts []regionCapacity) error { c.got = counts return nil } // Astra 2a round 2 finding 3, end to end: a stray box's edges file naming // ie in another AWS region (and sorting after the real box) does not move // ie's count out of the roster's eu-west-1. func TestPublishCapacityTakesTheRegionFromTheRoster(t *testing.T) { cx := haClaimCase(t, "") publishHABox(t, cx, "box-ie-default", false, nil) publishHABox(t, cx, "box-ie-stray", false, func(m map[string]any) { m["region"] = "eu-west-2" }) var pub capacityRecorder if err := cx.srv.publishCapacity(context.Background(), &pub); err != nil { t.Fatal(err) } want := []regionCapacity{{Placement: "ie", AWSRegion: "eu-west-1", Eligible: 1}} if !reflect.DeepEqual(pub.got, want) { t.Fatalf("published %+v, want %+v", pub.got, want) } } func TestPlacementChooserIsScopedLikeCreate(t *testing.T) { s := &Server{cfg: Config{RegionHA: true}} if s.placementChooser("") != nil { t.Fatal("a chooser with no claim runtime") } x := &hostops.ExecutorChooser{Namespace: "stale", Pool: "acme"} s.cfg.ClaimAuthority = &hostops.ProductionAuthority{Namespace: "platform", Executors: x} got := s.placementChooser("") if got == nil || got.Namespace != "platform" || got.Pool != "" || x.Namespace != "stale" { t.Fatalf("chooser %+v, original %+v", got, x) } s.cfg.RegionHA = false if s.placementChooser("") != nil { t.Fatal("a chooser with region HA off") } } func TestCloudWatchCapacityPublishesEachRegionInItsOwnRegion(t *testing.T) { now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) type call struct { region, ns, metric, dim string value float64 } var calls []call c := cloudWatchCapacity{now: func() time.Time { return now }, put: func(_ context.Context, region string, in *cloudwatch.PutMetricDataInput) error { d := in.MetricData[0] calls = append(calls, call{region, *in.Namespace, *d.MetricName, *d.Dimensions[0].Name + "=" + *d.Dimensions[0].Value, *d.Value}) if !d.Timestamp.Equal(now) || d.Unit != "Count" { t.Fatalf("datum %+v", d) } return nil }} err := c.PublishCapacity(context.Background(), []regionCapacity{{"ie", "eu-west-1", 2}, {"uk", "eu-west-2", 0}}) want := []call{{"eu-west-1", "Carlos/Region", "EligibleBoxes", "Region=ie", 2}, {"eu-west-2", "Carlos/Region", "EligibleBoxes", "Region=uk", 0}} if err != nil || !reflect.DeepEqual(calls, want) { t.Fatalf("calls %+v %v", calls, err) } if capacityPublisherFromEnv(context.Background(), func(string) string { return "" }) != nil { t.Fatal("published with CARLOS_REGION_METRICS unset") } } ```` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser' -count=1; nice -n 19 go test ./internal/console/ -run 'RegionCapacities|PlacementChooserIsScoped|CloudWatchCapacity|PublishCapacityTakes' -count=1` Expected: FAIL to build (`assignment mismatch: 3 variables but x.Capacity returns 2 values`; `undefined: regionCapacities`, `undefined: cloudWatchCapacity`). - [ ] **Step 3: Write the count, the pass and the adapter** ````diff --- a/internal/hostops/executor.go +++ b/internal/hostops/executor.go @@ -89,8 +89,8 @@ // PlaceableBoxes is every box in placement that could take a new // instance in the chooser's namespace and pool now, sorted: Choose's set, -// before the choice. The console's move door takes a move's destination -// from it (region HA spec 2 §7, H27). +// before the choice. The console's create and move doors ask it whether +// a region has room before they choose (region HA spec 2 §7, H27). func (x ExecutorChooser) PlaceableBoxes(ctx context.Context, placement string) ([]string, error) { r := Record{Holder: hostclaim.NewHolder{Region: placement}} o, cands, err := x.eligible(ctx, hostclaim.Observed{Absent: true}, r) @@ -104,12 +104,22 @@ return boxes, nil } -// Capacity is how many boxes PlaceableBoxes names. The console publishes -// it per region as the eligible-capacity metric (region HA spec 2 §4, -// A1-11). -func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error) { - boxes, err := x.PlaceableBoxes(ctx, placement) - return len(boxes), err +// Capacity is how many boxes PlaceableBoxes names, and the AWS region the +// signed roster names for placement. The console publishes the count per +// region as the eligible-capacity metric (region HA spec 2 §4, A1-11), in +// that AWS region: the roster's mapping, the one eligible reads, never an +// edges file's own claim (Astra 2a round 2 finding 3). +func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, string, error) { + r := Record{Holder: hostclaim.NewHolder{Region: placement}} + o, cands, err := x.eligible(ctx, hostclaim.Observed{Absent: true}, r) + if err != nil { + return 0, "", err + } + // eligible's own home mapping, so the region is the one it read. + if placement == park.HomeRegion && x.HomeName != "" && park.PlacementRegion(x.HomeName) == x.HomeName { + placement = x.HomeName + } + return len(x.placeable(o, cands)), o.Roster.Regions[placement].AWSRegion, nil } // placeable is region HA spec 2's policy over the eligible set (§7): ```` `internal/console/regioncapacity.go`: ````go package console import ( "context" "errors" "log" "sort" "time" "github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostops" ) // The eligible-capacity metric (region HA spec 2 §4, A1-11): once a // minute the console counts, for each region its boxes publish, the boxes // that could take a new instance now (hostops.ExecutorChooser.Capacity: // enrolled, active, unfenced and placeable), and publishes the counts. An // alarm on the count catches what the cloud's own group metrics cannot: a // box that is in service but never enrolled, and a launch loop. Where the // counts go is an adapter's business (awsregionmetric.go); a console with // none publishes nothing. // capacityInterval is how often the counts are published. const capacityInterval = time.Minute // regionCapacity is one region's count. Placement is the CARLOS placement // code the metric is keyed by; AWSRegion is where its boxes run. type regionCapacity struct { Placement, AWSRegion string Eligible int } // capacityPublisher sends the counts to a metrics service. type capacityPublisher interface { PublishCapacity(ctx context.Context, counts []regionCapacity) error } // regionCapacities counts each placement the snapshot's files name, sorted // by placement, and takes each one's AWS region from capacity's answer: // the signed roster's, never the files' own claim, so a stale or // misconfigured file naming another AWS region cannot send a region's // count elsewhere (Astra 2a round 2 finding 3). A placement the roster // does not know (no region entry, so no box there can be chosen) is left // out. Any other error stops the count: a partial set would read as // regions with no capacity. func regionCapacities(ctx context.Context, snap boxchoice.Snapshot, capacity func(context.Context, string) (int, string, error)) ([]regionCapacity, error) { named := map[string]bool{} for _, b := range snap.Boxes { if b.Placement != "" { named[b.Placement] = true } } placements := make([]string, 0, len(named)) for p := range named { placements = append(placements, p) } sort.Strings(placements) var out []regionCapacity for _, p := range placements { n, aws, err := capacity(ctx, p) if errors.Is(err, hostops.ErrContract) { continue } if err != nil { return nil, err } out = append(out, regionCapacity{Placement: p, AWSRegion: aws, Eligible: n}) } return out, nil } // publishCapacity counts and publishes once. func (s *Server) publishCapacity(ctx context.Context, pub capacityPublisher) error { x := s.placementChooser("") if x == nil { return hostops.ErrContract } snap, err := s.liveEdges(ctx) if err != nil { return err } counts, err := regionCapacities(ctx, boxSnapshot(snap), x.Capacity) if err != nil { return err } return pub.PublishCapacity(ctx, counts) } // runCapacityPass publishes every capacityInterval until ctx ends. A // failed pass is logged on its first failure and every tenth; the alarm // treats missing data as breaching, so a console that stops publishing is // itself an alert. func (s *Server) runCapacityPass(ctx context.Context, pub capacityPublisher) { log.Printf("console: eligible-capacity metric armed (every %s)", capacityInterval) t := time.NewTicker(capacityInterval) defer t.Stop() fails := 0 for { pass, cancel := context.WithTimeout(ctx, capacityInterval) err := s.publishCapacity(pass, pub) cancel() switch { case err != nil: fails++ if fails == 1 || fails%10 == 0 { log.Printf("console: eligible-capacity metric (failure #%d): %v", fails, err) } case fails > 0: log.Printf("console: eligible-capacity metric recovered after %d failure(s)", fails) fails = 0 } select { case <-ctx.Done(): return case <-t.C: } } } ```` `internal/console/awsregionmetric.go`: ````go package console import ( "context" "errors" "log" "strings" "time" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/service/cloudwatch" "github.com/aws/aws-sdk-go-v2/service/cloudwatch/types" "github.com/carlosframework/platform/internal/rootaws" ) // The AWS adapter of the eligible-capacity metric (region HA spec 2 §4, // §25): CloudWatch, Carlos/Region EligibleBoxes, keyed by the placement // code, in each region's own CloudWatch, where the region-host module's // alarm reads it. Armed by CARLOS_REGION_METRICS=cloudwatch on the // console. The platform box's role needs cloudwatch:PutMetricData on the // Carlos/Region namespace: plan 2c grants it. const ( capacityNamespace = "Carlos/Region" capacityMetric = "EligibleBoxes" ) // capacityData is the exact PutMetricData input for one region's count. func capacityData(c regionCapacity, now time.Time) *cloudwatch.PutMetricDataInput { return &cloudwatch.PutMetricDataInput{Namespace: aws.String(capacityNamespace), MetricData: []types.MetricDatum{{ MetricName: aws.String(capacityMetric), Dimensions: []types.Dimension{{Name: aws.String("Region"), Value: aws.String(c.Placement)}}, Timestamp: aws.Time(now), Unit: types.StandardUnitCount, Value: aws.Float64(float64(c.Eligible)), }}} } // cloudWatchCapacity publishes each count to its region's CloudWatch. type cloudWatchCapacity struct { put func(ctx context.Context, awsRegion string, in *cloudwatch.PutMetricDataInput) error now func() time.Time } func (c cloudWatchCapacity) PublishCapacity(ctx context.Context, counts []regionCapacity) error { var errs []error for _, n := range counts { errs = append(errs, c.put(ctx, n.AWSRegion, capacityData(n, c.now()))) } return errors.Join(errs...) } // capacityPublisherFromEnv is the console's publisher: CloudWatch when // CARLOS_REGION_METRICS is "cloudwatch", and nil otherwise, or when the // AWS configuration cannot be loaded (logged). func capacityPublisherFromEnv(ctx context.Context, getenv func(string) string) capacityPublisher { if strings.TrimSpace(getenv("CARLOS_REGION_METRICS")) != "cloudwatch" { return nil } cfg, err := rootaws.LoadDefaultConfig(ctx) if err != nil { log.Printf("console: eligible-capacity metric disabled: load aws config: %v", err) return nil } return cloudWatchCapacity{now: time.Now, put: func(ctx context.Context, awsRegion string, in *cloudwatch.PutMetricDataInput) error { _, err := cloudwatch.NewFromConfig(cfg, func(o *cloudwatch.Options) { o.Region = awsRegion }).PutMetricData(ctx, in) return err }} } ```` ````diff --- a/internal/console/hostclaimops.go +++ b/internal/console/hostclaimops.go @@ -4,6 +4,7 @@ "context" "errors" "log" + "os" "time" "github.com/carlosframework/platform/internal/hostclaim" @@ -43,6 +44,14 @@ log.Printf("console: claim recovery: %v", hostops.ErrContract) return } + // The eligible-capacity metric (region HA spec 2 §4, regioncapacity.go) + // runs beside claim recovery, on a host-claim console with region HA + // on and a metrics adapter armed. + if s.cfg.RegionHA { + if pub := capacityPublisherFromEnv(ctx, os.Getenv); pub != nil { + go s.runCapacityPass(ctx, pub) + } + } t := time.NewTicker(claimRecoveryInterval) defer t.Stop() for { ```` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser' -count=1 && nice -n 19 go test ./internal/console/ -run 'RegionCapacities|PlacementChooserIsScoped|CloudWatchCapacity|PublishCapacityTakes|ClaimRecovery' -count=1` Expected: PASS. - [ ] **Step 5: Boundary list and census** ````diff --- a/region_ha_boundary_test.go +++ b/region_ha_boundary_test.go @@ -29,12 +29,14 @@ "system_roles.go", "internal/console/nocapacity.go", "internal/hostops/chosen.go", + "internal/console/regioncapacity.go", } // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so // a reader can see where the provider lives. var regionHAAdapterFiles = []string{ "internal/host/awsaddr.go", + "internal/console/awsregionmetric.go", } // regionHAPackages are the packages spec 2 touches. Every S3 client built ```` Add this comment line after the overlay's last comment line: ````text # Task 13: the eligible-capacity metric (internal/console regioncapacity.go, counted through hostops.ExecutorChooser.Capacity, which now also answers the AWS region the signed roster names for the placement; awsregionmetric.go, the CloudWatch adapter), started beside claim recovery (StartClaimRecoveryPass). No root-read unit changed or was added: Capacity reads through eligible, unchanged. Envelope-only, and the seal stands. ```` Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: ````text @coverage internal/hostops a86df6341b8b1d44676da9ff1310a204f2b34380cb438cc05fe71ae1834834f3 @coverage internal/console 0a6e6c1d77b0c56955bf1b090a450ac6ec5d12f0ed60313e44049cb341a6788c ```` Run the census checks and `go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1`. Expected: PASS: `regioncapacity.go` imports no AWS package, and `awsregionmetric.go` is named for AWS. - [ ] **Step 6: Commit** ```bash git add internal/hostops/executor.go internal/hostops/executor_test.go internal/console/regioncapacity.go internal/console/regioncapacity_test.go internal/console/awsregionmetric.go internal/console/hostclaimops.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv git commit -m "region HA 2a: publish each region's eligible capacity" ``` --- ## Before handing off - [ ] `gofmt -l .` is empty, and `go build ./...` passes with `GOOS` linux, darwin and windows. - [ ] `GOOS=linux go vet ./...` and `GOOS=darwin go vet ./...` pass. - [ ] Every task's targeted tests pass again, then once: `nice -n 19 go test ./internal/console/ -run 'TestClaimConsoleMutationInvariant$' -count=1 -timeout 90m` and the census checks. - [ ] `go test . -run 'RegionHACopy|RegionHABuilt' -count=1` passes with every built row approved by Paul. A pending row means the branch is not ready. - [ ] Push the branch and read `amadan ci status`: CI's `make test` is the full run, `internal/host` included. - [ ] The rollout checklist for region HA gains: consoles before boxes (the console is the roster's only writer, and an older console's roster write would drop the roles); 2c's IAM grant, the IMDS path `network/interfaces/macs/*` and `CARLOS_REGION_METRICS` before the alarm exists; on a box outside AWS, `CARLOS_PRIVATE_IP` with `CARLOS_PRIVATE_CIDR` in `host.env`, or no private address is published. ## Open questions 1. **Hosting before 2b.** Task 9 marks a box hosting when the claim agent's wiring starts, which proves enrolment and a matching roster row but does not itself check that the floor guard is armed for that enrolment. 2b's mode selection makes it exact. Is that acceptable for 2a, given no load balancer reads the answer until 2c? 2. **`Server.chooseCreateFirst`: decided.** platform-1e approved it in the current structure, not the hook inside `beginClaimCreate`, on condition that the choice carries the claim observation it was decided on and `beginClaimCreate` never proceeds on a changed claim. Rev 4 does that and refuses on a change (Task 11). 3. **The census values (condition 3).** Every value here was computed at main `70e57449`. Plan 14 Parts 2 and 3 will move the base; the implementer recomputes every value there with the tooling above, and platform-1e's ruling covers the rows, not these hashes.