# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 19) > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18). **Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 15 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`). **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. ## Rev 19: round 18 dispositions Astra's round 18 (`CODEX-PLAN-REGION-HA-1BN-R18.md`) resolved all three round 17 findings, accepted rulings 1 to 27, and raised two P2 test gaps. Both were checked against the plan's tests and accepted; none is declined. Neither exposed a production defect: the new events and oracle found no failing seed in the unchanged code. Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed 200 seeds, niced, and then each lifecycle guard was removed in turn. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P2) | A | The property's events only started and completed operations. Its recovery oracle wanted every relocation to be ready again on this box. So it could not express a box excluded from an operation that goes on, and it would have rejected correct recovery there. | accepted | New events: `replaced` (this box, the target, is replaced within the same operation by `sp2`, and the relocation continues) and `probeFailedAndReleased`. In every fourth seed, a script follows round 18's trace: target installed, probe fails, release journaled, replaced, cleanup blocked, a pass, an agent crash, then the random events and recovery. The oracle now asks a participating box to be ready, and an excluded box to have finished its cleanup: no controller, no hostexec record, no journaled participation, with the operation still running. New invariant (i): nothing for an operation opens again on a box excluded from it. Invariants (a), cleanup before clear, and (h), persistence across a crash, stay. | Task 7 (`TestRelocationControllerLifecycleProperties`) | | 2 (P2) | A | `TestExecutorChooserSpreadsThenBalancesThenHashes` took the first choice as given and derived its later expectations from it. A chooser that hashed a constant instead of the record's host passed. | accepted | `TestExecutorChooserHashesTheRecordsHostOnEqualLoads` uses equal loads. It computes the winner independently (the greatest SHA-256 of host + "\x00" + box id, claim P5-D2) for `shop.bab.oncarlos.com` (box-b) and `cafe.bab.oncarlos.com` (box-a). The test itself asserts that the two winners differ, and requires `Choose` to return each one. Substituting a constant hostname in `ExecutorChooser.Choose` fails it (cafe chose box-b). | Task 2 | Mutation check (property at 200 seeds, `-failfast`): `Sweep` finishing a participation whose box is excluded, killed at seed 4 (the excluded box never finishes its cleanup). Rev 14's guards, rechecked under the new events, were all killed again: the cleanup before a clear (seed 1, (a)), the verified forget (seed 3, (b)), lost-controller recovery (seed 10), the journal before the open (seed 1, (b)), the close before the clear (seed 6), and the restart path's verification (seed 1, (b)). ## Rev 18: round 17 dispositions Astra's round 17 (`CODEX-PLAN-REGION-HA-1BN-R17.md`) rated F1 to F3 as only partly closed, with one P1 and two P2s. It accepted ruling 26 and its cost. All three findings were checked against the rev 17 code, reproduced, and accepted; none is declined. Stage A was dry-run on a clean 3407d114 in task order, and the lifecycle property passed 200 seeds, niced. Each new guard was removed in turn, and each removal failed its test. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P1, F2) | A, spec | Rev 17's `filesPin` returned the first manifest it found before checking any evidence. §5.6's pre-start copy can lack the `instance.key` the app mints after starting, so a nonempty database was pinned with no key. | accepted | `Pin` computes `needKey`: a writer was admitted and the pinned database is nonempty. `filesPin` then refuses with `ErrFilesUnevidenced` unless the pinned manifest has a live `instance.key` entry (`InstanceKeyPath`, `liveEntry`), and the worker marks the operation `stuck` (rev 17's path). Ruling 25 is amended. The durability window is recorded as the open request L21, with both candidate contracts, for Paul. | Task 5 (`TestStorePinnerRefusesAKeylessManifestForANonemptyDatabase`: keyless with a nonempty database refuses; keyed pins; keyless with an empty database at a release of 0 pins); Requests (L21) | | 2 (P2, F1) | A | `Worker.Step` sends a fenced target to `replace` before any phase check. `replace` returned the chooser's error for ever, so a target terminated after its release, with no spare, bypassed rev 17's bound. | accepted | New fence-consuming `hostclaim.FenceTargetWithoutSpare`, sharing `FenceExhaustedTarget`'s body (`fenceTargetStuck`): it retains the loss, puts the box on `lost`, revokes a dead executor and records `stuck` with its reason. `replace` calls it `ReleaseDeadline` after the fence's `observed_at` when no eligible box exists. | Task 1; Task 5 (`TestWorkerBoundsAProbeFailedWithNoSpare/target_terminated_after_its_release_true`: waiting before the deadline; after it, `stuck` with the fence consumed, `dst` lost and `op.exec` "none") | | 3 (P2, F3) | A | `ReplaceTarget`'s default branch (`probe-failed`, or after P's admission) deleted `prepared` without recording an unopened P. That happens when the probe deadline passes before any admission and the target is then terminated. | accepted | `abandonPrepared(c)` records the current P when no writer opened it. `install` and `ReplaceTarget`'s default branch call it before they replace or drop P. `FenceExhaustedTarget` and `FenceTargetWithoutSpare` keep an unopened P for the operator, so they drop nothing. | Task 1 (`TestRelocationParksAPDroppedFromProbeFailed`: probe deadline before admission, target terminated, replaced, nested relocation completed; P1 is parked) | Guards removed, each failing its test: the keyless refusal; `needKey` only for a nonempty database; the dead-target bound; the bound consuming the fence; `ReplaceTarget` recording the P it drops. That is five. ## Rev 17: two-box model findings The joint model (`origin/region-ha-hostlife-two-box`, model plan rev 2 at 6d25c54e) reported three gaps in rev 16 that review rounds 1 to 16 missed. Each comes with a direct test (`TestFailoverBeforeTheFirstManifest`, `TestProbeFailedInATwoBoxRegion`, `TestAbandonedPreparedGenerationParked`). I checked each against this plan's code first, and all three reproduce. All are accepted; none is declined. Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed its 200 seeds, niced. Each new guard was removed in turn, and each removal failed its test. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | F2 (data loss) | A, B | `StorePinner.filesPin` pinned an empty files set whenever no manifest existed in the admitted chain, whatever the holder's writer history. An install from it starts with no `instance.key`, and the app mints a new one over data sealed with the old (model INV-4). | accepted (ruling 25) | `filesPin` refuses with `ErrFilesUnevidenced` unless no writer was ever admitted (session 0) or the journaled release flushed nothing (`manifest_seq` 0). `Worker.pinUnderLease` marks the operation `stuck` on that error. Stage B: 6b's cold admission writes its manifest copy before the app starts (§5.6), with a named test. | Task 5 (`TestStorePinnerRefusesAnEmptyFilesPinAfterAWriter`, `TestStorePinnerPinsEmptyOnlyWithEvidence`, `TestWorkerMarksAnUnevidencedFilesPinStuck`); stage B re-plan | | F1 (liveness) | A | In `probe-failed` with the release journaled, `Worker.Step` returned the chooser's error forever ("a later pass may find one"). With two boxes no spare exists, and `replaceTargetSlot` refuses the source. | accepted (ruling 26: stuck, not the source) | `ReleaseDeadline` after `probe_failed_since` with still no eligible box, the worker marks the operation `stuck`. Before that it waits as before. | Task 5 (`TestWorkerBoundsAProbeFailedWithNoSpare`: waiting before the deadline, `stuck` after it) | | F3 (leak) | A | `CompleteRelocation` queued `park-local` chores only. A P replaced while unopened (at `installing`, or a deadline `probe-failed` before any admission) stayed in `holder.location` for the at-rest sweep alone. | accepted (ruling 27) | `install` journals every unopened P it replaces in `abandoned_prepared`. The serving CAS queues a `park-generation` for each in the region (`Owner` `region:`), deduplicated, never for an opened P. | Task 1 (`TestRelocationParksEveryAbandonedPreparedGeneration`: replaced at `installing`, completed on P2. P1 is parked and P2 is not.) | Guards removed, each failing its test: `install` journaling the abandoned P; the serving CAS queuing its park; the opened-P exclusion; `filesPin`'s refusal; the release-evidence exception; the worker's `stuck` on `ErrFilesUnevidenced`; the `probe-failed` bound; and waiting before the bound. That is eight. Checked and unchanged: a target terminated with no spare in `starting` or `releasing` also waits for a spare. The live source still serves there (it is `op.exec` until its admission fence closes), so that is not an outage. The model's finding is the `probe-failed` case only. ## Rev 16: round 15 dispositions Astra's round 15 (`CODEX-PLAN-REGION-HA-1BN-R15.md`) resolved all four round 14 items and raised two P2s. I checked both against spec §5.8 (line 847, "a box that finds its receipt missing after that point … the op goes to `stuck`") and against the rev 15 property code. Both are accepted; none is declined. Stage A was dry-run on a clean 3407d114 in task order. The lifecycle property passed 200 seeds, niced, in about four and a half minutes. Each new guard was then removed in turn. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P2) | A, B | Rev 15's `Step` called `EnsureLocal` only while P was unopened. After P opened it went straight to the next attempt, so a lost receipt never made the operation `stuck` as §5.8 requires. | accepted | Receipt verification is now separate from restoration. A new effect, `VerifyReceipt`, runs before every attempt once P is opened. It compares this box's durable receipt (the new `Receipt` provider: the install journal under its `install_stamp`) with the journaled `local_install`, and reads nothing else. If the receipt is missing or different, `Step` marks the operation `stuck` (`MarkStuck`, allowed in `probing` once P is opened) and makes no admission and no reinstall. Files, env and timers changed by the admitted writers are never compared or reverted. Stage B: the production `Receipt`, with named tests. | Task 6 (`TestAParticipantIsStuckWhenItsReceiptIsLostAfterPOpens`: P is opened and the agent restarts. With the receipt lost, the claim goes `stuck` and no admission follows. With it intact, the next attempt goes ahead and nothing is reinstalled or verified against content.) Task 7 (`TestVerifyReceiptChecksTheReceiptOnly`). Stage B re-plan | | 2 (P2) | A | Rev 15's `restart` repaired the faults it was meant to survive: it unblocked and un-failed the cleanup, and closed the old controllers in the background, so their hooks ran. | accepted | `restart` becomes `crash`, an abrupt death. No controller is closed and no hook runs or completes. A dead generation's cleanups, including ones waiting when it died, never return. Its clock never fires again, and the next generation uses fresh lock files: the kernel releases a dead process's locks. Durable records and every external fault persist; recovery stays its own later event. New invariant (h): a record goes only through a successful clear or a successor's open over it, never through a crash or any other step, and while a record exists root's §5.10 check refuses an executable update. | Task 7 (the property, 200 seeds) | | C1 | A | Found by the new crash event. In-process, a dead generation's hostexec goroutines still run: `publish` retries its "lost" write through a directory fd it already holds open. After a repair it recreated a record a live generation had just cleared (seed 75). | fixed in the test | Each generation reaches the data directory through its own symlink, and a crash removes the dead one, so newly opened paths fail. The repair waits 300 ms so a write already in flight through an open fd lands as part of the repair. A real dead process writes nothing, so production code is unchanged. | Task 7 (the property's `dataLink`, `recoverAll`) | Mutation check, with the property at 200 seeds and `-failfast`, plus targeted tests: | Guard removed | Result | |---|---| | `Step` marking `stuck` on a lost receipt after P opens | killed: `TestAParticipantIsStuck…` | | `VerifyReceipt` comparing the receipt exactly | killed: `TestVerifyReceiptChecksTheReceiptOnly` | | both `EnsureLocal` calls | killed, seed 182: (e) | | `EnsureLocal` before an attempt alone / before entering probing alone | each survived: the other covers it | | rev 14's ten lifecycle guards | all killed again; the crash event now kills "journal before the open" at (b) and the handover at (h) | While building C1 I also tried an in-flight-open guard: no forgetting while an abandoned open might still write. It survived 200 seeds, so I removed it. `ClearStale` already refuses while such an open holds the host's lock, and `recordIsMine` refuses on anything unclear. Astra's readings of rulings 1 to 24 stand. ## Rev 15: round 14 dispositions Astra's round 14 (`CODEX-PLAN-REGION-HA-1BN-R14.md`) resolved the lease-recovery finding. It left round 13's finding 2 partly resolved and raised three new P2s. All four are accepted; none is declined. Stage A was dry-run again on a clean 3407d114, applied in task order. The lifecycle property passed its 200 seeds (about three minutes, niced) and was re-run for every mutation. Each new guard was removed in turn; the results are below. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | R13-2 (partly) | A | Rev 14's `recordIsMine` decoded the JSON itself. A parseable record without its enrollment or incarnation, which hostexec's own reader refuses (`State.validate`, `Binding.valid`), was taken for another binding, and the participation was forgotten. | accepted | New `hostexec.ReadLocal(dataDir, host)` (`internal/hostexec/readlocal.go`) is hostexec's validated reader: the same checks `Open`, `CheckLocal` and `ClearStale` apply. `recordIsMine` uses it. Only an absent record, or a valid record of another binding, is "not mine"; anything hostexec refuses is an error, and the participation is kept. | Task 7 (`TestARecordHostexecRefusesKeepsItsParticipation`, for a missing enrollment and a missing incarnation: refused and kept, still kept after a restart, then the record is repaired and a cleanup and clear succeed) | | 1 (P2) | A, B | Claim rev 6.30 R18-1 and moves rev 7.22 §7.6 require the same thing: a create-only prepared manifest at sequence 0, bound to `{location, P, op id, installation stamp}`. The first admission consumes it by exact version, and the session publisher starts at sequence 1. Rev 14's `ToInstalled` required P's sequence to equal the pin's, and `CopyPinnedManifest` kept the source's sequence. | accepted | `CopyPinnedManifest` writes P's prepared manifest at sequence 0, keeping the pinned entries by their content-addressed objects. `PreparedAuthority` separates the manifest's identity from its provenance: `FilesSeq` must be 0, `SourceFilesSeq` must be the pin's sequence, and `InstallStamp` is bound by `ToInstalled` from `install_stamp`. Ruling 13 is corrected. Stage B: R1 has the first admission consume that manifest by its journaled digest as the publisher's baseline, and the re-plan names the tests. | Task 1 (`TestRelocationPreparedManifestIsSequenceZero`: a prepared manifest claimed at the source's sequence 7 is refused, one built from another source revision is refused, and the accepted one is bound to the stamp); Task 5 (`TestPreparedManifestIsSequenceZeroAndTheSessionPublishesOne`: source at 7, prepared at 0, first session publication at 1, inherited entries intact); stage B re-plan | | 2 (P2) | A, B | `Step` in `installed` with `local_install` journaled went straight to probing. Nothing checked that the install still existed after a lost lease's scoped cleanup retired the unopened P's local install. The property never reached installation. | accepted | `hostrelocate.Effects.EnsureLocal` runs before probing and before every attempt until P is opened. `host.RelocationEffects.EnsureLocal` verifies the receipt's install (`VerifyLocal`). If the install is retired, it restores it with the same `Install` under the same `install_stamp`, refuses a restore that is not the receipt's, and verifies again. It never touches an opened P. The property now also covers the target's role: half the seeds start with the local install journaled, and a new event hands the installation to this box. Its effects are stateful: `Install` makes the install, the scoped cleanup retires it unless P is opened, and `Admit` opens P. Two invariants are added: (f) P's first admission never meets a missing local install, and (g) an opened P is never reinstalled. Stage B: the production `VerifyLocal` and restore, with named tests. | Task 6 (`TestAParticipantRestoresARetiredLocalInstallBeforeTheFirstAdmission`), Task 7 (`TestEnsureLocalRestoresARetiredInstallOnlyBeforePOpens`, the property), stage B re-plan | | 3 (P2) | B | 6b's `runtime.Prerequisites` returns `ClaimDependencyError` for any probe or non-serving claim (6b plan, `if p.Probe \|\| o.Claim.Op.Kind != hostclaim.KindServing`). So a source that claim §4 and Task 1 let wake in `starting` and `releasing` could not, even after R1. | accepted (contract) | R1 now also asks for the source's cold-admission dispatch in `starting` and `releasing` while its admission fence is open, refused once the fence closes or the claim no longer names the box. A target's ordinary wake stays refused. The stage B checklist names the tests. | Requests (R1), stage B re-plan | Mutation check, with the lifecycle property at 200 seeds and `-failfast`, plus targeted tests: | Guard removed | Result | |---|---| | both `EnsureLocal` calls | killed, seed 32: (f) | | `EnsureLocal` before an attempt while P is unopened | killed, seed 142: (f) | | `EnsureLocal` before entering probing | survived: the attempt-time check covers it (defence in depth) | | `EnsureLocal` restoring a retired install / refusing a restore under another stamp | killed: `TestEnsureLocal…` | | `recordIsMine` through `hostexec.ReadLocal` | killed: `TestARecordHostexecRefusesKeepsItsParticipation` | | `recordIsMine` comparing the binding | killed, seed 3: (b) | | `ToInstalled` requiring sequence 0 / the pinned source sequence / binding the stamp | killed: `TestRelocationPreparedManifestIsSequenceZero` (each) | | `CopyPinnedManifest` writing sequence 0 | killed: `TestCopyPinnedManifestEnforcesThePin` | | rev 14's lifecycle guards (`finish`'s cleanup before a clear, verified forget, lost recovery, bounded open, journal before open, handover, close before clear, `ErrInactive` superseded, verified restart forget) | all killed again (seeds 2 to 46) | The `EnsureLocal` short-circuit for an opened P also survives, because the participant never calls `EnsureLocal` once P is opened. It is kept so the effect is safe on its own. Astra's readings of rulings 1 to 24 stand. Ruling 13 is corrected as finding 1 requires. ## Rev 14: round 13 dispositions Astra's round 13 (`CODEX-PLAN-REGION-HA-1BN-R13.md`) resolved both findings from round 12 and raised two P2s, both in the execution-controller lifecycle. Both are accepted; none is declined. This is the sixth round in a row to find a controller cleanup or recovery interleaving (rounds 2, 4, 5, 6, 12 and 13). So rev 14 also closes the whole class with a property test over real hostexec controllers, not only the two examples. The changes were dry-run on a clean 3407d114 with stage A applied in task order, building, vetting and running each task's tests after it, plus vet for darwin and windows and the cross-builds. The property test ran its 200 seeds niced, in about three minutes. Each lifecycle guard was then removed in turn (mutation check); the results are below. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P2) | A | One failed lease put makes hostexec latch the controller lost (controller.go, `run`: `loseLocked(putErr)`), and a lost controller never becomes ready again. Rev 13's `Ready` reported the error and kept the controller, `Step` reopens only a missing one, and `Sweep` keeps a running operation's controller. A transient storage fault in `installing` therefore stalled the relocation until the agent restarted. | accepted | `RelocationControllers.live`, used by `Ready` and `Acknowledge`, recovers within the same operation. If the open controller has lost its lease (`Lost()` is closed, newly on `RelocationController`), it is joined (`Close`, bounded by the pass) and the scoped cleanup is repeated. The object is then forgotten, but its journal entry is kept. `Ready` reports `ErrNoController`, and the participant's `Acknowledge` opens a fresh controller with fresh authority and its own lease. The claim keeps the first acknowledgement, as after a restart (rev 2, finding 10). The record on disk stays `lost` until the fresh controller replaces it, so denial is durable throughout. A failed join or cleanup keeps the lost controller, and the next pass repeats both. | Task 7 (`TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime`: lost lease, cleanup failing, then recovered and reopened, no restart); the property test | | 2 (P2) | A | `ClearStale` returns `ErrState` for an absent or invalid claim (controller.go:619) and for an unreadable or mismatched local record, as well as for a record that names another binding. Rev 13's `finish` treated every `ErrState` as replacement and forgot the participation while this binding's record stayed. | accepted | `finish` no longer reads anything into the error. It asks the record itself (`recordIsMine`: `/.hostexec/.json`, compared on box, enrollment, incarnation and operation, as `ClearStale` compares them). The participation is forgotten only when the record is absent or names another binding. An unreadable record, or this binding's record, keeps the participation and reports the error. `RelocationControllers.DataDir` is new; production sets it to the assembly's `Guard.DataDir`, and with it empty nothing is ever forgotten on a failed clear. | Task 7; the property test (claim read failure, absent claim, invalid local record, then restart and recovery) | | C1 | A | The class. | added | `TestRelocationControllerLifecycleProperties` (`internal/host/relocationlifecycle_linux_test.go`) runs over real hostexec controllers, a store that injects faults, and a clock the test drives. Each seed runs 30 random events: lease-renewal failure, a close that times out (a cleanup that blocks), cleanup failure, `ClearStale` failing for each of its causes (claim unreadable, claim absent, invalid local record, a successor that lists this box), agent restart, a successor starting before the sweep, completion, and recovery of the storage. Participant passes are the real `Participant.Step` and `Sweep`. After every step it asserts: (a) no clear without a successful scoped cleanup earlier in the same call; (b) no participation forgotten while its record exists; (c) at most one controller per host; (d) no pass past its deadline. Then (e), liveness: once faults stop, within eight passes the running relocation is ready again; after completion every record is cleared, no controller remains, and a new relocation opens. 200 seeds by default, 20 with `-short`. | Task 7 | | C2 | A | Found while building C1. `hostexec.Open` waits for its Cleanup hook without a context (`err = <-cleaned`), so rev 13's `openBounded`, which waited for `Open` to return, did not bound a pass whose cleanup blocks. | fixed | `openBounded` runs the open apart and returns when the pass's context ends. A controller that opens too late is closed, never kept. | Task 7; invariant (d) | | C3 | A | Found while building C1. After a handover, sweeping the predecessor called `ClearStale` while the successor's live controller held the host's lock, which waited out the whole pass. | fixed | With no controller object, `finish` reads the record first. If it is verifiably not this binding's (absent, or the successor's), the participation is forgotten without a clear: nothing of its own is left to clear. The handover no longer forgets superseded entries itself; the next sweep does, through that check. | Task 7 (`TestASuccessorRelocationRetiresItsPredecessorsController` now sweeps A after the handover) | | C4 | A | Found by the in-order dry run. `TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity` (rev 7) failed about once in five full-package runs. Main's `TestFDHoldersUnsharedThread` can run its unshare on the leader thread, which then keeps a private table for the rest of the process, so no thread shares the leader's. | fixed | The real-kernel test re-executes the test binary and runs in a fresh process. Its guards still fail it (the `KCMP_FILES` check was rechecked). | Task 4 | Mutation check: each lifecycle guard was removed in turn, and the property test was run with `-failfast`. | Guard removed | Result | |---|---| | `finish` repeating the scoped cleanup before a clear | killed, seed 1: invariant (a) | | `finish` forgetting only a verified other binding | killed, seed 3: invariant (b) | | `live` recovering a lost controller | killed, seed 13: invariant (e) (the relocation never recovers) | | `openBounded` | killed: invariant (d) (a pass never returns) | | the journal written before the open | killed, seed 3: invariant (b) | | the handover before a successor opens | killed, seed 53: invariant (e) | | `finish` closing before the clear | killed, seed 2: invariant (e) (never clean) | | `recordIsMine` comparing the binding | killed, seed 3: invariant (b) | | a handover treating hostexec's `ErrInactive` as superseded | killed, seed 7: invariant (e) | | the restart path verifying the record before forgetting | killed, seed 3: invariant (b) | | `live` repeating the scoped cleanup before the reopen | survived 200 seeds | The survivor is defence in depth, not a load-bearing guard. `hostexec.Open` runs the controller's Cleanup hook, which is the same scoped `relocationCleanup` (Task 8a), and fails if the cleanup fails. So no fresh controller, and no fresh authority, exists without a successful cleanup, and invariant (a) is about clears, which recovery never makes. The explicit cleanup is kept so that a failure surfaces at the pass that notices the loss. Astra's readings of rulings 1 to 24 stand. Rev 13's statement that the predecessor's entry is removed "once the successor's open succeeds" is replaced by C3. ## Rev 13: round 12 dispositions Astra's round 12 (`CODEX-PLAN-REGION-HA-1BN-R12.md`) resolved the file-origin finding and raised two P2s. Both are accepted; none is declined. Finding 1 is stage A code. It was dry-run on a clean 3407d114 with stage A applied in task order, building and running each task's tests after it, plus vet for darwin and windows and the cross-builds. Four new guards were removed in turn, and each removal failed its test (listed in the Self-review). Finding 2 is a stage B contract and has no code. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P2) | A | Rev 12's `Acknowledge` looked controllers up by host and operation, and opened a missing one under `e.Lifetime`. `hostexec.Open` loops on `ErrControllerBusy` until its context ends (controller.go, `Open`), and a predecessor's controller keeps the host's lifetime lock until `Close`. `ClearStale` refuses with `ErrInactive` whenever the running operation lists this box, whatever its id (controller.go:643). The regression reproduces it: without the handover, B's acknowledgement waits out the whole pass. | accepted | Before opening a successor's controller, `Acknowledge` calls `RelocationControllers.handover`. It retires every other participation of the same host under the pass's context: close, then the repeated scoped cleanup, then the clear. A clear refused with `ErrInactive` because the successor lists this box counts as superseded. The lock is free by then, and the successor's `Open` replaces the record under it; the predecessor's journal entry is removed only once that open succeeds. A clear that finds another binding's record (`ErrState`) means this participation has no record left. The open itself is bounded by `openBounded`: the controller still lives under the agent's lifetime, but waits for the lock no longer than the pass. A controller that opens in the same instant the pass ends is closed, never kept with a cancelled life. | Task 7 (`TestASuccessorRelocationRetiresItsPredecessorsController`, on real hostexec controllers: A acknowledged; A completes and B starts before A's sweep. B's acknowledgement succeeds, only B is retained, and B's record blocks updates. A lock held by a controller this box does not retain is waited for no longer than the pass, and nothing is kept.) | | 2 (P2) | B | 6b rev 5: `ClaimRoleAssembly.Preflight` returns `activator.ErrClaimIsolationUnavailable`. `runtime.Prerequisites` calls `Roles.Preflight` before dispatch. The "External completion gates" section requires "the reservation-bearing extension of main's isolated root launch protocol … before any tenant shape can run with claims enabled", which "is not supplied by the external owner adapters". | accepted (contract) | Request **R10** asks the claim session and the activator's owner for that protocol, with a relocation probe's `"probe"` work admitted through it. Task 8c is now gated on R10. Task 10 gains an activation gate: the passes stay dormant and refuse while the launcher lacks the protocol. The stage B re-plan names the production probe tests: success through the protocol, refusal without it before any attempt is counted, and the pass staying off. | Requests (R10), Task 8c, Task 10, stage B re-plan | Astra's readings of rulings 1 to 24 stand. ## Rev 12: round 11 dispositions Astra's round 11 (`CODEX-PLAN-REGION-HA-1BN-R11.md`) resolved the chooser-test finding and raised one P2. It is accepted; none is declined. The coordinator also asked for a check of every stage A use of `holder.admissions` as authority. That check found one more case, S1. Both changes were dry-run on a clean 3407d114 with stage A applied in task order (1, 2, 3, 4, 5, 6, 7, 9a), building and running each task's tests after it. Vet for darwin and windows and the cross-builds are clean. Four new guards were each removed in turn, and each removal failed its test (listed in the Self-review). | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P2) | A | `AdmitWriter` drops a covered, loss-archived head once more than eight admissions are kept (generation.go:154). Rev 11's `filesPin` built its origins only from `holder.generation` and the retained admissions. A file published at g1 and never changed is still `files/g1/objects/...` at g10, because `CopyManifest` carries the entry and `Stage` writes nothing for unchanged content. The pin accepted that manifest, but `CopyPinnedManifest` refused it with `ErrOrigin`. Claim §6.2a ("Files in the live set (L13)") makes the live set every object the live manifests reference, computed independently of the database baseline. | accepted | `filesPin` adds `referencedOrigins` to the retained chain: the files prefix of every generation whose objects the pinned manifest's entries and platform items reference. Each referenced object must lie under this holder's location, which binds it to the incarnation, in this region's data bucket (the region binding), at `files//objects/`. Any other reference refuses the pin (`ErrRefused`, `hostfiles.ErrOrigin`), so the pin never admits what the copy would refuse. `CopyPinnedManifest`'s check is unchanged. | Task 5 (`TestPinKeepsAFileOriginTheAdmissionListTrimmed`: ten real `AdmitWriter` and `PublishGeneration` cycles with real `CopyManifest` and publication. The admission list is trimmed to eight and no longer names g1, yet g10's manifest still references g1's object for the unchanged file. The pin and the copy succeed, and a reference to another incarnation refuses.) | | S1 | A | From the check. `preparedOpened`, the boundary that stops P being opened twice or its local install remade (B1-1, B2-5), scanned only the retained admissions. The same goes for `coverageFor`'s "P not opened" test. A covered P admission that is trimmed would let a later probe open P again. | fixed | The admission that opens P journals `prepared_opened` in the same CAS (`relocationAdmission`). `preparedOpened`, now exported as `PreparedOpened` for the participant, reads that key before the admission list. | Task 1 (`TestRelocationPOpenedOutlivesTheAdmissionTrim`, with P's admission removed as the trim would remove it: P's reopening is refused), Task 6 (`coverageFor`) | The check found no other authority taken from the trimmed list. `relocationAdmission` and `coverageFor` compare against the last admission, which the trim never removes: it drops only the head, and only past eight entries. `retainAutomaticLoss` records evidence, not authority. An admission the trim dropped was covered and loss-archived (`admissionLossArchived`), so nothing it could vouch for is lost; it falls back to the holder's live writer session. Astra's readings of rulings 1 to 24 stand. ## Rev 11: round 10 dispositions Astra's round 10 (`CODEX-PLAN-REGION-HA-1BN-R10.md`) resolved both findings from round 9 and raised one P2: a test that would still pass with its production wiring disconnected. It is accepted; none is declined. I then checked every stage A test for the same weakness. For each test of production wiring, the question was whether its assertions still hold with the wiring replaced by a stub. Two more tests failed that check. Both are fixed here, and the table lists them as S1 and S2. The changes were dry-run on a clean 3407d114, applying stage A in task order: build, vet for darwin and windows, cross-builds, and the targeted `hostops`, `console` and `cmd/console` tests. Four new guards were removed in turn, and each removal failed its test (listed in the Self-review). | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P2) | A | `TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims` read an empty bucket. Both readers returning nothing satisfied every assertion. | accepted | The bucket now holds evidence for two boxes in `ie`: their edges files (box-b advertising `region-ha-1`) and fresh status snapshots. It also holds three serving claims, two on box-a for bab/jam and one on box-b for bab/tea. The test reads through the constructed chooser's `Edges` and `Loads`. It asserts each box's identity, freshness, placement and capabilities, and each box's host and per-app counts. Then the evidence changes: box-a starts draining and goes stale, and a fourth host serves on box-b. The next read must show both changes. | Task 2 (`TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims`) | | S1 | A | Found in the scan. `TestProductionAuthoritySelectsOnlyWithAChooser` used a one-box fixture, so a `SelectExecutor` that ignored the chooser and returned a fixed box-a still passed. | fixed | Two boxes, with box-a draining. The selection must be box-b with its roster enrollment. With both draining it must be `ErrNoEligibleBox` and no selection. | Task 2 | | S2 | A | Found in the scan. `regionHAEnabled` sets `Config.RegionHA` from `CARLOS_REGION_HA`, and no test covered it. | fixed | `cmd/console/regionha_test.go`: on only for `strconv.ParseBool` true values, with whitespace trimmed. Unset, `0`, `false`, `no` and garbage are off. | Task 2 | The scan found nothing else in stage A. These tests already run against real wiring: - hostclaim's transitions, on a real store. - `ClaimLoads`, on real claims. - `Guard.AdmissionFenced` and Task 9a's fence observer, against a real `BeginFence`. - `RelocationPolicy`, through `Policy.Check`. - The writers-stopped scan, against the real kernel and a fixture `/proc`. - The worker, over a signed roster, `hostactors.Directory` and `hostfence.ClaimFenceAuthority`. - `StopReceiptReleases`, against stored receipts. - `ReleaseFlushFiles`, against a real store and registry. - `ReleasePositionsFile` and `RelocationControllers`, on disk. - `OpenRelocationController`, with a real hostexec controller. The remaining stubs are providers that Task 10 (stage B) supplies: the worker's `Choose`, `Eligible`, `Leases` and `Prepared`; `StorePinner.SelectDB`, whose production value is `genselect.Select` over the versioned bucket; and `RelocationEffects`' providers. The stage B re-plan has a row that requires each one to be tested with its production value. Astra's readings of rulings 1 to 24 stand. ## Rev 10: round 9 dispositions Astra's round 9 (`CODEX-PLAN-REGION-HA-1BN-R9.md`) resolved both round 8 findings and raised two new ones. Both are accepted; none is declined. This revision also adds a dry run that applies stage A in task order. The earlier dry runs applied every task at once, which is how finding 1 got through. This time a clean 3407d114 took Tasks 1, 2, 3, 4, 5, 6, 7 and 9a one at a time. After each task: `go build ./...`, vet of that task's packages, and that task's tests. With finding 1 fixed, every step built and passed and turned up no other ordering fault. The tree as a whole then passed vet for darwin and windows and the cross-builds. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P1) | A | Applied in order, Task 5 failed to build: `internal/hostrelocate/pin.go:259:23: undefined: hostfiles.ValidateManifest`. The helper was first defined in Task 7's `internal/hostfiles/platform.go`. | accepted | `ValidateManifest` now lives in its own file, `internal/hostfiles/manifestcheck.go`, which Task 5 creates (Files list, Step 4, the Step 5 run and the Step 6 `git add`). Task 7's `platform.go` keeps only `EnvItem` and `StagePlatformDelete` and uses the helper unchanged. | Task 5, Task 7; the in-order dry run | | 2 (P2) | A | Selection and completion in rev 9's test both named box-a, and the intent check called `BeginCreateOn` itself. The test still passed with `Attempt`'s selection removed. | accepted | The fake authority now has a selected box, the box completion offers, and a call counter. With selection on box-b and completion offering box-a, `Attempt` must refuse with `ErrWrongState`. The persisted claim must still be `creating`, with `selected_executor` box-b and `executor_enrollment` enrollment-b. `Complete` with box-b then serves on box-b, and selection must have run exactly once. Removing `Attempt`'s use of the selection (`if selected` made false) fails the test: the completion substitutes box-a. | Task 2 (`TestCreateIsTodaysWithoutASelection`) | Astra's readings of rulings 1 to 24 stand. ## Rev 9: round 8 dispositions Astra's round 8 (`CODEX-PLAN-REGION-HA-1BN-R8.md`) resolved the release-retry finding and added two P2s. I checked both against `internal/host/claimstop.go` and `internal/hostguard/fence.go` and accepted both; none is declined. Both fixes are stage A code. They were dry-run on a clean 3407d114: build, vet for darwin and windows, cross-builds, and the targeted `hostclaim`, `hostrelocate` and `host` tests. Two new guards were each removed in turn and their tests failed (listed in the Self-review). Ruling 23 is amended. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P2) | A | Rev 8's `FenceExhaustedTarget` always set `op.exec` to "none" and the phase to `pinning`. In `starting` and `releasing`, `op.exec` is the live source, and the verified fence is the target's. Root's stop refuses a requester that is not `op.exec` (claimstop.go:184, `observed.Claim.Op.Exec != req.Box`). | accepted | The fix depends on who holds `op.exec`. If another box holds it (the live source in `starting` or `releasing`), the CAS consumes the target's fence, retains its loss, puts it on `lost`, drops it from the participants and records `stuck`. It leaves `op.exec`, the phase, the stamp and every release record unchanged. The old move to a stuck `pinning` with `op.exec` "none" now happens only when `op.exec` is the dead target itself (its own fence revokes it) or is already "none". | Task 1 (`TestRelocationExhaustedTargetIsFencedIntoAStuckPinning/dead_while_the_source_executes_in_starting` and `…_in_releasing`, the latter with the source's release journaled first); ruling 23 | | 2 (P2) | A | Rev 8's `Release` restored `recorded.ExpectedDirectoryRevision` only when the boot or controller had changed. Root's stop compares that revision with its stop inventory (claimstop.go:191). The fence target also carries it (`hostguard.FenceTarget.DirectoryRevision`, fence.go). | accepted | When matching positions are recorded, `Release` always asks under the recorded `ExpectedDirectoryRevision`, recovery or not. Only the recovery branch takes the recorded boot and controller. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/root_fails_after_the_producers_closed`: the retry runs under the same boot and controller, with the roster at revision 5. Root is asked at revision 4, and only the producers' drain and the receipt run.) | Astra's readings of rulings 1 to 22 and 24 stand. Ruling 23 is amended as finding 1 requires. ## Rev 8: round 7 dispositions Astra's round 7 (`CODEX-PLAN-REGION-HA-1BN-R7.md`) resolved the kcmp finding, confirmed that stage B's contracts are complete, and added one P2. We checked it against `internal/host/claimstop.go` and accepted it; none is declined. The fix is stage A code. It was dry-run again on a clean 3407d114: build, vet for darwin and windows, cross-builds, and the targeted `internal/host` tests. Two new guards were removed in turn and each made the new test fail (listed in the Self-review). | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P2) | A | In `ClaimStopCoordinator`, `DrainClaim` runs before `Kernel.VerifyStopped` and `Guard.CompleteFence` (claimstop.go:219 onward). If either fails, root has no durable receipt. On retry, `BeginFence` returns the same fence, `FenceReceiptFor` returns `ErrNoFenceReceipt`, and root asks for the drain again. Rev 7's `Release` read the recorded positions but still built a fresh `releaseDrain`, which repeated `StopWriters`, the observation and both flushes against a replicator its own producer drain had already closed. | accepted | When this controller has already recorded positions for the exact stop (same boot and controller, root without a receipt), `Release` marks its drain `resumed` and carries the recorded TXID and sequence. A resumed drain goes straight to the producers' own drain, which is idempotent, and leaves verification and the receipt to root. A restarted agent keeps the recovery path; a replayed receipt keeps pairing with the record. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/root_fails_after_the_producers_closed`: the drain closes the producers, root fails before persisting its receipt, and the retry without a restart runs only `producers` and `receipt`, with no flush, record or writers stop, and returns 41 and 8) | Astra's readings of rulings 1 to 24 stand. ## Rev 7: round 6 dispositions Astra's round 6 (`CODEX-PLAN-REGION-HA-1BN-R6.md`, against `222ff1c6`) resolved all six of round 5's findings and added one. It was checked against the plan and the kernel before anything changed, and is accepted; none is declined. The fix is stage A code, dry-run again on a clean 3407d114 (build; vet for darwin, windows and linux/arm64; cross-builds; the stopevidence tests run five times against the real kernel), with 3 new guards removed in turn (listed in the Self-review). Stage B is unchanged: R3's root endpoint calls the same `WritersStopped`. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | 1 (P1) | A | `scanOne` exempted a thread's descriptor when the leader showed the same number and path. Confirmed on this kernel: a thread that runs `unshare(CLONE_FILES)` keeps a copied table with the database at the same number, and it can `dup3` an independent open onto that number; `/proc//task//fd/` reads the same path either way. kcmp(2) is the positive check: `KCMP_FILES` compares two tasks' descriptor tables and `KCMP_FILE` compares two descriptors' open file descriptions. golang.org/x/sys v0.46.0 has `unix.SYS_KCMP` on all 15 linux architectures but no `unix.Kcmp` and no constants, so the plan makes a raw `Syscall6` with `KCMP_FILE = 0` and `KCMP_FILES = 2` from linux/kcmp.h. kcmp needs ptrace read access to both tasks. The agent always has it within its own thread group, and root's complete scan has it through `CAP_SYS_PTRACE`. This box runs `CONFIG_KCMP=y` and Yama scope 0. | accepted | A thread's descriptor is the leader's only if three things hold: the leader shows the same number and path, `KCMP_FILES(pid, tid)` is 0 (the thread shares the leader's table), and `KCMP_FILE(pid, tid, n, n)` is 0 (the same open file description). Otherwise it is reported with FD -1, which no allow-list exempts. A copied table counts as foreign even when the description is shared, because the thread could still use that description after the replicator closes. If a task exits mid-scan (ESRCH), it is skipped as before. EBADF is treated as "not the leader's". Any other kcmp error, including ENOSYS and EPERM, is `ErrIncomplete`. kcmp takes pids in the caller's pid namespace, so it is used only for the caller's own `/proc`. Any other root fails closed unless a test replaces `kcmpFor`. The `//go:build linux` file and the `!linux` stub stay as they were. | Task 4 (`TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity` on the real kernel: a shared locked thread adds nothing; an unshared thread with the copied descriptor, and one that reopened the database and `dup3`'d it onto the leader's number, are each reported FD -1. `TestWritersStoppedRefusesAnyOtherHolder`'s new fixture cases: an unshared thread at the leader's number and path, a shared table whose descriptor is another description, and kcmp unavailable (ENOSYS, EPERM) as `ErrIncomplete` for both scans) | Astra's readings of rulings 1 to 24 stand. ## Rev 6: round 5 dispositions Astra's round 5 (`CODEX-PLAN-REGION-HA-1BN-R5.md`, against `996b24c3`) left R4-5 partly resolved and added five findings. Each was checked against the tree at 3407d114 and the plan before anything changed; all six are accepted and none is declined. Every one lands in stage A code, dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds for darwin, windows and linux/arm64, targeted tests), with 11 new guards removed in turn (listed in the Self-review); R4-5's restart obligations are carried into the stage B re-plan as well. | # | Stage | Evidence checked | Disposition | What changed | Where | |---|---|---|---|---|---| | R4-5 | A, B | `hostexec.Controller.Close` (controller.go:480) sets `closed` before returning `errors.Join(err, c.CleanupError())` and returns nil on every later call; rev 5's `Finish` cleared after any nil `Close`, and the restart path cleared from the journaled binding with no cleanup at all. Rev 5 also journaled only inside `Finish`, so a record left by a crash (or a failed `Open`, which writes `lost` once it holds the live lock) before the first `Finish` was never found. | accepted | Repeat, do not trust: `RelocationControllers.Cleanup(ctx, binding)` is the scoped cleanup (Task 8a's `relocationCleanup`, authenticated to the binding's box and enrollment) and `Finish` runs it, and needs it to succeed, in the same call as every clear, after `Close` and on the restart path alike; a failure keeps the record, the entry and the journal. The participation is journaled before the controller opens (`Acknowledge` refuses without a journal), so a restart finds every record it may have left; the journal write syncs its directory. | Task 7 (`TestFinishClearsTheDurableRecordAndRetriesAcrossARestart`: cleanup failure, a sweep after it, a restart while it still fails, then success; `TestARestartFindsAParticipationThatNeverReachedFinish`; `TestFinishClearsTheDurableRecordSoUpdatesAreEligible` with a real hostexec controller whose cleanup fails); stage B re-plan | | 1 | A | `hostops` `eligible` excluded only `Lost`; `Worker.next` excluded the source, `Lost` and replaced targets; `ReplaceTarget` reaches `install` directly in `installing` and `installed`, never through `ToInstalling`'s `park-local` check. | accepted | `hostclaim.OwesLocal` and `LocalChoreBoxes`. Selection excludes every box owing a `park-local` (the chooser's eligible set, so create, `Permits` and the worker's spare all agree; `Worker.next` too), and `replaceTargetSlot` refuses one with `ErrPendingChore` inside the CAS, for `ReplaceTarget` at every phase and `NameTarget`. | Task 1 (`TestRelocationReplacementRefusesABoxOwingALocalRetirement`, in `starting`, `installing` and `installed`), Task 2 (`TestExecutorChooserNeverNamesAFencedLostOrUnknownBox/owing_a_local_retirement`), Task 5 (`TestWorkerNeverReplacesATargetWithABoxOwingALocalRetirement`) | | 2 | A | `fenceBox` journaled the fence, added `lost`, discharged `park-local` and retained the loss, and touched no regional obligation; `lost.go:302` reassigns `box:`'s `park-generation` and `park-replica` to `region:` for a declared loss. | accepted | `fenceBox` passes every `park-generation` and `park-replica` owned by `box:` to `region:` in the same CAS, changing nothing but `owner` (an obligation with no region refuses, as in `DeclareLost`); the retained `AutomaticLoss` names them (`reassigned`, the D4 record's `reassign_regional`). Every automatic fence CAS shares it: start, `FenceSource`, `ReplaceTarget`, `FenceExhaustedTarget`. | Task 1 (`TestRelocationFenceReassignsTheDeadBoxsRegionalWork`, unqualified and qualified discharge); R5 | | 3 | A | Task 9a's `check` replaced `Serving` with `RelocationRow` unconditionally; with both callbacks nil a `moving` executor participant is `RowHeld` and passed, and `Install` went on to `Activate` and `RecordInventory` before `WorkSource.SelectAt` (work.go:94) refused. | accepted | `Reconciler.relocationAware()` is `FenceClosed != nil \|\| HoldRow != nil` (Task 10 sets both only with `CARLOS_REGION_HA`). Off, `check` is today's `Serving` and `Install` skips the row decision and the hold-only path entirely. | Task 9a (`TestReconcileWithoutAFenceObserverIsTodays`: a relocating source in `starting` and `releasing` and a `moving` executor, dry run and real, refused with the claim's ETag and every file under the guard's data directory unchanged) | | 4 | A | `journaled` decoded every directory entry, so `writeFileSynced`'s truncated `.json.tmp` failed the whole listing, and `Retained` dropped the error. | accepted | Only names ending `.json` are committed records; a `.tmp` is ignored (its participation opened nothing: the record precedes the open). A committed record that cannot be read is reported, joined into the error, beside every readable one. `Effects.Retained` returns `([]Participation, error)`; `Sweep` reports the error and still finishes every readable participation. | Task 7 (`TestARestartFindsAParticipationThatNeverReachedFinish`: one valid journal, one truncated `.tmp`, then one damaged committed entry), Task 6 (`TestAParticipantFinishesItsControllerOnceTheOperationIsOver`) | | 5 | A, B | `Flush` loads the existing manifest through `hostfiles.Open` and did nothing on `ENOENT`, so `Platform[EnvItem]` kept the old bytes. | accepted | `(*hostfiles.Publisher).StagePlatformDelete(name)` (in `platform.go`) drops the item at a new sequence; `Flush` stages it on `ENOENT`, so `ReadPlatform` on the final revision returns `os.ErrNotExist`. Stage B's local install treats that as "no env file" (re-plan). | Task 7 (`TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv`: published, removed, flushed again, then a no-op repeat) | Astra's readings of rulings 1 to 24 stand; ruling 20's cleanup is completed by R4-5 above. ## Rev 5: round 4 dispositions Astra's round 4 (`CODEX-PLAN-REGION-HA-1BN-R4.md`, against `418daf58`) left three earlier findings partly resolved and added six. All nine are accepted; none is declined. Under ruling 24, stage A findings are fixed in code (dry-run on a clean 3407d114, 7 new guards removed in turn, listed in the Self-review); stage B findings are fixed at the contract level: what must happen, which request covers it, which test proves it. | # | Stage | Disposition | What changed | Where | |---|---|---|---|---| | R1-9 | B | accepted | `Region` is 6b's field: Task 8a's `Collect` provider sets it on every request from the verified actor (the box's region never changes), initially and on recovery alike, beside `Recover` for the recorded owner's stop. | Task 8a | | N1 | B | accepted | The fixture's `withExistingRow` takes the hold everywhere it is called; the census literal carries 6b's actual fields (`Inventory`, `Registry`, `Launches`, `Box`, `Enrollment`, `DirectoryRevision`, `Roots`, `SharedPaths`), no `Guard`. | Task 9 | | N3 | B | accepted | R8's insertion point is after actor verification: 6b's `Build` calls an `Extend(a *ClaimAssembly, actor hostbox.Actor)` hook before it captures `Policy.Check`; `applyRegionHA` is that hook's body and binds the policy to `actor.Box`. | Task 10, R8 | | 1 | A | accepted | `Guard.AdmissionFenced(host, claim, captured)` takes no lock (one atomic record read; the claim is the caller's fresh read); `Reconciler.FenceClosed` takes the context and the observation and returns an error, which keeps rows and mints none; `holdOnly` re-decides on a fresh read under `WithLocal`. Tested with the real observer, a real `BeginFence`, and a bounded context on both the activation and the hold-only path. | Task 3, Task 9a (`TestReconcileRelocationSourceReadyThenHeldWithTheRealFenceObserver`) | | 2 | A, B | accepted | Policy half (A): a relocation target's probe is its own selected action, `probe`, which `Policy.Check` admits for a relocating claim and `RelocationPolicy` sends to `WakePhase(probe=true)`; an ordinary `spawn` by the target stays refused. Dispatch half (B): R1 extended, 6b's runtime selects `"probe"` work for a relocation probe. | Task 3 (`TestRelocationPolicyGivesEachActionItsRole`, `TestPolicyLetsAHeldParticipantReconcileItsRow`), R1 | | 3 | B | accepted | The restore destination (the host's files directory and env path) is created and inventoried under the installation's authority before `Restore`, through 6b's typed inventory (a `files` entry at `install_stamp`), never an untracked `MkdirAll`; tested with a fresh spare that has no host directory. | Task 8c | | 4 | B | accepted | Held release staging needs root: request R9 asks moves for a relocation-aware relay authorizer (bound to the operation, executor, install stamp and the pinned release); the install verifies staging before it records its receipt; tested under the edge unit's filesystem restrictions. | Task 8c, R9 | | 5 | A | accepted | `RelocationController.Clear` (hostexec's `ClearStale` with the controller's config); `Finish` journals the participation before closing, keeps the entry and the record on a failed close or clear, clears the exact record after a successful cleanup, and a restart retries through the journaled binding. A real hostexec controller on disk shows root's §5.10 check (`hostexecParticipation`) refusing before and admitting after. | Task 7 (`TestFinishClearsTheDurableRecordSoUpdatesAreEligible`, `TestFinishClearsTheDurableRecordAndRetriesAcrossARestart`) | | 6 | A | accepted | The obsolete-hold clearance in `Policy.Check` needs `Relocation` set (region HA on); off, today's refusal. Tested both ways on the same held serving row. | Task 3 (`TestPolicyLetsTheServingExecutorClearAnObsoleteHold`) | ## Rev 4: round 3 dispositions Astra's round 3 (`CODEX-PLAN-REGION-HA-1BN-R3.md`, against `d93c94d0`) left three earlier findings partly resolved, one unresolved, and added six. All ten are accepted; none is declined. The un-gated changes were dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds, targeted tests) and each new guard removed in turn (8 removals, each failing its test; listed in the Self-review). | # | Disposition | What changed | Where | |---|---|---|---| | R1-5 | accepted | `PreparedFromInstallManifest.Verify` compares P's entries against the pinner's retained copy (`pin.Files.Retained`), never the source manifest; a late source PUT changes nothing it reads. | Task 8b | | R1-9 | accepted | After a restart `Release` names the recorded owner's boot, controller and directory revision and offers `RecoveryReleaseDrain`; Task 8a's `Collect` provider sets 6b's `Recover` on that drain, so root's authenticated recovery path accepts the tuple and replays the receipt. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/after_a_restart…`), Task 8a | | R2-N4 | accepted | `Policy.Check`'s serving branch lets exactly the serving executor's `reconcile` through an obsolete hold (one naming no operation the claim runs), so the adapter's clear is reachable; every other write under a hold still waits. | Task 3 (`TestPolicyLetsTheServingExecutorClearAnObsoleteHold`), Task 9 | | R2-N5 | accepted | See N2. | Task 7 | | N1 | accepted | Task 9's adapter insertion re-reads the claim under the captured authority (`observed` is defined there, validated against the captured incarnation and stamp); the test fixture (`adapterFixture` and its methods) is this task's code, over 6b's adapter and registry. | Task 9 | | N2 | accepted | `Finish` keeps the map entry until `Close` succeeds and forgets only that same controller, so a close the pass's context cut short is finished by a later sweep. | Task 7 (`…RetainsTheControllerUnderTheAgentLifetime`: a cancelled close, then a successful one) | | N3 | accepted | `hostbox.RelocationPolicy(box)` (un-gated, tested) and `hostguard.Guard.AdmissionFenced` (un-gated, tested) are the two providers; Task 10 wires them into the assembly between 6b's `OpenProductionClaimAssembly` and `Build` (which captures `Policy.Check`), gated on `CARLOS_REGION_HA`; with it off neither is set and behaviour is 6b's. The insertion point is request R8. | Task 3, Task 10, R8 | | N4 | accepted | `Reconciler.Install` takes the hold-only path for a held row whenever the box is not the executor or its admission fence is closed, so a fenced source still named `op.exec` in `releasing` writes its hold under `WithLocal`, never through the guarded path `With` refuses. | Task 9 | | N5 | accepted | `FenceExhaustedTarget`: a target terminated after the replacements are used is fenced into a stuck `pinning` in one CAS (loss retained, box on `lost`, `op.exec` "none", receipt dropped, P's remote build kept when unopened, nested when opened); the worker uses it instead of `MarkStuck`, which stays the live target's rule. | Task 1 (`TestRelocationExhaustedTargetIsFencedIntoAStuckPinning`), Task 5 (`TestWorkerFencesAnExhaustedTargetThatDiedAfterItsLocalInstall`); ruling 23 | | N6 | accepted | `PinAfterRelease` from `releasing` requires the acknowledgement barrier, so a target replaced after the source's release acknowledges before the operation leaves the phase. | Task 1 (`TestRelocationReplacementInReleasingAcknowledgesBeforePinning`) | ## Rev 3: round 2 dispositions Astra's round 2 (`CODEX-PLAN-REGION-HA-1BN-R2.md`, against `ed992c46`) left seven round 1 findings partly resolved and added seven. All fourteen are accepted; none is declined. Every un-gated change was dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds, targeted tests) and each new guard removed in turn (11 removals, each failing its test; listed in the Self-review). | # | Disposition | What changed | Where | |---|---|---|---| | R1-2 | accepted | The build's authority is checked at every write, not once: `authorityStore` wraps the versioned store so each `PutIfAbsent` and `CopyVersion` re-reads the claim and the host-guard record (`hostguard.Capture` at `install_stamp`) first; the lease's loss cancels the rest. | Task 8b | | R1-4 | accepted | `AutomaticLoss` retains the holder's incarnation and stamp, the box's last writer session whether or not it is still the writer (its last admission record on the holder, `LastAdmission`), so a source fenced after the target's admission keeps its session and consequences. | Task 1 (`TestRelocationLossRetainsAFormerWritersSession`); R5 | | R1-5 | accepted | The pinner retains the exact manifest bytes in the control bucket (`RetainedManifestKey`, create-only) before the pin; `CopyPinnedManifest` writes P from that copy, validated by digest, sequence and origins (`hostfiles.ValidateManifest`), and never reads the source again. A late source PUT changes nothing. | Task 5 (`TestCopyPinnedManifestEnforcesThePin`), ruling 21 | | R1-9 | accepted | `ReleasePositions.Recorded` matches the installation and session, not the controller, and returns the request the flush was made under; after a restart `Release` asks root under that identity (so root finds the stop's durable receipt), offers no drain of its own, and pairs the replayed receipt with the recorded positions. A stop root never completed cannot be finished by a new process: the release deadline makes it stuck. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions/after_a_restart…`, `TestReleasePositionsFileBindsTheExactStop`) | | R1-13 | accepted | A later probe attempt whose predecessor generation holds no LTX file (`genselect.ErrEmpty`) is an ordinary empty cold admission: `Empty` set, no baseline, 6b's own empty authority. | Task 8c | | R1-14 | accepted | See new finding 2. | Task 9 | | R1-16 | accepted | Exhaustion is checked before a replacement is looked for, in both paths: with the replacements used the operation is stuck now, never retried on "no eligible box". | Task 5 (`TestWorkerMarksExhaustionStuckBeforeLookingForASpare`) | | N1 | accepted | `Prepared` is set only by the two branches that open P; the retry branch resets `Prepared` and `Empty` and selects its baseline as an ordinary cold wake, so its marker follows the first durable root. | Task 8c | | N2 | accepted | The adapter test separates "no row to mint" (a box with no role and no row: `ErrChanged`, nothing written) from "an existing unexplained row to keep" (held, never served, never deleted), and asserts `hold_op`, not a status the implementation does not write. | Task 9 | | N3 | accepted | `ProcessDescriptors` reads only `/proc/` (its own process; another process's unreadable table does not matter); the complete scan is root's: `RelocationEffects.Observe` is a provider, production asks root over the instance-control socket (`CollectWritersStopped`, R3's server half), and an agent-side scan fails closed on the first unreadable table. | Task 4 (`TestProcessDescriptorsNamesOneProcessesHolders`), Task 7, Task 8a, R3 | | N4 | accepted | A `RowReady` row whose `hold_op` names an operation the claim no longer runs has its hold cleared under the activation's verified ownership (`SetClaimHold(owner, old, "")`); a hold of the running operation is never touched. | Task 9 (`TestReconcileAdapterClearsTheHoldAfterCompletion`), ruling 20 | | N5 | accepted | `Effects.Retained` and `Finish`; `Participant.Sweep` finishes every retained controller whose operation the claim no longer runs with this box as a participant, once per pass; the relocation controller's `Cleanup` hook is scoped to the relocation's own partial work and is a no-op once this box has admitted under the operation or serves the host, so finishing never stops the serving writer. | Task 6 (`TestAParticipantFinishesItsControllerOnceTheOperationIsOver`), Task 7 (`Finish` in `TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime`), Task 8a, Task 10; ruling 20 | | N6 | accepted | `MaxProbeAttempts = 3`; `BeginProbeAttempt` counts the attempt in the claim before it is made (per box and operation, so a replacement starts at zero); the participant enters `probe-failed` on the fourth. Durable across restarts and lost responses. | Task 1 (`TestRelocationProbeAttemptsAreBounded`), Task 6 (`TestAParticipantGivesUpAfterThreeProbeAttempts`); ruling 19 | | N7 | accepted | A qualified discharge completes each `park-local` with `CompleteObligation`'s own semantics, so a discovery operation's final chore marks discovery complete exactly as a retirement would. | Task 1 (`TestRelocationDischargeCompletesDiscoveryLikeARetirement`) | Astra's readings of rulings 1 to 4, 6 to 8 and 10 to 17 stand; rulings 5, 9 and 18 are completed by R1-5, N6 and R1-4 above. ## Rev 2: round 1 dispositions Astra's round 1 (`CODEX-PLAN-REGION-HA-1BN-R1.md`, against `eb5e9051`) returned sixteen findings, "not ready". Each was checked against the tree at 3407d114 and the cited plans before anything changed. Fifteen are accepted in full and one in part (8: the un-gated half is built, the production body is a request to the claim session). None is declined. Every un-gated change below was dry-run again: applied to a clean 3407d114, built, vetted for darwin and windows, cross-built, tested, and each new guard removed in turn (25 removals, each failing its test; listed in the Self-review). | # | Disposition | What changed | Where | |---|---|---|---| | 1 | accepted | Task 8 is three tasks against the declared types: `moves.SealedObject{Key, VersionID, Size}`, `moves.InstallManifest` (on main), `moves.BuildInputs` with `Now` from this plan's config, 6b's `localStopIdentity`, `Wake.Producers`, `CollectClaimStop`. What no plan supplies is now this plan's code (`RelocationWakeInputs`, the install journal, `PreparedFromInstallManifest` reading `moves.InstallManifestKey`) or a named request (R1 the probe dispatch 6b refuses today, R2, R3). | Tasks 8a, 8b, 8c; Requests | | 2 | accepted | The worker holds the replica lease on `holder.location` from pin selection to the journal CAS (`Worker.Leases`, `pinUnderLease`); a lease lost before the CAS refuses. The build runs under the same lease, re-reads the claim before each durable step and is cancelled when the lease is lost. | Task 5 (`TestWorkerDrivesADrain`'s order, `TestWorkerRefusesToJournalAPinAfterTheLeaseIsLost`); Task 8b (`TestRelocationBuildStopsWhenAuthorityChanges`) | | 3 | accepted | `ReleaseFlushFiles`: `Open`, `StageWake(ExportWake)`, `StagePlatform(EnvItem)`, `SyncDir`, `Commit` as one revision; `hostfiles.EnvItem` added. | Task 7 (`TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv`) | | 4 | accepted | D4 retention outside the operation: `Claim.EvidenceRefs []AutomaticLoss` (fence reference, observation time, box, role, last writer session, discharge verdict, timers `unknown`, reporting pending) and the loss id in `LossRefs`, written by `fenceBox` in the consuming CAS with a deterministic id; both survive `CompleteRelocation`, `free` and holder replacement. `Claim.Validate` checks them. | Task 1 (the failover test's completion checks, `TestRelocationRetainsEveryFencedBoxAcrossCompletion`); ruling 18 | | 5 | accepted | `FilesPin.Origins` (the admitted chain as pinned); `hostrelocate.CopyPinnedManifest` verifies the source bytes against the pin's digest and sequence, copies with the pinned origins, and refuses a copy whose entries differ from the pinned ones; the build and the verifier use it. | Task 5 (`TestCopyPinnedManifestEnforcesThePin`: a late revision, different bytes at the same sequence, inherited objects); Tasks 8b | | 6 | accepted | `PinObject.N` is gone. `ClaimGen` names the observation; the build takes `pin.DB.MaxTXID`; the empty branch is `DB == nil`. | Task 5, Task 8b | | 7 | accepted | `ToInstalled` requires `FilesSeq == pin.FilesSeq` and a `FilesManifestSHA256` always (an empty pin included); the build writes P's files manifest before the database; the verifier reads both back. | Task 1 (`TestRelocationDrainEndToEnd`, `TestRelocationEmptyPinNeedsItsEvidence`); Task 8b (`TestPreparedRefusesAPartialBuild`) | | 8 | partly | `RelocationEffects.StopWriters` runs first in the drain, before the observation; its production body is request R2, because 6b has no writers-only stop and its `ClaimProducers.DrainClaim` cancels every producer. The live-process test is Task 8a's, gated on R2. | Task 7 (`TestRelocationReleaseRunsStepsOneToThreeInsideTheDrain`); Task 8a; Requests | | 9 | accepted | `ReleasePositions`: the flushed positions are recorded durably, bound to the exact stop request, before the producers close; a replayed receipt is paired with that record or refused. `ReleasePositionsFile` on disk. | Task 7 (`TestRelocationReleaseReplaysOnlyRecordedPositions`, `TestReleasePositionsFileBindsTheExactStop`) | | 10 | accepted | `RelocationControllers` retains controllers under `Lifetime` (the agent's context); `Effects.Ready` runs before every effect; a restarted agent reopens a controller and never re-journals the acknowledgement. | Task 6 (`TestAParticipantChecksItsControllerBeforeEveryEffectAndReopensAfterARestart`); Task 7 (`TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime`); Task 10 | | 11 | accepted | The pin's key names the claim generation it was selected against (`hostclaim.RecoveryPinKey`, `pin-.json`); `JournalPin` binds it, so a pin whose CAS lost is abandoned and the retry selects afresh; every bound is checked before the object is written; the counter is gone. | Task 1, Task 5 (`TestStorePinnerPinsTheNewestFilesManifestInTheChain`, `TestStorePinnerChecksTheBoundsBeforeItWrites`); ruling 16 | | 12 | accepted | `Worker.Eligible` (`ExecutorChooser.Permits`: the complete eligible set, so stale edges, a missing capability or key, a draining box refuse) and `Worker.Relocatable` (the record's shape) at `Start`; a sidecar family is refused outright. | Task 2 (`TestExecutorChooserPermitsOnlyTheEligibleSet`), Task 5 (`TestWorkerStartRefusesAnIneligibleTargetAndANonRelocatableHost`), Task 10 | | 13 | accepted, narrowed | An empty pin is evidenced only by a journaled release at `final_txid` 0 or a holder that never admitted a writer; any other `ErrEmpty` refuses (ruling 15). A drain of a session-0 host is refused at `Start` (ruling 11 amended, request R4). `FlushDB` reports 0 on `ErrEmpty` after the sync; the verifier requires only the files manifest for an empty pin; the first probe is a cold admission into an empty generation. | Task 1 (`TestRelocationEmptyPinNeedsItsEvidence`), Task 5 (`TestStorePinnerPinsEmptyOnlyWithEvidence`, `TestWorkerRefusesToDrainAHostThatNeverHadAWriter`), Tasks 8a to 8c | | 14 | accepted | `Policy.Check` gains the `reconcile` action for a relocating participant (un-gated); Task 9 is rewritten around three authorities: activation only for the executor, `WithLocal` plus a conditional `SetClaimHold` for a kept row, and no mint for a target before `installing` (ruling 17). Its test drives 6b's `ReconcileAdapter.Install` for the source and the target in every phase. | Task 3 (`TestPolicyLetsAHeldParticipantReconcileItsRow`), Task 9 | | 15 | accepted | `Authority.SelectExecutor` returns `selected`; without one `Attempt` runs today's `BeginCreate`. `ProductionAuthority.Executors` nil means no selection; the console's chooser is nil unless `Config.RegionHA` (parsed by `strconv.ParseBool`) is on. | Task 2 (`TestCreateIsTodaysWithoutASelection`, `TestProductionAuthoritySelectsOnlyWithAChooser`, `TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims`) | | 16 | accepted | A live target's exhausted replacement goes through `stuckOn` from `probe-failed`; `ReleaseDeadline` marks a release never journaled stuck in `releasing` and `probe-failed` (ruling 14); `relocationAdmission` checks `stuck` before any phase. | Task 1 (`a stuck source does not wake`), Task 5 (`TestWorkerMarksALiveTargetsThirdReplacementStuck`, `TestWorkerMarksAnOverdueReleaseStuck`) | Astra's readings of rulings 1 to 13 stand; ruling 5's exact-files enforcement is finding 5's fix, and ruling 9's promised behaviour is finding 16's. ## Requests to the claim and moves sessions Interfaces this plan needs that no merged or declared plan supplies. Each gated task names the request it waits for; nothing here is invented silently. | # | To | Request | Waited for by | |---|---|---|---| | R1 | claim 6b | `ClaimWakeAssembly.Build`'s runtime refuses every probe and every non-`serving` claim with a `ClaimDependencyError` naming moves C1. Region HA supplies `host.RelocationWakeInputs` (Task 8c) for `relocating/probing` where `op.exec` is this box. Ask: dispatch by `Op.Kind` to that provider in `Prerequisites`, `Admit` and `Prepare`, allowing `p.Probe` there, and expose one probe attempt as `(*ClaimWakeAssembly).Probe(ctx, obs, StartPolicy) (session uint64, passed bool, err error)` returning the attempt's exact health. *Rev 5:* for that attempt the runtime selects its spawn work as action `"probe"` (`source.SelectAt(host, "probe", captured)`), the action Task 3's policy gives probe authority to; ordinary `"spawn"` work for a relocation target stays refused. *Rev 15 (round 14 finding 3):* claim §4 and Task 1's phase table also let the **source** admit, cold, in `starting` and `releasing`, but 6b's `runtime.Prerequisites` refuses every non-serving claim (`p.Probe || o.Claim.Op.Kind != KindServing`, 6b's `ClaimDependencyError` naming moves C1). Ask: dispatch a relocating claim's ordinary wake to the existing wake inputs when this box is `op.exec` and the source, in `starting` or `releasing`, with its admission fence open (Task 3's `WakePhase` and `RelocationPolicy`); refuse it once the fence closes or the claim no longer names this box. A relocation target's ordinary wake stays refused; its only wake is the probe. *Rev 15 (round 14 finding 1):* the probe's first admission consumes P's prepared manifest (sequence 0) as the session publisher's baseline, by the digest `prepared_authority.files_manifest_sha256` names (the store has no version-id read, ruling 5). The publisher starts at sequence 1. | Task 8c | | R2 | claim 6b | A writers-only stop on the admitted attempt, `(*claimWakeAttempt).StopWriters(ctx) (ClaimWritersDrained, error)`: close admission, settle spawn reservations, cancel and join the app's role launch, drain the schedule runner's in-flight deliveries for the host, wait for the host's cgroup to empty; keep the replicator and the files publisher open; idempotent once stopped; report the three drained facts, the cgroup and the database paths. `ClaimProducers.DrainClaim` stays the full drain. | Task 8a (finding 8) | | R3 | claim 6b | Two halves of L10's observation, split at the privilege boundary (rev 3). Agent side: record the replicator's descriptors when it opens, `stopevidence.ProcessDescriptors(ctx, "/proc", os.Getpid(), dbPaths)` immediately before and after `held.Replicate` (it reads only `/proc/self`), the difference kept on the attempt as `replicatorFDs` with `claimStopLocal.ProcessStart`. Root side: an authenticated instance-control endpoint `POST /claim/writers-stopped` taking Task 8a's `WritersStoppedRequest` (host, drained facts, database paths, replicator pid, start and descriptors) from the carlos uid, running `stopevidence.WritersStopped` with root's cgroup manager and the complete `/proc` scan, and returning the evidence. The agent cannot read other processes' descriptor tables (`carlos-edge.service`: uid carlos, `CAP_NET_BIND_SERVICE` only); an agent-side scan fails closed. | Task 8a | | R4 | claim Plan 5 | The positive no-process witness for an installation with `writer_session = 0` (6b's `collectCapturedStop` already names it). Until it lands a drain of a never-woken host is refused at `Start` (ruling 11); a failover relocates it. | Task 5's ruling 11 | | R5 | claim spec, console | Adopt `evidence_refs` as the claim field for D3's "enrollment-qualified fence references": this plan's `hostclaim.AutomaticLoss` shape, one per (operation, box, fence), with the loss id in `loss_refs`, carrying the fence reference and observation time, the holder's incarnation and stamp, the box's role, its last writer session and that session's admission record (coverage, declarations), the discharge verdict and timer knowledge (rev 3: complete retained consequences), and the regional obligations the fence passed to the region (rev 6: `reassigned`, D4's `reassign_regional`). The console's claim-recovery pass materializes each into `control/host-loss/.json` (D4's automatic variant, `write_sources` from `last_admission`, timers `unknown` or `partial`) and sets `reported`; until then the claim keeps it. | After this plan | | R6 | claim spec | What evidences an empty generation on a failover of a host whose writer session is above 0 and whose listing holds no LTX file. Ruling 15 refuses it (the operation waits for an operator); if the claim session wants automatic recovery there, name the evidence. | ruling 15 | | R8 | claim 6b | A hook on `ClaimAssembly`, `Extend func(*ClaimAssembly, hostbox.Actor)`, called by `Build` after `a.Identity.Verify` and before the `WorkSource` captures `Policy.Check` (6b's Build at the line that sets `source := hostbox.WorkSource{…Policy: a.Policy.Check…}`), so the verified box identity is available. Task 10's `applyRegionHA` is its body: with `CARLOS_REGION_HA` on it sets `a.Policy.Relocation = hostbox.RelocationPolicy(actor.Box)` and `a.Reconcile.Reconciler.FenceClosed` (over `Guard.AdmissionFenced`); off, it touches nothing. (Rev 4's insertion after `OpenProductionClaimAssembly` was before the box was known.) | Task 10 | | R9 | moves Plan 1 Task 7b | A relocation-aware held-release relay: `release.ServeHeldStageRequests`' authorizer accepts a request bound to a `relocating` operation (`Op.Kind == relocating`, `Op.Exec == req.Box`, `op.journal.install_stamp` equal to the request's stamp, and the pinned release equal to the version the local install recorded), beside the `moving`/`landing` requests it accepts today. Without it `MaterializeHeld` returns `ErrStageBlocked` under the edge unit's `ProtectSystem=strict` and the local install cannot complete. | Task 8c | | R10 | claim 6b (and the activator's owner) | The reservation-bearing extension of main's isolated root launch protocol, which 6b's own external completion gate names as required "before any tenant shape can run with claims enabled" and as "not supplied by the external owner adapters" (6b rev 5, "External completion gates"). Until it exists, `ClaimRoleAssembly.Preflight` returns `activator.ErrClaimIsolationUnavailable` for every launch (6b, `Preflight`), and the runtime's `Prerequisites` calls `Roles.Preflight` before any dispatch (6b, `runtime.Prerequisites`). Even with R1 every production probe would therefore be refused. The ask: the protocol binds each supported role launch to the durable claim reservation and preserves main's namespace isolation; `Preflight` admits a relocation probe's `"probe"` work through it, bound to `op.exec`, the operation and `install_stamp`; without the protocol, `Preflight` refuses before any probe attempt is counted. | Task 8c, Task 10 | | L21 | region HA spec (Paul) | **Open: the durability of the always-continuous items between an app minting them and their first publish.** §5.6 has every cold admission copy its manifest before the app starts, and H6 makes `instance.key`, `report.token` and the env files continuous. Continuous is not synchronous: an app can mint `instance.key`, write database rows sealed with it, and replicate them before the publisher's first manifest with the key lands. A box terminated in that window leaves a nonempty database whose key exists nowhere. Rev 18's pinner refuses that case and the operation goes `stuck` for an operator (ruling 25), but the database is not recoverable by any relocation. Closing the window needs an upstream durability contract, and there are two candidates, neither chosen here: the platform pre-provisions `instance.key` (and `report.token`) into the files directory and publishes them before the app's first start, or the replicator holds its first upload until the first manifest carrying the key has landed. **Decided 2026-10-09 (Paul): the platform pre-provisions the key.** At creation the platform generates it on the box (CSPRNG), writes it create-only (never overwriting) into the instance's overlay env file as `CARLOS_INSTANCE_KEY`, and publishes it before the app's first start. That file is platform-managed and always continuous. Existing instances get one forced publish before region HA is enabled. The app side is a framework contract, requested as rastrillo/rastrillo #66: detect the platform, use the app's own existing key first and the platform's only when it has none, never mint on the platform, and keep today's behaviour off it. The platform side is built only once #66 agrees the contract. The pinner's refusal (ruling 25) stays as the backstop. Residual: an app that later replaces its key reopens the window for the new key. | ruling 25 | | R7 | moves Plan 2 | Task 8's `moves.InstallManifestKey`, `SnapshotKey`, `AnchorKey` as declared, and `BuildInputs.Source` taking the pinned generation's full keys as `genselect` lists them (level and TXID range parsed from the key's tail by `SealedClient`). | Task 8b | ## Rulings made while planning Each was a conflict between the authorities or a gap in them. Each says what it costs if it is wrong. 1. **Relocation lives in `internal/hostclaim`.** Main already has `KindRelocating`, its phase validator and `ConvertMovingToRelocating` there, and Plan 6b's boundary table gives "`relocating`, released coverage, once-selected executor and two-box witnesses" to region HA. One implementation per concept: no second package grows claim transitions. *Cost if wrong:* the claim session moves the functions; the names stay. 2. **The relocation release's stop request names the obligation `relocation-release`.** Root's `FenceAndStop` requires an obligation component, and 6b's `collectCapturedStop` builds only retirement stops (it looks for this box's `park-local` under the operation). A release retires nothing. `host.StopReceiptReleases` accepts only receipts carrying `relocation-release`, so a retirement stop can never be offered as a release, nor the reverse. *Cost if wrong:* a one-line constant, and the 6b session may prefer another literal. 3. **Discharge stays off until the roster records an enrollment time.** `boxvolumes.Discharge` needs the enrollment time (§24 item 4) and `hostactors.Box` carries none. The production `Discharge` provider returns "the roster records no enrollment time", so a fenced box's `park-local` chores stay open for an operator. H19: this costs cleanup, never recovery. *Ask of the claim session:* add `enrolled_at` to box entries (D7); then the provider reads `boxvolumes.List` and calls `boxvolumes.Discharge`. 4. **A box-started relocation needs claim §5.1's app-admission registration**, with a `{box, enrollment, controller}` owner. Main has no `apps///admission.json` registry yet. `hostrelocate.Worker` takes an `Admission` interface and refuses to start without one (`ErrUnsupported`). Task 10 is gated on claim Plan 5's app gate as well as 6b. *Cost if wrong:* none until that lands; nothing starts. 5. **The pinned recovery set is an immutable object** at `control/relocations///pin-.json` (rev 2, ruling 16: the generation it was selected against), written create-only and named in the journal by key and SHA-256 (spec §18's first correction: the store has no version-id read, so content is identified by digest). The files pin names its manifest by prefix, sequence and digest. *Cost if wrong:* a deny statement for that prefix in `deployment-buckets` is a follow-up; a rewritten pin is already caught, because every reader checks the journaled digest. 6. **`installed` is the worker's CAS**, made once a `Prepared` verifier reads P's install manifest complete (§5.8's phase table). The target enters `probing` itself. *Cost if wrong:* who makes one CAS; the claim rules are unchanged. 7. **A converted move names its target in `pinning`** (`hostclaim.NameTarget`), before `installing`, so the target acknowledges (D6) before it is handed `op.exec` (moves §7.10: "the replacement box `choose()` names when the relocation leaves `pinning`"). 8. **The final TXID N is read from the remote after `SyncAndWait` returns with every writer stopped**, as the newest level-0 `MaxTXID` (§5.8 release step 2). With writers stopped and the sync acknowledged, the stopped database's position and that listing agree; a sync error is returned, never swallowed (claim P6-D4). 9. **`stuck` is a journal entry, not a phase.** The claim says exhaustion "records `stuck` without resetting the count"; `validateRelocating` lists no `stuck` phase. `MarkStuck` writes `journal.stuck` and every relocation transition refuses afterwards; it refuses between `installed` and a probe. 10. **The probe deadline starts at the `probing` CAS** (`journal.probing_since`), and the worker moves a live target to `probe-failed` 10 minutes later. 11. **A host whose source never had a writer** (session 0, `writer_box` empty) is relocated with coverage `""`, which declares nothing (claim §6.4a: "a created incarnation's first admission … declares nothing"), never `released`. *Rev 2:* only by failover. A drain of such a host is refused at `Worker.Start`: root refuses stop evidence for writer session 0 (`validateRequest`), and 6b's `collectCapturedStop` waits for claim Plan 5's no-process witness (R4). Its failover pin is an evidenced empty one (ruling 15). *Cost if wrong:* a box with never-woken hosts cannot be drained until R4 lands; nothing is lost. 12. **L18's owner.** The coordination post on #236 says moves owns `internal/sim/hostlife`; claim rev 6.30 §1.1 calls the two-box model "region HA's own … as part of region HA's plan"; the moves conflicts note defers it to "a joint round with the region HA session". This plan writes no model and ends with a merge gate that names the cases; whichever session builds the model, relocation does not merge until that round passes for a one-box and a two-box region. 13. **Moves revision.** The brief cites moves rev 7.20; the branch reads 7.22. Rev 7.21 and 7.22 bind the prepared files manifest to P (R18-1, R19-1). *Corrected in rev 15 (round 14 finding 1):* until rev 14 this plan did not follow that binding, because it copied the source manifest's sequence into P. Under claim rev 6.30 (R18-1) and moves rev 7.22 §7.6, P's prepared manifest is create-only at sequence 0 and bound to `{location, P, op id, installation stamp}`. The first admission consumes it by exact version, and the session publisher starts at sequence 1. The source pin's sequence is provenance only (`PreparedAuthority.SourceFilesSeq`). `CopyPinnedManifest` writes sequence 0, and `ToInstalled` requires it and binds the stamp. Nothing in L14 to L17 changed against it. 14. **The release deadline is 10 minutes** (`hostrelocate.ReleaseDeadline`, rev 2). §5.8 says a flush that never reaches N, and a `probe-failed` target that cannot finish its release, go to `stuck`, and names no bound. `ToReleasing` and `ToProbeFailed` journal when the phase began; a worker marks the operation stuck once that long has passed with no release journaled. The host stays where it is: nothing moves away from a live box without its stop evidence. *Cost if wrong:* a number; the stuck state and the operator path are the claim's. 15. **An empty pin needs evidence a writer can give** (rev 2; claim §6.4a "Empty bootstrap": no LTX file exists until the first commit, so a listing that finds nothing is a definite observation but says nothing about who wrote). A pin is `Empty` only when the journaled release reports `final_txid` 0 (the stopped writer flushed and found nothing), or the holder never admitted a writer (session 0, failover). A failover of a host whose session is above 0 with an empty listing refuses and waits for an operator; `JournalPin` enforces it, the pinner never writes it. *Cost if wrong:* that one case recovers by hand until R6 names its evidence. 16. **A pin is bound to the claim generation it was selected against** (rev 2; B2-3). Its key is `control/relocations///pin-.json` and `JournalPin` requires `gen` to be the observed claim's. A pin whose journal CAS lost is abandoned where it lies (never referenced, cheap, reaped by the storage follow-up of ruling 5); the next pass selects afresh at the new generation. Every bound `JournalPin` checks (the release's positions, the failover window) is checked before the object is written, so a doomed pin is never published. *Cost if wrong:* abandoned objects under a prefix nothing reads. 17. **Reconcile has three authorities for a relocation row** (rev 2; §5.3, 6b's `ClaimOwnership.Valid` needs a stamp). The executor (the source until its fence closes, the target from `installing` on) activates its own installation stamp and mints or keeps its row through the guarded path. A kept row of a box that is no longer the executor (the source after its release, a replaced target) gets its hold through `registry.SetClaimHold` under the flock, conditional on the row's recorded ownership: a hold grants nothing, so it needs no installation authority. A target before `installing` has no installation and mints nothing; the edges file already withholds the host from a box that is not `op.exec` (§24). *Cost if wrong:* a request that reaches a pre-installation target meets the box's unknown-host answer instead of a hold, for the seconds between `starting` and `installing`. 19. **Probe attempts are counted before they are made** (rev 3; §5.8 "limits": at most three, then `probe-failed`). `BeginProbeAttempt` is one CAS per attempt, keyed by box, so a crash, a lost response or a restart never buys a fourth and a replacement target starts at zero; the participant enters `probe-failed` when refused. *Cost if wrong:* an attempt lost to a crash before its admission still counts; the bound is the spec's. 20. **A relocation's execution controller is finished when its participation ends** (rev 3). `hostexec` keeps the host's lifetime lock until `Close`, so the participant sweeps its retained controllers once per pass and closes every one whose operation the claim no longer runs with this box as a participant. The relocation controller's `Cleanup` hook is scoped: it retires only the relocation's own partial installation (an unopened P's local install) and does nothing once this box has admitted a writer under the operation or serves the host, so closing it after a successful relocation stops nothing. *Rev 6:* a durable record is cleared only straight after that scoped cleanup succeeds in the same call (`Finish` repeats it after `Close`, and on a restart), because hostexec reports a cleanup failure from `Close` once and nil afterwards; the participation is journaled before its controller opens, so a restart finds every record. The serving writer belongs to the wake assembly's own controller from its admission on. A completed operation's `hold_op` on the serving executor's row is cleared by reconcile under the activation's verified ownership; a running operation's hold is never touched. *Cost if wrong:* a scope rule in one hook; the lock release is hostexec's. 21. **The pinned files manifest is retained by the pinner** (rev 3; B2-4). Before the pin object exists, the exact manifest bytes are written create-only to `control/relocations///manifest-.json`, and P's manifest is written from that copy, never from the source prefix again: a late revision of the source changes nothing an admitted pin reads. *Cost if wrong:* one small object per pin under the prefix ruling 5's storage follow-up covers. 22. **The writers-stopped observation is root's** (rev 3; L10 "a complete privileged descriptor scan"). The agent, uid carlos with `CAP_NET_BIND_SERVICE` alone, can read only its own descriptor table; it records the replicator's descriptors from `/proc/self` at open and asks root, over the authenticated instance-control socket, to make the complete scan with its cgroup manager (R3). An agent-side scan fails closed on the first unreadable table. *Cost if wrong:* one more root endpoint, beside the stop collector's. 23. **An exhausted target that dies is fenced into a stuck pinning** (rev 4; §5.8 "The bound": "never between `installed` and a probe" is a live target's safe point). *Rev 9:* that holds only when the dead target is `op.exec`, or `op.exec` is already "none". While the live source is `op.exec` (`starting`, `releasing`), the CAS consumes the target's fence and records `stuck`. It keeps the source's execution identity, phase and release evidence, because a target's fence is no evidence against the source (claim §4), and root's stop needs the source to remain `op.exec`. Otherwise `FenceExhaustedTarget` consumes the fence in one CAS, retains the loss, revokes the dead installation's authority and leaves the operation stuck at `pinning` with its pin, and P's remote build when nothing opened it, for the operator's replacement; nothing chooses a third target. *Cost if wrong:* the stuck state's shape; the operator path is the claim's. 24. **Two stages** (controller ruling, rev 5). Rounds 3 and 4 found most of their defects in the gated tasks' integration with 6b, itself unmerged and changing; polishing code against a moving plan does not converge. Stage A (Tasks 1 to 7, 9a) is build-ready: compiled, dry-run, every finding fixed in code. Stage B (8a, 8b, 8c, 9, 10) keeps its design and contracts; its code is rewritten against the real 6b code after it merges and re-reviewed then. Stage B findings are answered at the contract level: what must happen, which request covers it, which test proves it. The "Stage B re-plan" step lists every stage B obligation from rounds 1 to 4, so none is lost. *Cost if wrong:* a second review round on stage B after 6b lands; the alternative is tuning against text that will not survive. 25. **An empty files pin needs positive evidence; otherwise the operation is stuck** (rev 17, two-box model finding F2). H6 makes `instance.key`, `report.token` and `.env` continuous on every plan, and §5.6 has every cold admission write its manifest create-only before the app starts. So a holder with an admitted writer and no manifest anywhere in its chain cannot be shown never to have minted them. Examples are an adopted legacy host whose files never published, or a first manifest that never landed. Installing an empty set there would mint a fresh key over data sealed with the old one. `StorePinner` pins an empty files set only when no writer was ever admitted (session 0), or when the stopped writer's release flush, which publishes every file whatever the setting, reported `manifest_seq` 0. Otherwise it returns `ErrFilesUnevidenced`, and the worker marks the operation `stuck` for an operator, as ruling 15 and R6 do for the database. The alternative, requiring the first manifest that carries `instance.key` to land before the first admission, cannot be built here. The key is written by the app, not the platform, and §5.6 already orders the admission's manifest copy before the app starts. That ordering is a stage B obligation on 6b's cold admission (the re-plan names its test). The window between the app minting a key and the publisher's next publish is §5.6's continuous-sync bound, raised for the joint round. *Cost if wrong:* a host with no manifest and no key is stuck until an operator resolves it, rather than relocated empty. *Rev 18 (round 17 finding 1):* an existing manifest is not evidence either. When a writer was admitted and the pinned database is nonempty, the pinned manifest must carry a live `instance.key`, the minimum of H6's always-continuous items. A manifest copied before the app started, with the key minted after and never published, refuses the pin, and the operation is `stuck`. The durability window itself is not closed by this plan. It is the open request L21, for Paul to choose between the platform pre-provisioning the key and the replicator holding until it lands. *Cost if wrong (rev 18):* an app that never writes `instance.key` and has a nonempty database is stuck on every relocation until an operator resolves it; H6 names the key as platform-managed on every plan, so such an app should not exist. 26. **A drain's probe-failed with no eligible spare is stuck after `ReleaseDeadline`, never sent back to the source** (rev 17, finding F1). §5.8 excludes `to` from the choice after a failed probe, and a relocation's target is never its source (the start CAS, `replaceTargetSlot`). Letting the live source be the "replacement" would put a fresh install of P' into the same host paths that the source's own `park-local` chore retires once the operation completes (§5.8, "What a relocation leaves behind"). The same box would then hold the retiring installation and the new one under one name. So the worker waits for a spare and, `ReleaseDeadline` after `probe_failed_since`, marks the operation `stuck`. This follows the spec's own bound ("It alerts, and the op waits for `carlos ops host resolve`"). The operator's resolve is the claim's: it may return the host to the source, as the model's ruling 13 does. *Cost if wrong:* in a two-box region a failed drain waits ten minutes plus an operator instead of an automatic return. 27. **Every abandoned prepared generation is parked by the operation that built it** (rev 17, finding F3). §5.8 says "an abandoned `P` is parked by the at-rest sweep". This plan adds an earlier owner: `install` journals every unopened P it replaces (`abandoned_prepared`), and `CompleteRelocation` queues a `park-generation` chore for each in the region, beside its `park-local` chores. The at-rest sweep stays the backstop for an operation that never completes. A P that any admission opened is part of the writer lineage and is never on the list. *Cost if wrong:* one chore per abandoned build. Its objects are outside every live set (a prepared manifest names the source's objects, not P's), so parking them retires nothing live. 18. **D4's automatic loss is retained as `evidence_refs`** (rev 2; claim §7.3 D3 "enrollment-qualified fence references in `evidence_refs` and stable loss identities in `loss_refs`"). `hostclaim.AutomaticLoss` holds the fence reference, its observation time, the box, enrollment and role, the box's last writer session, the discharge verdict, timer knowledge `unknown` and whether the console has reported it; the id is deterministic over (operation, box, fence) so a replayed transition dedupes. It lives beside `loss_refs`, outside `op`, and so survives the serving CAS. The console's materialization is after this plan (R5). *Cost if wrong:* a field name; the retention is what D4 requires. ## Why this is 1b-next, and what it leaves out Plan 1b built every part of rev 5.4 that needed none of L1 to L20. Since then claim Plans 1 to 8b and 5 landed on main (3407d114): the roster (`hostactors.Directory`), the fence writer (`hostfence.TerminationWorker`), the fence reader for claims (`hostclaim.VerifyFence`, `hostfence.ClaimFenceAuthority`), the execution controller (`hostexec`), stop receipts (`host.ClaimStopCoordinator`), the box-started census (`Directory.CensusBoxRelocation`) and H14's start check (`Directory.CheckBoxStart`). This plan uses all of them and duplicates none. Left out, with where it goes: - **The placer loop that decides what to relocate** (spec 3): reading the termination queue, relocating every host off a dead box in bounded parallel, rebalancing. This plan's worker starts a relocation when asked and replays any it finds; Task 10's pass only replays. - **Automatic enrollment and the cross-region confirmer** (spec 4, H16, H17). - **The console's copy of a fence into a D4 loss record and the roster's retirement of the box** (claim D4 "Automatic loss persistence"). The fence CAS keeps the reference and the discharge verdict in the operation journal and puts the box on `lost`; materializing the loss record is the console's, and is listed under "After this plan". - **Storage policy for `control/relocations/`** (ruling 5). ## Global Constraints - **Dormant.** Nothing changes on a box or console where `CARLOS_REGION_HA` is unset or not true (`strconv.ParseBool`), and it is read from `/etc/carlos/host.env` by `regionHAEnabled` (`internal/host/boxrecord.go`). With one box per region every rule is a no-op (spec §4, §8 step 2). - **Activation gate (claim §1.1, L18).** "`relocating` and the rules it depends on (L1, L9 to L13) must not be enabled until region HA's own two-box-per-region model (L18) … validate[s] them." Relocation may not merge until Task 11's joint round passes. - **One implementation per concept; main's wins (#236).** Do not write a termination-fence writer (`hostfence.TerminationWorker` writes `control/box-fences/.json` using `termfence.Verify`). Do not write a roster reader (`hostactors.Directory`). Read fences only through `hostclaim.VerifyFence` (in claim CASes) and `hostactors.ValidateTerminationFence` (before them). - **H19, verbatim:** "The failover fence rests on the `terminated` read alone; disk evidence only decides discharge of local chores." No volume record gates a fence, a relocation step or an admission. - **Absence is never evidence (§5.9).** A fence that cannot be read or does not validate is an error, never "not fenced"; a box missing from the roster refuses. - **Every phase change is one claim CAS conditional on `op.id`, the phase and the ETag (claim C2).** Two workers racing make the same CAS and one gets `hostclaim.ErrConflict`. - **Exact values.** Failover pin delay **30 seconds** after the fence's `observed_at` (`hostclaim.FailoverPinDelay`); probe deadline **10 minutes** (`hostrelocate.ProbeDeadline`); at most **2** target replacements per operation (`hostclaim.MaxTargetReplacements`); reconcile's kept-row sweep **10 minutes** (H9); release deadline **10 minutes** (`hostrelocate.ReleaseDeadline`, ruling 14); probe attempts **3** per target installation (`hostclaim.MaxProbeAttempts`, ruling 19); create intent keys **`selected_executor`** and **`executor_enrollment`** (claim P5-D6); journal fence key **`fence.`** holding `{"enrollment","key","sha256"}` (claim §7.3); release obligation **`relocation-release`**; pin objects **`control/relocations///pin-.json`** (`hostclaim.RecoveryPinKey`, ruling 16); loss ids **`auto-<32 hex>`** over (op, box, fence); the retained manifest **`control/relocations///manifest-.json`** (ruling 21); the env platform item **`host.env`** (`hostfiles.EnvItem`). - **Gated tasks.** Tasks 8a, 8b, 8c, 9 and 10 start with a "Gated on" line. Their code is written against the interfaces claim Plan 6b rev 5 declares (`ClaimAssembly`, `ClaimWakeAssembly`, `ClaimWakeInputs`, `CollectClaimStop`, `localStopIdentity`, `hostbox.ReconcileAdapter`, `registry.SetClaimHold`), moves Phase 1 Plan 2 Task 8 declares (`moves.BuildGeneration`, `BuildInputs`, `SealedObject`, the key helpers) and moves Plan 1 Task 7b declares (`RenderHeld`, `MaterializeHeld`), plus the requests R1 to R3 and R7. They were not compiled. When 6b and moves land, re-read their merged code and bind the same contracts to the names they actually have; the un-gated tasks do not change. - **No new module dependencies. Do not add a method to `release.Store`.** Linux-only syscalls in `*_linux.go` with a `!linux` stub; CI cross-compiles darwin, windows and linux/arm64. - **Copy.** No user-facing copy. Errors and log lines are operator-facing and follow the surrounding style. If a task finds it needs console or CLI text, stop: it waits for Paul's copy review. - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. - **Gate before each commit:** `go build ./... && go vet .// && gofmt -l ` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 `. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. - **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one, rev 11's four, rev 12's four, rev 13's four, rev 14's ten killed mutations of the lifecycle property, rev 15's twelve, rev 16's, rev 17's eight, rev 18's five and rev 19's two (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. ## Review Focus 1. **`released` coverage naming the wrong predecessor.** A drain's first probe admission declares nothing; if the session it vouches for is not exactly the one whose release was journaled (another box, another session, another generation), a real loss goes unrecorded. Expect refusal. Task 1, `TestRelocationAdmissionRules/released_names_another_predecessor`, `…/released_names_another_predecessor_box`, `…/released_after_P_was_opened`. 2. **P opened twice.** A restarted probe that restores P again loses everything attempt two replicated (B1-1). Expect every attempt after the first to take a fresh generation, and P refused. Task 1, `TestRelocationAdmissionRules/a_third_attempt_reopens_P`. 3. **A failover pinned too early.** Requests S3 accepted from the terminated box can still land; a pin taken before `observed_at` + 30 s widens H3's window. Expect the worker to wait and the claim to refuse. Task 1 `TestRelocationFailoverCitesTheFenceAndDeclaresTheLoss`, Task 5 `TestWorkerPinsNoEarlierThanThirtySecondsAfterTheFence`. 4. **A fence that cannot be read taken as no fence.** A garbled `control/box-fences/` object must stop the drain and the create choice, not let them proceed as if the box were alive. Task 2 `TestExecutorChooserNeverNamesAFencedLostOrUnknownBox/an_unreadable_fence`, Task 5 `TestWorkerTreatsAnUnreadableFenceAsAnErrorNeverAsNoFence`. 5. **A database descriptor in an unshared thread exempted as the replicator's.** Writers-stopped allows only the replicator's identified descriptors; a thread whose table no longer matches the leader's must count as a writer. A matching number and path is not enough: the kernel must say the table and the description are shared (rev 7). Task 4, `TestWritersStoppedRefusesAnyOtherHolder/a_thread's_descriptor_the_leader's_table_does_not_show`, `…/an_unshared_thread_at_the_leader's_number_and_path`, `TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity`. ## File Structure | Path | Responsibility | Task | |---|---|---| | `internal/hostclaim/relocation.go` | The relocation transitions: start (drain, failover), source loss, barrier, release journal, pin, install, local install receipt, probe, serving, target replacement, stuck; fences consumed in the same CAS | 1 | | `internal/hostclaim/generation.go`, `claim.go` | `AdmitWriter`'s relocating phase table and `released` coverage (L1, L9); `evidence_refs`, the retained automatic loss (D4) | 1 | | `internal/hostclaim/create.go` | `ExecutorSelection`, `BeginCreateOn`; `CompleteCreate` never substitutes | 2 | | `internal/console/server.go`, `cmd/console/main.go` | `Config.RegionHA` from `CARLOS_REGION_HA` | 2 | | `internal/hostops/executor.go`, `loads.go`, `operations.go`, `backfill.go`, `backfill_controller.go` | `ExecutorChooser` over the roster, the edges snapshot and claim loads, and `Permits`; `Authority.SelectExecutor` before the create CAS, or today's create without one | 2 | | `internal/hostbox/relocationrows.go`, `wakepolicy.go`, `internal/hostguard/admissionfence.go` | §5.3's row cases; who may wake under `relocating`; the policy's relocation owner, its `reconcile` action and `RelocationPolicy`; the admission-fence reader | 3 | | `internal/stopevidence/writers*.go` | L10's writers-stopped observation; one process's descriptors on the database | 4 | | `internal/hostrelocate/worker.go`, `pin.go`, `internal/hostfiles/manifestcheck.go` | The regional worker: start (eligibility, shape), one replayed CAS per step, the pin under the location lease, the release deadline; the pinned recovery set by claim generation, `ReadPin`, `CopyPinnedManifest` | 5 | | `internal/hostrelocate/participant.go` | A participant box's transitions, ordered over its local effects | 6 | | `internal/host/relocationreleases.go`, `relocationeffects.go`, `relocationflush.go`, `internal/hostfiles/platform.go` | Stop receipts as release evidence; the release sequence inside root's stop collection (stop writers, observe, flush, record, drain); retained controllers; durable positions; the full files flush; `EnvItem` | 7 | | `internal/host/relocationassembly.go` | Gated: the release providers bound to 6b's assembly | 8a | | `internal/host/relocationbuild.go` | Gated: P's build under the lease; the `Prepared` verifier | 8b | | `internal/host/relocationinstall.go`, `relocationwake.go` | Gated: the local install under its stamp; the probe's inputs | 8c | | `internal/hostbox/reconcile.go`, 6b's `reconcile_adapter.go` | Gated: held rows and the 10-minute sweep | 9 | | `internal/console/hostclaimexecutor.go` | The console's create chooser (constructed, not yet used by a door) | 2 | | `internal/host/relocatepass.go` | Gated: the passes, behind `CARLOS_REGION_HA`, and the create door's chooser | 10 | --- ### Task 1: The `relocating` operation in the claim library (L1, L2 to L4, L9, L11) Spec §5.8 and claim §4. Every transition is a `func(*Claim) error` for `hostclaim.Update`, except the four that cite evidence read from storage (`StartRelocation`, `FenceSource`, `PinAfterRelease`, `ReplaceTarget`), which verify it first and then make one CAS, as `ConvertMovingToRelocating` does. A fence is consumed only through `VerifyFence` against the exact bytes its reference names; the CAS that relies on it journals `fence.`, puts the box on `lost`, and records the discharge verdict (L3), discharging the box's `park-local` chores only when the verdict qualifies; the same CAS passes the box's regional work (`park-generation`, `park-replica` owned by `box:`) to `region:`, as `DeclareLost` does (rev 6). A box owing a `park-local` is never named a target (`replaceTargetSlot`, rev 6). **Files:** - Create: `internal/hostclaim/relocation.go` - Modify: `internal/hostclaim/generation.go` (the `AdmitWriter` kind switch, the coverage switch, a new `relocationAdmission`), `internal/hostclaim/claim.go` (`EvidenceRefs`, validated) - Test: `internal/hostclaim/relocation_test.go` **Interfaces:** - Consumes (main): `Update`, `RequireOp`, `VerifyFence`, `FenceRef`, `FenceAuthority`, `AcknowledgeExecution`, `AdmitWriter`, `validBoxReference`, `isHex64`, `KindRelocating`, `validateRelocating`. - Produces: constants `ReasonDrain|ReasonRebalance|ReasonFailover`, `PhaseStarting … PhaseProbeFailed`, `MaxTargetReplacements = 2`, `FailoverPinDelay = 30s`; errors `ErrRelocation`, `ErrReplacementsExceeded`, `ErrAlreadyRecorded`; types `RelocationStart`, `LocalDischarge`, `Release`, `ReleaseAuthority` (`VerifyRelease(ctx, Claim, Release) error`), `RecoveryPin` (with `Empty`), `PreparedAuthority` (with `FilesSeq`, `FilesManifestSHA256`), `LocalInstall`, `Probe`, `Target{Box, Enrollment}`, `AutomaticLoss` (with `Incarnation`, `Stamp`, `LastAdmission`, `Reassigned`) and `Claim.EvidenceRefs`; `OwesLocal(c, box) bool`, `LocalChoreBoxes(c) []string`, `PreparedOpened(c) bool`; `MaxProbeAttempts = 3`, `ErrProbeAttemptsExhausted`; functions `RecoveryPinKey(host, op string, gen uint64) string`, `BeginProbeAttempt(opID, box string)`, `FenceExhaustedTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `RelocationStartGuard(c, region, source, target string, sourceFenced bool) error`, `StartRelocation(ctx, s, observed, RelocationStart, FenceAuthority) (Observed, error)`, `FenceSource(ctx, s, observed, opID, FenceRef, *LocalDischarge, FenceAuthority) (Observed, error)`, `ToReleasing(opID, now time.Time)`, `JournalRelease(opID, Release)`, `PinAfterRelease(ctx, s, observed, opID, ReleaseAuthority, *Target) (Observed, error)`, `PinFailover(opID)`, `JournalPin(opID, RecoveryPin)`, `NameTarget(opID, Target)`, `ToInstalling(opID, Target, prepared string)`, `ToInstalled(opID, PreparedAuthority)`, `RecordLocalInstall(opID, LocalInstall)`, `ToProbing(opID, now time.Time)`, `RecordProbe(opID, Probe)`, `ToProbeFailed(opID, now time.Time)`, `CompleteRelocation(opID, newID func() string)`, `ReplaceTarget(ctx, s, observed, opID, FenceRef, *LocalDischarge, next Target, prepared string, FenceAuthority) (Observed, error)`, `MarkStuck(opID, reason)`. - Journal keys later tasks read: `exec_ack.`, `fence.`, `fence..observed_at`, `discharge.`, `reason`, `releasing_since`, `release`, `release.phase`, `released_by_target`, `pin`, `prepared`, `install_stamp`, `installed_boxes`, `prepared_authority`, `local_install`, `probing_since`, `probe_attempts.`, `probe`, `probe_failed_since`, `replacements`, `excluded_targets`, `stuck`, `prepared_opened` (rev 12). Claim fields: `evidence_refs`, `loss_refs`. - [ ] **Step 1: Write the failing tests** `internal/hostclaim/relocation_test.go`: ```go package hostclaim import ( "context" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "reflect" "slices" "strings" "testing" "time" "github.com/carlosframework/platform/internal/boxid" "github.com/carlosframework/platform/internal/release" "github.com/carlosframework/platform/internal/termfence" ) const ( relSource = "adi-knitting" relTarget = "bri-knitting" relSpare = "cai-knitting" relSpare2 = "dov-knitting" ) var relObserved = time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) func relEnrollment(box string) string { return strings.Repeat(string(box[0]), 32) } // relAuthority accepts any fence record for a box in this region. type relAuthority struct{} func (relAuthority) ValidateFence(_ context.Context, f termfence.Record) (FenceBinding, error) { if f.Enrollment != relEnrollment(f.Box) { return FenceBinding{}, ErrFence } return FenceBinding{Region: "ie", RosterRevision: 9, RosterETag: "roster"}, nil } type relReleases struct{ err error } func (r relReleases) VerifyRelease(context.Context, Claim, Release) error { return r.err } func putFence(t *testing.T, s release.Store, box string) FenceRef { t.Helper() f := termfence.Record{V: 1, Box: box, Enrollment: relEnrollment(box), State: "terminated", Cloud: boxid.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + box}, ObservedAt: relObserved.Format(time.RFC3339Nano), Via: "describe", RequestID: "req", WrittenBy: termfence.WrittenBy{Box: "worker", Enrollment: strings.Repeat("e", 32)}} raw, err := json.Marshal(f) if err != nil { t.Fatal(err) } sum := sha256.Sum256(raw) ref := FenceRef{Enrollment: f.Enrollment, Key: "control/box-fences/" + f.Enrollment + ".json", SHA256: hex.EncodeToString(sum[:])} if _, err := s.Put(context.Background(), ref.Key, raw, ""); err != nil && !errors.Is(err, release.ErrCASConflict) { t.Fatal(err) } return ref } // relServing is a serving claim on relSource whose source has admitted one // cold writer session (generation g1). func relServing(t *testing.T) (release.Store, Observed) { t.Helper() ctx := context.Background() s := release.NewDirStore(t.TempDir()) o := servingClaim(t, ctx, s) o, err := Update(ctx, s, o, AdmitWriter(Admission{Box: relSource, Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) if err != nil { t.Fatal(err) } return s, o } func relStart(reason string, fence *FenceRef) RelocationStart { return RelocationStart{OpID: "rel-1", By: "worker", Now: relObserved, Region: "ie", Reason: reason, SourceBox: relSource, SourceEnrollment: relEnrollment(relSource), TargetBox: relTarget, TargetEnrollment: relEnrollment(relTarget), DirectoryRevision: 9, AdmissionID: "adm-1", Fence: fence} } func step(t *testing.T, s release.Store, o Observed, tr func(*Claim) error) Observed { t.Helper() next, err := Update(context.Background(), s, o, tr) if err != nil { t.Fatal(err) } return next } func ack(box string) func(*Claim) error { return AcknowledgeExecution(box, ExecutionAcknowledgement{Op: "rel-1", Enrollment: relEnrollment(box), BootID: "boot", ControllerID: "ctl", LeaseETag: "lease", Expires: relObserved.Add(time.Hour)}) } // relPin is a pin selected against o: its key names o's claim generation. func relPin(o Observed, maxTXID uint64, seq int64, at time.Time) RecoveryPin { return RecoveryPin{Key: RecoveryPinKey(o.Claim.Host, "rel-1", o.Claim.Gen), SHA256: strings.Repeat("a", 64), Generation: "g1", MaxTXID: maxTXID, FilesManifest: "instances/x/files/g1/manifest.json", FilesSeq: seq, PinnedAt: at, Empty: maxTXID == 0} } func relPrepared(gen string, maxTXID uint64, filesSeq int64) PreparedAuthority { a := PreparedAuthority{Generation: gen, SnapshotTXID: maxTXID, FilesSeq: 0, SourceFilesSeq: filesSeq, FilesManifestSHA256: strings.Repeat("f", 64)} if maxTXID > 0 { a.InstallManifest, a.SHA256 = "m", strings.Repeat("c", 64) } return a } func localInstall(c *Claim) LocalInstall { return LocalInstall{Box: c.Op.Exec, Enrollment: relEnrollment(c.Op.Exec), InstallStamp: journalUint(c.Op, "install_stamp"), FilesManifestVersion: 3, JournalSHA256: strings.Repeat("b", 64)} } // toProbing drives an operation at pinning (with its pin) through // installing, installed and the local install receipt into probing. func toProbing(t *testing.T, s release.Store, o Observed, prepared string, maxTXID uint64) Observed { t.Helper() target := Target{Box: o.Claim.Op.Intent["target_box"], Enrollment: o.Claim.Op.Intent["target_enrollment"]} o = step(t, s, o, ToInstalling("rel-1", target, prepared)) var pin RecoveryPin readJournal(o.Claim.Op, "pin", &pin) o = step(t, s, o, ToInstalled("rel-1", relPrepared(prepared, maxTXID, pin.FilesSeq))) o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) return step(t, s, o, ToProbing("rel-1", relObserved)) } func TestRelocationDrainEndToEnd(t *testing.T) { ctx := context.Background() s, o := relServing(t) o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{}) if err != nil { t.Fatal(err) } if c := o.Claim; c.Op.Exec != relSource || c.Op.Phase != PhaseStarting || !slices.Equal(c.Op.Participants, []string{relSource, relTarget}) || c.Op.Intent["stamp_from"] == "" { t.Fatalf("start: %+v", c.Op) } if _, err := Update(ctx, s, o, ToReleasing("rel-1", relObserved)); !errors.Is(err, ErrRelocation) { t.Fatalf("barrier without acknowledgements: %v", err) } o = step(t, s, o, ack(relSource)) o = step(t, s, o, ack(relTarget)) o = step(t, s, o, ToReleasing("rel-1", relObserved)) if o.Claim.Op.Journal["releasing_since"] == "" { t.Fatal("releasing did not start its deadline") } h := o.Claim.Holder rel := Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "control/stop/r.json", ReceiptSHA256: strings.Repeat("d", 64)} o = step(t, s, o, JournalRelease("rel-1", rel)) if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{err: errors.New("bad receipt")}, nil); err == nil { t.Fatal("pinned on an unverified receipt") } if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil { t.Fatal(err) } stamp := o.Claim.Holder.Stamp if o.Claim.Op.Exec != "none" || o.Claim.Op.Phase != PhasePinning { t.Fatalf("pinning: %+v", o.Claim.Op) } if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 39, 7, relObserved))); !errors.Is(err, ErrRelocation) { t.Fatalf("pin short of the release: %v", err) } stale := relPin(o, 40, 7, relObserved) stale.Key = RecoveryPinKey(o.Claim.Host, "rel-1", o.Claim.Gen-1) if _, err := Update(ctx, s, o, JournalPin("rel-1", stale)); !errors.Is(err, ErrRelocation) { t.Fatalf("a pin selected against an earlier observation was journaled: %v", err) } o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved))) if o.Claim.Holder.Stamp != stamp { t.Fatal("pinning changed the stamp") } o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1")) for name, a := range map[string]PreparedAuthority{ "no files manifest": {Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 40}, "another files seq": {Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 40, FilesSeq: 6, FilesManifestSHA256: strings.Repeat("f", 64)}, "no install manifest": {Generation: "P1", SnapshotTXID: 40, FilesSeq: 7, FilesManifestSHA256: strings.Repeat("f", 64)}, "another snapshot txid": {Generation: "P1", InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: 41, FilesSeq: 7, FilesManifestSHA256: strings.Repeat("f", 64)}, } { if _, err := Update(ctx, s, o, ToInstalled("rel-1", a)); !errors.Is(err, ErrRelocation) { t.Fatalf("%s: installed on an incomplete build: %v", name, err) } } o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) o = step(t, s, o, ToProbing("rel-1", relObserved)) install := journalUint(o.Claim.Op, "install_stamp") if o.Claim.Op.Exec != relTarget || o.Claim.Holder.Stamp != install || !slices.Contains(o.Claim.Touched.Boxes, relTarget) { t.Fatalf("installation: exec %s stamp %d install %d", o.Claim.Op.Exec, o.Claim.Holder.Stamp, install) } o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1] if last.Coverage != "released" || last.WritesLost != nil || o.Claim.Holder.Stamp != install { t.Fatalf("released admission: %+v stamp %d", last, o.Claim.Holder.Stamp) } o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relTarget, Session: o.Claim.Holder.Session})) o = step(t, s, o, CompleteRelocation("rel-1", counter())) c := o.Claim if c.Op.Kind != KindServing || c.Op.ID != "" || c.Op.Exec != relTarget || !slices.Equal(c.Op.Participants, []string{relTarget}) { t.Fatalf("serving: %+v", c.Op) } if len(c.Obligations) != 1 || c.Obligations[0].Kind != "park-local" || c.Obligations[0].Box != relSource || c.Obligations[0].Op != "rel-1" { t.Fatalf("the source's park-local: %+v", c.Obligations) } } func TestRelocationFailoverCitesTheFenceAndDeclaresTheLoss(t *testing.T) { ctx := context.Background() for _, qualified := range []bool{false, true} { s, o := relServing(t) o = step(t, s, o, func(c *Claim) error { // a chore the dead source already owed c.Obligations = append(c.Obligations, Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: relSource, Retirement: "r", Owner: "box:" + relSource}) return nil }) ref := putFence(t, s, relSource) sourceSession := o.Claim.Holder.Session in := relStart(ReasonFailover, &ref) in.Discharge = &LocalDischarge{Qualified: qualified, Records: []string{"control/box-volumes/x/1.json"}} o, err := StartRelocation(ctx, s, o, in, relAuthority{}) if err != nil { t.Fatal(err) } c := o.Claim if c.Op.Exec != "none" || !slices.Equal(c.Op.Participants, []string{relTarget}) || !slices.Contains(c.Lost, relSource) || c.Op.Journal["fence."+relSource] == "" { t.Fatalf("failover start: %+v lost %v", c.Op, c.Lost) } if owes := slices.ContainsFunc(c.Obligations, func(o Obligation) bool { return o.Box == relSource }); owes == qualified { t.Fatalf("discharge qualified=%v left chores %+v", qualified, c.Obligations) } if _, err := Update(ctx, s, o, ToReleasing("rel-1", relObserved)); err == nil { t.Fatal("a failover entered releasing") } o = step(t, s, o, ack(relTarget)) o = step(t, s, o, PinFailover("rel-1")) if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(29*time.Second)))); !errors.Is(err, ErrRelocation) { t.Fatalf("pinned inside the 30-second window: %v", err) } o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(FailoverPinDelay)))) o = toProbing(t, s, o, "P1", 40) probe := Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"} if _, err := Update(ctx, s, o, AdmitWriter(probe)); !errors.Is(err, ErrWrongState) { t.Fatalf("a failover admission claimed released coverage: %v", err) } probe.Coverage = "" o = step(t, s, o, AdmitWriter(probe)) last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1] if last.WritesLost == nil || last.WritesLost.Box != relSource || last.WritesLost.Generation != "g1" { t.Fatalf("failover must declare the source's session: %+v", last) } o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relTarget, Session: o.Claim.Holder.Session})) o = step(t, s, o, CompleteRelocation("rel-1", counter())) owes := slices.ContainsFunc(o.Claim.Obligations, func(ob Obligation) bool { return ob.Kind == "park-local" && ob.Box == relSource }) if owes == qualified { t.Fatalf("qualified=%v: source chores at serving %+v", qualified, o.Claim.Obligations) } // D4: the loss outlives the operation, with the console still to // materialize it (nothing here stands in for the console). c = o.Claim if len(c.EvidenceRefs) != 1 || len(c.LossRefs) != 1 || c.EvidenceRefs[0].ID != c.LossRefs[0] { t.Fatalf("retained losses after serving: %+v refs %v", c.EvidenceRefs, c.LossRefs) } l := c.EvidenceRefs[0] if l.Box != relSource || l.Fence != ref || l.ObservedAt != relObserved.Format(time.RFC3339Nano) || l.Op != "rel-1" || l.Role != "source" || l.Session != sourceSession || l.Generation != "g1" || l.Discharge.Qualified != qualified || l.Timers != TimersUnknown || l.Reported { t.Fatalf("retained loss: %+v", l) } if c.Op.Journal != nil || !slices.Contains(c.Lost, relSource) { t.Fatalf("serving kept the journal, or lost the box: %+v %v", c.Op, c.Lost) } } } func TestRelocationSourceLossBecomesFailover(t *testing.T) { ctx := context.Background() for _, tc := range []struct { name string prepare func(*testing.T, release.Store, Observed) Observed wantPhase string wantExec string wantParts []string }{ {"starting before acknowledging", func(t *testing.T, s release.Store, o Observed) Observed { return o }, PhaseStarting, "none", []string{relTarget}}, {"starting after acknowledging", func(t *testing.T, s release.Store, o Observed) Observed { return step(t, s, o, ack(relSource)) }, PhaseStarting, "none", []string{relTarget}}, {"releasing before its receipt", func(t *testing.T, s release.Store, o Observed) Observed { o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget)) return step(t, s, o, ToReleasing("rel-1", relObserved)) }, PhasePinning, "none", []string{relTarget}}, {"releasing after its receipt", func(t *testing.T, s release.Store, o Observed) Observed { o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget)) o = step(t, s, o, ToReleasing("rel-1", relObserved)) h := o.Claim.Holder return step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 1, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) }, PhaseReleasing, relSource, []string{relSource, relTarget}}, } { t.Run(tc.name, func(t *testing.T) { s, o := relServing(t) o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{}) if err != nil { t.Fatal(err) } o = tc.prepare(t, s, o) stamp := o.Claim.Holder.Stamp o, err = FenceSource(ctx, s, o, "rel-1", putFence(t, s, relSource), nil, relAuthority{}) if err != nil { t.Fatal(err) } c := o.Claim if c.Op.Phase != tc.wantPhase || c.Op.Exec != tc.wantExec || !slices.Equal(c.Op.Participants, tc.wantParts) || c.Holder.Stamp != stamp || !slices.Contains(c.Lost, relSource) { t.Fatalf("phase %s exec %s parts %v stamp %d lost %v", c.Op.Phase, c.Op.Exec, c.Op.Participants, c.Holder.Stamp, c.Lost) } if c.Op.Intent["reason"] != ReasonDrain { t.Fatal("the intent was rewritten") } }) } } // relAtPhase starts a drain and drives it to phase. "probing+P" is probing // after the first admission opened P. func relAtPhase(t *testing.T, phase string) (release.Store, Observed) { t.Helper() ctx := context.Background() s, o := relServing(t) o, err := StartRelocation(ctx, s, o, relStart(ReasonDrain, nil), relAuthority{}) if err != nil { t.Fatal(err) } o = step(t, s, step(t, s, o, ack(relSource)), ack(relTarget)) if phase == PhaseStarting { return s, o } o = step(t, s, o, ToReleasing("rel-1", relObserved)) if phase == PhaseReleasing { return s, o } h := o.Claim.Holder o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil { t.Fatal(err) } o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved))) if phase == PhasePinning { return s, o } o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1")) if phase == PhaseInstalling { return s, o } o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) if phase == PhaseInstalled { return s, o } o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) o = step(t, s, o, ToProbing("rel-1", relObserved)) if phase == PhaseProbing { return s, o } o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) return s, o } func TestRelocationReplacementTable(t *testing.T) { ctx := context.Background() for _, tc := range []struct { at, wantPhase, wantExec string newStamp, keepP bool }{ {PhaseStarting, PhaseStarting, relSource, false, false}, {PhaseReleasing, PhaseReleasing, relSource, false, false}, {PhasePinning, PhasePinning, "none", false, false}, {PhaseInstalling, PhaseInstalling, relSpare, true, false}, {PhaseInstalled, PhaseInstalled, relSpare, true, true}, {PhaseProbing, PhaseInstalled, relSpare, true, true}, {"probing+P", PhasePinning, "none", false, false}, } { t.Run(tc.at, func(t *testing.T) { s, o := relAtPhase(t, tc.at) before := o.Claim.Holder.Stamp o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}) if err != nil { t.Fatal(err) } c := o.Claim if c.Op.Phase != tc.wantPhase || c.Op.Exec != tc.wantExec || c.Op.Intent["target_box"] != relSpare || !slices.Contains(c.Lost, relTarget) || slices.Contains(c.Op.Participants, relTarget) { t.Fatalf("phase %s exec %s intent %v lost %v parts %v", c.Op.Phase, c.Op.Exec, c.Op.Intent, c.Lost, c.Op.Participants) } if (c.Holder.Stamp != before) != tc.newStamp { t.Fatalf("stamp %d -> %d", before, c.Holder.Stamp) } if tc.keepP && c.Op.Journal["prepared"] != "P1" { t.Fatalf("the unopened P must be reused: %q", c.Op.Journal["prepared"]) } if c.Op.Journal["local_install"] != "" { t.Fatal("a replacement inherited its predecessor's receipt") } if tc.wantPhase == PhaseInstalled { if _, err := Update(ctx, s, o, ToProbing("rel-1", relObserved)); err == nil { t.Fatal("probing without the replacement's own local install") } } }) } } func TestRelocationTwoReplacementsThenStuck(t *testing.T) { ctx := context.Background() s, o := relAtPhase(t, PhasePinning) var err error for _, next := range []string{relSpare, relSpare2} { old := o.Claim.Op.Intent["target_box"] if o, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, old), nil, Target{next, relEnrollment(next)}, "", relAuthority{}); err != nil { t.Fatal(err) } } if _, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relSpare2), nil, Target{"eve-knitting", relEnrollment("eve-knitting")}, "", relAuthority{}); !errors.Is(err, ErrReplacementsExceeded) { t.Fatalf("third replacement: %v", err) } o = step(t, s, o, MarkStuck("rel-1", "two target replacements used")) if journalUint(o.Claim.Op, "replacements") != 2 { t.Fatal("stuck reset the count") } if _, err := Update(ctx, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P9")); !errors.Is(err, ErrRelocation) { t.Fatalf("a stuck operation moved: %v", err) } } func TestRelocationReplacementInReleasingAcknowledgesBeforePinning(t *testing.T) { ctx := context.Background() s, o := relAtPhase(t, PhaseReleasing) h := o.Claim.Holder o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{}) if err != nil || o.Claim.Op.Phase != PhaseReleasing { t.Fatalf("replacement in releasing: %v %+v", err, o.Claim.Op) } if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); !errors.Is(err, ErrRelocation) { t.Fatalf("left releasing before the replacement acknowledged: %v", err) } o = step(t, s, o, ack(relSpare)) if o, err = PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil); err != nil || o.Claim.Op.Phase != PhasePinning { t.Fatalf("after the acknowledgement: %v %+v", err, o.Claim.Op) } } func TestRelocationExhaustedTargetIsFencedIntoAStuckPinning(t *testing.T) { ctx := context.Background() exhaust := func(t *testing.T, s release.Store, o Observed) Observed { var err error for _, next := range []string{relSpare, relSpare2} { old := o.Claim.Op.Intent["target_box"] if o, err = ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, old), nil, Target{next, relEnrollment(next)}, "", relAuthority{}); err != nil { t.Fatal(err) } } return o } t.Run("before exhaustion it is a replacement", func(t *testing.T) { s, o := relAtPhase(t, PhasePinning) if _, err := FenceExhaustedTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, relAuthority{}); !errors.Is(err, ErrRelocation) { t.Fatalf("fenced as exhausted with replacements left: %v", err) } }) // Round 8 finding 1: while the live source is still op.exec, the // target's fence consumes the target and records stuck; the source's // execution identity, phase and release evidence stay. for _, phase := range []string{PhaseStarting, PhaseReleasing} { t.Run("dead while the source executes in "+phase, func(t *testing.T) { s, o := relAtPhase(t, phase) if phase == PhaseReleasing { h := o.Claim.Holder o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) } o = exhaust(t, s, o) before := o.Claim ref := putFence(t, s, relSpare2) o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{}) if err != nil { t.Fatal(err) } c := o.Claim if c.Op.Phase != phase || c.Op.Exec != relSource || c.Holder.Stamp != before.Holder.Stamp || c.Op.Journal["stuck"] == "" || !slices.Contains(c.Op.Participants, relSource) || slices.Contains(c.Op.Participants, relSpare2) || !slices.Contains(c.Lost, relSpare2) { t.Fatalf("the live source was revoked: %+v lost %v", c.Op, c.Lost) } for _, k := range []string{"release", "release.phase", "releasing_since", "exec_ack." + relSource} { if c.Op.Journal[k] != before.Op.Journal[k] { t.Fatalf("%s changed: %q -> %q", k, before.Op.Journal[k], c.Op.Journal[k]) } } if phase == PhaseReleasing && c.Op.Journal["release"] == "" { t.Fatal("the source's release evidence was dropped") } if l := c.EvidenceRefs[len(c.EvidenceRefs)-1]; l.Box != relSpare2 || l.Fence != ref { t.Fatalf("the dead target's loss: %+v", c.EvidenceRefs) } if err := c.Validate(); err != nil { t.Fatal(err) } }) } t.Run("dead after its local install, P unopened", func(t *testing.T) { s, o := relAtPhase(t, PhasePinning) o = exhaust(t, s, o) o = step(t, s, o, ack(relSpare2)) o = step(t, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P1")) o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) if _, err := Update(ctx, s, o, MarkStuck("rel-1", "x")); !errors.Is(err, ErrRelocation) { t.Fatalf("MarkStuck between installed and a probe: %v", err) } ref := putFence(t, s, relSpare2) o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{}) if err != nil { t.Fatal(err) } c := o.Claim if c.Op.Phase != PhasePinning || c.Op.Exec != "none" || c.Op.Journal["stuck"] == "" || slices.Contains(c.Op.Participants, relSpare2) || !slices.Contains(c.Lost, relSpare2) || c.Op.Journal["local_install"] != "" || c.Op.Journal["pin"] == "" || c.Op.Journal["prepared"] != "P1" || c.Op.Journal["prepared_authority"] == "" { t.Fatalf("exhausted and dead: %+v lost %v", c.Op, c.Lost) } if len(c.EvidenceRefs) != 3 || c.EvidenceRefs[2].Box != relSpare2 || c.EvidenceRefs[2].Fence != ref { t.Fatalf("the dead target's loss: %+v", c.EvidenceRefs) } if _, err := Update(ctx, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P2")); !errors.Is(err, ErrRelocation) { t.Fatalf("a stuck operation moved: %v", err) } if _, err := FenceExhaustedTarget(ctx, s, o, "rel-1", ref, nil, relAuthority{}); err == nil { t.Fatal("the fence was consumed twice") } }) t.Run("dead after P was opened", func(t *testing.T) { s, o := relAtPhase(t, "probing+P") o = exhaust(t, s, o) // after P's admission each replacement is a nested pinning o = step(t, s, o, ack(relSpare2)) o = step(t, s, o, JournalPin("rel-1", relPin(o, 60, 9, relObserved.Add(FailoverPinDelay)))) o = step(t, s, o, ToInstalling("rel-1", Target{relSpare2, relEnrollment(relSpare2)}, "P3")) o = step(t, s, o, ToInstalled("rel-1", relPrepared("P3", 60, 9))) o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) o = step(t, s, o, ToProbing("rel-1", relObserved)) o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relSpare2, Mode: "cold", NewGeneration: "P3", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"})) o, err := FenceExhaustedTarget(ctx, s, o, "rel-1", putFence(t, s, relSpare2), nil, relAuthority{}) if err != nil { t.Fatal(err) } c := o.Claim if c.Op.Phase != PhasePinning || c.Op.Exec != "none" || c.Op.Journal["stuck"] == "" || c.Op.Journal["pin"] != "" || c.Op.Journal["prepared"] != "" { t.Fatalf("nested stuck pinning: %+v", c.Op) } }) } func TestRelocationProbeFailureIsANestedRelocation(t *testing.T) { ctx := context.Background() s, o := relAtPhase(t, "probing+P") o = step(t, s, o, ToProbeFailed("rel-1", relObserved)) if o.Claim.Op.Journal["probe_failed_since"] == "" { t.Fatal("probe-failed did not start its release deadline") } h := o.Claim.Holder rel := Release{ReleasedBy: relTarget, Enrollment: relEnrollment(relTarget), Generation: h.Generation, Session: h.Session, FinalTXID: 55, ManifestSeq: 9, Receipt: "r2", ReceiptSHA256: strings.Repeat("d", 64)} o = step(t, s, o, JournalRelease("rel-1", rel)) if _, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, &Target{relTarget, relEnrollment(relTarget)}); err == nil { t.Fatal("the failed target was named again") } o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, &Target{relSpare, relEnrollment(relSpare)}) if err != nil { t.Fatal(err) } if o.Claim.Op.Phase != PhasePinning || o.Claim.Op.Exec != "none" { t.Fatalf("nested relocation: %+v", o.Claim.Op) } if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 54, 9, relObserved))); !errors.Is(err, ErrRelocation) { t.Fatalf("pin lost the probe's writes: %v", err) } o = step(t, s, o, JournalPin("rel-1", relPin(o, 55, 9, relObserved))) o = step(t, s, o, ack(relSpare)) o = toProbing(t, s, o, "P2", 55) // released coverage on P2's opening: the predecessor is the failed // target's probe session, whose release this operation journaled. o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) o = step(t, s, o, CompleteRelocation("rel-1", counter())) var owed []string for _, ob := range o.Claim.Obligations { owed = append(owed, ob.Box) } slices.Sort(owed) if !slices.Equal(owed, []string{relSource, relTarget}) { t.Fatalf("park-local owed by the source and the abandoned target: %v", owed) } } func TestRelocationAdmissionRules(t *testing.T) { ctx := context.Background() probe := func(o Observed, gen, coverage string) Admission { return Admission{ObservedOpID: "rel-1", ObservedPhase: o.Claim.Op.Phase, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: o.Claim.Op.Exec, Mode: "cold", NewGeneration: gen, Coverage: coverage, Region: "ie", PredecessorRegion: "ie"} } for _, tc := range []struct { name, at string admit func(Observed) Admission }{ {"the target admits in installing", PhaseInstalling, func(o Observed) Admission { return probe(o, "P1", "") }}, {"the target admits in installed", PhaseInstalled, func(o Observed) Admission { return probe(o, "P1", "") }}, {"the first probe skips P", PhaseProbing, func(o Observed) Admission { return probe(o, "other", "") }}, {"P is opened twice", "probing+P", func(o Observed) Admission { return probe(o, "P1", "") }}, {"released after P was opened", "probing+P", func(o Observed) Admission { return probe(o, "P3", "released") }}, {"warm probe", PhaseProbing, func(o Observed) Admission { a := probe(o, "", "") a.Mode, a.PredecessorCertified = "warm", true return a }}, {"the source claims released", PhaseStarting, func(o Observed) Admission { return probe(o, "g2", "released") }}, } { t.Run(tc.name, func(t *testing.T) { s, o := relAtPhase(t, tc.at) if _, err := Update(ctx, s, o, AdmitWriter(tc.admit(o))); !errors.Is(err, ErrWrongState) { t.Fatalf("admitted: %v", err) } }) } t.Run("released names another predecessor", func(t *testing.T) { s, o := relAtPhase(t, PhaseProbing) o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1].Session++ if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", "released"))); !errors.Is(err, ErrWrongState) { t.Fatalf("admitted: %v", err) } }) t.Run("released names another predecessor box", func(t *testing.T) { s, o := relAtPhase(t, PhaseProbing) o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1].Box = relSpare if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", "released"))); !errors.Is(err, ErrWrongState) { t.Fatalf("admitted: %v", err) } }) t.Run("a third attempt reopens P", func(t *testing.T) { s, o := relAtPhase(t, "probing+P") o = step(t, s, o, AdmitWriter(probe(o, "P3", ""))) // attempt two: a fresh generation if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "P1", ""))); !errors.Is(err, ErrWrongState) { t.Fatalf("P was opened again: %v", err) } }) t.Run("released outside relocating", func(t *testing.T) { s, o := relServing(t) if _, err := Update(ctx, s, o, AdmitWriter(Admission{ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relSource, Mode: "cold", NewGeneration: "g2", Coverage: "released", Region: "ie"})); !errors.Is(err, ErrWrongState) { t.Fatalf("admitted: %v", err) } }) t.Run("the source wakes in starting", func(t *testing.T) { s, o := relAtPhase(t, PhaseStarting) if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "g2", ""))); err != nil { t.Fatal(err) } }) t.Run("a stuck source does not wake", func(t *testing.T) { s, o := relAtPhase(t, PhaseStarting) o = step(t, s, o, MarkStuck("rel-1", "operator")) if _, err := Update(ctx, s, o, AdmitWriter(probe(o, "g2", ""))); !errors.Is(err, ErrWrongState) { t.Fatalf("admitted: %v", err) } }) } func TestRelocationEmptyPinNeedsItsEvidence(t *testing.T) { ctx := context.Background() t.Run("a release at txid 0 vouches", func(t *testing.T) { s, o := relAtPhase(t, PhaseReleasing) h := o.Claim.Holder o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 0, ManifestSeq: 0, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil) if err != nil { t.Fatal(err) } o = step(t, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved))) o = step(t, s, o, ToInstalling("rel-1", Target{relTarget, relEnrollment(relTarget)}, "P1")) if _, err := Update(ctx, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 0))); !errors.Is(err, ErrRelocation) { t.Fatalf("an empty pin accepted a database build: %v", err) } if _, err := Update(ctx, s, o, ToInstalled("rel-1", PreparedAuthority{Generation: "P1", FilesSeq: 0})); !errors.Is(err, ErrRelocation) { t.Fatalf("an empty P without its files manifest: %v", err) } step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 0, 0))) }) t.Run("a release at txid 40 does not", func(t *testing.T) { s, o := relAtPhase(t, PhaseReleasing) h := o.Claim.Holder o = step(t, s, o, JournalRelease("rel-1", Release{ReleasedBy: relSource, Enrollment: relEnrollment(relSource), Generation: h.Generation, Session: h.Session, FinalTXID: 40, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) o, err := PinAfterRelease(ctx, s, o, "rel-1", relReleases{}, nil) if err != nil { t.Fatal(err) } if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved))); !errors.Is(err, ErrRelocation) { t.Fatalf("an empty pin against a release at 40: %v", err) } }) t.Run("a failover of a host that had a writer does not", func(t *testing.T) { s, o := relServing(t) ref := putFence(t, s, relSource) o, err := StartRelocation(ctx, s, o, relStart(ReasonFailover, &ref), relAuthority{}) if err != nil { t.Fatal(err) } o = step(t, s, step(t, s, o, ack(relTarget)), PinFailover("rel-1")) if _, err := Update(ctx, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved.Add(FailoverPinDelay)))); !errors.Is(err, ErrRelocation) { t.Fatalf("a listing that found nothing was taken as empty: %v", err) } }) t.Run("a failover of a never-woken host does", func(t *testing.T) { s := release.NewDirStore(t.TempDir()) o := servingClaim(t, ctx, s) // session 0: no writer was ever admitted ref := putFence(t, s, relSource) o, err := StartRelocation(ctx, s, o, relStart(ReasonFailover, &ref), relAuthority{}) if err != nil { t.Fatal(err) } o = step(t, s, step(t, s, o, ack(relTarget)), PinFailover("rel-1")) step(t, s, o, JournalPin("rel-1", relPin(o, 0, 0, relObserved.Add(FailoverPinDelay)))) }) } func TestRelocationProbeAttemptsAreBounded(t *testing.T) { ctx := context.Background() s, o := relAtPhase(t, PhaseProbing) for i := 0; i < MaxProbeAttempts; i++ { o = step(t, s, o, BeginProbeAttempt("rel-1", relTarget)) } if journalUint(o.Claim.Op, "probe_attempts."+relTarget) != MaxProbeAttempts { t.Fatalf("count: %v", o.Claim.Op.Journal) } if _, err := Update(ctx, s, o, BeginProbeAttempt("rel-1", relTarget)); !errors.Is(err, ErrProbeAttemptsExhausted) { t.Fatalf("a fourth attempt: %v", err) } if _, err := Update(ctx, s, o, BeginProbeAttempt("rel-1", relSpare)); !errors.Is(err, ErrRelocation) { t.Fatalf("a non-executor counted an attempt: %v", err) } // A replacement target has its own count. o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{}) if err != nil { t.Fatal(err) } if journalUint(o.Claim.Op, "probe_attempts."+relSpare) != 0 { t.Fatal("the replacement inherited attempts") } } func TestRelocationDischargeCompletesDiscoveryLikeARetirement(t *testing.T) { ctx := context.Background() s, o := relServing(t) o = step(t, s, o, func(c *Claim) error { c.Touched.DiscoveryOp = "disc" c.Obligations = append(c.Obligations, Obligation{ID: "d1", Op: "disc", Kind: "park-local", Box: relSource, Retirement: "r", Owner: "box:" + relSource}) return nil }) ref := putFence(t, s, relSource) in := relStart(ReasonFailover, &ref) in.Discharge = &LocalDischarge{Qualified: true} o, err := StartRelocation(ctx, s, o, in, relAuthority{}) if err != nil { t.Fatal(err) } if len(o.Claim.Obligations) != 0 || !o.Claim.Touched.Complete || o.Claim.Touched.DiscoveryOp != "" { t.Fatalf("discharging the final discovery chore: %+v touched %+v", o.Claim.Obligations, o.Claim.Touched) } } func TestRelocationLossRetainsAFormerWritersSession(t *testing.T) { // The source is fenced after the target's probe admission made the // target the writer: the loss still names the source's last session. ctx := context.Background() s, o := relAtPhase(t, "probing+P") before := o.Claim.Holder var sourceAdmission AdmissionRecord for _, a := range before.Admissions { if a.Box == relSource { sourceAdmission = a } } o, err := FenceSource(ctx, s, o, "rel-1", putFence(t, s, relSource), nil, relAuthority{}) if err != nil { t.Fatal(err) } l := o.Claim.EvidenceRefs[0] if l.Box != relSource || l.Session != sourceAdmission.Session || l.Generation != sourceAdmission.Generation || l.LastAdmission == nil || !reflect.DeepEqual(*l.LastAdmission, sourceAdmission) || l.Incarnation != before.Incarnation || l.Stamp != before.Stamp { t.Fatalf("retained: %+v, source admission %+v", l, sourceAdmission) } if err := o.Claim.Validate(); err != nil { t.Fatal(err) } } func TestRelocationRetainsEveryFencedBoxAcrossCompletion(t *testing.T) { // A drain whose target is terminated in installing: the replacement // completes, and the dead target's loss is still on the claim. ctx := context.Background() s, o := relAtPhase(t, PhaseInstalling) ref := putFence(t, s, relTarget) o, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}) if err != nil { t.Fatal(err) } o = step(t, s, o, ack(relSpare)) o = step(t, s, o, ToInstalled("rel-1", relPrepared("P2", 40, 7))) o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) o = step(t, s, o, ToProbing("rel-1", relObserved)) o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) o = step(t, s, o, CompleteRelocation("rel-1", counter())) c := o.Claim if len(c.EvidenceRefs) != 1 || c.EvidenceRefs[0].Box != relTarget || c.EvidenceRefs[0].Role != "target" || c.EvidenceRefs[0].Session != 0 || !slices.Contains(c.LossRefs, c.EvidenceRefs[0].ID) { t.Fatalf("the abandoned target's loss: %+v refs %v", c.EvidenceRefs, c.LossRefs) } if err := c.Validate(); err != nil { t.Fatal(err) } } func TestRelocationStartGuard(t *testing.T) { serving := func(obs ...Obligation) *Claim { return &Claim{Holder: &Holder{Incarnation: "i"}, Op: Op{Kind: KindServing, Exec: relSource}, Obligations: obs} } for _, tc := range []struct { name string ob Obligation fenced bool want error }{ {"park-generation in the region (L11)", Obligation{Kind: "park-generation", Region: "ie"}, false, nil}, {"park-replica in the region", Obligation{Kind: "park-replica", Region: "ie"}, false, ErrPendingChore}, {"park-replica elsewhere", Obligation{Kind: "park-replica", Region: "uk"}, false, nil}, {"park-local on the target", Obligation{Kind: "park-local", Box: relTarget}, true, ErrPendingChore}, {"park-local on a live source", Obligation{Kind: "park-local", Box: relSource}, false, ErrPendingChore}, {"park-local on a fenced source", Obligation{Kind: "park-local", Box: relSource}, true, nil}, } { t.Run(tc.name, func(t *testing.T) { if err := RelocationStartGuard(serving(tc.ob), "ie", relSource, relTarget, tc.fenced); !errors.Is(err, tc.want) { t.Fatalf("got %v, want %v", err, tc.want) } }) } } func TestRelocationLocalInstallReceiptIsBound(t *testing.T) { ctx := context.Background() for _, tc := range []struct { name string mutate func(*LocalInstall) }{ {"another box", func(l *LocalInstall) { l.Box = relSpare }}, {"another enrollment", func(l *LocalInstall) { l.Enrollment = relEnrollment(relSpare) }}, {"a later writer stamp", func(l *LocalInstall) { l.InstallStamp++ }}, } { t.Run(tc.name, func(t *testing.T) { s, o := relAtPhase(t, PhaseInstalled) l := localInstall(o.Claim) tc.mutate(&l) if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", l)); !errors.Is(err, ErrRelocation) { t.Fatalf("recorded: %v", err) } }) } s, o := relAtPhase(t, PhaseInstalled) l := localInstall(o.Claim) o = step(t, s, o, RecordLocalInstall("rel-1", l)) if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", l)); !errors.Is(err, ErrAlreadyRecorded) { t.Fatalf("replay: %v", err) } if _, err := Update(ctx, s, o, MarkStuck("rel-1", "operator")); !errors.Is(err, ErrRelocation) { t.Fatalf("stuck between installed and a probe: %v", err) } // B1-2: probing is refused on another box's receipt, even one in the journal. foreign := l foreign.Box = relSpare raw, _ := json.Marshal(foreign) o.Claim.Op.Journal["local_install"] = string(raw) if _, err := Update(ctx, s, o, ToProbing("rel-1", relObserved)); !errors.Is(err, ErrRelocation) { t.Fatalf("probing on another box's receipt: %v", err) } } // Round 5 finding 1: a spare that still owes a local retirement for the // host is never named a replacement target, whatever phase the // replacement resumes at (installing and installed hand it op.exec in the // same CAS); a clean spare is. func TestRelocationReplacementRefusesABoxOwingALocalRetirement(t *testing.T) { ctx := context.Background() for _, phase := range []string{PhaseStarting, PhaseInstalling, PhaseInstalled} { t.Run(phase, func(t *testing.T) { s, o := relAtPhase(t, phase) o = step(t, s, o, func(c *Claim) error { c.Obligations = append(c.Obligations, Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: relSpare, Retirement: "r", Owner: "box:" + relSpare}) return nil }) ref := putFence(t, s, relTarget) if _, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}); !errors.Is(err, ErrPendingChore) { t.Fatalf("a box owing park-local was named: %v", err) } if _, err := Update(ctx, s, o, NameTarget("rel-1", Target{relSpare, relEnrollment(relSpare)})); err == nil { t.Fatal("NameTarget named a box owing park-local") } o, err := ReplaceTarget(ctx, s, o, "rel-1", ref, nil, Target{relSpare2, relEnrollment(relSpare2)}, "P2", relAuthority{}) if err != nil { t.Fatal(err) } c := o.Claim if c.Op.Intent["target_box"] != relSpare2 || slices.Contains(c.Op.Participants, relSpare) || slices.Contains(c.Touched.Boxes, relSpare) { t.Fatalf("the clean spare was not chosen: %+v touched %v", c.Op, c.Touched.Boxes) } if phase != PhaseStarting && c.Op.Exec != relSpare2 { t.Fatalf("exec %s", c.Op.Exec) } }) } } // Round 5 finding 2: the automatic fence passes the dead box's regional // work (park-generation, park-replica owned by box:) to its region in // the same CAS, as DeclareLost does for a declared loss, keeping every // other field; another box's work is untouched; local discharge, qualified // or not, changes nothing about it. func TestRelocationFenceReassignsTheDeadBoxsRegionalWork(t *testing.T) { ctx := context.Background() gen := Obligation{ID: "pg", Op: "old-op", Kind: "park-generation", Region: "ie", Generation: "g0", Retirement: "rg", Owner: "box:" + relSource} rep := Obligation{ID: "pr", Op: "old-op", Kind: "park-replica", Region: "uk", Retirement: "rr", Owner: "box:" + relSource} other := Obligation{ID: "po", Op: "old-op", Kind: "park-generation", Region: "ie", Generation: "g0b", Retirement: "ro", Owner: "box:" + relSpare} local := Obligation{ID: "pl", Op: "old-op", Kind: "park-local", Box: relSource, Retirement: "rl", Owner: "box:" + relSource} for _, qualified := range []bool{false, true} { s, o := relServing(t) o = step(t, s, o, func(c *Claim) error { c.Obligations = append(c.Obligations, gen, rep, other, local) return nil }) ref := putFence(t, s, relSource) in := relStart(ReasonFailover, &ref) in.Discharge = &LocalDischarge{Qualified: qualified, Records: []string{"control/box-volumes/x/1.json"}} o, err := StartRelocation(ctx, s, o, in, relAuthority{}) if err != nil { t.Fatal(err) } byID := map[string]Obligation{} for _, ob := range o.Claim.Obligations { byID[ob.ID] = ob } wantGen, wantRep := gen, rep wantGen.Owner, wantRep.Owner = "region:ie", "region:uk" if !reflect.DeepEqual(byID["pg"], wantGen) || !reflect.DeepEqual(byID["pr"], wantRep) || !reflect.DeepEqual(byID["po"], other) { t.Fatalf("qualified=%v: regional work %+v", qualified, o.Claim.Obligations) } if _, owes := byID["pl"]; owes == qualified { t.Fatalf("qualified=%v: the local chore %+v", qualified, o.Claim.Obligations) } want := []ObligationRef{{ID: "pg", Op: "old-op", Retirement: "rg"}, {ID: "pr", Op: "old-op", Retirement: "rr"}} if len(o.Claim.EvidenceRefs) != 1 || !reflect.DeepEqual(o.Claim.EvidenceRefs[0].Reassigned, want) { t.Fatalf("qualified=%v: the retained loss does not name the reassignment: %+v", qualified, o.Claim.EvidenceRefs) } if err := o.Claim.Validate(); err != nil { t.Fatal(err) } } } // Round 11's sweep: P's opening is journaled with the admission that made // it, so it survives AdmitWriter's trim of holder.admissions to eight. The // removal below stands in for that trim (generation.go drops a covered, // loss-archived head once more than eight are kept); with P's admission // gone from the list, P is still opened: never reopened, and the local // install is never remade. func TestRelocationPOpenedOutlivesTheAdmissionTrim(t *testing.T) { ctx := context.Background() s, o := relAtPhase(t, "probing+P") if o.Claim.Op.Journal["prepared_opened"] != "P1" { t.Fatalf("the opening was not journaled: %+v", o.Claim.Op.Journal) } o = step(t, s, o, func(c *Claim) error { c.Holder.Admissions = slices.DeleteFunc(c.Holder.Admissions, func(a AdmissionRecord) bool { return a.Generation == "P1" }) return nil }) if !PreparedOpened(o.Claim) { t.Fatal("P forgotten with its admission") } again := Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relTarget, Mode: "cold", NewGeneration: "P1", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"} if _, err := Update(ctx, s, o, AdmitWriter(again)); !errors.Is(err, ErrWrongState) { t.Fatalf("P reopened after the trim: %v", err) } if _, err := Update(ctx, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))); err == nil { t.Fatal("the local install was remade after P was opened") } } // Round 14 finding 1: P's prepared manifest is at sequence 0 whatever the // source's; the pin's sequence (7 here) is its provenance. ToInstalled // refuses a prepared manifest claimed at the source's sequence, and one // built from another source revision, and binds the one it accepts to // the installation stamp. func TestRelocationPreparedManifestIsSequenceZero(t *testing.T) { ctx := context.Background() s, o := relAtPhase(t, PhaseInstalling) var pin RecoveryPin readJournal(o.Claim.Op, "pin", &pin) if pin.FilesSeq != 7 { t.Fatalf("pin at %d", pin.FilesSeq) } atSource := relPrepared("P1", 40, 7) atSource.FilesSeq = 7 if _, err := Update(ctx, s, o, ToInstalled("rel-1", atSource)); !errors.Is(err, ErrRelocation) { t.Fatalf("a prepared manifest at the source's sequence: %v", err) } if _, err := Update(ctx, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 6))); !errors.Is(err, ErrRelocation) { t.Fatalf("a prepared manifest from another source revision: %v", err) } o = step(t, s, o, ToInstalled("rel-1", relPrepared("P1", 40, 7))) var a PreparedAuthority readJournal(o.Claim.Op, "prepared_authority", &a) if a.FilesSeq != 0 || a.SourceFilesSeq != 7 || a.InstallStamp == 0 || a.InstallStamp != journalUint(o.Claim.Op, "install_stamp") { t.Fatalf("journaled authority: %+v", a) } } // Two-box model finding F3: a target replaced in installing leaves its P // unopened; the replacement builds P'. The serving CAS queues a // park-generation for P in the region, and none for the opened P'. func TestRelocationParksEveryAbandonedPreparedGeneration(t *testing.T) { ctx := context.Background() s, o := relAtPhase(t, PhaseInstalling) o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "P2", relAuthority{}) if err != nil { t.Fatal(err) } var abandoned []string if !readJournal(o.Claim.Op, "abandoned_prepared", &abandoned) || !slices.Equal(abandoned, []string{"P1"}) { t.Fatalf("abandoned: %v", abandoned) } o = step(t, s, o, ack(relSpare)) o = step(t, s, o, ToInstalled("rel-1", relPrepared("P2", 40, 7))) o = step(t, s, o, RecordLocalInstall("rel-1", localInstall(o.Claim))) o = step(t, s, o, ToProbing("rel-1", relObserved)) o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Coverage: "released", Region: "ie", PredecessorRegion: "ie"})) o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) o = step(t, s, o, CompleteRelocation("rel-1", counter())) var parks []string for _, ob := range o.Claim.Obligations { if ob.Kind == "park-generation" && ob.Region == "ie" && ob.Owner == "region:ie" { parks = append(parks, ob.Generation) } } if !slices.Contains(parks, "P1") || slices.Contains(parks, "P2") { t.Fatalf("park-generation chores: %v (%+v)", parks, o.Claim.Obligations) } if err := o.Claim.Validate(); err != nil { t.Fatal(err) } } // Round 17 finding 3: the target's probe deadline passes before any // admission (P1 unopened), it releases in probe-failed, then it is // terminated; the replacement's default branch drops P1. It is recorded // first, so the completion of the nested relocation parks it. func TestRelocationParksAPDroppedFromProbeFailed(t *testing.T) { ctx := context.Background() s, o := relAtPhase(t, PhaseProbing) if PreparedOpened(o.Claim) { t.Fatal("P1 was opened") } o = step(t, s, o, ToProbeFailed("rel-1", relObserved)) o, err := ReplaceTarget(ctx, s, o, "rel-1", putFence(t, s, relTarget), nil, Target{relSpare, relEnrollment(relSpare)}, "", relAuthority{}) if err != nil { t.Fatal(err) } if o.Claim.Op.Phase != PhasePinning || o.Claim.Op.Journal["prepared"] != "" { t.Fatalf("nested: %+v", o.Claim.Op) } var abandoned []string if !readJournal(o.Claim.Op, "abandoned_prepared", &abandoned) || !slices.Equal(abandoned, []string{"P1"}) { t.Fatalf("abandoned: %v", abandoned) } o = step(t, s, o, ack(relSpare)) o = step(t, s, o, JournalPin("rel-1", relPin(o, 40, 7, relObserved.Add(FailoverPinDelay)))) o = toProbing(t, s, o, "P2", 40) o = step(t, s, o, AdmitWriter(Admission{ObservedOpID: "rel-1", ObservedPhase: PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: relSpare, Mode: "cold", NewGeneration: "P2", KeepStamp: true, Region: "ie", PredecessorRegion: "ie"})) o = step(t, s, o, RecordProbe("rel-1", Probe{Box: relSpare, Session: o.Claim.Holder.Session})) o = step(t, s, o, CompleteRelocation("rel-1", counter())) parked := false for _, ob := range o.Claim.Obligations { if ob.Kind == "park-generation" && ob.Generation == "P1" && ob.Region == "ie" { parked = true } } if !parked { t.Fatalf("P1 was not parked: %+v", o.Claim.Obligations) } } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostclaim/ -run 'Relocation' -count=1` Expected: FAIL to compile: `undefined: RelocationStart`, `undefined: StartRelocation`, and the rest. - [ ] **Step 3: Write the transitions** `internal/hostclaim/relocation.go`: ```go package hostclaim import ( "context" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "fmt" "slices" "strconv" "time" "github.com/carlosframework/platform/internal/release" ) // Relocation (claim §4 "Same-region relocation", region HA spec 1 §5.8): // the reasons, the phases and the bound on target replacements. const ( ReasonDrain = "drain" ReasonRebalance = "rebalance" ReasonFailover = "failover" PhaseStarting = "starting" PhaseReleasing = "releasing" PhasePinning = "pinning" PhaseInstalling = "installing" PhaseInstalled = "installed" PhaseProbing = "probing" PhaseProbeFailed = "probe-failed" // MaxTargetReplacements is spec 1 §5.8's bound: after two // replacements of the target in one operation it goes to stuck. MaxTargetReplacements = 2 // MaxProbeAttempts is §5.8's probe bound: at most three admissions // per target installation within the probe deadline; running out is // probe-failed. Attempts are counted in the claim before they are // made, so a crash or a lost response never buys a fourth. MaxProbeAttempts = 3 // FailoverPinDelay is §5.8's failover pin rule: pin no earlier than // the fence's observed_at plus 30 seconds, so requests S3 had already // accepted from the terminated box have completed (H3's window). FailoverPinDelay = 30 * time.Second ) var ( ErrRelocation = errors.New("hostclaim: relocation transition refused") ErrReplacementsExceeded = errors.New("hostclaim: relocation has used its target replacements") ErrAlreadyRecorded = errors.New("hostclaim: relocation journal entry already recorded") ErrProbeAttemptsExhausted = errors.New("hostclaim: the target has used its probe attempts") ) // RelocationStart is the start CAS's intent (spec 1 §5.8, claim §4). Fence // is set for a failover only and names the source's termination fence. type RelocationStart struct { OpID, By string Now time.Time Region, Reason string SourceBox, SourceEnrollment string TargetBox, TargetEnrollment string DirectoryRevision uint64 AdmissionID string Fence *FenceRef Discharge *LocalDischarge } // LocalDischarge is a fenced box's volume evidence (claim D5 as amended by // L3, spec 1 §5.9, H19): the volume records the caller read and judged with // boxvolumes.Discharge. When it is nil, or Qualified is false, the box's // park-local chores stay open for the operator path. It never gates the // fence or the relocation. type LocalDischarge struct { Qualified bool `json:"qualified"` Records []string `json:"records,omitempty"` Reason string `json:"reason,omitempty"` } // Release is the journal entry a live box writes after its release // sequence (spec 1 §5.8 steps 1 to 4): its stopped database's final TXID, // its final files manifest sequence and its full stop receipt. type Release struct { ReleasedBy string `json:"released_by"` Enrollment string `json:"enrollment"` Generation string `json:"generation"` Session uint64 `json:"session"` FinalTXID uint64 `json:"final_txid"` ManifestSeq int64 `json:"manifest_seq"` Receipt string `json:"receipt"` ReceiptSHA256 string `json:"receipt_sha256"` } // ReleaseAuthority verifies a journaled release's full stop receipt: the // immutable object at Receipt hashes to ReceiptSHA256 and is complete stop // evidence (claim §6.1, P6-D1) for this host, operation, box, incarnation // and session. Plan 6b's receipt format is its only implementation. type ReleaseAuthority interface { VerifyRelease(context.Context, Claim, Release) error } // RecoveryPin is the journal's reference to one pinned recovery set // (§5.8 "The pinned recovery set"): the immutable pin object holding the // exact versions, its digest, the database position it reaches, the files // manifest it pins, and when it was taken. type RecoveryPin struct { Key string `json:"key"` SHA256 string `json:"sha256"` Generation string `json:"generation"` MaxTXID uint64 `json:"max_txid"` FilesManifest string `json:"files_manifest"` FilesSeq int64 `json:"files_seq"` PinnedAt time.Time `json:"pinned_at"` // Empty is the evidenced empty bootstrap (claim §6.4a "Empty // bootstrap", spec 1 §5.8 "A host with nothing replicated"): no LTX // file exists because no transaction was ever committed. It is legal // only when a journaled release reports final_txid 0, or the holder // never admitted a writer (session 0). A listing that merely finds // nothing is never evidence (JournalPin refuses it). Empty bool `json:"empty,omitempty"` } // RecoveryPinKey is where the pin selected against claim generation gen // lives: control/relocations///pin-.json. The key binds the // object to the exact observation it was selected against, so a pin whose // journal CAS lost (the claim moved) is abandoned, never reused, and a // retry selects afresh at the new generation (B2-3's round binding). func RecoveryPinKey(host, op string, gen uint64) string { return "control/relocations/" + host + "/" + op + "/pin-" + strconv.FormatUint(gen, 10) + ".json" } // PreparedAuthority is the `installed` CAS's record of the remote build // (§5.8 "Building the prepared generation"): the install manifest of // generation P, verified complete by a fresh read. type PreparedAuthority struct { Generation string `json:"generation"` InstallManifest string `json:"install_manifest"` SHA256 string `json:"sha256"` SnapshotTXID uint64 `json:"snapshot_txid"` // FilesSeq and FilesManifestSHA256 identify P's prepared files // manifest (claim rev 6.30 R18-1, moves rev 7.22 §7.6): create-only, // at sequence 0, never replaced; FilesManifestSHA256 is the digest of // the exact bytes a fresh read verified, the exact version the first // admission consumes as its baseline (the session publisher then // starts at sequence 1). SourceFilesSeq is provenance, not identity: // the pinned source manifest's sequence it was built from, which must // be the pin's (rev 15, round 14 finding 1). Required always, an empty // bootstrap included: a P without its files manifest is not built. FilesSeq int64 `json:"files_seq"` SourceFilesSeq int64 `json:"source_files_seq"` FilesManifestSHA256 string `json:"files_manifest_sha256"` // InstallStamp binds the prepared manifest to the installation that // built it ({location, P, op id, installation stamp}): ToInstalled // sets it from install_stamp in the CAS that journals the authority. InstallStamp uint64 `json:"install_stamp,omitempty"` } // AutomaticLoss is claim §7.3 D4's "Automatic loss persistence": what the // transition that consumed a termination fence retains on the claim itself, // outside the replaceable operation, until the console materializes the // D4 host loss record. It survives CompleteRelocation, free and holder // replacement because it lives beside loss_refs, not in op.journal. type AutomaticLoss struct { ID string `json:"id"` // the loss identity, also in loss_refs Box string `json:"box"` Enrollment string `json:"enrollment"` Fence FenceRef `json:"fence"` ObservedAt string `json:"observed_at"` Op string `json:"op"` Kind string `json:"kind"` Phase string `json:"phase"` Role string `json:"role"` // "source" | "target" // Incarnation and Stamp bind the loss to the holder it was taken // against; Generation and Session are the box's last writer session, // whether it is still the writer or was superseded, and LastAdmission // is that session's admission record when the claim holds one (the // consequences the console reports: its coverage and declarations). Incarnation string `json:"incarnation"` Stamp uint64 `json:"stamp"` Generation string `json:"generation,omitempty"` Session uint64 `json:"session,omitempty"` LastAdmission *AdmissionRecord `json:"last_admission,omitempty"` Discharge LocalDischarge `json:"discharge"` // Reassigned is the regional work (park-generation, park-replica) // the box owned that the same CAS passed to its region (L3, as // DeclareLost's reassign_regional): the console's D4 record names it. Reassigned []ObligationRef `json:"reassigned,omitempty"` Timers TimerKnowledge `json:"timers"` // Reported is set by the console once the D4 record holds this loss // and its alert work exists. Until then reporting is pending. Reported bool `json:"reported,omitempty"` } // automaticLossID is deterministic for one fence consumed by one // operation for one box, so a replayed transition names the same loss and // the console's materialization deduplicates on it (D4: "Retries // deduplicate by the retained loss identity"). func automaticLossID(opID, box string, ref FenceRef) string { sum := sha256.Sum256([]byte(opID + "\x00" + box + "\x00" + ref.Enrollment + "\x00" + ref.SHA256)) return "auto-" + hex.EncodeToString(sum[:16]) } // LocalInstall is one box's local install receipt (§5.8, B1-2, B2-5). type LocalInstall struct { Box string `json:"box"` Enrollment string `json:"enrollment"` InstallStamp uint64 `json:"install_stamp"` FilesManifestVersion int64 `json:"files_manifest_version"` JournalSHA256 string `json:"journal_sha256"` } // Probe is the target's journaled probe result for its admitted session. type Probe struct { Box string `json:"box"` Session uint64 `json:"session"` } // Target names a box for the relocation's target slot. type Target struct { Box, Enrollment string } func journalJSON(c *Claim, key string, v any) error { raw, err := json.Marshal(v) if err != nil { return err } if c.Op.Journal == nil { c.Op.Journal = map[string]string{} } c.Op.Journal[key] = string(raw) return nil } func readJournal(op Op, key string, v any) bool { raw, ok := op.Journal[key] return ok && json.Unmarshal([]byte(raw), v) == nil } func journalUint(op Op, key string) uint64 { n, _ := strconv.ParseUint(op.Journal[key], 10, 64) return n } func requireRelocation(c *Claim, opID string, phases ...string) error { if err := RequireOp(opID)(c); err != nil { return err } if c.Holder == nil || c.Op.Kind != KindRelocating || !slices.Contains(phases, c.Op.Phase) { return fmt.Errorf("%w: %s/%s", ErrWrongState, c.Op.Kind, c.Op.Phase) } if c.Op.Journal["stuck"] != "" { return fmt.Errorf("%w: operation is stuck", ErrRelocation) } ensureJournal(c) return nil } // ensureJournal makes the journal writable: an empty one round-trips as nil. func ensureJournal(c *Claim) { if c.Op.Journal == nil { c.Op.Journal = map[string]string{} } } // requireAcks is the D6 barrier: every current participant has recorded // its execution acknowledgement before the operation leaves its phase. func requireAcks(c *Claim) error { for _, box := range c.Op.Participants { if c.Op.Journal["exec_ack."+box] == "" { return fmt.Errorf("%w: %s has not acknowledged", ErrRelocation, box) } } return nil } func failover(op Op) bool { return op.Intent["reason"] == ReasonFailover || op.Journal["reason"] == ReasonFailover } // RelocationStartGuard is §4's start guard with the relocation exception // (L11): a park-generation chore in the region does not block, a // park-replica in the region does, a park-local on the target does, and a // park-local on a fenced source does not block a failover. func RelocationStartGuard(c *Claim, region, source, target string, sourceFenced bool) error { if c.Holder == nil || c.Op.Kind != KindServing || c.Op.ID != "" { return fmt.Errorf("%w: kind %q", ErrWrongState, c.Op.Kind) } for _, o := range c.Obligations { switch { case o.Kind == "park-replica" && o.Region == region: return fmt.Errorf("%w: park-replica in %s", ErrPendingChore, region) case o.Kind == "park-local" && o.Box == target: return fmt.Errorf("%w: park-local on %s", ErrPendingChore, target) case o.Kind == "park-local" && o.Box == source && !sourceFenced: return fmt.Errorf("%w: park-local on %s", ErrPendingChore, source) } } return nil } // OwesLocal reports whether box owes a park-local chore on this claim: a // local retirement not yet done (spec 1 §2: such a box is never chosen to // install the host, and §4: a park-local on the target refuses). func OwesLocal(c *Claim, box string) bool { for _, o := range c.Obligations { if o.Kind == "park-local" && o.Box == box { return true } } return false } // LocalChoreBoxes lists every box that owes a park-local chore on the // claim, for a chooser's exclusions. func LocalChoreBoxes(c *Claim) []string { var out []string for _, o := range c.Obligations { if o.Kind == "park-local" && o.Box != "" && !slices.Contains(out, o.Box) { out = append(out, o.Box) } } return out } // fenceBox puts box on lost with its fence reference (L2, L4), passes the // regional work the box owned (park-generation, park-replica owned by // box:) to its region, region:, keeping each obligation's id, // operation, retirement, region, generation and park scope (L3, exactly // as DeclareLost does for a declared loss, lost.go), and, when the volume // evidence qualifies (L3), discharges that box's park-local chores for // this host, all in the same CAS. It never removes anything else. D4: the // same CAS retains the loss outside the operation (evidence_refs, // loss_refs) with the box's last writer session, the discharge verdict and // timer knowledge "unknown", so the console can materialize the loss record // after the operation is long gone. func fenceBox(c *Claim, box string, ref FenceRef, observedAt string, d *LocalDischarge) error { key := "fence." + box raw, err := json.Marshal(ref) if err != nil { return err } if old, ok := c.Op.Journal[key]; ok && old != string(raw) { return fmt.Errorf("%w: fence reference already bound", ErrFence) } if c.Op.Journal == nil { c.Op.Journal = map[string]string{} } c.Op.Journal[key] = string(raw) c.Op.Journal[key+".observed_at"] = observedAt if !slices.Contains(c.Lost, box) { c.Lost = append(c.Lost, box) } var reassigned []ObligationRef for i := range c.Obligations { o := &c.Obligations[i] if (o.Kind == "park-generation" || o.Kind == "park-replica") && o.Owner == "box:"+box { if o.Region == "" { return fmt.Errorf("%w: regional obligation %q names no region", ErrWrongState, o.ID) } o.Owner = "region:" + o.Region reassigned = append(reassigned, ObligationRef{ID: o.ID, Op: o.Op, Retirement: o.Retirement}) } } if d == nil { d = &LocalDischarge{Reason: "no volume evidence supplied"} } if err := journalJSON(c, "discharge."+box, d); err != nil { return err } if d.Qualified { // Each chore is completed with CompleteObligation's own semantics, // so a discovery operation whose final chore this discharges is // marked complete exactly as a retirement would (delete.go). for { var chore *Obligation for i := range c.Obligations { if c.Obligations[i].Kind == "park-local" && c.Obligations[i].Box == box { chore = &c.Obligations[i] break } } if chore == nil { break } if err := CompleteObligation(chore.Op, chore.ID)(c); err != nil { return err } } } return retainAutomaticLoss(c, box, ref, observedAt, *d, reassigned) } // retainAutomaticLoss is D4's retention, written once per (operation, box, // fence): a replay finds its record and changes nothing. func retainAutomaticLoss(c *Claim, box string, ref FenceRef, observedAt string, d LocalDischarge, reassigned []ObligationRef) error { id := automaticLossID(c.Op.ID, box, ref) for _, l := range c.EvidenceRefs { if l.ID == id { return nil } } role := "target" if box == c.Op.Intent["source_box"] { role = "source" } loss := AutomaticLoss{ID: id, Box: box, Enrollment: ref.Enrollment, Fence: ref, ObservedAt: observedAt, Op: c.Op.ID, Kind: c.Op.Kind, Phase: c.Op.Phase, Role: role, Discharge: d, Reassigned: reassigned, Timers: TimersUnknown} if h := c.Holder; h != nil { loss.Incarnation, loss.Stamp = h.Incarnation, h.Stamp // The box's last writer session: the live one when it is still // the writer, otherwise its last admission on this holder. if h.WriterBox == box { loss.Generation, loss.Session = h.Generation, h.Session } for i := len(h.Admissions) - 1; i >= 0; i-- { if h.Admissions[i].Box == box { last := h.Admissions[i] loss.LastAdmission = &last if loss.Session == 0 { loss.Generation, loss.Session = last.Generation, last.Session } break } } } c.EvidenceRefs = append(c.EvidenceRefs, loss) if !slices.Contains(c.LossRefs, id) { c.LossRefs = append(c.LossRefs, id) } return nil } func removeParticipant(c *Claim, box string) { c.Op.Participants = slices.DeleteFunc(slices.Clone(c.Op.Participants), func(b string) bool { return b == box }) } func addParticipant(c *Claim, box string) { if !slices.Contains(c.Op.Participants, box) { c.Op.Participants = append(c.Op.Participants, box) } } // installedBoxes is every box this operation handed op.exec for an // installation; each gets a park-local unless discharged (claim §4: // "Every abandoned local installation receives an exact park-local"). func noteInstalledBox(c *Claim, box string) error { var boxes []string readJournal(c.Op, "installed_boxes", &boxes) if !slices.Contains(boxes, box) { boxes = append(boxes, box) } return journalJSON(c, "installed_boxes", boxes) } // StartRelocation is the start CAS (spec 1 §5.8, claim §4). A drain or // rebalance starts with op.exec on the source; a failover cites the // source's termination fence, starts with op.exec "none", keeps the stamp, // and puts the source on lost. The caller has already checked the roster // (hostactors.Directory.CheckBoxStart), the target's eligibility and the // app-admission registration; this CAS records what it checked. func StartRelocation(ctx context.Context, s release.Store, observed Observed, in RelocationStart, authority FenceAuthority) (Observed, error) { if observed.Absent || observed.Claim == nil || observed.ETag == "" { return Observed{}, ErrWrongState } switch in.Reason { case ReasonDrain, ReasonRebalance: if in.Fence != nil { return Observed{}, fmt.Errorf("%w: only a failover cites a fence", ErrRelocation) } case ReasonFailover: if in.Fence == nil { return Observed{}, fmt.Errorf("%w: a failover cites the source's fence", ErrRelocation) } default: return Observed{}, fmt.Errorf("%w: reason %q", ErrRelocation, in.Reason) } if in.OpID == "" || in.Region == "" || in.AdmissionID == "" || in.DirectoryRevision == 0 || in.Now.IsZero() || !validBoxReference(in.SourceBox) || !validBoxReference(in.TargetBox) || in.SourceBox == "none" || in.TargetBox == "none" || in.SourceBox == in.TargetBox || in.SourceEnrollment == "" || in.TargetEnrollment == "" { return Observed{}, fmt.Errorf("%w: incomplete start", ErrRelocation) } var observedAt string if in.Fence != nil { v, err := VerifyFence(ctx, s, authority, *in.Fence) if err != nil { return Observed{}, err } if v.record.Box != in.SourceBox || v.record.Enrollment != in.SourceEnrollment || v.binding.Region != in.Region { return Observed{}, ErrFence } observedAt = v.record.ObservedAt } return Update(ctx, s, observed, func(c *Claim) error { if c.Op.Exec != in.SourceBox || slices.Contains(c.Lost, in.TargetBox) { return fmt.Errorf("%w: the source is not the executor, or the target is lost", ErrWrongState) } if err := RelocationStartGuard(c, in.Region, in.SourceBox, in.TargetBox, in.Fence != nil); err != nil { return err } exec, participants := in.SourceBox, []string{in.SourceBox, in.TargetBox} if in.Fence != nil { exec, participants = "none", []string{in.TargetBox} } c.Op = Op{Kind: KindRelocating, ID: in.OpID, Phase: PhaseStarting, By: in.By, StartedAt: in.Now.UTC(), Exec: exec, Participants: participants, Intent: map[string]string{ "region": in.Region, "reason": in.Reason, "source_box": in.SourceBox, "source_enrollment": in.SourceEnrollment, "target_box": in.TargetBox, "target_enrollment": in.TargetEnrollment, "stamp_from": strconv.FormatUint(c.Holder.Stamp, 10), "generation_from": c.Holder.Generation, "directory_revision": strconv.FormatUint(in.DirectoryRevision, 10), "admission_id": in.AdmissionID, }, Journal: map[string]string{}} if in.Fence != nil { return fenceBox(c, in.SourceBox, *in.Fence, observedAt, in.Discharge) } return nil }) } // FenceSource is §5.8's source-loss rule (B1-6): a drain or rebalance // whose source is terminated becomes a failover in one CAS citing the // source's fence. In starting it stays in starting with the source off the // barrier; in releasing before a release is journaled it goes to pinning // with the fence as evidence; after a release (pinning on) the source only // goes on lost. Every case keeps the stamp and the intent. func FenceSource(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority) (Observed, error) { if observed.Absent || observed.Claim == nil { return Observed{}, ErrWrongState } before := observed.Claim.Op source := before.Intent["source_box"] v, err := VerifyFence(ctx, s, authority, ref) if err != nil { return Observed{}, err } if v.record.Box != source || v.record.Enrollment != before.Intent["source_enrollment"] || v.binding.Region != before.Intent["region"] { return Observed{}, ErrFence } return Update(ctx, s, observed, func(c *Claim) error { if err := RequireOp(opID)(c); err != nil { return err } if c.Op.Kind != KindRelocating || c.Op.Intent["source_box"] != source { return ErrWrongState } ensureJournal(c) // Before its release is journaled the source's writes are not // kept by a release, so the op becomes a failover. After it, only // the source's liveness changes. if c.Op.Phase == PhaseStarting || (c.Op.Phase == PhaseReleasing && c.Op.Journal["release"] == "") { c.Op.Journal["reason"] = ReasonFailover removeParticipant(c, source) c.Op.Exec = "none" if c.Op.Phase == PhaseReleasing { c.Op.Phase = PhasePinning } } return fenceBox(c, source, ref, v.record.ObservedAt, d) }) } // ToReleasing is a drain's barrier CAS: from starting, once every // participant has acknowledged. A failover never enters releasing. now // starts the release deadline (hostrelocate.ReleaseDeadline): a source // that never journals its release goes to stuck. func ToReleasing(opID string, now time.Time) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseStarting); err != nil { return err } if failover(c.Op) { return fmt.Errorf("%w: a failover skips releasing", ErrRelocation) } if err := requireAcks(c); err != nil { return err } if now.IsZero() { return fmt.Errorf("%w: releasing needs its start time", ErrRelocation) } c.Op.Phase = PhaseReleasing c.Op.Journal["releasing_since"] = now.UTC().Format(time.RFC3339Nano) return nil } } // JournalRelease records a live box's completed release sequence. The // source records it in releasing; a failed target in probe-failed. The // release must be of the box's own current writer session. func JournalRelease(opID string, r Release) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseReleasing, PhaseProbeFailed); err != nil { return err } h := c.Holder writerMismatch := h.Session != 0 && h.WriterBox != r.ReleasedBy // a host never woken has no writer if r.ReleasedBy == "" || r.ReleasedBy != c.Op.Exec || r.Enrollment == "" || r.Receipt == "" || !isHex64(r.ReceiptSHA256) || r.Generation != h.Generation || r.Session != h.Session || writerMismatch { return fmt.Errorf("%w: release is not the executor's current session", ErrRelocation) } want := c.Op.Intent["source_enrollment"] if c.Op.Phase == PhaseProbeFailed { want = c.Op.Intent["target_enrollment"] } if r.Enrollment != want { return fmt.Errorf("%w: release enrollment", ErrRelocation) } var old Release if readJournal(c.Op, "release", &old) && c.Op.Journal["release.phase"] == c.Op.Phase { if old == r { return ErrAlreadyRecorded } return fmt.Errorf("%w: a different release is journaled", ErrRelocation) } c.Op.Journal["release.phase"] = c.Op.Phase return journalJSON(c, "release", r) } } // PinAfterRelease leaves releasing (a drain) or probe-failed (a failed // target, §5.8's nested relocation) for pinning, citing the journaled // release's verified receipt. op.exec becomes "none" and the stamp is kept // (claim R6-4, D5's non-installing form). From probe-failed, next names // the replacement target, which must not be the failed one. func PinAfterRelease(ctx context.Context, s release.Store, observed Observed, opID string, authority ReleaseAuthority, next *Target) (Observed, error) { if observed.Absent || observed.Claim == nil || authority == nil { return Observed{}, ErrWrongState } var r Release if !readJournal(observed.Claim.Op, "release", &r) { return Observed{}, fmt.Errorf("%w: no journaled release", ErrRelocation) } if err := authority.VerifyRelease(ctx, *observed.Claim, r); err != nil { return Observed{}, err } return Update(ctx, s, observed, func(c *Claim) error { if err := requireRelocation(c, opID, PhaseReleasing, PhaseProbeFailed); err != nil { return err } var now Release if !readJournal(c.Op, "release", &now) || now != r || c.Op.Journal["release.phase"] != c.Op.Phase || r.ReleasedBy != c.Op.Exec { return fmt.Errorf("%w: release changed", ErrRelocation) } if c.Op.Phase == PhaseReleasing { if next != nil { return fmt.Errorf("%w: a drain keeps its target", ErrRelocation) } // §5.8: a target replaced in releasing acknowledges before the // operation leaves the phase. if err := requireAcks(c); err != nil { return err } } else { if next == nil || next.Box == r.ReleasedBy { return fmt.Errorf("%w: a nested relocation names another target", ErrRelocation) } if err := replaceTargetSlot(c, *next); err != nil { return err } c.Op.Journal["released_by_target"] = r.ReleasedBy } c.Op.Exec, c.Op.Phase = "none", PhasePinning delete(c.Op.Journal, "pin") return nil }) } // PinFailover leaves starting for pinning on a failover: the source's // fence is journaled and every remaining participant has acknowledged. func PinFailover(opID string) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseStarting); err != nil { return err } if !failover(c.Op) || c.Op.Journal["fence."+c.Op.Intent["source_box"]] == "" || c.Op.Exec != "none" { return fmt.Errorf("%w: pinning without a release needs the source's fence", ErrRelocation) } if err := requireAcks(c); err != nil { return err } c.Op.Phase = PhasePinning return nil } } // JournalPin records the pinned recovery set in pinning. The pin's key // must be RecoveryPinKey for this claim generation: the object was selected // against exactly this observation (B2-3). After a release it must reach // the journaled final TXID and files sequence; on a failover it must be // taken no earlier than the fence's observed_at plus FailoverPinDelay. An // Empty pin needs its evidence: a release at final_txid 0, or a holder // that never admitted a writer. func JournalPin(opID string, p RecoveryPin) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhasePinning); err != nil { return err } if p.Key != RecoveryPinKey(c.Host, opID, c.Gen) { return fmt.Errorf("%w: pin %s was not selected against this observation (gen %d)", ErrRelocation, p.Key, c.Gen) } if !isHex64(p.SHA256) || p.FilesManifest == "" || p.PinnedAt.IsZero() || (p.Empty != (p.MaxTXID == 0)) { return fmt.Errorf("%w: incomplete pin", ErrRelocation) } var r Release released := readJournal(c.Op, "release", &r) if released { if p.MaxTXID < r.FinalTXID || p.FilesSeq < r.ManifestSeq { return fmt.Errorf("%w: pin %d/%d is short of the release's %d/%d", ErrRelocation, p.MaxTXID, p.FilesSeq, r.FinalTXID, r.ManifestSeq) } } if p.Empty && !((released && r.FinalTXID == 0) || (!released && c.Holder.Session == 0)) { return fmt.Errorf("%w: an empty pin needs a release at txid 0 or a holder that never had a writer", ErrRelocation) } for key, at := range c.Op.Journal { if len(key) > len(".observed_at") && key[len(key)-len(".observed_at"):] == ".observed_at" { t, err := time.Parse(time.RFC3339Nano, at) if err != nil || p.PinnedAt.Before(t.Add(FailoverPinDelay)) { return fmt.Errorf("%w: pinned before the fence's observation plus %s", ErrRelocation, FailoverPinDelay) } } } if readJournal(c.Op, "pin", new(RecoveryPin)) { return ErrAlreadyRecorded } return journalJSON(c, "pin", p) } } // NameTarget names the target of an operation that has none: a move // converted to relocating before its flip (claim §4, moves §7.10). The // named box acknowledges (D6) before ToInstalling hands it op.exec. func NameTarget(opID string, t Target) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhasePinning); err != nil { return err } if c.Op.Intent["target_box"] != "" { return fmt.Errorf("%w: the operation already has a target", ErrRelocation) } return replaceTargetSlot(c, t) } } // ToInstalling hands op.exec to the target for a new installation: a new // stamp (this CAS's gen, R6-4), journaled as install_stamp, a fresh // prepared generation id, and the target added to touched.boxes before its // first effect. A converted operation with no target names it here. func ToInstalling(opID string, t Target, prepared string) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhasePinning); err != nil { return err } if !readJournal(c.Op, "pin", new(RecoveryPin)) || prepared == "" || prepared == c.Holder.Generation { return fmt.Errorf("%w: installing needs a pin and a fresh prepared generation", ErrRelocation) } if t.Box != c.Op.Intent["target_box"] || t.Enrollment != c.Op.Intent["target_enrollment"] || slices.Contains(c.Lost, t.Box) { return fmt.Errorf("%w: not the operation's target", ErrRelocation) } if OwesLocal(c, t.Box) { return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box) } if err := requireAcks(c); err != nil { return err } return install(c, t.Box, prepared, PhaseInstalling) } } // abandonPrepared journals the operation's current P as abandoned when // no writer opened it, before a transition replaces or drops it: nothing // will open it now, and the serving CAS parks it (rev 17, finding F3; // rev 18: every transition that discards an unopened P calls it). func abandonPrepared(c *Claim) error { old := c.Op.Journal["prepared"] if old == "" || preparedOpened(c) { return nil } var abandoned []string readJournal(c.Op, "abandoned_prepared", &abandoned) if slices.Contains(abandoned, old) { return nil } return journalJSON(c, "abandoned_prepared", append(abandoned, old)) } func install(c *Claim, box, prepared, phase string) error { if c.Op.Journal["prepared"] != prepared { if err := abandonPrepared(c); err != nil { return err } } stamp := c.Gen + 1 // the resulting gen: Update sets gen = observed+1 c.Op.Exec, c.Op.Phase = box, phase c.Holder.Stamp = stamp c.Op.Journal["install_stamp"] = strconv.FormatUint(stamp, 10) c.Op.Journal["prepared"] = prepared delete(c.Op.Journal, "local_install") if !slices.Contains(c.Touched.Boxes, box) { c.Touched.Boxes = append(c.Touched.Boxes, box) } return noteInstalledBox(c, box) } // ToInstalled records the verified remote build of P (§5.8, the // no-reinstall boundary): the install manifest at the pin's N (none for an // empty pin) and, always, P's files manifest at the pinned sequence, each // verified by a fresh read. Nobody builds P again after this CAS. func ToInstalled(opID string, a PreparedAuthority) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseInstalling); err != nil { return err } var pin RecoveryPin readJournal(c.Op, "pin", &pin) if a.Generation == "" || a.Generation != c.Op.Journal["prepared"] || a.SnapshotTXID != pin.MaxTXID || (pin.MaxTXID > 0 && (a.InstallManifest == "" || !isHex64(a.SHA256))) || (pin.MaxTXID == 0 && (a.InstallManifest != "" || a.SHA256 != "")) { return fmt.Errorf("%w: prepared authority is not this pin's build of P", ErrRelocation) } if a.FilesSeq != 0 || a.SourceFilesSeq != pin.FilesSeq || !isHex64(a.FilesManifestSHA256) { return fmt.Errorf("%w: prepared authority lacks P's prepared files manifest (sequence 0, built from the pinned sequence)", ErrRelocation) } if err := requireAcks(c); err != nil { return err } c.Op.Phase = PhaseInstalled a.InstallStamp = journalUint(c.Op, "install_stamp") return journalJSON(c, "prepared_authority", a) } } // PreparedOpened reports whether P has been opened: journaled as // prepared_opened by the admission that opened it (rev 12: AdmitWriter // trims holder.admissions to eight, so the admission list alone could // forget it), or named by a retained admission. func PreparedOpened(c *Claim) bool { return preparedOpened(c) } func preparedOpened(c *Claim) bool { p := c.Op.Journal["prepared"] if p == "" || c.Holder == nil { return false } if c.Op.Journal["prepared_opened"] == p { return true } for _, a := range c.Holder.Admissions { if a.Generation == p { return true } } return false } // RecordLocalInstall journals the executor's local install receipt in // installed (B1-2). It is bound to op.exec, the target's enrollment and the // journaled install_stamp, and refused once any admission has named P // (B2-5): after that the local install is never remade. func RecordLocalInstall(opID string, l LocalInstall) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseInstalled); err != nil { return err } if l.Box != c.Op.Exec || l.Enrollment != c.Op.Intent["target_enrollment"] || l.InstallStamp == 0 || l.InstallStamp != journalUint(c.Op, "install_stamp") || !isHex64(l.JournalSHA256) { return fmt.Errorf("%w: receipt is not this installation's", ErrRelocation) } if preparedOpened(c) { return fmt.Errorf("%w: P was admitted; the local install is never remade", ErrRelocation) } var old LocalInstall if readJournal(c.Op, "local_install", &old) { if old == l { return ErrAlreadyRecorded } return fmt.Errorf("%w: a different receipt is journaled", ErrRelocation) } return journalJSON(c, "local_install", l) } } // ToProbing enters probing, which requires a local install receipt for // exactly op.exec, its enrollment and the journaled install_stamp. now // starts the probe deadline (§5.8: 10 minutes). func ToProbing(opID string, now time.Time) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseInstalled); err != nil { return err } var l LocalInstall if !readJournal(c.Op, "local_install", &l) || l.Box != c.Op.Exec || l.Enrollment != c.Op.Intent["target_enrollment"] || l.InstallStamp != journalUint(c.Op, "install_stamp") { return fmt.Errorf("%w: no local install receipt for the executor", ErrRelocation) } if err := requireAcks(c); err != nil { return err } if now.IsZero() { return fmt.Errorf("%w: probing needs its start time", ErrRelocation) } c.Op.Phase = PhaseProbing c.Op.Journal["probing_since"] = now.UTC().Format(time.RFC3339Nano) return nil } } // BeginProbeAttempt counts one probe attempt for the executor before it // is made (§5.8 "limits"): the fourth is refused with // ErrProbeAttemptsExhausted, and the participant then enters // probe-failed. The count is per box and per operation, so a replacement // target starts at zero and a fenced one is never counted again. func BeginProbeAttempt(opID, box string) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseProbing); err != nil { return err } if box == "" || box != c.Op.Exec { return fmt.Errorf("%w: only the executor probes", ErrRelocation) } key := "probe_attempts." + box n := journalUint(c.Op, key) if n >= MaxProbeAttempts { return ErrProbeAttemptsExhausted } c.Op.Journal[key] = strconv.FormatUint(n+1, 10) return nil } } // RecordProbe journals the executor's passing probe of its admitted session. func RecordProbe(opID string, p Probe) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseProbing); err != nil { return err } if p.Box != c.Op.Exec || p.Box != c.Holder.WriterBox || p.Session == 0 || p.Session != c.Holder.Session || p.Session < journalUint(c.Op, "install_stamp") { return fmt.Errorf("%w: probe is not of the executor's admitted session", ErrRelocation) } return journalJSON(c, "probe", p) } } // ToProbeFailed records that the live target's probe failed or missed its // deadline. The target then runs its release sequence (JournalRelease); // now starts its release deadline. func ToProbeFailed(opID string, now time.Time) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseProbing); err != nil { return err } if now.IsZero() { return fmt.Errorf("%w: probe-failed needs its start time", ErrRelocation) } c.Op.Phase = PhaseProbeFailed c.Op.Journal["probe_failed_since"] = now.UTC().Format(time.RFC3339Nano) delete(c.Op.Journal, "release") delete(c.Op.Journal, "release.phase") return nil } } // CompleteRelocation is the serving CAS: the target's probe passed for its // admitted session, so the host serves on it alone. Every other box this // operation installed on, and a live source, owes a park-local unless a // fence discharged it. func CompleteRelocation(opID string, newID func() string) func(*Claim) error { return func(c *Claim) error { if err := requireRelocation(c, opID, PhaseProbing); err != nil { return err } var p Probe if !readJournal(c.Op, "probe", &p) || p.Box != c.Op.Exec || p.Session != c.Holder.Session || c.Holder.WriterBox != p.Box { return fmt.Errorf("%w: no passing probe of the current session", ErrRelocation) } target := c.Op.Exec var owed []string readJournal(c.Op, "installed_boxes", &owed) owed = append([]string{c.Op.Intent["source_box"]}, owed...) for _, box := range owed { if box == target || box == "" { continue } var d LocalDischarge if readJournal(c.Op, "discharge."+box, &d) && d.Qualified { continue } exists := false for _, o := range c.Obligations { if o.Kind == "park-local" && o.Box == box { exists = true } } if !exists { c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-local", Box: box, Retirement: newID(), Owner: "box:" + box}) } } // Every prepared generation this operation built and no writer // opened is parked here, in the region, by the claim's own // park-generation chore (rev 17, finding F3); the at-rest sweep // stays the backstop the spec names. var abandoned []string readJournal(c.Op, "abandoned_prepared", &abandoned) region := c.Op.Intent["region"] for _, g := range abandoned { if g == "" || g == c.Holder.Generation || region == "" { continue } queued := false for _, o := range c.Obligations { if o.Kind == "park-generation" && o.Region == region && o.Generation == g { queued = true } } if !queued { c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-generation", Region: region, Generation: g, Retirement: newID(), Owner: "region:" + region}) } } c.Op = Op{Kind: KindServing, Exec: target, Participants: []string{target}} return nil } } // replaceTargetSlot names t as the target, counting a replacement when it // replaces another box. A replaced box is never named again for this host. func replaceTargetSlot(c *Claim, t Target) error { if !validBoxReference(t.Box) || t.Box == "none" || t.Enrollment == "" || t.Box == c.Op.Intent["source_box"] || slices.Contains(c.Lost, t.Box) { return fmt.Errorf("%w: invalid target", ErrRelocation) } var excluded []string readJournal(c.Op, "excluded_targets", &excluded) if slices.Contains(excluded, t.Box) { return fmt.Errorf("%w: %s was replaced in this operation", ErrRelocation, t.Box) } if OwesLocal(c, t.Box) { // Spec 1 §2: a box owing a local retirement for this host is never // handed an installation, in any phase a replacement resumes at // (install is called below without ToInstalling's check). return fmt.Errorf("%w: park-local on %s", ErrPendingChore, t.Box) } old := c.Op.Intent["target_box"] if old != "" && old != t.Box { n := journalUint(c.Op, "replacements") if n >= MaxTargetReplacements { return ErrReplacementsExceeded } c.Op.Journal["replacements"] = strconv.FormatUint(n+1, 10) excluded = append(excluded, old) if err := journalJSON(c, "excluded_targets", excluded); err != nil { return err } removeParticipant(c, old) } c.Op.Intent["target_box"], c.Op.Intent["target_enrollment"] = t.Box, t.Enrollment addParticipant(c, t.Box) return nil } // ReplaceTarget is §5.8's replacement table for a target terminated by its // fence. It records the fence (the old target goes on lost), names next, // and resumes at the phase the table gives: // // starting, releasing, pinning -> the same phase // installing -> installing: next is op.exec, new stamp, fresh P' // installed, or probing before P -> installed: next is op.exec, new stamp, same P, no receipt // probing after P, or probe-failed -> pinning: a nested failover from holder.generation // // prepared is P' for installing and ignored otherwise. A third // replacement is refused with ErrReplacementsExceeded; the worker then // marks the operation stuck. func ReplaceTarget(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, next Target, prepared string, authority FenceAuthority) (Observed, error) { if observed.Absent || observed.Claim == nil { return Observed{}, ErrWrongState } old := observed.Claim.Op.Intent["target_box"] v, err := VerifyFence(ctx, s, authority, ref) if err != nil { return Observed{}, err } if old == "" || v.record.Box != old || v.record.Enrollment != observed.Claim.Op.Intent["target_enrollment"] || v.binding.Region != observed.Claim.Op.Intent["region"] { return Observed{}, ErrFence } return Update(ctx, s, observed, func(c *Claim) error { if err := requireRelocation(c, opID, PhaseStarting, PhaseReleasing, PhasePinning, PhaseInstalling, PhaseInstalled, PhaseProbing, PhaseProbeFailed); err != nil { return err } if c.Op.Intent["target_box"] != old { return ErrStaleOp } phase := c.Op.Phase if err := fenceBox(c, old, ref, v.record.ObservedAt, d); err != nil { return err } if err := replaceTargetSlot(c, next); err != nil { return err } switch { case phase == PhaseStarting || phase == PhaseReleasing || phase == PhasePinning: return nil case phase == PhaseInstalling: if prepared == "" || prepared == c.Op.Journal["prepared"] || prepared == c.Holder.Generation { return fmt.Errorf("%w: a replacement in installing builds a fresh P'", ErrRelocation) } return install(c, next.Box, prepared, PhaseInstalling) case (phase == PhaseInstalled || phase == PhaseProbing) && !preparedOpened(c): return install(c, next.Box, c.Op.Journal["prepared"], PhaseInstalled) default: // After P's first admission, or from probe-failed: nested // failover from pinning, pinning from holder.generation as // observed. A P no writer opened (a probe deadline before any // admission) is recorded before it is dropped. if err := abandonPrepared(c); err != nil { return err } c.Op.Exec, c.Op.Phase = "none", PhasePinning for _, k := range []string{"pin", "prepared", "prepared_authority", "local_install", "probe", "release", "release.phase"} { delete(c.Op.Journal, k) } return nil } }) } // FenceExhaustedTarget is the terminal transition for a target terminated // after the operation has used its replacements (§5.8 "The bound"; round 3 // finding 5): it consumes the fence (the loss is retained, the box goes on // lost) and records stuck. While another box is op.exec (the live source // in starting or releasing; round 8 finding 1) nothing else changes: the // target's fence revokes nothing of the source's. Otherwise (op.exec is // the dead target, or none) it revokes the dead installation's authority (op.exec "none", the // target off the barrier, its receipt dropped), leaves the operation at // pinning with its pin and, when P was never opened, its remote build kept // for the operator's replacement, or a nested pinning when P was opened, // and records stuck in the same CAS. No third target is chosen. MarkStuck's // "never between installed and a probe" is for a live target; a dead one // has no probe to make. func FenceExhaustedTarget(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority) (Observed, error) { return fenceTargetStuck(ctx, s, observed, opID, ref, d, authority, "") } // FenceTargetWithoutSpare is FenceExhaustedTarget's transition for a // terminated target with replacements left but no eligible box to name // (rev 18, round 17 finding 2): the worker calls it once its bound has // passed. The fence is consumed in the same CAS (loss retained, the box // on lost, a dead executor's authority revoked), and the operation is // stuck for an operator with reason, never left waiting unbounded. func FenceTargetWithoutSpare(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority, reason string) (Observed, error) { if reason == "" { return Observed{}, fmt.Errorf("%w: a stuck operation names its reason", ErrRelocation) } return fenceTargetStuck(ctx, s, observed, opID, ref, d, authority, reason) } // fenceTargetStuck is both: noSpare "" is the exhausted bound. func fenceTargetStuck(ctx context.Context, s release.Store, observed Observed, opID string, ref FenceRef, d *LocalDischarge, authority FenceAuthority, noSpare string) (Observed, error) { if observed.Absent || observed.Claim == nil { return Observed{}, ErrWrongState } old := observed.Claim.Op.Intent["target_box"] v, err := VerifyFence(ctx, s, authority, ref) if err != nil { return Observed{}, err } if old == "" || v.record.Box != old || v.record.Enrollment != observed.Claim.Op.Intent["target_enrollment"] || v.binding.Region != observed.Claim.Op.Intent["region"] { return Observed{}, ErrFence } return Update(ctx, s, observed, func(c *Claim) error { if err := RequireOp(opID)(c); err != nil { return err } if c.Op.Kind != KindRelocating || c.Op.Intent["target_box"] != old { return ErrStaleOp } ensureJournal(c) if c.Op.Journal["stuck"] != "" { return ErrAlreadyRecorded } reason := noSpare if reason == "" { if journalUint(c.Op, "replacements") < MaxTargetReplacements { return fmt.Errorf("%w: replacements remain; replace the target instead", ErrRelocation) } reason = fmt.Sprintf("%s terminated after %d target replacements", old, MaxTargetReplacements) } if err := fenceBox(c, old, ref, v.record.ObservedAt, d); err != nil { return err } removeParticipant(c, old) if c.Op.Exec != old && c.Op.Exec != "none" { // The executor is another box (the live source in starting or // releasing): the target's fence is no evidence against it. // Its execution identity, phase and release records stay, so // root still accepts its own stop (claimstop.go requires the // requester to be op.exec); only stuck is new. c.Op.Journal["stuck"] = reason return nil } opened := preparedOpened(c) c.Op.Exec, c.Op.Phase = "none", PhasePinning keys := []string{"local_install", "probe", "release", "release.phase", "probing_since", "probe_failed_since"} if opened { // Nested: pinning again from holder.generation, as a replacement // after P's first admission would. keys = append(keys, "pin", "prepared", "prepared_authority") } // An unopened P is kept for the operator's replacement; it is on // the abandoned list only if a later transition drops it. for _, k := range keys { delete(c.Op.Journal, k) } if c.Op.Journal["stuck"] == "" { c.Op.Journal["stuck"] = reason } return nil }) } // MarkStuck stops the operation at its current phase for an operator // (`carlos ops host resolve`). It never resets the replacement count and // never invents evidence; every later relocation transition refuses. func MarkStuck(opID, reason string) func(*Claim) error { return func(c *Claim) error { if err := RequireOp(opID)(c); err != nil { return err } if c.Op.Kind != KindRelocating || reason == "" { return ErrWrongState } ensureJournal(c) if c.Op.Phase == PhaseInstalled && c.Op.Journal["local_install"] != "" && !preparedOpened(c) { return fmt.Errorf("%w: never stop between installed and a probe", ErrRelocation) } if c.Op.Journal["stuck"] != "" { return ErrAlreadyRecorded } c.Op.Journal["stuck"] = reason return nil } } ``` - [ ] **Step 4: Teach `AdmitWriter` the phase table and `released`, and the claim its retained losses** Apply to `internal/hostclaim/claim.go` (`git apply`): `EvidenceRefs`, D4's retained automatic losses, validated beside `loss_refs`: ```diff diff --git a/internal/hostclaim/claim.go b/internal/hostclaim/claim.go index 23d17858..50e55006 100644 --- a/internal/hostclaim/claim.go +++ b/internal/hostclaim/claim.go @@ -162,6 +162,7 @@ type Claim struct { Lost []string `json:"lost,omitempty"` LostRevisions map[string]uint64 `json:"lost_revisions,omitempty"` LossRefs []string `json:"loss_refs,omitempty"` + EvidenceRefs []AutomaticLoss `json:"evidence_refs,omitempty"` RetiredIncarnations []string `json:"retired_incarnations,omitempty"` QuietAfter map[string]time.Time `json:"quiet_after,omitempty"` CancelledCreates []CancelledCreate `json:"cancelled_creates,omitempty"` @@ -289,6 +290,16 @@ func (c *Claim) Validate() error { } seenLossRefs[id] = true } + seenEvidence := map[string]bool{} + for _, l := range c.EvidenceRefs { + // D4: a retained automatic loss names a loss identity the claim + // also lists, a box, and a well-formed fence reference. + if !seenLossRefs[l.ID] || seenEvidence[l.ID] || l.Box == "" || !validFenceRef(l.Fence) || l.Enrollment != l.Fence.Enrollment || l.Op == "" || + (l.Role != "source" && l.Role != "target") || (l.Timers != TimersComplete && l.Timers != TimersPartial && l.Timers != TimersUnknown) { + return fmt.Errorf("%w: invalid automatic loss evidence", ErrMalformed) + } + seenEvidence[l.ID] = true + } if len(c.RetiredIncarnations) > retiredCap { return fmt.Errorf("%w: retired_incarnations over cap", ErrMalformed) } ``` Apply to `internal/hostclaim/generation.go` (`git apply`): ```diff diff --git a/internal/hostclaim/generation.go b/internal/hostclaim/generation.go index ffd7e058..74947b56 100644 --- a/internal/hostclaim/generation.go +++ b/internal/hostclaim/generation.go @@ -43,6 +43,10 @@ func AdmitWriter(in Admission) func(*Claim) error { switch c.Op.Kind { case KindCreating, KindDeleting, KindParking, KindPurging, KindPlacing: return fmt.Errorf("%w: op kind %q cannot admit a writer", ErrWrongState, c.Op.Kind) + case KindRelocating: + if err := relocationAdmission(c, in); err != nil { + return err + } } h := c.Holder if in.LossID != "" && !in.lossVerified { @@ -87,6 +91,10 @@ func AdmitWriter(in Admission) func(*Claim) error { switch in.Coverage { case "", "sealed", "clean-return": + case "released": // checked by relocationAdmission (L9) + if c.Op.Kind != KindRelocating { + return ErrWrongState + } default: return ErrWrongState } @@ -261,3 +269,60 @@ func CoverParkedPredecessors(observedOpID, observedPhase string, region string, return nil } } + +// relocationAdmission is §4's phase table for relocating (L1) and §6.4a's +// `released` coverage (L9). The source may admit in starting and releasing +// (its local admission fence closes releasing); only op.exec admits in +// probing, cold; nobody admits in any other phase. The first probe +// admission opens the prepared generation P, which is opened at most once. +// `released` is legal only on that opening, when the predecessor is exactly +// the session whose release this operation journaled and its pin reaches +// that release's final position. +func relocationAdmission(c *Claim, in Admission) error { + if c.Op.Journal["stuck"] != "" { + // Ruling 9: a stuck operation runs nothing, the source's wake + // included; the operator resolves it (carlos ops host resolve). + return fmt.Errorf("%w: relocation is stuck", ErrWrongState) + } + switch c.Op.Phase { + case PhaseStarting, PhaseReleasing: + if in.Box != c.Op.Intent["source_box"] || in.Coverage == "released" { + return fmt.Errorf("%w: only the source admits before pinning", ErrWrongState) + } + return nil + case PhaseProbing: + default: + return fmt.Errorf("%w: relocating/%s admits no writer", ErrWrongState, c.Op.Phase) + } + if in.Mode != "cold" || in.Box != c.Op.Exec { + return fmt.Errorf("%w: a probe admission is the executor's, cold", ErrWrongState) + } + p, opened := c.Op.Journal["prepared"], preparedOpened(c) + if p == "" || (!opened && in.NewGeneration != p) || (opened && in.NewGeneration == p) { + return fmt.Errorf("%w: the first probe opens P, and only it", ErrWrongState) + } + if !opened { + // The opening is journaled with the admission, so it outlives the + // admission list's trim (rev 12): P is never opened twice. + ensureJournal(c) + c.Op.Journal["prepared_opened"] = p + } + if in.Coverage != "released" { + return nil + } + var r Release + var pin RecoveryPin + if opened || in.PredecessorCertified || !readJournal(c.Op, "release", &r) || !readJournal(c.Op, "pin", &pin) || + pin.MaxTXID < r.FinalTXID || pin.FilesSeq < r.ManifestSeq { + return fmt.Errorf("%w: released coverage needs this operation's release and a pin reaching it", ErrWrongState) + } + h := c.Holder + predecessor := AdmissionRecord{Session: h.Session, Box: h.WriterBox, Generation: h.Generation} + if len(h.Admissions) > 0 { + predecessor = h.Admissions[len(h.Admissions)-1] + } + if predecessor.Box != r.ReleasedBy || predecessor.Session != r.Session || predecessor.Generation != r.Generation { + return fmt.Errorf("%w: released coverage names another predecessor", ErrWrongState) + } + return nil +} ``` - [ ] **Step 5: Run the tests** Run: `nice -n 19 go test ./internal/hostclaim/ -count=1` Expected: PASS, every existing test included (`TestC7…` covers the conversion from `moving`, unchanged). - [ ] **Step 6: Commit** ```bash gofmt -l internal/hostclaim git add internal/hostclaim/relocation.go internal/hostclaim/relocation_test.go internal/hostclaim/generation.go internal/hostclaim/claim.go git commit -m "hostclaim: relocating within one region, released by stop evidence or a termination fence, with released coverage on P's one opening and the fenced box's loss retained Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 2: Create names its executor once (H12, L8, P5-D2, P5-D6) Spec §5.2. The executor is `boxchoice.Choose(boxchoice.Eligible(...))` over the roster's active, unfenced boxes in the record's placement, chosen before the create CAS and recorded in its intent as `selected_executor` and `executor_enrollment`. `CompleteCreate` refuses any other box, so recovery never substitutes. Identity, placement and AWS region come from the roster (`Roster.Regions[placement].AWSRegion`); the edges snapshot gives liveness and configuration only; claim loads only balance. A fence that cannot be read refuses the choice. `Permits` is the same read without the choice: whether one named box is in the eligible set (Task 5's start check). **Dormant:** `SelectExecutor` reports whether it selected at all; with no chooser (`ProductionAuthority.Executors` nil, which the console leaves nil unless `Config.RegionHA` is on) the create is byte-for-byte today's, `BeginCreate` and `Executor` at completion, claim-enabled one-box regions included. **Files:** - Modify: `internal/hostclaim/create.go`, `internal/hostclaim/create_test.go` - Create: `internal/hostops/executor.go`, `internal/hostops/loads.go`, `internal/hostops/executor_test.go` - Modify: `internal/hostops/operations.go` (`Authority.SelectExecutor`, `Attempt`), `internal/hostops/backfill_controller.go`, `internal/hostops/backfill.go` (`ProductionAuthority.Executors`, `SelectExecutor`), `internal/hostops/operations_test.go` (the fake authority) - Create: `internal/console/hostclaimexecutor.go`, `internal/console/hostclaimexecutor_test.go` (the console's chooser; the create door uses it once claim Plan 5's door lands, Task 10) - Modify: `internal/console/server.go` (`Config.RegionHA`), `cmd/console/main.go` (sets it from `CARLOS_REGION_HA` with `strconv.ParseBool`) - Test: `cmd/console/regionha_test.go` (rev 11: the switch itself) **Interfaces:** - Consumes: `boxchoice.Eligible`, `boxchoice.Choose`, `boxchoice.Snapshot`, `boxchoice.Load`, `hostactors.Directory.Read`, `Directory.TerminationFence`, `hostactors.Roster.Regions`; Task 1's `hostclaim.LocalChoreBoxes` (the eligible set excludes a box owing a `park-local`, rev 6). - Produces: `hostclaim.ExecutorSelection{Box, Enrollment}`, `hostclaim.BeginCreateOn(nh, body, opID, by, now, sel)`; `hostops.ErrNoEligibleBox`; `hostops.ExecutorChooser{Directory, Edges func(ctx) (boxchoice.Snapshot, error), Loads func(ctx) (map[string]boxchoice.Load, error), SecretsKeyFP, HomeName, Now}` with `Choose(ctx, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, error)` and `Permits(ctx, hostclaim.Observed, Record, box string) error`; `hostops.ClaimLoads(ctx, release.Store) (map[string]boxchoice.Load, error)`; `Authority.SelectExecutor(ctx, hostclaim.Observed, Record) (sel hostclaim.ExecutorSelection, selected bool, err error)`; `ProductionAuthority.Executors *ExecutorChooser`; `console.Config.RegionHA`; `(*console.Server).executorChooser(hostactors.Directory) *hostops.ExecutorChooser` (nil with HA off). - [ ] **Step 1: Write the failing tests** Append to `internal/hostclaim/create_test.go`: ```diff diff --git a/internal/hostclaim/create_test.go b/internal/hostclaim/create_test.go index 53968923..c9a100e6 100644 --- a/internal/hostclaim/create_test.go +++ b/internal/hostclaim/create_test.go @@ -276,3 +276,27 @@ func TestCompleteCreateRefusesExecutionAuthority(t *testing.T) { }) } } + +func TestCreateCompletesOnlyOnTheSelectedExecutor(t *testing.T) { + ctx := context.Background() + s := release.NewDirStore(t.TempDir()) + now := time.Date(2026, 10, 4, 1, 0, 1, 0, time.UTC) + reg, _ := Read(ctx, s, testHost) + sel := ExecutorSelection{Box: "bri-knitting", Enrollment: "enrollment-b"} + if _, err := Update(ctx, s, reg, BeginCreateOn(newHolder(), record, "op_c1", "console", now, ExecutorSelection{})); !errors.Is(err, ErrMalformed) { + t.Fatalf("create with no selection: %v", err) + } + o, err := Update(ctx, s, reg, BeginCreateOn(newHolder(), record, "op_c1", "console", now, sel)) + if err != nil { + t.Fatal(err) + } + if o.Claim.Op.Intent["selected_executor"] != sel.Box || o.Claim.Op.Intent["executor_enrollment"] != sel.Enrollment || o.Claim.Op.Exec != "" { + t.Fatalf("intent: %+v", o.Claim.Op) + } + if _, err := Update(ctx, s, o, CompleteCreate("op_c1", "adi-knitting")); !errors.Is(err, ErrWrongState) { + t.Fatalf("completion substituted a box: %v", err) + } + if o, err = Update(ctx, s, o, CompleteCreate("op_c1", sel.Box)); err != nil || o.Claim.Op.Exec != sel.Box { + t.Fatalf("complete: %v %+v", err, o.Claim.Op) + } +} ``` `internal/hostops/executor_test.go`: ```go package hostops import ( "context" "crypto/ed25519" "crypto/rand" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "strings" "testing" "time" "github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/release" "github.com/carlosframework/platform/internal/termfence" ) func chooserFixture(t *testing.T, boxes ...string) (ExecutorChooser, release.Store, *boxchoice.Snapshot, map[string]boxchoice.Load) { t.Helper() s := release.NewDirStore(t.TempDir()) pub, key, err := ed25519.GenerateKey(rand.Reader) if err != nil { t.Fatal(err) } roster := hostactors.Roster{V: 1, DeploymentBucket: "bucket", Revision: 1, Regions: map[string]hostactors.Region{"ie": {Account: "123", AWSRegion: "eu-west-1"}}, Boxes: map[string]hostactors.Box{}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}} snap := &boxchoice.Snapshot{Complete: true} yes := true for i, ref := range boxes { roster.Boxes[ref] = hostactors.Box{Enrollment: strings.Repeat(string("abcdef"[i]), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default", Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"} snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true, HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}}) } wire, err := release.SignRootEnvelope(release.RootSigner(key), roster) if err != nil { t.Fatal(err) } if _, err := s.Put(context.Background(), hostactors.Key, wire, ""); err != nil { t.Fatal(err) } loads := map[string]boxchoice.Load{} x := ExecutorChooser{Directory: hostactors.Directory{Store: s, PublicKey: pub, Bucket: "bucket"}, SecretsKeyFP: "fp", Edges: func(context.Context) (boxchoice.Snapshot, error) { return *snap, nil }, Loads: func(context.Context) (map[string]boxchoice.Load, error) { return loads, nil }, Now: func() time.Time { return time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) }} return x, s, snap, loads } func chooserRecord() Record { return Record{Holder: hostclaim.NewHolder{Family: "instance", Account: "bab", App: "jam", Host: "shop.bab.oncarlos.com", Region: "ie"}} } func TestExecutorChooserSpreadsThenBalancesThenHashes(t *testing.T) { ctx := context.Background() x, _, _, loads := chooserFixture(t, "box-a", "box-b") absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} first, err := x.Choose(ctx, absent, chooserRecord()) if err != nil { t.Fatal(err) } other := map[string]string{"box-a": "box-b", "box-b": "box-a"}[first.Box] if first.Enrollment != map[string]string{"box-a": strings.Repeat("a", 32), "box-b": strings.Repeat("b", 32)}[first.Box] { t.Fatalf("enrollment not the roster's: %+v", first) } // The hash winner already runs this app: spread prefers the other box. loads[first.Box] = boxchoice.Load{Hosts: 1, Apps: map[string]int{"bab/jam": 1}} if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != other { t.Fatalf("soft spread: got %s, want %s", got.Box, other) } // Both run the app: the fewest hosts wins. loads[other] = boxchoice.Load{Hosts: 5, Apps: map[string]int{"bab/jam": 1}} if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != first.Box { t.Fatalf("headroom: got %s, want %s", got.Box, first.Box) } } // Round 18 finding 2: with equal loads the winner is the greatest // SHA-256 of host + "\x00" + box id (claim P5-D2), computed here // independently of boxchoice. The two hostnames are chosen so that their // winners differ: a chooser that hashed anything but the record's host // would get at least one of them wrong. func TestExecutorChooserHashesTheRecordsHostOnEqualLoads(t *testing.T) { ctx := context.Background() winner := func(host string) string { a := sha256.Sum256([]byte(host + "\x00box-a")) b := sha256.Sum256([]byte(host + "\x00box-b")) if hex.EncodeToString(a[:]) > hex.EncodeToString(b[:]) { return "box-a" } return "box-b" } hosts := []string{"shop.bab.oncarlos.com", "cafe.bab.oncarlos.com"} if winner(hosts[0]) == winner(hosts[1]) { t.Fatal("the fixture's hostnames must choose opposite boxes") } for _, host := range hosts { x, _, _, _ := chooserFixture(t, "box-a", "box-b") r := chooserRecord() r.Holder.Host = host got, err := x.Choose(ctx, hostclaim.Observed{Host: host, Absent: true}, r) if err != nil || got.Box != winner(host) { t.Fatalf("%s: chose %+v %v, want %s", host, got, err, winner(host)) } } } func TestExecutorChooserNeverNamesAFencedLostOrUnknownBox(t *testing.T) { ctx := context.Background() absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} t.Run("lost", func(t *testing.T) { x, _, _, _ := chooserFixture(t, "box-a") obs := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Claim: &hostclaim.Claim{Lost: []string{"box-a"}}} if _, err := x.Choose(ctx, obs, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) { t.Fatalf("chose a lost box: %v", err) } }) t.Run("owing a local retirement", func(t *testing.T) { // Spec 1 §2: a box whose old installation of the host is not yet // retired (park-local) is never chosen; the clean one is. x, _, _, _ := chooserFixture(t, "box-a", "box-b") obs := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Claim: &hostclaim.Claim{Obligations: []hostclaim.Obligation{{ID: "old", Op: "old-op", Kind: "park-local", Box: "box-a", Retirement: "r", Owner: "box:box-a"}}}} for range 4 { if got, err := x.Choose(ctx, obs, chooserRecord()); err != nil || got.Box != "box-b" { t.Fatalf("got %+v %v, want the box with no chore", got, err) } } if err := x.Permits(ctx, obs, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) { t.Fatalf("permitted a box owing park-local: %v", err) } }) t.Run("fenced", func(t *testing.T) { x, s, _, _ := chooserFixture(t, "box-a", "box-w") f := termfence.Record{V: 1, Box: "box-a", Enrollment: strings.Repeat("a", 32), State: "terminated", Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-box-a"}, ObservedAt: "2026-10-07T08:00:00Z", Via: termfence.ViaDescribe, RequestID: "req", WrittenBy: termfence.WrittenBy{Box: "box-w", Enrollment: strings.Repeat("b", 32)}} raw, _ := json.Marshal(f) if _, err := s.Put(ctx, "control/box-fences/"+strings.Repeat("a", 32)+".json", raw, ""); err != nil { t.Fatal(err) } got, err := x.Choose(ctx, absent, chooserRecord()) if err != nil || got.Box != "box-w" { t.Fatalf("got %+v %v, want the unfenced box", got, err) } }) t.Run("an edges file for a box the roster does not hold", func(t *testing.T) { x, _, snap, _ := chooserFixture(t) yes := true snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: "stray", EIP: "203.0.113.9", Region: "eu-west-1", Placement: "ie", Fresh: true, HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}}) if _, err := x.Choose(ctx, absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) { t.Fatalf("chose an unenrolled box: %v", err) } }) t.Run("an incomplete snapshot", func(t *testing.T) { x, _, snap, _ := chooserFixture(t, "box-a") snap.Complete = false if _, err := x.Choose(ctx, absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) { t.Fatalf("chose from a partial read: %v", err) } }) t.Run("an unreadable fence", func(t *testing.T) { x, s, _, _ := chooserFixture(t, "box-a") if _, err := s.Put(ctx, "control/box-fences/"+strings.Repeat("a", 32)+".json", []byte("{"), ""); err != nil { t.Fatal(err) } if _, err := x.Choose(ctx, absent, chooserRecord()); err == nil || errors.Is(err, ErrNoEligibleBox) { t.Fatalf("an unreadable fence must refuse, not skip: %v", err) } }) } func TestExecutorChooserPermitsOnlyTheEligibleSet(t *testing.T) { ctx := context.Background() absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} for _, tc := range []struct { name string change func(*boxchoice.Box) }{ {"a stale edges file", func(b *boxchoice.Box) { b.Fresh = false }}, {"no region-ha-1 capability", func(b *boxchoice.Box) { b.Capabilities = nil }}, {"another secrets key", func(b *boxchoice.Box) { b.SecretsKeyFP = "other" }}, {"a draining box", func(b *boxchoice.Box) { b.Draining = true }}, } { t.Run(tc.name, func(t *testing.T) { x, _, snap, _ := chooserFixture(t, "box-a", "box-b") if err := x.Permits(ctx, absent, chooserRecord(), "box-a"); err != nil { t.Fatal(err) } tc.change(&snap.Boxes[0]) if err := x.Permits(ctx, absent, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) { t.Fatalf("permitted: %v", err) } if err := x.Permits(ctx, absent, chooserRecord(), "box-b"); err != nil { t.Fatalf("the other box: %v", err) } }) } t.Run("an unknown box", func(t *testing.T) { x, _, _, _ := chooserFixture(t, "box-a") if err := x.Permits(ctx, absent, chooserRecord(), "box-z"); !errors.Is(err, ErrNoEligibleBox) { t.Fatalf("permitted: %v", err) } }) } func TestClaimLoadsCountsExecutorsPerBoxAndApp(t *testing.T) { ctx := context.Background() s := release.NewDirStore(t.TempDir()) created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) for i, tc := range []struct{ host, app, box string }{ {"a.bab.oncarlos.com", "jam", "box-a"}, {"b.bab.oncarlos.com", "jam", "box-a"}, {"c.bab.oncarlos.com", "tea", "box-b"}, } { nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/" + tc.app + "/instances/" + tc.host + ".json", Account: "bab", App: tc.app, Host: tc.host, Region: "ie", RecordRegion: "ie", CreatedAt: created} body := []byte(`{"host":"` + tc.host + `","account":"bab","app":"` + tc.app + `","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) o, _ := hostclaim.Read(ctx, s, tc.host) o, err := hostclaim.Update(ctx, s, o, hostclaim.BeginCreateOn(nh, body, "c", "console", created, hostclaim.ExecutorSelection{Box: tc.box, Enrollment: "e"})) if err != nil { t.Fatal(err) } if i < 2 { // the third is still creating: it has no executor yet if _, err := hostclaim.Update(ctx, s, o, hostclaim.CompleteCreate("c", tc.box)); err != nil { t.Fatal(err) } } } if _, err := s.Put(ctx, "control/hosts/broken.bab.oncarlos.com.json", []byte("{"), ""); err != nil { t.Fatal(err) } loads, err := ClaimLoads(ctx, s) if err != nil { t.Fatal(err) } if loads["box-a"].Hosts != 2 || loads["box-a"].Apps["bab/jam"] != 2 || loads["box-b"].Hosts != 0 { t.Fatalf("loads: %+v", loads) } } func TestProductionAuthoritySelectsOnlyWithAChooser(t *testing.T) { // No chooser (CARLOS_REGION_HA off): no selection and no error, so the // create is today's. sel, selected, err := (ProductionAuthority{}).SelectExecutor(context.Background(), hostclaim.Observed{}, Record{}) if err != nil || selected || sel != (hostclaim.ExecutorSelection{}) { t.Fatalf("no chooser: %+v %v %v", sel, selected, err) } // With a chooser the selection is the chooser's, read from its // evidence: box-a is draining, so box-b with its roster enrollment; // with both draining, no box and an error, never a selection. x, _, snap, _ := chooserFixture(t, "box-a", "box-b") snap.Boxes[0].Draining = true absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} sel, selected, err = (ProductionAuthority{Executors: &x}).SelectExecutor(context.Background(), absent, chooserRecord()) if err != nil || !selected || sel.Box != "box-b" || sel.Enrollment != strings.Repeat("b", 32) { t.Fatalf("with a chooser: %+v %v %v", sel, selected, err) } snap.Boxes[1].Draining = true if sel, selected, err = (ProductionAuthority{Executors: &x}).SelectExecutor(context.Background(), absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) || selected { t.Fatalf("no eligible box: %+v %v %v", sel, selected, err) } } ``` `internal/console/hostclaimexecutor_test.go`: ```go package console import ( "context" "slices" "testing" "time" "github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/release" ) // The console's chooser reads the bucket through its two production // readers: the live edges (each box's edges file joined with its status // snapshot) and the serving claims' loads. The bucket holds real evidence, // the assertions name boxes and counts, and a change to the stored // evidence shows in the next read, so neither reader can be replaced by // one returning nothing (round 10). func TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims(t *testing.T) { ctx := context.Background() store := release.NewDirStore(t.TempDir()) off := &Server{cfg: Config{Releases: store, HomeRegion: "ie"}} if off.executorChooser(hostactors.Directory{Store: store, Bucket: "bucket"}) != nil { t.Fatal("a chooser with CARLOS_REGION_HA off: create would change") } s := &Server{cfg: Config{Releases: store, HomeRegion: "ie", RegionHA: true}} x := s.executorChooser(hostactors.Directory{Store: store, Bucket: "bucket"}) if x == nil || x.Edges == nil || x.Loads == nil || x.Now == nil || x.HomeName != "ie" || x.SecretsKeyFP != s.expectedSecretsFP() { t.Fatalf("chooser: %+v", x) } // Two live boxes in ie, each with its edges file and a fresh status // snapshot; box-b advertises the region-ha-1 capability. edges := func(box, extra string) string { return `{"v":1,"box":"` + box + `","box_id":"` + box + `","eip":"203.0.113.1","region":"eu-west-1","placement":"ie","hibernation_configured":true` + extra + `,"hosts":{}}` } for box, body := range map[string]string{"box-a": edges("box-a", ""), "box-b": edges("box-b", `,"capabilities":["`+boxchoice.Capability+`"]`)} { setBoxSnapshot(t, store, box, time.Now()) if _, err := store.Put(ctx, "control/edges/"+box+".json", []byte(body), ""); err != nil { t.Fatal(err) } } // Two serving claims on box-a for bab/jam, one on box-b for bab/tea. created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) serve := func(host, app, box string) { t.Helper() nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/" + app + "/instances/" + host + ".json", Account: "bab", App: app, Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created} body := []byte(`{"host":"` + host + `","account":"bab","app":"` + app + `","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) o, err := hostclaim.Read(ctx, store, host) if err != nil { t.Fatal(err) } if o, err = hostclaim.Update(ctx, store, o, hostclaim.BeginCreate(nh, body, "c", "console", created)); err != nil { t.Fatal(err) } if _, err = hostclaim.Update(ctx, store, o, hostclaim.CompleteCreate("c", box)); err != nil { t.Fatal(err) } } serve("a.bab.oncarlos.com", "jam", "box-a") serve("b.bab.oncarlos.com", "jam", "box-a") serve("c.bab.oncarlos.com", "tea", "box-b") read := func() (map[string]boxchoice.Box, map[string]boxchoice.Load) { t.Helper() snap, err := x.Edges(ctx) if err != nil || !snap.Complete { t.Fatalf("edges: %+v %v", snap, err) } boxes := map[string]boxchoice.Box{} for _, b := range snap.Boxes { boxes[b.BoxID] = b } loads, err := x.Loads(ctx) if err != nil { t.Fatal(err) } return boxes, loads } boxes, loads := read() if len(boxes) != 2 || !boxes["box-a"].Fresh || !boxes["box-b"].Fresh || boxes["box-a"].Placement != "ie" || slices.Contains(boxes["box-a"].Capabilities, boxchoice.Capability) || !slices.Contains(boxes["box-b"].Capabilities, boxchoice.Capability) { t.Fatalf("live edges: %+v", boxes) } if loads["box-a"].Hosts != 2 || loads["box-a"].Apps["bab/jam"] != 2 || loads["box-b"].Hosts != 1 || loads["box-b"].Apps["bab/tea"] != 1 { t.Fatalf("loads: %+v", loads) } // The evidence changes: box-a starts draining and stops reporting // (stale), and a fourth host serves on box-b. The next read says so. _, etag, err := store.Get(ctx, "control/edges/box-a.json") if err != nil { t.Fatal(err) } if _, err := store.Put(ctx, "control/edges/box-a.json", []byte(edges("box-a", `,"draining":true`)), etag); err != nil { t.Fatal(err) } setBoxSnapshot(t, store, "box-a", time.Now().Add(-time.Hour)) serve("d.bab.oncarlos.com", "tea", "box-b") boxes, loads = read() if !boxes["box-a"].Draining || boxes["box-a"].Fresh || !boxes["box-b"].Fresh { t.Fatalf("live edges after the change: %+v", boxes) } if loads["box-b"].Hosts != 2 || loads["box-b"].Apps["bab/tea"] != 2 || loads["box-a"].Hosts != 2 { t.Fatalf("loads after the change: %+v", loads) } } ``` `cmd/console/regionha_test.go` (the switch that sets `Config.RegionHA`): ```go package main import "testing" // CARLOS_REGION_HA reaches Config.RegionHA through regionHAEnabled // (strconv.ParseBool): only a true value turns the console's create-time // chooser on; unset, false and garbage leave create as today. func TestRegionHAEnabledParsesTheSwitch(t *testing.T) { for v, want := range map[string]bool{"": false, "0": false, "false": false, "no": false, "garbage": false, "1": true, "t": true, "true": true, "TRUE": true, " true ": true} { got := regionHAEnabled(func(k string) string { if k != "CARLOS_REGION_HA" { t.Fatalf("read %s", k) } return v }) if got != want { t.Errorf("CARLOS_REGION_HA=%q: %v, want %v", v, got, want) } } } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostclaim/ ./internal/hostops/ ./internal/console/ -run 'SelectedExecutor|ExecutorChooser|ClaimLoads|ProductionAuthoritySelects|CreateIsTodays' -count=1` Expected: FAIL to compile: `undefined: ExecutorSelection`, `undefined: BeginCreateOn`, `undefined: ExecutorChooser`, `undefined: ClaimLoads`, `s.executorChooser undefined`. - [ ] **Step 3: Record the selection in the create intent** Apply to `internal/hostclaim/create.go`: ```diff diff --git a/internal/hostclaim/create.go b/internal/hostclaim/create.go index c8fe97cc..39badc04 100644 --- a/internal/hostclaim/create.go +++ b/internal/hostclaim/create.go @@ -80,6 +80,32 @@ func BeginCreate(nh NewHolder, recordBody []byte, opID, by string, now time.Time } } +// ExecutorSelection is the box create selects once, with +// boxchoice.Choose over the eligible set (claim P5-D2, region HA §5.2, +// H12), and its roster enrollment. +type ExecutorSelection struct { + Box, Enrollment string +} + +// BeginCreateOn is BeginCreate recording the selected executor in the +// create intent, in the same step 2 CAS, before any effect (claim §5.1, +// P5-D6 `selected_executor`, `executor_enrollment`). CompleteCreate then +// sets op.exec to exactly that box; recovery never substitutes another. +func BeginCreateOn(nh NewHolder, recordBody []byte, opID, by string, now time.Time, sel ExecutorSelection) func(*Claim) error { + begin := BeginCreate(nh, recordBody, opID, by, now) + return func(c *Claim) error { + if !validBoxReference(sel.Box) || sel.Box == "none" || sel.Enrollment == "" { + return fmt.Errorf("%w: create needs a selected executor", ErrMalformed) + } + if err := begin(c); err != nil { + return err + } + c.Op.Intent["selected_executor"] = sel.Box + c.Op.Intent["executor_enrollment"] = sel.Enrollment + return nil + } +} + func CompleteCreate(opID, execBox string) func(*Claim) error { return func(c *Claim) error { if err := RequireOp(opID)(c); err != nil { @@ -88,6 +114,9 @@ func CompleteCreate(opID, execBox string) func(*Claim) error { if c.Op.Kind != KindCreating || execBox == "" { return fmt.Errorf("%w: kind %q", ErrWrongState, c.Op.Kind) } + if selected := c.Op.Intent["selected_executor"]; selected != "" && execBox != selected { + return fmt.Errorf("%w: create selected %s; completion never substitutes %s", ErrWrongState, selected, execBox) + } if err := requireNoCreateAuthority(c); err != nil { return err } ``` - [ ] **Step 4: The chooser and the loads** `internal/hostops/executor.go`: ```go package hostops import ( "context" "errors" "fmt" "time" "github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/hostclaim" ) // ErrNoEligibleBox is a create refused because no box in the placement is // eligible (region HA §5.2). The console maps it to today's refusal. var ErrNoEligibleBox = errors.New("hostops: region has no box that can take this") // ExecutorChooser is create's executor selection: claim P5-D2's choose() // over the complete eligible set (region HA spec 1 §5.2, H12, L8). It is // made once, before the create CAS, and recorded in the create intent. type ExecutorChooser struct { Directory hostactors.Directory // Edges is one read of every box's edges file with the caller's // liveness verdict per box. An incomplete snapshot selects nothing. Edges func(context.Context) (boxchoice.Snapshot, error) // Loads counts claimed hosts per box and per "acct/app", from the // claim-recovery listing (claim §5.1 step 5). A stale count only // affects balance, never safety. Loads func(context.Context) (map[string]boxchoice.Load, error) SecretsKeyFP string HomeName string Now func() time.Time } // Choose returns the box, and its roster enrollment, that a new host in // r's placement is created on. Identity, placement and AWS region come from // the roster; a box with a valid termination fence, or not active, is never // chosen; a box on the claim's lost list, or owing a park-local chore on // it, is never chosen. func (x ExecutorChooser) Choose(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, error) { o, cands, err := x.eligible(ctx, obs, r) if err != nil { return hostclaim.ExecutorSelection{}, err } loads, err := x.Loads(ctx) if err != nil { return hostclaim.ExecutorSelection{}, err } best, ok := boxchoice.Choose(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App) if !ok { return hostclaim.ExecutorSelection{}, ErrNoEligibleBox } return hostclaim.ExecutorSelection{Box: best.BoxID, Enrollment: o.Roster.Boxes[best.BoxID].Enrollment}, nil } // Permits reports whether box is in the complete eligible set for a host // in r's placement now (spec 1 §2, §5.8 "The start CAS ... refused ... // when `to` is not eligible for the host"): the same read Choose makes, // without choosing. A stale edges file, a missing capability or secrets // key, a draining box, a fence, or an incomplete snapshot refuses. func (x ExecutorChooser) Permits(ctx context.Context, obs hostclaim.Observed, r Record, box string) error { _, cands, err := x.eligible(ctx, obs, r) if err != nil { return err } for _, c := range cands { if c.BoxID == box { return nil } } return fmt.Errorf("%w: %s", ErrNoEligibleBox, box) } // eligible is one read of the roster, the fences and the edges files, // reduced to boxchoice.Eligible's candidates. func (x ExecutorChooser) eligible(ctx context.Context, obs hostclaim.Observed, r Record) (hostactors.Observed, []boxchoice.Candidate, error) { placement := r.Holder.Region if x.Edges == nil || x.Loads == nil || x.Now == nil || placement == "" { return hostactors.Observed{}, nil, ErrContract } o, err := x.Directory.Read(ctx) if err != nil { return hostactors.Observed{}, nil, err } region, ok := o.Roster.Regions[placement] if !ok || region.AWSRegion == "" { return hostactors.Observed{}, nil, ErrContract } roster := map[string]boxchoice.RosterBox{} for ref, b := range o.Roster.Boxes { if b.State != "active" || b.Region != placement { continue } fenced, err := x.Directory.TerminationFence(ctx, o, ref, x.Now()) if err != nil { return hostactors.Observed{}, nil, err // C6: a read error never means "not fenced" } if !fenced { roster[ref] = boxchoice.RosterBox{Placement: b.Region, Cloud: b.Cloud} } } snap, err := x.Edges(ctx) if err != nil { return hostactors.Observed{}, nil, err } // Never a box on the claim's lost list, nor one that still owes a // park-local chore for this host (spec 1 §2: its old installation is // not yet retired). excluded := map[string]bool{} if !obs.Absent && obs.Claim != nil { for _, box := range obs.Claim.Lost { excluded[box] = true } for _, box := range hostclaim.LocalChoreBoxes(obs.Claim) { excluded[box] = true } } return o, boxchoice.Eligible(snap, roster, placement, region.AWSRegion, x.HomeName, x.SecretsKeyFP, excluded), nil } ``` `internal/hostops/loads.go`: ```go package hostops import ( "context" "log" "strings" "github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/release" ) // ClaimLoads counts, from one listing of control/hosts/, the hosts whose // claim names each box as op.exec, in total and per "acct/app": choose()'s // second rule until spec 2 (region HA §5.2, claim P5-D2). The count only // balances, so a claim that cannot be read is logged and left out; a // listing that fails is an error, because a count of nothing would send // every create to one box. func ClaimLoads(ctx context.Context, s release.Store) (map[string]boxchoice.Load, error) { keys, err := s.List(ctx, "control/hosts/") if err != nil { return nil, err } out := map[string]boxchoice.Load{} for _, key := range keys { name, ok := strings.CutPrefix(key, "control/hosts/") host, isJSON := strings.CutSuffix(name, ".json") if !ok || !isJSON || strings.Contains(host, "/") { continue } obs, err := hostclaim.Read(ctx, s, host) if err != nil { log.Printf("hostops: claim loads: %s: %v", host, err) continue } if obs.Absent || obs.Claim.Holder == nil || obs.Claim.Op.Exec == "" || obs.Claim.Op.Exec == "none" { continue } box, h := obs.Claim.Op.Exec, obs.Claim.Holder l := out[box] if l.Apps == nil { l.Apps = map[string]int{} } l.Hosts++ l.Apps[h.Account+"/"+h.App]++ out[box] = l } return out, nil } ``` - [ ] **Step 5: Select before the create CAS, or create as today** Apply to `internal/hostops/operations.go`: ```diff diff --git a/internal/hostops/operations.go b/internal/hostops/operations.go index 8313b80e..d3f1665a 100644 --- a/internal/hostops/operations.go +++ b/internal/hostops/operations.go @@ -14,6 +14,12 @@ import ( // These callbacks are authority providers, not optional best-effort hints. // Production construction waits for the decisions named in this plan. type Authority interface { + // SelectExecutor chooses the new host's executor once, before the + // create CAS (claim P5-D2, region HA §5.2). selected false means this + // deployment does not choose at create (CARLOS_REGION_HA off): the + // create proceeds exactly as today, and Executor names the box at + // completion. Production is ExecutorChooser.Choose behind that switch. + SelectExecutor(context.Context, hostclaim.Observed, Record) (sel hostclaim.ExecutorSelection, selected bool, err error) Create(context.Context, hostclaim.Observed, Record, string) error Executor(context.Context, hostclaim.Observed, Record) (string, error) StartDelete(context.Context, hostclaim.Observed, Record) error @@ -46,11 +52,21 @@ func (c Controller) Attempt(ctx context.Context, before hostclaim.Observed, r Re if before.Host != r.Holder.Host || r.Holder.CreatedAt.IsZero() || id == "" { return hostclaim.Observed{}, ErrIdentity } + // With region HA on, the executor is selected once, before the CAS, + // and recorded in it; off, the create is today's. + sel, selected, err := c.Authority.SelectExecutor(ctx, before, r) + if err != nil { + return hostclaim.Observed{}, err + } + begin := hostclaim.BeginCreate(r.Holder, r.Body, id, by, c.Now()) + if selected { + begin = hostclaim.BeginCreateOn(r.Holder, r.Body, id, by, c.Now(), sel) + } // Last pre-CAS read of registration, open admission and purge reservation. if err := c.Authority.Create(ctx, before, r, id); err != nil { return hostclaim.Observed{}, err } - obs, err := hostclaim.Update(ctx, c.Store, before, hostclaim.BeginCreate(r.Holder, r.Body, id, by, c.Now())) + obs, err := hostclaim.Update(ctx, c.Store, before, begin) if err != nil { return hostclaim.Observed{}, err } ``` Apply to `internal/hostops/backfill_controller.go` (the backfill grants no create): ```diff diff --git a/internal/hostops/backfill_controller.go b/internal/hostops/backfill_controller.go index 08ebd856..dcbf750e 100644 --- a/internal/hostops/backfill_controller.go +++ b/internal/hostops/backfill_controller.go @@ -23,6 +23,9 @@ func (a BackfillAuthority) Adoption(ctx context.Context, r Record) (string, stri } func (a BackfillAuthority) ClaimBackfill() *Backfill { return a.p.Backfill } +func (a BackfillAuthority) SelectExecutor(context.Context, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, bool, error) { + return hostclaim.ExecutorSelection{}, false, nil // the backfill grants no create (Create refuses) +} func (a BackfillAuthority) Create(context.Context, hostclaim.Observed, Record, string) error { return ErrContract } ``` Apply to `internal/hostops/backfill.go`: ```diff diff --git a/internal/hostops/backfill.go b/internal/hostops/backfill.go index e20b1982..7c727f9d 100644 --- a/internal/hostops/backfill.go +++ b/internal/hostops/backfill.go @@ -89,6 +89,22 @@ type ProductionAuthority struct { Signer release.Signer Evidence hostactors.Evidence Backfill *Backfill + // Executors is create's executor selection (region HA §5.2, L8). The + // create door sets it only with CARLOS_REGION_HA on; nil keeps today's + // create path, so a claim-enabled deployment with HA off (every one-box + // region included) creates exactly as before. + Executors *ExecutorChooser +} + +// SelectExecutor is Authority's create-time selection: choose() once, +// before the create CAS, when a chooser is configured; otherwise no +// selection, and the create is today's. +func (a ProductionAuthority) SelectExecutor(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, bool, error) { + if a.Executors == nil { + return hostclaim.ExecutorSelection{}, false, nil + } + sel, err := a.Executors.Choose(ctx, obs, r) + return sel, err == nil, err } func reportKey(reservation, box string) (string, error) { ``` Apply to `internal/hostops/operations_test.go` (the fake authority implements the new method; `TestCreateIsTodaysWithoutASelection` pins dormancy and, rev 10, drives `Attempt` with a selection that differs from what completion offers): ```diff diff --git a/internal/hostops/operations_test.go b/internal/hostops/operations_test.go index 42ebf9d1..f37e17c3 100644 --- a/internal/hostops/operations_test.go +++ b/internal/hostops/operations_test.go @@ -15,10 +15,32 @@ import ( "github.com/carlosframework/platform/internal/release" ) -type authority struct{ err error } +type authority struct { + err error + // selects is region HA's create-time choice; false is today's create. + // selected is the box it chooses (box-a when empty); executor is the + // box completion offers (box-a when empty); selections counts calls. + selects bool + selected, executor string + selections int +} +func (a *authority) SelectExecutor(context.Context, hostclaim.Observed, Record) (hostclaim.ExecutorSelection, bool, error) { + a.selections++ + if !a.selects { + return hostclaim.ExecutorSelection{}, false, a.err + } + box := a.selected + if box == "" { + box = "box-a" + } + return hostclaim.ExecutorSelection{Box: box, Enrollment: "enrollment-" + box[len("box-"):]}, true, a.err +} func (a *authority) Create(context.Context, hostclaim.Observed, Record, string) error { return a.err } func (a *authority) Executor(context.Context, hostclaim.Observed, Record) (string, error) { + if a.executor != "" { + return a.executor, a.err + } return "box-a", a.err } func (a *authority) StartDelete(context.Context, hostclaim.Observed, Record) error { return a.err } @@ -138,6 +160,47 @@ func TestAttemptUsesPreRegistrationObservation(t *testing.T) { t.Fatal("deleted claim") } } + +// TestCreateIsTodaysWithoutASelection pins dormancy (region HA §8 step 2): +// with no create-time choice the claim carries no selected executor and +// completion takes the box Executor names. With one, Attempt itself +// records the selection in the create CAS (round 9 finding 2): selection +// chooses box-b while completion offers box-a, so the completion refuses, +// the persisted intent names box-b and its enrollment, and a completion +// with box-b serves without selecting again. +func TestCreateIsTodaysWithoutASelection(t *testing.T) { + f := newFixture(t) + obs := f.serving(t, "off.bab.example.com") + if obs.Claim.Op.Exec != "box-a" || obs.Claim.Op.Intent["selected_executor"] != "" || obs.Claim.Op.Intent["executor_enrollment"] != "" { + t.Fatalf("HA off: %+v", obs.Claim.Op) + } + f.auth.selects, f.auth.selected, f.auth.selections = true, "box-b", 0 + const host = "on.bab.example.com" + before, err := hostclaim.Read(f.ctx, f.store, host) + if err != nil { + t.Fatal(err) + } + id := f.c.NewID() + if _, err := f.c.Attempt(f.ctx, before, f.record(t, host), id, "test"); !errors.Is(err, hostclaim.ErrWrongState) { + t.Fatalf("completion substituted box-a for the selected box-b: %v", err) + } + pending, err := hostclaim.Read(f.ctx, f.store, host) + if err != nil { + t.Fatal(err) + } + if pending.Claim.Op.Kind != hostclaim.KindCreating || pending.Claim.Op.Intent["selected_executor"] != "box-b" || pending.Claim.Op.Intent["executor_enrollment"] != "enrollment-b" { + t.Fatalf("HA on, the persisted intent: %+v", pending.Claim.Op) + } + f.auth.executor = "box-b" + done, err := f.c.Complete(f.ctx, host, id) + if err != nil || done.Claim.Op.Kind != hostclaim.KindServing || done.Claim.Op.Exec != "box-b" { + t.Fatalf("completion with the selected box: %+v %v", done.Claim, err) + } + if f.auth.selections != 1 { + t.Fatalf("selection ran %d times, want once", f.auth.selections) + } +} + func TestRecoverCreateAbsentAndLateOrphan(t *testing.T) { f := newFixture(t) host := "shop.bab.example.com" ``` Apply to `internal/console/server.go` (`Config.RegionHA`): ```diff diff --git a/internal/console/server.go b/internal/console/server.go index ea7f7a7c..8f89bd8e 100644 --- a/internal/console/server.go +++ b/internal/console/server.go @@ -493,6 +493,11 @@ type Config struct { // struct. "" is legal — a local CARLOS_DEPLOYMENT_DIR deployment has no // bucket name to draw — and simply draws no bucket node. Bucket string + // RegionHA is CARLOS_REGION_HA as strconv.ParseBool reads it (region HA + // spec 1 §8 step 2): with it off the console chooses no executor at + // create and relocation has no console half. A deployment fact, on + // Config for the same reason Bucket is. + RegionHA bool // HomeRegion NAMES the deployment's home placement (CONSOLE_HOME_REGION, // e.g. "ie") for consumer pickers (issue #346). It is a NAME, not a // scope: records for the home keep region "" (the unconstrained value ``` Apply to `cmd/console/main.go`: ```diff diff --git a/cmd/console/main.go b/cmd/console/main.go index 76d01905..ad7af881 100644 --- a/cmd/console/main.go +++ b/cmd/console/main.go @@ -302,6 +302,7 @@ func serve() { // correctly draws no bucket node. Bucket: os.Getenv("CARLOS_DEPLOYMENT_BUCKET"), HomeRegion: strings.TrimSpace(os.Getenv("CONSOLE_HOME_REGION")), + RegionHA: regionHAEnabled(os.Getenv), AppRegions: appRegionsSwitch(os.Getenv("CONSOLE_APP_REGIONS")), // GeoIP (geoip spec §11.3): the CONSOLE_GEOIP_DB override when set, // otherwise the follower of this box's own edge mirror. @@ -1341,3 +1342,11 @@ func geoLocator(sw *geo.Swappable) geo.Locator { } return sw } + +// regionHAEnabled is CARLOS_REGION_HA as strconv.ParseBool reads it: on +// for "1", "t", "true" in any case, off for "", "0", "false" and anything +// unparseable. The same rule as the box's (internal/host/boxrecord.go). +func regionHAEnabled(getenv func(string) string) bool { + on, err := strconv.ParseBool(strings.TrimSpace(getenv("CARLOS_REGION_HA"))) + return err == nil && on +} ``` `internal/console/hostclaimexecutor.go`: ```go package console import ( "context" "time" "github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/hostops" ) // executorChooser is create's executor choice (region HA §5.2, L8) over // console's live edges read and the claim listing's loads. The create // door's hostops.ProductionAuthority uses it as Executors once that door // lands; nothing calls it before then. With Config.RegionHA off it is nil, // and the create path is today's (spec §8 step 2: dormant). func (s *Server) executorChooser(d hostactors.Directory) *hostops.ExecutorChooser { if !s.cfg.RegionHA { return nil } return &hostops.ExecutorChooser{Directory: d, Edges: func(ctx context.Context) (boxchoice.Snapshot, error) { snap, err := s.liveEdges(ctx) if err != nil { return boxchoice.Snapshot{}, err } return boxSnapshot(snap), nil }, Loads: func(ctx context.Context) (map[string]boxchoice.Load, error) { return hostops.ClaimLoads(ctx, s.cfg.Releases) }, SecretsKeyFP: s.expectedSecretsFP(), HomeName: s.cfg.HomeRegion, Now: time.Now} } ``` - [ ] **Step 6: Run the tests** Run: `nice -n 19 go test ./internal/hostclaim/ ./internal/hostops/ -count=1 && nice -n 19 go test ./internal/console/ -run 'CreateExecutorChooser|Claim|BoxChoice' -count=1 && nice -n 19 go test ./cmd/console/ -run RegionHA -count=1 && go vet ./internal/console/ ./internal/host/ ./cmd/console/` Expected: PASS; vet clean (no other `hostops.Authority` implementation exists on main). - [ ] **Step 7: Commit** ```bash git add internal/hostclaim/create.go internal/hostclaim/create_test.go internal/hostops/ internal/console/hostclaimexecutor.go internal/console/hostclaimexecutor_test.go internal/console/server.go cmd/console/main.go cmd/console/regionha_test.go git commit -m "hostops: with region HA on, create chooses its executor once with boxchoice over the roster and records it in the create intent Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 3: Reconcile's row cases and who may wake (§5.3, L12) `RelocationRow` is §5.3's table as a pure decision over a fresh claim read: ready on the serving executor and on the source until its admission fence closes; held with `hold_op` = the operation for any other participant; retiring (held, never minted) for a box owing a `park-local`; nothing, or an unexplained held row, otherwise. A box on `lost` is never ready. `WakePhase` gains `relocating`: the source wakes in `starting` and `releasing`, the target only probes, in `probing`, and a stuck operation runs nothing. `Policy` routes relocating actions to its own owner, never to moves', and admits one new action, `reconcile`, for any participant of the operation that is not lost, executor or not: the hold write Task 9 makes for a held row (finding 14). Every other action keeps the executor rule. **Files:** - Create: `internal/hostbox/relocationrows.go`, `internal/hostbox/relocationrows_test.go` - Modify: `internal/hostbox/wakepolicy.go` **Interfaces:** - Consumes: Task 1's phase constants. - Produces: `hostbox.RowCase` (`RowNone`, `RowReady`, `RowHeld`, `RowRetiring`, `RowUnexplained`), `hostbox.RowDecision{Case, HoldOp}`, `hostbox.RelocationRow(c *hostclaim.Claim, box string, exists, fenceClosed bool) RowDecision`; `hostbox.Policy.Relocation func(ctx, *hostclaim.Claim, Selection, hostwrite.Identity) error`, `hostbox.RelocationPolicy(box string)` (its production value); `Policy.Check`'s `reconcile` and `probe` actions, and the serving executor's `reconcile` through an obsolete hold (only with `Relocation` set); `hostguard.Guard.AdmissionFenced(host string, c *hostclaim.Claim, Captured) (bool, error)`, lock-free. - [ ] **Step 1: Write the failing tests** `internal/hostbox/relocationrows_test.go`: ```go package hostbox import ( "context" "errors" "testing" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostwrite" ) func relocating(phase, exec string) *hostclaim.Claim { return &hostclaim.Claim{Holder: &hostclaim.Holder{}, Op: hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: phase, Exec: exec, Participants: []string{"src", "dst"}, Intent: map[string]string{"source_box": "src", "target_box": "dst", "region": "ie"}}} } func TestRelocationRowCases(t *testing.T) { withChore := func(c *hostclaim.Claim, box string) *hostclaim.Claim { c.Obligations = append(c.Obligations, hostclaim.Obligation{ID: "o", Op: "rel-0", Kind: "park-local", Box: box, Owner: "box:" + box, Retirement: "r"}) return c } serving := &hostclaim.Claim{Holder: &hostclaim.Holder{}, Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "src", Participants: []string{"src"}}} lostSource := relocating(hostclaim.PhaseStarting, "src") lostSource.Lost = []string{"src"} for _, tc := range []struct { name string c *hostclaim.Claim box string exists, fenceClosed bool want RowDecision }{ {"the serving executor", serving, "src", false, false, RowDecision{Case: RowReady}}, {"a serving host elsewhere, no row", serving, "dst", false, false, RowDecision{Case: RowNone}}, {"a serving host elsewhere, a row", serving, "dst", true, false, RowDecision{Case: RowUnexplained}}, {"the source in starting", relocating(hostclaim.PhaseStarting, "src"), "src", true, false, RowDecision{Case: RowReady}}, {"the source releasing, fence open", relocating(hostclaim.PhaseReleasing, "src"), "src", true, false, RowDecision{Case: RowReady}}, {"the source releasing, fence closed", relocating(hostclaim.PhaseReleasing, "src"), "src", true, true, RowDecision{Case: RowHeld, HoldOp: "rel-1"}}, {"the source in pinning", relocating(hostclaim.PhasePinning, "none"), "src", true, false, RowDecision{Case: RowHeld, HoldOp: "rel-1"}}, {"the target in starting", relocating(hostclaim.PhaseStarting, "src"), "dst", false, false, RowDecision{Case: RowHeld, HoldOp: "rel-1"}}, {"the target executing in probing", relocating(hostclaim.PhaseProbing, "dst"), "dst", true, false, RowDecision{Case: RowHeld, HoldOp: "rel-1"}}, {"a lost source", lostSource, "src", true, false, RowDecision{Case: RowUnexplained}}, {"a former executor owing park-local", withChore(serving, "old"), "old", true, false, RowDecision{Case: RowRetiring, HoldOp: "rel-0"}}, {"a former executor with no row", withChore(serving, "old"), "old", false, false, RowDecision{Case: RowNone}}, {"no claim, a row", nil, "src", true, false, RowDecision{Case: RowUnexplained}}, } { t.Run(tc.name, func(t *testing.T) { if got := RelocationRow(tc.c, tc.box, tc.exists, tc.fenceClosed); got != tc.want { t.Fatalf("got %+v, want %+v", got, tc.want) } }) } } func TestWakePhaseRelocating(t *testing.T) { for _, tc := range []struct { phase, exec, box string probe, want bool }{ {hostclaim.PhaseStarting, "src", "src", false, true}, {hostclaim.PhaseReleasing, "src", "src", false, true}, {hostclaim.PhaseStarting, "src", "src", true, false}, {hostclaim.PhaseInstalling, "dst", "dst", false, false}, {hostclaim.PhaseInstalled, "dst", "dst", true, false}, {hostclaim.PhaseProbing, "dst", "dst", true, true}, {hostclaim.PhaseProbing, "dst", "dst", false, false}, {hostclaim.PhaseProbeFailed, "dst", "dst", true, false}, {hostclaim.PhasePinning, "none", "src", false, false}, } { if got := WakePhase(relocating(tc.phase, tc.exec), tc.box, tc.probe) == nil; got != tc.want { t.Fatalf("%+v: %v", tc, got) } } stuck := relocating(hostclaim.PhaseProbing, "dst") stuck.Op.Journal = map[string]string{"stuck": "operator"} if WakePhase(stuck, "dst", true) == nil { t.Fatal("a stuck operation probed") } } func TestPolicyRoutesRelocationToItsOwner(t *testing.T) { c := relocating(hostclaim.PhaseStarting, "src") held := func(context.Context, string) (string, error) { return "", nil } s := Selection{Host: "x.example", Box: "src", Op: "rel-1", Action: "spawn"} moves := func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error { return nil } if err := (Policy{Held: held, Moves: moves}).Check(context.Background(), c, s, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { t.Fatalf("moves' policy authorized a relocation: %v", err) } called := false reloc := func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error { called = true return nil } if err := (Policy{Held: held, Relocation: reloc}).Check(context.Background(), c, s, hostwrite.Identity{}); err != nil || !called { t.Fatalf("relocation policy: %v called=%v", err, called) } } func TestPolicyLetsTheServingExecutorClearAnObsoleteHold(t *testing.T) { serving := &hostclaim.Claim{Holder: &hostclaim.Holder{}, Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "dst", Participants: []string{"dst"}}} obsolete := func(context.Context, string) (string, error) { return "rel-1", nil } reconcile := Selection{Host: "x.example", Box: "dst", Action: "reconcile"} // Region HA off (no Relocation policy): today's refusal, unchanged. if err := (Policy{Held: obsolete}).Check(context.Background(), serving, reconcile, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { t.Fatalf("with HA off a held serving row was reconciled: %v", err) } policy := Policy{Held: obsolete, Relocation: RelocationPolicy("dst")} if err := policy.Check(context.Background(), serving, reconcile, hostwrite.Identity{}); err != nil { t.Fatalf("the serving executor could not reconcile an obsolete hold away: %v", err) } for name, s := range map[string]Selection{ "a spawn under the hold": {Host: "x.example", Box: "dst", Action: "spawn"}, "another box's reconcile": {Host: "x.example", Box: "src", Action: "reconcile"}, } { if err := policy.Check(context.Background(), serving, s, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { t.Fatalf("%s: %v", name, err) } } } func TestRelocationPolicyGivesEachActionItsRole(t *testing.T) { policy := RelocationPolicy("dst") at := func(phase, exec string) *hostclaim.Claim { return relocating(phase, exec) } for name, tc := range map[string]struct { c *hostclaim.Claim action string want error }{ "a participant reconciles": {at(hostclaim.PhaseStarting, "src"), "reconcile", nil}, "a non-executor's held config": {at(hostclaim.PhaseStarting, "src"), "held-config", ErrChanged}, "the executor's held config": {at(hostclaim.PhaseInstalling, "dst"), "held-config", nil}, "the executor installs": {at(hostclaim.PhaseInstalled, "dst"), "install", nil}, "the target spawns in installing": {at(hostclaim.PhaseInstalling, "dst"), "spawn", ErrChanged}, "the target spawns in probing": {at(hostclaim.PhaseProbing, "dst"), "spawn", ErrChanged}, // only a probe "the target probes in probing": {at(hostclaim.PhaseProbing, "dst"), "probe", nil}, "the target probes in installed": {at(hostclaim.PhaseInstalled, "dst"), "probe", ErrChanged}, "the source probes": {at(hostclaim.PhaseProbing, "src"), "probe", ErrChanged}, "an unknown action": {at(hostclaim.PhaseProbing, "dst"), "purge", ErrChanged}, } { if err := policy(context.Background(), tc.c, Selection{Host: "x.example", Box: "dst", Op: "rel-1", Action: tc.action}, hostwrite.Identity{}); !errors.Is(err, tc.want) { t.Fatalf("%s: %v, want %v", name, err, tc.want) } } if err := policy(context.Background(), at(hostclaim.PhaseStarting, "src"), Selection{Host: "x.example", Box: "src", Op: "rel-1", Action: "reconcile"}, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { t.Fatalf("another box's selection: %v", err) } } func TestPolicyLetsAHeldParticipantReconcileItsRow(t *testing.T) { held := func(context.Context, string) (string, error) { return "rel-1", nil } reloc := func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error { return nil } policy := Policy{Held: held, Relocation: reloc} row := func(box, op string) Selection { return Selection{Host: "x.example", Box: box, Op: op, Action: "reconcile"} } c := relocating(hostclaim.PhaseStarting, "src") for name, s := range map[string]Selection{"the target, not the executor": row("dst", "rel-1"), "the source": row("src", "rel-1")} { if err := policy.Check(context.Background(), c, s, hostwrite.Identity{}); err != nil { t.Fatalf("%s: %v", name, err) } } lost := relocating(hostclaim.PhaseStarting, "src") lost.Lost = []string{"src"} for name, tc := range map[string]struct { c *hostclaim.Claim s Selection }{ "a box with no role": {c, row("zzz", "rel-1")}, "another operation": {c, row("dst", "rel-0")}, "a lost participant": {lost, row("src", "rel-1")}, "a spawn by the target": {c, Selection{Host: "x.example", Box: "dst", Op: "rel-1", Action: "spawn"}}, } { if err := policy.Check(context.Background(), tc.c, tc.s, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { t.Fatalf("%s: %v", name, err) } } // A target probe passes the whole policy; its ordinary spawn does not. probing := relocating(hostclaim.PhaseProbing, "dst") full := Policy{Held: held, Relocation: RelocationPolicy("dst")} if err := full.Check(context.Background(), probing, Selection{Host: "x.example", Box: "dst", Op: "rel-1", Action: "probe"}, hostwrite.Identity{}); err != nil { t.Fatalf("the target's probe: %v", err) } if err := full.Check(context.Background(), probing, Selection{Host: "x.example", Box: "dst", Op: "rel-1", Action: "spawn"}, hostwrite.Identity{}); !errors.Is(err, ErrChanged) { t.Fatalf("an ordinary spawn by the probing target: %v", err) } moving := &hostclaim.Claim{Holder: &hostclaim.Holder{}, Op: hostclaim.Op{Kind: hostclaim.KindMoving, ID: "mv-1", Phase: "sealed", Exec: "src", Participants: []string{"src", "dst"}}} if err := (Policy{Held: held, Moves: reloc, Relocation: reloc}).Check(context.Background(), moving, row("dst", "mv-1"), hostwrite.Identity{}); !errors.Is(err, ErrChanged) { t.Fatalf("a move's participant reconciled through the relocation action: %v", err) } } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostbox/ -run 'RelocationRow|WakePhaseRelocating|PolicyRoutesRelocation|PolicyLetsAHeldParticipant' -count=1` Expected: FAIL to compile: `undefined: RelocationRow`, `unknown field Relocation in struct literal`. - [ ] **Step 3: The row decision** `internal/hostbox/relocationrows.go`: ```go package hostbox import ( "slices" "github.com/carlosframework/platform/internal/hostclaim" ) // RowCase is what reconcile does with one relocatable host's registry row // on this box (region HA spec 1 §5.3, claim C4 as amended by L12). type RowCase int const ( // RowNone: mint nothing, and there is no row to keep. RowNone RowCase = iota // RowReady: mint the row, or keep it, ready. RowReady // RowHeld: mint the row, or keep it, held with HoldOp. RowHeld // RowRetiring: keep an existing row held; mint none. A park-local chore // this box owes retires it under the host guard (claim P6-D2). RowRetiring // RowUnexplained: the claim gives this box no role, yet a row exists. // Keep it held, never serve it, and alert. Never delete it by hostname. RowUnexplained ) // RowDecision is RelocationRow's answer. type RowDecision struct { Case RowCase HoldOp string } // RelocationRow decides one row from a fresh claim read. box is this box's // reference; exists says whether the registry holds a row for the host; // fenceClosed says whether this box's local admission fence for the host is // closed (the source's release has begun, §5.8's release sequence step 1). // // op.exec, serving ready // relocating source in starting, or releasing // until its admission fence closes ready // any other participant of a running operation held, hold_op = op id // owner of an open park-local chore for the host retiring (held, no mint) // none of these none; an existing row is unexplained // // A box on the claim's lost list is never ready. A nil or holderless claim // (an error is the caller's: claim C6, keep rows and mint none) gives none. func RelocationRow(c *hostclaim.Claim, box string, exists, fenceClosed bool) RowDecision { if c == nil || c.Holder == nil || box == "" { return unexplained(exists) } lost := slices.Contains(c.Lost, box) switch { case lost: case c.Op.Kind == hostclaim.KindServing && c.Op.ID == "" && c.Op.Exec == box: return RowDecision{Case: RowReady} case c.Op.Kind == hostclaim.KindRelocating && c.Op.Exec == box && c.Op.Intent["source_box"] == box && (c.Op.Phase == hostclaim.PhaseStarting || (c.Op.Phase == hostclaim.PhaseReleasing && !fenceClosed)): return RowDecision{Case: RowReady} case c.Op.Kind != hostclaim.KindServing && c.Op.ID != "" && slices.Contains(c.Op.Participants, box): return RowDecision{Case: RowHeld, HoldOp: c.Op.ID} } for _, o := range c.Obligations { if o.Kind == "park-local" && o.Box == box { if !exists { return RowDecision{Case: RowNone} } return RowDecision{Case: RowRetiring, HoldOp: o.Op} } } return unexplained(exists) } func unexplained(exists bool) RowDecision { if exists { return RowDecision{Case: RowUnexplained} } return RowDecision{Case: RowNone} } ``` - [ ] **Step 4: Wake phase and policy** Apply to `internal/hostbox/wakepolicy.go`: ```diff diff --git a/internal/hostbox/wakepolicy.go b/internal/hostbox/wakepolicy.go index 9c3c00d6..5dec10df 100644 --- a/internal/hostbox/wakepolicy.go +++ b/internal/hostbox/wakepolicy.go @@ -2,6 +2,7 @@ package hostbox import ( "context" + "slices" "time" "github.com/carlosframework/platform/internal/hostclaim" @@ -27,6 +28,17 @@ func WakePhase(c *hostclaim.Claim, box string, probe bool) error { if c.Op.ID != "" && ((!probe && c.Op.Phase == "committed") || (probe && c.Op.Phase == "probing")) { return nil } + case hostclaim.KindRelocating: + // Region HA §5.8: the source keeps serving in starting and in + // releasing (its local admission fence closes releasing); the + // target runs only probes, in probing. Nobody else, ever. + if c.Op.ID == "" || c.Op.Journal["stuck"] != "" { + break + } + source := c.Op.Intent["source_box"] == box && (c.Op.Phase == hostclaim.PhaseStarting || c.Op.Phase == hostclaim.PhaseReleasing) + if (!probe && source) || (probe && c.Op.Phase == hostclaim.PhaseProbing) { + return nil + } case hostclaim.KindReverting, hostclaim.KindRestoring: if c.Op.ID != "" && ((!probe && (c.Op.Phase == "probed" || c.Op.Phase == "opened")) || (probe && c.Op.Phase == "probing")) { return nil @@ -37,10 +49,13 @@ func WakePhase(c *hostclaim.Claim, box string, probe bool) error { // Policy separates ordinary writes from the moves-owned held actions. // Moves validates roles, held inputs and phase-specific rights with the same capture. +// Relocation does the same for a relocating claim (region HA §5.8); with it +// nil, every relocating action is refused. type Policy struct { - Inputs func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error - Moves func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error - Held func(context.Context, string) (string, error) + Inputs func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error + Moves func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error + Relocation func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error + Held func(context.Context, string) (string, error) } func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id hostwrite.Identity) error { @@ -61,14 +76,43 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h } } if hold != "" { - return ErrChanged + // A hold of an operation the claim no longer runs is obsolete. + // Only the serving executor's reconcile may pass it, and only + // to clear it (round 3 finding N4), and only where region HA + // is on (Relocation set): with it off, today's refusal. + if p.Relocation == nil || s.Action != "reconcile" || hold == c.Op.ID || c.Op.ID != "" { + return ErrChanged + } } return Serving(c, s) } - if p.Moves == nil || s.Op == "" || c.Op.ID != s.Op || c.Op.Exec != s.Box { + owner := p.Moves + if c.Op.Kind == hostclaim.KindRelocating { + owner = p.Relocation + } + if s.Action == "reconcile" { + // Region HA §5.3: reconcile mints or keeps a held row for any + // participant of a running relocation, executor or not, never + // for a lost box. The row's hold is the operation's; the owner + // policy permits the hold write and nothing else for a + // non-executor. Other kinds keep refusing the action here. + if c.Op.Kind != hostclaim.KindRelocating || owner == nil || s.Op == "" || c.Op.ID != s.Op || + !slices.Contains(c.Op.Participants, s.Box) || slices.Contains(c.Lost, s.Box) { + return ErrChanged + } + return owner(ctx, c, s, id) + } + if owner == nil || s.Op == "" || c.Op.ID != s.Op || c.Op.Exec != s.Box { return ErrChanged } switch s.Action { + case "probe": + // A relocation target's probe spawn (§5.8, R1): selected as its own + // action, so the policy can give it probe authority where an + // ordinary spawn has none. + if c.Op.Kind != hostclaim.KindRelocating || (hold != "" && hold != s.Op) { + return ErrChanged + } case "held-config", "held-release", "held-timers": if hold != s.Op { return ErrChanged @@ -84,7 +128,37 @@ func (p Policy) Check(ctx context.Context, c *hostclaim.Claim, s Selection, id h default: return ErrChanged } - return p.Moves(ctx, c, s, id) + return owner(ctx, c, s, id) +} + +// RelocationPolicy is the Policy.Relocation owner for one box: the role +// and phase rights of each action under a relocating claim, after +// Policy.Check has matched the operation and the participant. A +// reconcile is the hold write any participant makes; held inputs are the +// executor's; execution actions are what WakePhase allows the executor, +// and "probe" is the target's probe spawn in probing (WakePhase with +// probe set), which an ordinary spawn never gets. Production sets it with +// CARLOS_REGION_HA on (Task 10); nil refuses. +func RelocationPolicy(box string) func(context.Context, *hostclaim.Claim, Selection, hostwrite.Identity) error { + return func(_ context.Context, c *hostclaim.Claim, s Selection, _ hostwrite.Identity) error { + if box == "" || s.Box != box || c.Op.Kind != hostclaim.KindRelocating { + return ErrChanged + } + switch s.Action { + case "reconcile": + return nil + case "held-config", "held-release", "held-timers", "install", "restore": + if c.Op.Exec != box { + return ErrChanged + } + return nil + case "spawn", "seed", "writer": + return WakePhase(c, box, false) + case "probe": + return WakePhase(c, box, true) + } + return ErrChanged + } } const RequestLimit = 120 * time.Second ``` - [ ] **Step 4b: The admission-fence reader** `internal/hostguard/admissionfence.go` (the reading `Reconciler.FenceClosed` is wired to in Task 10; it grants nothing): ```go package hostguard import "github.com/carlosframework/platform/internal/hostclaim" // AdmissionFenced reports whether the host's durable admission fence is // closed for the installation and session captured names on claim c: a // stop root began (ActiveFence) or completed (a FenceJournal receipt) for // that session. It is the same reading With refuses on, exposed for // region HA §5.3's row decision ("the source in releasing until its // admission fence closes"). It takes no lock: the record is replaced // atomically, and callers already hold the host lock (Activate's reader, // WithLocal) or need no lock for one consistent read. The claim is the // caller's fresh read. It grants nothing. func (g Guard) AdmissionFenced(host string, c *hostclaim.Claim, captured Captured) (bool, error) { if c == nil || c.Holder == nil { return false, ErrNoHolder } r, err := readRecord(g.DataDir, host) if err != nil { return false, err } if r.ActiveFence != nil && r.ActiveFence.matchesSession(c, captured) { return true, nil } for _, receipt := range r.FenceJournal { if receipt.Target.matchesSession(c, captured) { return true, nil } } return false, nil } ``` `internal/hostguard/admissionfence_test.go`: ```go //go:build unix package hostguard import ( "errors" "testing" ) func TestAdmissionFencedFollowsTheDurableFence(t *testing.T) { f, target, captured := fenceFixture(t) if fenced, err := f.g.AdmissionFenced(target.Host, f.obs.Claim, captured); err != nil || fenced { t.Fatalf("before any fence: %v %v", fenced, err) } if _, err := f.g.BeginFence(t.Context(), target); err != nil { t.Fatal(err) } // Read inside the host lock, as Activate's reader and WithLocal do: // no second acquisition, no deadlock. err := f.g.WithLocal(t.Context(), target.Host, func() error { fenced, err := f.g.AdmissionFenced(target.Host, f.obs.Claim, captured) if err != nil || !fenced { return errors.New("after BeginFence, under the lock: not fenced") } return nil }) if err != nil { t.Fatal(err) } other := captured other.Stamp++ if fenced, err := f.g.AdmissionFenced(target.Host, f.obs.Claim, other); err != nil || fenced { t.Fatalf("another installation: %v %v", fenced, err) } if _, err := f.g.AdmissionFenced("no.such.host", f.obs.Claim, captured); !errors.Is(err, ErrNoRecord) { t.Fatalf("a host with no record: %v", err) } } ``` - [ ] **Step 5: Run the tests** Run: `nice -n 19 go test ./internal/hostbox/ ./internal/hostguard/ ./internal/activator/ -run 'Relocation|WakePhase|Policy|ClaimWake|Wake|AdmissionFenced' -count=1` Expected: PASS (`activator`'s claim wake calls `WakePhase`; serving and moves cases are unchanged). - [ ] **Step 6: Commit** ```bash git add internal/hostbox/relocationrows.go internal/hostbox/relocationrows_test.go internal/hostbox/wakepolicy.go internal/hostguard/admissionfence.go internal/hostguard/admissionfence_test.go git commit -m "hostbox: reconcile's relocation row cases, and only the source or a probing target wakes under relocating Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 4: Writers stopped, with the replicator still open (L10) Claim §6.1, "Release before full stop": release step 1 needs the admission fence, settled spawns and drained work (the drain's own facts), an empty cgroup with no members, and a complete descriptor scan in which the only holders of the database, WAL and SHM files are descriptors positively identified as the replicator's: its process (PID and start time, field 22 of `/proc//stat`) and each descriptor number and target. A descriptor seen in a thread whose table the leader's does not match is never exempt. *Rev 7:* that match is the kernel's word, never a number and a path. kcmp(2) `KCMP_FILES` must show the thread shares the leader's table, and `KCMP_FILE` must show the descriptor is the leader's open file description. A comparison that cannot be made fails closed as `ErrIncomplete`. `Collect` is unchanged: full stop evidence follows once the replicator closes. **Files:** - Create: `internal/stopevidence/writers.go`, `internal/stopevidence/writers_linux.go`, `internal/stopevidence/writers_other.go` - Test: `internal/stopevidence/writers_linux_test.go` **Interfaces:** - Consumes: `realFDOps`, `readProcDir`, `procGone`, `ErrIncomplete`, `ErrInvalidInput`, `ErrUnsupported`, `ErrNoCgroupEvidence`, `ErrPopulated`, `ErrMembers` (all on main); `unix.SYS_KCMP`, `unix.Syscall6`, `unix.Unshare`, `unix.CLONE_FILES`, `unix.Dup3` (golang.org/x/sys v0.46.0, already a dependency). - Produces: `stopevidence.Descriptor{PID, FD int; Target string}`, `stopevidence.WritersInputs`, `stopevidence.WritersEvidence`, `stopevidence.WritersStopped(ctx, WritersInputs) (WritersEvidence, error)`, `stopevidence.ProcessDescriptors(ctx, procRoot string, pid int, dbs []string) ([]Descriptor, error)` (one process's descriptors on the database, reading only `/proc/`, taken by the agent for its own pid before and after the replicator opens: R3's agent half), `ErrWorkOpen`, `ErrForeignHolder`. `WritersStopped`'s complete scan of every process is root's (ruling 22). - [ ] **Step 1: Write the failing tests** `internal/stopevidence/writers_linux_test.go` (it reuses `evidenceGroup` from `evidence_linux_test.go`): ```go //go:build linux package stopevidence import ( "errors" "os" "os/exec" "path/filepath" "runtime" "slices" "strconv" "strings" "testing" "golang.org/x/sys/unix" ) // fixtureUnshared names the fixture threads that do not share their // leader's descriptor table; fixtureKcmpErr makes every comparison fail. var ( fixtureUnshared map[int]bool fixtureKcmpErr error ) // fixtureKcmp answers kcmp(2) for a /proc fixture as the kernel would: a // thread shares the leader's table unless fixtureUnshared names it, and a // shared table's descriptor n is the leader's n; a descriptor missing from // either table is EBADF. func fixtureKcmp(root string) kcmpFunc { return func(pid, tid, typ int, idx1, idx2 uintptr) (int, error) { if fixtureKcmpErr != nil { return 0, fixtureKcmpErr } switch typ { case kcmpFiles: if fixtureUnshared[tid] { return 1, nil } return 0, nil case kcmpFile: fd := func(task int, n uintptr) (string, error) { return os.Readlink(filepath.Join(root, strconv.Itoa(pid), "task", strconv.Itoa(task), "fd", strconv.Itoa(int(n)))) } a, errA := fd(pid, idx1) b, errB := fd(tid, idx2) if errA != nil || errB != nil { return 0, unix.EBADF } if fixtureUnshared[tid] || a != b { return 1, nil } return 0, nil } return 0, unix.EINVAL } } // writersProc builds a /proc fixture: the agent (pid 20, threads 20 and // 7, sharing the table) holds the database on fds 5 and 6. func writersProc(t *testing.T) (string, string) { t.Helper() root, db := t.TempDir(), filepath.Join(t.TempDir(), "data.db") fixtureUnshared, fixtureKcmpErr = map[int]bool{}, nil prior := kcmpFor kcmpFor = func(r string) kcmpFunc { if r == root { return fixtureKcmp(root) } return prior(r) } t.Cleanup(func() { kcmpFor = prior }) for _, tid := range []string{"20", "7"} { dir := filepath.Join(root, "20", "task", tid, "fd") if err := os.MkdirAll(dir, 0700); err != nil { t.Fatal(err) } for fd, target := range map[int]string{5: db, 6: db + "-wal", 9: "/dev/null"} { if err := os.Symlink(target, filepath.Join(dir, strconv.Itoa(fd))); err != nil { t.Fatal(err) } } } if err := os.WriteFile(filepath.Join(root, "20", "stat"), []byte("20 (carlos edge) S 1 20 20 0 -1 4194560 1 0 0 0 0 0 0 0 20 0 9 0 777 0 0"), 0600); err != nil { t.Fatal(err) } return root, db } func writersInputs(root, db string) WritersInputs { return WritersInputs{Host: "shop.example", FenceClosed: true, SpawnsSettled: true, WorkDrained: true, Cgroup: &evidenceGroup{}, DBPaths: []string{db}, ReplicatorPID: 20, ReplicatorStart: "777", ProcRoot: root, Replicator: []Descriptor{{PID: 20, FD: 5, Target: db}, {PID: 20, FD: 6, Target: db + "-wal"}}} } func TestWritersStoppedAllowsOnlyTheReplicatorsDescriptors(t *testing.T) { root, db := writersProc(t) ev, err := WritersStopped(t.Context(), writersInputs(root, db)) if err != nil || len(ev.Held) != 2 { t.Fatalf("the replicator alone: %+v %v", ev, err) } for _, tc := range []struct { name string change func(*WritersInputs) want error }{ {"admission open", func(in *WritersInputs) { in.FenceClosed = false }, ErrWorkOpen}, {"timer work running", func(in *WritersInputs) { in.WorkDrained = false }, ErrWorkOpen}, {"cgroup populated", func(in *WritersInputs) { in.Cgroup = &evidenceGroup{populated: true} }, ErrPopulated}, {"cgroup members", func(in *WritersInputs) { in.Cgroup = &evidenceGroup{members: []int{8}} }, ErrMembers}, {"one replicator descriptor not identified", func(in *WritersInputs) { in.Replicator = in.Replicator[:1] }, ErrForeignHolder}, {"another process start", func(in *WritersInputs) { in.ReplicatorStart = "778" }, ErrForeignHolder}, {"descriptor of another process", func(in *WritersInputs) { in.Replicator[0].PID = 21 }, ErrInvalidInput}, } { t.Run(tc.name, func(t *testing.T) { in := writersInputs(root, db) tc.change(&in) if _, err := WritersStopped(t.Context(), in); !errors.Is(err, tc.want) { t.Fatalf("got %v, want %v", err, tc.want) } }) } } func TestProcessDescriptorsNamesOneProcessesHolders(t *testing.T) { root, db := writersProc(t) // Another process holds the database too; it is not pid 20's. dir := filepath.Join(root, "30", "task", "30", "fd") if err := os.MkdirAll(dir, 0700); err != nil { t.Fatal(err) } if err := os.Symlink(db, filepath.Join(dir, "4")); err != nil { t.Fatal(err) } own, err := ProcessDescriptors(t.Context(), root, 20, []string{db}) if err != nil || len(own) != 2 || own[0] != (Descriptor{PID: 20, FD: 5, Target: db}) || own[1] != (Descriptor{PID: 20, FD: 6, Target: db + "-wal"}) { t.Fatalf("own: %+v %v", own, err) } if _, err := ProcessDescriptors(t.Context(), filepath.Join(root, "missing"), 20, []string{db}); !errors.Is(err, ErrIncomplete) { t.Fatalf("an unreadable table: %v", err) } // Another process's table being unreadable (root-owned, as the agent // sees it) does not matter: only /proc/ is read. if err := os.Chmod(filepath.Join(root, "30", "task"), 0); err != nil { t.Fatal(err) } t.Cleanup(func() { _ = os.Chmod(filepath.Join(root, "30", "task"), 0o700) }) if again, err := ProcessDescriptors(t.Context(), root, 20, []string{db}); err != nil || len(again) != 2 { t.Fatalf("own descriptors with another table unreadable: %+v %v", again, err) } if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrIncomplete) { t.Fatalf("the complete scan with an unreadable table must fail closed: %v", err) } if err := os.Chmod(filepath.Join(root, "30", "task"), 0o700); err != nil { t.Fatal(err) } // The difference around an open is the replicator's set, and // WritersStopped accepts exactly it. in := writersInputs(root, db) in.Replicator = own if _, err := WritersStopped(t.Context(), in); !errors.Is(err, ErrForeignHolder) { t.Fatalf("pid 30 still holds the database: %v", err) } if err := os.RemoveAll(filepath.Join(root, "30")); err != nil { t.Fatal(err) } if _, err := WritersStopped(t.Context(), in); err != nil { t.Fatal(err) } } func TestWritersStoppedRefusesAnyOtherHolder(t *testing.T) { for _, tc := range []struct{ name, pid, tid string }{ {"another process", "30", "30"}, {"an unshared thread of the agent", "20", "8"}, } { t.Run(tc.name, func(t *testing.T) { root, db := writersProc(t) dir := filepath.Join(root, tc.pid, "task", tc.tid, "fd") if err := os.MkdirAll(dir, 0700); err != nil { t.Fatal(err) } if err := os.Symlink(db+"-shm", filepath.Join(dir, "11")); err != nil { t.Fatal(err) } if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrForeignHolder) { t.Fatalf("got %v", err) } }) } t.Run("a thread's descriptor the leader's table does not show", func(t *testing.T) { root, db := writersProc(t) if err := os.Remove(filepath.Join(root, "20", "task", "20", "fd", "5")); err != nil { t.Fatal(err) } if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrForeignHolder) { t.Fatalf("got %v", err) } }) // Round 6 finding 1: a thread holding the database at the leader's // number and path is the leader's only on the kernel's word. t.Run("an unshared thread at the leader's number and path", func(t *testing.T) { root, db := writersProc(t) fixtureUnshared[7] = true if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrForeignHolder) { t.Fatalf("got %v", err) } }) t.Run("a shared thread whose descriptor is another description", func(t *testing.T) { // The table compared shared, then the descriptor did not: the // thread unshared and reopened between the two comparisons. root, db := writersProc(t) prior := kcmpFor kcmpFor = func(string) kcmpFunc { return func(pid, tid, typ int, a, b uintptr) (int, error) { if typ == kcmpFile { return 2, nil } return 0, nil } } defer func() { kcmpFor = prior }() if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrForeignHolder) { t.Fatalf("got %v", err) } }) for _, kerr := range []error{unix.ENOSYS, unix.EPERM} { t.Run("kcmp unavailable: "+kerr.Error(), func(t *testing.T) { root, db := writersProc(t) fixtureKcmpErr = kerr if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrIncomplete) { t.Fatalf("got %v", err) } if _, err := ProcessDescriptors(t.Context(), root, 20, []string{db}); !errors.Is(err, ErrIncomplete) { t.Fatalf("own descriptors: %v", err) } }) } t.Run("an unreadable descriptor directory", func(t *testing.T) { root, db := writersProc(t) if _, err := WritersStopped(t.Context(), writersInputs(filepath.Join(root, "missing"), db)); !errors.Is(err, ErrIncomplete) { t.Fatalf("got %v", err) } }) } // threadWith runs act on a new OS thread that has unshared its descriptor // table (CLONE_FILES) when unshare is set, and keeps the thread alive // until the returned release is called. The thread is never unlocked, so // it exits with its goroutine and the runtime never reuses its table. func threadWith(t *testing.T, unshare bool, act func() error) (tid int, release func()) { t.Helper() type started struct { tid int err error } ready, done := make(chan started), make(chan struct{}) var run func() run = func() { runtime.LockOSThread() if unix.Gettid() == os.Getpid() { // The leader's own thread: unsharing it would move the // leader's table, not a thread's. Hold it and start again on // another thread. go run() <-done runtime.UnlockOSThread() return } if unshare { if err := unix.Unshare(unix.CLONE_FILES); err != nil { ready <- started{err: err} return } } if err := act(); err != nil { ready <- started{err: err} return } ready <- started{tid: unix.Gettid()} <-done } go run() s := <-ready if s.err != nil { t.Fatal(s.err) } return s.tid, func() { close(done) } } // The real kernel, on this test process: a thread // sharing the table adds nothing; a thread that unshared it holds the // database at the leader's number and path, either by the copied // description or by reopening it independently there, and is reported // with FD -1, never as the leader's. func TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity(t *testing.T) { // The kernel facts are this process's own threads, so the test runs in // a fresh copy of the test binary: another test in the package // (main's TestFDHoldersUnsharedThread) may have unshared the leader // thread's table for good, and then no thread shares the leader's. if os.Getenv("STOPEVIDENCE_KCMP_CHILD") != "1" { cmd := exec.Command(os.Args[0], "-test.run=^TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity$", "-test.count=1", "-test.v") cmd.Env = append(os.Environ(), "STOPEVIDENCE_KCMP_CHILD=1") out, err := cmd.CombinedOutput() if err != nil || !strings.Contains(string(out), "--- PASS: TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity") { t.Fatalf("the fresh process: %v\n%s", err, out) } return } db := filepath.Join(t.TempDir(), "data.db") f, err := os.OpenFile(db, os.O_RDWR|os.O_CREATE, 0o600) if err != nil { t.Fatal(err) } defer f.Close() n, pid := int(f.Fd()), os.Getpid() own := Descriptor{PID: pid, FD: n, Target: db} scan := func() []Descriptor { t.Helper() // scanOne through the one-process scan: the complete scan root // makes (scanDescriptors) reads each process the same way. got, err := scanProcess(t.Context(), "/proc", pid, []string{db}) if err != nil { t.Fatal(err) } return got } tid, release := threadWith(t, false, func() error { return nil }) if tid == pid { t.Fatal("the helper thread is the leader") } if got := scan(); !slices.Equal(got, []Descriptor{own}) { t.Fatalf("a shared thread: %+v", got) } release() for _, tc := range []struct { name string act func() error }{ {"the copied description", func() error { return nil }}, {"reopened independently at the same number", func() error { fd, err := unix.Open(db, unix.O_RDWR|unix.O_CLOEXEC, 0) if err != nil { return err } defer unix.Close(fd) return unix.Dup3(fd, n, unix.O_CLOEXEC) }}, } { t.Run(tc.name, func(t *testing.T) { _, release := threadWith(t, true, tc.act) defer release() got := scan() if !slices.Contains(got, own) || !slices.Contains(got, Descriptor{PID: pid, FD: -1, Target: db}) { t.Fatalf("an unshared thread passed as the leader's: %+v", got) } }) } } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/stopevidence/ -run 'Writers|ProcessDescriptors' -count=1` Expected: FAIL to compile: `undefined: WritersInputs`. - [ ] **Step 3: The observation** `internal/stopevidence/writers.go`: ```go package stopevidence import ( "context" "errors" "path/filepath" "sort" "time" "github.com/carlosframework/platform/internal/controlname" ) var ( ErrWorkOpen = errors.New("stopevidence: admission, spawns or work not drained") ErrForeignHolder = errors.New("stopevidence: a descriptor other than the replicator's holds the database") ) // Descriptor is one open file descriptor of one process, as /proc shows it. type Descriptor struct { PID int FD int Target string } // WritersInputs is claim §6.1's writers-stopped observation (L10, region // HA spec 1 §5.8 release step 1). The caller's drain acknowledgement // supplies the three drained facts; this package checks the kernel. type WritersInputs struct { Host string // FenceClosed, SpawnsSettled and WorkDrained are the release // controller's durable admission fence, settled spawn reservations, and // drained application and timer work. FenceClosed, SpawnsSettled, WorkDrained bool Cgroup interface { Populated() (bool, error) Members() ([]int, error) } DBPaths []string // Replicator names the only descriptors that may hold the database: // the agent's own replicator for this host, by PID, its process start // time (field 22 of /proc//stat), and each descriptor. ReplicatorPID int ReplicatorStart string Replicator []Descriptor ProcRoot string } // WritersEvidence records what WritersStopped observed. type WritersEvidence struct { Host string CheckedAt time.Time CheckedPaths []string Held []Descriptor // the replicator descriptors still open ProcRoot string } // WritersStopped passes only when admission is fenced, spawns settled, // work drained, the host's cgroup empty with no members, and every holder // of the database, WAL or SHM files is one of the replicator's identified // descriptors in the replicator's own process, started at // ReplicatorStart. An incomplete scan, an unreadable start time, or any // other holder fails. It grants no handover: full stop evidence (Collect) // still follows, after the replicator closes. func WritersStopped(ctx context.Context, in WritersInputs) (WritersEvidence, error) { if ctx == nil || !controlname.ValidHostShape(in.Host) || len(in.DBPaths) == 0 || in.ReplicatorPID <= 0 || in.ReplicatorStart == "" { return WritersEvidence{}, ErrInvalidInput } if !in.FenceClosed || !in.SpawnsSettled || !in.WorkDrained { return WritersEvidence{}, ErrWorkOpen } if in.Cgroup == nil { return WritersEvidence{}, ErrNoCgroupEvidence } paths, err := dbPaths(in.DBPaths) if err != nil { return WritersEvidence{}, err } if populated, err := in.Cgroup.Populated(); err != nil { return WritersEvidence{}, err } else if populated { return WritersEvidence{}, ErrPopulated } if members, err := in.Cgroup.Members(); err != nil { return WritersEvidence{}, err } else if len(members) != 0 { return WritersEvidence{}, ErrMembers } root := in.ProcRoot if root == "" { root = "/proc" } held, err := scanDescriptors(ctx, root, paths) if err != nil { return WritersEvidence{}, err } allowed := map[Descriptor]bool{} for _, d := range in.Replicator { if d.PID != in.ReplicatorPID { return WritersEvidence{}, ErrInvalidInput } allowed[Descriptor{PID: d.PID, FD: d.FD, Target: filepath.Clean(d.Target)}] = true } for _, d := range held { if !allowed[d] { return WritersEvidence{}, ErrForeignHolder } } if len(held) > 0 { start, err := processStart(root, in.ReplicatorPID) if err != nil { return WritersEvidence{}, errors.Join(ErrIncomplete, err) } if start != in.ReplicatorStart { return WritersEvidence{}, ErrForeignHolder } } if err := ctx.Err(); err != nil { return WritersEvidence{}, err } return WritersEvidence{Host: in.Host, CheckedAt: time.Now().UTC(), CheckedPaths: paths, Held: held, ProcRoot: root}, nil } // ProcessDescriptors lists the descriptors of one process (its leader's // table) that hold one of dbs' database, WAL or SHM files, as /proc shows // them. It reads only /proc/, never another process's table, so the // agent (uid carlos, no CAP_SYS_PTRACE) can take it for its own pid // immediately before and after it opens a host's replicator: the // difference is that replicator's descriptor set, which // WritersInputs.Replicator names at release (L10: "descriptors positively // identified as belonging to this host's admitted replicator"). An // unreadable table is ErrIncomplete, never an empty set. The complete // scan of every process (WritersStopped) stays root's. func ProcessDescriptors(ctx context.Context, procRoot string, pid int, dbs []string) ([]Descriptor, error) { if ctx == nil || pid <= 0 { return nil, ErrInvalidInput } paths, err := dbPaths(dbs) if err != nil { return nil, err } if procRoot == "" { procRoot = "/proc" } held, err := scanProcess(ctx, procRoot, pid, paths) if err != nil { return nil, err } var own []Descriptor for _, d := range held { if d.FD >= 0 { own = append(own, d) } } return own, nil } func dbPaths(dbs []string) ([]string, error) { seen := map[string]bool{} for _, db := range dbs { if !filepath.IsAbs(db) { return nil, ErrInvalidInput } db = filepath.Clean(db) seen[db], seen[db+"-wal"], seen[db+"-shm"] = true, true, true } paths := make([]string, 0, len(seen)) for path := range seen { paths = append(paths, path) } sort.Strings(paths) return paths, nil } ``` `internal/stopevidence/writers_linux.go`: ```go //go:build linux package stopevidence import ( "context" "errors" "fmt" "os" "path/filepath" "sort" "strconv" "strings" "golang.org/x/sys/unix" ) // kcmp(2)'s comparison types (linux/kcmp.h). golang.org/x/sys has the // syscall number (unix.SYS_KCMP) on every linux architecture but no // wrapper or constants. const ( kcmpFile = 0 // the same open file description: idx1, idx2 are fds kcmpFiles = 2 // the same file descriptor table ) // kcmpFunc is kcmp(2): 0 when the two tasks' resources are the same, a // positive ordering (1, 2) or 3 (not comparable) when they are not. type kcmpFunc func(pid1, pid2, typ int, idx1, idx2 uintptr) (int, error) func sysKcmp(pid1, pid2, typ int, idx1, idx2 uintptr) (int, error) { r, _, errno := unix.Syscall6(unix.SYS_KCMP, uintptr(pid1), uintptr(pid2), uintptr(typ), idx1, idx2, 0) if errno != 0 { return 0, errno } return int(r), nil } // kcmpFor is the kernel's identity check for the tasks under root. // kcmp(2) takes pids in the caller's own pid namespace, so it is used // only for the caller's /proc; any other root (a test fixture) has none, // which fails closed as ErrIncomplete. Tests replace it. kcmp needs // ptrace read access to both tasks: always granted within the caller's // own process (ProcessDescriptors), and to root's complete scan through // CAP_SYS_PTRACE; an EPERM (a Yama scope that forbids it), or ENOSYS (a // kernel built without CONFIG_KCMP), is an incomplete observation. var kcmpFor = func(root string) kcmpFunc { if filepath.Clean(root) == "/proc" { return sysKcmp } return func(int, int, int, uintptr, uintptr) (int, error) { return 0, unix.ENOSYS } } // scanDescriptors lists every descriptor under root that resolves to one // of paths, deduplicated by process and descriptor number. A descriptor // seen only in a task that does not share the leader's table (an unshared // thread) is reported against the leader's PID with FD -1, so no // allow-list can exempt it. func scanDescriptors(ctx context.Context, root string, paths []string) ([]Descriptor, error) { ops := realFDOps() if !filepath.IsAbs(root) { return nil, ErrInvalidInput } entries, err := ops.readDir(root) if err != nil { return nil, fmt.Errorf("%w: %w", ErrIncomplete, err) } var out []Descriptor for _, entry := range entries { if err := ctx.Err(); err != nil { return nil, err } pid, err := strconv.Atoi(entry.Name()) if err != nil || pid <= 0 { continue } held, err := scanOne(ctx, root, pid, paths, ops, kcmpFor(root)) if err != nil { return nil, err } out = append(out, held...) } sort.Slice(out, func(i, j int) bool { if out[i].PID != out[j].PID { return out[i].PID < out[j].PID } return out[i].FD < out[j].FD }) return out, nil } // scanProcess is scanDescriptors for one process: it reads only // /proc/, so it needs no privilege over any other process. func scanProcess(ctx context.Context, root string, pid int, paths []string) ([]Descriptor, error) { if !filepath.IsAbs(root) { return nil, ErrInvalidInput } ops := realFDOps() if _, err := ops.stat(filepath.Join(root, strconv.Itoa(pid))); err != nil { // A process asking about itself cannot have vanished: an absent // or unreadable /proc/ is an incomplete observation. return nil, fmt.Errorf("%w: pid %d: %w", ErrIncomplete, pid, err) } out, err := scanOne(ctx, root, pid, paths, ops, kcmpFor(root)) if err != nil { return nil, err } sort.Slice(out, func(i, j int) bool { return out[i].FD < out[j].FD }) return out, nil } // scanOne lists pid's descriptors on paths, deduplicated by descriptor // number. A thread's descriptor counts as the leader's only on positive // kernel evidence (round 6 finding 1): the leader shows the same number // for the same path, kcmp(2) KCMP_FILES says the thread shares the // leader's descriptor table, and KCMP_FILE says that descriptor is the // leader's open file description. A number and a path alone prove // nothing: a thread that unshared its table (CLONE_FILES) can reopen the // database at the same number. Any other thread's descriptor is reported // with FD -1, so no allow-list can exempt it. A comparison that cannot be // made (ENOSYS, EPERM, anything but a vanished task) is ErrIncomplete. A // process or thread that vanished mid-scan is skipped; an unreadable // table is ErrIncomplete. func scanOne(ctx context.Context, root string, pid int, paths []string, ops fdOps, cmp kcmpFunc) ([]Descriptor, error) { wanted := make(map[string]bool, len(paths)) for _, p := range paths { wanted[filepath.Clean(p)] = true } pidDir := filepath.Join(root, strconv.Itoa(pid)) tasks, err := readProcDir(filepath.Join(pidDir, "task"), pidDir, ops) if err != nil { return nil, fmt.Errorf("%w: pid %d: %w", ErrIncomplete, pid, err) } // The leader's table first: a thread's descriptor is the leader's // only when the leader shows the same number for the same file. leaderName := strconv.Itoa(pid) sort.SliceStable(tasks, func(i, j int) bool { return tasks[i].Name() == leaderName && tasks[j].Name() != leaderName }) leader := map[int]string{} seen := map[Descriptor]bool{} var out []Descriptor for _, task := range tasks { if err := ctx.Err(); err != nil { return nil, err } tid, err := strconv.Atoi(task.Name()) if err != nil || tid <= 0 { continue } taskDir := filepath.Join(pidDir, "task", task.Name()) fdDir := filepath.Join(taskDir, "fd") fds, err := readProcDir(fdDir, taskDir, ops) if err != nil { return nil, fmt.Errorf("%w: pid %d tid %d: %w", ErrIncomplete, pid, tid, err) } shared := -1 // this thread's KCMP_FILES verdict, asked once, only if needed descriptors: for _, fd := range fds { n, err := strconv.Atoi(fd.Name()) if err != nil { continue } target, err := ops.readlink(filepath.Join(fdDir, fd.Name())) if procGone(err) { continue } if err != nil { return nil, fmt.Errorf("%w: pid %d tid %d fd %d: %w", ErrIncomplete, pid, tid, n, err) } clean := filepath.Clean(strings.TrimSuffix(target, " (deleted)")) if !filepath.IsAbs(target) || !wanted[clean] { continue } d := Descriptor{PID: pid, FD: n, Target: clean} if tid == pid { leader[n] = clean } else { ok, gone, err := leadersDescriptor(cmp, pid, tid, n, leader[n] == clean, &shared) if err != nil { return nil, fmt.Errorf("%w: pid %d tid %d fd %d: kcmp: %w", ErrIncomplete, pid, tid, n, err) } if gone { break descriptors // the thread exited mid-scan } if !ok { d.FD = -1 // not provably the leader's descriptor } } if !seen[d] { seen[d] = true out = append(out, d) } } } return out, nil } // leadersDescriptor is the kernel's evidence that thread tid's descriptor // n is the leader's own: the same table (KCMP_FILES, asked once per thread // and kept in *shared) and the same open file description (KCMP_FILE). // sameName is the leader showing n on the same path. gone reports a task // that exited (ESRCH); EBADF (n closed in one of them since its listing) // is "not the leader's"; any other error is returned. func leadersDescriptor(cmp kcmpFunc, pid, tid, n int, sameName bool, shared *int) (ok, gone bool, err error) { if !sameName { return false, false, nil } if *shared < 0 { r, err := cmp(pid, tid, kcmpFiles, 0, 0) if errors.Is(err, unix.ESRCH) { return false, true, nil } if err != nil { return false, false, err } *shared = r } if *shared != 0 { return false, false, nil } r, err := cmp(pid, tid, kcmpFile, uintptr(n), uintptr(n)) switch { case errors.Is(err, unix.ESRCH): return false, true, nil case errors.Is(err, unix.EBADF): return false, false, nil case err != nil: return false, false, err } return r == 0, false, nil } // processStart is field 22 of /proc//stat: the process start time in // clock ticks since boot, which with the boot ID names one process. func processStart(root string, pid int) (string, error) { raw, err := os.ReadFile(filepath.Join(root, strconv.Itoa(pid), "stat")) if err != nil { return "", err } s := string(raw) end := strings.LastIndexByte(s, ')') if end < 0 { return "", ErrIncomplete } fields := strings.Fields(s[end+1:]) if len(fields) < 20 { // fields 3.. start after the comm return "", ErrIncomplete } return fields[19], nil } ``` `internal/stopevidence/writers_other.go`: ```go //go:build !linux package stopevidence import "context" func scanDescriptors(context.Context, string, []string) ([]Descriptor, error) { return nil, ErrUnsupported } func scanProcess(context.Context, string, int, []string) ([]Descriptor, error) { return nil, ErrUnsupported } func processStart(string, int) (string, error) { return "", ErrUnsupported } ``` - [ ] **Step 4: Run the tests and the cross-builds** Run: `nice -n 19 go test ./internal/stopevidence/ -count=1 && GOOS=darwin go vet ./internal/stopevidence/ && GOOS=windows go vet ./internal/stopevidence/` Expected: PASS; both vets clean. - [ ] **Step 5: Commit** ```bash git add internal/stopevidence/writers.go internal/stopevidence/writers_linux.go internal/stopevidence/writers_other.go internal/stopevidence/writers_linux_test.go git commit -m "stopevidence: writers-stopped allows only the replicator's identified descriptors, so a release can flush before full stop Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 5: The regional worker (`internal/hostrelocate`) Spec §5.8 "Who acts". `Worker.Start` checks that this box is an active, unfenced worker of the region; that the claim serves on the source and the host is an instance of a relocatable shape (`Relocatable`); that the target is in the complete eligible set (`Eligible`, §2); the source's fence (a failover needs one, a drain refuses one); that a drain's source has a writer session to release (ruling 11); `Directory.CheckBoxStart` for the starter, the target and a live source (active, unfenced, no executable update reserved: H14); the box-started census (L5, L20); and the app admission registration; then `StartRelocation`. `Worker.Step` makes at most one worker CAS from the claim alone: a fenced source converts the operation (`FenceSource`), a fenced target is replaced (`ReplaceTarget`, or `MarkStuck` once two are used), then the phase's own step: the barrier, the pin after a release or after the failover window (selected and journaled under the replica lease on `holder.location`, finding 2), naming and installing the target, recording `installed` once `Prepared` verifies P, serving after a passing probe, `probe-failed` past the probe deadline, `stuck` past the release deadline (ruling 14) or on a live target's third replacement. `StorePinner` selects the database plan with `genselect.Select` and the files pin (the newest manifest in the admitted chain, independently of the database baseline, B2-4, with the chain as its origins), checks every bound before it writes, and writes the pin object create-only at the key of the claim generation it selected against (ruling 16); a retry at the same generation reuses its own object; an empty pin needs its evidence (ruling 15). `ReadPin` is every reader's digest-checked read; the pinner retains the pinned manifest's exact bytes in the control bucket (ruling 21) and `CopyPinnedManifest` writes P's files manifest from that copy and nothing else. A target replacement is looked for only while replacements remain; exhaustion is stuck at once. **Files:** - Create: `internal/hostrelocate/worker.go`, `internal/hostrelocate/pin.go`, `internal/hostfiles/manifestcheck.go` (`ValidateManifest`, rev 10) - Test: `internal/hostrelocate/worker_test.go`, `internal/hostrelocate/pin_test.go` **Interfaces:** - Consumes: Task 1; `hostactors.Directory` (`Read`, `TerminationFence`, `CheckBoxStart`), `hostactors.ValidateTerminationFence`, `hostfence.FenceKey`, `hostfence.ClaimFenceAuthority` (tests), `genselect.Request`, `genselect.Plan`, `genselect.ErrEmpty`, `hostfiles.ManifestKey`, `hostfiles.Manifest`, `hostfiles.CopyManifest`, `release.CreateExact`. - Produces: `hostrelocate.Worker{Claims, Directory, Fences, Releases, Admission, Census, Choose, Eligible, Relocatable, Leases, Pins, Prepared, Discharge, Notify, Self, Region, Controller, NewID, Now}` with `Start(ctx, host, Request) (hostclaim.Observed, error)` and `Step(ctx, host) (hostclaim.Observed, error)`; `Request{Reason, Source, Target}`; interfaces `Admission`, `Pinner`, `Prepared`, `Lease`; `Owner`; `ProbeDeadline`, `ReleaseDeadline`; errors `ErrUnsupported`, `ErrNotWorker`, `ErrNotYet`, `ErrRefused`, `ErrNoController`; `StorePinner{Claims, Data, SelectDB, Bucket, Region}`, `PinObject{V, Host, Op, ClaimGen, DB, Empty, Files, PinnedAt}`, `FilesPin{Prefix, Seq, SHA256, Origins, Retained}`, `PinKey(host, op, gen)`, `RetainedManifestKey(host, op, sha)`, `ReadPin(ctx, claims, RecoveryPin) (PinObject, error)`, `CopyPinnedManifest(ctx, claims, data, FilesPin, toPrefix) (hostfiles.Manifest, error)`; `hostfiles.ValidateManifest(m, origins) error`. - [ ] **Step 1: Write the failing tests** `internal/hostrelocate/worker_test.go`: ```go package hostrelocate import ( "context" "crypto/ed25519" "crypto/rand" "encoding/json" "errors" "fmt" "slices" "strings" "testing" "time" "github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostfence" "github.com/carlosframework/platform/internal/release" "github.com/carlosframework/platform/internal/termfence" ) const host = "shop.bab.oncarlos.com" var boxes = map[string]string{"wrk": "a", "src": "b", "dst": "c", "spa": "d", "sp2": "e", "sp3": "f"} func enrollment(box string) string { return strings.Repeat(boxes[box], 32) } type fixture struct { pinErr error t *testing.T s release.Store w Worker now time.Time notified int sign release.Signer roster hostactors.Roster // leases records the lease provider's calls in order with the pinner's. leases []string loseLease bool ineligible map[string]bool } // fakeLease is one held replica lease; lost is closed when the test takes // it away before the pin is journaled. type fakeLease struct { f *fixture lost chan struct{} } func (l *fakeLease) Lost() <-chan struct{} { return l.lost } func (l *fakeLease) Release(context.Context) error { l.f.leases = append(l.f.leases, "release") return nil } type okReleases struct{} func (okReleases) VerifyRelease(context.Context, hostclaim.Claim, hostclaim.Release) error { return nil } type admission struct{ registered []string } func (a *admission) Register(_ context.Context, _ hostclaim.Observed, o Owner, id string) (string, error) { if o.Box != "wrk" || o.Enrollment != enrollment("wrk") { return "", errors.New("wrong owner") } a.registered = append(a.registered, id) return "adm-" + id, nil } func (a *admission) Recheck(context.Context, string) error { return nil } // built reports P built once the target has said so (the participant's // half), as moves' install routine would leave it. type built struct{} func (built) Verify(_ context.Context, o hostclaim.Observed) (hostclaim.PreparedAuthority, bool, error) { c := o.Claim if c.Op.Journal["test.built"] != c.Op.Journal["prepared"] { return hostclaim.PreparedAuthority{}, false, nil } var pin hostclaim.RecoveryPin _ = json.Unmarshal([]byte(c.Op.Journal["pin"]), &pin) return hostclaim.PreparedAuthority{Generation: c.Op.Journal["prepared"], InstallManifest: "m", SHA256: strings.Repeat("c", 64), SnapshotTXID: pin.MaxTXID, SourceFilesSeq: pin.FilesSeq, FilesManifestSHA256: strings.Repeat("f", 64)}, true, nil } type pins struct{ f *fixture } func (p pins) Pin(_ context.Context, o hostclaim.Observed, now time.Time) (hostclaim.RecoveryPin, error) { p.f.leases = append(p.f.leases, "pin") if p.f.pinErr != nil { return hostclaim.RecoveryPin{}, p.f.pinErr } var r hostclaim.Release _ = json.Unmarshal([]byte(o.Claim.Op.Journal["release"]), &r) return hostclaim.RecoveryPin{Key: PinKey(o.Claim.Host, o.Claim.Op.ID, o.Claim.Gen), SHA256: strings.Repeat("a", 64), Generation: o.Claim.Holder.Generation, MaxTXID: max(r.FinalTXID, 40), FilesManifest: "m", FilesSeq: max(r.ManifestSeq, 7), PinnedAt: now}, nil } func newFixture(t *testing.T) *fixture { t.Helper() ctx := context.Background() f := &fixture{t: t, s: release.NewDirStore(t.TempDir()), now: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)} pub, key, err := ed25519.GenerateKey(rand.Reader) if err != nil { t.Fatal(err) } f.sign = release.RootSigner(key) f.roster = hostactors.Roster{V: 1, DeploymentBucket: "bucket", Revision: 1, Regions: map[string]hostactors.Region{"ie": {Account: "123", AWSRegion: "eu-west-1"}}, Boxes: map[string]hostactors.Box{}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{}} for ref := range boxes { f.roster.Boxes[ref] = hostactors.Box{Enrollment: enrollment(ref), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default", Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"} } f.writeRoster() d := hostactors.Directory{Store: f.s, PublicKey: pub, Bucket: "bucket"} clock := func() time.Time { return f.now } n := 0 f.w = Worker{Claims: f.s, Directory: d, Fences: hostfence.ClaimFenceAuthority{Directory: d, Now: clock}, Releases: okReleases{}, Admission: &admission{}, Census: func(context.Context, string, []string) error { return nil }, Choose: func(_ context.Context, _ hostclaim.Observed, exclude []string) (hostclaim.Target, error) { for _, b := range []string{"dst", "spa", "sp2", "sp3"} { if !slices.Contains(exclude, b) { return hostclaim.Target{Box: b, Enrollment: enrollment(b)}, nil } } return hostclaim.Target{}, errors.New("no eligible box") }, Eligible: func(_ context.Context, _ hostclaim.Observed, box string) error { if f.ineligible[box] { return errors.New("not eligible: " + box) } return nil }, Relocatable: func(context.Context, hostclaim.Observed) error { return nil }, Leases: func(context.Context, string) (Lease, error) { f.leases = append(f.leases, "acquire") l := &fakeLease{f: f, lost: make(chan struct{})} if f.loseLease { close(l.lost) } return l, nil }, Pins: pins{f}, Prepared: built{}, Discharge: func(context.Context, hostactors.Box, termfence.Record) *hostclaim.LocalDischarge { return nil }, Notify: func(context.Context) { f.notified++ }, Self: termfence.WrittenBy{Box: "wrk", Enrollment: enrollment("wrk")}, Region: "ie", Controller: "ctl", NewID: func() string { n++; return "id" + string(rune('0'+n)) }, Now: clock} f.ineligible = map[string]bool{} // A host serving on src, whose source admitted one cold session. created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + host + ".json", Account: "bab", App: "jam", Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created} body := []byte(`{"host":"` + host + `","account":"bab","app":"jam","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) o, err := hostclaim.Read(ctx, f.s, host) if err != nil { t.Fatal(err) } o = f.step(o, hostclaim.BeginCreateOn(nh, body, "c1", "console", created, hostclaim.ExecutorSelection{Box: "src", Enrollment: enrollment("src")})) o = f.step(o, hostclaim.CompleteCreate("c1", "src")) f.step(o, hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) return f } func (f *fixture) writeRoster() { f.t.Helper() raw, err := release.SignRootEnvelope(f.sign, f.roster) if err != nil { f.t.Fatal(err) } _, etag, err := f.s.Get(context.Background(), hostactors.Key) if err != nil { etag = "" } if _, err := f.s.Put(context.Background(), hostactors.Key, raw, etag); err != nil { f.t.Fatal(err) } } func (f *fixture) step(o hostclaim.Observed, tr func(*hostclaim.Claim) error) hostclaim.Observed { f.t.Helper() next, err := hostclaim.Update(context.Background(), f.s, o, tr) if err != nil { f.t.Fatal(err) } return next } func (f *fixture) claim() hostclaim.Observed { f.t.Helper() o, err := hostclaim.Read(context.Background(), f.s, host) if err != nil { f.t.Fatal(err) } return o } // participant applies a box-side transition (Plan 6b's half) to the claim. func (f *fixture) participant(tr func(*hostclaim.Claim) error) { f.t.Helper() f.step(f.claim(), tr) } func (f *fixture) ack(box string) { f.participant(hostclaim.AcknowledgeExecution(box, hostclaim.ExecutionAcknowledgement{Op: f.claim().Claim.Op.ID, Enrollment: enrollment(box), BootID: "boot", ControllerID: "ctl", LeaseETag: "lease", Expires: f.now.Add(time.Hour)})) } // terminate writes box's termination fence as hostfence.TerminationWorker // would, observed a minute ago by wrk. func (f *fixture) terminate(box string) { f.t.Helper() b := f.roster.Boxes[box] rec := termfence.Record{V: 1, Box: box, Enrollment: b.Enrollment, Cloud: b.Cloud, State: "terminated", ObservedAt: f.now.Add(-time.Minute).Format(time.RFC3339Nano), Via: termfence.ViaDescribe, RequestID: "req", WrittenBy: termfence.WrittenBy{Box: "wrk", Enrollment: enrollment("wrk")}} raw, _ := json.Marshal(rec) if _, err := f.s.Put(context.Background(), hostfence.FenceKey(b.Enrollment), raw, ""); err != nil { f.t.Fatal(err) } } func (f *fixture) worker(want error) hostclaim.Observed { f.t.Helper() o, err := f.w.Step(context.Background(), host) if !errors.Is(err, want) { f.t.Fatalf("step: %v, want %v", err, want) } return o } func (f *fixture) installAndProbe(target string) { f.t.Helper() c := f.claim().Claim id, p := c.Op.ID, c.Op.Journal["prepared"] f.worker(ErrNotYet) // P is not built yet f.participant(func(c *hostclaim.Claim) error { c.Op.Journal["test.built"] = p; return nil }) if o := f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseInstalled { f.t.Fatalf("the worker records installed once P verifies: %s", o.Claim.Op.Phase) } stamp := f.claim().Claim.Op.Journal["install_stamp"] var n uint64 _ = json.Unmarshal([]byte(stamp), &n) f.participant(hostclaim.RecordLocalInstall(id, hostclaim.LocalInstall{Box: target, Enrollment: enrollment(target), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("d", 64)})) f.participant(hostclaim.ToProbing(id, f.now)) o := f.claim() f.participant(hostclaim.AdmitWriter(hostclaim.Admission{ObservedOpID: id, ObservedPhase: hostclaim.PhaseProbing, ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: target, Mode: "cold", NewGeneration: p, KeepStamp: true, Region: "ie", PredecessorRegion: "ie"})) } func TestWorkerDrivesADrain(t *testing.T) { f := newFixture(t) o, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}) if err != nil { t.Fatal(err) } if o.Claim.Op.Intent["admission_id"] == "" || o.Claim.Op.Intent["directory_revision"] != "1" { t.Fatalf("intent: %+v", o.Claim.Op.Intent) } f.worker(ErrNotYet) // no acknowledgements yet f.ack("src") f.ack("dst") if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseReleasing { t.Fatalf("phase %s", o.Claim.Op.Phase) } f.worker(ErrNotYet) // the source has not released h := f.claim().Claim.Holder f.participant(hostclaim.JournalRelease(o.Claim.Op.ID, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhasePinning || o.Claim.Op.Exec != "none" { t.Fatalf("pinning: %+v", o.Claim.Op) } if o = f.worker(nil); o.Claim.Op.Journal["pin"] == "" { t.Fatal("no pin journaled") } if want := []string{"acquire", "pin", "release"}; !slices.Equal(f.leases, want) { t.Fatalf("the pin was not selected and journaled under the location lease: %v", f.leases) } before := f.notified if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseInstalling || o.Claim.Op.Exec != "dst" || f.notified != before+1 { t.Fatalf("installing: %+v notified %d", o.Claim.Op, f.notified-before) } f.installAndProbe("dst") f.worker(ErrNotYet) // probing, no probe result, within the deadline o = f.claim() f.participant(hostclaim.RecordProbe(o.Claim.Op.ID, hostclaim.Probe{Box: "dst", Session: o.Claim.Holder.Session})) if o = f.worker(nil); o.Claim.Op.Kind != hostclaim.KindServing || o.Claim.Op.Exec != "dst" { t.Fatalf("serving: %+v", o.Claim.Op) } } func TestWorkerFailoverWaitsForThePinWindow(t *testing.T) { f := newFixture(t) ctx := context.Background() if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); !errors.Is(err, ErrRefused) { t.Fatalf("failover with no fence: %v", err) } f.terminate("src") if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); !errors.Is(err, ErrRefused) { t.Fatalf("drain of a fenced source: %v", err) } o, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}) if err != nil { t.Fatal(err) } if o.Claim.Op.Exec != "none" || !slices.Contains(o.Claim.Lost, "src") { t.Fatalf("failover start: %+v", o.Claim.Op) } f.ack("dst") if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhasePinning { t.Fatalf("phase %s", o.Claim.Op.Phase) } // The fence was observed a minute ago, so the 30-second window is open. if o = f.worker(nil); o.Claim.Op.Journal["pin"] == "" { t.Fatal("not pinned") } } func TestWorkerPinsNoEarlierThanThirtySecondsAfterTheFence(t *testing.T) { f := newFixture(t) ctx := context.Background() f.terminate("src") f.now = f.now.Add(-time.Minute + 10*time.Second) // ten seconds after the observation if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } f.ack("dst") f.worker(nil) // pinning f.worker(ErrNotYet) f.now = f.now.Add(20 * time.Second) if o := f.worker(nil); o.Claim.Op.Journal["pin"] == "" { t.Fatal("not pinned at observation plus thirty seconds") } } func TestWorkerConvertsADrainWhoseSourceIsTerminated(t *testing.T) { f := newFixture(t) if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } f.ack("dst") f.terminate("src") o := f.worker(nil) if o.Claim.Op.Phase != hostclaim.PhaseStarting || o.Claim.Op.Exec != "none" || slices.Contains(o.Claim.Op.Participants, "src") || !slices.Contains(o.Claim.Lost, "src") { t.Fatalf("conversion: %+v lost %v", o.Claim.Op, o.Claim.Lost) } if o = f.worker(nil); o.Claim.Op.Phase != hostclaim.PhasePinning { t.Fatalf("the barrier now needs only the target: %s", o.Claim.Op.Phase) } } func TestWorkerReplacesATerminatedTargetTwiceThenStops(t *testing.T) { f := newFixture(t) if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } for _, tc := range []struct{ dead, next string }{{"dst", "spa"}, {"spa", "sp2"}} { f.terminate(tc.dead) o := f.worker(nil) if o.Claim.Op.Intent["target_box"] != tc.next || !slices.Contains(o.Claim.Lost, tc.dead) { t.Fatalf("replacement of %s: %+v", tc.dead, o.Claim.Op.Intent) } } f.terminate("sp2") o := f.worker(nil) if o.Claim.Op.Journal["stuck"] == "" { t.Fatalf("a third replacement must stop the operation: %+v", o.Claim.Op.Journal) } before := o.Claim.Gen if o = f.worker(nil); o.Claim.Gen != before { t.Fatal("a worker moved a stuck operation") } } func TestWorkerFailsAProbePastItsDeadline(t *testing.T) { f := newFixture(t) if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } f.ack("src") f.ack("dst") f.worker(nil) h := f.claim().Claim.Holder f.participant(hostclaim.JournalRelease(f.claim().Claim.Op.ID, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) f.worker(nil) f.worker(nil) f.worker(nil) f.installAndProbe("dst") f.now = f.now.Add(ProbeDeadline + time.Second) if o := f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseProbeFailed { t.Fatalf("phase %s", o.Claim.Op.Phase) } f.worker(ErrNotYet) // the failed target has not released } func TestWorkerRefusesWhenItIsNotAnActiveWorker(t *testing.T) { f := newFixture(t) b := f.roster.Boxes["wrk"] b.Region = "uk" f.roster.Boxes["wrk"] = b f.roster.Revision++ f.writeRoster() before := f.claim().ETag if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); !errors.Is(err, ErrNotWorker) { t.Fatalf("start: %v", err) } if f.claim().ETag != before { t.Fatal("a refused start wrote the claim") } } func TestWorkerStartRefusesAnUpdateReservedForAParticipant(t *testing.T) { f := newFixture(t) f.roster.Activations["upd-1"] = hostactors.Reservation{ID: "upd-1", FloorSHA256: strings.Repeat("f", 64), Actors: map[string]string{"box:dst": enrollment("dst")}, Releases: map[string]hostactors.Report{"box:dst": {Enrollment: enrollment("dst")}}} f.roster.Revision++ f.writeRoster() before := f.claim().ETag if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err == nil { t.Fatal("started while an executable update is reserved for the target (H14)") } if f.claim().ETag != before { t.Fatal("a refused start wrote the claim") } } func TestWorkerTreatsAnUnreadableFenceAsAnErrorNeverAsNoFence(t *testing.T) { f := newFixture(t) if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } f.ack("src") f.ack("dst") if _, err := f.s.Put(context.Background(), hostfence.FenceKey(enrollment("src")), []byte("{"), ""); err != nil { t.Fatal(err) } before := f.claim().ETag if _, err := f.w.Step(context.Background(), host); err == nil || errors.Is(err, ErrNotYet) { t.Fatalf("an unreadable fence let the drain proceed: %v", err) } if f.claim().ETag != before { t.Fatal("the claim moved") } } func TestWorkerRefusesToJournalAPinAfterTheLeaseIsLost(t *testing.T) { f := newFixture(t) f.terminate("src") if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } f.ack("dst") f.worker(nil) // pinning f.loseLease = true o := f.worker(ErrNotYet) if o.Claim.Op.Journal["pin"] != "" { t.Fatal("a pin was journaled after its lease was lost: a park could have retired its versions") } } func TestWorkerStartRefusesAnIneligibleTargetAndANonRelocatableHost(t *testing.T) { f := newFixture(t) before := f.claim().ETag f.ineligible["dst"] = true if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err == nil { t.Fatal("started on a target the eligible set does not hold") } f.ineligible["dst"] = false shape := errors.New("unit-backed route") f.w.Relocatable = func(context.Context, hostclaim.Observed) error { return shape } if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); !errors.Is(err, shape) { t.Fatalf("started a non-relocatable host: %v", err) } if f.claim().ETag != before { t.Fatal("a refused start wrote the claim") } } func TestWorkerRefusesToDrainAHostThatNeverHadAWriter(t *testing.T) { f := newFixture(t) // A second host created on src and never woken: session 0. ctx := context.Background() const quiet = "quiet.bab.oncarlos.com" nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + quiet + ".json", Account: "bab", App: "jam", Host: quiet, Region: "ie", RecordRegion: "ie", CreatedAt: time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC)} body := []byte(`{"host":"` + quiet + `","account":"bab","app":"jam","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) o, err := hostclaim.Read(ctx, f.s, quiet) if err != nil { t.Fatal(err) } o = f.step(o, hostclaim.BeginCreateOn(nh, body, "c2", "console", f.now, hostclaim.ExecutorSelection{Box: "src", Enrollment: enrollment("src")})) f.step(o, hostclaim.CompleteCreate("c2", "src")) if _, err := f.w.Start(ctx, quiet, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); !errors.Is(err, ErrRefused) { t.Fatalf("drained a never-woken host: %v", err) } f.terminate("src") if _, err := f.w.Start(ctx, quiet, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); err != nil { t.Fatalf("a failover of a never-woken host: %v", err) } } func TestWorkerMarksALiveTargetsThirdReplacementStuck(t *testing.T) { f := newFixture(t) ctx := context.Background() if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } // Two terminated targets use the replacements; the third target is // alive but fails its probe, and its release names a fourth box: stuck. for _, dead := range []string{"dst", "spa"} { f.terminate(dead) f.worker(nil) } f.ack("src") f.ack("sp2") f.worker(nil) // releasing h := f.claim().Claim.Holder id := f.claim().Claim.Op.ID f.participant(hostclaim.JournalRelease(id, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) f.worker(nil) // pinning f.worker(nil) // pin f.worker(nil) // installing on sp2 f.installAndProbe("sp2") f.now = f.now.Add(ProbeDeadline + time.Second) if o := f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseProbeFailed { t.Fatalf("phase %s", o.Claim.Op.Phase) } h = f.claim().Claim.Holder f.participant(hostclaim.JournalRelease(id, hostclaim.Release{ReleasedBy: "sp2", Enrollment: enrollment("sp2"), Generation: h.Generation, Session: h.Session, FinalTXID: 41, ManifestSeq: 7, Receipt: "r2", ReceiptSHA256: strings.Repeat("d", 64)})) if o := f.worker(nil); o.Claim.Op.Journal["stuck"] == "" || o.Claim.Op.Phase != hostclaim.PhaseProbeFailed { t.Fatalf("a live target's third replacement must stop the operation: %+v", o.Claim.Op) } } func TestWorkerMarksExhaustionStuckBeforeLookingForASpare(t *testing.T) { // Two replacements used; no box is eligible for a third. The // operation is stuck at once, not retried forever on "no eligible box". f := newFixture(t) if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } for _, dead := range []string{"dst", "spa"} { f.terminate(dead) f.worker(nil) } f.w.Choose = func(context.Context, hostclaim.Observed, []string) (hostclaim.Target, error) { return hostclaim.Target{}, errors.New("no eligible box") } f.terminate("sp2") if o := f.worker(nil); o.Claim.Op.Journal["stuck"] == "" { t.Fatalf("exhaustion with no spare was not stuck: %+v", o.Claim.Op.Journal) } } func TestWorkerFencesAnExhaustedTargetThatDiedAfterItsLocalInstall(t *testing.T) { f := newFixture(t) ctx := context.Background() if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } for _, dead := range []string{"dst", "spa"} { f.terminate(dead) f.worker(nil) } f.ack("src") f.ack("sp2") f.worker(nil) // releasing h := f.claim().Claim.Holder id := f.claim().Claim.Op.ID f.participant(hostclaim.JournalRelease(id, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) f.worker(nil) // pinning f.worker(nil) // pin f.worker(nil) // installing on sp2 c := f.claim().Claim f.participant(func(c *hostclaim.Claim) error { c.Op.Journal["test.built"] = c.Op.Journal["prepared"]; return nil }) f.worker(nil) // installed var n uint64 _ = json.Unmarshal([]byte(f.claim().Claim.Op.Journal["install_stamp"]), &n) f.participant(hostclaim.RecordLocalInstall(c.Op.ID, hostclaim.LocalInstall{Box: "sp2", Enrollment: enrollment("sp2"), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("d", 64)})) f.terminate("sp2") // dead between installed and its probe o := f.worker(nil) if o.Claim.Op.Journal["stuck"] == "" || o.Claim.Op.Phase != hostclaim.PhasePinning || o.Claim.Op.Exec != "none" || !slices.Contains(o.Claim.Lost, "sp2") { t.Fatalf("an exhausted dead target was not fenced into a stuck pinning: %+v", o.Claim.Op) } before := o.Claim.Gen if o = f.worker(nil); o.Claim.Gen != before { t.Fatal("a worker moved a stuck operation") } } func TestWorkerMarksAnOverdueReleaseStuck(t *testing.T) { f := newFixture(t) if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } f.ack("src") f.ack("dst") f.worker(nil) // releasing f.now = f.now.Add(ReleaseDeadline) f.worker(ErrNotYet) // at the deadline: still waiting f.now = f.now.Add(time.Second) o := f.worker(nil) if o.Claim.Op.Journal["stuck"] == "" || o.Claim.Op.Phase != hostclaim.PhaseReleasing || o.Claim.Op.Exec != "src" { t.Fatalf("a source that never released must leave the operation stuck on the source: %+v", o.Claim.Op) } } // Round 5 finding 1: a spare that owes an older local retirement for the // host is never offered to the chooser; the worker replaces the dead // target with a clean spare (the claim refuses the other in its CAS). func TestWorkerNeverReplacesATargetWithABoxOwingALocalRetirement(t *testing.T) { f := newFixture(t) if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } f.participant(func(c *hostclaim.Claim) error { c.Obligations = append(c.Obligations, hostclaim.Obligation{ID: "old", Op: "old-op", Kind: "park-local", Box: "spa", Retirement: "r", Owner: "box:spa"}) return nil }) f.terminate("dst") o := f.worker(nil) if o.Claim.Op.Intent["target_box"] != "sp2" || slices.Contains(o.Claim.Op.Participants, "spa") { t.Fatalf("replacement: %+v participants %v", o.Claim.Op.Intent, o.Claim.Op.Participants) } } // Two-box model finding F2: a pin that cannot show the platform files // never existed (StorePinner's ErrFilesUnevidenced) stops the operation // for an operator instead of retrying or pinning empty. func TestWorkerMarksAnUnevidencedFilesPinStuck(t *testing.T) { f := newFixture(t) if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } f.ack("src") f.ack("dst") f.worker(nil) // releasing h := f.claim().Claim.Holder f.participant(hostclaim.JournalRelease(f.claim().Claim.Op.ID, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) f.worker(nil) // pinning f.pinErr = fmt.Errorf("%w: test", ErrFilesUnevidenced) if o := f.worker(nil); o.Claim.Op.Journal["stuck"] == "" || o.Claim.Op.Journal["pin"] != "" { t.Fatalf("an unevidenced files pin: %+v", o.Claim.Op) } } // Two-box model finding F1: a drain's target fails its probe and // releases; with no spare (two boxes: the source is excluded) the worker // waits, then, ReleaseDeadline after the target failed, marks the // operation stuck for an operator. func TestWorkerBoundsAProbeFailedWithNoSpare(t *testing.T) { for _, terminated := range []bool{false, true} { t.Run(fmt.Sprintf("target terminated after its release %v", terminated), func(t *testing.T) { boundsAProbeFailed(t, terminated) }) } } func boundsAProbeFailed(t *testing.T, terminated bool) { f := newFixture(t) ctx := context.Background() if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } f.ack("src") f.ack("dst") f.worker(nil) // releasing h := f.claim().Claim.Holder id := f.claim().Claim.Op.ID f.participant(hostclaim.JournalRelease(id, hostclaim.Release{ReleasedBy: "src", Enrollment: enrollment("src"), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)})) f.worker(nil) // pinning f.worker(nil) // pin f.worker(nil) // installing on dst f.installAndProbe("dst") f.now = f.now.Add(ProbeDeadline + time.Second) if o := f.worker(nil); o.Claim.Op.Phase != hostclaim.PhaseProbeFailed { t.Fatalf("phase %s", o.Claim.Op.Phase) } h = f.claim().Claim.Holder f.participant(hostclaim.JournalRelease(id, hostclaim.Release{ReleasedBy: "dst", Enrollment: enrollment("dst"), Generation: h.Generation, Session: h.Session, FinalTXID: 41, ManifestSeq: 7, Receipt: "r2", ReceiptSHA256: strings.Repeat("d", 64)})) noSpare := errors.New("no eligible box") f.w.Choose = func(context.Context, hostclaim.Observed, []string) (hostclaim.Target, error) { return hostclaim.Target{}, noSpare } if _, err := f.w.Step(ctx, host); !errors.Is(err, noSpare) || f.claim().Claim.Op.Journal["stuck"] != "" { t.Fatalf("before the deadline: %v %+v", err, f.claim().Claim.Op.Journal) } if terminated { // Round 17 finding 2: the released target is terminated. Its // fence goes to the replacement path first; with no spare that // path is bounded too, and consumes the fence as it stops. f.terminate("dst") if _, err := f.w.Step(ctx, host); !errors.Is(err, noSpare) || f.claim().Claim.Op.Journal["stuck"] != "" { t.Fatalf("a dead target, before the deadline: %v %+v", err, f.claim().Claim.Op.Journal) } } f.now = f.now.Add(ReleaseDeadline + time.Second) o := f.worker(nil) if o.Claim.Op.Journal["stuck"] == "" { t.Fatalf("no spare after the deadline: %+v", o.Claim.Op) } if terminated && (!slices.Contains(o.Claim.Lost, "dst") || o.Claim.Op.Journal["fence.dst"] == "" || o.Claim.Op.Exec != "none") { t.Fatalf("the dead target's fence was not consumed: %+v lost %v", o.Claim.Op, o.Claim.Lost) } if !terminated && o.Claim.Op.Phase != hostclaim.PhaseProbeFailed { t.Fatalf("a live failed target: %+v", o.Claim.Op) } } ``` `internal/hostrelocate/pin_test.go`: ```go package hostrelocate import ( "context" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "fmt" "strings" "testing" "time" "github.com/carlosframework/platform/internal/genselect" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostfiles" "github.com/carlosframework/platform/internal/release" ) const pinLocation = "instances/" + host + "/inc" func pinClaim() hostclaim.Observed { return hostclaim.Observed{Host: host, Claim: &hostclaim.Claim{Host: host, Gen: 12, Op: hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: hostclaim.PhasePinning, Journal: map[string]string{}}, Holder: &hostclaim.Holder{Location: pinLocation, Generation: "g3", Session: 3, Admissions: []hostclaim.AdmissionRecord{{Generation: "g2", Predecessor: "g1"}, {Generation: "g3", Predecessor: "g2"}}}}} } // keyed adds the host's instance.key (H6), as every manifest of a host // whose database is nonempty must carry (rev 18). func keyed(m hostfiles.Manifest) hostfiles.Manifest { sha := strings.Repeat("b", 64) m.Entries["instance.key"] = hostfiles.Entry{Object: pinLocation + "/files/g1/objects/0000000000000001-" + sha, SHA256: sha, Size: 32, Mode: 0o600, Seq: 1} return m } func putManifest(t *testing.T, data release.Store, gen string, m hostfiles.Manifest) (string, []byte) { t.Helper() raw, _ := json.Marshal(m) key := hostfiles.ManifestKey(pinLocation + "/files/" + gen + "/") if _, err := data.Put(context.Background(), key, raw, ""); err != nil { t.Fatal(err) } return key, raw } func selectAt(txid uint64) func(context.Context, genselect.Request) (*genselect.Plan, error) { return func(_ context.Context, req genselect.Request) (*genselect.Plan, error) { return &genselect.Plan{Generation: req.Holder.Generation, MaxTXID: txid}, nil } } func TestStorePinnerPinsTheNewestFilesManifestInTheChain(t *testing.T) { ctx := context.Background() claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) // g3 (holder.generation) never published files; g2 did, then g1 earlier. for gen, seq := range map[string]int64{"g2": 5, "g1": 9} { putManifest(t, data, gen, keyed(hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: seq, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}})) } selects := 0 p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: func(ctx context.Context, req genselect.Request) (*genselect.Plan, error) { selects++ return selectAt(41)(ctx, req) }} at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) pin, err := p.Pin(ctx, pinClaim(), at) if err != nil { t.Fatal(err) } if pin.FilesManifest != hostfiles.ManifestKey(pinLocation+"/files/g2/") || pin.FilesSeq != 5 || pin.MaxTXID != 41 || pin.Key != PinKey(host, "rel-1", 12) || pin.Empty { t.Fatalf("pin: %+v", pin) } obj, err := ReadPin(ctx, claims, pin) if err != nil { t.Fatal(err) } if want := []string{pinLocation + "/files/g3/", pinLocation + "/files/g2/", pinLocation + "/files/g1/"}; !equalStrings(obj.Files.Origins, want) || obj.ClaimGen != 12 { t.Fatalf("origins %v gen %d", obj.Files.Origins, obj.ClaimGen) } // The exact bytes are retained in the control bucket before the pin. kept, _, kerr := claims.Get(ctx, obj.Files.Retained) if kerr != nil || obj.Files.Retained != RetainedManifestKey(host, "rel-1", obj.Files.SHA256) { t.Fatalf("retained copy: %s %v", obj.Files.Retained, kerr) } if sum := sha256.Sum256(kept); hex.EncodeToString(sum[:]) != obj.Files.SHA256 { t.Fatal("the retained copy is not the pinned bytes") } // A retry at the same observation reuses the object unchanged. again, err := p.Pin(ctx, pinClaim(), at.Add(time.Minute)) if err != nil || again != pin || selects != 1 { t.Fatalf("replay: %+v %v selects=%d", again, err, selects) } // The claim moved (the journal CAS lost): the next pin is a fresh // selection at the new generation; the old object is never reused. moved := pinClaim() moved.Claim.Gen = 13 next, err := p.Pin(ctx, moved, at.Add(2*time.Minute)) if err != nil || next.Key != PinKey(host, "rel-1", 13) || selects != 2 || next.PinnedAt.Equal(pin.PinnedAt) { t.Fatalf("after the claim moved: %+v %v selects=%d", next, err, selects) } // An object at the key that is not this operation's is refused. if _, err := claims.Put(ctx, PinKey(host, "rel-2", 12), []byte(`{"v":1,"host":"other","op":"rel-2","claim_gen":12,"db":{}}`), ""); err != nil { t.Fatal(err) } other := pinClaim() other.Claim.Op.ID = "rel-2" if _, err := p.Pin(ctx, other, at); !errors.Is(err, ErrRefused) { t.Fatalf("foreign pin object: %v", err) } // A rewritten object is caught by the journaled digest. tampered := pin tampered.SHA256 = strings.Repeat("0", 64) if _, err := ReadPin(ctx, claims, tampered); !errors.Is(err, ErrRefused) { t.Fatalf("a pin that does not hash to its digest was read: %v", err) } } func equalStrings(a, b []string) bool { if len(a) != len(b) { return false } for i := range a { if a[i] != b[i] { return false } } return true } func TestStorePinnerChecksTheBoundsBeforeItWrites(t *testing.T) { ctx := context.Background() at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) release := func(txid uint64, seq int64) string { raw, _ := json.Marshal(hostclaim.Release{ReleasedBy: "src", Enrollment: "e", Generation: "g3", Session: 3, FinalTXID: txid, ManifestSeq: seq, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}) return string(raw) } for _, tc := range []struct { name string journal map[string]string txid uint64 seq int64 now time.Time }{ {"the listing is short of the release", map[string]string{"release": release(45, 5)}, 41, 5, at}, {"the files manifest is short of the release", map[string]string{"release": release(41, 6)}, 41, 5, at}, {"inside the failover window", map[string]string{"fence.src.observed_at": at.Format(time.RFC3339Nano)}, 41, 5, at.Add(29 * time.Second)}, } { t.Run(tc.name, func(t *testing.T) { claims, data := release_NewDirStore(t), release_NewDirStore(t) putManifest(t, data, "g3", keyed(hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: tc.seq, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}})) o := pinClaim() o.Claim.Op.Journal = tc.journal p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: selectAt(tc.txid)} if _, err := p.Pin(ctx, o, tc.now); !errors.Is(err, ErrNotYet) { t.Fatalf("got %v, want ErrNotYet", err) } if keys, _ := claims.List(ctx, "control/relocations/"); len(keys) != 0 { t.Fatalf("a pin that cannot be journaled was written: %v", keys) } }) } } func release_NewDirStore(t *testing.T) release.Store { return release.NewDirStore(t.TempDir()) } func TestStorePinnerPinsEmptyOnlyWithEvidence(t *testing.T) { ctx := context.Background() at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) empty := func(context.Context, genselect.Request) (*genselect.Plan, error) { return nil, genselect.ErrEmpty } released := func(txid uint64) string { raw, _ := json.Marshal(hostclaim.Release{ReleasedBy: "src", Enrollment: "e", Generation: "g3", Session: 3, FinalTXID: txid, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}) return string(raw) } for _, tc := range []struct { name string session uint64 journal map[string]string want string }{ {"a release at txid 0", 3, map[string]string{"release": released(0)}, "release"}, {"a never-woken host", 0, map[string]string{}, "session-0"}, {"a release at txid 40 and an empty listing", 3, map[string]string{"release": released(40)}, ""}, {"a failover of a host that had a writer", 3, map[string]string{}, ""}, } { t.Run(tc.name, func(t *testing.T) { claims, data := release_NewDirStore(t), release_NewDirStore(t) o := pinClaim() o.Claim.Holder.Session, o.Claim.Op.Journal = tc.session, tc.journal p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: empty} pin, err := p.Pin(ctx, o, at) if tc.want == "" { if !errors.Is(err, genselect.ErrEmpty) { t.Fatalf("an unevidenced empty listing was pinned: %+v %v", pin, err) } return } if err != nil || !pin.Empty || pin.MaxTXID != 0 || pin.Generation != "" { t.Fatalf("empty pin: %+v %v", pin, err) } obj, err := ReadPin(ctx, claims, pin) if err != nil || obj.Empty != tc.want || obj.DB != nil { t.Fatalf("object: %+v %v", obj, err) } }) } } func TestStorePinnerRefusesAnUnreadableManifest(t *testing.T) { ctx := context.Background() claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) if _, err := data.Put(ctx, hostfiles.ManifestKey(pinLocation+"/files/g3/"), []byte(`{"v":1}`), ""); err != nil { t.Fatal(err) } p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: selectAt(41)} if _, err := p.Pin(ctx, pinClaim(), time.Now()); !errors.Is(err, ErrRefused) { t.Fatalf("a version 1 manifest was pinned: %v", err) } } // pinnedManifest publishes gen's manifest whose one entry names an object // under origin (an earlier generation's prefix), and returns its pin. func pinnedManifest(t *testing.T, claims, data release.Store, gen, origin string, seq int64) (FilesPin, hostfiles.Manifest) { t.Helper() sum := sha256.Sum256([]byte("hello")) sha := hex.EncodeToString(sum[:]) object := origin + "objects/" + strings.Repeat("0", 15) + "1-" + sha if _, err := data.Put(context.Background(), object, []byte("hello"), ""); err != nil { t.Fatal(err) } m := hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: seq, Entries: map[string]hostfiles.Entry{"a.txt": {Object: object, SHA256: sha, Size: 5, Mode: 0o644, Seq: 1}}, Platform: map[string]hostfiles.Entry{}} _, raw := putManifest(t, data, gen, m) rawSum := sha256.Sum256(raw) manifestSHA := hex.EncodeToString(rawSum[:]) retained := RetainedManifestKey(host, "rel-1", manifestSHA) if _, err := claims.Put(context.Background(), retained, raw, ""); err != nil { t.Fatal(err) } prefix := pinLocation + "/files/" + gen + "/" return FilesPin{Prefix: prefix, Seq: seq, SHA256: manifestSHA, Origins: []string{prefix, origin}, Retained: retained}, m } func TestCopyPinnedManifestEnforcesThePin(t *testing.T) { ctx := context.Background() t.Run("a second relocation keeps objects of an earlier generation", func(t *testing.T) { claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) pin, want := pinnedManifest(t, claims, data, "P1", pinLocation+"/files/g1/", 3) got, err := CopyPinnedManifest(ctx, claims, data, pin, pinLocation+"/files/P2/") if err != nil || got.Seq != 0 || want.Seq != 3 || got.Entries["a.txt"] != want.Entries["a.txt"] { t.Fatalf("copy: %+v %v", got, err) } // A replay is a success; the copy's entries still name g1's object. if _, err := CopyPinnedManifest(ctx, claims, data, pin, pinLocation+"/files/P2/"); err != nil { t.Fatalf("replay: %v", err) } // Without the earlier generation among the origins the copy refuses. narrow := pin narrow.Origins = []string{pin.Prefix} if _, err := CopyPinnedManifest(ctx, claims, data, narrow, pinLocation+"/files/P3/"); !errors.Is(err, hostfiles.ErrOrigin) { t.Fatalf("an inherited object outside the origins: %v", err) } }) t.Run("a source manifest that moved after the pin is not read", func(t *testing.T) { // A late PUT to the source changes nothing: P is written from the // retained bytes, so an admitted pin stays usable (B2-4). claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) pin, m := pinnedManifest(t, claims, data, "P1", pinLocation+"/files/g1/", 3) m.Seq = 4 m.Entries["late.txt"] = m.Entries["a.txt"] raw, _ := json.Marshal(m) _, etag, _ := data.Get(ctx, hostfiles.ManifestKey(pin.Prefix)) if _, err := data.Put(ctx, hostfiles.ManifestKey(pin.Prefix), raw, etag); err != nil { t.Fatal(err) } got, err := CopyPinnedManifest(ctx, claims, data, pin, pinLocation+"/files/P2/") if _, late := got.Entries["late.txt"]; err != nil || got.Seq != 0 || late || len(got.Entries) != 1 { t.Fatalf("the pinned revision: %+v %v", got, err) } }) t.Run("a retained copy that is not the pinned bytes", func(t *testing.T) { claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) pin, m := pinnedManifest(t, claims, data, "P1", pinLocation+"/files/g1/", 3) e := m.Entries["a.txt"] e.Size = 6 m.Entries["a.txt"] = e raw, _ := json.Marshal(m) tampered := release.NewDirStore(t.TempDir()) if _, err := tampered.Put(ctx, pin.Retained, raw, ""); err != nil { t.Fatal(err) } if _, err := CopyPinnedManifest(ctx, tampered, data, pin, pinLocation+"/files/P2/"); !errors.Is(err, ErrRefused) { t.Fatalf("bytes the pin did not vouch for were copied: %v", err) } if _, _, err := data.Get(ctx, hostfiles.ManifestKey(pinLocation+"/files/P2/")); err == nil { t.Fatal("P's manifest was written from bytes the pin did not admit") } }) t.Run("a host that never published files", func(t *testing.T) { claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) pin := FilesPin{Prefix: pinLocation + "/files/g3/", Origins: []string{pinLocation + "/files/g3/"}} m, err := CopyPinnedManifest(ctx, claims, data, pin, pinLocation+"/files/P1/") if err != nil || m.Seq != 0 || len(m.Entries) != 0 { t.Fatalf("empty manifest: %+v %v", m, err) } if _, err := CopyPinnedManifest(ctx, claims, data, pin, pinLocation+"/files/P1/"); err != nil { t.Fatalf("replay: %v", err) } }) } // Round 11: holder.admissions keeps eight entries, but an unchanged file // published at g1 is still referenced by every later generation's // manifest (each copies the last, and unchanged content is never // rewritten). Ten real admissions and publications later, with g1 trimmed // from the admission list, the pin still admits g1's object and the copy // succeeds; a reference outside this incarnation's files refuses. func TestPinKeepsAFileOriginTheAdmissionListTrimmed(t *testing.T) { ctx := context.Background() claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + host + ".json", Account: "bab", App: "jam", Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created} body := []byte(`{"host":"` + host + `","account":"bab","app":"jam","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) step := func(o hostclaim.Observed, tr func(*hostclaim.Claim) error) hostclaim.Observed { t.Helper() next, err := hostclaim.Update(ctx, claims, o, tr) if err != nil { t.Fatal(err) } return next } o, err := hostclaim.Read(ctx, claims, host) if err != nil { t.Fatal(err) } o = step(o, hostclaim.BeginCreate(nh, body, "c", "console", created)) o = step(o, hostclaim.CompleteCreate("c", "src")) loc := o.Claim.Holder.Location n := 0 newID := func() string { n++; return fmt.Sprintf("id%d", n) } var gens []string prev, prevSeq := "", int64(0) for i := 1; i <= 10; i++ { gen := fmt.Sprintf("g%d", i) o = step(o, hostclaim.AdmitWriter(hostclaim.Admission{Region: "ie", PredecessorRegion: "ie", ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: "src", Mode: "cold", NewGeneration: gen, PredecessorCertified: true})) prefix := loc + "/files/" + gen + "/" gens = append(gens, prefix) if prev != "" { if _, err := hostfiles.CopyManifest(ctx, data, prev, prefix, prevSeq, gens); err != nil { t.Fatal(err) } } p, err := hostfiles.Open(ctx, data, prefix) if err != nil { t.Fatal(err) } // kept.txt and instance.key never change after g1; changed.txt // changes every time. if err := p.Stage(ctx, "kept.txt", []byte("written once"), 0o644, false); err != nil { t.Fatal(err) } if err := p.Stage(ctx, "instance.key", []byte("the host's key"), 0o600, false); err != nil { t.Fatal(err) } if err := p.Stage(ctx, "changed.txt", []byte(gen), 0o644, false); err != nil { t.Fatal(err) } if prevSeq, err = p.Commit(ctx); err != nil { t.Fatal(err) } prev = prefix o = step(o, hostclaim.PublishGeneration("", "", o.Claim.Holder.Session, gen, "ie", newID)) } h := o.Claim.Holder if len(h.Admissions) != 8 { t.Fatalf("admissions not trimmed to eight: %d", len(h.Admissions)) } for _, a := range h.Admissions { if a.Generation == "g1" || a.Predecessor == "g1" { t.Fatalf("g1 is still in the admission list: %+v", h.Admissions) } } m, _, err := hostfiles.Load(ctx, data, loc+"/files/g10/") if err != nil || !strings.HasPrefix(m.Entries["kept.txt"].Object, loc+"/files/g1/objects/") { t.Fatalf("the unchanged file is not g1's object: %+v %v", m.Entries["kept.txt"], err) } obs := o obs.Claim.Op.ID = "rel-1" p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: selectAt(41)} pin, err := p.Pin(ctx, obs, time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)) if err != nil { t.Fatal(err) } obj, err := ReadPin(ctx, claims, pin) if err != nil { t.Fatal(err) } got, err := CopyPinnedManifest(ctx, claims, data, obj.Files, loc+"/files/P1/") if err != nil || got.Entries["kept.txt"] != m.Entries["kept.txt"] { t.Fatalf("copy with g1 trimmed from the admissions: %+v %v", got.Entries, err) } // A reference outside this incarnation's files is never admitted. foreign := m foreign.Entries = map[string]hostfiles.Entry{"instance.key": m.Entries["instance.key"], "x.txt": {Object: "instances/" + host + "/other-inc/files/g1/objects/" + strings.Repeat("a", 64), Seq: 1, SHA256: strings.Repeat("a", 64)}} raw, _ := json.Marshal(foreign) _, etag, _ := data.Get(ctx, hostfiles.ManifestKey(loc+"/files/g10/")) if _, err := data.Put(ctx, hostfiles.ManifestKey(loc+"/files/g10/"), raw, etag); err != nil { t.Fatal(err) } obs.Claim.Gen++ if _, err := p.Pin(ctx, obs, time.Date(2026, 10, 7, 9, 1, 0, 0, time.UTC)); !errors.Is(err, hostfiles.ErrOrigin) { t.Fatalf("a foreign reference was pinned: %v", err) } } // Round 14 finding 1: the source's files manifest is at sequence 8; P's // prepared manifest is a create-only copy of its entries at sequence 0 // (claim rev 6.30 R18-1, moves rev 7.22 §7.6); the session publisher that // starts after the first admission takes it as its baseline and publishes // sequence 1, the inherited entries intact. func TestPreparedManifestIsSequenceZeroAndTheSessionPublishesOne(t *testing.T) { ctx := context.Background() claims, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir()) src := pinLocation + "/files/g3/" p, err := hostfiles.Open(ctx, data, src) if err != nil { t.Fatal(err) } for i := 1; i <= 7; i++ { if i == 7 { // The seventh revision also publishes the host's key (H6). if err := p.Stage(ctx, "instance.key", []byte("the host's key"), 0o600, false); err != nil { t.Fatal(err) } } if err := p.Stage(ctx, "a.txt", []byte(fmt.Sprintf("revision %d", i)), 0o644, false); err != nil { t.Fatal(err) } if _, err := p.Commit(ctx); err != nil { t.Fatal(err) } } pinner := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: selectAt(41)} ref, err := pinner.Pin(ctx, pinClaim(), time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)) if err != nil || ref.FilesSeq != 8 { // seven revisions of a.txt and the key: eight objects t.Fatalf("pin: %+v %v", ref, err) } obj, err := ReadPin(ctx, claims, ref) if err != nil { t.Fatal(err) } prepared := pinLocation + "/files/P1/" m, err := CopyPinnedManifest(ctx, claims, data, obj.Files, prepared) if err != nil || m.Seq != 0 { t.Fatalf("prepared manifest: %+v %v", m, err) } stored, _, err := hostfiles.Load(ctx, data, prepared) if err != nil || stored.Seq != 0 || stored.Entries["a.txt"].Object == "" || !strings.HasPrefix(stored.Entries["a.txt"].Object, src+"objects/") { t.Fatalf("stored prepared manifest: %+v %v", stored, err) } session, err := hostfiles.Open(ctx, data, prepared) if err != nil { t.Fatal(err) } if err := session.Stage(ctx, "b.txt", []byte("first session write"), 0o644, false); err != nil { t.Fatal(err) } seq, err := session.Commit(ctx) if err != nil || seq != 1 { t.Fatalf("the first session publication: %d %v", seq, err) } after, _, err := hostfiles.Load(ctx, data, prepared) if err != nil || after.Seq != 1 || after.Entries["a.txt"] != stored.Entries["a.txt"] || after.Entries["b.txt"].Object == "" { t.Fatalf("after the first publication: %+v %v", after, err) } } // Two-box model finding F2: no files manifest anywhere in the chain, and // a writer was admitted (session 3): the pin refuses rather than pin an // empty set an install would mint a fresh instance.key over. A holder // that never admitted a writer pins empty. func TestStorePinnerRefusesAnEmptyFilesPinAfterAWriter(t *testing.T) { ctx := context.Background() at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) for _, tc := range []struct { session uint64 want error }{{3, ErrFilesUnevidenced}, {0, nil}} { claims, data := release_NewDirStore(t), release_NewDirStore(t) o := pinClaim() o.Claim.Holder.Session = tc.session p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: selectAt(41)} pin, err := p.Pin(ctx, o, at) if !errors.Is(err, tc.want) { t.Fatalf("session %d: %+v %v", tc.session, pin, err) } } } // Round 17 finding 1 (P1): a manifest exists, copied before the app // started, without the instance.key the app minted afterwards; the writer // was admitted and the database is nonempty. The pin refuses (the worker // marks the operation stuck); with the key in the manifest, or an empty // database, it pins. func TestStorePinnerRefusesAKeylessManifestForANonemptyDatabase(t *testing.T) { ctx := context.Background() at := time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC) empty := func(context.Context, genselect.Request) (*genselect.Plan, error) { return nil, genselect.ErrEmpty } released0 := func() string { raw, _ := json.Marshal(hostclaim.Release{ReleasedBy: "src", Enrollment: "e", Generation: "g3", Session: 3, FinalTXID: 0, ManifestSeq: 4, Receipt: "r", ReceiptSHA256: strings.Repeat("d", 64)}) return string(raw) } for _, tc := range []struct { name string key bool db func(context.Context, genselect.Request) (*genselect.Plan, error) journal map[string]string want error }{ {"keyless, nonempty database", false, selectAt(41), map[string]string{}, ErrFilesUnevidenced}, {"keyed, nonempty database", true, selectAt(41), map[string]string{}, nil}, {"keyless, empty database at a release of 0", false, empty, map[string]string{"release": released0()}, nil}, } { t.Run(tc.name, func(t *testing.T) { claims, data := release_NewDirStore(t), release_NewDirStore(t) m := hostfiles.Manifest{V: hostfiles.ManifestVersion, Seq: 4, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}} if tc.key { m = keyed(m) } putManifest(t, data, "g3", m) o := pinClaim() o.Claim.Op.Journal = tc.journal p := StorePinner{Claims: claims, Data: data, Bucket: "data", Region: "ie", SelectDB: tc.db} if _, err := p.Pin(ctx, o, at); !errors.Is(err, tc.want) { t.Fatalf("got %v, want %v", err, tc.want) } }) } } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostrelocate/ -count=1` Expected: FAIL: `no non-test Go files` / `undefined: Worker`. - [ ] **Step 3: The worker** `internal/hostrelocate/worker.go`: ```go // Package hostrelocate is the regional worker's half of relocation (region // HA spec 1 §5.8, claim §4 L1): starting a relocation, and replaying every // worker CAS from the claim alone. Any active enrolled box of the region // may act (claim P6-D5); correctness never depends on which, because every // step is a claim CAS conditional on the operation, its phase and the // claim's ETag. The participants' halves (acknowledgement, release, // install, probe) are not here: they run on the boxes, under the host // guard, through Plan 6b's box assembly. // // This package only READS termination fences. hostfence.TerminationWorker // writes them (control/box-fences/.json). package hostrelocate import ( "context" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "fmt" "os" "slices" "strconv" "time" "github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostfence" "github.com/carlosframework/platform/internal/release" "github.com/carlosframework/platform/internal/termfence" ) const ( // ProbeDeadline is §5.8's probe deadline: a live target whose probe has // not passed this long after entering probing goes to probe-failed. ProbeDeadline = 10 * time.Minute // ReleaseDeadline is this plan's stopping policy for a release that // never completes (§5.8: "a flush that never reaches N goes to stuck"; // "a probe-failed target that cannot finish its release sequence also // goes to stuck"). The spec names no number; this one matches the // probe deadline. A live box whose release is not journaled this long // after entering releasing or probe-failed leaves the operation stuck // for the operator: nothing moves away from a live box without its // stop evidence. ReleaseDeadline = 10 * time.Minute ) var ( ErrUnsupported = errors.New("hostrelocate: a required provider is missing") ErrNotWorker = errors.New("hostrelocate: this box is not an active, unfenced worker of the region") ErrNotYet = errors.New("hostrelocate: waiting for a participant or a time bound") ErrRefused = errors.New("hostrelocate: relocation refused") ErrNoController = errors.New("hostrelocate: no execution controller is open for this participation") ) // Lease is the replica lease on holder.location (claim C3), as // lease.Acquire returns it. The worker holds it from selecting a pin until // the pin is journaled, so a park worker (which takes the same lease to // retire) cannot retire the versions in between (claim §4: "New pins must // be established under the required leases and fresh claim authority // before conflicting retirement can proceed"). type Lease interface { Lost() <-chan struct{} Release(context.Context) error } // Request asks for one relocation. Target is required; Source must be the // claim's executor. Reason is drain, rebalance or failover. type Request struct { Reason, Source, Target string } // Owner is the app-admission registration owner of a box-started // operation (claim §5.1, "A regional worker starting relocating"). type Owner struct { Box, Enrollment, Controller string } // Admission is claim §5.1's app-admission registration for a box-started // relocation: Register binds the saved claim observation and holder // incarnation while admission is open; Recheck is the final check before // the single start CAS. Claim Plan 5's app gate implements it. type Admission interface { Register(context.Context, hostclaim.Observed, Owner, string) (string, error) Recheck(context.Context, string) error } // Pinner selects, writes and returns one pinned recovery set (§5.8). type Pinner interface { Pin(context.Context, hostclaim.Observed, time.Time) (hostclaim.RecoveryPin, error) } // Prepared verifies the remote build of the operation's prepared // generation P (§5.8 "Building the prepared generation"): done is true only // when a fresh read verifies P's snapshot, level-0 anchor and completed // install manifest (or, for a host with nothing replicated, P's files // manifest alone). Moves' shared install routine (L16) writes them. type Prepared interface { Verify(context.Context, hostclaim.Observed) (a hostclaim.PreparedAuthority, done bool, err error) } // Worker drives relocation for one region. Every field is required. type Worker struct { Claims release.Store // the deployment (control) bucket Directory hostactors.Directory Fences hostclaim.FenceAuthority // hostfence.ClaimFenceAuthority Releases hostclaim.ReleaseAuthority // Plan 6b's stop receipt reader Admission Admission // Census is the box-started start census (claim D7 with L5 and L20): // hostactors.Directory.CensusBoxRelocation with the deployment's // evidence and floor. Census func(ctx context.Context, starter string, participants []string) error // Choose names a target in the operation's region, never one in // exclude (hostops.ExecutorChooser behind an adapter). Choose func(ctx context.Context, obs hostclaim.Observed, exclude []string) (hostclaim.Target, error) // Eligible refuses a named target that the complete eligible set // (§2, boxchoice.Eligible over the roster and the edges files) does // not hold: stale edges, a missing capability or secrets key, a // draining box (hostops.ExecutorChooser.Permits). Eligible func(ctx context.Context, obs hostclaim.Observed, box string) error // Relocatable refuses a host whose authoritative record is not a // relocatable shape: a unit- or socket-backed route, a tunnel, a // sidecar (the same shapes Plan 6b's wake refuses). Relocatable func(ctx context.Context, obs hostclaim.Observed) error // Leases acquires the replica lease on the host's location in the // region's data bucket (lease.Acquire). Leases func(ctx context.Context, location string) (Lease, error) Pins Pinner Prepared Prepared // Discharge judges a fenced box's volume records (boxvolumes.Discharge). // It never gates anything: nil, or an unqualified verdict, leaves the // box's park-local chores open for an operator (H19). Discharge func(context.Context, hostactors.Box, termfence.Record) *hostclaim.LocalDischarge // Notify bumps the control serial after an executor CAS (§5.3, // "Prompt changes"). A failed bump costs cadence, never correctness. Notify func(context.Context) Self termfence.WrittenBy Region string Controller string NewID func() string Now func() time.Time } func (w Worker) ready() error { if w.Claims == nil || w.Directory.Store == nil || w.Fences == nil || w.Releases == nil || w.Admission == nil || w.Census == nil || w.Choose == nil || w.Eligible == nil || w.Relocatable == nil || w.Leases == nil || w.Pins == nil || w.Prepared == nil || w.Discharge == nil || w.Notify == nil || w.Self.Box == "" || w.Self.Enrollment == "" || w.Region == "" || w.Controller == "" || w.NewID == nil || w.Now == nil { return ErrUnsupported } return nil } // self checks, on every call, that this box is an active, unfenced worker // enrolled in the region (§5.8 "Each worker first checks its own roster // entry is active in this region"). func (w Worker) self(ctx context.Context) (hostactors.Observed, error) { o, err := w.Directory.Read(ctx) if err != nil { return hostactors.Observed{}, err } b, ok := o.Roster.Boxes[w.Self.Box] if !ok || b.State != "active" || b.Enrollment != w.Self.Enrollment || b.Region != w.Region { return hostactors.Observed{}, ErrNotWorker } fenced, err := w.Directory.TerminationFence(ctx, o, w.Self.Box, w.Now()) if err != nil || fenced { return hostactors.Observed{}, errors.Join(ErrNotWorker, err) } return o, nil } // fence reads box's termination fence, if any, as the exact bytes a claim // CAS will cite. Absence is never evidence: no fence is (nil, nil); an // unreadable or invalid one is an error, never "not fenced". func (w Worker) fence(ctx context.Context, o hostactors.Observed, box string) (*hostclaim.FenceRef, termfence.Record, error) { b, ok := o.Roster.Boxes[box] if !ok { return nil, termfence.Record{}, fmt.Errorf("%w: %s is not in the roster", ErrRefused, box) } key := hostfence.FenceKey(b.Enrollment) raw, _, err := w.Directory.Store.Get(ctx, key) if errors.Is(err, os.ErrNotExist) { return nil, termfence.Record{}, nil } if err != nil { return nil, termfence.Record{}, err } var f termfence.Record if err := json.Unmarshal(raw, &f); err != nil { return nil, termfence.Record{}, err } if err := hostactors.ValidateTerminationFence(o, box, f, w.Now()); err != nil { return nil, termfence.Record{}, err } sum := sha256.Sum256(raw) return &hostclaim.FenceRef{Enrollment: b.Enrollment, Key: key, SHA256: hex.EncodeToString(sum[:])}, f, nil } // Start writes the start CAS for req (§5.8 "The start CAS", §5.10). It // refuses unless the claim serves on req.Source; the roster shows the // starter, the target and (for a drain) the source active, unfenced and // with no executable update reserved (H14); the census passes; and the // app admits the registration. A failover requires the source's fence. func (w Worker) Start(ctx context.Context, host string, req Request) (hostclaim.Observed, error) { if err := w.ready(); err != nil { return hostclaim.Observed{}, err } o, err := w.self(ctx) if err != nil { return hostclaim.Observed{}, err } obs, err := hostclaim.Read(ctx, w.Claims, host) if err != nil { return hostclaim.Observed{}, err } if obs.Absent || obs.Claim.Op.Kind != hostclaim.KindServing || obs.Claim.Op.Exec != req.Source || req.Target == req.Source { return hostclaim.Observed{}, fmt.Errorf("%w: %s does not serve on %s", ErrRefused, host, req.Source) } if obs.Claim.Holder.Family != "instance" { return hostclaim.Observed{}, fmt.Errorf("%w: a %s host is not relocatable", ErrRefused, obs.Claim.Holder.Family) } if err := w.Relocatable(ctx, obs); err != nil { return hostclaim.Observed{}, err } if err := w.Eligible(ctx, obs, req.Target); err != nil { return hostclaim.Observed{}, err } source, sok := o.Roster.Boxes[req.Source] target, tok := o.Roster.Boxes[req.Target] if !sok || !tok || source.Region != w.Region || target.Region != w.Region { return hostclaim.Observed{}, fmt.Errorf("%w: source and target must be enrolled in %s", ErrRefused, w.Region) } ref, record, err := w.fence(ctx, o, req.Source) if err != nil { return hostclaim.Observed{}, err } refs := []string{w.Self.Box, req.Target} switch req.Reason { case hostclaim.ReasonFailover: if ref == nil { return hostclaim.Observed{}, fmt.Errorf("%w: a failover needs the source's termination fence", ErrRefused) } case hostclaim.ReasonDrain, hostclaim.ReasonRebalance: if ref != nil { return hostclaim.Observed{}, fmt.Errorf("%w: the source is fenced; start a failover", ErrRefused) } if obs.Claim.Holder.Session == 0 { // Ruling 11 (rev 2): a never-woken source has no writer session // to stop, and root refuses stop evidence for session 0 until // claim Plan 5's no-process witness lands. Until then only a // failover relocates such a host. return hostclaim.Observed{}, fmt.Errorf("%w: %s never admitted a writer on %s; a drain has nothing to release", ErrRefused, host, req.Source) } refs = append(refs, req.Source) default: return hostclaim.Observed{}, fmt.Errorf("%w: reason %q", ErrRefused, req.Reason) } refs = slices.Compact(slices.Sorted(slices.Values(refs))) revision, err := w.Directory.CheckBoxStart(ctx, refs...) if err != nil { return hostclaim.Observed{}, err } if err := w.Census(ctx, w.Self.Box, refs); err != nil { return hostclaim.Observed{}, err } id := w.NewID() admission, err := w.Admission.Register(ctx, obs, Owner{Box: w.Self.Box, Enrollment: w.Self.Enrollment, Controller: w.Controller}, id) if err != nil { return hostclaim.Observed{}, err } if err := w.Admission.Recheck(ctx, admission); err != nil { return hostclaim.Observed{}, err } in := hostclaim.RelocationStart{OpID: id, By: "box:" + w.Self.Box, Now: w.Now(), Region: w.Region, Reason: req.Reason, SourceBox: req.Source, SourceEnrollment: source.Enrollment, TargetBox: req.Target, TargetEnrollment: target.Enrollment, DirectoryRevision: revision, AdmissionID: admission, Fence: ref} if ref != nil { in.Discharge = w.Discharge(ctx, source, record) } next, err := hostclaim.StartRelocation(ctx, w.Claims, obs, in, w.Fences) if err == nil && ref != nil { w.Notify(ctx) } return next, err } // Step makes at most one worker CAS for host's relocation, the one its // phase and evidence call for, and returns the claim it left. ErrNotYet // means a participant or a time bound is awaited; nothing was written. // A host that is not relocating, or whose operation is stuck, is left // alone. func (w Worker) Step(ctx context.Context, host string) (hostclaim.Observed, error) { if err := w.ready(); err != nil { return hostclaim.Observed{}, err } o, err := w.self(ctx) if err != nil { return hostclaim.Observed{}, err } obs, err := hostclaim.Read(ctx, w.Claims, host) if err != nil || obs.Absent { return obs, err } c := obs.Claim if c.Op.Kind != hostclaim.KindRelocating || c.Op.Intent["region"] != w.Region || c.Op.Journal["stuck"] != "" { return obs, nil } id := c.Op.ID // Evidence first: a fenced participant changes what every phase may do. if src := c.Op.Intent["source_box"]; !slices.Contains(c.Lost, src) { ref, record, err := w.fence(ctx, o, src) if err != nil { return obs, err } if ref != nil { next, err := hostclaim.FenceSource(ctx, w.Claims, obs, id, *ref, w.Discharge(ctx, o.Roster.Boxes[src], record), w.Fences) if err == nil { w.Notify(ctx) } return next, err } } if dst := c.Op.Intent["target_box"]; dst != "" && !slices.Contains(c.Lost, dst) { ref, record, err := w.fence(ctx, o, dst) if err != nil { return obs, err } if ref != nil { return w.replace(ctx, obs, *ref, w.Discharge(ctx, o.Roster.Boxes[dst], record), record.ObservedAt) } } switch c.Op.Phase { case hostclaim.PhaseStarting: if !acked(c) { return obs, ErrNotYet } if c.Op.Exec == "none" { return w.update(ctx, obs, hostclaim.PinFailover(id)) } return w.update(ctx, obs, hostclaim.ToReleasing(id, w.Now())) case hostclaim.PhaseReleasing: if c.Op.Journal["release"] == "" { return w.releaseOverdue(ctx, obs, "releasing_since") } return hostclaim.PinAfterRelease(ctx, w.Claims, obs, id, w.Releases, nil) case hostclaim.PhaseProbeFailed: if c.Op.Journal["release"] == "" { return w.releaseOverdue(ctx, obs, "probe_failed_since") } if exhausted(c) { return w.stuckOn(ctx, obs, hostclaim.ErrReplacementsExceeded) } next, err := w.next(ctx, obs) if err != nil { // No eligible box yet; a later pass may find one, but not for // ever: in a two-box region the source is excluded and there // may be none (rev 17, finding F1). ReleaseDeadline after the // target failed, the operation is stuck for an operator. since, perr := time.Parse(time.RFC3339Nano, c.Op.Journal["probe_failed_since"]) if perr == nil && w.Now().Sub(since) > ReleaseDeadline { return w.update(ctx, obs, hostclaim.MarkStuck(id, fmt.Sprintf("no eligible box for the nested relocation after %s: %v", ReleaseDeadline, err))) } return obs, err } out, err := hostclaim.PinAfterRelease(ctx, w.Claims, obs, id, w.Releases, &next) if err != nil { // A live target's replacement counts too: the third is stuck. return w.stuckOn(ctx, obs, err) } return out, nil case hostclaim.PhasePinning: return w.pinning(ctx, obs) case hostclaim.PhaseProbing: since, err := time.Parse(time.RFC3339Nano, c.Op.Journal["probing_since"]) if err != nil { return obs, err } if c.Op.Journal["probe"] != "" { next, err := w.update(ctx, obs, hostclaim.CompleteRelocation(id, w.NewID)) if err == nil { w.Notify(ctx) } return next, err } if w.Now().Sub(since) > ProbeDeadline { return w.update(ctx, obs, hostclaim.ToProbeFailed(id, w.Now())) } return obs, ErrNotYet case hostclaim.PhaseInstalling: a, done, err := w.Prepared.Verify(ctx, obs) if err != nil || !done || !acked(c) { return obs, errors.Join(ErrNotYet, err) } return w.update(ctx, obs, hostclaim.ToInstalled(id, a)) } // installed is the target's: its local install and its probe (Plan 6b). return obs, ErrNotYet } func (w Worker) pinning(ctx context.Context, obs hostclaim.Observed) (hostclaim.Observed, error) { c := obs.Claim id := c.Op.ID if c.Op.Intent["target_box"] == "" { next, err := w.next(ctx, obs) if err != nil { return obs, err } return w.update(ctx, obs, hostclaim.NameTarget(id, next)) } if c.Op.Journal["pin"] == "" { for key, at := range c.Op.Journal { if len(key) > len(".observed_at") && key[len(key)-len(".observed_at"):] == ".observed_at" { t, err := time.Parse(time.RFC3339Nano, at) if err != nil { return obs, err } if w.Now().Before(t.Add(hostclaim.FailoverPinDelay)) { return obs, ErrNotYet } } } return w.pinUnderLease(ctx, obs) } t := hostclaim.Target{Box: c.Op.Intent["target_box"], Enrollment: c.Op.Intent["target_enrollment"]} if !acked(c) { return obs, ErrNotYet } next, err := w.update(ctx, obs, hostclaim.ToInstalling(id, t, w.NewID())) if err == nil { w.Notify(ctx) } return next, err } // pinUnderLease selects and journals the pin while holding the replica // lease on holder.location, so no park can retire the pinned versions // between the selection and the journal CAS (claim §4). A lease lost // before the CAS refuses: the next pass selects afresh. func (w Worker) pinUnderLease(ctx context.Context, obs hostclaim.Observed) (hostclaim.Observed, error) { held, err := w.Leases(ctx, obs.Claim.Holder.Location) if err != nil { return obs, err } defer func() { _ = held.Release(ctx) }() pin, err := w.Pins.Pin(ctx, obs, w.Now()) if errors.Is(err, ErrFilesUnevidenced) { // H6: instance.key, report.token and the env are continuous; a pin // that cannot show them never existed waits for an operator. return w.update(ctx, obs, hostclaim.MarkStuck(obs.Claim.Op.ID, err.Error())) } if err != nil { return obs, err } select { case <-held.Lost(): return obs, fmt.Errorf("%w: the location lease was lost before the pin was journaled", ErrNotYet) default: } return w.update(ctx, obs, hostclaim.JournalPin(obs.Claim.Op.ID, pin)) } // releaseOverdue marks the operation stuck when a live box has not // journaled its release ReleaseDeadline after entering the phase whose // start key names; before that it waits. func (w Worker) releaseOverdue(ctx context.Context, obs hostclaim.Observed, sinceKey string) (hostclaim.Observed, error) { since, err := time.Parse(time.RFC3339Nano, obs.Claim.Op.Journal[sinceKey]) if err != nil { return obs, err } if w.Now().Sub(since) <= ReleaseDeadline { return obs, ErrNotYet } return w.update(ctx, obs, hostclaim.MarkStuck(obs.Claim.Op.ID, fmt.Sprintf("%s has not released %s after %s", obs.Claim.Op.Exec, ReleaseDeadline, obs.Claim.Op.Phase))) } // next names the operation's next target: never the source, a lost box, // a box this operation already replaced, or a box that owes a park-local // chore for the host (spec 1 §2; ReplaceTarget refuses one in its CAS). func (w Worker) next(ctx context.Context, obs hostclaim.Observed) (hostclaim.Target, error) { c := obs.Claim exclude := append([]string{c.Op.Intent["source_box"]}, c.Lost...) exclude = append(exclude, hostclaim.LocalChoreBoxes(c)...) if t := c.Op.Intent["target_box"]; t != "" { exclude = append(exclude, t) } var replaced []string if raw := c.Op.Journal["excluded_targets"]; raw != "" { if err := json.Unmarshal([]byte(raw), &replaced); err != nil { return hostclaim.Target{}, err } } return w.Choose(ctx, obs, append(exclude, replaced...)) } func (w Worker) replace(ctx context.Context, obs hostclaim.Observed, ref hostclaim.FenceRef, d *hostclaim.LocalDischarge, observedAt string) (hostclaim.Observed, error) { // Exhaustion is checked before a box is chosen (§5.8 "The bound"): // with the replacements used, no spare is looked for; the dead // target's fence is consumed into a stuck pinning (the loss retained, // its installation's authority revoked, no probe to wait for). if exhausted(obs.Claim) { out, err := hostclaim.FenceExhaustedTarget(ctx, w.Claims, obs, obs.Claim.Op.ID, ref, d, w.Fences) if err == nil { w.Notify(ctx) } return out, err } next, err := w.next(ctx, obs) if err != nil { // No eligible box to replace the dead target. A later pass may // find one, but not for ever (rev 18, round 17 finding 2): // ReleaseDeadline after the fence's observation the fence is // consumed and the operation is stuck for an operator. at, perr := time.Parse(time.RFC3339Nano, observedAt) if perr == nil && w.Now().Sub(at) > ReleaseDeadline { out, ferr := hostclaim.FenceTargetWithoutSpare(ctx, w.Claims, obs, obs.Claim.Op.ID, ref, d, w.Fences, fmt.Sprintf("%s terminated and no eligible box replaced it after %s: %v", obs.Claim.Op.Intent["target_box"], ReleaseDeadline, err)) if ferr == nil { w.Notify(ctx) } return out, ferr } return obs, err } out, err := hostclaim.ReplaceTarget(ctx, w.Claims, obs, obs.Claim.Op.ID, ref, d, next, w.NewID(), w.Fences) if errors.Is(err, hostclaim.ErrReplacementsExceeded) { return w.stuckOn(ctx, obs, err) } if err == nil { w.Notify(ctx) } return out, err } // stuckOn marks the operation stuck when it has used its replacements; // any other error is returned for a later pass to retry. func (w Worker) stuckOn(ctx context.Context, obs hostclaim.Observed, cause error) (hostclaim.Observed, error) { if !errors.Is(cause, hostclaim.ErrReplacementsExceeded) { return obs, cause } return w.update(ctx, obs, hostclaim.MarkStuck(obs.Claim.Op.ID, cause.Error())) } func (w Worker) update(ctx context.Context, obs hostclaim.Observed, tr func(*hostclaim.Claim) error) (hostclaim.Observed, error) { return hostclaim.Update(ctx, w.Claims, obs, tr) } // exhausted reports whether the operation has used its target replacements. func exhausted(c *hostclaim.Claim) bool { n, _ := strconv.ParseUint(c.Op.Journal["replacements"], 10, 64) return n >= hostclaim.MaxTargetReplacements } // acked is the D6 barrier as the worker reads it before a CAS that needs it. func acked(c *hostclaim.Claim) bool { for _, box := range c.Op.Participants { if c.Op.Journal["exec_ack."+box] == "" { return false } } return true } ``` - [ ] **Step 4: The pin** `internal/hostrelocate/pin.go`: ```go package hostrelocate import ( "context" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "fmt" "os" "sort" "strings" "time" "github.com/carlosframework/platform/internal/genselect" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostfiles" "github.com/carlosframework/platform/internal/release" ) // PinObject is the immutable pinned recovery set (§5.8): the database // plan genselect selected, with every file's exact version, and the files // pin, the newest manifest in the admitted chain, by digest (spec §18: the // store has no version-id read, so content is identified by SHA-256). // ClaimGen is the claim generation the set was selected against; the // object's key names it (hostclaim.RecoveryPinKey), so a pin whose journal // CAS lost is never reused. DB is nil for an evidenced empty bootstrap. type PinObject struct { V int `json:"v"` Host string `json:"host"` Op string `json:"op"` ClaimGen uint64 `json:"claim_gen"` DB *genselect.Plan `json:"db,omitempty"` Empty string `json:"empty,omitempty"` // "release" | "session-0": the evidence Files FilesPin `json:"files"` PinnedAt time.Time `json:"pinned_at"` } // FilesPin names one generation's files manifest by prefix, sequence and // the digest of its exact bytes, and the origins its entries may name: // every generation prefix in the admitted chain as observed at the pin, // the manifest's own included (contracts note §1). type FilesPin struct { Prefix string `json:"prefix"` Seq int64 `json:"seq"` SHA256 string `json:"sha256"` Origins []string `json:"origins"` // Retained is the pinner's immutable copy of the exact manifest bytes // (RetainedManifestKey), written create-only before the pin object. A // late revision of the source manifest changes nothing the build // reads: P is written from this copy, never from the source again. // Empty for a host that never published files. Retained string `json:"retained,omitempty"` } // RetainedManifestKey is where the pinner keeps the pinned manifest's // bytes: control/relocations///manifest-.json. func RetainedManifestKey(host, op, sha string) string { return "control/relocations/" + host + "/" + op + "/manifest-" + sha + ".json" } // PinKey is hostclaim.RecoveryPinKey. func PinKey(host, op string, gen uint64) string { return hostclaim.RecoveryPinKey(host, op, gen) } // StorePinner selects the pinned recovery set against the claim // observation it is given (§5.8 "The pinned recovery set") and writes it // create-only at the key for that observation's claim generation. It // checks every bound JournalPin will check before it writes anything, so // a doomed pin is never published; a retry at the same generation reuses // its own object unchanged. The caller holds the location lease (claim §4: // "New pins must be established under the required leases"). type StorePinner struct { Claims release.Store // control bucket: pin objects Data release.Store // the region's data bucket: files manifests SelectDB func(context.Context, genselect.Request) (*genselect.Plan, error) Bucket string // the region's data bucket name, for genselect Region string } func (p StorePinner) Pin(ctx context.Context, obs hostclaim.Observed, now time.Time) (hostclaim.RecoveryPin, error) { if p.Claims == nil || p.Data == nil || p.SelectDB == nil || p.Bucket == "" || p.Region == "" || obs.Claim == nil || obs.Claim.Holder == nil { return hostclaim.RecoveryPin{}, ErrUnsupported } c := obs.Claim key := PinKey(c.Host, c.Op.ID, c.Gen) if raw, _, err := p.Claims.Get(ctx, key); err == nil { return p.reuse(c, key, raw) } else if !errors.Is(err, os.ErrNotExist) { return hostclaim.RecoveryPin{}, err } var rel hostclaim.Release released := json.Unmarshal([]byte(c.Op.Journal["release"]), &rel) == nil && rel.ReleasedBy != "" for k, at := range c.Op.Journal { if len(k) > len(".observed_at") && k[len(k)-len(".observed_at"):] == ".observed_at" { t, err := time.Parse(time.RFC3339Nano, at) if err != nil { return hostclaim.RecoveryPin{}, err } if now.Before(t.Add(hostclaim.FailoverPinDelay)) { return hostclaim.RecoveryPin{}, ErrNotYet } } } obj := PinObject{V: 1, Host: c.Host, Op: c.Op.ID, ClaimGen: c.Gen, PinnedAt: now.UTC()} plan, err := p.SelectDB(ctx, genselect.Request{Bucket: p.Bucket, Region: p.Region, Host: c.Host, Holder: *c.Holder}) switch { case err == nil: obj.DB = plan case errors.Is(err, genselect.ErrEmpty) && released && rel.FinalTXID == 0: obj.Empty = "release" // the stopped writer flushed and reports position 0 case errors.Is(err, genselect.ErrEmpty) && !released && c.Holder.Session == 0: obj.Empty = "session-0" // no writer was ever admitted, so nothing was written default: // A listing that finds nothing after a writer ran is not evidence // of an empty database (§5.8, claim §6.4a): refuse, never bootstrap. return hostclaim.RecoveryPin{}, err } if released && obj.DB != nil && obj.DB.MaxTXID < rel.FinalTXID { return hostclaim.RecoveryPin{}, fmt.Errorf("%w: the listing is at %d, the release at %d", ErrNotYet, obj.DB.MaxTXID, rel.FinalTXID) } // A nonempty database written by an admitted writer is sealed with // that writer's instance.key: the pinned manifest must carry it (rev // 18, round 17 finding 1). needKey := c.Holder.Session != 0 && obj.DB != nil && obj.DB.MaxTXID > 0 files, retained, err := p.filesPin(ctx, c.Holder, released && rel.ManifestSeq == 0, needKey) if err != nil { return hostclaim.RecoveryPin{}, err } obj.Files = files if released && obj.Files.Seq < rel.ManifestSeq { return hostclaim.RecoveryPin{}, fmt.Errorf("%w: the files manifest is at %d, the release at %d", ErrNotYet, obj.Files.Seq, rel.ManifestSeq) } if retained != nil { // The exact bytes, kept where nothing rewrites them, before the // pin that names them exists. obj.Files.Retained = RetainedManifestKey(c.Host, c.Op.ID, files.SHA256) if err := release.CreateExact(ctx, p.Claims, obj.Files.Retained, retained); err != nil && !errors.Is(err, release.ErrCASConflict) { return hostclaim.RecoveryPin{}, err } } raw, err := json.Marshal(obj) if err != nil { return hostclaim.RecoveryPin{}, err } if err := release.CreateExact(ctx, p.Claims, key, raw); err != nil { if !errors.Is(err, release.ErrCASConflict) { return hostclaim.RecoveryPin{}, err } got, _, gerr := p.Claims.Get(ctx, key) // another worker pinned this generation first if gerr != nil { return hostclaim.RecoveryPin{}, gerr } return p.reuse(c, key, got) } return pinRef(key, raw, obj), nil } func (p StorePinner) reuse(c *hostclaim.Claim, key string, raw []byte) (hostclaim.RecoveryPin, error) { var obj PinObject if err := json.Unmarshal(raw, &obj); err != nil || obj.V != 1 || obj.Host != c.Host || obj.Op != c.Op.ID || obj.ClaimGen != c.Gen || (obj.DB == nil) == (obj.Empty == "") { return hostclaim.RecoveryPin{}, fmt.Errorf("%w: pin object %s is not this operation's at this generation", ErrRefused, key) } return pinRef(key, raw, obj), nil } func pinRef(key string, raw []byte, obj PinObject) hostclaim.RecoveryPin { sum := sha256.Sum256(raw) ref := hostclaim.RecoveryPin{Key: key, SHA256: hex.EncodeToString(sum[:]), FilesManifest: hostfiles.ManifestKey(obj.Files.Prefix), FilesSeq: obj.Files.Seq, PinnedAt: obj.PinnedAt, Empty: obj.DB == nil} if obj.DB != nil { ref.Generation, ref.MaxTXID = obj.DB.Generation, obj.DB.MaxTXID } return ref } // InstanceKeyPath is H6's always-continuous instance.key, at the root of // the host's files directory. const InstanceKeyPath = "instance.key" func liveEntry(m hostfiles.Manifest, path string) bool { e, ok := m.Entries[path] return ok && !e.Deleted && e.Skipped == "" && e.Object != "" } // ErrFilesUnevidenced is a pin refused because its files set cannot be // shown to hold the host's platform files: a writer was admitted and no // files manifest exists in the admitted chain (rev 17), or the database // is nonempty and the pinned manifest has no instance.key (rev 18). The // worker marks the operation stuck. var ErrFilesUnevidenced = errors.New("hostrelocate: the files evidence does not cover the host's platform files") // filesPin is §5.6's files pin: the manifest of holder.generation as // observed or, if that generation has none, of the nearest predecessor in // holder.admissions that has one. It is chosen independently of the // database baseline (B2-4). A read error refuses; no manifest anywhere in // the chain is a host that never published files, pinned at sequence 0. // Origins is the retained chain plus every generation of this incarnation // whose objects the pinned manifest references (rev 12, round 11): the // claim's §6.2a files live set is the objects the live manifests // reference, independent of holder.admissions, which AdmitWriter trims to // eight. An unchanged file published at g1 and carried by every later copy // stays restorable at g10. Every referenced object must lie under this // holder's location (its incarnation) in this region's data bucket, at // files//objects/; any other reference refuses the pin. func (p StorePinner) filesPin(ctx context.Context, h *hostclaim.Holder, releasedEmpty, needKey bool) (FilesPin, []byte, error) { chain := []string{h.Generation} for i := len(h.Admissions) - 1; i >= 0; i-- { chain = append(chain, h.Admissions[i].Generation, h.Admissions[i].Predecessor) } seen := map[string]bool{} var origins []string for _, gen := range chain { if gen == "" || gen == hostclaim.RootGeneration || seen[gen] { continue } seen[gen] = true origins = append(origins, h.Location+"/files/"+gen+"/") } for _, prefix := range origins { raw, _, err := p.Data.Get(ctx, hostfiles.ManifestKey(prefix)) if errors.Is(err, os.ErrNotExist) { continue } if err != nil { return FilesPin{}, nil, err } var m hostfiles.Manifest // decoded from the bytes the digest names if err := json.Unmarshal(raw, &m); err != nil || m.V != hostfiles.ManifestVersion { return FilesPin{}, nil, errors.Join(fmt.Errorf("%w: files manifest %s", ErrRefused, prefix), err) } if needKey && !liveEntry(m, InstanceKeyPath) { // The manifest was copied before the app started and the // key it minted afterwards never published (§5.6's // continuous-sync window, L21): installing this set would mint // a new key over data sealed with the old one. return FilesPin{}, nil, fmt.Errorf("%w: %s's pinned manifest has no %s for a nonempty database", ErrFilesUnevidenced, prefix, InstanceKeyPath) } if origins, err = referencedOrigins(h.Location, origins, m); err != nil { return FilesPin{}, nil, err } sum := sha256.Sum256(raw) return FilesPin{Prefix: prefix, Seq: m.Seq, SHA256: hex.EncodeToString(sum[:]), Origins: origins}, raw, nil } // No manifest anywhere in the chain. An empty files pin needs the // same kind of evidence an empty database pin does (ruling 15): no // writer was ever admitted on this holder (session 0), so no app ever // ran to mint instance.key, report.token or an env (H6); or the // stopped writer's release flush, which publishes every file whatever // the setting, reported nothing (manifest_seq 0). Any other admitted // writer without a manifest (an adopted host whose files never // published, a first manifest that never landed, a trimmed chain) // refuses: the worker marks the operation stuck for an operator (rev // 17, finding F2), never an install that mints a fresh key over data // sealed with the old one. if h.Session != 0 && !releasedEmpty { return FilesPin{}, nil, fmt.Errorf("%w: %s has admitted a writer but no files manifest exists in its chain", ErrFilesUnevidenced, h.Location) } gen := h.Generation if gen == "" { gen = hostclaim.RootGeneration } prefix := h.Location + "/files/" + gen + "/" return FilesPin{Prefix: prefix, Origins: []string{prefix}}, nil, nil } // referencedOrigins adds to origins the files prefix of every generation // whose objects m references, each of which must be an object this // incarnation's location published: /files//objects/. func referencedOrigins(location string, origins []string, m hostfiles.Manifest) ([]string, error) { base := location + "/files/" have := make(map[string]bool, len(origins)) for _, o := range origins { have[o] = true } var extra []string check := func(name string, e hostfiles.Entry) error { if e.Object == "" { return nil // a tombstone or a skipped entry names no object } gen, rest, ok := strings.Cut(strings.TrimPrefix(e.Object, base), "/") if !strings.HasPrefix(e.Object, base) || !ok || gen == "" || gen == "." || gen == ".." || !strings.HasPrefix(rest, "objects/") { return fmt.Errorf("%w: %s references %s, outside %s: %w", ErrRefused, name, e.Object, base, hostfiles.ErrOrigin) } if o := base + gen + "/"; !have[o] { have[o] = true extra = append(extra, o) } return nil } for name, e := range m.Entries { if err := check(name, e); err != nil { return nil, err } } for name, e := range m.Platform { if err := check(name, e); err != nil { return nil, err } } sort.Strings(extra) return append(origins, extra...), nil } // ReadPin reads and verifies the pin object a journaled RecoveryPin names: // the exact bytes hash to the journaled digest (ruling 5). Every reader of // a pin goes through it. func ReadPin(ctx context.Context, claims release.Store, ref hostclaim.RecoveryPin) (PinObject, error) { raw, _, err := claims.Get(ctx, ref.Key) if err != nil { return PinObject{}, err } if sum := sha256.Sum256(raw); hex.EncodeToString(sum[:]) != ref.SHA256 { return PinObject{}, fmt.Errorf("%w: pin object %s does not hash to its journaled digest", ErrRefused, ref.Key) } var obj PinObject if err := json.Unmarshal(raw, &obj); err != nil || obj.V != 1 { return PinObject{}, fmt.Errorf("%w: pin object %s", ErrRefused, ref.Key) } return obj, nil } // CopyPinnedManifest writes the pinned files manifest into toPrefix // (§5.8 "write P's files manifest, create-only, as the pinned manifest's // entries"), from the pinner's retained copy of its exact bytes (claims: // the control bucket), never from the source prefix again: a late // revision of the source manifest cannot move the pin (B2-4). The bytes // must hash to the pin's digest and sit at the pinned sequence; every // entry is validated against the pinned origins (L13); the write is // create-only, and a retry finding the same bytes at toPrefix succeeds. // A host that never published files gets an empty manifest. func CopyPinnedManifest(ctx context.Context, claims, data release.Store, pin FilesPin, toPrefix string) (hostfiles.Manifest, error) { if pin.Prefix == "" || toPrefix == "" || len(pin.Origins) == 0 || (pin.SHA256 == "") != (pin.Retained == "") { return hostfiles.Manifest{}, ErrUnsupported } var pinned hostfiles.Manifest if pin.Retained == "" { // A host that never published files: P starts an empty manifest. pinned = hostfiles.Manifest{V: hostfiles.ManifestVersion, Entries: map[string]hostfiles.Entry{}, Platform: map[string]hostfiles.Entry{}} } else { raw, _, err := claims.Get(ctx, pin.Retained) if err != nil { return hostfiles.Manifest{}, err } if sum := sha256.Sum256(raw); hex.EncodeToString(sum[:]) != pin.SHA256 { return hostfiles.Manifest{}, fmt.Errorf("%w: retained manifest %s is not the pinned bytes", ErrRefused, pin.Retained) } if err := json.Unmarshal(raw, &pinned); err != nil || pinned.Seq != pin.Seq { return hostfiles.Manifest{}, fmt.Errorf("%w: retained manifest %s", ErrRefused, pin.Retained) } if err := hostfiles.ValidateManifest(pinned, pin.Origins); err != nil { return hostfiles.Manifest{}, err } } // P's prepared manifest (claim rev 6.30 R18-1, moves rev 7.22 §7.6): // the pinned entries, by their exact content-addressed objects, at // sequence 0. The source's sequence is provenance (the pin keeps it), // never P's: the session publisher starts at 1 after the first // admission. The binding {location, P, op, install stamp} is the // prefix (location, P) and the claim's prepared_authority (op, stamp, // this manifest's digest). pinned.Seq = 0 body, err := json.Marshal(pinned) if err != nil { return hostfiles.Manifest{}, err } if err := release.CreateExact(ctx, data, hostfiles.ManifestKey(toPrefix), body); err != nil { if errors.Is(err, release.ErrCASConflict) { return hostfiles.Manifest{}, hostfiles.ErrManifestExists } return hostfiles.Manifest{}, err } return pinned, nil } ``` `internal/hostfiles/manifestcheck.go` (`CopyPinnedManifest` needs it; Task 7's `platform.go` sits beside it in the same package): ```go package hostfiles import "fmt" // ValidateManifest checks every entry and platform item of m against the // admitted origins, as Restore and CopyManifest do before they touch // anything (contracts note §1). A caller holding a manifest's exact bytes // from somewhere other than its prefix (a relocation's retained copy) // validates them with it before writing them under a new generation. func ValidateManifest(m Manifest, origins []string) error { if m.V != ManifestVersion { return fmt.Errorf("hostfiles: manifest is version %d; this binary reads %d", m.V, ManifestVersion) } if len(origins) == 0 { return ErrOrigin } for _, o := range origins { if err := checkPrefix(o); err != nil { return err } } return checkManifest(m, origins) } ``` - [ ] **Step 5: Run the tests** Run: `nice -n 19 go test ./internal/hostrelocate/ ./internal/hostfiles/ -count=1 -v 2>&1 | grep -E '^(---|ok|FAIL)'` Expected: PASS for all twenty-eight `hostrelocate` tests (worker and pin), and `hostfiles` unchanged. - [ ] **Step 6: Commit** ```bash git add internal/hostrelocate/worker.go internal/hostrelocate/pin.go internal/hostrelocate/worker_test.go internal/hostrelocate/pin_test.go internal/hostfiles/manifestcheck.go git commit -m "hostrelocate: a regional worker starts a relocation and replays it one claim CAS at a time, reading fences only Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 6: The participant's half, ordered (`internal/hostrelocate`) A participant box acknowledges first (D6, after §5.10's pending-state and update-begun check, inside `Effects.Acknowledge`); before every later effect it checks its execution controller is still open and active (`Effects.Ready`), and after an agent restart (`ErrNoController`) it reopens one and journals nothing, the first acknowledgement standing; then, as `op.exec`, it releases in `releasing` or `probe-failed`, builds P in `installing`, makes its local install and enters `probing` in `installed`, and probes in `probing`, counting each attempt in the claim first and entering `probe-failed` on the fourth (ruling 19). `Sweep`, once per pass, finishes every retained controller whose operation the claim no longer runs with this box as a participant (ruling 20). Coverage for the probe admission is `released` only when this operation journaled a release whose session is the predecessor writer and P has not been opened; `AdmitWriter` checks the rest. A box on `lost`, or not a participant, does nothing. The two end-to-end tests drive the worker and both participants against one claim store: a drain completes with `released` and no declared loss; a failover completes and declares the source's session. **Files:** - Create: `internal/hostrelocate/participant.go` - Test: `internal/hostrelocate/participant_test.go` **Interfaces:** - Consumes: Tasks 1 and 5 (the worker fixture in `worker_test.go`). - Produces: `hostrelocate.Effects` (`Acknowledge`, `Ready`, `Release`, `Build`, `InstallLocal`, `EnsureLocal` (rev 15), `VerifyReceipt` (rev 16), `Probe`, `Retained() ([]Participation, error)`, `Finish`), `hostrelocate.Participation{Host, Op}`, `hostrelocate.Participant{Claims, Box, Enrollment, Effects, Now}` with `Step(ctx, host) (hostclaim.Observed, error)` and `Sweep(ctx) error`. - [ ] **Step 1: Write the failing tests** `internal/hostrelocate/participant_test.go`: ```go package hostrelocate import ( "context" "errors" "fmt" "strconv" "strings" "testing" "time" "github.com/carlosframework/platform/internal/hostclaim" ) // fakeBox is one box's local effects. Its writes to the claim stand in for // what the real effects leave in storage (P built, a probe admitted). type fakeBox struct { f *fixture box string coverages []string // opened counts controllers opened; restarted makes Ready report none // until the next Acknowledge, as a restarted agent would. opened int restarted bool effects []string op string finished bool // failProbes makes every probe admission fail readiness. failProbes bool // retainedErr is a damaged journal entry Retained reports. retainedErr error // retired: a scoped cleanup removed the local install; restores, // reinstalls and probedMissing count what EnsureLocal and Probe saw. retired bool restores, reinstalls, probedMissing int // receiptLost: the box's durable local install receipt is gone. receiptLost bool } func (b *fakeBox) Acknowledge(_ context.Context, o hostclaim.Observed) (hostclaim.ExecutionAcknowledgement, error) { b.opened++ b.restarted, b.finished, b.op = false, false, o.Claim.Op.ID return hostclaim.ExecutionAcknowledgement{Op: o.Claim.Op.ID, Enrollment: enrollment(b.box), BootID: "boot", ControllerID: "ctl-" + b.box, LeaseETag: "lease", Expires: b.f.now.Add(15 * 60e9)}, nil } func (b *fakeBox) Retained() ([]Participation, error) { if b.opened == 0 || b.finished { return nil, b.retainedErr } return []Participation{{Host: host, Op: b.op}}, b.retainedErr } func (b *fakeBox) Finish(_ context.Context, h, op string) error { b.finished = true b.effects = append(b.effects, "finish "+op) return nil } func (b *fakeBox) Ready(context.Context, hostclaim.Observed) error { b.effects = append(b.effects, "ready") if b.restarted || b.opened == 0 { return ErrNoController } return nil } func (b *fakeBox) Release(_ context.Context, o hostclaim.Observed) (hostclaim.Release, error) { h := o.Claim.Holder return hostclaim.Release{ReleasedBy: b.box, Enrollment: enrollment(b.box), Generation: h.Generation, Session: h.Session, FinalTXID: 40, ManifestSeq: 7, Receipt: "r-" + b.box, ReceiptSHA256: strings.Repeat("d", 64)}, nil } func (b *fakeBox) Build(context.Context, hostclaim.Observed) error { b.effects = append(b.effects, "build") b.f.participant(func(c *hostclaim.Claim) error { c.Op.Journal["test.built"] = c.Op.Journal["prepared"]; return nil }) return nil } func (b *fakeBox) InstallLocal(_ context.Context, o hostclaim.Observed) (hostclaim.LocalInstall, error) { b.effects = append(b.effects, "install") n, _ := strconv.ParseUint(o.Claim.Op.Journal["install_stamp"], 10, 64) return hostclaim.LocalInstall{Box: b.box, Enrollment: enrollment(b.box), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("e", 64)}, nil } func (b *fakeBox) EnsureLocal(_ context.Context, o hostclaim.Observed) error { b.effects = append(b.effects, "ensure-local") if hostclaim.PreparedOpened(o.Claim) { b.reinstalls++ // never: an opened P is not reinstalled } if b.retired { b.retired, b.restores = false, b.restores+1 } return nil } func (b *fakeBox) VerifyReceipt(context.Context, hostclaim.Observed) (bool, error) { b.effects = append(b.effects, "verify-receipt") return !b.receiptLost, nil } func (b *fakeBox) Probe(ctx context.Context, o hostclaim.Observed, coverage string) (uint64, bool, error) { if b.retired { b.probedMissing++ // a probe admission over a missing local install } b.coverages = append(b.coverages, coverage) c := o.Claim next, err := hostclaim.Update(ctx, b.f.s, o, hostclaim.AdmitWriter(hostclaim.Admission{ObservedOpID: c.Op.ID, ObservedPhase: c.Op.Phase, ObservedSession: c.Holder.Session, ObservedGeneration: c.Holder.Generation, Box: b.box, Mode: "cold", NewGeneration: c.Op.Journal["prepared"], KeepStamp: true, Coverage: coverage, Region: "ie", PredecessorRegion: "ie"})) if err != nil { return 0, false, err } if b.failProbes { return next.Claim.Holder.Session, false, nil } return next.Claim.Holder.Session, true, nil } func (f *fixture) clock() time.Time { return f.now } // run steps the worker and both participants until the host serves again. func run(t *testing.T, f *fixture, parts ...Participant) hostclaim.Observed { t.Helper() ctx := context.Background() for i := 0; i < 60; i++ { if o := f.claim(); o.Claim.Op.Kind == hostclaim.KindServing { return o } if _, err := f.w.Step(ctx, host); err != nil && !errors.Is(err, ErrNotYet) { t.Fatalf("worker: %v", err) } for _, p := range parts { if _, err := p.Step(ctx, host); err != nil && !errors.Is(err, ErrNotYet) { t.Fatalf("%s: %v", p.Box, err) } } } t.Fatalf("no progress: %+v", f.claim().Claim.Op) return hostclaim.Observed{} } func TestADrainCompletesWithBothParticipants(t *testing.T) { f := newFixture(t) src, dst := &fakeBox{f: f, box: "src"}, &fakeBox{f: f, box: "dst"} if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } o := run(t, f, Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock}, Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock}) if o.Claim.Op.Exec != "dst" || len(dst.coverages) != 1 || dst.coverages[0] != "released" { t.Fatalf("exec %s coverages %v", o.Claim.Op.Exec, dst.coverages) } last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1] if last.WritesLost != nil { t.Fatalf("a drain declared a loss: %+v", last) } } func TestAFailoverCompletesAndDeclaresTheSource(t *testing.T) { f := newFixture(t) f.terminate("src") dst := &fakeBox{f: f, box: "dst"} if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } o := run(t, f, Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock}) last := o.Claim.Holder.Admissions[len(o.Claim.Holder.Admissions)-1] if dst.coverages[0] != "" || last.WritesLost == nil || last.WritesLost.Box != "src" { t.Fatalf("coverages %v last %+v", dst.coverages, last) } } func TestALostParticipantDoesNothing(t *testing.T) { f := newFixture(t) f.terminate("src") if _, err := f.w.Start(context.Background(), host, Request{Reason: hostclaim.ReasonFailover, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } src := &fakeBox{f: f, box: "src"} before := f.claim().ETag if _, err := (Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock}).Step(context.Background(), host); !errors.Is(err, ErrNotYet) || f.claim().ETag != before { t.Fatalf("a fenced source acted: %v", err) } } func TestAParticipantChecksItsControllerBeforeEveryEffectAndReopensAfterARestart(t *testing.T) { f := newFixture(t) ctx := context.Background() src, dst := &fakeBox{f: f, box: "src"}, &fakeBox{f: f, box: "dst"} if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } ps := Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock} pd := Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock} for _, p := range []Participant{ps, pd} { if _, err := p.Step(ctx, host); err != nil { t.Fatal(err) } } f.worker(nil) // releasing // The target's agent restarts: its next step opens a controller and // journals nothing; the one after acts. dst.restarted = true before := f.claim().ETag if _, err := pd.Step(ctx, host); !errors.Is(err, ErrNotYet) || dst.opened != 2 || f.claim().ETag != before { t.Fatalf("restart: %v opened=%d wrote=%v", err, dst.opened, f.claim().ETag != before) } if n := len(f.claim().Claim.Op.Journal["exec_ack.dst"]); n == 0 { t.Fatal("the first acknowledgement was dropped") } // The source releases: Ready ran before the effect. src.effects = nil if _, err := ps.Step(ctx, host); err != nil { t.Fatal(err) } if len(src.effects) == 0 || src.effects[0] != "ready" { t.Fatalf("release ran without checking the controller: %v", src.effects) } // A lost controller refuses the effect and writes nothing. o := run(t, f, ps, pd) if o.Claim.Op.Exec != "dst" { t.Fatalf("exec %s", o.Claim.Op.Exec) } for i, e := range dst.effects { if e == "build" && (i == 0 || dst.effects[i-1] != "ready") { t.Fatalf("build without a readiness check first: %v", dst.effects) } } } func TestAParticipantGivesUpAfterThreeProbeAttempts(t *testing.T) { f := newFixture(t) ctx := context.Background() src, dst := &fakeBox{f: f, box: "src"}, &fakeBox{f: f, box: "dst", failProbes: true} if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } ps := Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock} pd := Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock} for i := 0; i < 40; i++ { o := f.claim() if o.Claim.Op.Phase == hostclaim.PhaseProbeFailed { if n := len(dst.coverages); n != hostclaim.MaxProbeAttempts { t.Fatalf("%d probe admissions, want %d", n, hostclaim.MaxProbeAttempts) } if journal := o.Claim.Op.Journal; journal["probe_attempts.dst"] != "3" { t.Fatalf("attempts journaled: %v", journal["probe_attempts.dst"]) } return } f.w.Step(ctx, host) ps.Step(ctx, host) pd.Step(ctx, host) } t.Fatalf("never probe-failed: %+v", f.claim().Claim.Op) } func TestAParticipantFinishesItsControllerOnceTheOperationIsOver(t *testing.T) { f := newFixture(t) ctx := context.Background() src, dst := &fakeBox{f: f, box: "src"}, &fakeBox{f: f, box: "dst"} if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } ps := Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock} pd := Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock} run(t, f, ps, pd) if err := pd.Sweep(ctx); err != nil || !dst.finished { t.Fatalf("the serving target kept its relocation controller: %v finished=%v", err, dst.finished) } // A damaged journal entry beside it is reported, and the readable // participation is still finished. src.retainedErr = errors.New("a damaged journal entry") if err := ps.Sweep(ctx); !errors.Is(err, src.retainedErr) || !src.finished { t.Fatalf("the former source kept its controller, or the damaged entry was hidden: %v finished=%v", err, src.finished) } src.retainedErr = nil // A second relocation opens a fresh controller. spa := &fakeBox{f: f, box: "spa"} if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "dst", Target: "spa"}); err != nil { t.Fatal(err) } before := dst.opened run(t, f, pd, Participant{Claims: f.s, Box: "spa", Enrollment: enrollment("spa"), Effects: spa, Now: f.clock}) if dst.opened != before+1 { t.Fatalf("the second relocation did not open its own controller: %d -> %d", before, dst.opened) } // Mid-operation, Sweep keeps the controller. if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "spa", Target: "sp2"}); err != nil { t.Fatal(err) } pspa := Participant{Claims: f.s, Box: "spa", Enrollment: enrollment("spa"), Effects: spa, Now: f.clock} spa.finished = false if _, err := pspa.Step(ctx, host); err != nil { t.Fatal(err) } if err := pspa.Sweep(ctx); err != nil || spa.finished { t.Fatalf("a running operation's controller was finished: %v", err) } } // Round 14 finding 2: the local install is journaled, the lease is lost, // and the scoped cleanup retires the unopened P's local install. Before // probing, and before P's first admission, the participant has the // install verified and restored; once P is opened it is never touched. func TestAParticipantRestoresARetiredLocalInstallBeforeTheFirstAdmission(t *testing.T) { f := newFixture(t) ctx := context.Background() src, dst := &fakeBox{f: f, box: "src"}, &fakeBox{f: f, box: "dst"} if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } ps := Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock} pd := Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock} for i := 0; i < 40 && f.claim().Claim.Op.Journal["local_install"] == ""; i++ { _, _ = f.w.Step(ctx, host) ps.Step(ctx, host) pd.Step(ctx, host) } if f.claim().Claim.Op.Journal["local_install"] == "" { t.Fatalf("never installed locally: %+v", f.claim().Claim.Op) } dst.retired = true // the lease was lost; the scoped cleanup retired it run(t, f, ps, pd) if dst.restores != 1 || dst.probedMissing != 0 || dst.reinstalls != 0 { t.Fatalf("restores %d, probes over a missing install %d, reinstalls after P %d", dst.restores, dst.probedMissing, dst.reinstalls) } } // Round 15 finding 1: after P's first admission the local install is // never remade, and a box that has lost its receipt cannot rebuild it from // P: the next admission is refused and the operation is stuck (§5.8, // B2-5), across a restart. With the receipt intact, the later attempt goes // ahead and nothing is reinstalled or reverted: the files, env and timers // the admitted writers changed are theirs. func TestAParticipantIsStuckWhenItsReceiptIsLostAfterPOpens(t *testing.T) { for _, lost := range []bool{true, false} { t.Run(fmt.Sprintf("receipt lost %v", lost), func(t *testing.T) { f := newFixture(t) ctx := context.Background() src, dst := &fakeBox{f: f, box: "src"}, &fakeBox{f: f, box: "dst", failProbes: true} if _, err := f.w.Start(ctx, host, Request{Reason: hostclaim.ReasonDrain, Source: "src", Target: "dst"}); err != nil { t.Fatal(err) } ps := Participant{Claims: f.s, Box: "src", Enrollment: enrollment("src"), Effects: src, Now: f.clock} pd := Participant{Claims: f.s, Box: "dst", Enrollment: enrollment("dst"), Effects: dst, Now: f.clock} for i := 0; i < 60 && len(dst.coverages) == 0; i++ { _, _ = f.w.Step(ctx, host) ps.Step(ctx, host) pd.Step(ctx, host) } if len(dst.coverages) != 1 || !hostclaim.PreparedOpened(f.claim().Claim) { t.Fatalf("P was not opened: %d admissions", len(dst.coverages)) } dst.receiptLost, dst.restarted = lost, true // and the agent restarts dst.effects = nil for i := 0; i < 6; i++ { pd.Step(ctx, host) } c := f.claim().Claim for _, e := range dst.effects { if e == "ensure-local" || e == "install" { t.Fatalf("the local install was touched after P opened: %v", dst.effects) } } if lost { if c.Op.Journal["stuck"] == "" || len(dst.coverages) != 1 { t.Fatalf("a lost receipt after P opened: stuck %q, admissions %d", c.Op.Journal["stuck"], len(dst.coverages)) } return } if c.Op.Journal["stuck"] != "" || len(dst.coverages) < 2 { t.Fatalf("an intact receipt: stuck %q, admissions %d", c.Op.Journal["stuck"], len(dst.coverages)) } }) } } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostrelocate/ -run 'Completes|LostParticipant|ChecksItsController|ThreeProbeAttempts|FinishesItsController' -count=1` Expected: FAIL to compile: `undefined: Participant`. - [ ] **Step 3: The participant** `internal/hostrelocate/participant.go`: ```go package hostrelocate import ( "context" "encoding/json" "errors" "slices" "time" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/release" ) // Effects are a participant box's local effects, each under the host // guard and bound to the claim observation it is given. Plan 6b's box // assembly implements them (ClaimAssembly); this package only orders them // and records each result in the claim. type Effects interface { // Acknowledge writes this box's pending controller state, refuses while // root has an update begun (§5.10), takes the execution lease // (hostexec.Open under the agent's lifetime, then Ready) and returns // the acknowledgement to journal. After an agent restart it opens a // fresh controller for the same participation; the journaled // acknowledgement (the first lease, D6) is left as it is. Acknowledge(context.Context, hostclaim.Observed) (hostclaim.ExecutionAcknowledgement, error) // Ready reports whether this box's execution controller for the // operation is open and active now: ErrNoController when none is // retained (the agent restarted), hostexec's error when it was lost. // Every effect after the acknowledgement runs only after Ready. Ready(context.Context, hostclaim.Observed) error // Release runs §5.8's release sequence for this box's current writer // session: stop every writer (stopevidence.WritersStopped), flush the // database to its final TXID, publish every pending file and wake-state // mutation, close the replicator and publisher, and complete full stop // evidence (ClaimStopCoordinator.FenceAndStop). It returns the journal // entry; it is idempotent for one session. Release(context.Context, hostclaim.Observed) (hostclaim.Release, error) // Build builds the prepared generation P from the journaled pin with // moves' shared install routine (L16), under the replica lease. Build(context.Context, hostclaim.Observed) error // InstallLocal makes this box's local install from P's files manifest // and returns the receipt (§5.8 "The local install"). InstallLocal(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error) // EnsureLocal is called before P's first admission, each time: it // verifies that the local install the journaled receipt names is // complete on this box and, when it is not (a lost lease's scoped // cleanup retires an unopened P's local install), restores it under // install_stamp from P's files manifest. It is never called once P is // opened: an opened P is never reinstalled (rev 15, round 14). EnsureLocal(context.Context, hostclaim.Observed) error // VerifyReceipt is called before every admission after P is opened: // it reports whether this box's durable local install receipt is // still the one the claim journaled (box, enrollment, install_stamp, // journal digest). It checks the receipt and its binding only, never // the files, env or timers, which the admitted writers have changed // legitimately since. A missing or different receipt is false: the // local state cannot be rebuilt from P, so the operation is stuck // (§5.8, B2-5; rev 16, round 15). VerifyReceipt(context.Context, hostclaim.Observed) (bool, error) // Probe makes one cold probe admission (AdmitWriter, with coverage the // caller names) and runs the probe. passed is false while it runs. Probe(ctx context.Context, obs hostclaim.Observed, coverage string) (session uint64, passed bool, err error) // Retained lists the participations whose execution controllers this // box still holds; Finish closes one and releases its lifetime lock. // A controller is finished once the claim no longer runs that // operation with this box as a participant (Sweep), so the next // relocation of the host can open its own (hostexec keeps the lock // until Close). Retained also lists every participation journaled on // disk whose durable record a restart still has to clear; an entry it // cannot read is the error, returned beside every one it can. Retained() ([]Participation, error) Finish(ctx context.Context, host, op string) error } // Participation names one retained execution controller. type Participation struct { Host, Op string } // Participant is one box's half of relocation. Step makes at most one // participant transition for one host, chosen from the claim alone. type Participant struct { Claims release.Store Box string Enrollment string Effects Effects Now func() time.Time } // Step acts for host if this box has work in its relocation, and reports // ErrNotYet when it has none. func (p Participant) Step(ctx context.Context, host string) (hostclaim.Observed, error) { if p.Claims == nil || p.Box == "" || p.Enrollment == "" || p.Effects == nil || p.Now == nil { return hostclaim.Observed{}, ErrUnsupported } obs, err := hostclaim.Read(ctx, p.Claims, host) if err != nil || obs.Absent { return obs, err } c := obs.Claim if c.Op.Kind != hostclaim.KindRelocating || c.Op.Journal["stuck"] != "" || !slices.Contains(c.Op.Participants, p.Box) || slices.Contains(c.Lost, p.Box) { return obs, ErrNotYet } id := c.Op.ID if c.Op.Journal["exec_ack."+p.Box] == "" { ack, err := p.Effects.Acknowledge(ctx, obs) if err != nil { return obs, err } return p.update(ctx, obs, hostclaim.AcknowledgeExecution(p.Box, ack)) } // The controller is checked before every effect, never inferred from // the historical acknowledgement: a restarted agent reopens one (the // claim keeps the first lease's record) and acts on the next pass. if err := p.Effects.Ready(ctx, obs); errors.Is(err, ErrNoController) { _, err := p.Effects.Acknowledge(ctx, obs) return obs, errors.Join(ErrNotYet, err) } else if err != nil { return obs, err } executor := c.Op.Exec == p.Box switch { case executor && (c.Op.Phase == hostclaim.PhaseReleasing || c.Op.Phase == hostclaim.PhaseProbeFailed) && c.Op.Journal["release"] == "": r, err := p.Effects.Release(ctx, obs) if err != nil { return obs, err } return p.update(ctx, obs, hostclaim.JournalRelease(id, r)) case executor && c.Op.Phase == hostclaim.PhaseInstalling: return obs, errors.Join(ErrNotYet, p.Effects.Build(ctx, obs)) case executor && c.Op.Phase == hostclaim.PhaseInstalled: if c.Op.Journal["local_install"] != "" { // The receipt is a past fact; the files, env and timers it // names are checked (and restored) before probing. if err := p.Effects.EnsureLocal(ctx, obs); err != nil { return obs, err } // The target enters probing itself (§5.8 phase table). return p.update(ctx, obs, hostclaim.ToProbing(id, p.Now())) } l, err := p.Effects.InstallLocal(ctx, obs) if err != nil { return obs, err } if l.Box != p.Box || l.Enrollment != p.Enrollment { return obs, ErrRefused } return p.update(ctx, obs, hostclaim.RecordLocalInstall(id, l)) case executor && c.Op.Phase == hostclaim.PhaseProbing && c.Op.Journal["probe"] == "": if !hostclaim.PreparedOpened(c) { // Before P's first admission the local install must be whole. if err := p.Effects.EnsureLocal(ctx, obs); err != nil { return obs, err } } else { // After it, the install is never remade; its receipt must // still be this box's, or the operation is stuck. ok, err := p.Effects.VerifyReceipt(ctx, obs) if err != nil { return obs, err } if !ok { return p.update(ctx, obs, hostclaim.MarkStuck(id, "the local install receipt is missing after P was admitted (§5.8, B2-5)")) } } // The attempt is counted in the claim before it is made (§5.8: // at most three); the fourth is probe-failed, by this CAS. counted, err := hostclaim.Update(ctx, p.Claims, obs, hostclaim.BeginProbeAttempt(id, p.Box)) if errors.Is(err, hostclaim.ErrProbeAttemptsExhausted) { return p.update(ctx, obs, hostclaim.ToProbeFailed(id, p.Now())) } if err != nil { return obs, err } obs, c = counted, counted.Claim session, passed, err := p.Effects.Probe(ctx, obs, coverageFor(c)) if err != nil || !passed { return obs, errors.Join(ErrNotYet, err) } // The probe's admission wrote the claim: record the result on a // fresh read, which RecordProbe binds to the admitted session. fresh, err := hostclaim.Read(ctx, p.Claims, host) if err != nil || fresh.Absent { return obs, errors.Join(ErrNotYet, err) } return p.update(ctx, fresh, hostclaim.RecordProbe(id, hostclaim.Probe{Box: p.Box, Session: session})) } return obs, ErrNotYet } // Sweep finishes every retained controller whose participation is over: // the claim no longer runs that operation, or this box is no longer a // participant of it. It runs once per pass, for hosts the pass itself // skips (a completed relocation is a serving claim). func (p Participant) Sweep(ctx context.Context) error { if p.Claims == nil || p.Effects == nil { return ErrUnsupported } retained, err := p.Effects.Retained() errs := []error{err} // reported, while every readable participation is still finished for _, r := range retained { obs, err := hostclaim.Read(ctx, p.Claims, r.Host) if err != nil { errs = append(errs, err) continue } if !obs.Absent && obs.Claim.Op.Kind == hostclaim.KindRelocating && obs.Claim.Op.ID == r.Op && slices.Contains(obs.Claim.Op.Participants, p.Box) && !slices.Contains(obs.Claim.Lost, p.Box) { continue } if err := p.Effects.Finish(ctx, r.Host, r.Op); err != nil { errs = append(errs, err) } } return errors.Join(errs...) } // coverageFor is the probe admission's coverage (§5.8, L9): `released` // only for the admission that opens P, when this operation journaled a // release whose box is the predecessor writer; "" otherwise (a failover, // a later attempt, or a host that never had a writer). AdmitWriter checks // the rest. func coverageFor(c *hostclaim.Claim) string { var r hostclaim.Release if json.Unmarshal([]byte(c.Op.Journal["release"]), &r) != nil || r.ReleasedBy == "" { return "" } if hostclaim.PreparedOpened(c) { return "" } pred := hostclaim.AdmissionRecord{Session: c.Holder.Session, Box: c.Holder.WriterBox, Generation: c.Holder.Generation} if n := len(c.Holder.Admissions); n > 0 { pred = c.Holder.Admissions[n-1] } if pred.Box != r.ReleasedBy || pred.Session != r.Session || pred.Generation != r.Generation { return "" } return "released" } func (p Participant) update(ctx context.Context, obs hostclaim.Observed, tr func(*hostclaim.Claim) error) (hostclaim.Observed, error) { next, err := hostclaim.Update(ctx, p.Claims, obs, tr) if errors.Is(err, hostclaim.ErrAlreadyRecorded) || errors.Is(err, hostclaim.ErrAckAlreadyApplied) { return obs, nil } return next, err } ``` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/hostrelocate/ -count=1` Expected: PASS. - [ ] **Step 5: Commit** ```bash git add internal/hostrelocate/participant.go internal/hostrelocate/participant_test.go git commit -m "hostrelocate: a participant acknowledges, releases, installs and probes in the order its phase allows Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 7: Release evidence on the box (`internal/host`) `StopReceiptReleases` is `hostclaim.ReleaseAuthority` over main's stop receipts: the receipt the journal names hashes to the journaled digest, is canonical, sits at its own key, and is the full stop (`kind: stopped`) of exactly that host, incarnation, operation, phase, box, enrollment and writer session, with obligation `relocation-release` (ruling 2). `RelocationEffects` is `hostrelocate.Effects` over providers: `Release` asks root for stop evidence and runs steps 1 to 3 inside the drain root calls back (stop the writers, observe them stopped, database flush, files flush), records the flushed positions durably against the exact stop, and only then lets the producers drain; a failed step stops everything after it; a receipt root replays is paired with the positions this exact stop recorded, or refused, never guessed at. The observation (`Observe`) is a provider because the complete descriptor scan is root's (ruling 22). After a restart, `Release` asks root under the identity the flush was recorded with, offers no drain of its own, and pairs root's durable receipt with the record. *Rev 8:* a retry by the same controller, after root failed between the drain and its durable receipt, resumes at the producers' idempotent drain with the recorded positions and never flushes again. *Rev 9:* whenever positions are recorded for the stop, the request keeps the recorded directory revision, even if the roster has moved on. `ReleaseFlushFiles` is step 3 in full: the files directory, the wake state and the env file in one manifest revision. Controllers are opened under the agent's lifetime and retained (`RelocationControllers`); `Ready` checks one before every effect. *Rev 6:* each participation is journaled before its controller opens, and a durable record is cleared only straight after the scoped cleanup succeeds in the same `Finish` (round 5, R4-5); the journal listing ignores uncommitted `.tmp` files and reports a damaged committed entry beside the readable ones (finding 4); a removed env file is flushed as a deletion (finding 5). *Rev 13:* a successor's controller opens only after this box's earlier participations of the host are retired (`handover`), and its wait for the host's lifetime lock is bounded by the pass (`openBounded`). **Files:** - Create: `internal/host/relocationreleases.go`, `internal/host/relocationeffects.go`, `internal/host/relocationflush.go`, `internal/hostfiles/platform.go` - Test: `internal/host/relocationreleases_test.go`, `internal/host/relocationeffects_test.go`, `internal/host/relocationcontrollers_linux_test.go`, `internal/host/relocationlifecycle_linux_test.go` (rev 14) - Create: `internal/hostexec/readlocal.go` (rev 15: `hostexec.ReadLocal`) **Interfaces:** - Consumes: Tasks 1, 4 and 6; main's `StopReceipt`, `StopRequest`, `StopResult`, `StopDrainer`, `DrainAcknowledgement`, `mustMarshalStopReceipt`, `ErrClaimStopUnavailable`, `ErrClaimStopReceipt`, `hostbox.Actor`, `hostexec.Binding`, `hostexec.State`, `hostfiles.Open`, `ExportWake`, `StageWake`, `StagePlatform`, `SyncDir`, `Commit`, `ReadPlatform`, `registry.Registry`, `hostexec.ClearStale`. - Produces: `host.RelocationReleaseObligation`, `host.StopReceiptReleases{Evidence}`, `host.RelocationIdentity{Actor, BootID, ControllerID}`, `host.RelocationController` (hostexec's `Ready` and `Close`), `host.RelocationControllers` (`Close`), `host.ReleasePositions` (`Record`, `Recorded` returning the recorded request), `host.ReleasePositionsFile{Dir}`, `host.RecoveryReleaseDrain` (the drain a restarted agent offers; Task 8a's `Collect` marks its request a recovery), `host.ReleaseFlushFiles{Data, Registry, FilesDir, EnvPath, Skip}.Flush`, `*host.RelocationEffects{Lifetime, Controllers, Identity, Controller, Producers, Collect, StopWriters, Writers, Observe, FlushDB, FlushFiles, Positions, BuildP, Install, Admit}` with `Retained` and `Finish`; `host.RelocationController` (`Ready`, `Close`, `Clear`), `host.OpenRelocationController(ctx, execctl.Config)` (hostexec's controller with `ClearStale` as its `Clear`), `RelocationControllers{Journal, Cleanup, Clear}` with the participation journal (written before each open) and `Retained() ([]hostrelocate.Participation, error)`; `hostfiles.EnvItem`, `(*hostfiles.Publisher).StagePlatformDelete(name)`. `StopRequest.Region` is 6b's field: Task 8a's `Collect` sets it on every request. Note: `internal/host`'s tests declare a function named `hostexec`, so this file imports the package as `execctl`. - [ ] **Step 1: Write the failing tests** `internal/host/relocationreleases_test.go`: ```go package host import ( "context" "crypto/sha256" "encoding/hex" "errors" "strings" "testing" "time" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/release" ) func TestStopReceiptReleasesBindsTheReceiptToTheRelease(t *testing.T) { ctx := context.Background() inc := strings.Repeat("a", 64) c := hostclaim.Claim{Host: "shop.example", Holder: &hostclaim.Holder{Incarnation: inc}, Op: hostclaim.Op{ID: "rel-1", Kind: hostclaim.KindRelocating, Phase: hostclaim.PhaseReleasing}} receipt := StopReceipt{V: 1, ID: "r1", Host: "shop.example", Incarnation: inc, WriterSession: 12, OperationID: "rel-1", Phase: hostclaim.PhaseReleasing, ObligationID: RelocationReleaseObligation, Box: "src", Enrollment: "enr", Kind: "stopped", EvidenceAt: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC)} put := func(s release.Store, r StopReceipt) hostclaim.Release { body := mustMarshalStopReceipt(r) key := "control/hosts/shop.example/evidence/" + r.OperationID + "/" + r.Box + "/" + r.ID + ".json" if _, err := s.Put(ctx, key, body, ""); err != nil { t.Fatal(err) } sum := sha256.Sum256(body) return hostclaim.Release{ReleasedBy: "src", Enrollment: "enr", Session: 12, Receipt: key, ReceiptSHA256: hex.EncodeToString(sum[:])} } s := release.NewDirStore(t.TempDir()) a := StopReceiptReleases{Evidence: s} if err := a.VerifyRelease(ctx, c, put(s, receipt)); err != nil { t.Fatal(err) } for _, tc := range []struct { name string mutate func(*StopReceipt, *hostclaim.Release) }{ {"another session", func(r *StopReceipt, _ *hostclaim.Release) { r.WriterSession = 11 }}, {"another box", func(r *StopReceipt, _ *hostclaim.Release) { r.Box = "dst" }}, {"another operation", func(r *StopReceipt, _ *hostclaim.Release) { r.OperationID = "rel-0" }}, {"a retirement's stop", func(r *StopReceipt, _ *hostclaim.Release) { r.ObligationID = "park-1" }}, {"another phase", func(r *StopReceipt, _ *hostclaim.Release) { r.Phase = hostclaim.PhaseProbeFailed }}, {"another incarnation", func(r *StopReceipt, _ *hostclaim.Release) { r.Incarnation = strings.Repeat("b", 64) }}, {"a digest that is not the bytes", func(_ *StopReceipt, rel *hostclaim.Release) { rel.ReceiptSHA256 = strings.Repeat("0", 64) }}, } { t.Run(tc.name, func(t *testing.T) { s := release.NewDirStore(t.TempDir()) r := receipt var mutated hostclaim.Release tc.mutate(&r, &mutated) rel := put(s, r) if mutated.ReceiptSHA256 != "" { rel.ReceiptSHA256 = mutated.ReceiptSHA256 } if err := (StopReceiptReleases{Evidence: s}).VerifyRelease(ctx, c, rel); !errors.Is(err, ErrClaimStopReceipt) { t.Fatalf("accepted: %v", err) } }) } } ``` `internal/host/relocationeffects_test.go`: ```go package host import ( "context" "encoding/json" "errors" "os" "path/filepath" "slices" "strings" "testing" "time" "github.com/carlosframework/platform/internal/hostbox" "github.com/carlosframework/platform/internal/hostclaim" execctl "github.com/carlosframework/platform/internal/hostexec" "github.com/carlosframework/platform/internal/hostfiles" "github.com/carlosframework/platform/internal/hostrelocate" "github.com/carlosframework/platform/internal/registry" "github.com/carlosframework/platform/internal/release" "github.com/carlosframework/platform/internal/stopevidence" ) type emptyGroup struct{} func (emptyGroup) Populated() (bool, error) { return false, nil } func (emptyGroup) Members() ([]int, error) { return nil, nil } type recordingDrain struct{ calls *[]string } func (d recordingDrain) DrainClaim(context.Context, StopRequest, string) (DrainAcknowledgement, error) { *d.calls = append(*d.calls, "producers") return DrainAcknowledgement{FenceID: "fence"}, nil } // readyController is one opened controller: it reports the context it was // opened under, so a test can see it was the agent's, not a pass's. type readyController struct { st execctl.State opened context.Context closed bool cleared bool lost error clearErr error closeErr error } func (c *readyController) Clear(context.Context) error { if c.clearErr != nil { return c.clearErr } c.cleared = true return nil } func (c *readyController) Lost() <-chan struct{} { ch := make(chan struct{}) if c.lost != nil { close(ch) } return ch } func (c *readyController) Ready(context.Context) (execctl.State, error) { if c.lost != nil { return execctl.State{}, c.lost } return c.st, nil } func (c *readyController) Close(ctx context.Context) error { if err := ctx.Err(); err != nil { return err // hostexec retains the lifetime lock on a cut-short join } if c.closed { return nil // hostexec: every Close after the first returns nil } c.closed = true return c.closeErr // hostexec: a cleanup failure is returned once } // testControllers is a RelocationControllers with a journal on disk and a // scoped cleanup that succeeds. func testControllers(t *testing.T) *RelocationControllers { return &RelocationControllers{Journal: filepath.Join(t.TempDir(), "participations"), Cleanup: func(context.Context, execctl.Binding) error { return nil }} } // kept is Retained's list, for tests whose journal holds nothing damaged. func kept(r interface { Retained() ([]hostrelocate.Participation, error) }) []hostrelocate.Participation { got, _ := r.Retained() return got } type memPositions struct { calls *[]string req StopRequest txid uint64 seq int64 ok bool } func (m *memPositions) Record(_ context.Context, r StopRequest, txid uint64, seq int64) error { *m.calls = append(*m.calls, "record") m.req, m.txid, m.seq, m.ok = r, txid, seq, true return nil } func (m *memPositions) Recorded(_ context.Context, r StopRequest) (StopRequest, uint64, int64, bool, error) { if !m.ok || !sameStop(m.req, r) { return StopRequest{}, 0, 0, false, nil } return m.req, m.txid, m.seq, true, nil } func effectsFixture(t *testing.T) (*RelocationEffects, hostclaim.Observed, *[]string, *StopRequest) { t.Helper() calls := &[]string{} seen := &StopRequest{} proc := t.TempDir() // no process holds anything exp := time.Date(2026, 10, 7, 10, 0, 0, 0, time.UTC) e := &RelocationEffects{ Lifetime: context.Background(), Controllers: testControllers(t), Identity: func(context.Context) (RelocationIdentity, error) { return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: "enr", Revision: 4}, BootID: "boot", ControllerID: "ctl"}, nil }, Controller: func(ctx context.Context, b execctl.Binding) (RelocationController, error) { return &readyController{opened: ctx, st: execctl.State{Binding: execctl.Binding{Enrollment: "enr", BootID: "boot", ControllerID: "c9"}, LeaseETag: "lease-1", Expires: exp}}, nil }, Producers: func(context.Context, hostclaim.Observed) (StopDrainer, error) { return recordingDrain{calls}, nil }, StopWriters: func(context.Context, hostclaim.Observed) error { *calls = append(*calls, "stop-writers") return nil }, Writers: func(context.Context, hostclaim.Observed) (stopevidence.WritersInputs, error) { *calls = append(*calls, "writers") return stopevidence.WritersInputs{Host: "shop.example", FenceClosed: true, SpawnsSettled: true, WorkDrained: true, Cgroup: emptyGroup{}, DBPaths: []string{"/data/shop.db"}, ReplicatorPID: 9, ReplicatorStart: "1", ProcRoot: proc}, nil }, FlushDB: func(context.Context, hostclaim.Observed) (uint64, error) { *calls = append(*calls, "db") return 41, nil }, FlushFiles: func(context.Context, hostclaim.Observed) (int64, error) { *calls = append(*calls, "files") return 8, nil }, Positions: &memPositions{calls: calls}, Observe: stopevidence.WritersStopped, Collect: func(ctx context.Context, r StopRequest, d StopDrainer) (StopResult, error) { *seen = r if _, err := d.DrainClaim(ctx, r, "fence"); err != nil { return StopResult{}, err } *calls = append(*calls, "receipt") return StopResult{Key: "control/hosts/shop.example/evidence/rel-1/src/r.json", Digest: strings.Repeat("a", 64)}, nil }, } obs := hostclaim.Observed{Host: "shop.example", ETag: "etag-7", Claim: &hostclaim.Claim{Host: "shop.example", Holder: &hostclaim.Holder{Incarnation: "inc", Stamp: 5, Session: 6, Generation: "g6", WriterBox: "src"}, Op: hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: hostclaim.PhaseReleasing, Exec: "src"}}} return e, obs, calls, seen } func TestRelocationReleaseRunsStepsOneToThreeInsideTheDrain(t *testing.T) { e, obs, calls, seen := effectsFixture(t) r, err := e.Release(context.Background(), obs) if err != nil { t.Fatal(err) } if want := []string{"stop-writers", "writers", "db", "files", "record", "producers", "receipt"}; !slices.Equal(*calls, want) { t.Fatalf("order %v, want %v", *calls, want) } if r.FinalTXID != 41 || r.ManifestSeq != 8 || r.Session != 6 || r.Generation != "g6" || r.ReleasedBy != "src" || r.Receipt == "" { t.Fatalf("release: %+v", r) } if seen.ObligationID != RelocationReleaseObligation || seen.Phase != hostclaim.PhaseReleasing || seen.ExpectedClaimETag != "etag-7" || seen.ExpectedDirectoryRevision != "4" { t.Fatalf("stop request: %+v", *seen) } } func TestRelocationReleaseStopsAtTheFirstFailedStep(t *testing.T) { failure := errors.New("sync failed") for _, tc := range []struct { name string change func(*RelocationEffects) want []string }{ {"the writers cannot be stopped", func(e *RelocationEffects) { e.StopWriters = func(context.Context, hostclaim.Observed) error { return failure } }, nil}, {"writers still running", func(e *RelocationEffects) { w := e.Writers e.Writers = func(ctx context.Context, o hostclaim.Observed) (stopevidence.WritersInputs, error) { in, err := w(ctx, o) in.WorkDrained = false return in, err } }, []string{"stop-writers", "writers"}}, {"the database flush", func(e *RelocationEffects) { e.FlushDB = func(context.Context, hostclaim.Observed) (uint64, error) { return 0, failure } }, []string{"stop-writers", "writers"}}, {"the files flush", func(e *RelocationEffects) { e.FlushFiles = func(context.Context, hostclaim.Observed) (int64, error) { return 0, failure } }, []string{"stop-writers", "writers", "db"}}, } { t.Run(tc.name, func(t *testing.T) { e, obs, calls, _ := effectsFixture(t) tc.change(e) if _, err := e.Release(context.Background(), obs); err == nil { t.Fatal("released") } if !slices.Equal(*calls, tc.want) { t.Fatalf("ran %v, want %v: nothing after a failed step", *calls, tc.want) } }) } } func TestRelocationReleaseReplaysOnlyRecordedPositions(t *testing.T) { replayed := func(context.Context, StopRequest, StopDrainer) (StopResult, error) { return StopResult{Key: "k", Digest: strings.Repeat("a", 64)}, nil // root replayed; no drain ran here } t.Run("no record", func(t *testing.T) { e, obs, _, _ := effectsFixture(t) e.Collect = replayed if _, err := e.Release(context.Background(), obs); !errors.Is(err, ErrClaimStopReceipt) { t.Fatalf("got %v", err) } }) t.Run("the lost response", func(t *testing.T) { // The first call flushed and recorded; root completed its receipt; // the response was lost. The retry pairs the replayed receipt with // the recorded positions. e, obs, _, _ := effectsFixture(t) collect := e.Collect e.Collect = func(ctx context.Context, r StopRequest, d StopDrainer) (StopResult, error) { _, err := collect(ctx, r, d) return StopResult{}, errors.Join(err, errors.New("connection reset")) } if _, err := e.Release(context.Background(), obs); err == nil { t.Fatal("the lost response released") } e.Collect = replayed r, err := e.Release(context.Background(), obs) if err != nil || r.FinalTXID != 41 || r.ManifestSeq != 8 { t.Fatalf("replay: %+v %v", r, err) } }) t.Run("root fails after the producers closed", func(t *testing.T) { // Round 7: the drain flushed, recorded and closed the producers; // root then failed before its receipt was durable (kernel // verification or CompleteFence). The retry, same controller, no // restart, resumes at the producers' drain with the recorded // positions: no step of the release runs again. e, obs, calls, seen := effectsFixture(t) collect := e.Collect e.Collect = func(ctx context.Context, r StopRequest, d StopDrainer) (StopResult, error) { if _, err := d.DrainClaim(ctx, r, "fence"); err != nil { return StopResult{}, err } return StopResult{}, errors.New("verify stopped: cgroup not yet empty") // no receipt persisted } if _, err := e.Release(context.Background(), obs); err == nil { t.Fatal("a release without a receipt succeeded") } *calls = nil e.Collect = collect // Round 8 finding 2: the roster moved on between the attempts // (same boot, same controller); the retry still names the // revision the stop was flushed under. e.Identity = func(context.Context) (RelocationIdentity, error) { return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: "enr", Revision: 5}, BootID: "boot", ControllerID: "ctl"}, nil } r, err := e.Release(context.Background(), obs) if err != nil || r.FinalTXID != 41 || r.ManifestSeq != 8 { t.Fatalf("retry: %+v %v", r, err) } if want := []string{"producers", "receipt"}; !slices.Equal(*calls, want) { t.Fatalf("the retry ran %v, want %v (no flush, no record, no writers stop again)", *calls, want) } if seen.ExpectedDirectoryRevision != "4" || seen.BootID != "boot" || seen.ControllerID != "ctl" { t.Fatalf("the retry asked root under another stop: %+v", *seen) } }) t.Run("after a restart the recorded identity asks root", func(t *testing.T) { // The agent flushed and recorded under controller "ctl", root // completed its receipt, the agent restarted. The new process asks // root under the recorded boot and controller (so root finds that // stop's durable receipt), offers no drain of its own, and pairs // the replayed receipt with the recorded positions. e, obs, _, _ := effectsFixture(t) if _, err := e.Release(context.Background(), obs); err != nil { t.Fatal(err) } e.Identity = func(context.Context) (RelocationIdentity, error) { return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: "enr", Revision: 5}, BootID: "boot-2", ControllerID: "ctl-2"}, nil } var asked StopRequest recovery := false e.Collect = func(_ context.Context, r StopRequest, d StopDrainer) (StopResult, error) { asked = r _, recovery = d.(RecoveryReleaseDrain) // Task 8a sets req.Recover from this if _, err := d.DrainClaim(context.Background(), r, "fence"); !errors.Is(err, ErrClaimStopIdentity) { return StopResult{}, errors.New("a restarted agent offered a drain for another process's stop") } return StopResult{Key: "k", Digest: strings.Repeat("a", 64)}, nil } r, err := e.Release(context.Background(), obs) if err != nil || r.FinalTXID != 41 || r.ManifestSeq != 8 { t.Fatalf("after restart: %+v %v", r, err) } if asked.BootID != "boot" || asked.ControllerID != "ctl" || asked.ExpectedDirectoryRevision != "4" || !recovery { t.Fatalf("root was not asked as the recorded owner's recovery: %+v recovery=%v", asked, recovery) } }) t.Run("a record of another session", func(t *testing.T) { e, obs, _, _ := effectsFixture(t) if _, err := e.Release(context.Background(), obs); err != nil { t.Fatal(err) } e.Collect = replayed obs.Claim.Holder.Session = 7 if _, err := e.Release(context.Background(), obs); !errors.Is(err, ErrClaimStopReceipt) { t.Fatalf("another session's positions were used: %v", err) } }) } func TestReleasePositionsFileBindsTheExactStop(t *testing.T) { ctx := context.Background() f := ReleasePositionsFile{Dir: filepath.Join(t.TempDir(), "relocate")} r := StopRequest{Host: "shop.example", Incarnation: "inc", InstallationStamp: 5, WriterSession: 6, OperationID: "rel-1", Phase: "releasing", ObligationID: RelocationReleaseObligation, Box: "src", Enrollment: "enr", BootID: "boot", ControllerID: "ctl", ExpectedClaimETag: "e1", ExpectedDirectoryRevision: "4"} if _, _, _, ok, err := f.Recorded(ctx, r); ok || err != nil { t.Fatalf("before any record: %v %v", ok, err) } if err := f.Record(ctx, r, 41, 8); err != nil { t.Fatal(err) } moved := r moved.ExpectedClaimETag, moved.ExpectedDirectoryRevision = "e2", "5" // a target replacement moved the claim if rec, txid, seq, ok, err := f.Recorded(ctx, moved); !ok || err != nil || txid != 41 || seq != 8 || rec.ControllerID != "ctl" { t.Fatalf("recorded: %d %d %v %v", txid, seq, ok, err) } // A restarted controller of the same installation finds the record, // and gets the identity the flush was made under. restarted := r restarted.BootID, restarted.ControllerID = "boot-2", "ctl-2" if rec, _, _, ok, err := f.Recorded(ctx, restarted); !ok || err != nil || rec.BootID != "boot" || rec.ControllerID != "ctl" { t.Fatalf("after a restart: %+v %v %v", rec, ok, err) } for name, change := range map[string]func(*StopRequest){ "session": func(r *StopRequest) { r.WriterSession = 7 }, "stamp": func(r *StopRequest) { r.InstallationStamp = 6 }, "operation": func(r *StopRequest) { r.OperationID = "rel-2" }, "phase": func(r *StopRequest) { r.Phase = "probe-failed" }, "box": func(r *StopRequest) { r.Box = "dst" }, } { other := r change(&other) if _, _, _, ok, _ := f.Recorded(ctx, other); ok { t.Fatalf("%s: another stop's positions were returned", name) } } if err := os.WriteFile(filepath.Join(f.Dir, "shop.example.json"), []byte("{"), 0o600); err != nil { t.Fatal(err) } if _, _, _, _, err := f.Recorded(ctx, r); !errors.Is(err, ErrClaimStopReceipt) { t.Fatalf("a damaged record: %v", err) } } func TestRelocationAcknowledgeRetainsTheControllerUnderTheAgentLifetime(t *testing.T) { e, obs, _, _ := effectsFixture(t) var opened []*readyController exp := time.Date(2026, 10, 7, 10, 0, 0, 0, time.UTC) var bound execctl.Binding e.Controller = func(ctx context.Context, b execctl.Binding) (RelocationController, error) { bound = b c := &readyController{opened: ctx, st: execctl.State{Binding: execctl.Binding{Enrollment: "enr", BootID: "boot", ControllerID: "c9"}, LeaseETag: "lease-1", Expires: exp}} opened = append(opened, c) return c, nil } pass, cancel := context.WithCancel(context.Background()) ack, err := e.Acknowledge(pass, obs) if err != nil { t.Fatal(err) } cancel() // the pass ends; the controller must not if bound.Op != "rel-1" || bound.Host != "shop.example" || bound.Incarnation != "inc" || ack.LeaseETag != "lease-1" || ack.ControllerID != "c9" || !ack.Expires.Equal(exp) { t.Fatalf("binding %+v ack %+v", bound, ack) } if opened[0].opened.Err() != nil { t.Fatal("the controller was opened under the pass's context") } // A second pass reuses the retained controller; Ready sees it. if _, err := e.Acknowledge(context.Background(), obs); err != nil || len(opened) != 1 { t.Fatalf("second pass: %v opened=%d", err, len(opened)) } if err := e.Ready(context.Background(), obs); err != nil { t.Fatal(err) } // A lost lease is recovered without a restart (round 13 finding 1): // Ready joins the lost controller and repeats the scoped cleanup, // reports no controller, and the participant's Acknowledge opens a // fresh one; a cleanup that fails keeps the lost one for the next pass. opened[0].lost = execctl.ErrLost cleanupErr := errors.New("cleanup: a process would not stop") e.Controllers.Cleanup = func(context.Context, execctl.Binding) error { return cleanupErr } if err := e.Ready(context.Background(), obs); !errors.Is(err, cleanupErr) || !opened[0].closed { t.Fatalf("lost controller, cleanup failing: %v closed=%v", err, opened[0].closed) } e.Controllers.Cleanup = func(context.Context, execctl.Binding) error { return nil } if err := e.Ready(context.Background(), obs); !errors.Is(err, hostrelocate.ErrNoController) { t.Fatalf("lost controller: %v", err) } if _, err := e.Acknowledge(context.Background(), obs); err != nil || len(opened) != 2 { t.Fatalf("recovery: %v opened=%d", err, len(opened)) } if err := e.Ready(context.Background(), obs); err != nil { t.Fatalf("after recovery: %v", err) } // An agent restart has no retained controller: Ready says so, and // Acknowledge opens a fresh one. restarted := *e restarted.Controllers = testControllers(t) if err := restarted.Ready(context.Background(), obs); !errors.Is(err, hostrelocate.ErrNoController) { t.Fatalf("after a restart: %v", err) } if _, err := restarted.Acknowledge(context.Background(), obs); err != nil || len(opened) != 3 { t.Fatalf("reopen: %v opened=%d", err, len(opened)) } // Finishing a participation closes its controller and forgets it; a // later Acknowledge for the same host opens a new one. if got := kept(&restarted); len(got) != 1 || got[0] != (hostrelocate.Participation{Host: "shop.example", Op: "rel-1"}) { t.Fatalf("retained: %+v", got) } // A close cut short by the pass's context keeps the entry, so a later // sweep can finish it; then it is forgotten. cut, cancel := context.WithCancel(context.Background()) cancel() if err := restarted.Finish(cut, "shop.example", "rel-1"); err == nil || opened[2].closed || len(kept(&restarted)) != 1 { t.Fatalf("a cut-short close forgot the controller: %v closed=%v retained=%v", err, opened[2].closed, kept(&restarted)) } if err := restarted.Finish(context.Background(), "shop.example", "rel-1"); err != nil || !opened[2].closed || !opened[2].cleared || len(kept(&restarted)) != 0 { t.Fatalf("finish: %v closed=%v cleared=%v retained=%v", err, opened[2].closed, opened[2].cleared, kept(&restarted)) } if err := restarted.Ready(context.Background(), obs); !errors.Is(err, hostrelocate.ErrNoController) { t.Fatalf("after finish: %v", err) } obs.Claim.Op.ID = "rel-2" if _, err := restarted.Acknowledge(context.Background(), obs); err != nil || len(opened) != 4 { t.Fatalf("the next relocation: %v opened=%d", err, len(opened)) } if err := restarted.Controllers.Close(context.Background()); err != nil || !opened[3].closed { t.Fatalf("close: %v closed=%v", err, opened[3].closed) } } func TestReleaseFlushFilesCarriesTheLastWakeStateAndEnv(t *testing.T) { ctx := context.Background() data := release.NewDirStore(t.TempDir()) reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db")) if err != nil { t.Fatal(err) } t.Cleanup(func() { reg.Close() }) const host = "shop.example" if err := reg.Upsert(registry.Route{Host: host, Kind: "service", Addr: "10.0.0.1:3000", Status: "ready", Hibernate: true}); err != nil { t.Fatal(err) } dir, envDir := t.TempDir(), t.TempDir() if err := os.WriteFile(filepath.Join(dir, "a.txt"), []byte("hello"), 0o644); err != nil { t.Fatal(err) } if err := os.WriteFile(filepath.Join(envDir, host+".env"), []byte("KEY=v\n"), 0o600); err != nil { t.Fatal(err) } f := ReleaseFlushFiles{Data: data, Registry: reg, FilesDir: func(string) string { return dir }, EnvPath: func(h string) string { return filepath.Join(envDir, h+".env") }} obs := hostclaim.Observed{Host: host, Claim: &hostclaim.Claim{Host: host, Holder: &hostclaim.Holder{Location: "instances/" + host + "/inc", Generation: "g6"}}} first, err := f.Flush(ctx, obs) if err != nil || first == 0 { t.Fatalf("first flush: %d %v", first, err) } // A timer registered immediately before the release, and a new file. if err := reg.ReplaceTimers(host, []registry.Timer{{Host: host, Name: "late", Kind: "schedule", Spec: "0 3 * * *", Path: "/n", NextAt: 5000, CreatedAt: 10, DueAt: 5000, Registration: "reg-late"}}); err != nil { t.Fatal(err) } if err := reg.SetNextDue(host, 7777); err != nil { t.Fatal(err) } if err := os.WriteFile(filepath.Join(dir, "b.txt"), []byte("later"), 0o644); err != nil { t.Fatal(err) } seq, err := f.Flush(ctx, obs) if err != nil || seq <= first { t.Fatalf("second flush: %d %v", seq, err) } prefix := "instances/" + host + "/inc/files/g6/" m, ok, err := hostfiles.Load(ctx, data, prefix) if err != nil || !ok || m.Seq != seq { t.Fatalf("manifest: %+v %v %v", m, ok, err) } if _, ok := m.Entries["b.txt"]; !ok { t.Fatal("the file written before the release is not in the manifest") } ws, ok, err := hostfiles.LoadWake(ctx, data, prefix, m, nil) if err != nil || !ok || len(ws.Timers) != 1 || ws.Timers[0].Name != "late" || ws.NextDue != 7777 { t.Fatalf("wake state: %+v %v %v", ws, ok, err) } env, err := hostfiles.ReadPlatform(ctx, data, prefix, m, hostfiles.EnvItem, nil) if err != nil || string(env) != "KEY=v\n" { t.Fatalf("env: %q %v", env, err) } // Nothing changed: the same sequence, still the final state. again, err := f.Flush(ctx, obs) if err != nil || again != seq { t.Fatalf("repeat: %d %v", again, err) } // The env file is removed before a later release: the flush stages its // deletion, so the final state carries no env, not the old KEY=v. if err := os.Remove(filepath.Join(envDir, host+".env")); err != nil { t.Fatal(err) } gone, err := f.Flush(ctx, obs) if err != nil || gone <= seq { t.Fatalf("flush after the env file was removed: %d %v", gone, err) } m, ok, err = hostfiles.Load(ctx, data, prefix) if err != nil || !ok || m.Seq != gone { t.Fatalf("manifest after removal: %+v %v %v", m, ok, err) } if env, err := hostfiles.ReadPlatform(ctx, data, prefix, m, hostfiles.EnvItem, nil); !errors.Is(err, os.ErrNotExist) { t.Fatalf("a removed env file was still published: %q %v", env, err) } if again, err := f.Flush(ctx, obs); err != nil || again != gone { t.Fatalf("repeat after removal: %d %v", again, err) } var raw map[string]any body, _, _ := data.Get(ctx, hostfiles.ManifestKey(prefix)) _ = json.Unmarshal(body, &raw) if raw["v"] != float64(hostfiles.ManifestVersion) { t.Fatalf("manifest version: %v", raw["v"]) } } func TestFinishClearsTheDurableRecordAndRetriesAcrossARestart(t *testing.T) { journal := filepath.Join(t.TempDir(), "participations") e, obs, _, _ := effectsFixture(t) cleanups := 0 var cleanupErr error cleanup := func(_ context.Context, b execctl.Binding) error { if b.Op != "rel-1" || b.Box != "src" || b.Host != "shop.example" || b.Incarnation != "inc" { t.Fatalf("cleanup for another binding: %+v", b) } cleanups++ return cleanupErr } e.Controllers = &RelocationControllers{Journal: journal, Cleanup: cleanup} var ctl *readyController exp := time.Date(2026, 10, 7, 10, 0, 0, 0, time.UTC) e.Controller = func(ctx context.Context, b execctl.Binding) (RelocationController, error) { ctl = &readyController{opened: ctx, st: execctl.State{Binding: b, LeaseETag: "lease-1", Expires: exp}} return ctl, nil } // Without a journal nothing opens: a record no restart could find. unjournaled := *e unjournaled.Controllers = &RelocationControllers{Cleanup: cleanup} if _, err := unjournaled.Acknowledge(context.Background(), obs); !errors.Is(err, ErrClaimStopUnavailable) || ctl != nil { t.Fatalf("a controller opened without a journal: %v", err) } if _, err := e.Acknowledge(context.Background(), obs); err != nil { t.Fatal(err) } // The controller's cleanup fails. hostexec's Close reports it once. ctl.closeErr = errors.New("cleanup: a process would not stop") cleanupErr = ctl.closeErr if err := e.Finish(context.Background(), "shop.example", "rel-1"); err == nil || !ctl.closed || ctl.cleared || len(kept(e)) != 1 { t.Fatalf("a failed cleanup: %v closed=%v cleared=%v retained=%v", err, ctl.closed, ctl.cleared, kept(e)) } // The next sweep: Close now returns nil, which proves nothing. The // repeated cleanup still fails, so the record stays. if err := e.Finish(context.Background(), "shop.example", "rel-1"); err == nil || ctl.cleared || len(kept(e)) != 1 { t.Fatalf("a nil Close after a failed cleanup cleared the record: %v cleared=%v retained=%v", err, ctl.cleared, kept(e)) } // The clear itself fails: the entry and the journal stay. cleanupErr = nil ctl.clearErr = errors.New("claim read failed") if err := e.Finish(context.Background(), "shop.example", "rel-1"); err == nil || ctl.cleared || len(kept(e)) != 1 { t.Fatalf("a failed clear: %v cleared=%v retained=%v", err, ctl.cleared, kept(e)) } // The agent restarts: no controller object, the journal remains. The // cleanup is repeated first; while it fails nothing is cleared. cleared := []execctl.Binding{} clear := func(_ context.Context, b execctl.Binding) error { cleared = append(cleared, b); return nil } again := &RelocationControllers{Journal: journal, Cleanup: cleanup, Clear: clear} if got := kept(again); len(got) != 1 || got[0].Op != "rel-1" { t.Fatalf("after a restart the journaled participation is forgotten: %+v", got) } cleanupErr = errors.New("cleanup: still failing") if err := again.Finish(context.Background(), "shop.example", "rel-1"); err == nil || len(cleared) != 0 || len(kept(again)) != 1 { t.Fatalf("a restart cleared without a successful cleanup: %v cleared=%+v", err, cleared) } cleanupErr = nil before := cleanups if err := again.Finish(context.Background(), "shop.example", "rel-1"); err != nil || cleanups != before+1 || len(cleared) != 1 || cleared[0].Op != "rel-1" || cleared[0].Box != "src" { t.Fatalf("retry after restart: %v cleanups=%d cleared=%+v", err, cleanups-before, cleared) } if got := kept(again); len(got) != 0 { t.Fatalf("the journal entry outlived the clear: %+v", got) } } // A crash before the first Finish (or a failed Open after hostexec wrote // its record) leaves no controller and no finishing state, only the // participation journaled before the open: a restart finds it, and an // uncommitted, truncated journal write beside it hides nothing. func TestARestartFindsAParticipationThatNeverReachedFinish(t *testing.T) { journal := filepath.Join(t.TempDir(), "participations") e, obs, _, _ := effectsFixture(t) e.Controllers = &RelocationControllers{Journal: journal, Cleanup: func(context.Context, execctl.Binding) error { return nil }} if _, err := e.Acknowledge(context.Background(), obs); err != nil { t.Fatal(err) } // The crash: the controller object is gone. Another participation's // journal write was cut short, leaving a truncated temporary file. if err := os.WriteFile(filepath.Join(journal, "spa.example~rel-9.json.tmp"), []byte(`{"v":1,"ho`), 0o600); err != nil { t.Fatal(err) } cleaned, cleared := []execctl.Binding{}, []execctl.Binding{} again := &RelocationControllers{Journal: journal, Cleanup: func(_ context.Context, b execctl.Binding) error { cleaned = append(cleaned, b); return nil }, Clear: func(_ context.Context, b execctl.Binding) error { cleared = append(cleared, b); return nil }} got, err := again.Retained() if err != nil || len(got) != 1 || got[0] != (hostrelocate.Participation{Host: "shop.example", Op: "rel-1"}) { t.Fatalf("restart with a truncated temporary file: %+v %v", got, err) } // A committed entry that cannot be read is reported, never hiding the // readable one. if err := os.WriteFile(filepath.Join(journal, "bad.example~rel-3.json"), []byte("{"), 0o600); err != nil { t.Fatal(err) } got, err = again.Retained() if !errors.Is(err, ErrClaimStopUnavailable) || len(got) != 1 || got[0].Host != "shop.example" { t.Fatalf("a damaged committed entry: %+v %v", got, err) } if err := again.Finish(context.Background(), "shop.example", "rel-1"); err != nil || len(cleaned) != 1 || len(cleared) != 1 || cleared[0].Host != "shop.example" || cleared[0].Incarnation != "inc" { t.Fatalf("finish after the crash: %v cleaned=%+v cleared=%+v", err, cleaned, cleared) } if got, _ := again.Retained(); len(got) != 0 { t.Fatalf("the participation outlived its clear: %+v", got) } } // Round 14 finding 2: EnsureLocal verifies the receipt's install, restores // a retired one under the same stamp and verifies it again, refuses a // restore that is not the receipt's, and never touches an opened P. func TestEnsureLocalRestoresARetiredInstallOnlyBeforePOpens(t *testing.T) { ctx := context.Background() e, obs, _, _ := effectsFixture(t) receipt := hostclaim.LocalInstall{Box: "dst", Enrollment: "enr-d", InstallStamp: 9, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("e", 64)} raw, _ := json.Marshal(receipt) obs.Claim.Op.Phase = hostclaim.PhaseProbing obs.Claim.Op.Journal = map[string]string{"local_install": string(raw), "prepared": "P1", "install_stamp": "9"} present, installs := true, 0 e.VerifyLocal = func(context.Context, hostclaim.Observed, hostclaim.LocalInstall) (bool, error) { return present, nil } restored := receipt e.Install = func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error) { installs++ present = true return restored, nil } if err := e.EnsureLocal(ctx, obs); err != nil || installs != 0 { t.Fatalf("a complete install: %v installs=%d", err, installs) } present = false if err := e.EnsureLocal(ctx, obs); err != nil || installs != 1 || !present { t.Fatalf("a retired install: %v installs=%d", err, installs) } present, restored.InstallStamp = false, 10 if err := e.EnsureLocal(ctx, obs); !errors.Is(err, ErrClaimStopReceipt) { t.Fatalf("a restore under another stamp: %v", err) } present, installs = false, 0 obs.Claim.Op.Journal["prepared_opened"] = "P1" if err := e.EnsureLocal(ctx, obs); err != nil || installs != 0 { t.Fatalf("an opened P was reinstalled: %v installs=%d", err, installs) } } // Round 15 finding 1: after P opens, VerifyReceipt compares the box's // durable receipt with the journaled one and nothing else: a missing or // different receipt is false (the participant marks the operation stuck); // changed files never trigger a restore. func TestVerifyReceiptChecksTheReceiptOnly(t *testing.T) { ctx := context.Background() e, obs, _, _ := effectsFixture(t) receipt := hostclaim.LocalInstall{Box: "dst", Enrollment: "enr-d", InstallStamp: 9, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("e", 64)} raw, _ := json.Marshal(receipt) obs.Claim.Op.Journal = map[string]string{"local_install": string(raw), "prepared": "P1", "prepared_opened": "P1", "install_stamp": "9"} stored, present := receipt, true e.Receipt = func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, bool, error) { return stored, present, nil } touched := 0 e.VerifyLocal = func(context.Context, hostclaim.Observed, hostclaim.LocalInstall) (bool, error) { touched++ return false, nil } e.Install = func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error) { touched++ return receipt, nil } if ok, err := e.VerifyReceipt(ctx, obs); !ok || err != nil || touched != 0 { t.Fatalf("an intact receipt over changed files: %v %v touched=%d", ok, err, touched) } present = false if ok, err := e.VerifyReceipt(ctx, obs); ok || err != nil { t.Fatalf("a missing receipt: %v %v", ok, err) } present, stored.InstallStamp = true, 8 if ok, err := e.VerifyReceipt(ctx, obs); ok || err != nil { t.Fatalf("another installation's receipt: %v %v", ok, err) } } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/host/ -run 'StopReceiptReleases|RelocationRelease|RelocationAcknowledge|ReleasePositions|ReleaseFlushFiles|Finish|ARestartFinds' -count=1` Expected: FAIL to compile: `undefined: StopReceiptReleases`, `undefined: RelocationEffects`. - [ ] **Step 3: The receipt verifier** `internal/host/relocationreleases.go`: ```go package host import ( "bytes" "context" "crypto/sha256" "encoding/hex" "encoding/json" "fmt" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/release" ) // RelocationReleaseObligation is the stop request's obligation component // for a relocation's release (§5.8 release step 4). A release retires // nothing, so it names no park-local chore; root's FenceAndStop requires // the component, and the receipt carries it so a retirement stop can never // be offered as a release, nor a release as a retirement. const RelocationReleaseObligation = "relocation-release" // StopReceiptReleases is hostclaim.ReleaseAuthority over this package's // stop receipts (ClaimStopCoordinator.Publish): a relocation's journaled // release counts only when the immutable receipt it names hashes to the // journaled digest and is the full stop of exactly that box, enrollment, // writer session, host, incarnation, operation and phase (region HA spec 1 // §5.8 release step 4, claim §6.1). type StopReceiptReleases struct { Evidence release.Store } var _ hostclaim.ReleaseAuthority = StopReceiptReleases{} func (a StopReceiptReleases) VerifyRelease(ctx context.Context, c hostclaim.Claim, r hostclaim.Release) error { if a.Evidence == nil || c.Holder == nil { return ErrClaimStopUnavailable } raw, _, err := a.Evidence.Get(ctx, r.Receipt) if err != nil { return err } sum := sha256.Sum256(raw) var receipt StopReceipt if hex.EncodeToString(sum[:]) != r.ReceiptSHA256 || json.Unmarshal(raw, &receipt) != nil || !bytes.Equal(raw, mustMarshalStopReceipt(receipt)) { return ErrClaimStopReceipt } key := fmt.Sprintf("control/hosts/%s/evidence/%s/%s/%s.json", receipt.Host, receipt.OperationID, receipt.Box, receipt.ID) if receipt.V != 1 || receipt.Kind != "stopped" || r.Receipt != key || receipt.Host != c.Host || receipt.Incarnation != c.Holder.Incarnation || receipt.OperationID != c.Op.ID || receipt.Phase != c.Op.Phase || receipt.ObligationID != RelocationReleaseObligation || receipt.Box != r.ReleasedBy || receipt.Enrollment != r.Enrollment || receipt.WriterSession != r.Session || receipt.EvidenceAt.IsZero() { return ErrClaimStopReceipt } return nil } ``` - [ ] **Step 4: The release sequence** `internal/hostfiles/platform.go`: ```go package hostfiles import ( "fmt" "strings" ) // EnvItem is the host's env file (.env) among the manifest's // platform items. A release flush stages it beside the wake state so a // relocation's prepared manifest carries it (region HA spec 1 §5.6: // always continuous, whatever the setting); the local install restores it // with ReadPlatform. const EnvItem = "host.env" // StagePlatformDelete records that a platform item is gone: the next // Commit publishes a revision without it, at a new sequence, so a reader // of that revision gets os.ErrNotExist from ReadPlatform instead of the // last bytes it published. Deleting an item the manifest does not hold // is a no-op, like StageDelete. func (p *Publisher) StagePlatformDelete(name string) error { if name == "" || strings.ContainsAny(name, "/\x00") { return fmt.Errorf("hostfiles: platform item %q is not a plain name", name) } p.mu.Lock() defer p.mu.Unlock() if _, ok := p.man.Platform[name]; !ok { return nil } p.next++ delete(p.man.Platform, name) p.dirty = true return nil } ``` `internal/host/relocationflush.go`: ```go package host import ( "context" "encoding/json" "errors" "fmt" "os" "path/filepath" "github.com/carlosframework/platform/internal/controlname" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostfiles" "github.com/carlosframework/platform/internal/registry" "github.com/carlosframework/platform/internal/release" ) // ReleaseFlushFiles is release step 3 (§5.8, §5.6 "Release flush"): one // complete publish of the host's files directory, its wake state (timer // rows, NextDue, Replication) and its env file into the session's // generation, as one manifest revision, whatever the app's // files_durability. hostfiles.Publish leaves platform items out, so this // is the Open, StagePlatform, SyncDir, Commit sequence the contracts note // §2 names. It runs after StopWriters, so nothing changes under it. type ReleaseFlushFiles struct { Data release.Store Registry *registry.Registry FilesDir func(host string) string EnvPath func(host string) string // Skip is the files publisher's own exclusion (nil publishes every // regular file). Skip func(rel string, isDir bool) bool } // Flush publishes and returns the committed manifest sequence. The env // file is staged when it exists; when it does not, its deletion is staged // (StagePlatformDelete), so a host whose env file was removed after an // earlier flush publishes none rather than the old bytes. A // manifest revision is committed even when nothing changed (Commit // returns the current sequence), so the release's manifest_seq is always // the sequence that holds this box's final state. func (f ReleaseFlushFiles) Flush(ctx context.Context, obs hostclaim.Observed) (int64, error) { if f.Data == nil || f.Registry == nil || f.FilesDir == nil || f.EnvPath == nil || obs.Claim == nil || obs.Claim.Holder == nil { return 0, ErrClaimStopUnavailable } h := obs.Claim.Holder if h.Generation == "" || h.Location == "" || !controlname.ValidHostShape(obs.Host) { return 0, ErrClaimStopUnavailable } p, err := hostfiles.Open(ctx, f.Data, h.Location+"/files/"+h.Generation+"/") if err != nil { return 0, err } ws, err := hostfiles.ExportWake(f.Registry, obs.Host) if err != nil { return 0, err } if err := hostfiles.StageWake(ctx, p, ws); err != nil { return 0, err } env, err := os.ReadFile(f.EnvPath(obs.Host)) switch { case err == nil: if err := p.StagePlatform(ctx, hostfiles.EnvItem, env); err != nil { return 0, err } case errors.Is(err, os.ErrNotExist): // The env file is gone: the release's final state has none, so a // previously published one is deleted, never carried forward. if err := p.StagePlatformDelete(hostfiles.EnvItem); err != nil { return 0, err } default: return 0, err } st, err := hostfiles.SyncDir(ctx, p, f.FilesDir(obs.Host), f.Skip) if err != nil { return 0, err } if len(st.Unreadable) > 0 || len(st.Retry) > 0 || len(st.Torn) > 0 { // Writers are stopped (step 1), so an unstable file is a fault, // not a race: the release does not vouch for an image it could // not read whole. return 0, fmt.Errorf("hostfiles: release flush of %s: unreadable %q replaced %q torn %q", obs.Host, st.Unreadable, st.Retry, st.Torn) } return p.Commit(ctx) } // releasePosition is one recorded release: the exact stop request and // the positions its flush reached. type releasePosition struct { V int `json:"v"` Request StopRequest `json:"request"` TXID uint64 `json:"txid"` Seq int64 `json:"seq"` } // ReleasePositionsFile is ReleasePositions on disk: one file per host under // Dir, written whole and renamed into place, read back against the exact // stop request. The claim's ETag and the directory revision are not part // of the match: a target replacement moves both while the source's stop // stays the same stop. type ReleasePositionsFile struct { Dir string } func (f ReleasePositionsFile) path(host string) (string, error) { if f.Dir == "" || !controlname.ValidHostShape(host) { return "", ErrClaimStopUnavailable } return filepath.Join(f.Dir, host+".json"), nil } func (f ReleasePositionsFile) Record(_ context.Context, r StopRequest, txid uint64, seq int64) error { path, err := f.path(r.Host) if err != nil { return err } if err := os.MkdirAll(f.Dir, 0o700); err != nil { return err } raw, err := json.Marshal(releasePosition{V: 1, Request: r, TXID: txid, Seq: seq}) if err != nil { return err } tmp := path + ".tmp" w, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600) if err != nil { return err } if _, err := w.Write(raw); err != nil { _ = w.Close() return err } if err := w.Sync(); err != nil { _ = w.Close() return err } if err := w.Close(); err != nil { return err } if err := os.Rename(tmp, path); err != nil { return err } dir, err := os.Open(f.Dir) if err != nil { return err } defer dir.Close() return dir.Sync() } // sameStop is the installation and session, not the controller. func sameStop(a, b StopRequest) bool { return a.Host == b.Host && a.Incarnation == b.Incarnation && a.InstallationStamp == b.InstallationStamp && a.WriterSession == b.WriterSession && a.OperationID == b.OperationID && a.Phase == b.Phase && a.ObligationID == b.ObligationID && a.Box == b.Box && a.Enrollment == b.Enrollment } func (f ReleasePositionsFile) Recorded(_ context.Context, r StopRequest) (StopRequest, uint64, int64, bool, error) { path, err := f.path(r.Host) if err != nil { return StopRequest{}, 0, 0, false, err } raw, err := os.ReadFile(path) if errors.Is(err, os.ErrNotExist) { return StopRequest{}, 0, 0, false, nil } if err != nil { return StopRequest{}, 0, 0, false, err } var p releasePosition if err := json.Unmarshal(raw, &p); err != nil || p.V != 1 { return StopRequest{}, 0, 0, false, fmt.Errorf("%w: release positions for %s", ErrClaimStopReceipt, r.Host) } if !sameStop(p.Request, r) { return StopRequest{}, 0, 0, false, nil } return p.Request, p.TXID, p.Seq, true, nil } ``` `internal/host/relocationcontrollers_linux_test.go` (a real hostexec controller on disk; root's §5.10 check is `hostexecParticipation`): ```go //go:build linux package host import ( "context" "encoding/json" "errors" "os" "path/filepath" "strings" "testing" "time" "github.com/carlosframework/platform/internal/hostbox" "github.com/carlosframework/platform/internal/hostclaim" execctl "github.com/carlosframework/platform/internal/hostexec" "github.com/carlosframework/platform/internal/hostguard" "github.com/carlosframework/platform/internal/hostquarantine" "github.com/carlosframework/platform/internal/release" ) type activeDirectory struct{} func (activeDirectory) Lookup(context.Context, string) (hostquarantine.Actor, error) { return hostquarantine.Actor{Enrollment: strings.Repeat("a", 32), State: "active"}, nil } // A real hostexec controller on disk: after the relocation completes, // Finish closes it and clears its durable record, so root's §5.10 check // (hostexecParticipation) lets an executable update through. While the // operation still runs, the clear refuses and the record stays. A cleanup // that failed keeps the record however often Close is called afterwards // (hostexec returns the failure once), and across a restart, until a // repeated cleanup succeeds. func TestFinishClearsTheDurableRecordSoUpdatesAreEligible(t *testing.T) { ctx := context.Background() dataDir := t.TempDir() for _, name := range []string{"hostguard", ".hostexec", "locks", "exec-locks"} { if err := os.Mkdir(filepath.Join(dataDir, name), 0o770); err != nil { t.Fatal(err) } } store := release.NewDirStore(t.TempDir()) const host = "shop.bab.oncarlos.com" created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + host + ".json", Account: "bab", App: "jam", Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created} body := []byte(`{"host":"` + host + `","account":"bab","app":"jam","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) obs, err := hostclaim.Read(ctx, store, host) if err != nil { t.Fatal(err) } step := func(o hostclaim.Observed, tr func(*hostclaim.Claim) error) hostclaim.Observed { t.Helper() next, err := hostclaim.Update(ctx, store, o, tr) if err != nil { t.Fatal(err) } return next } obs = step(obs, hostclaim.BeginCreateOn(nh, body, "c1", "console", created, hostclaim.ExecutorSelection{Box: "src", Enrollment: strings.Repeat("a", 32)})) obs = step(obs, hostclaim.CompleteCreate("c1", "src")) obs = step(obs, hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) obs = step(obs, func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: hostclaim.PhaseStarting, Exec: "src", Participants: []string{"src", "dst"}, Intent: map[string]string{"region": "ie", "source_box": "src", "target_box": "dst"}} return nil }) failing := false scoped := func(context.Context) error { if failing { return errors.New("cleanup: the partial install would not go") } return nil } cfg := execctl.Config{Store: store, Guard: hostguard.Guard{DataDir: dataDir, LockDir: filepath.Join(dataDir, "locks"), Claims: hostguard.StoreReader{Store: store}}, Directory: activeDirectory{}, Clock: execctl.SystemClock{}, LockDir: filepath.Join(dataDir, "exec-locks"), CloseAdmission: func() {}, CancelWork: func() {}, Cleanup: scoped} journal := filepath.Join(dataDir, "relocate", "participations") controllers := func() *RelocationControllers { return &RelocationControllers{Journal: journal, DataDir: dataDir, Cleanup: func(ctx context.Context, _ execctl.Binding) error { return scoped(ctx) }, Clear: func(ctx context.Context, b execctl.Binding) error { c := cfg c.Binding = b return execctl.ClearStale(ctx, c) }} } e, _, _, _ := effectsFixture(t) e.Controllers = controllers() e.Identity = func(context.Context) (RelocationIdentity, error) { return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: strings.Repeat("a", 32), Revision: 4}, BootID: "boot", ControllerID: "ctl"}, nil } e.Controller = func(ctx context.Context, b execctl.Binding) (RelocationController, error) { c := cfg c.Binding = b return OpenRelocationController(ctx, c) } if _, err := e.Acknowledge(ctx, obs); err != nil { t.Fatal(err) } if err := hostexecParticipation(dataDir); err == nil { t.Fatal("a participant's record did not block an update") } // The controller's cleanup fails: Close reports it, the record stays. failing = true if err := e.Finish(ctx, host, "rel-1"); err == nil { t.Fatal("a failed cleanup was not reported") } if err := hostexecParticipation(dataDir); err == nil || len(kept(e)) != 1 { t.Fatalf("a failed cleanup cleared the record: %v retained=%v", err, kept(e)) } // Still relocating: the clear refuses, the record and the entry stay. failing = false if err := e.Finish(ctx, host, "rel-1"); !errors.Is(err, execctl.ErrInactive) { t.Fatalf("finish during the operation: %v", err) } if err := hostexecParticipation(dataDir); err == nil || len(kept(e)) != 1 { t.Fatalf("the record was cleared while the operation ran: %v retained=%v", err, kept(e)) } // Served on the target. The cleanup fails again: hostexec's Close now // returns nil, and the next sweep must still not clear. step(obs, func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "dst", Participants: []string{"dst"}} return nil }) failing = true if err := e.Finish(ctx, host, "rel-1"); err == nil { t.Fatal("a sweep after a failed cleanup reported success") } if err := hostexecParticipation(dataDir); err == nil || len(kept(e)) != 1 { t.Fatalf("a nil Close cleared the record: %v retained=%v", err, kept(e)) } // The agent restarts with the cleanup still failing: nothing clears. restarted := controllers() if err := restarted.Finish(ctx, host, "rel-1"); err == nil { t.Fatal("a restart cleared without a successful cleanup") } if err := hostexecParticipation(dataDir); err == nil || len(kept(restarted)) != 1 { t.Fatalf("a restart cleared the record: %v retained=%v", err, kept(restarted)) } // The cleanup succeeds: the retry clears the record and the journal. failing = false if err := restarted.Finish(ctx, host, "rel-1"); err != nil { t.Fatal(err) } if err := hostexecParticipation(dataDir); err != nil { t.Fatalf("after the relocation an update is still refused: %v", err) } if len(kept(restarted)) != 0 { t.Fatalf("retained: %v", kept(restarted)) } } // Round 12 finding 1: relocation A completes and B starts before this box // sweeps A. A's real controller still holds the host's lifetime lock; // Acknowledge for B retires it first (hostexec refuses to clear A's record // while B lists this box, so B's Open replaces it), under the pass's // bounded context, and B's controller opens. A lock held by a controller // this box does not retain is waited for no longer than the pass. func TestASuccessorRelocationRetiresItsPredecessorsController(t *testing.T) { ctx := context.Background() dataDir := t.TempDir() for _, name := range []string{"hostguard", ".hostexec", "locks", "exec-locks"} { if err := os.Mkdir(filepath.Join(dataDir, name), 0o770); err != nil { t.Fatal(err) } } store := release.NewDirStore(t.TempDir()) const host = "shop.bab.oncarlos.com" created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + host + ".json", Account: "bab", App: "jam", Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created} body := []byte(`{"host":"` + host + `","account":"bab","app":"jam","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) obs, err := hostclaim.Read(ctx, store, host) if err != nil { t.Fatal(err) } step := func(o hostclaim.Observed, tr func(*hostclaim.Claim) error) hostclaim.Observed { t.Helper() next, err := hostclaim.Update(ctx, store, o, tr) if err != nil { t.Fatal(err) } return next } relocating := func(id, from, to string) func(*hostclaim.Claim) error { return func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: id, Phase: hostclaim.PhaseStarting, Exec: from, Participants: []string{from, to}, Intent: map[string]string{"region": "ie", "source_box": from, "target_box": to}} return nil } } obs = step(obs, hostclaim.BeginCreateOn(nh, body, "c1", "console", created, hostclaim.ExecutorSelection{Box: "src", Enrollment: strings.Repeat("a", 32)})) obs = step(obs, hostclaim.CompleteCreate("c1", "src")) obs = step(obs, hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) obs = step(obs, relocating("rel-1", "src", "dst")) cfg := execctl.Config{Store: store, Guard: hostguard.Guard{DataDir: dataDir, LockDir: filepath.Join(dataDir, "locks"), Claims: hostguard.StoreReader{Store: store}}, Directory: activeDirectory{}, Clock: execctl.SystemClock{}, LockDir: filepath.Join(dataDir, "exec-locks"), CloseAdmission: func() {}, CancelWork: func() {}, Cleanup: func(context.Context) error { return nil }} lifetime, stop := context.WithCancel(ctx) t.Cleanup(stop) e, _, _, _ := effectsFixture(t) e.Lifetime = lifetime e.Controllers = &RelocationControllers{Journal: filepath.Join(dataDir, "relocate", "participations"), DataDir: dataDir, Cleanup: func(context.Context, execctl.Binding) error { return nil }, Clear: func(ctx context.Context, b execctl.Binding) error { c := cfg c.Binding = b return execctl.ClearStale(ctx, c) }} e.Identity = func(context.Context) (RelocationIdentity, error) { return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: strings.Repeat("a", 32), Revision: 4}, BootID: "boot", ControllerID: "ctl"}, nil } e.Controller = func(ctx context.Context, b execctl.Binding) (RelocationController, error) { c := cfg c.Binding = b return OpenRelocationController(ctx, c) } if _, err := e.Acknowledge(ctx, obs); err != nil { t.Fatal(err) } // A completes on dst, and B (dst back to src) starts before this box's // sweep has finished A. obs = step(obs, func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: "dst", Participants: []string{"dst"}} return nil }) obs = step(obs, relocating("rel-2", "dst", "src")) pass, cancel := context.WithTimeout(ctx, 10*time.Second) defer cancel() ack, err := e.Acknowledge(pass, obs) if err != nil { t.Fatalf("the successor's acknowledgement: %v", err) } if ack.Op != "rel-2" { t.Fatalf("ack: %+v", ack) } // A's journal entry stays until a sweep verifies the record is B's // (rev 14), then it goes; B's stays. if err := e.Finish(pass, host, "rel-1"); err != nil { t.Fatalf("sweeping A after the handover: %v", err) } got, err := e.Retained() if err != nil || len(got) != 1 || got[0].Op != "rel-2" { t.Fatalf("retained after the handover: %+v %v", got, err) } if err := hostexecParticipation(dataDir); err == nil { t.Fatal("B's record did not block an update") } // A lock held by a controller this box does not retain (here, another // operation's opened directly): the next open waits no longer than the // pass, and opens nothing. if err := e.Controllers.Close(ctx); err != nil { t.Fatal(err) } obs = step(obs, relocating("rel-3", "dst", "src")) foreign := cfg foreign.Binding = execctl.Binding{Box: "src", Enrollment: strings.Repeat("a", 32), Host: host, Incarnation: obs.Claim.Holder.Incarnation, Op: "rel-3", BootID: "boot"} held, err := execctl.Open(lifetime, foreign) if err != nil { t.Fatal(err) } defer held.Close(ctx) other := *e other.Controllers = &RelocationControllers{Journal: filepath.Join(dataDir, "relocate", "other"), Cleanup: func(context.Context, execctl.Binding) error { return nil }} short, cancelShort := context.WithTimeout(ctx, 300*time.Millisecond) defer cancelShort() began := time.Now() if _, err := other.Acknowledge(short, obs); err == nil { t.Fatal("opened while another controller held the host") } if waited := time.Since(began); waited > 5*time.Second { t.Fatalf("the open waited %v, past the pass's bound", waited) } if got, _ := other.Controllers.Retained(); len(got) != 1 || got[0].Op != "rel-3" { t.Fatalf("the bounded open retained a controller: %+v", got) // only the journal entry } other.Controllers.mu.Lock() n := len(other.Controllers.open) other.Controllers.mu.Unlock() if n != 0 { t.Fatal("a controller was kept from a cancelled open") } } // Round 14: a failed clear forgets a participation only on hostexec's own // validated reading of the record. A parseable record missing its // enrollment or incarnation is one hostexec refuses, so it is never // "another binding": the participation is kept, across a restart, until // the record is repaired and a cleanup and clear succeed. func TestARecordHostexecRefusesKeepsItsParticipation(t *testing.T) { ctx := context.Background() const host = "shop.bab.oncarlos.com" b := execctl.Binding{Box: "src", Enrollment: strings.Repeat("a", 32), Host: host, Incarnation: "inc", Op: "rel-1", BootID: "boot", ControllerID: "ctl"} valid := execctl.State{V: 1, Binding: b, Expires: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC), SafeUntil: 1, State: "lost"} for _, missing := range []string{"enrollment", "incarnation"} { t.Run(missing, func(t *testing.T) { dataDir := t.TempDir() if err := os.Mkdir(filepath.Join(dataDir, ".hostexec"), 0o770); err != nil { t.Fatal(err) } path := filepath.Join(dataDir, ".hostexec", host+".json") write := func(st execctl.State, drop string) { t.Helper() raw, _ := json.Marshal(st) var m map[string]any _ = json.Unmarshal(raw, &m) delete(m, drop) raw, _ = json.Marshal(m) if err := os.WriteFile(path, raw, 0o660); err != nil { t.Fatal(err) } } write(valid, missing) cleared, cleanups := 0, 0 clearErr := execctl.ErrState // ClearStale refuses the invalid record controllers := func() *RelocationControllers { return &RelocationControllers{Journal: filepath.Join(dataDir, "relocate", "participations"), DataDir: dataDir, Cleanup: func(context.Context, execctl.Binding) error { cleanups++; return nil }, Clear: func(context.Context, execctl.Binding) error { if clearErr != nil { return clearErr } cleared++ return os.Remove(path) }} } r := controllers() if err := r.record(host, "rel-1", b); err != nil { t.Fatal(err) } if err := r.Finish(ctx, host, "rel-1"); err == nil { t.Fatal("a record hostexec refuses was taken as another binding") } if got := kept(&RelocationEffects{Controllers: r}); len(got) != 1 { t.Fatalf("the participation was forgotten: %+v", got) } // A restart: the journal still names it, and still nothing clears. r = controllers() if err := r.Finish(ctx, host, "rel-1"); err == nil || len(kept(&RelocationEffects{Controllers: r})) != 1 { t.Fatalf("after a restart: %v", err) } // Repaired: the record is valid and this binding's; the cleanup // and the clear run, and only then is the participation gone. write(valid, "") clearErr = nil before := cleanups if err := r.Finish(ctx, host, "rel-1"); err != nil || cleared != 1 || cleanups != before+1 { t.Fatalf("after the repair: %v cleared=%d cleanups=%d", err, cleared, cleanups-before) } if got := kept(&RelocationEffects{Controllers: r}); len(got) != 0 { t.Fatalf("retained after the clear: %+v", got) } }) } } ``` `internal/host/relocationlifecycle_linux_test.go` (rev 14: the controller lifecycle as a property over real hostexec controllers; 200 seeds, `-short` 20): ```go //go:build linux package host import ( "context" "encoding/json" "errors" "fmt" "math/rand" "os" "path/filepath" "slices" "strconv" "strings" "sync" "testing" "time" "github.com/carlosframework/platform/internal/hostbox" "github.com/carlosframework/platform/internal/hostclaim" execctl "github.com/carlosframework/platform/internal/hostexec" "github.com/carlosframework/platform/internal/hostguard" "github.com/carlosframework/platform/internal/hostrelocate" "github.com/carlosframework/platform/internal/release" ) // lifeSeeds is the property test's seed count (rev 14: 200 by default). // Run it niced; it is not a sweep. const lifeSeeds = 200 // lifeStore is the bucket with faults: lease puts that fail (a renewal // that cannot reach the store) and claim reads that fail or find nothing. type lifeStore struct { release.Store mu sync.Mutex failLease bool claimErr error claimKey string } func (s *lifeStore) Put(ctx context.Context, key string, b []byte, etag string) (string, error) { s.mu.Lock() fail := s.failLease && strings.Contains(key, "/exec/") s.mu.Unlock() if fail { return "", errors.New("lease put: store unreachable") } return s.Store.Put(ctx, key, b, etag) } func (s *lifeStore) Get(ctx context.Context, key string) ([]byte, string, error) { s.mu.Lock() err := s.claimErr s.mu.Unlock() if err != nil && key == s.claimKey { return nil, "", err } return s.Store.Get(ctx, key) } func (s *lifeStore) DeleteIfMatch(ctx context.Context, key, etag string) error { return release.DeleteIfMatch(ctx, s.Store, key, etag) } // lifeClock is hostexec's clock under the test's hand: every After fires // on tick, so a renewal happens exactly when a step says so. type lifeClock struct { mu sync.Mutex wall time.Time boot int64 waiters []chan time.Time } func (c *lifeClock) Now() (time.Time, int64, error) { c.mu.Lock() defer c.mu.Unlock() return c.wall, c.boot, nil } func (c *lifeClock) After(time.Duration) <-chan time.Time { ch := make(chan time.Time, 1) c.mu.Lock() c.waiters = append(c.waiters, ch) c.mu.Unlock() return ch } func (c *lifeClock) tick() { c.mu.Lock() c.wall, c.boot = c.wall.Add(time.Second), c.boot+int64(time.Second) ws := c.waiters c.waiters = nil now := c.wall c.mu.Unlock() for _, w := range ws { w <- now } } // lifeWorld is one box ("src") taking part in a host's relocations over // real hostexec controllers, with the faults a seed injects. type lifeWorld struct { t *testing.T ctx context.Context inner release.Store store *lifeStore clock *lifeClock dataDir string host string cfg execctl.Config e *RelocationEffects p hostrelocate.Participant ops int mu sync.Mutex cleanErr bool // both cleanups fail block chan struct{} // non-nil: both cleanups wait for it (a close that times out) cleanOK map[string]bool // installed is this box's local install per operation (rev 15): the // Install provider makes it, a scoped cleanup of an operation whose P // is unopened retires it, VerifyLocal reads it. installed map[string]bool // gen is the agent process's generation: a crash starts the next one. // A dead generation's hooks never complete (rev 16). gen int // clears and opens count durable records cleared and controllers // opened in the current step, for invariant (h). clears, opens int // the record as the previous step left it. prevExists, prevReadable bool prevOp string saved []byte // a record the seed corrupted, to restore on recovery violated []string } const lifeDeadline = 400 * time.Millisecond func newLifeWorld(t *testing.T) *lifeWorld { t.Helper() ctx := context.Background() w := &lifeWorld{t: t, ctx: ctx, host: "shop.bab.oncarlos.com", clock: &lifeClock{wall: time.Date(2026, 10, 7, 9, 0, 0, 0, time.UTC), boot: int64(time.Hour)}, cleanOK: map[string]bool{}, installed: map[string]bool{}} w.dataDir = t.TempDir() for _, name := range []string{"hostguard", ".hostexec", "locks", "exec-locks"} { if err := os.Mkdir(filepath.Join(w.dataDir, name), 0o770); err != nil { t.Fatal(err) } } w.inner = release.NewDirStore(t.TempDir()) key, _ := hostclaim.Key(w.host) w.store = &lifeStore{Store: w.inner, claimKey: key} created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + w.host + ".json", Account: "bab", App: "jam", Host: w.host, Region: "ie", RecordRegion: "ie", CreatedAt: created} body := []byte(`{"host":"` + w.host + `","account":"bab","app":"jam","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) w.claim(hostclaim.BeginCreateOn(nh, body, "c1", "console", created, hostclaim.ExecutorSelection{Box: "src", Enrollment: strings.Repeat("a", 32)})) w.claim(hostclaim.CompleteCreate("c1", "src")) w.claim(hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) w.cfg = execctl.Config{Store: w.store, Guard: hostguard.Guard{DataDir: w.dataLink(0), LockDir: filepath.Join(w.dataDir, "locks"), Claims: hostguard.StoreReader{Store: w.store}}, Directory: activeDirectory{}, Clock: w.clock, LockDir: filepath.Join(w.dataDir, "exec-locks"), CloseAdmission: func() {}, CancelWork: func() {}} lifetime, stop := context.WithCancel(ctx) t.Cleanup(stop) w.e = &RelocationEffects{Lifetime: lifetime, Identity: func(context.Context) (RelocationIdentity, error) { return RelocationIdentity{Actor: hostbox.Actor{Box: "src", Enrollment: strings.Repeat("a", 32), Revision: 4}, BootID: "boot", ControllerID: "ctl"}, nil }, Controller: func(ctx context.Context, b execctl.Binding) (RelocationController, error) { c := w.cfg c.Binding = b // hostexec's own Cleanup hook is the same scoped cleanup, // bound to the binding (Task 8a); it does not count for (a). if cur := w.current(); cur.Op.ID == b.Op && !slices.Contains(cur.Op.Participants, "src") { w.violate(fmt.Sprintf("(i) %s opened again after this box was excluded from it", b.Op)) } gen := w.generation() c.Cleanup = func(context.Context) error { return w.cleanupScoped(context.Background(), b.Op, false, gen) } opened, err := OpenRelocationController(ctx, c) if err != nil { return nil, err } w.mu.Lock() w.opens++ w.mu.Unlock() return &lifeController{RelocationController: opened, w: w, op: b.Op}, nil }, Install: func(_ context.Context, o hostclaim.Observed) (hostclaim.LocalInstall, error) { c := w.current() if hostclaim.PreparedOpened(c) { w.violate("(g) the local install was remade after P was opened") } w.mu.Lock() w.installed[c.Op.ID] = true w.mu.Unlock() n, _ := strconv.ParseUint(c.Op.Journal["install_stamp"], 10, 64) return hostclaim.LocalInstall{Box: "src", Enrollment: strings.Repeat("a", 32), InstallStamp: n, FilesManifestVersion: 1, JournalSHA256: strings.Repeat("e", 64)}, nil }, Receipt: func(_ context.Context, o hostclaim.Observed) (hostclaim.LocalInstall, bool, error) { var l hostclaim.LocalInstall _ = json.Unmarshal([]byte(o.Claim.Op.Journal["local_install"]), &l) w.mu.Lock() defer w.mu.Unlock() return l, w.installed[o.Claim.Op.ID], nil }, VerifyLocal: func(_ context.Context, o hostclaim.Observed, _ hostclaim.LocalInstall) (bool, error) { w.mu.Lock() defer w.mu.Unlock() return w.installed[o.Claim.Op.ID], nil }, Admit: func(_ context.Context, o hostclaim.Observed, _ string) (uint64, bool, error) { c := w.current() w.mu.Lock() present := w.installed[c.Op.ID] w.mu.Unlock() if !hostclaim.PreparedOpened(c) && !present { w.violate("(f) P's first admission over a missing local install") } // The admission opens P (the claim journals it). if _, err := hostclaim.Update(context.Background(), w.inner, o, func(c *hostclaim.Claim) error { c.Op.Journal["prepared_opened"] = c.Op.Journal["prepared"] return nil }); err != nil { return 0, false, err } return 0, false, nil // the probe is still running }} w.e.Controllers = w.controllers() w.p = hostrelocate.Participant{Claims: w.store, Box: "src", Enrollment: strings.Repeat("a", 32), Effects: w.e, Now: func() time.Time { return w.clock.wall }} return w } // lifeController checks invariant (a) at every clear of a live // controller's record. type lifeController struct { RelocationController w *lifeWorld op string } func (c *lifeController) Clear(ctx context.Context) error { c.w.clearing(c.op) err := c.RelocationController.Clear(ctx) c.w.cleared(err) return err } func (w *lifeWorld) cleared(err error) { if err == nil { w.mu.Lock() w.clears++ w.mu.Unlock() } } func (w *lifeWorld) generation() int { w.mu.Lock() defer w.mu.Unlock() return w.gen } func (w *lifeWorld) controllers() *RelocationControllers { gen := w.generation() return &RelocationControllers{Journal: filepath.Join(w.dataDir, "relocate", "participations"), DataDir: w.dataDir, Cleanup: func(ctx context.Context, b execctl.Binding) error { return w.cleanupScoped(ctx, b.Op, true, gen) }, Clear: func(ctx context.Context, b execctl.Binding) error { w.clearing(b.Op) c := w.cfg c.Binding = b err := execctl.ClearStale(ctx, c) w.cleared(err) return err }} } // cleanupScoped is the scoped cleanup (Task 8a's relocationCleanup): // it retires the operation's local install unless the claim has opened // that operation's P (it never stops what serves). counts says whether it // is RelocationControllers' own call, the one invariant (a) asks for. func (w *lifeWorld) cleanupScoped(ctx context.Context, op string, counts bool, gen int) error { w.mu.Lock() block, fail, dead := w.block, w.cleanErr, gen != w.gen w.mu.Unlock() if dead { select {} // a crashed process's hooks never complete } w.mu.Lock() if counts { w.cleanOK[op] = false } w.mu.Unlock() if block != nil { select { case <-block: case <-ctx.Done(): return ctx.Err() } } if gen != w.generation() { select {} // the process died while its cleanup waited } if fail { return errors.New("cleanup: a partial install would not go") } c, err := hostclaim.Read(context.Background(), w.inner, w.host) opened := err == nil && !c.Absent && c.Claim.Op.ID == op && hostclaim.PreparedOpened(c.Claim) w.mu.Lock() if !opened { w.installed[op] = false } if counts { w.cleanOK[op] = true } w.mu.Unlock() return nil } // clearing is invariant (a): a record is cleared only straight after a // scoped cleanup of the same participation succeeded, in the same call. func (w *lifeWorld) clearing(op string) { w.mu.Lock() defer w.mu.Unlock() if !w.cleanOK[op] { w.violated = append(w.violated, "(a) clear of "+op+" without a successful scoped cleanup first") } w.cleanOK[op] = false } func (w *lifeWorld) claim(tr func(*hostclaim.Claim) error) { w.t.Helper() o, err := hostclaim.Read(w.ctx, w.inner, w.host) if err != nil { w.t.Fatal(err) } if _, err := hostclaim.Update(w.ctx, w.inner, o, tr); err != nil { w.t.Fatal(err) } } func (w *lifeWorld) current() *hostclaim.Claim { w.t.Helper() o, err := hostclaim.Read(w.ctx, w.inner, w.host) if err != nil { w.t.Fatal(err) } return o.Claim } // startRelocation starts the next relocation, src taking part (as source // or as target), whatever the claim was doing. func (w *lifeWorld) startRelocation() { w.ops++ id := fmt.Sprintf("rel-%d", w.ops) from, to := "src", "dst" if exec := w.current().Op.Exec; exec != "" && exec != "src" && exec != "none" { from, to = exec, "src" } w.claim(func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: id, Phase: hostclaim.PhaseStarting, Exec: from, Participants: []string{from, to}, Intent: map[string]string{"region": "ie", "source_box": from, "target_box": to, "target_enrollment": map[string]string{"src": strings.Repeat("a", 32), "dst": strings.Repeat("d", 32), "sp2": strings.Repeat("e", 32)}[to]}} return nil }) } // replaced is the worker replacing this box, the target, within the same // operation (rev 19, round 18 finding 1): after its release in // probe-failed, or at any target phase, the relocation goes on with sp2 // and this box is no longer a participant. Its controller, record and // journal must go, and nothing for this operation may open again here. func (w *lifeWorld) replaced() bool { c := w.current() if c.Op.Kind != hostclaim.KindRelocating || c.Op.Intent["target_box"] != "src" { return false } w.claim(func(c *hostclaim.Claim) error { if c.Op.Journal == nil { c.Op.Journal = map[string]string{} } c.Op.Participants = append(slices.DeleteFunc(slices.Clone(c.Op.Participants), func(b string) bool { return b == "src" }), "sp2") c.Op.Intent["target_box"], c.Op.Intent["target_enrollment"] = "sp2", strings.Repeat("e", 32) if c.Op.Phase != hostclaim.PhaseStarting && c.Op.Phase != hostclaim.PhaseReleasing { c.Op.Exec, c.Op.Phase = "none", hostclaim.PhasePinning for _, k := range []string{"local_install", "probe", "prepared", "prepared_opened", "install_stamp", "release", "release.phase", "probing_since", "probe_failed_since"} { delete(c.Op.Journal, k) } } c.Op.Journal["excluded_targets"] = `["src"]` c.Op.Journal["exec_ack.sp2"] = "{}" return nil }) return true } // probeFailedAndReleased moves this box, the target, to probe-failed with // its release journaled (the worker's and its own half, in short). func (w *lifeWorld) probeFailedAndReleased() { w.claim(func(c *hostclaim.Claim) error { if c.Op.Journal == nil { c.Op.Journal = map[string]string{} } c.Op.Phase = hostclaim.PhaseProbeFailed c.Op.Journal["probe_failed_since"] = w.clock.wall.Format(time.RFC3339Nano) c.Op.Journal["release"] = `{"released_by":"src"}` c.Op.Journal["release.phase"] = hostclaim.PhaseProbeFailed return nil }) } // install is the worker's half while this box is the target: P built and // the operation handed to it (installed), its local install still to make. func (w *lifeWorld) install() { c := w.current() if c.Op.Kind != hostclaim.KindRelocating || c.Op.Intent["target_box"] != "src" || (c.Op.Phase != hostclaim.PhaseStarting && c.Op.Phase != hostclaim.PhaseReleasing) { return } w.claim(func(c *hostclaim.Claim) error { c.Op.Phase, c.Op.Exec = hostclaim.PhaseInstalled, "src" if c.Op.Journal == nil { c.Op.Journal = map[string]string{} } c.Op.Journal["install_stamp"] = strconv.FormatUint(c.Gen+1, 10) c.Op.Journal["prepared"] = "P-" + c.Op.ID if c.Op.Journal["exec_ack.dst"] == "" { c.Op.Journal["exec_ack.dst"] = "{}" } return nil }) } func (w *lifeWorld) complete() { if c := w.current(); c.Op.Kind == hostclaim.KindRelocating { box := c.Op.Intent["target_box"] w.claim(func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: box, Participants: []string{box}} return nil }) } } // pass is one participant pass, Step then Sweep as the agent runs them, // under a deadline; (d) is that it returns within it. func (w *lifeWorld) pass() { w.mu.Lock() w.cleanOK = map[string]bool{} // "the same call": nothing carries over from an earlier pass w.mu.Unlock() done := make(chan struct{}) start := time.Now() go func() { defer close(done) ctx, cancel := context.WithTimeout(w.ctx, lifeDeadline) defer cancel() _, _ = w.p.Step(ctx, w.host) _ = w.p.Sweep(ctx) }() select { case <-done: if took := time.Since(start); took > lifeDeadline+time.Second { w.violate(fmt.Sprintf("(d) a pass took %v past its %v deadline", took, lifeDeadline)) } case <-time.After(lifeDeadline + 3*time.Second): w.t.Fatalf("(d) a pass never returned within %v", lifeDeadline+3*time.Second) } } func (w *lifeWorld) violate(s string) { w.mu.Lock() w.violated = append(w.violated, s) w.mu.Unlock() } // renew fires hostexec's timers so every controller renews (or loses). func (w *lifeWorld) renew() { w.clock.tick() time.Sleep(20 * time.Millisecond) } func (w *lifeWorld) record() (execctl.State, bool, bool) { raw, err := os.ReadFile(filepath.Join(w.dataDir, ".hostexec", w.host+".json")) if errors.Is(err, os.ErrNotExist) { return execctl.State{}, false, true } var st execctl.State if err != nil || json.Unmarshal(raw, &st) != nil { return st, true, false } return st, true, true } func (w *lifeWorld) open() []RelocationController { r := w.e.Controllers r.mu.Lock() defer r.mu.Unlock() var out []RelocationController for _, c := range r.open { out = append(out, c) } return out } // check is invariants (a) to (c) after a step. func (w *lifeWorld) check(step string) { st, exists, readable := w.record() if exists && readable { retained, _ := w.e.Controllers.Retained() if !slices.ContainsFunc(retained, func(p hostrelocate.Participation) bool { return p.Host == w.host && p.Op == st.Op }) { w.violate(fmt.Sprintf("(b) after %s: %s's record exists but its participation is forgotten (retained %+v)", step, st.Op, retained)) } } if n := len(w.open()); n > 1 { w.violate(fmt.Sprintf("(c) after %s: %d controllers for one host", step, n)) } // (h): a record goes only by a clear or by a successor's open over it, // never by a crash or any other step; while one exists, an executable // update is refused (root's §5.10 check). w.mu.Lock() clears, opens := w.clears, w.opens w.clears, w.opens = 0, 0 w.mu.Unlock() if w.prevExists && w.prevReadable && (!exists || (readable && st.Op != w.prevOp)) && clears == 0 && opens == 0 { w.violate(fmt.Sprintf("(h) after %s: %s's record went with no clear and no successor's open", step, w.prevOp)) } if exists && hostexecParticipation(w.dataDir) == nil { w.violate(fmt.Sprintf("(h) after %s: a record exists but an executable update is not refused", step)) } w.prevExists, w.prevReadable, w.prevOp = exists, readable, st.Op } // dataLink is generation gen's path to the data directory: a symlink, so // a crash can cut a dead generation off from it. A dead process writes // nothing; in-process, a dead controller's goroutines are still running // (hostexec's save loop retries a "lost" write until it lands), so the // link it opens through is removed. func (w *lifeWorld) dataLink(gen int) string { link := filepath.Join(filepath.Dir(w.dataDir), fmt.Sprintf("data-%d", gen)) if err := os.Symlink(w.dataDir, link); err != nil && !os.IsExist(err) { w.t.Fatal(err) } return link } // crash is the agent process dying abruptly (rev 16, round 15): no // controller is closed and none of its hooks run or complete; its durable // records stay, and every external fault (a blocked or failing cleanup, a // store that refuses) stays as it was. The kernel releases a dead // process's locks: the next generation uses fresh lock files, and the dead // generation's clock never fires again, so its controllers never renew, // lose or clean up. Recovery is a separate, later event. func (w *lifeWorld) crash() { w.mu.Lock() w.gen++ gen := w.gen w.mu.Unlock() now, boot, _ := w.clock.Now() w.clock = &lifeClock{wall: now, boot: boot} for _, d := range []string{fmt.Sprintf("exec-locks-%d", gen), fmt.Sprintf("locks-%d", gen)} { if err := os.MkdirAll(filepath.Join(w.dataDir, d), 0o770); err != nil { w.t.Fatal(err) } } w.cfg.Clock = w.clock w.cfg.LockDir = filepath.Join(w.dataDir, fmt.Sprintf("exec-locks-%d", gen)) w.cfg.Guard.LockDir = filepath.Join(w.dataDir, fmt.Sprintf("locks-%d", gen)) if err := os.Remove(w.cfg.Guard.DataDir); err != nil { w.t.Fatal(err) } w.cfg.Guard.DataDir = w.dataLink(gen) w.e.Controllers = w.controllers() } func (w *lifeWorld) recoverCleanup() { w.mu.Lock() if w.block != nil { close(w.block) w.block = nil } w.cleanErr = false w.mu.Unlock() } func (w *lifeWorld) recoverAll() { w.recoverCleanup() w.store.mu.Lock() w.store.failLease, w.store.claimErr = false, nil w.store.mu.Unlock() if w.saved != nil { path := filepath.Join(w.dataDir, ".hostexec", w.host+".json") if raw, err := os.ReadFile(path); err == nil && string(raw) == "{" { _ = os.WriteFile(path, w.saved, 0o660) // hostexec retries a "lost" write every 100ms through a // directory it already holds open, so a controller that met // the invalid record (a dead generation's included: in-process // its goroutine outlives the crash) lands its write now, as // part of the repair, never after a later clear. time.Sleep(300 * time.Millisecond) } w.saved = nil } } func (w *lifeWorld) event(r *rand.Rand) string { switch n := r.Intn(16); n { case 15: if w.replaced() { return "this box replaced in the same operation" } return "no replacement (not the target)" case 14: w.install() return "the target's installation (P built)" case 0, 1, 2, 3: w.pass() return "pass" case 4: w.renew() return "renew" case 5: w.store.mu.Lock() w.store.failLease = true w.store.mu.Unlock() w.renew() return "lease renewal fails" case 6: w.mu.Lock() if w.block == nil { w.block = make(chan struct{}) } w.mu.Unlock() return "cleanup blocks (close times out)" case 7: w.mu.Lock() w.cleanErr = true w.mu.Unlock() return "cleanup fails" case 8: w.store.mu.Lock() if r.Intn(2) == 0 { w.store.claimErr = errors.New("claim read: store unreachable") } else { w.store.claimErr = os.ErrNotExist // ClearStale: ErrState for an absent claim } w.store.mu.Unlock() return "claim reads fail" case 9: path := filepath.Join(w.dataDir, ".hostexec", w.host+".json") if raw, err := os.ReadFile(path); err == nil && w.saved == nil { w.saved = raw _ = os.WriteFile(path, []byte("{"), 0o660) // ClearStale: an invalid local record } return "local record invalid" case 10: w.crash() return "agent crash (no hooks run, faults persist)" case 11: w.complete() w.startRelocation() // the successor starts before this box's sweep return "completion, successor before the sweep" case 12: w.complete() return "completion" default: w.recoverAll() return "storage recovers" } } // TestRelocationControllerLifecycleProperties drives seeded random // sequences of lease-renewal failures, close timeouts, cleanup failures, // ClearStale failures (claim unreadable, claim absent, invalid local // record, a successor that lists this box), agent restarts, successors // before the sweep, completions and recovery, over real hostexec // controllers, and checks after every step: (a) no clear without a // successful scoped cleanup in the same call; (b) no participation // forgotten while its record exists; (c) at most one controller per host; // (d) no pass past its deadline; and (e) once faults stop, within eight // passes the current relocation is ready again, then after completion // every record is cleared and no controller remains, and a new relocation // opens. func TestRelocationControllerLifecycleProperties(t *testing.T) { seeds := lifeSeeds if testing.Short() { seeds = 20 } for seed := 1; seed <= seeds; seed++ { t.Run(fmt.Sprint(seed), func(t *testing.T) { r := rand.New(rand.NewSource(int64(seed))) w := newLifeWorld(t) defer w.recoverAll() w.startRelocation() var trace []string if seed%2 == 0 { // Every other seed starts as the target with its local // install journaled (rev 15), so the random events fall on // an installation that a lost lease's cleanup can retire // before P's first admission. w.complete() w.startRelocation() // dst back to src: this box is the target w.pass() w.install() w.pass() trace = append(trace, "target, installed, local install journaled") if w.current().Op.Journal["local_install"] == "" { t.Fatalf("seed %d: the local install was not journaled: %+v", seed, w.current().Op) } if seed%4 == 2 { // Round 18's trace: the target's probe fails, it // releases, it is replaced while the relocation goes // on elsewhere, its cleanup blocks, the agent crashes. w.pass() w.probeFailedAndReleased() if !w.replaced() { t.Fatalf("seed %d: this box was not the target to replace: %+v", seed, w.current().Op) } w.mu.Lock() w.block = make(chan struct{}) w.mu.Unlock() w.pass() w.check("replaced with its cleanup blocked") w.crash() w.check("agent crash after the replacement") trace = append(trace, "probe failed, released, replaced, cleanup blocked, crash") } } for i := 0; i < 30; i++ { ev := w.event(r) trace = append(trace, ev) w.check(ev) if len(w.violated) > 0 { t.Fatalf("seed %d: %v\ntrace: %v", seed, w.violated, trace) } } // (e) Faults stop. w.recoverAll() trace = append(trace, "faults stop") // A participating box is ready again; an excluded one (rev 19) // has finished its cleanup: no controller, no record, no // journaled participation, while the relocation goes on. ready := func() bool { c := w.current() if c.Op.Kind != hostclaim.KindRelocating { return true } if !slices.Contains(c.Op.Participants, "src") { _, exists, _ := w.record() retained, err := w.e.Controllers.Retained() return !exists && err == nil && len(retained) == 0 && len(w.open()) == 0 } if c.Op.Journal["exec_ack.src"] == "" { return false } obs, err := hostclaim.Read(w.ctx, w.inner, w.host) return err == nil && w.e.Ready(w.ctx, obs) == nil } ok := false for i := 0; i < 8 && !ok; i++ { w.renew() w.pass() w.check("recovery pass") ok = ready() } if !ok || len(w.violated) > 0 { t.Fatalf("seed %d: the running relocation did not recover within 8 passes: %v\ntrace: %v", seed, w.violated, trace) } w.complete() clean := false for i := 0; i < 8 && !clean; i++ { w.renew() w.pass() w.check("sweep pass") _, exists, _ := w.record() retained, err := w.e.Controllers.Retained() clean = !exists && len(retained) == 0 && err == nil && len(w.open()) == 0 } if !clean || len(w.violated) > 0 { st, exists, _ := w.record() retained, _ := w.e.Controllers.Retained() t.Fatalf("seed %d: not clean within 8 passes after completion: record %v %+v retained %+v open %d %v\ntrace: %v", seed, exists, st, retained, len(w.open()), w.violated, trace) } w.startRelocation() for i := 0; i < 3; i++ { w.pass() } w.check("new relocation") if !ready() || len(w.open()) != 1 || len(w.violated) > 0 { t.Fatalf("seed %d: a new relocation did not open: open %d %v\ntrace: %v", seed, len(w.open()), w.violated, trace) } for _, c := range w.open() { _ = c.Close(context.Background()) } }) } } ``` `internal/hostexec/readlocal.go` (rev 15: hostexec's validated reader of its own local record, for `recordIsMine`): ```go package hostexec // ReadLocal reads host's durable local record (/.hostexec/.json) // with the validation Open, CheckLocal and ClearStale apply to it: a // regular file, a complete binding (box, enrollment, host, incarnation, // op, boot and controller ids), a known state. An absent record is // os.ErrNotExist; anything it would refuse is an error, never a State. // It takes no lock and grants nothing: a caller deciding whether a record // is still its own uses it, and keeps its own state on any error. func ReadLocal(dataDir, host string) (State, error) { d, err := openLocal(dataDir) if err != nil { return State{}, err } defer d.close() return d.read(host) } ``` `internal/host/relocationeffects.go`: ```go package host import ( "context" "encoding/json" "errors" "fmt" "os" "path/filepath" "sort" "strings" "sync" "github.com/carlosframework/platform/internal/hostbox" "github.com/carlosframework/platform/internal/hostclaim" execctl "github.com/carlosframework/platform/internal/hostexec" "github.com/carlosframework/platform/internal/hostrelocate" "github.com/carlosframework/platform/internal/stopevidence" ) // RelocationIdentity is this box's verified actor and stop identity, as // Plan 6b's assembly reports it (claimStopLocal). type RelocationIdentity struct { Actor hostbox.Actor BootID, ControllerID string } // RelocationController is the execution controller as this package uses // it: hostexec.Controller's Ready and Close, and Clear, which removes the // controller's durable local record once its cleanup is done // (hostexec.ClearStale with the controller's config): Close leaves the // record in "lost", and under §5.10 any record blocks an executable // update until it is cleared. type RelocationController interface { Ready(context.Context) (execctl.State, error) Close(context.Context) error Clear(context.Context) error // Lost is closed once the controller has lost its lease for good // (hostexec latches the first failure: a failed renewal, a lost // authority, a clock fault). Lost() <-chan struct{} } // participation is one journaled participation whose controller may have // left a durable record: written before the controller is opened // (Acknowledge), removed only after the record is cleared (Finish). A // crash anywhere in between leaves the entry, so a restarted agent finds // every record it may have left, including one from a controller that // never reached its first Finish or whose Open failed after writing. type participation struct { V int `json:"v"` Host string `json:"host"` Op string `json:"op"` Binding execctl.Binding `json:"binding"` } // RelocationControllers retains the execution controllers this agent has // opened for its participations, one per {host, op}, for the agent's // lifetime. A controller is never bound to a pass's context (claim D6: // the lease lives as long as the participation). A restarted agent has // none and opens a fresh one; the claim keeps the first acknowledgement. type RelocationControllers struct { mu sync.Mutex open map[string]RelocationController // Journal is where participations are recorded before their // controller opens (/relocate/participations); Finish // retries a journaled one after a restart, when no controller object // exists, through the binding it recorded. Required: Acknowledge // opens nothing without it. Journal string // Cleanup is the participation's scoped cleanup for one binding // (Task 8a's relocationCleanup: it retires only that operation's own // partial installation of that host and incarnation, after checking // the binding's box and enrollment are this box's verified actor, and // does nothing once this box has admitted a writer under the // operation or serves the host). Finish repeats it, and needs it to // succeed, in the same call as every clear: hostexec.Controller.Close // returns a cleanup failure once and nil on every later call, and a // restarted agent has no controller at all, so neither a nil Close // nor a journal entry proves the cleanup ran. Nil clears nothing. Cleanup func(context.Context, execctl.Binding) error // Clear removes a journaled binding's durable record after a restart // (hostexec.ClearStale with the assembly's config for the binding). Clear func(context.Context, execctl.Binding) error // DataDir is hostexec's data directory (Guard.DataDir), whose // .hostexec/.json is the durable record. When a clear fails, // the participation is forgotten only if that record is verifiably // not this binding's: absent, or naming another binding (rev 14). // Empty: never verifiable, so a failed clear always keeps it. DataDir string // bindings remembers each open controller's binding. bindings map[string]execctl.Binding } func controllerKey(host, op string) string { return host + "\x00" + op } func (r *RelocationControllers) get(host, op string) (RelocationController, bool) { r.mu.Lock() defer r.mu.Unlock() c, ok := r.open[controllerKey(host, op)] return c, ok } func (r *RelocationControllers) put(host, op string, b execctl.Binding, c RelocationController) (RelocationController, bool) { r.mu.Lock() defer r.mu.Unlock() if r.open == nil { r.open, r.bindings = map[string]RelocationController{}, map[string]execctl.Binding{} } if prior, ok := r.open[controllerKey(host, op)]; ok { return prior, false } r.open[controllerKey(host, op)], r.bindings[controllerKey(host, op)] = c, b return c, true } const journalSuffix = ".json" func (r *RelocationControllers) journalPath(host, op string) string { return filepath.Join(r.Journal, host+"~"+op+journalSuffix) } // record journals a participation before its controller opens. func (r *RelocationControllers) record(host, op string, b execctl.Binding) error { if r.Journal == "" { return ErrClaimStopUnavailable } raw, err := json.Marshal(participation{V: 1, Host: host, Op: op, Binding: b}) if err != nil { return err } if err := os.MkdirAll(r.Journal, 0o700); err != nil { return err } return writeFileSynced(r.journalPath(host, op), raw, 0o600) } // journaled lists the committed participations on disk. Only names // ending in .json are committed records: writeFileSynced's .json.tmp is // an uncommitted write a crash may have truncated, and is ignored (its // participation opened nothing, because the record is written before the // controller opens). A committed record that cannot be read is reported, // joined into the error, and every readable one is still returned, so // one damaged entry never hides the others' cleanup. func (r *RelocationControllers) journaled() ([]participation, error) { if r.Journal == "" { return nil, nil } entries, err := os.ReadDir(r.Journal) if errors.Is(err, os.ErrNotExist) { return nil, nil } if err != nil { return nil, err } var out []participation var errs []error for _, e := range entries { if e.IsDir() || !strings.HasSuffix(e.Name(), journalSuffix) { continue } raw, err := os.ReadFile(filepath.Join(r.Journal, e.Name())) if err != nil { errs = append(errs, err) continue } var f participation if err := json.Unmarshal(raw, &f); err != nil || f.V != 1 || f.Host == "" || f.Op == "" { errs = append(errs, fmt.Errorf("%w: participation journal %s", ErrClaimStopUnavailable, e.Name())) continue } out = append(out, f) } return out, errors.Join(errs...) } // Retained lists the participations whose controllers are open, and the // journaled ones whose durable record may still need clearing. A damaged // journal entry is returned as the error beside every readable one. func (r *RelocationControllers) Retained() ([]hostrelocate.Participation, error) { r.mu.Lock() seen := map[string]bool{} var out []hostrelocate.Participation for key := range r.open { host, op, _ := strings.Cut(key, "\x00") seen[key] = true out = append(out, hostrelocate.Participation{Host: host, Op: op}) } r.mu.Unlock() journaled, err := r.journaled() for _, f := range journaled { if !seen[controllerKey(f.Host, f.Op)] { seen[controllerKey(f.Host, f.Op)] = true out = append(out, hostrelocate.Participation{Host: f.Host, Op: f.Op}) } } sort.Slice(out, func(i, j int) bool { return out[i].Host < out[j].Host || (out[i].Host == out[j].Host && out[i].Op < out[j].Op) }) return out, err } // Finish closes and forgets one participation's controller, releasing the // host's lifetime lock for the next operation, and clears its durable // record. Closing runs hostexec's Cleanup hook: the relocation's // controller is opened with a cleanup scoped to the relocation's own // partial work (Task 8a), never the serving writer's session, so // finishing after a successful relocation stops nothing that serves. // // The entry stays until Close succeeds: hostexec retains the lifetime // lock on a timed-out join and needs another Close, so a controller whose // close the pass's context cut short is found by the next sweep. Then the // scoped cleanup is repeated (Cleanup) and the record is cleared only if // that cleanup succeeded in this call: a nil Close after an earlier // failed one, or a restart with no controller, proves nothing. The // journal entry goes only once the record is gone, so a restart in // between retries the cleanup and the clear through the recorded binding. func (r *RelocationControllers) Finish(ctx context.Context, host, op string) error { return r.finish(ctx, host, op, false) } // errSuperseded is finish's answer, for a successor's handover, when the // predecessor's controller is closed and cleaned up but hostexec refuses // to clear its record because the claim's running operation lists this // box (hostexec.ClearStale, controller.go: "a fresh successor operation // may exclude this box"; here it includes it). The lifetime lock is free, // and the successor's Open replaces the record under it. var errSuperseded = errors.New("relocation: the predecessor's record passes to its successor") func (r *RelocationControllers) finish(ctx context.Context, host, op string, successor bool) error { key := controllerKey(host, op) r.mu.Lock() c, ok := r.open[key] b := r.bindings[key] r.mu.Unlock() if !ok { // No controller object: a restart, or a crash before the first // Finish. Retry through the journaled binding. journaled, err := r.journaled() found := false for _, f := range journaled { if f.Host == host && f.Op == op { found, b = true, f.Binding } } if !found { return err // nil: nothing journaled; else the damaged entry may be this one } } if r.Cleanup == nil || (!ok && r.Clear == nil) { return ErrClaimStopUnavailable } if ok { if err := c.Close(ctx); err != nil { return err // cleanup failed or was cut short: the record stays, and so does the entry } } if err := r.Cleanup(ctx, b); err != nil { return err // the record stays: no clear without a cleanup that succeeded now } clear := func() error { return r.Clear(ctx, b) } if ok { clear = func() error { return c.Clear(ctx) } } if !ok { // No controller object: when the record is verifiably not this // binding's (gone, or a successor's that holds the host's lock), // there is nothing to clear, and ClearStale would only wait out // the pass on the successor's lock. if mine, verr := r.recordIsMine(host, b); verr == nil && !mine { return r.forget(host, op) } } superseded := false if err := clear(); err != nil { // Whatever the error, the participation goes only if the record // itself shows it is not this binding's (round 13 finding 2: // ClearStale's ErrState also means an absent claim or an invalid // local record, and those leave this binding's record in place). mine, verr := r.recordIsMine(host, b) switch { case verr != nil: return errors.Join(err, verr) case !mine: // Verified: no record, or another binding's (a successor // opened over it). case successor && errors.Is(err, execctl.ErrInactive): superseded = true // the lock is free; the journal stays until the successor opens default: return err } } r.mu.Lock() if ok && r.open[key] == c { delete(r.open, key) delete(r.bindings, key) } r.mu.Unlock() if superseded { return errSuperseded } return r.forget(host, op) } // recordIsMine reads hostexec's durable record for host through // hostexec's own validated reader (ReadLocal) and reports whether it is // b's binding (box, enrollment, host, incarnation, operation: what // ClearStale matches). Only an absent record, or a valid record of another // binding, is "not mine"; a record hostexec would refuse (one missing its // enrollment or incarnation, say) is an error, and the participation is // kept (round 14). func (r *RelocationControllers) recordIsMine(host string, b execctl.Binding) (bool, error) { if r.DataDir == "" { return false, fmt.Errorf("%w: no hostexec data directory to verify the record", ErrClaimStopUnavailable) } st, err := execctl.ReadLocal(r.DataDir, host) if errors.Is(err, os.ErrNotExist) { return false, nil } if err != nil { return false, fmt.Errorf("hostexec record for %s: %w", host, err) } return st.Box == b.Box && st.Enrollment == b.Enrollment && st.Incarnation == b.Incarnation && st.Op == b.Op, nil } // live returns the open controller for {host, op}, retiring it first if it // has lost its lease (round 13 finding 1): hostexec latches the first // failure, so a lost controller never becomes ready again. Retiring joins // it (Close, bounded by ctx), repeats the scoped cleanup, and forgets the // object but keeps the journal: the record on disk stays "lost" (durable // denial) until a fresh controller, opened with fresh authority and its // own lease, replaces it. ok false: none open (or just retired). func (r *RelocationControllers) live(ctx context.Context, host, op string) (RelocationController, bool, error) { key := controllerKey(host, op) r.mu.Lock() c, ok := r.open[key] b := r.bindings[key] r.mu.Unlock() if !ok { return nil, false, nil } select { case <-c.Lost(): default: return c, true, nil } if r.Cleanup == nil { return nil, false, ErrClaimStopUnavailable } if err := c.Close(ctx); err != nil { return nil, false, err // joined later; the entry stays } if err := r.Cleanup(ctx, b); err != nil { return nil, false, err // the entry stays: the next pass repeats both } r.mu.Lock() if r.open[key] == c { delete(r.open, key) delete(r.bindings, key) } r.mu.Unlock() return nil, false, nil } // forget removes one participation's journal entry. func (r *RelocationControllers) forget(host, op string) error { if r.Journal != "" { if err := os.Remove(r.journalPath(host, op)); err != nil && !errors.Is(err, os.ErrNotExist) { return err } } return nil } // handover retires every other participation of host this box still holds // before op's controller opens (round 12 finding 1): hostexec keeps one // controller per host, and a predecessor whose sweep has not yet run still // holds the host's lifetime lock, so the successor's Open would wait on it // for the whole pass. Each predecessor is closed and its scoped cleanup // repeated under the pass's context (bounded). One whose record hostexec // will not clear because the successor lists this box (errSuperseded) // keeps its journal entry: the successor's Open replaces the record under // the freed lock, and a later sweep forgets the entry once the record is // verifiably the successor's (rev 14). func (r *RelocationControllers) handover(ctx context.Context, host, op string) error { retained, _ := r.Retained() // a damaged entry is Sweep's to report for _, p := range retained { if p.Host != host || p.Op == op { continue } if err := r.finish(ctx, host, p.Op, true); err != nil && !errors.Is(err, errSuperseded) { return fmt.Errorf("relocation: retiring %s's controller for %s before %s: %w", p.Op, host, op, err) } } return nil } // writeFileSynced writes path whole: a synced temporary file renamed into // place, then the directory synced, so a committed entry survives a crash. func writeFileSynced(path string, body []byte, mode os.FileMode) error { tmp := path + ".tmp" if err := os.WriteFile(tmp, body, mode); err != nil { return err } f, err := os.Open(tmp) if err != nil { return err } if err := f.Sync(); err != nil { _ = f.Close() return err } _ = f.Close() if err := os.Rename(tmp, path); err != nil { return err } dir, err := os.Open(filepath.Dir(path)) if err != nil { return err } defer dir.Close() return dir.Sync() } // Close closes every retained controller (agent shutdown). func (r *RelocationControllers) Close(ctx context.Context) error { r.mu.Lock() open := r.open r.open = nil r.mu.Unlock() var first error for _, c := range open { if err := c.Close(ctx); err != nil && first == nil { first = err } } return first } // ReleasePositions keeps the flushed positions of one release durably, // bound to the exact stop request (host, incarnation, stamp, session, // operation, phase, box, enrollment, boot, controller), from before root // completes its receipt. A replayed receipt (root's durable fence after a // lost response or a crash before JournalRelease) is paired with them; // without a record for that exact request the release is refused rather // than guessed. // Recorded matches the stop's installation and session (host, incarnation, // stamp, session, operation, phase, obligation, box, enrollment), not its // controller identity: a restarted agent asks with a new boot and // controller id, and gets back the request the flush was made under, so // it can ask root for that stop's durable receipt. type ReleasePositions interface { Record(ctx context.Context, r StopRequest, txid uint64, seq int64) error Recorded(ctx context.Context, r StopRequest) (recorded StopRequest, txid uint64, seq int64, ok bool, err error) } // RelocationEffects is hostrelocate.Effects over providers Plan 6b's box // assembly supplies. It owns only the order of §5.8's release sequence and // the shape of each record; every provider acts under the host guard. type RelocationEffects struct { // Lifetime is the agent's context: controllers are opened under it, // never under a pass's. Lifetime context.Context Controllers *RelocationControllers Identity func(context.Context) (RelocationIdentity, error) // Controller is Plan 8's execution controller for one binding // (hostexec.Open with the assembly's CloseAdmission, CancelWork and // Cleanup). It writes pending state, checks root's update-begun marker, // and takes the execution lease (§5.10, claim D6). Controller func(context.Context, execctl.Binding) (RelocationController, error) // Producers is the admitted session's drain (ClaimWakeAssembly.Producers). Producers func(context.Context, hostclaim.Observed) (StopDrainer, error) // Collect asks root for full stop evidence (Plan 6b's CollectClaimStop // over the instance-control socket). Collect func(context.Context, StopRequest, StopDrainer) (StopResult, error) // StopWriters is release step 1's action (§5.8, A4-1): close admission, // settle spawn attempts, drain in-flight timer deliveries, stop the // host's process tree and wait for it to empty, keeping the replicator // and the files publisher open. Writers reports what it left. StopWriters func(context.Context, hostclaim.Observed) error // Writers reports the drained facts and the replicator's descriptors // for release step 1's observation; Observe makes it. The complete // descriptor scan reads every process's table, which the agent (uid // carlos) cannot: production's Observe asks root over the // instance-control socket (R3); an agent-side scan fails closed on // the first unreadable table. Writers func(context.Context, hostclaim.Observed) (stopevidence.WritersInputs, error) Observe func(context.Context, stopevidence.WritersInputs) (stopevidence.WritersEvidence, error) // FlushDB is release step 2: final checkpoint and SyncAndWait, then the // stopped database's final TXID N, returned only once the newest // level-0 file's MaxTXID is N. A sync error is returned, never swallowed. FlushDB func(context.Context, hostclaim.Observed) (uint64, error) // FlushFiles is release step 3: publish every pending file, the wake // state and the env file into the session's generation, whatever // files_durability says (ReleaseFlushFiles), and return the manifest's // final sequence. FlushFiles func(context.Context, hostclaim.Observed) (int64, error) // Positions is the durable record of the flushed positions. Positions ReleasePositions // BuildP, Install and Admit are the target's (§5.8): moves' shared // install routine for P, the local install from P's files manifest, // and one cold probe admission followed by the probe. BuildP func(context.Context, hostclaim.Observed) error Install func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error) // VerifyLocal reports whether the local install the receipt names is // complete on this box now: the files directory as P's prepared // manifest lists it, the env file and the wake state, under the // receipt's install_stamp (Task 8c's production body). VerifyLocal func(context.Context, hostclaim.Observed, hostclaim.LocalInstall) (bool, error) // Receipt reads this box's durable local install receipt for the // claim's host (Task 8c: the install journal written under its // install_stamp); false when there is none. Receipt func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, bool, error) Admit func(context.Context, hostclaim.Observed, string) (uint64, bool, error) } var _ hostrelocate.Effects = (*RelocationEffects)(nil) // Acknowledge opens this box's execution controller for the operation // under the agent's lifetime, retains it, and returns the lease it took. // A controller already open for the participation is reused. func (e *RelocationEffects) Acknowledge(ctx context.Context, obs hostclaim.Observed) (hostclaim.ExecutionAcknowledgement, error) { if e.Identity == nil || e.Controller == nil || e.Controllers == nil || e.Lifetime == nil || obs.Claim == nil || obs.Claim.Holder == nil { return hostclaim.ExecutionAcknowledgement{}, ErrClaimStopUnavailable } c := obs.Claim ctl, ok, err := e.Controllers.live(ctx, c.Host, c.Op.ID) if err != nil { return hostclaim.ExecutionAcknowledgement{}, err } if !ok { id, err := e.Identity(ctx) if err != nil { return hostclaim.ExecutionAcknowledgement{}, err } b := execctl.Binding{Box: id.Actor.Box, Enrollment: id.Actor.Enrollment, Host: c.Host, Incarnation: c.Holder.Incarnation, Op: c.Op.ID, BootID: id.BootID} // Journaled before the open: hostexec writes its durable record // inside Open, so a crash or a failed Open leaves a record only // the journal lets a restarted agent find and clear. if err := e.Controllers.record(c.Host, c.Op.ID, b); err != nil { return hostclaim.ExecutionAcknowledgement{}, err } // A predecessor this box has not swept yet still holds the host's // lifetime lock: retire it first, under this pass's context. if err := e.Controllers.handover(ctx, c.Host, c.Op.ID); err != nil { return hostclaim.ExecutionAcknowledgement{}, err } opened, err := openBounded(ctx, e.Lifetime, func(octx context.Context) (RelocationController, error) { return e.Controller(octx, b) }) if err != nil { return hostclaim.ExecutionAcknowledgement{}, err } if ctl, ok = e.Controllers.put(c.Host, c.Op.ID, b, opened); !ok { _ = opened.Close(ctx) // another pass opened one first } } st, err := ctl.Ready(ctx) if err != nil { return hostclaim.ExecutionAcknowledgement{}, err } return hostclaim.ExecutionAcknowledgement{Op: c.Op.ID, Enrollment: st.Enrollment, BootID: st.BootID, ControllerID: st.ControllerID, LeaseETag: st.LeaseETag, Expires: st.Expires}, nil } // openBounded opens a controller that lives under lifetime (claim D6) but // waits for the host's lifetime lock no longer than the pass's ctx: // hostexec.Open takes one context for both. If ctx ends first, the open is // cancelled (Open returns, releasing what it took); a controller that // opened in the same instant is closed, never kept with a cancelled life. func openBounded(ctx, lifetime context.Context, open func(context.Context) (RelocationController, error)) (RelocationController, error) { octx, cancel := context.WithCancel(lifetime) type result struct { c RelocationController err error } ch := make(chan result, 1) // hostexec.Open waits on its Cleanup hook without a context, so the // open runs apart and the pass waits for it no longer than ctx. go func() { c, err := open(octx) ch <- result{c, err} }() select { case res := <-ch: if res.err != nil { cancel() return nil, res.err } context.AfterFunc(lifetime, cancel) // octx ends only with lifetime return res.c, nil case <-ctx.Done(): cancel() go func() { if res := <-ch; res.c != nil { _ = res.c.Close(context.Background()) // opened too late: never kept } }() return nil, ctx.Err() } } // Ready checks the retained controller for the operation is active now. func (e *RelocationEffects) Ready(ctx context.Context, obs hostclaim.Observed) error { if e.Controllers == nil || obs.Claim == nil { return ErrClaimStopUnavailable } ctl, ok, err := e.Controllers.live(ctx, obs.Claim.Host, obs.Claim.Op.ID) if err != nil { return err } if !ok { return hostrelocate.ErrNoController // the participant reopens one (Acknowledge) } _, err = ctl.Ready(ctx) return err } // Release runs the release sequence inside root's stop collection: root // records its durable fence and asks the agent to drain; the drain does // steps 1 to 3 before it closes the producers and acknowledges (§5.8, // claim P6-D1). The receipt is step 4. The flushed positions are durable // before the producers close, so a receipt root replays later still // carries them. func (e *RelocationEffects) Release(ctx context.Context, obs hostclaim.Observed) (hostclaim.Release, error) { if e.Identity == nil || e.Producers == nil || e.Collect == nil || e.StopWriters == nil || e.Writers == nil || e.Observe == nil || e.FlushDB == nil || e.FlushFiles == nil || e.Positions == nil || obs.Claim == nil || obs.Claim.Holder == nil { return hostclaim.Release{}, ErrClaimStopUnavailable } id, err := e.Identity(ctx) if err != nil { return hostclaim.Release{}, err } c, h := obs.Claim, obs.Claim.Holder req := StopRequest{Host: c.Host, Incarnation: h.Incarnation, InstallationStamp: h.Stamp, WriterSession: h.Session, OperationID: c.Op.ID, Phase: c.Op.Phase, ObligationID: RelocationReleaseObligation, Box: id.Actor.Box, Enrollment: id.Actor.Enrollment, BootID: id.BootID, ControllerID: id.ControllerID, ExpectedClaimETag: obs.ETag, ExpectedDirectoryRevision: fmt.Sprint(id.Actor.Revision)} // A flush this box already made for this stop, by any controller of // this installation: after a restart the request is made under the // identity that flushed, so root finds that stop's durable receipt. recorded, txid, seq, flushed, err := e.Positions.Recorded(ctx, req) if err != nil { return hostclaim.Release{}, err } if flushed { // The stop is the recorded one: root's fence and its inventory // check name the directory revision the flush was made under, so // a retry keeps it even when the roster has moved on since // (round 8 finding 2), whether or not this is a recovery. req.ExpectedDirectoryRevision = recorded.ExpectedDirectoryRevision } var d StopDrainer = RecoveryReleaseDrain{} drain := &releaseDrain{e: e, obs: obs} if flushed && (recorded.BootID != req.BootID || recorded.ControllerID != req.ControllerID) { // The flushing process is gone with its producers: only root's // durable receipt for the recorded identity can finish this // release. The request names the recorded owner (boot, controller, // directory revision) and the drain refuses; the Collect provider // marks it a recovery request (6b's authenticated Recover path, // Task 8a), under which root accepts another tuple for this box's // session and replays the receipt. req.BootID, req.ControllerID = recorded.BootID, recorded.ControllerID } else { if drain.inner, err = e.Producers(ctx, obs); err != nil { return hostclaim.Release{}, err } if flushed { // This process already flushed and recorded this stop, and // root has no durable receipt yet (it failed after the drain // acknowledged, before CompleteFence: claimstop.go). Steps 1 // to 3 are done and the replicator may be closed: resume at // the producers' idempotent drain with the recorded positions, // and let root verify and persist the receipt. drain.resumed, drain.txid, drain.seq = true, txid, seq } d = drain } result, err := e.Collect(ctx, req, d) if err != nil { return hostclaim.Release{}, err } if drain.done { txid, seq = drain.txid, drain.seq } else if !flushed { // Root replayed a durable receipt and nothing recorded a flush for // this stop: never guess. return hostclaim.Release{}, fmt.Errorf("%w: a replayed receipt has no recorded flushed positions for this stop", ErrClaimStopReceipt) } return hostclaim.Release{ReleasedBy: id.Actor.Box, Enrollment: id.Actor.Enrollment, Generation: h.Generation, Session: h.Session, FinalTXID: txid, ManifestSeq: seq, Receipt: result.Key, ReceiptSHA256: result.Digest}, nil } // releaseDrain is the agent's drain for a release: step 1's action, its // observation, steps 2 and 3, the durable positions, and only then the // producers' own drain and acknowledgement. type releaseDrain struct { e *RelocationEffects obs hostclaim.Observed inner StopDrainer txid uint64 seq int64 done bool // resumed: this stop's positions are already durable (an earlier // attempt by this controller flushed and recorded them), so the // drain goes straight to the producers' own drain. resumed bool } // RecoveryReleaseDrain refuses: a restarted agent's producers are not the // stop's, and root must replay the durable receipt or nothing. The Collect // provider recognises it and submits the request as a recovery (6b). type RecoveryReleaseDrain struct{} func (RecoveryReleaseDrain) DrainClaim(context.Context, StopRequest, string) (DrainAcknowledgement, error) { return DrainAcknowledgement{}, fmt.Errorf("%w: the process that flushed this stop is gone; only its durable receipt can finish the release", ErrClaimStopIdentity) } func (d *releaseDrain) DrainClaim(ctx context.Context, r StopRequest, fence string) (DrainAcknowledgement, error) { if d.resumed { ack, err := d.inner.DrainClaim(ctx, r, fence) if err != nil { return DrainAcknowledgement{}, err } d.done = true return ack, nil } if err := d.e.StopWriters(ctx, d.obs); err != nil { return DrainAcknowledgement{}, err } in, err := d.e.Writers(ctx, d.obs) if err != nil { return DrainAcknowledgement{}, err } if _, err := d.e.Observe(ctx, in); err != nil { return DrainAcknowledgement{}, err } if d.txid, err = d.e.FlushDB(ctx, d.obs); err != nil { return DrainAcknowledgement{}, err } if d.seq, err = d.e.FlushFiles(ctx, d.obs); err != nil { return DrainAcknowledgement{}, err } if err := d.e.Positions.Record(ctx, r, d.txid, d.seq); err != nil { return DrainAcknowledgement{}, err } ack, err := d.inner.DrainClaim(ctx, r, fence) if err != nil { return DrainAcknowledgement{}, err } d.done = true return ack, nil } func (e *RelocationEffects) Retained() ([]hostrelocate.Participation, error) { if e.Controllers == nil { return nil, nil } return e.Controllers.Retained() } func (e *RelocationEffects) Finish(ctx context.Context, host, op string) error { if e.Controllers == nil { return ErrClaimStopUnavailable } return e.Controllers.Finish(ctx, host, op) } func (e *RelocationEffects) Build(ctx context.Context, obs hostclaim.Observed) error { if e.BuildP == nil { return ErrClaimStopUnavailable } return e.BuildP(ctx, obs) } func (e *RelocationEffects) InstallLocal(ctx context.Context, obs hostclaim.Observed) (hostclaim.LocalInstall, error) { if e.Install == nil { return hostclaim.LocalInstall{}, ErrClaimStopUnavailable } return e.Install(ctx, obs) } // EnsureLocal is hostrelocate.Effects' check before P's first admission: // the journaled receipt's install is verified and, if a lost lease's scoped // cleanup retired it, restored by the same Install provider under the same // install_stamp, then verified again. An opened P is never reinstalled. func (e *RelocationEffects) EnsureLocal(ctx context.Context, obs hostclaim.Observed) error { if e.VerifyLocal == nil || e.Install == nil || obs.Claim == nil { return ErrClaimStopUnavailable } c := obs.Claim if hostclaim.PreparedOpened(c) { return nil } var l hostclaim.LocalInstall if err := json.Unmarshal([]byte(c.Op.Journal["local_install"]), &l); err != nil || l.InstallStamp == 0 { return fmt.Errorf("%w: no local install receipt to verify", ErrClaimStopReceipt) } if ok, err := e.VerifyLocal(ctx, obs, l); err != nil || ok { return err } got, err := e.Install(ctx, obs) if err != nil { return err } if got.Box != l.Box || got.Enrollment != l.Enrollment || got.InstallStamp != l.InstallStamp { return fmt.Errorf("%w: the restored install is not the receipt's", ErrClaimStopReceipt) } ok, err := e.VerifyLocal(ctx, obs, l) if err == nil && !ok { err = fmt.Errorf("%w: the restored install did not verify", ErrClaimStopReceipt) } return err } // VerifyReceipt is hostrelocate.Effects' check before every admission // after P is opened: the box's durable receipt must be the one the claim // journaled, exactly. It reads the receipt alone: the files, env and // timers are the admitted writers' now, and nothing is restored. func (e *RelocationEffects) VerifyReceipt(ctx context.Context, obs hostclaim.Observed) (bool, error) { if e.Receipt == nil || obs.Claim == nil { return false, ErrClaimStopUnavailable } var want hostclaim.LocalInstall if err := json.Unmarshal([]byte(obs.Claim.Op.Journal["local_install"]), &want); err != nil || want.InstallStamp == 0 { return false, nil // no journaled receipt to match: nothing vouches for the local state } got, ok, err := e.Receipt(ctx, obs) if err != nil { return false, err } return ok && got == want, nil } func (e *RelocationEffects) Probe(ctx context.Context, obs hostclaim.Observed, coverage string) (uint64, bool, error) { if e.Admit == nil { return 0, false, ErrClaimStopUnavailable } return e.Admit(ctx, obs, coverage) } // execController is hostexec.Controller as a RelocationController: Clear // is hostexec.ClearStale with the controller's own config, the durable // record's removal once the claim no longer runs the operation with this // box (CC9). It is the Controller provider's production value (Task 8a // fills the assembly's hooks into cfg). type execController struct { *execctl.Controller cfg execctl.Config } func (c *execController) Clear(ctx context.Context) error { return execctl.ClearStale(ctx, c.cfg) } func (c *execController) Lost() <-chan struct{} { return c.Controller.Lost() } // OpenRelocationController opens hostexec's controller for cfg and wraps // it with its durable clear. func OpenRelocationController(ctx context.Context, cfg execctl.Config) (RelocationController, error) { c, err := execctl.Open(ctx, cfg) if err != nil { return nil, err } return &execController{Controller: c, cfg: cfg}, nil // a pointer: the retained set compares controllers by identity } ``` - [ ] **Step 5: Run the tests** Run: `nice -n 19 go test ./internal/host/ -run 'StopReceiptReleases|Relocation|ReleasePositions|ReleaseFlushFiles|ClaimStop|Finish|ARestartFinds|ASuccessor' -count=1 && nice -n 19 go test ./internal/hostfiles/ -count=1` (the lifecycle property, `TestRelocationControllerLifecycleProperties`, runs its 200 seeds in about three minutes niced; `-short` runs 20) Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/host/relocationreleases.go internal/host/relocationreleases_test.go internal/host/relocationeffects.go internal/host/relocationeffects_test.go internal/host/relocationcontrollers_linux_test.go internal/host/relocationlifecycle_linux_test.go internal/hostexec/readlocal.go internal/host/relocationflush.go internal/hostfiles/platform.go git commit -m "host: a release is proved by its full stop receipt; it stops the writers, flushes files, wake state and env, records its positions and only then drains Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 9a: Reconcile's row decision on main (stage A) The half of Task 9 that compiles on main: `Reconciler.check` decides a relocatable host's row with `RelocationRow` on a fresh read, the fence observer is consulted inside the already-locked operation (never reacquiring the host lock: round 4 finding 1), its errors keep rows and mint none (C6), and a held row whose box is not the executor, or whose admission fence is closed, takes the hold-only path under the flock (`WithLocal`, `HoldRow`) where the guarded path would refuse it (`ErrFenced`). 6b's adapter supplies `HoldRow` and the registry's conditional hold write (Task 9, stage B). With `FenceClosed` and `HoldRow` both nil (region HA off) the reconciler is today's to the letter: the first check is main's `Serving`, so an executor of any other operation is refused before the guard is activated or inventory recorded, and `Install` never takes the row decision or the hold-only path (rev 6, round 5 finding 3). **Files:** - Modify: `internal/hostbox/reconcile.go` - Test: `internal/hostbox/reconcile_relocation_test.go` **Interfaces:** - Consumes: Task 3's `RelocationRow`, `RelocationPolicy`, `hostguard.Guard.AdmissionFenced`, `Guard.WithLocal`. - Produces: `Reconciler.FenceClosed func(ctx, hostclaim.Observed) (bool, error)`, `Reconciler.HoldRow func(ctx, host, holdOp string) error`, `(Reconciler) relocationAware() bool`, `(Reconciler) rowDecision(ctx, obs) (RowDecision, error)`, `(Reconciler) holdOnly(ctx, selected, RowDecision) error`. - [ ] **Step 1: Write the failing tests** `internal/hostbox/reconcile_relocation_test.go`: ```go //go:build unix package hostbox import ( "context" "errors" "io/fs" "maps" "os" "path/filepath" "testing" "time" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostguard" "github.com/carlosframework/platform/internal/hostwrite" "github.com/carlosframework/platform/internal/park" ) // relocationReconciler is the reconciler over workFixture's guard and // store, with the real fence observer (Guard.AdmissionFenced) and a // recording hold writer. The claim relocates the host off "box". func relocationReconciler(t *testing.T) (Reconciler, RecordSelection, *[]string, hostguard.Guard) { t.Helper() source, store, obs := workFixture(t) source.Policy = Policy{Held: func(context.Context, string) (string, error) { return "", nil }, Relocation: RelocationPolicy("box")}.Check // One cold writer session, so the source has a session a stop can fence. obs, err := hostclaim.Update(context.Background(), store, obs, hostclaim.AdmitWriter(hostclaim.Admission{Box: "box", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) if err != nil { t.Fatal(err) } h := obs.Claim.Holder selected := RecordSelection{Host: obs.Host, Key: h.Key, ETag: "record-etag", Incarnation: h.Incarnation, Family: h.Family, Layout: "flat", Paths: []string{"shop.example.db"}} holds := &[]string{} r := Reconciler{Claims: store, Guard: source.Guard, Local: park.Local{Guard: source.Guard, Claims: store, DataDir: source.Guard.DataDir, Box: "box"}, Writes: hostwrite.Config{Enabled: true, Selector: source}, Box: "box", Region: "ie", Now: func() time.Time { return time.Unix(100, 0) }, CheckRecord: func(context.Context, RecordSelection) error { return nil }, CheckLocal: func(context.Context, hostclaim.Observed, RecordSelection) error { return nil }, HoldRow: func(_ context.Context, _ string, op string) error { *holds = append(*holds, op); return nil }} r.FenceClosed = func(_ context.Context, o hostclaim.Observed) (bool, error) { captured, err := hostguard.Capture(source.Guard.DataDir, o.Host) if err != nil { return false, err } return source.Guard.AdmissionFenced(o.Host, o.Claim, captured) } return r, selected, holds, source.Guard } func relocate(t *testing.T, r Reconciler, host, phase string) hostclaim.Observed { t.Helper() ctx := context.Background() obs, err := hostclaim.Read(ctx, r.Claims, host) if err != nil { t.Fatal(err) } obs, err = hostclaim.Update(ctx, r.Claims, obs, func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: phase, Exec: "box", Participants: []string{"box", "dst"}, Intent: map[string]string{"region": "ie", "source_box": "box", "target_box": "dst"}, Journal: map[string]string{"exec_ack.box": "{}", "exec_ack.dst": "{}"}} return nil }) if err != nil { t.Fatal(err) } return obs } // The source in releasing takes the activation path (ready) until root // begins its stop; then the hold-only path, under the flock, with the // observer read inside the existing locked operation: a bounded context // finishes both, so neither reacquires the host lock. func TestReconcileRelocationSourceReadyThenHeldWithTheRealFenceObserver(t *testing.T) { r, selected, holds, guard := relocationReconciler(t) obs := relocate(t, r, selected.Host, hostclaim.PhaseReleasing) ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() writes := 0 if err := r.Install(ctx, selected, func() error { writes++; return nil }); err != nil { t.Fatalf("the source before its fence closed: %v", err) } if writes != 1 || len(*holds) != 0 { t.Fatalf("writes %d holds %v", writes, *holds) } // Root begins the source's stop: the durable admission fence closes. // (The inventory write above moved the claim: read it fresh.) captured, err := hostguard.Capture(guard.DataDir, selected.Host) if err != nil { t.Fatal(err) } obs, err = hostclaim.Read(ctx, r.Claims, selected.Host) if err != nil { t.Fatal(err) } target := hostguard.FenceTarget{FenceID: "fence-1", Host: selected.Host, Incarnation: captured.Incarnation, Stamp: captured.Stamp, Session: obs.Claim.Holder.Session, Op: "rel-1", Phase: hostclaim.PhaseReleasing, Box: "box", Enrollment: "enrollment-1", BootID: "boot", ControllerID: "ctl", ClaimETag: obs.ETag, DirectoryRevision: "4"} if _, err := guard.BeginFence(ctx, target); err != nil { t.Fatal(err) } if err := r.Install(ctx, selected, func() error { writes++; return nil }); err != nil { t.Fatalf("the fenced source's hold: %v", err) } if writes != 1 || len(*holds) != 1 || (*holds)[0] != "rel-1" { t.Fatalf("after the fence: writes %d holds %v (the guarded path must not run)", writes, *holds) } // A box with no role mints nothing; an observer error keeps rows and // mints none. r.Box = "zzz" if err := r.Install(ctx, selected, func() error { writes++; return nil }); !errors.Is(err, ErrChanged) || writes != 1 { t.Fatalf("no role: %v writes %d", err, writes) } r.Box = "box" broken := errors.New("record unreadable") r.FenceClosed = func(context.Context, hostclaim.Observed) (bool, error) { return false, broken } if err := r.Install(ctx, selected, func() error { writes++; return nil }); !errors.Is(err, broken) || writes != 1 { t.Fatalf("an observer error: %v writes %d", err, writes) } } // With no observer (region HA off) the reconciler is today's: the serving // executor is ready; an executor of any other operation is refused by the // first check, before the guard is activated or inventory recorded, dry // run or not (round 5 finding 3). func TestReconcileWithoutAFenceObserverIsTodays(t *testing.T) { r, selected, _, guard := relocationReconciler(t) r.FenceClosed, r.HoldRow = nil, nil ctx := context.Background() writes := 0 if err := r.Install(ctx, selected, func() error { writes++; return nil }); err != nil || writes != 1 { t.Fatalf("serving: %v writes %d", err, writes) } moving := func() hostclaim.Observed { obs, err := hostclaim.Read(ctx, r.Claims, selected.Host) if err != nil { t.Fatal(err) } obs, err = hostclaim.Update(ctx, r.Claims, obs, func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindMoving, ID: "mv-1", Phase: "copying", Exec: "box", Participants: []string{"box", "dst"}, Intent: map[string]string{"admission_id": "adm"}} return nil }) if err != nil { t.Fatal(err) } return obs } for _, phase := range []string{hostclaim.PhaseStarting, hostclaim.PhaseReleasing, "moving"} { var obs hostclaim.Observed if phase == "moving" { obs = moving() // a held executor participant with HA on (RowHeld) } else { obs = relocate(t, r, selected.Host, phase) // ready with HA on (RowReady) } before := snapshotDir(t, guard.DataDir) for _, dry := range []bool{true, false} { sel := selected sel.DryRun = dry if err := r.Install(ctx, sel, func() error { writes++; return nil }); !errors.Is(err, ErrChanged) || writes != 1 { t.Fatalf("%s (dry run %v) with HA off: %v writes %d", phase, dry, err, writes) } } after, err := hostclaim.Read(ctx, r.Claims, selected.Host) if err != nil { t.Fatal(err) } if after.ETag != obs.ETag { t.Fatalf("%s: the claim (inventory) was written with HA off", phase) } if got := snapshotDir(t, guard.DataDir); !maps.Equal(got, before) { t.Fatalf("%s: the guard's local state changed with HA off", phase) } } } // snapshotDir is every regular file under dir, by relative path. func snapshotDir(t *testing.T, dir string) map[string]string { t.Helper() out := map[string]string{} err := filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error { if err != nil || !d.Type().IsRegular() { return err } raw, err := os.ReadFile(path) if err != nil { return err } out[path] = string(raw) return nil }) if err != nil { t.Fatal(err) } return out } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostbox/ -run 'ReconcileRelocation|ReconcileWithoutAFenceObserver' -count=1` Expected: FAIL to compile: `unknown field FenceClosed in struct literal`. - [ ] **Step 3: The decision, the observer and the hold-only path** Apply to `internal/hostbox/reconcile.go` (`git apply`): ```diff diff --git a/internal/hostbox/reconcile.go b/internal/hostbox/reconcile.go index 7cf5d4a5..cacefbc0 100644 --- a/internal/hostbox/reconcile.go +++ b/internal/hostbox/reconcile.go @@ -30,7 +30,38 @@ type Reconciler struct { CheckLocal func(context.Context, hostclaim.Observed, RecordSelection) error // Adopt is Plan 5's #232 backfill coordinator, never a second adoption rule. Adopt func(context.Context, RecordSelection) (hostclaim.Observed, error) + // FenceClosed reports whether this box's admission fence for the host + // is closed on the claim given (region HA §5.3: the source in + // releasing is ready until then). It is called under the host lock + // and must not take it (hostguard.Guard.AdmissionFenced). nil: never + // closed, today's behaviour. An error keeps rows and mints none (C6). + FenceClosed func(context.Context, hostclaim.Observed) (bool, error) + // HoldRow keeps an existing row held for an operation without + // installation authority (ruling 17); 6b's adapter supplies it. + HoldRow func(ctx context.Context, host, holdOp string) error } + +// relocationAware is region HA on: Task 10 sets FenceClosed and HoldRow +// only with CARLOS_REGION_HA. With neither, reconcile is today's: the +// serving executor alone (Serving), and no hold-only path. +func (r Reconciler) relocationAware() bool { return r.FenceClosed != nil || r.HoldRow != nil } + +func (r Reconciler) fenceClosed(ctx context.Context, obs hostclaim.Observed) (bool, error) { + if r.FenceClosed == nil { + return false, nil + } + return r.FenceClosed(ctx, obs) +} + +// rowDecision is §5.3's decision for this box on a fresh read. +func (r Reconciler) rowDecision(ctx context.Context, obs hostclaim.Observed) (RowDecision, error) { + closed, err := r.fenceClosed(ctx, obs) + if err != nil { + return RowDecision{}, err + } + return RelocationRow(obs.Claim, r.Box, true, closed), nil +} + type activationReader struct { r Reconciler selected RecordSelection @@ -57,8 +88,20 @@ func (r Reconciler) check(ctx context.Context, obs hostclaim.Observed, selected if obs.Host != selected.Host || h.Key != selected.Key || h.Incarnation != selected.Incarnation || h.Family != selected.Family || h.Stamp == 0 { return ErrChanged } - if err := Serving(obs.Claim, Selection{Box: r.Box}); err != nil { - return err + if !r.relocationAware() { + if err := Serving(obs.Claim, Selection{Box: r.Box}); err != nil { + return err + } + } else { + d, err := r.rowDecision(ctx, obs) + if err != nil { + return err + } + switch d.Case { + case RowReady, RowHeld: + default: + return ErrChanged + } } if until, ok := obs.Claim.QuietAfter[r.Region]; ok && !r.Now().After(until) { return ErrChanged @@ -100,6 +143,21 @@ func (r Reconciler) Install(ctx context.Context, selected RecordSelection, write if selected.DryRun { return nil } + if r.relocationAware() { + d, err := r.rowDecision(ctx, obs) + if err != nil { + return err + } + closed, _ := r.fenceClosed(ctx, obs) + if d.Case == RowHeld && (obs.Claim.Op.Exec != r.Box || closed) { + // No installation of ours to activate, or one whose admission + // fence is closed (the source in releasing, still op.exec, + // which With refuses): a kept row's hold only, under the + // flock. A target before installing has no row and mints none + // (ruling 17). + return r.holdOnly(ctx, selected, d) + } + } // Activate's participant callback is not an admission validator for executors. // The reader validates EVERY fresh read while Activate holds hostguard. guard := r.Guard @@ -131,3 +189,32 @@ func (r Reconciler) Install(ctx context.Context, selected RecordSelection, write } return work.With(ctx, hostwrite.Identity{}, write) } + +// holdOnly keeps an existing row held for the operation, with no +// installation authority: the write is conditional on the row's recorded +// ownership and its observed hold (6b's registry.SetClaimHold, through +// HoldRow), under the host's flock. The decision is remade on a fresh +// read under that lock. ErrChanged when there is no row: nothing is +// minted. +func (r Reconciler) holdOnly(ctx context.Context, selected RecordSelection, d RowDecision) error { + if r.HoldRow == nil { + return ErrUnsupported + } + return r.Guard.WithLocal(ctx, selected.Host, func() error { + fresh, err := hostclaim.Read(ctx, r.Claims, selected.Host) + if err != nil { + return err + } + if fresh.Absent || fresh.Claim.Holder == nil || fresh.Claim.Holder.Incarnation != selected.Incarnation { + return ErrChanged + } + again, err := r.rowDecision(ctx, fresh) + if err != nil { + return err + } + if again != d { + return ErrChanged + } + return r.HoldRow(ctx, selected.Host, d.HoldOp) + }) +} ``` - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/hostbox/ -count=1` Expected: PASS, main's reconcile tests included (no observer: today's behaviour). - [ ] **Step 5: Commit** ```bash git add internal/hostbox/reconcile.go internal/hostbox/reconcile_relocation_test.go git commit -m "hostbox: reconcile decides a relocation row on a fresh read, observes the fence inside its lock, and keeps a fenced source's row held without installation authority Co-Authored-By: Claude Opus 5.5 " ``` --- ## Stage B re-plan (ruling 24) When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c, 9 and 10 against their merged code, then re-review. Every obligation rounds 1 to 5 placed on stage B, with the request that covers it and the test that proves it: | Obligation | Round | Covered by | Proved by | |---|---|---|---| | Release providers bound to the real attempt: identity, producers, collect, writers-only stop, descriptors recorded at open | 1 (1, 8), 2 (N3) | 6b's `localStopIdentity`, `Wake.Producers`, `CollectClaimStop`; R2; R3 (agent half) | `TestRelocationStopWritersLeavesTheReplicatorOpen`, `TestRelocationEffectsAreBound` | | Every stage A provider a test stubs is tested with its production value once wired: the worker's `Choose` and `Eligible` (`ExecutorChooser` adapters), `Leases` (`*lease.Held`), `Prepared` (`PreparedFromInstallManifest`), `StorePinner.SelectDB` (`genselect.Select` over the versioned data bucket), and `RelocationEffects`' providers; replacing any with a stub that returns nothing must fail a test | 10 (scan) | Task 10's adapters; Tasks 8a to 8c | a test per provider over real evidence (genselect's versioned-bucket fixture for the pinner) | | A production probe launches only through the reservation-bearing isolated root launch protocol. It succeeds through it, and without it is refused at `Preflight` before any attempt is counted and before any admission. The passes stay off until the protocol exists. | 12 (2) | R10; Task 8c's gate; Task 10's activation gate | `TestRelocationProbeLaunchesThroughTheIsolatedRootProtocol` (the probe's `"probe"` work launched under the reservation, admitted and healthy, under the edge unit's restrictions); `TestRelocationProbeRefusesWithoutTheIsolatedRootProtocol` (`ErrClaimIsolationUnavailable`, no `BeginProbeAttempt`, no admission, no write); `TestRelocatePassStaysOffWithoutTheIsolatedRootProtocol` | | P's prepared files manifest is create-only at sequence 0, bound to `{location, P, op, install_stamp}`; the verifier (`PreparedFromInstallManifest`) reports `FilesSeq` 0, `SourceFilesSeq` the pin's, and the digest it read; the first admission takes that exact manifest, by digest, as the publisher's baseline, and the session's first publication is sequence 1 | 14 (1) | Task 5's `CopyPinnedManifest`, Task 1's `ToInstalled`; R1 (rev 15) | `TestPreparedFromInstallManifestReportsSequenceZero`, `TestRelocationProbeAdmissionConsumesThePreparedManifestByDigest` (a digest mismatch refuses the admission; the first commit is sequence 1) | | Before P's first admission, `EnsureLocal`'s production body: `VerifyLocal` checks the receipt's files (against P's prepared manifest), env file and wake state under its `install_stamp`; a retired or partial install is restored by the same `Install` under the same stamp, then verified; an opened P is never reinstalled | 14 (2) | Task 7's `EnsureLocal`; Task 8c's `VerifyLocal` and `Install` | `TestRelocationLocalInstallIsRestoredAfterALostLeaseBeforeTheProbe` (journal the receipt, lose the lease, scoped cleanup retires the install, recover, the probe sees a complete install); `TestRelocationOpenedPIsNeverReinstalled` | | A source's ordinary cold wake in `starting` and `releasing` dispatches through the wake inputs while its admission fence is open; refused once the fence closes or the claim no longer names it; a target's ordinary wake stays refused | 14 (3) | R1 (rev 15) | `TestRelocationSourceWakesInStarting`, `TestRelocationSourceWakesInReleasing`, `TestRelocationSourceWakeRefusedAfterItsFenceCloses`, `TestRelocationSourceWakeRefusedWithoutAuthority`, `TestRelocationTargetOrdinaryWakeStaysRefused` | | After P's first admission, `Receipt`'s production body reads this box's durable local install receipt (the install journal written under its `install_stamp`) and `VerifyReceipt` compares it with the journaled one; a missing or different receipt marks the operation `stuck` before any later admission, never a reinstall; legitimate post-admission changes to files, env and timers are never compared or reverted | 15 (1) | Task 6's participant, Task 7's `VerifyReceipt`; Task 8c's `Receipt` | `TestRelocationLostReceiptAfterPOpensIsStuckAcrossARestart` (open P, delete the install journal, restart the agent, the next probe attempt is refused and the claim is stuck); `TestRelocationPostAdmissionChangesSurviveTheReceiptCheck` | | Every cold admission writes its files manifest (a copy of the files pin) create-only before the app starts (§5.6), so a session that ran its app always leaves a manifest; the first manifest carrying a freshly minted `instance.key` is published by the continuous publisher (H6) | 17 (F2) | 6b's cold admission; R1 | `TestColdAdmissionWritesItsManifestBeforeTheAppStarts` (a store that refuses the manifest PUT: the app never starts) | | The producers' drain is idempotent: a second `DrainClaim` for the same stop and fence, after the producers have closed, acknowledges again with the same digest (Task 7's resumed drain relies on it after root fails between the drain and its receipt) | 7 (1) | 6b's `Wake.Producers` / `ClaimProducers.DrainClaim` | over the assembly: drain, root fails before `CompleteFence`, retry; one flush, two acknowledgements, a receipt | | The complete writers-stopped scan is root's, authenticated, over the control socket | 2 (N3) | R3 (root half), `CollectWritersStopped` | the root endpoint under the production uid | | `Region` on every stop request; `Recover` for the recorded owner after a restart | 2 (9), 3 (R1-9), 4 (R1-9) | 6b's `StopRequest.Region`, `Recover` | `…ReplaysOnlyRecordedPositions/after_a_restart…` against 6b's coordinator | | The relocation controller's cleanup scoped to its own partial work; `Clear` is `ClearStale` with its config | 3 (N5), 4 (5) | `relocationCleanup`, `OpenRelocationController` | `TestRelocationCleanupNeverStopsAnAdmittedWriter`, the on-disk eligibility test over the assembly | | Clearing rests on a cleanup that succeeded now, never on `Close`'s nil (hostexec returns a cleanup failure once): the assembly sets `RelocationControllers.Cleanup` to `relocationCleanup` bound to the journaled binding (it checks the binding's box and enrollment against the verified actor, touches only that operation's partial install of that host and incarnation, and is a no-op once this box admitted a writer under the operation or serves the host) and `Clear` to `ClearStale` with the assembly's config for the binding; `Journal` is `/relocate/participations`, written before every open, so a restart (or a crash before the first `Finish`, or a failed `Open`) finds every record | 5 (R4-5, 4) | Task 7's `Finish`, `record`, `journaled`; R8's `applyRegionHA` | over the assembly: cleanup fails, a sweep, a restart with it still failing (the record stays and `hostexecParticipation` refuses), then success (cleared); a crash between `record` and `Open`, and between `Open` and `Finish`, each cleared after the restart; a truncated `.tmp` beside a valid entry | | The local install restores the env file from P's manifest exactly: `ReadPlatform(EnvItem)` returning `os.ErrNotExist` means the host has no env file, and any env file at the destination is removed under the install's authority, never kept | 5 (5) | Task 7's `StagePlatformDelete` on `ENOENT`; 6b's typed inventory | `…RestoresNoEnvFileWhenTheReleaseRemovedIt` | | P built under the location lease with authority re-checked at every write; files manifest first; the verifier reads both manifests, the pinned one from the retained copy | 1 (2, 5, 7), 3 (R1-2), 4 (R1-5) | `authorityStore`, `CopyPinnedManifest`, `PreparedFromInstallManifest`; R7 | `TestRelocationBuildStopsWhenAuthorityChanges`, `TestPreparedRefusesAPartialBuild` | | The local install under `install_stamp`; destination directories created and inventoried first; held staging through root's relay; never remade after P's admission | 1 (1), 4 (3, 4) | 6b's typed inventory; R9; moves Plan 1 Task 7b | `TestRelocationInstallIsBoundToTheInstallStamp`, `…CreatesAndInventoriesTheDestination`, `…WaitsForRootsHeldStaging` | | The probe: inputs open P once with `Prepared`, later attempts ordinary (empty included), selected as `"probe"` work, one attempt per `BeginProbeAttempt` | 1 (1), 2 (N1, N6), 3 (R1-13), 4 (2) | R1 (dispatch, `Probe`, `"probe"` action) | `TestRelocationWakeInputsOpenPOnceThenRestoreThePredecessor`, `TestASecondProbeAttemptPublishesNoMarkerBeforeItsRoot` | | Reconcile through 6b's adapter: hold writes conditional on ownership, the completed hold cleared, nothing minted for a box with no role, an unexplained row kept | 1 (14), 2 (N2, N4), 3 (N1, N4), 4 (N1) | `holdRowFor`, the adapter's hold switch; Task 9a's decision | `TestReconcileAdapterInstallFollowsTheRowCases`, `…HoldIsConditional`, `…ClearsTheHoldAfterCompletion` | | Production wiring: the policy and the fence observer bound to the verified actor, flag-gated; the passes with the sweep; create's chooser only with HA on | 2 (15), 3 (N3), 4 (N3) | R8 (`Extend`), `applyRegionHA`, `runRelocatePass` | `TestApplyRegionHASetsTheProvidersOnlyWithTheFlagOn`, `TestRelocatePassIsOffUnlessBothFlagsAreOn` | Nothing in stage A changes for stage B's rewrite: the contracts stage B consumes (`hostrelocate.Effects`, `Prepared`, `Pinner`, `Lease`, `hostclaim`'s transitions, `RelocationRow`, `RelocationPolicy`, `AdmissionFenced`, `ReleasePositions`, `RelocationControllers`) are stage A's and tested there. --- ### Task 8a: The release providers on the claim assembly **Gated on:** claim Plan 6b (`docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md` on `origin/design-replica-ownership`, rev 5) merged to main, for `ClaimAssembly` (its `Wake *ClaimWakeAssembly`, `Guard`, `Claims`, `Reconcile`), `(*ClaimWakeAssembly).Producers`, `(*ClaimWakeAssembly).get(host) *claimWakeAttempt` and its `rep *lease.Replicator`, `(ClaimAssembly).localStopIdentity` returning `claimStopLocal{Actor, BootID, ControllerID, PID, ProcessStart}`, and `CollectClaimStop(ctx, path, StopRequest, StopDrainer)`; and on requests **R2** (a writers-only stop on the attempt) and **R3** (the replicator's descriptors recorded at open), which 6b does not declare. Not compiled in the dry run. Each provider of Task 7's `RelocationEffects` gets its production body. Nothing here admits a writer, spawns, or touches the registry outside the assembly's guarded paths. **Files:** - Create: `internal/host/relocationassembly.go` - Test: `internal/host/relocationassembly_test.go` **Interfaces:** - Consumes (6b rev 5, by name): `ClaimAssembly`, `(*ClaimWakeAssembly).Producers(ctx, hostclaim.Observed) (*ClaimProducers, error)`, `(*ClaimWakeAssembly).get(host)`, `(*claimWakeAttempt).rep`, `(ClaimAssembly).localStopIdentity(ctx) (claimStopLocal, error)`, `CollectClaimStop`, `controlsock.Path`; R2's `(*claimWakeAttempt).StopWriters(ctx) (drained ClaimWritersDrained, err error)` with `ClaimWritersDrained{FenceClosed, SpawnsSettled, WorkDrained bool; Cgroup stopevidence cgroup; DBPaths []string}`; R3's `(*claimWakeAttempt).replicatorFDs []stopevidence.Descriptor`. Main: `hostexec.Open`, `genselect.Select`, `stopevidence.WritersInputs`; Task 7's `ReleaseFlushFiles`, `ReleasePositionsFile`, `RelocationControllers`. - Produces: `RelocationAssemblyConfig`, `(ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig) *RelocationEffects`. - [ ] **Step 1: Write the failing tests** With 6b's seams (`stopLocal`, `collectStop`) and a `release.NewDirStore`: ```go package host import ( "context" "errors" "testing" "github.com/carlosframework/platform/internal/hostclaim" ) // The release provider refuses a host whose admitted attempt has no // replicator: there is nothing to flush, and N must not be guessed. func TestRelocationFlushRefusesWithoutTheAdmittedReplicator(t *testing.T) { a := ClaimAssembly{Wake: &ClaimWakeAssembly{}} _, err := a.relocationFlushDB(RelocationAssemblyConfig{})(context.Background(), hostclaim.Observed{Host: "shop.example", Claim: &hostclaim.Claim{Host: "shop.example", Holder: &hostclaim.Holder{Session: 3}}}) if !errors.Is(err, ErrClaimStopUnavailable) { t.Fatalf("got %v", err) } } // Every provider is bound, under the agent's lifetime, not a pass's. func TestRelocationEffectsAreBoundToTheAssembly(t *testing.T) { a := ClaimAssembly{Wake: &ClaimWakeAssembly{}} e := a.RelocationEffects(RelocationAssemblyConfig{Lifetime: context.Background(), ReleaseDir: t.TempDir()}) if e.Lifetime == nil || e.Controllers == nil || e.Identity == nil || e.Controller == nil || e.Producers == nil || e.Collect == nil || e.StopWriters == nil || e.Writers == nil || e.Observe == nil || e.FlushDB == nil || e.FlushFiles == nil || e.Positions == nil || e.BuildP == nil || e.Install == nil || e.Admit == nil { t.Fatalf("an unbound provider: %+v", e) } } ``` A third test, written once R2 lands: `TestRelocationStopWritersLeavesTheReplicatorOpen` starts a real attempt with 6b's test launcher (a process in the host's cgroup and a timer delivery in flight through `schedule`), calls `StopWriters`, and checks the cgroup is empty, the delivery drained, and `rep` still open (its `Lost()` not closed); then root's `writers-stopped` endpoint (R3) with the attempt's recorded descriptors passes, run as the production uid against a root-owned process holding another file in `/proc`. It replaces the pre-set booleans Task 7's tests use. A fourth, `TestRelocationCleanupNeverStopsAnAdmittedWriter`, closes the relocation controller after the probe's admission and after serving and checks the wake assembly's session is untouched (its producers still registered, `rep` open), and that before any admission it retires the partial local install. - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/host/ -run 'RelocationFlush|RelocationEffectsAreBound' -count=1` Expected: FAIL to compile: `undefined: RelocationAssemblyConfig`. - [ ] **Step 3: The bindings** `internal/host/relocationassembly.go`: ```go package host import ( "bytes" "context" "encoding/json" "errors" "fmt" "io" "net" "net/http" "os" "path/filepath" "time" "github.com/carlosframework/platform/internal/controlsock" "github.com/carlosframework/platform/internal/genselect" "github.com/carlosframework/platform/internal/hostclaim" execctl "github.com/carlosframework/platform/internal/hostexec" "github.com/carlosframework/platform/internal/lease" "github.com/carlosframework/platform/internal/registry" "github.com/carlosframework/platform/internal/release" "github.com/carlosframework/platform/internal/stopevidence" ) // RelocationAssemblyConfig is what the relocation providers need beyond // the assembly itself. type RelocationAssemblyConfig struct { Lifetime context.Context // the agent's context: controllers live under it Open func(context.Context, execctl.Config) (*execctl.Controller, error) // execctl.Open Exec execctl.Config // the assembly's hooks; Binding is filled per host Data release.Store // the region's data bucket (files manifests) Versions lease.VersionedStore // the same bucket, versioned (LTX files) Bucket string // its name Region string // the lease region for lease.Acquire Registry *registry.Registry FilesDir func(host string) string // the host's files directory on this box EnvPath func(host string) string // the host's env file on this box DataDir string // the guard's data dir (hostguard.Guard.DataDir) WorkDir string // scratch for the build (/relocate/build) ReleaseDir string // ReleasePositionsFile.Dir (/relocate/release) Now func() time.Time } // RelocationEffects binds every provider to this assembly. One value per // agent: its Controllers outlive every pass. func (a ClaimAssembly) RelocationEffects(cfg RelocationAssemblyConfig) *RelocationEffects { if cfg.Now == nil { cfg.Now = time.Now } return &RelocationEffects{ Lifetime: cfg.Lifetime, // Rev 6 (round 5, R4-5): the journal is written before every open, // and Finish clears only straight after Cleanup succeeds. The // re-plan authenticates the binding against the verified actor // before relocationCleanup runs (stage B re-plan table). Controllers: &RelocationControllers{Journal: filepath.Join(cfg.DataDir, "relocate", "participations"), DataDir: cfg.Exec.Guard.DataDir, Cleanup: func(ctx context.Context, b execctl.Binding) error { return a.relocationCleanup(ctx, cfg, b.Host, b.Op) }, Clear: func(ctx context.Context, b execctl.Binding) error { c := cfg.Exec; c.Binding = b; return execctl.ClearStale(ctx, c) }}, Identity: func(ctx context.Context) (RelocationIdentity, error) { l, err := a.localStopIdentity(ctx) // 6b: the verified actor, boot id and controller id if err != nil { return RelocationIdentity{}, err } return RelocationIdentity{Actor: l.Actor, BootID: l.BootID, ControllerID: l.ControllerID}, nil }, Controller: func(ctx context.Context, b execctl.Binding) (RelocationController, error) { c := cfg.Exec c.Binding = b // Ruling 20: this controller's cleanup is the relocation's own. c.Cleanup = func(ctx context.Context) error { return a.relocationCleanup(ctx, cfg, b.Host, b.Op) } return OpenRelocationController(ctx, c) // Clear is ClearStale with this config (round 4 finding 5) }, Producers: func(ctx context.Context, o hostclaim.Observed) (StopDrainer, error) { return a.Wake.Producers(ctx, o) }, Collect: func(ctx context.Context, r StopRequest, d StopDrainer) (StopResult, error) { // Region is 6b's field and part of its fence identity: set on every // request from the verified actor (a box never changes region), so // the initial request and a recovery name the same stop. local, err := a.localStopIdentity(ctx) if err != nil { return StopResult{}, err } r.Region = local.Actor.Region if _, ok := d.(RecoveryReleaseDrain); ok { // A restarted agent finishing the recorded owner's stop: 6b's // authenticated recovery path accepts another tuple for this // box's session and replays the durable receipt (R1-9). r.Recover = true } return CollectClaimStop(ctx, controlsock.Path, r, d) }, StopWriters: a.relocationStopWriters(), Writers: a.relocationWriters(), Observe: func(ctx context.Context, in stopevidence.WritersInputs) (stopevidence.WritersEvidence, error) { return CollectWritersStopped(ctx, controlsock.Path, in) // root's complete scan (R3, ruling 22) }, FlushDB: a.relocationFlushDB(cfg), FlushFiles: ReleaseFlushFiles{Data: cfg.Data, Registry: cfg.Registry, FilesDir: cfg.FilesDir, EnvPath: cfg.EnvPath}.Flush, Positions: ReleasePositionsFile{Dir: cfg.ReleaseDir}, BuildP: a.relocationBuild(cfg), // Task 8b Install: a.relocationInstall(cfg), // Task 8c Admit: a.relocationAdmit(cfg), // Task 8c } } // relocationStopWriters is release step 1's action (R2): the attempt // closes admission, settles its spawns, drains the host's in-flight timer // deliveries, stops the host's process tree and waits for it to empty, // and leaves its replicator and files publisher open. func (a ClaimAssembly) relocationStopWriters() func(context.Context, hostclaim.Observed) error { return func(ctx context.Context, o hostclaim.Observed) error { att := a.Wake.get(o.Host) if att == nil { return ErrClaimStopUnavailable } _, err := att.StopWriters(ctx) // R2 return err } } // relocationWriters reports what StopWriters left: the three drained facts // and the host's cgroup from the attempt, and the replicator's descriptors // as recorded when it opened (R3: stopevidence.ProcessDescriptors before // and after held.Replicate, the difference kept on the attempt), bound to // this process's start time. func (a ClaimAssembly) relocationWriters() func(context.Context, hostclaim.Observed) (stopevidence.WritersInputs, error) { return func(ctx context.Context, o hostclaim.Observed) (stopevidence.WritersInputs, error) { att := a.Wake.get(o.Host) if att == nil { return stopevidence.WritersInputs{}, ErrClaimStopUnavailable } drained, err := att.StopWriters(ctx) // idempotent once stopped: reports the facts if err != nil { return stopevidence.WritersInputs{}, err } local, err := a.localStopIdentity(ctx) if err != nil { return stopevidence.WritersInputs{}, err } if len(att.replicatorFDs) == 0 { return stopevidence.WritersInputs{}, fmt.Errorf("%w: the replicator's descriptors were not recorded at open", ErrClaimStopUnavailable) } return stopevidence.WritersInputs{Host: o.Host, FenceClosed: drained.FenceClosed, SpawnsSettled: drained.SpawnsSettled, WorkDrained: drained.WorkDrained, Cgroup: drained.Cgroup, DBPaths: drained.DBPaths, ReplicatorPID: local.PID, ReplicatorStart: local.ProcessStart, Replicator: att.replicatorFDs}, nil } } // relocationFlushDB is release step 2 (ruling 8): SyncAndWait with every // writer stopped, then N is the newest level-0 MaxTXID, which must equal // the plan's MaxTXID. A sync error is returned, never swallowed. A listing // with no LTX file after the sync is the stopped writer vouching that it // never committed (claim §6.4a "Empty bootstrap"): N is 0, and the release // at final_txid 0 is what lets the pin be empty (ruling 15). func (a ClaimAssembly) relocationFlushDB(cfg RelocationAssemblyConfig) func(context.Context, hostclaim.Observed) (uint64, error) { return func(ctx context.Context, o hostclaim.Observed) (uint64, error) { att := a.Wake.get(o.Host) if att == nil || att.rep == nil || o.Claim == nil || o.Claim.Holder == nil { return 0, ErrClaimStopUnavailable } if err := att.rep.SyncAndWait(ctx); err != nil { return 0, err } plan, err := genselect.Select(ctx, cfg.Versions, genselect.Request{Bucket: cfg.Bucket, Region: o.Claim.Op.Intent["region"], Host: o.Host, Holder: *o.Claim.Holder}) if errors.Is(err, genselect.ErrEmpty) { return 0, nil } if err != nil { return 0, err } if plan.MaxLevel0TXID != plan.MaxTXID { return 0, fmt.Errorf("%w: level 0 at %d, the database at %d", ErrClaimStopUnavailable, plan.MaxLevel0TXID, plan.MaxTXID) } return plan.MaxTXID, nil } } // relocationCleanup is the relocation controller's Cleanup hook (hostexec // runs it when the controller is lost or closed): it retires only this // operation's partial local installation, the unopened P's files, env and // wake import made under install_stamp, through the assembly's // retire-artifacts path; and it does nothing once this box has admitted a // writer under the operation or serves the host (the wake assembly's own // controller and producers own that session, ruling 20). func (a ClaimAssembly) relocationCleanup(ctx context.Context, cfg RelocationAssemblyConfig, host, op string) error { obs, err := hostclaim.Read(ctx, a.Claims, host) if err != nil { return err } local, err := a.localStopIdentity(ctx) if err != nil { return err } c := obs.Claim if obs.Absent || c.Holder == nil { return nil } if c.Holder.WriterBox == local.Actor.Box || (c.Op.Kind == hostclaim.KindServing && c.Op.Exec == local.Actor.Box) { return nil // a writer this box admitted: not the relocation's to stop } for _, adm := range c.Holder.Admissions { if adm.Box == local.Actor.Box && c.Op.ID == op { return nil } } if c.Op.Kind == hostclaim.KindRelocating && c.Op.ID == op && c.Op.Exec == local.Actor.Box && c.Op.Journal["local_install"] != "" { return a.Wake.RetireArtifacts(ctx, obs, c.Op.Journal["prepared"]) // 6b's retire-artifacts under the host guard } return nil } // CollectWritersStopped asks root for L10's observation over the // instance-control socket (R3's root half): root runs // stopevidence.WritersStopped with its cgroup manager and the complete // /proc scan for the request's host, and answers with the evidence. The // request carries everything but the cgroup, which root supplies. It // mirrors CollectClaimStop's transport. func CollectWritersStopped(ctx context.Context, path string, in stopevidence.WritersInputs) (stopevidence.WritersEvidence, error) { req := WritersStoppedRequest{Host: in.Host, FenceClosed: in.FenceClosed, SpawnsSettled: in.SpawnsSettled, WorkDrained: in.WorkDrained, DBPaths: in.DBPaths, ReplicatorPID: in.ReplicatorPID, ReplicatorStart: in.ReplicatorStart, Replicator: in.Replicator} ctx, cancel := context.WithTimeout(ctx, 2*time.Minute) defer cancel() body, err := json.Marshal(req) if err != nil { return stopevidence.WritersEvidence{}, err } transport := &http.Transport{DisableKeepAlives: true, DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { return (&net.Dialer{}).DialContext(ctx, "unix", path) }} defer transport.CloseIdleConnections() request, err := http.NewRequestWithContext(ctx, http.MethodPost, "http://control/claim/writers-stopped", bytes.NewReader(body)) if err != nil { return stopevidence.WritersEvidence{}, err } response, err := (&http.Client{Transport: transport}).Do(request) if err != nil { return stopevidence.WritersEvidence{}, err } defer response.Body.Close() if response.StatusCode != http.StatusOK { return stopevidence.WritersEvidence{}, fmt.Errorf("%w: writers-stopped: %s", ErrClaimStopUnavailable, response.Status) } var ev stopevidence.WritersEvidence dec := json.NewDecoder(io.LimitReader(response.Body, 1<<20)) dec.DisallowUnknownFields() if err := dec.Decode(&ev); err != nil { return stopevidence.WritersEvidence{}, err } if ev.Host != in.Host || ev.CheckedAt.IsZero() { return stopevidence.WritersEvidence{}, fmt.Errorf("%w: writers-stopped evidence is not for %s", ErrClaimStopUnavailable, in.Host) } return ev, nil } // WritersStoppedRequest is the wire form of stopevidence.WritersInputs // without the cgroup, which root supplies from its own manager. type WritersStoppedRequest struct { Host string `json:"host"` FenceClosed bool `json:"fence_closed"` SpawnsSettled bool `json:"spawns_settled"` WorkDrained bool `json:"work_drained"` DBPaths []string `json:"db_paths"` ReplicatorPID int `json:"replicator_pid"` ReplicatorStart string `json:"replicator_start"` Replicator []stopevidence.Descriptor `json:"replicator"` } // relocationJournalDir is where a box journals its own relocation work for // one host and operation. func relocationJournalDir(cfg RelocationAssemblyConfig, host, op string) string { return filepath.Join(cfg.DataDir, "relocate", host, op) } func ensureDir(path string) error { return os.MkdirAll(path, 0o700) } ``` Bind when 6b lands: `localStopIdentity` is 6b's (`claimstop_assembly.go`, with the `stopLocal` seam); `a.Wake.get` and `att.rep` are 6b's `ClaimWakeAssembly`; `StopWriters` and `replicatorFDs` are requests R2 and R3 (below), and this task does not merge without them. - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/host/ -run 'RelocationFlush|RelocationEffectsAreBound|RelocationStopWriters' -count=1` Expected: PASS. - [ ] **Step 5: Commit** ```bash git add internal/host/relocationassembly.go internal/host/relocationassembly_test.go git commit -m "host: the relocation release runs on the claim assembly's attempt: stop writers, observe, flush, record, then drain Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 8b: Build P and verify it **Gated on:** moves Phase 1 Plan 2 Task 8 (`docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md` on `origin/design-moves-any-region`) merged, for `moves.BuildGeneration(ctx, moves.BuildInputs) (moves.InstallManifest, error)`, `moves.BuildInputs{Store lease.VersionedStore; Bucket string; Source []moves.SealedObject; N uint64; GenerationPrefix, Op, Host, WorkDir string; Now func() time.Time; Verify func(string) error}`, `moves.SealedObject{Key, VersionID string; Size int64}`, `moves.InstallManifestKey(prefix)`, `moves.SnapshotKey`, `moves.AnchorKey`; and on Task 8a. `moves.InstallManifest` is on main (`internal/moves/install_contract.go`). Not compiled in the dry run. The build is §5.8's remote build under the replica lease on `holder.location`, with fresh phase authority checked before each durable step and the lease's loss cancelling the rest: P's files manifest first (`hostrelocate.CopyPinnedManifest`, which enforces the pin's digest, sequence and origins), then the database build from the pinned versions at the pin's `MaxTXID`. The verifier reads both back and refuses a P missing either. **Files:** - Create: `internal/host/relocationbuild.go` - Test: `internal/host/relocationbuild_test.go` **Interfaces:** - Consumes: Task 5's `hostrelocate.ReadPin`, `CopyPinnedManifest`; `lease.Acquire`, `lease.CheckDB`; moves Plan 2's names above; `genselect.File`. - Produces: `(ClaimAssembly) relocationBuild(cfg)`, `host.PreparedFromInstallManifest{Claims, Files release.Store; Data lease.VersionedStore; Bucket string}` implementing `hostrelocate.Prepared`. - [ ] **Step 1: Write the failing tests** ```go package host import ( "context" "errors" "testing" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostrelocate" ) // Prepared is done only when P's files manifest and (for a non-empty pin) // its install manifest both verify. An unconfigured verifier reports // nothing built. func TestPreparedFromInstallManifestNeedsBothManifests(t *testing.T) { p := PreparedFromInstallManifest{} if _, done, err := p.Verify(context.Background(), hostclaim.Observed{Claim: &hostclaim.Claim{Op: hostclaim.Op{Journal: map[string]string{"prepared": "P1"}}, Holder: &hostclaim.Holder{Location: "instances/x/i"}}}); done || err == nil { t.Fatalf("an unconfigured verifier reported P built: %v", err) } } // A crash after the files manifest and before the install manifest, and // the reverse, each leave P unverified. func TestPreparedRefusesAPartialBuild(t *testing.T) { // Fixture: a dir store with a pinned manifest copied to P (files only) // and a versioned fake with no install manifest; then the reverse, // an install manifest at N with no files manifest at P. for _, tc := range []struct{ name, present string }{{"files manifest only", "files"}, {"install manifest only", "db"}} { t.Run(tc.name, func(t *testing.T) { f := buildFixture(t, tc.present) if _, done, err := f.verifier.Verify(context.Background(), f.obs); done || err != nil { t.Fatalf("done=%v err=%v", done, err) } }) } } // A build whose phase authority moved under it stops before the next // durable write, wherever it is: between the two steps, and between the // builder's own snapshot and anchor PUTs (the lease still held, the claim // replaced the target, or the guard record retired the stamp). func TestRelocationBuildStopsWhenAuthorityChanges(t *testing.T) { for _, tc := range []struct{ name, after string }{{"after the files manifest", "files"}, {"after the snapshot PUT", "snapshot"}, {"after the stamp is retired", "stamp"}} { t.Run(tc.name, func(t *testing.T) { f := buildFixture(t, "") f.moveClaimAfter(tc.after) // ReplaceTarget to another box, or Guard.RetireInstallation, at that point if err := f.build(context.Background(), f.obs); !errors.Is(err, hostrelocate.ErrRefused) { t.Fatalf("the build went on after its authority changed: %v", err) } if f.installManifestExists() { t.Fatal("the install manifest was written for a box that is no longer op.exec") } }) } } ``` `buildFixture` is this task's: a `release.NewDirStore` for files, an in-memory `lease.VersionedStore` (moves Plan 2's test fake, or `leasetest`'s), a claim at `installing` with a journaled pin, and hooks after each step. - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/host/ -run 'PreparedFromInstallManifest|PreparedRefuses|RelocationBuild' -count=1` Expected: FAIL to compile: `undefined: PreparedFromInstallManifest`. - [ ] **Step 3: The build and the verifier** `internal/host/relocationbuild.go`: ```go package host import ( "context" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "fmt" "os" "path/filepath" "strconv" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostfiles" "github.com/carlosframework/platform/internal/hostguard" "github.com/carlosframework/platform/internal/hostrelocate" "github.com/carlosframework/platform/internal/lease" "github.com/carlosframework/platform/internal/moves" "github.com/carlosframework/platform/internal/release" ) // relocationBuild builds P from the journaled pin (§5.8 "Building the // prepared generation", L16, L17): under the replica lease on // holder.location, with the claim re-read before each durable step and // required to still be this operation at installing with this box as // op.exec and the same pin; the lease's loss cancels what follows. P's // files manifest is written first, so a crash between the two steps // leaves an install manifest never ahead of its files manifest. func (a ClaimAssembly) relocationBuild(cfg RelocationAssemblyConfig) func(context.Context, hostclaim.Observed) error { return func(ctx context.Context, o hostclaim.Observed) error { if cfg.Versions == nil || cfg.Data == nil || cfg.Bucket == "" || o.Claim == nil || o.Claim.Holder == nil { return ErrClaimStopUnavailable } c := o.Claim var ref hostclaim.RecoveryPin if err := json.Unmarshal([]byte(c.Op.Journal["pin"]), &ref); err != nil { return err } pin, err := hostrelocate.ReadPin(ctx, a.Claims, ref) if err != nil { return err } local, err := a.localStopIdentity(ctx) if err != nil { return err } p := c.Op.Journal["prepared"] stamp := c.Op.Journal["install_stamp"] held, err := lease.Acquire(ctx, cfg.Bucket, c.Holder.Location, cfg.Region) if err != nil { return err } defer func() { _ = held.Release(ctx) }() alive, stop := context.WithCancel(ctx) defer stop() go func() { select { case <-held.Lost(): stop() case <-alive.Done(): } }() // authority re-reads the claim and the host-guard record: the same // operation, phase, executor, stamp and pin, and this box's // activated installation still at install_stamp, or nothing more // is written. It runs before every durable step, including every // write the builder makes (authorityStore below). authority := func() error { fresh, err := hostclaim.Read(alive, a.Claims, o.Host) if err != nil { return err } if fresh.Absent || fresh.Claim.Op.ID != c.Op.ID || fresh.Claim.Op.Phase != hostclaim.PhaseInstalling || fresh.Claim.Op.Exec != local.Actor.Box || fresh.Claim.Op.Journal["prepared"] != p || fresh.Claim.Op.Journal["install_stamp"] != stamp || fresh.Claim.Op.Journal["pin"] != c.Op.Journal["pin"] || fresh.Claim.Op.Journal["stuck"] != "" { return fmt.Errorf("%w: the build's authority changed", hostrelocate.ErrRefused) } captured, err := hostguard.Capture(a.Guard.DataDir, o.Host) if err != nil { return err } if captured.Incarnation != c.Holder.Incarnation || strconv.FormatUint(captured.Stamp, 10) != stamp { return fmt.Errorf("%w: this box's installation is not the one the installing CAS issued", hostrelocate.ErrRefused) } return alive.Err() } prefix := c.Holder.Location + "/" + p + "/" if err := authority(); err != nil { return err } if _, err := hostrelocate.CopyPinnedManifest(alive, a.Claims, cfg.Data, pin.Files, c.Holder.Location+"/files/"+p+"/"); err != nil { return err } if pin.DB == nil { return nil // an evidenced empty bootstrap: no snapshot, no anchor (claim P6-D3 forbids [1, 0]) } if err := authority(); err != nil { return err } src := make([]moves.SealedObject, 0, len(pin.DB.Files)) for _, f := range pin.DB.Files { src = append(src, moves.SealedObject{Key: f.Key, VersionID: f.VersionID, Size: f.Size}) } work := filepath.Join(cfg.WorkDir, c.Op.ID) if err := os.MkdirAll(work, 0o700); err != nil { return err } _, err = moves.BuildGeneration(alive, moves.BuildInputs{Store: authorityStore{VersionedStore: cfg.Versions, check: authority}, Bucket: cfg.Bucket, Source: src, N: pin.DB.MaxTXID, GenerationPrefix: prefix, Op: c.Op.ID, Host: o.Host, WorkDir: work, Now: cfg.Now, Verify: lease.CheckDB}) return err } } // authorityStore is the versioned store the builder writes through: every // write re-checks the build's authority first (R1-2), so a delayed build // whose phase, executor, stamp or lease moved writes nothing more. Reads // are the store's own. type authorityStore struct { lease.VersionedStore check func() error } func (s authorityStore) PutIfAbsent(ctx context.Context, bucket, key string, body []byte) (string, error) { if err := s.check(); err != nil { return "", err } return s.VersionedStore.PutIfAbsent(ctx, bucket, key, body) } func (s authorityStore) CopyVersion(ctx context.Context, sb, sk, sv, db, dk string, createOnly bool) (string, string, error) { if err := s.check(); err != nil { return "", "", err } return s.VersionedStore.CopyVersion(ctx, sb, sk, sv, db, dk, createOnly) } func (s authorityStore) DeleteVersion(context.Context, string, string, string) error { return fmt.Errorf("%w: the build never deletes", hostrelocate.ErrRefused) } // PreparedFromInstallManifest is hostrelocate.Prepared: P is built when a // fresh read verifies its files manifest at the pinned sequence with the // pinned entries and, unless the pin is empty, its completed install // manifest at the pin's N whose snapshot and anchor versions exist. type PreparedFromInstallManifest struct { Claims release.Store // pin objects Files release.Store // files manifests Data lease.VersionedStore // LTX files and install manifests Bucket string } func (v PreparedFromInstallManifest) Verify(ctx context.Context, o hostclaim.Observed) (hostclaim.PreparedAuthority, bool, error) { if v.Claims == nil || v.Files == nil || v.Data == nil || v.Bucket == "" || o.Claim == nil || o.Claim.Holder == nil { return hostclaim.PreparedAuthority{}, false, ErrClaimStopUnavailable } c := o.Claim var ref hostclaim.RecoveryPin if err := json.Unmarshal([]byte(c.Op.Journal["pin"]), &ref); err != nil { return hostclaim.PreparedAuthority{}, false, err } pin, err := hostrelocate.ReadPin(ctx, v.Claims, ref) if err != nil { return hostclaim.PreparedAuthority{}, false, err } p := c.Op.Journal["prepared"] filesPrefix := c.Holder.Location + "/files/" + p + "/" raw, _, err := v.Files.Get(ctx, hostfiles.ManifestKey(filesPrefix)) if errors.Is(err, os.ErrNotExist) { return hostclaim.PreparedAuthority{}, false, nil // not built yet } if err != nil { return hostclaim.PreparedAuthority{}, false, err } var m hostfiles.Manifest if err := json.Unmarshal(raw, &m); err != nil || m.V != hostfiles.ManifestVersion || m.Seq != pin.Files.Seq { return hostclaim.PreparedAuthority{}, false, fmt.Errorf("%w: P's files manifest is not the pinned revision", hostrelocate.ErrRefused) } if pin.Files.Retained != "" { // The pinner's retained copy of the pinned bytes, never the source // prefix (ruling 21): P's entries must be exactly theirs. pinnedRaw, _, err := v.Claims.Get(ctx, pin.Files.Retained) if err != nil { return hostclaim.PreparedAuthority{}, false, err } if sum := sha256.Sum256(pinnedRaw); hex.EncodeToString(sum[:]) != pin.Files.SHA256 { return hostclaim.PreparedAuthority{}, false, fmt.Errorf("%w: the pinned files manifest changed", hostrelocate.ErrRefused) } var pinned hostfiles.Manifest if err := json.Unmarshal(pinnedRaw, &pinned); err != nil || !sameEntries(m, pinned) { return hostclaim.PreparedAuthority{}, false, fmt.Errorf("%w: P's files manifest is not the pinned manifest", hostrelocate.ErrRefused) } } filesSum := sha256.Sum256(raw) auth := hostclaim.PreparedAuthority{Generation: p, FilesSeq: m.Seq, FilesManifestSHA256: hex.EncodeToString(filesSum[:])} if pin.DB == nil { return auth, true, nil } prefix := c.Holder.Location + "/" + p + "/" key := moves.InstallManifestKey(prefix) versions, err := v.Data.ListVersions(ctx, v.Bucket, key) if err != nil { return hostclaim.PreparedAuthority{}, false, err } var current *lease.ObjectVersion for i := range versions { if versions[i].Key == key && versions[i].IsLatest && !versions[i].DeleteMarker { current = &versions[i] } } if current == nil { return hostclaim.PreparedAuthority{}, false, nil // the build has not completed } body, err := v.Data.GetVersion(ctx, v.Bucket, key, current.VersionID) if err != nil { return hostclaim.PreparedAuthority{}, false, err } var im moves.InstallManifest if err := json.Unmarshal(body, &im); err != nil || im.Op != c.Op.ID || im.Host != o.Host || im.N != pin.DB.MaxTXID || im.SnapshotKey != moves.SnapshotKey(prefix, im.N) || im.AnchorKey != moves.AnchorKey(prefix, im.N) || im.SnapshotVersion == "" || im.AnchorVersion == "" || im.IntegrityCheck != "ok" { return hostclaim.PreparedAuthority{}, false, fmt.Errorf("%w: install manifest %s is not this pin's build of P", hostrelocate.ErrRefused, key) } for _, want := range []struct{ key, version string }{{im.SnapshotKey, im.SnapshotVersion}, {im.AnchorKey, im.AnchorVersion}} { vs, err := v.Data.ListVersions(ctx, v.Bucket, want.key) if err != nil { return hostclaim.PreparedAuthority{}, false, err } found := false for _, ov := range vs { if ov.Key == want.key && ov.VersionID == want.version && !ov.DeleteMarker { found = true } } if !found { return hostclaim.PreparedAuthority{}, false, fmt.Errorf("%w: %s@%s named by the install manifest is not there", hostrelocate.ErrRefused, want.key, want.version) } } sum := sha256.Sum256(body) auth.InstallManifest, auth.SHA256, auth.SnapshotTXID = key, hex.EncodeToString(sum[:]), im.N return auth, true, nil } func sameEntries(a, b hostfiles.Manifest) bool { ra, _ := json.Marshal(struct { E map[string]hostfiles.Entry P map[string]hostfiles.Entry }{a.Entries, a.Platform}) rb, _ := json.Marshal(struct { E map[string]hostfiles.Entry P map[string]hostfiles.Entry }{b.Entries, b.Platform}) return string(ra) == string(rb) } var _ = os.ErrNotExist ``` `Source` carries the pinned generation's full keys as `genselect` lists them (`///-.ltx`); moves' `SealedClient` parses the level and TXID range from the key's tail (`ltx.ParseFilename`), so `SealedObject` needs nothing more than key, version and size (R7). Replay from `installing` after a crash: `CopyPinnedManifest` is a success on its own earlier copy; `BuildGeneration` reuses its intent, snapshot and anchor by construction (moves §7.6). An empty pin builds no snapshot or anchor. - [ ] **Step 4: Run the tests** Run: `nice -n 19 go test ./internal/host/ -run 'PreparedFromInstallManifest|PreparedRefuses|RelocationBuild' -count=1` Expected: PASS. - [ ] **Step 5: Commit** ```bash git add internal/host/relocationbuild.go internal/host/relocationbuild_test.go git commit -m "host: P is built under the replica lease from the pinned versions, files manifest first, and verified by reading both back Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 8c: The local install and the probe **Gated on:** claim Plan 6b (`ClaimWakeInputs`, `ClaimLaunchInputs`, `activator.StartPolicy.Probe`, `ClaimWakeAssembly.Build`'s runtime) and request **R1** (6b's runtime dispatches `relocating/probing` to this provider instead of refusing every probe); request **R10** (the reservation-bearing isolated root launch protocol, without which 6b's `Preflight` refuses every launch: rev 13); moves Phase 1 Plan 1 Task 7b (`appconfig.RenderHeld(ctx, s, priv, dataDir, rt, op, box) (appconfig.HeldRender, error)`, `release.MaterializeHeld(ctx, s, optDir, rt, version) (*release.Manifest, error)`); Tasks 8a and 8b. Not compiled in the dry run. The local install is §5.8's "The local install (B1-2)", under the host guard with the stamp the `installing` CAS issued: files and env from P's manifest (origins: the pin's chain plus P), the wake state imported, `RenderHeld` and `MaterializeHeld` with current inputs, every step journaled locally, then the receipt. The probe is one cold admission bound to `op.id` and `probing`, through 6b's wake runtime with `StartPolicy{Probe: true}`, whose inputs this task supplies. **Files:** - Create: `internal/host/relocationinstall.go`, `internal/host/relocationwake.go` - Test: `internal/host/relocationinstall_test.go` **Interfaces:** - Consumes: `hostguard.Guard.Activate`, `Guard.With`, `hostguard.Capture`; `hostfiles.Restore`, `LoadWake`, `ImportWake`, `ReadPlatform`, `EnvItem`; `genselect.SelectPrepared`, `genselect.Select`; 6b's `ClaimWakeInputs`, `ClaimLaunchInputs`, `activator.StartPolicy`; moves' `RenderHeld`, `MaterializeHeld`; Task 6's `coverageFor` rule (restated here against the observed claim). - Produces: `(ClaimAssembly) relocationInstall(cfg)`, `(ClaimAssembly) relocationAdmit(cfg)`, `host.RelocationWakeInputs` (a `ClaimWakeInputs` for `relocating/probing`). - [ ] **Step 1: Write the failing tests** ```go package host // The local install runs only under the stamp the installing CAS issued, // and never once any admission has named P. func TestRelocationInstallIsBoundToTheInstallStamp(t *testing.T) { /* Guard record at another stamp: ErrStampMismatch; P opened: hostrelocate.ErrRefused; happy path: the receipt's InstallStamp is the journaled one and JournalSHA256 is the journal file's */ } // A fresh spare with no host directory and a non-empty files manifest: // the directories are created and inventoried at install_stamp before // Restore, and retirement finds them (round 4 finding 3). func TestRelocationInstallCreatesAndInventoriesTheDestination(t *testing.T) { /* no files dir; after install the dir exists, the inventory holds a files entry at the stamp, and 6b's census refuses an untracked sibling */ } // Under the edge unit's filesystem restrictions MaterializeHeld is // blocked: the receipt waits (ErrNotYet) until root's relay (R9) has // pointed live/ at the pinned release, then is recorded. func TestRelocationInstallWaitsForRootsHeldStaging(t *testing.T) { /* a read-only optDir; the relay served; the receipt only after LiveVersion == rt.Version */ } // The probe's inputs: the first opening restores P from its prepared pin // with KeepStamp, Prepared and the operation's coverage; a later attempt // restores the predecessor with a fresh generation, no KeepStamp, coverage // "", Prepared false whatever the ordinary provider answered; a later // attempt over an empty predecessor is Empty, not Prepared. func TestRelocationWakeInputsOpenPOnceThenRestoreThePredecessor(t *testing.T) { /* table over a probing claim before and after P's admission, with an Ordinary that returns Prepared true */ } // A second attempt that crashes before its first replication publishes no // marker: with Prepared false 6b's runtime publishes only after the first // durable root (N1). func TestASecondProbeAttemptPublishesNoMarkerBeforeItsRoot(t *testing.T) { /* 6b's wake runtime with these inputs; crash injected in Replicate before the root; no generation.json under the attempt's generation */ } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/host/ -run 'RelocationInstall|RelocationWakeInputs' -count=1` Expected: FAIL to compile. - [ ] **Step 3: The local install** `internal/host/relocationinstall.go`: ```go package host import ( "context" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "fmt" "os" "path/filepath" "strconv" "github.com/carlosframework/platform/internal/appconfig" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostfiles" "github.com/carlosframework/platform/internal/hostguard" "github.com/carlosframework/platform/internal/hostrelocate" "github.com/carlosframework/platform/internal/release" ) // 6b's StopInventoryDB.EnsureDirectories is the typed inventory's directory // creation; if 6b's merged inventory names it otherwise, bind that name. // relocationInstallJournal is the box's local record of its install steps // for one operation (/relocate///install.json). Its // digest is the receipt's journal_sha256. type relocationInstallJournal struct { V int `json:"v"` Host string `json:"host"` Op string `json:"op"` Box string `json:"box"` InstallStamp uint64 `json:"install_stamp"` Prepared string `json:"prepared"` Steps map[string]string `json:"steps"` // files, wake, env, render, materialize } func (a ClaimAssembly) relocationInstall(cfg RelocationAssemblyConfig) func(context.Context, hostclaim.Observed) (hostclaim.LocalInstall, error) { return func(ctx context.Context, o hostclaim.Observed) (hostclaim.LocalInstall, error) { if cfg.Data == nil || cfg.Registry == nil || o.Claim == nil || o.Claim.Holder == nil { return hostclaim.LocalInstall{}, ErrClaimStopUnavailable } c := o.Claim local, err := a.localStopIdentity(ctx) if err != nil { return hostclaim.LocalInstall{}, err } stamp, err := strconv.ParseUint(c.Op.Journal["install_stamp"], 10, 64) if err != nil || stamp == 0 { return hostclaim.LocalInstall{}, fmt.Errorf("%w: no install stamp", hostrelocate.ErrRefused) } var ref hostclaim.RecoveryPin if err := json.Unmarshal([]byte(c.Op.Journal["pin"]), &ref); err != nil { return hostclaim.LocalInstall{}, err } pin, err := hostrelocate.ReadPin(ctx, a.Claims, ref) if err != nil { return hostclaim.LocalInstall{}, err } p := c.Op.Journal["prepared"] prefix := c.Holder.Location + "/files/" + p + "/" origins := append(append([]string(nil), pin.Files.Origins...), prefix) // The stamp the installing CAS issued is this box's installation: // activate it (the executor's own activation, idempotent on the // same stamp), then capture and work under it. if err := a.Guard.Activate(ctx, o.Host, local.Actor.Box, nil); err != nil { return hostclaim.LocalInstall{}, err } captured, err := hostguard.Capture(a.Guard.DataDir, o.Host) if err != nil { return hostclaim.LocalInstall{}, err } if captured.Stamp != stamp { return hostclaim.LocalInstall{}, hostguard.ErrStampMismatch } check := func(fresh *hostclaim.Claim) error { if fresh.Op.ID != c.Op.ID || fresh.Op.Phase != hostclaim.PhaseInstalled || fresh.Op.Exec != local.Actor.Box || fresh.Op.Journal["prepared"] != p || fresh.Op.Journal["install_stamp"] != c.Op.Journal["install_stamp"] || fresh.Op.Journal["local_install"] != "" || fresh.Op.Journal["stuck"] != "" { return fmt.Errorf("%w: the install's authority changed", hostrelocate.ErrRefused) } for _, adm := range fresh.Holder.Admissions { if adm.Generation == p { return fmt.Errorf("%w: P was admitted; the local install is never remade", hostrelocate.ErrRefused) // B2-5 } } return nil } dir := relocationJournalDir(cfg, o.Host, c.Op.ID) journal := relocationInstallJournal{V: 1, Host: o.Host, Op: c.Op.ID, Box: local.Actor.Box, InstallStamp: stamp, Prepared: p, Steps: map[string]string{}} var receipt hostclaim.LocalInstall err = a.Guard.With(ctx, o.Host, captured, check, func() error { if err := ensureDir(dir); err != nil { return err } // A fresh spare has no host directory yet (round 4 finding 3): // the files directory and the env path's parent are created // and inventoried under this installation's authority through // 6b's typed inventory (a "files" entry at install_stamp), so // retirement knows them; never an untracked MkdirAll. if err := a.Reconcile.Inventory.EnsureDirectories(ctx, o.Host, captured, stamp, cfg.FilesDir(o.Host), filepath.Dir(cfg.EnvPath(o.Host))); err != nil { return err } m, _, err := hostfiles.Restore(ctx, cfg.Data, prefix, cfg.FilesDir(o.Host), origins) if err != nil { return err } journal.Steps["files"] = strconv.FormatInt(m.Seq, 10) if ws, ok, err := hostfiles.LoadWake(ctx, cfg.Data, prefix, m, origins); err != nil { return err } else if ok { if err := hostfiles.ImportWake(cfg.Registry, o.Host, ws); err != nil { return err } journal.Steps["wake"] = "imported" } else { journal.Steps["wake"] = "none" } if _, ok := m.Platform[hostfiles.EnvItem]; ok { env, err := hostfiles.ReadPlatform(ctx, cfg.Data, prefix, m, hostfiles.EnvItem, origins) if err != nil { return err } if err := writeFileSynced(cfg.EnvPath(o.Host), env, 0o600); err != nil { return err } journal.Steps["env"] = "written" } rt, err := cfg.Registry.Get(o.Host) if err != nil || rt == nil { return fmt.Errorf("%w: no route for %s", hostrelocate.ErrRefused, o.Host) } // CC13: the held config and the held release, with current // inputs, recorded as this operation's (moves Plan 1 Task 7b). rendered, err := appconfig.RenderHeld(ctx, a.Claims, cfg.SecretsKey, a.Guard.DataDir, *rt, c.Op.ID, local.Actor.Box) if err != nil { return err } journal.Steps["render"] = rendered.SHA256 // The edge unit cannot write optDir (ProtectSystem=strict): // MaterializeHeld returns ErrStageBlocked and the held stage // request is relayed to root, whose authorizer accepts a // relocation bound to this operation, executor, install stamp // and the pinned release (R9). The receipt is recorded only // once live/ names that version (release.LiveVersion). if _, err := release.MaterializeHeld(ctx, a.Claims, release.OptDir(), *rt, rt.Version); err != nil && !errors.Is(err, release.ErrStageBlocked) { return err } if live, err := release.LiveVersion(release.OptDir(), o.Host); err != nil || live != rt.Version { return fmt.Errorf("%w: the held release %s is not staged yet (live %q)", hostrelocate.ErrNotYet, rt.Version, live) } journal.Steps["materialize"] = rt.Version raw, err := json.Marshal(journal) if err != nil { return err } if err := writeFileSynced(filepath.Join(dir, "install.json"), raw, 0o600); err != nil { return err } sum := sha256.Sum256(raw) receipt = hostclaim.LocalInstall{Box: local.Actor.Box, Enrollment: local.Actor.Enrollment, InstallStamp: stamp, FilesManifestVersion: m.Seq, JournalSHA256: hex.EncodeToString(sum[:])} return nil }) return receipt, err } } func writeFileSynced(path string, body []byte, mode os.FileMode) error { tmp := path + ".tmp" if err := os.WriteFile(tmp, body, mode); err != nil { return err } f, err := os.Open(tmp) if err != nil { return err } if err := f.Sync(); err != nil { _ = f.Close() return err } _ = f.Close() return os.Rename(tmp, path) } ``` `RenderHeld`'s arguments are moves Plan 1 Task 7b's (`priv *ecdh.PrivateKey, dataDir string`); `cfg.SecretsKey` is the box's secrets key, a `RelocationAssemblyConfig` field Task 8a's config gains (`SecretsKey *ecdh.PrivateKey`). The version `MaterializeHeld` pins is the route's current `Version` (the held release at the moment of install), recorded in the journal (P6-D3). - [ ] **Step 4: The probe's inputs** `internal/host/relocationwake.go`: ```go package host import ( "context" "encoding/json" "errors" "fmt" "time" "github.com/carlosframework/platform/internal/activator" "github.com/carlosframework/platform/internal/genselect" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostrelocate" "github.com/carlosframework/platform/internal/hostwrite" "github.com/carlosframework/platform/internal/lease" "github.com/carlosframework/platform/internal/moves" "github.com/carlosframework/platform/internal/registry" "github.com/carlosframework/platform/internal/release" ) // RelocationWakeInputs is 6b's ClaimWakeInputs for a relocation's probe // (§5.8 "The probe's admission"): the first attempt opens P from its // prepared pin with KeepStamp and the operation's coverage; every later // attempt is an ordinary cold admission into a fresh generation from the // predecessor, no KeepStamp, coverage "". Ordinary is the assembly's own // provider for everything that is not the admission and the baseline. type RelocationWakeInputs struct { Ordinary ClaimWakeInputs // 6b's provider: executable, args, env, limit, identity Claims release.Store Data lease.VersionedStore Bucket string Box string NewID func() string } func (r RelocationWakeInputs) Select(ctx context.Context, rt registry.Route, o hostclaim.Observed, p activator.StartPolicy) (ClaimLaunchInputs, error) { c := o.Claim if c == nil || c.Holder == nil || c.Op.Kind != hostclaim.KindRelocating || c.Op.Phase != hostclaim.PhaseProbing || c.Op.Exec != r.Box || !p.Probe || c.Op.Journal["stuck"] != "" { return ClaimLaunchInputs{}, fmt.Errorf("%w: not this box's relocation probe", hostrelocate.ErrRefused) } in, err := r.Ordinary.Select(ctx, rt, o, p) if err != nil { return ClaimLaunchInputs{}, err } var ref hostclaim.RecoveryPin if err := json.Unmarshal([]byte(c.Op.Journal["pin"]), &ref); err != nil { return ClaimLaunchInputs{}, err } pin, err := hostrelocate.ReadPin(ctx, r.Claims, ref) if err != nil { return ClaimLaunchInputs{}, err } prepared := c.Op.Journal["prepared"] opened := false for _, adm := range c.Holder.Admissions { if adm.Generation == prepared { opened = true } } req := genselect.Request{Bucket: r.Bucket, Region: c.Op.Intent["region"], Host: o.Host, Holder: *c.Holder} adm := hostclaim.Admission{ObservedOpID: c.Op.ID, ObservedPhase: hostclaim.PhaseProbing, ObservedSession: c.Holder.Session, ObservedGeneration: c.Holder.Generation, Box: r.Box, Mode: "cold", Region: c.Op.Intent["region"], PredecessorRegion: c.Op.Intent["region"]} // Nothing is inherited from the ordinary provider's answer: each branch // states its prepared and empty flags (N1). in.Prepared, in.Empty, in.Baseline = false, false, nil switch { case !opened && pin.DB == nil: // An evidenced empty bootstrap: a fresh, empty generation, opened // as P (its marker precedes replication, as a prepared one's). adm.NewGeneration, adm.KeepStamp, adm.Coverage = prepared, true, coverageOf(c) in.Empty, in.Prepared = true, true case !opened: var im moves.InstallManifest if err := readInstallManifest(ctx, r.Data, r.Bucket, c.Holder.Location+"/"+prepared+"/", &im); err != nil { return ClaimLaunchInputs{}, err } plan, err := genselect.SelectPrepared(ctx, r.Data, req, genselect.PreparedPin{Generation: prepared, SnapshotKey: im.SnapshotKey, SnapshotVersion: im.SnapshotVersion, SnapshotTXID: im.N, SnapshotCRC64: im.SnapshotCRC64}) if err != nil { return ClaimLaunchInputs{}, err } adm.NewGeneration, adm.KeepStamp, adm.Coverage, adm.RemoteMaxTXID = prepared, true, coverageOf(c), pin.DB.MaxTXID in.Baseline, in.Prepared = plan, true default: // A later attempt: cold from the predecessor, as any cold wake. Its // marker follows the first durable root (Prepared false). A // predecessor that committed nothing (no LTX file, claim §6.4a // "Empty bootstrap") is an ordinary empty cold admission under 6b's // own empty authority (R1-13). adm.NewGeneration = r.NewID() plan, err := genselect.Select(ctx, r.Data, req) switch { case errors.Is(err, genselect.ErrEmpty): in.Empty = true case err != nil: return ClaimLaunchInputs{}, err default: adm.RemoteMaxTXID, in.Baseline = plan.MaxTXID, plan } } in.Admission = adm return in, nil } // coverageOf is Task 6's coverageFor: released only for the admission // that opens P when this operation journaled a release whose session is // the predecessor writer. func coverageOf(c *hostclaim.Claim) string { var rel hostclaim.Release if json.Unmarshal([]byte(c.Op.Journal["release"]), &rel) != nil || rel.ReleasedBy == "" { return "" } pred := hostclaim.AdmissionRecord{Session: c.Holder.Session, Box: c.Holder.WriterBox, Generation: c.Holder.Generation} if n := len(c.Holder.Admissions); n > 0 { pred = c.Holder.Admissions[n-1] } if pred.Box != rel.ReleasedBy || pred.Session != rel.Session || pred.Generation != rel.Generation { return "" } return "released" } func (r RelocationWakeInputs) Materialize(ctx context.Context, o hostclaim.Observed, in ClaimLaunchInputs, w hostwrite.Work) error { return r.Ordinary.Materialize(ctx, o, in, w) // RenderHeld and MaterializeHeld ran at the local install; the session publisher starts from P's manifest (moves §7.6) } func (r RelocationWakeInputs) EmptyAuthority(ctx context.Context, o hostclaim.Observed, in ClaimLaunchInputs) error { if in.Prepared { // The opening of P: only an evidenced empty pin. var ref hostclaim.RecoveryPin if json.Unmarshal([]byte(o.Claim.Op.Journal["pin"]), &ref) != nil || !ref.Empty { return fmt.Errorf("%w: no evidenced empty pin", hostrelocate.ErrRefused) } return nil } return r.Ordinary.EmptyAuthority(ctx, o, in) // a later attempt: the assembly's own rule for an empty predecessor } func (r RelocationWakeInputs) Publish(ctx context.Context, o hostclaim.Observed, in ClaimLaunchInputs) error { return r.Ordinary.Publish(ctx, o, in) // generation.json, then PublishGeneration (claim §6.4a item 5) } // relocationAdmit is one probe attempt: 6b's wake runtime with // StartPolicy{Probe: true} and this provider's inputs. (*ClaimWakeAssembly).Probe // is request R1's name: one probe attempt (Prerequisites, Admit, Prepare, // the readiness window) for a non-serving claim, returning the admitted // session and the attempt's exact health. passed is that health. func (a ClaimAssembly) relocationAdmit(cfg RelocationAssemblyConfig) func(context.Context, hostclaim.Observed, string) (uint64, bool, error) { return func(ctx context.Context, o hostclaim.Observed, coverage string) (uint64, bool, error) { since, err := time.Parse(time.RFC3339Nano, o.Claim.Op.Journal["probing_since"]) if err != nil { return 0, false, err } return a.Wake.Probe(ctx, o, activator.StartPolicy{MaxStarts: 5, Window: 30 * time.Second, Probe: true, Deadline: since.Add(hostrelocate.ProbeDeadline)}) // R1 } } ``` `readInstallManifest` is Task 8b's read of the current install manifest version (factored out of `PreparedFromInstallManifest.Verify`). The coverage the participant computed (Task 6's `coverageFor`) and `coverageOf` here are the same rule; `AdmitWriter` is the authority for both. - [ ] **Step 5: Run the tests** Run: `nice -n 19 go test ./internal/host/ -run 'RelocationInstall|RelocationWakeInputs' -count=1` Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/host/relocationinstall.go internal/host/relocationwake.go internal/host/relocationinstall_test.go git commit -m "host: the target's local install under its install stamp, and the probe's admission that opens P once Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 9: Reconcile keeps relocation rows held (stage B) **Gated on:** claim Plan 6b merged, for `hostbox.ReconcileAdapter`, `registry.Route.HoldOp`, `registry.SetClaimHold`, `registry.ClaimOwnership.Valid` (which requires a non-zero stamp), `hostbox.RegistryHoldPolicy` and `provision.ReconcileConfig.ClaimInstall`/`ClaimRetire`. Task 3's `Policy.Check` action `reconcile` is un-gated and already lets a relocating participant through the write policy. Not compiled in the dry run. `Reconciler.check` today admits only `Serving`, and `Reconciler.Install` activates the box as the executor (`Guard.Activate(…, nil)` refuses any other). Under `relocating` reconcile must do exactly what `RelocationRow` says, with three distinct authorities (ruling 17): | `RelocationRow` | this box is | authority | effect | |---|---|---|---| | `RowReady` | the serving executor, or the source in `starting`/`releasing` before its fence closes | today's: `Activate` as `op.exec`, captured stamp, `SelectAt(…, "reconcile", captured)`, 6b's inventory and census | mint or keep `ready`, `hold_op` empty | | `RowHeld`, `op.exec == box` | the target from `installing` on | the same activation path: the stamp is `install_stamp`, this box's own installation (§5.8) | mint or keep the row held, `hold_op = op.id`, through `SetClaimHold` under the captured work | | `RowHeld`, `op.exec != box` or this box's admission fence closed, a row exists | the source after its fence closed (still `op.exec` in `releasing`, or in `pinning` and later); a replaced target with its old installation | `Guard.WithLocal` (the flock, no installation authority) around `registry.SetClaimHold` conditional on the row's recorded ownership and observed hold; the guarded path would refuse a fenced session (`With`: `ErrFenced`) | keep the row, set `hold_op = op.id`; never `ready` | | `RowHeld`, `op.exec != box`, no row | the target before `installing` | none | mint nothing (log once); the edges file withholds the host from a box that is not `op.exec` (§24), so no request routes here | | `RowRetiring` | the owner of an open `park-local` | none here: the chore retires it under the host guard (P6-D2) | keep, held | | `RowUnexplained` | nobody | none | keep, held, alert once per pass; never delete by hostname | **Files:** - Modify: `internal/hostbox/reconcile.go` (`Reconciler.check`, `Reconciler.Install`), 6b's `internal/hostbox/reconcile_adapter.go` (`Install`), 6b's `internal/provision/reconcile.go` (the claim branch and the sweep) - Test: `internal/hostbox/reconcile_relocation_test.go` **Interfaces:** - Consumes: Task 3's `RelocationRow`, `Policy.Check`'s `reconcile` action; 6b's `ReconcileAdapter{Registry, Box, Reconciler}`, `registry.SetClaimHold(ctx, host, owner, expected, next string)`, `registry.ClaimOwnership`; main's `hostguard.Guard.WithLocal`. - Produces: `(ReconcileAdapter) Configure` wiring `HoldRow` (`holdRowFor`), the adapter's hold writes after `write(owner)`. Task 9a (stage A) owns `Reconciler.FenceClosed`, `HoldRow`, `rowDecision` and `holdOnly`. - [ ] **Step 1: Write the failing test** ```go package hostbox import ( "context" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "fmt" "path/filepath" "strconv" "strings" "testing" "time" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostguard" "github.com/carlosframework/platform/internal/hostwrite" "github.com/carlosframework/platform/internal/provision" "github.com/carlosframework/platform/internal/registry" "github.com/carlosframework/platform/internal/release" ) // Every phase, for the source and the target, through the real adapter: // what the registry holds afterwards is the §5.3 table. Two things are // distinct: a box with no role and no row mints nothing (ErrChanged, // nothing written); a box with no role and an existing row keeps it, held, // never deleted (RowUnexplained). A hold is hold_op; nothing here writes a // status. func TestReconcileAdapterInstallFollowsTheRowCases(t *testing.T) { for _, tc := range []struct { name, phase, exec, box string fenceClosed bool existing bool // a row exists before the pass wantRow bool // a row exists after it wantHold string // its hold_op wantErr error // nil, or ErrChanged for "nothing minted" }{ {"source starting", hostclaim.PhaseStarting, "src", "src", false, true, true, "", nil}, {"target before installing, no row", hostclaim.PhaseStarting, "src", "dst", false, false, false, "", ErrChanged}, {"source releasing, fence open", hostclaim.PhaseReleasing, "src", "src", false, true, true, "", nil}, {"source releasing, fence closed", hostclaim.PhaseReleasing, "src", "src", true, true, true, "rel-1", nil}, // still op.exec; kept, held, under WithLocal (a real durable fence: f.beginFence) {"source pinning", hostclaim.PhasePinning, "none", "src", true, true, true, "rel-1", nil}, {"target installing, first pass", hostclaim.PhaseInstalling, "dst", "dst", false, false, true, "rel-1", nil}, // minted held at install_stamp {"target installed", hostclaim.PhaseInstalled, "dst", "dst", false, true, true, "rel-1", nil}, {"target probing", hostclaim.PhaseProbing, "dst", "dst", false, true, true, "rel-1", nil}, {"source probing", hostclaim.PhaseProbing, "dst", "src", true, true, true, "rel-1", nil}, {"no role, no row", hostclaim.PhaseProbing, "dst", "zzz", false, false, false, "", ErrChanged}, {"no role, an existing row is kept held", hostclaim.PhaseProbing, "dst", "zzz", false, true, true, "rel-0", ErrChanged}, // its earlier hold, untouched } { t.Run(tc.name, func(t *testing.T) { f := adapterFixture(t, tc.box) // 6b's reconcile_adapter_layout_test fixture: registry, inventory, guard dir, claims store f.claimAt(tc.phase, tc.exec) if tc.existing { f.withExistingRow("rel-0") // minted by an earlier pass, with an earlier hold where the case says } if tc.fenceClosed { f.beginFence() // root's durable BeginFence for the captured session; FenceClosed reads it through Guard.AdmissionFenced } f.adapter.Reconciler.FenceClosed = f.fenceClosed err := f.adapter.Install(context.Background(), f.record(), f.write) if !errors.Is(err, tc.wantErr) { t.Fatalf("err %v, want %v", err, tc.wantErr) } row, _ := f.registry.Get(f.host) switch { case !tc.wantRow && row != nil: t.Fatalf("a row for a box with no role: %+v", row) case tc.wantRow && row == nil: t.Fatal("the row was deleted") case tc.wantRow && row.HoldOp != tc.wantHold: t.Fatalf("hold %q, want %q", row.HoldOp, tc.wantHold) } }) } } // The full sequence on the target: minted held in installing, kept held // through probing, and after the serving CAS the next reconcile clears // the completed operation's hold under the serving executor's verified // ownership (N4), so the row routes. func TestReconcileAdapterClearsTheHoldAfterCompletion(t *testing.T) { f := adapterFixture(t, "dst") for _, phase := range []string{hostclaim.PhaseInstalling, hostclaim.PhaseInstalled, hostclaim.PhaseProbing} { f.claimAt(phase, "dst") if err := f.adapter.Install(context.Background(), f.record(), f.write); err != nil { t.Fatalf("%s: %v", phase, err) } if row, _ := f.registry.Get(f.host); row == nil || row.HoldOp != "rel-1" { t.Fatalf("%s: row %+v", phase, row) } } f.serving("dst") // CompleteRelocation if err := f.adapter.Install(context.Background(), f.record(), f.write); err != nil { t.Fatal(err) } if row, _ := f.registry.Get(f.host); row == nil || row.HoldOp != "" { t.Fatalf("the serving target is still held: %+v", row) } // A hold of a running operation is never cleared by a ready decision // for another: a second relocation holds the row again, and the // source's own pass leaves it. f.claimAt(hostclaim.PhaseStarting, "dst") f.setOp("rel-2") if err := f.adapter.Install(context.Background(), f.record(), f.write); err != nil { t.Fatal(err) } if row, _ := f.registry.Get(f.host); row == nil || row.HoldOp != "" { t.Fatalf("the source in starting is ready: %+v", row) } } // adapterFixture is this task's: 6b's ReconcileAdapter over a real // registry, inventory, guard directory and a dir-store claim, with one // host whose record is rec. Its methods drive the claim and the registry // so each case starts where the table says. type adapterTestFixture struct { t *testing.T host string box string claims release.Store registry *registry.Registry adapter ReconcileAdapter rec provision.ClaimRecord guard hostguard.Guard } func adapterFixture(t *testing.T, box string) *adapterTestFixture { t.Helper() dataDir := t.TempDir() reg, err := registry.Open(filepath.Join(dataDir, "registry.db")) if err != nil { t.Fatal(err) } t.Cleanup(func() { reg.Close() }) claims := release.NewDirStore(t.TempDir()) inv, err := OpenStopInventoryDB(filepath.Join(dataDir, "inventory.db")) // 6b's if err != nil { t.Fatal(err) } launches, err := OpenLaunchJournal(filepath.Join(dataDir, "launches")) // 6b's if err != nil { t.Fatal(err) } optDir, unitsDir := t.TempDir(), t.TempDir() guard := hostguard.Guard{DataDir: dataDir, Claims: hostguard.StoreReader{Store: claims}} f := &adapterTestFixture{t: t, host: "shop.bab.oncarlos.com", box: box, claims: claims, registry: reg, guard: guard} f.adapter = ReconcileAdapter{Registry: reg, Inventory: inv, Box: box, Enrollment: strings.Repeat(box[0:1], 32), DirectoryRevision: "4", Reconciler: Reconciler{Claims: claims, Guard: guard, Box: box, Region: "ie", Now: time.Now, Writes: hostwrite.Config{Enabled: true, Selector: WorkSource{Guard: guard, Box: box, Region: "ie", Now: time.Now, Policy: Policy{Held: RegistryHoldPolicy(reg), Relocation: RelocationPolicy(box)}.Check, Local: func(context.Context, *hostclaim.Claim, Selection) error { return nil }}}, CheckRecord: func(context.Context, RecordSelection) error { return nil }, CheckLocal: func(context.Context, hostclaim.Observed, RecordSelection) error { return nil }}, Census: &EmptyInstallationCensus{Inventory: inv, Registry: reg, Launches: launches, Box: box, Enrollment: strings.Repeat(box[0:1], 32), DirectoryRevision: "4", Roots: map[string]string{"data": dataDir, "opt": optDir, "units": unitsDir}, SharedPaths: map[string][]string{}}} // 6b's fields, nothing else f.adapter.Reconciler.HoldRow = holdRowFor(f.adapter) f.adapter.Reconciler.FenceClosed = f.fenceClosed created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) record := provision.Record{Schema: 1, Account: "bab", App: "jam", Host: f.host, Slug: "shop", Channel: "edge", Region: "ie", CreatedAt: created} body, _ := json.Marshal(record) key := provision.RecordKey("bab", "jam", f.host) if _, err := claims.Put(context.Background(), key, body, ""); err != nil { t.Fatal(err) } f.rec = provision.ClaimRecord{Host: f.host, Key: key, Family: "instance", Account: "bab", App: "jam", CreatedAt: created, Value: record, Route: registry.Route{Host: f.host, Kind: "instance", Account: "bab", App: "jam", Provisioned: true}} // The claim: created on src, one cold session, as Task 5's fixture. nh := hostclaim.NewHolder{Family: "instance", Key: key, Account: "bab", App: "jam", Host: f.host, Region: "ie", RecordRegion: "ie", CreatedAt: created} o, _ := hostclaim.Read(context.Background(), claims, f.host) o = f.step(o, hostclaim.BeginCreateOn(nh, body, "c1", "console", created, hostclaim.ExecutorSelection{Box: "src", Enrollment: strings.Repeat("s", 32)})) o = f.step(o, hostclaim.CompleteCreate("c1", "src")) f.step(o, hostclaim.AdmitWriter(hostclaim.Admission{Box: "src", Mode: "cold", NewGeneration: "g1", Region: "ie", KeepStamp: true})) return f } func (f *adapterTestFixture) step(o hostclaim.Observed, tr func(*hostclaim.Claim) error) hostclaim.Observed { f.t.Helper() next, err := hostclaim.Update(context.Background(), f.claims, o, tr) if err != nil { f.t.Fatal(err) } return next } func (f *adapterTestFixture) claim() hostclaim.Observed { o, err := hostclaim.Read(context.Background(), f.claims, f.host) if err != nil { f.t.Fatal(err) } return o } // claimAt rewrites the operation to phase with exec, keeping the holder, // as the relocation transitions would leave it (install_stamp journaled // for the target's phases, the stamp activated on this box when it is // the executor). func (f *adapterTestFixture) claimAt(phase, exec string) { f.t.Helper() f.step(f.claim(), func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindRelocating, ID: "rel-1", Phase: phase, Exec: exec, Participants: []string{"src", "dst"}, Intent: map[string]string{"region": "ie", "source_box": "src", "target_box": "dst", "source_enrollment": strings.Repeat("s", 32), "target_enrollment": strings.Repeat("d", 32)}, Journal: map[string]string{"exec_ack.src": "{}", "exec_ack.dst": "{}"}} if exec == "dst" { c.Holder.Stamp = c.Gen + 1 c.Op.Journal["install_stamp"] = strconv.FormatUint(c.Holder.Stamp, 10) } return nil }) if exec == f.box { if err := f.guard.Activate(context.Background(), f.host, f.box, nil); err != nil { f.t.Fatal(err) } } } // serving is the completed relocation: serving on box. func (f *adapterTestFixture) serving(box string) { f.t.Helper() f.step(f.claim(), func(c *hostclaim.Claim) error { c.Op = hostclaim.Op{Kind: hostclaim.KindServing, Exec: box, Participants: []string{box}} return nil }) } func (f *adapterTestFixture) setOp(id string) { f.step(f.claim(), func(c *hostclaim.Claim) error { c.Op.ID = id; return nil }) } // withExistingRow mints the row an earlier pass would have, owned by this // box's current installation, with hold. func (f *adapterTestFixture) withExistingRow(hold string) { f.t.Helper() captured, err := hostguard.Capture(f.guard.DataDir, f.host) if err != nil { if err := f.guard.Activate(context.Background(), f.host, f.box, func(*hostclaim.Claim) bool { return true }); err != nil { f.t.Fatal(err) } captured, err = hostguard.Capture(f.guard.DataDir, f.host) if err != nil { f.t.Fatal(err) } } digest := sha256.Sum256([]byte(fmt.Sprintf("%s\x00%s\x00%d\x00%s", f.host, captured.Incarnation, captured.Stamp, f.box))) rt := f.rec.Route rt.ClaimOwnership = registry.ClaimOwnership{ClaimIncarnation: captured.Incarnation, ClaimStamp: captured.Stamp, ClaimOwner: hex.EncodeToString(digest[:]), ClaimRowID: "row-1"} rt.HoldOp = hold if err := f.registry.Upsert(rt); err != nil { f.t.Fatal(err) } } func (f *adapterTestFixture) record() provision.ClaimRecord { return f.rec } // fenceClosed is Task 10's observer: the captured installation's fence, // read without the lock, on the observation given. func (f *adapterTestFixture) fenceClosed(_ context.Context, o hostclaim.Observed) (bool, error) { captured, err := hostguard.Capture(f.guard.DataDir, o.Host) if err != nil { return false, err } return f.guard.AdmissionFenced(o.Host, o.Claim, captured) } // beginFence closes this box's admission fence for the captured session, // as root's stop collector does (hostguard.BeginFence). func (f *adapterTestFixture) beginFence() { f.t.Helper() o := f.claim() captured, err := hostguard.Capture(f.guard.DataDir, f.host) if err != nil { f.t.Fatal(err) } target := hostguard.FenceTarget{FenceID: "fence-1", Host: f.host, Incarnation: captured.Incarnation, Stamp: captured.Stamp, Session: o.Claim.Holder.Session, Op: o.Claim.Op.ID, Phase: o.Claim.Op.Phase, Box: f.box, Enrollment: f.adapter.Enrollment, BootID: "boot", ControllerID: "ctl", ClaimETag: o.ETag, DirectoryRevision: "4"} if _, err := f.guard.BeginFence(context.Background(), target); err != nil { f.t.Fatal(err) } } // write is provision's claim-branch write: the row with its ownership. func (f *adapterTestFixture) write(owner registry.ClaimOwnership) error { rt := f.rec.Route rt.ClaimOwnership = owner return f.registry.Upsert(rt) } // A held row's hold is never cleared or moved by another operation, and // a kept row's hold write needs the row's recorded ownership. func TestReconcileAdapterHoldIsConditional(t *testing.T) { f := adapterFixture(t, "src") f.claimAt(hostclaim.PhasePinning, "none") f.withExistingRow("") if err := f.adapter.Install(context.Background(), f.record(), f.write); err != nil { t.Fatal(err) } other := registry.ClaimOwnership{ClaimIncarnation: "other", ClaimStamp: 9, ClaimOwner: "x", ClaimRowID: "y"} if err := f.registry.SetClaimHold(context.Background(), f.host, other, "rel-1", ""); !errors.Is(err, registry.ErrClaimOwnership) { t.Fatalf("another owner cleared the hold: %v", err) } } ``` - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/hostbox/ -run 'ReconcileAdapterInstall|ReconcileAdapterHold|ReconcileAdapterClears' -count=1` Expected: FAIL to compile: `f.adapter.Reconciler.FenceClosed undefined`. - [ ] **Step 3: The decision in `Reconciler`** Task 9a (stage A) built it: `Reconciler.check` decides with `RelocationRow` on a fresh read, `FenceClosed(ctx, obs) (bool, error)` is consulted inside the locked operation and an error keeps rows, and a held row whose box is not the executor or whose admission fence is closed goes through `holdOnly` under `WithLocal` with the adapter's `HoldRow`. Nothing here changes `reconcile.go` again. - [ ] **Step 4: The adapter's hold writes** In 6b's `ReconcileAdapter.Install`, after the row is written through `write(owner)` (the `RowReady` and executor `RowHeld` paths), set its hold from the same decision, conditional on the captured ownership and the observed hold, never clearing another operation's: ```go // The claim under the captured authority, read here: the earlier // `observed` is scoped to the census block above (6b). current, err := hostclaim.Read(ctx, a.Reconciler.Claims, rec.Host) if err != nil { return err } if current.Absent || current.Claim.Holder == nil || current.Claim.Holder.Incarnation != captured.Incarnation || current.Claim.Holder.Stamp != captured.Stamp { return ErrChanged } closed, err := a.Reconciler.fenceClosed(ctx, current) if err != nil { return err } d := RelocationRow(current.Claim, a.Box, true, closed) switch { case d.Case == RowHeld && row.HoldOp != d.HoldOp: // Held for the running operation; an earlier operation's hold is // replaced, never a different running one's (the claim runs one). if err := a.Registry.SetClaimHold(ctx, rec.Host, owner, row.HoldOp, d.HoldOp); err != nil { return err } case d.Case == RowReady && row.HoldOp != "" && row.HoldOp != current.Claim.Op.ID: // The operation that held this row is over and this box is the // serving executor with the captured stamp (the activation above // verified both): the row routes again (N4). if err := a.Registry.SetClaimHold(ctx, rec.Host, owner, row.HoldOp, ""); err != nil { return err } } ``` and give the reconciler its `HoldRow`: ```go func (a ReconcileAdapter) Configure(cfg *provision.ReconcileConfig) { a.Reconciler.HoldRow = holdRowFor(a) cfg.ClaimEnabled = a.Reconciler.Writes.Enabled cfg.ClaimInstall = a.Install cfg.ClaimRetire = a.Retire } // holdRowFor is the adapter's hold-only write for a kept row. func holdRowFor(a ReconcileAdapter) func(ctx context.Context, host, holdOp string) error { return func(ctx context.Context, host, holdOp string) error { row, err := a.Registry.Get(host) if err != nil { return err } if row == nil || !row.ClaimOwnership.Valid() { return ErrChanged // nothing to keep: a kept row is one this box installed } if row.HoldOp == holdOp { return nil } return a.Registry.SetClaimHold(ctx, host, row.ClaimOwnership, row.HoldOp, holdOp) } } ``` A hold is `hold_op` alone: non-empty is "held" for the edge (moves §5.2), and the status column is not written here. In 6b's `provision/reconcile.go` claim branch, a record whose `RelocationRow` is `RowRetiring` or `RowUnexplained` is kept, held, and left for typed retirement (`ClaimRetire`); an unexplained row is logged once per pass (`provision: %s: claim gives this box no role — held, not served`). Kept rows are re-read on a 10-minute sweep (`claimSweepEvery = 10 * time.Minute`), and at once when the control serial changes (§5.3 "Prompt changes"); a read error keeps the row and mints none (C6). - [ ] **Step 5: Run the tests** Run: `nice -n 19 go test ./internal/hostbox/ ./internal/provision/ -run 'Reconcile|Relocation|Claim' -count=1` Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/hostbox/ internal/provision/reconcile.go git commit -m "hostbox: reconcile mints and keeps relocation rows held by the operation and never deletes a former executor's row Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 10: Switch the passes on, behind `CARLOS_REGION_HA` **Gated on:** claim Plan 6b merged (Tasks 8a to 8c's effects, Task 9's rows); claim Plan 5's app gate merged (the `Admission` provider, ruling 4); and claim Plan 5's create door merged (`ProductionAuthority` as the console's create `Authority`); and request **R10**. *Activation gate (rev 13):* `applyRegionHA` turns nothing on, and `runRelocatePass` refuses with `activator.ErrClaimIsolationUnavailable`, while the assembly's launcher lacks R10's protocol. A box with `CARLOS_REGION_HA` on but no protocol logs the refusal once and stays dormant; it never starts a relocation whose probes cannot run. Not compiled in the dry run. On a box with `CARLOS_HOST_CLAIM` and `CARLOS_REGION_HA` both on, one pass every 30 seconds, and at once on a control serial change: for every claim under `control/hosts/` whose operation is `relocating` in this box's region, run `Worker.Step` (any active worker may), and for every claim naming this box as a participant, `Participant.Step`. The pass never starts a relocation by itself (spec 3's placer does); `carlos ops host relocate` (later, with Paul's copy) and the spec 3 placer call `Worker.Start`. The participant's effects are one `*RelocationEffects` per agent, opened with the agent's lifetime, so a pass's context bounds the pass and nothing else; each pass ends with `Participant.Sweep`, which finishes the controllers of participations the claim no longer runs (ruling 20). On the console, create's `Authority.SelectExecutor` is the `ExecutorChooser` over console's live edges read, and only when `Config.RegionHA` is on (Task 2); off, `Executors` is nil and the create is today's. **Files:** - Create: `internal/host/relocatepass.go` - Modify: `internal/host/agent.go` (start the pass next to `runHostFencePass`); the create door's construction of `hostops.ProductionAuthority` (claim Plan 5) - Test: `internal/host/relocatepass_test.go` - [ ] **Step 1: Write the failing tests** ```go package host import "testing" func TestRelocatePassIsOffUnlessBothFlagsAreOn(t *testing.T) { for _, tc := range []struct { claim, ha string want bool }{{"", "1", false}, {"1", "", false}, {"1", "0", false}, {"1", "yes", false}, {"1", "true", true}, {"0", "TRUE", true}} { env := map[string]string{"CARLOS_HOST_CLAIM": tc.claim, "CARLOS_REGION_HA": tc.ha} if got := relocatePassEnabled(func(k string) string { return env[k] }); got != tc.want { t.Fatalf("%+v: %v", tc, got) } } } ``` (`CARLOS_HOST_CLAIM` uses main's trimmed-nonempty test, where `"0"` enables it, as 6b states; `CARLOS_REGION_HA` is `strconv.ParseBool`.) - [ ] **Step 2: Run them to see them fail** Run: `nice -n 19 go test ./internal/host/ -run 'RelocatePass' -count=1` Expected: FAIL to compile: `undefined: relocatePassEnabled`. - [ ] **Step 3: The pass** `internal/host/relocatepass.go`: ```go package host import ( "context" "errors" "log" "os" "strings" "time" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostrelocate" "github.com/carlosframework/platform/internal/release" ) // relocatePassEnabled: the claim and region HA both on (spec §8 step 2). func relocatePassEnabled(getenv func(string) string) bool { return strings.TrimSpace(getenv("CARLOS_HOST_CLAIM")) != "" && regionHAEnabled(getenv) } // runRelocatePass replays every relocation in this box's region and this // box's participations. It starts none. Each pass is bounded; the // participant's controllers are not (they live under the agent's // lifetime, Task 7). func runRelocatePass(ctx context.Context, store release.Store, w hostrelocate.Worker, p hostrelocate.Participant, serial <-chan struct{}) { if !relocatePassEnabled(os.Getenv) { return } tick := time.NewTicker(30 * time.Second) defer tick.Stop() for { pass, cancel := context.WithTimeout(ctx, 2*time.Minute) keys, err := store.List(pass, "control/hosts/") if err != nil { log.Printf("relocate: list claims: %v", err) } for _, key := range keys { host, ok := strings.CutSuffix(strings.TrimPrefix(key, "control/hosts/"), ".json") if !ok || strings.Contains(host, "/") { continue } obs, err := hostclaim.Read(pass, store, host) if err != nil || obs.Absent || obs.Claim.Op.Kind != hostclaim.KindRelocating { continue } if _, err := w.Step(pass, host); err != nil && !errors.Is(err, hostrelocate.ErrNotYet) { log.Printf("relocate %s: worker: %v", host, err) } if _, err := p.Step(pass, host); err != nil && !errors.Is(err, hostrelocate.ErrNotYet) { log.Printf("relocate %s: %s: %v", host, p.Box, err) } } // Participations that ended this pass (a serving claim is skipped // above) release their controllers and the host's lifetime lock. if err := p.Sweep(pass); err != nil { log.Printf("relocate: sweep controllers: %v", err) } cancel() select { case <-ctx.Done(): return case <-tick.C: case <-serial: // the control serial changed (§5.3 "Prompt changes") } } } ``` `applyRegionHA` is Task 10's and is the body of request R8's `Extend` hook, which 6b's `Build` calls after actor verification and before it captures `Policy.Check` (round 4 N3: the adapter's box is set only there), gated on the flag: ```go // applyRegionHA gives the assembly region HA's two providers (round 3 // finding N3), only with CARLOS_REGION_HA on; off, the assembly is 6b's. // It is ClaimAssembly.Extend (R8), called with the verified actor. func applyRegionHA(getenv func(string) string) func(*ClaimAssembly, hostbox.Actor) { return func(a *ClaimAssembly, actor hostbox.Actor) { if !regionHAEnabled(getenv) || a == nil || actor.Box == "" { return } a.Policy.Relocation = hostbox.RelocationPolicy(actor.Box) guard := a.Guard // Read inside the caller's locked operation, never reacquiring the // host lock (round 4 finding 1); an error keeps rows and mints none. a.Reconcile.Reconciler.FenceClosed = func(_ context.Context, o hostclaim.Observed) (bool, error) { captured, err := hostguard.Capture(guard.DataDir, o.Host) if errors.Is(err, hostguard.ErrNoRecord) { return false, nil // no installation of ours: nothing to be fenced } if err != nil { return false, err } return guard.AdmissionFenced(o.Host, o.Claim, captured) } } } ``` Its test, `TestApplyRegionHASetsTheProvidersOnlyWithTheFlagOn`, builds a `ClaimAssembly` with 6b's test wiring and `Extend: applyRegionHA(getenv)`, runs `Build` with the flag off (both nil, `Build` behaves as 6b's) and on (both set, bound to the verified actor's box; `Policy.Check` admits a relocating participant's `reconcile` and the target's `probe`), and reconciles one host under a deadline with the real observer. In `internal/host/agent.go`, next to `go runHostFencePass(agentCtx)`, build the worker and participant from the assembly once it exists and start `go runRelocatePass(agentCtx, store, worker, participant, serialChanges)`. The worker's providers: - `Fences: hostfence.ClaimFenceAuthority{Directory: d, Now: time.Now}`; `Releases: StopReceiptReleases{Evidence: store}`; - `Admission`: claim Plan 5's app gate; - `Census: func(ctx, starter, parts) error { floor, err := hostfloor.Read(ctx, store, pub, d.Bucket); …; _, err = d.CensusBoxRelocation(ctx, hostactors.ExactEvidence{…}, floor, hostactors.Scope{StarterBox: starter, Participants: parts, Namespace: …, Regions: []string{region}, Pools: …}, time.Now()); return err }`; - `Choose`: an adapter that hides the excluded boxes behind `lost` and calls `chooser.Choose`; `Eligible: func(ctx, obs, box) error { return chooser.Permits(ctx, obs, recordOf(obs), box) }` where `chooser` is `hostops.ExecutorChooser{Directory: d, Edges: , Loads: func(ctx) { return hostops.ClaimLoads(ctx, store) }, SecretsKeyFP, HomeName, Now}` and `recordOf` builds `hostops.Record{Holder: hostclaim.NewHolder{Family, Key, Account, App, Host: obs.Host, Region: obs.Claim.Op.Intent["region"]}}` from the holder; - `Relocatable: func(ctx, obs) error`: `hostops.ReadRecord(ctx, store, obs.Claim.Holder.Key)`, refused (`hostrelocate.ErrRefused`) when the record's `Kind` is not `"instance"`, or it names a `Unit` or an `Addr` (a unit- or socket-backed route is not relocatable, as 6b's wake refuses `rt.Backing`, `rt.Unit`, `rt.Tunnel` and sidecars), or the holder's family is not `instance`; - `Leases: func(ctx, location) (hostrelocate.Lease, error) { return lease.Acquire(ctx, dataBucket, location, region) }` (`*lease.Held` has `Lost` and `Release`); - `Pins: hostrelocate.StorePinner{Claims: store, Data: dataStore, SelectDB: func(ctx, r) { return genselect.Select(ctx, versioned, r) }, Bucket: dataBucket, Region: region}`; - `Prepared: PreparedFromInstallManifest{Claims: store, Files: dataStore, Data: versioned, Bucket: dataBucket}`; - `Discharge`: ruling 3's provider (unqualified, "the roster records no enrollment time"); - `Notify: func(ctx) { _ = release.BumpSerial(ctx, store, time.Now()) }`; `Self` from `/etc/carlos/host-actor.json` (`hostfloor.ReadEnrollment`). The participant: `hostrelocate.Participant{Claims: store, Box, Enrollment, Effects: assembly.RelocationEffects(RelocationAssemblyConfig{Lifetime: agentCtx, Open: execctl.Open, Exec: , Data: dataStore, Versions: versioned, Bucket: dataBucket, Region: region, Registry: reg, FilesDir, EnvPath, DataDir: guard.DataDir, WorkDir: /relocate/build, ReleaseDir: /relocate/release, SecretsKey, Now: time.Now}), Now: time.Now}`; its `Controllers.Close` runs at agent shutdown. - [ ] **Step 4: The console's create door uses the chooser** Where claim Plan 5's create door constructs `hostops.ProductionAuthority` for the console, set `Executors` to Task 2's chooser, which is nil with `Config.RegionHA` off: ```go authority := hostops.ProductionAuthority{ /* the door's fields */ Executors: s.executorChooser(hostactors.Directory{Store: s.cfg.Releases, PublicKey: s.cfg.ReleasePub, Bucket: s.cfg.Bucket})} ``` - [ ] **Step 5: Run the tests** Run: `nice -n 19 go test ./internal/host/ ./internal/console/ -run 'RelocatePass|CreateExecutorChooser|Claim' -count=1` Expected: PASS. - [ ] **Step 6: Commit** ```bash git add internal/host/relocatepass.go internal/host/relocatepass_test.go internal/host/agent.go internal/console/ git commit -m "host: with CARLOS_REGION_HA on, every box replays its region's relocations and its own participations Co-Authored-By: Claude Opus 5.5 " ``` --- ### Task 11: The merge gate (L18) Claim rev 6.30 §1.1: "`relocating` and the rules it depends on (L1, L9 to L13) must not be enabled until region HA's own two-box-per-region model (L18) and the analysis's extra witnesses (§11) validate them." The #236 coordination rules: relocation may not merge until a joint simulation round passes on `internal/sim/hostlife` covering a one-box and a two-box region. Moves owns that model. This plan writes none of it. - [ ] **Step 1: Confirm the joint round has passed** The round's report must name `internal/sim/hostlife` at a commit containing this branch's claim transitions (Task 1) or a faithful model of them, and pass, with no open finding against relocation, for both a one-box region and a two-box region, covering spec §9's L18 list: create spread; relocation by drain; termination mid-serve and in every relocation phase (`starting`, `releasing`, `pinning`, `installing`, `installed`, `probing`, `probe-failed`); nested failover; a stale edges file; an unenrolled reference; a volume without `DeleteOnTermination`; a crash mid-install; late file uploads; and late PUTs after termination. Also claim §11's extra witnesses: a relocation start racing purge or app-placement admission closure; generation parking while relocation pins referenced files; the first `released` admission followed by an uncertified failed probe; and the one-box conversion of a pre-flip move holding at `pinning`. If any is missing or failing, stop. The branch stays unmerged; record the gap on amadan #236. - [ ] **Step 2: Run the branch gate** ```bash mkdir -p /tmp/k go build ./... && TMPDIR=/tmp/k nice -n 19 make test GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./... ``` Expected: PASS, apart from the socket tests the sandbox refuses (run those with the sandbox off, or read CI). - [ ] **Step 3: Push and read CI** ```bash git push origin region-ha-1b-next amadan ci status carlos/platform -branch region-ha-1b-next ``` Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log and fix. Merging is Paul's call, through `amadan branch merge`, after re-merging main and waiting for CI on the merged tip. --- ## After this plan | What | Waits for | |---|---| | The console materializing each `evidence_refs` entry into a D4 loss record (timer knowledge `unknown`, or `partial` from a usable snapshot), setting `reported`, and retiring the roster entry | the console's claim-recovery pass (claim D4 "Automatic loss persistence", R5) | | Automatic discharge of a fenced box's `park-local` chores | `enrolled_at` in the roster (ruling 3) | | `control/relocations/` create-only and never deleted in `deployment-buckets` | a tofu change with its policy test (ruling 5) | | The placer that decides what to relocate, `awaiting-evidence`, bounded parallel failover | spec 3 | | `carlos ops host relocate` and the console's view of a relocation | Paul's copy review | | A move into a multi-box region choosing its target with `ExecutorChooser` (L15) and a terminated move target taken off by the fence (L14) | moves Phase 1 Plans 2 and 4 | ## What the dry run could not check - Stage B (Tasks 8a, 8b, 8c, 9 and 10) was not compiled: it binds to Plan 6b rev 5's declared names, moves Plan 2 Task 8's `BuildGeneration` and key helpers, moves Plan 1 Task 7b's `RenderHeld` and `MaterializeHeld`, claim Plan 5's app gate and create door, and requests R1 to R3 and R7, none on main at 3407d114. - `StopWriters` against a live attempt (R2), root's writers-stopped endpoint (R3) and the probe through 6b's runtime (R1): their un-gated halves are tested with recorders and the fixture `/proc`; the live tests, under the production uid, are Tasks 8a and 8c's. - `WritersStopped` against a live agent with an embedded Litestream replicator. Its tests use a fixture `/proc`; the descriptor identification at replicator open belongs to Task 8. - `genselect.Select` inside `StorePinner` against a real versioned bucket. The pin tests stub `SelectDB`. - Socket-backed tests in `internal/host` inside the sandbox (the existing list in plan 1b's Global Constraints). ## Self-review - **Spec coverage.** §5.8: the start CAS and its refusals (Tasks 1, 5), the phase table and who acts (1, 3, 5, 6), the release sequence (4, 6, 7, 8), the pinned recovery set (1, 5), building P and the local install receipt (1, 6, 8), the probe admission with `KeepStamp`, per-admission coverage and the three-attempt bound's first-opening rule (1, 6), the no-reinstall boundary (1), the replacement table and the two-replacement bound (1, 5), source loss (1, 5), what a relocation leaves behind (1: `park-local` for the source and every abandoned installation). §5.9 consumption: fences read only through `VerifyFence`/`ValidateTerminationFence`, `lost`, journal reference, discharge verdict (1, 5), the census exclusion already on main (`CheckBoxStart`, `CensusBoxRelocation`). §5.10: the worker's roster read (5, `CheckBoxStart`); the participant half and root's half are main's (`hostexec`, `systemparticipation.go`). §5.2: Task 2. §5.3: Tasks 3 and 9. §10's relocation unit tests: every replacement-table row, the bound, the probe opening P with `KeepStamp` and a restarted probe taking a fresh generation, a failover declaring `writes_lost` and a drain declaring nothing, the start refused when the roster reserves an update, a `probing` CAS refused with another box's receipt, `from` terminated in `starting` before and after acknowledging. The two-agent integration test is Task 6's worker-plus-participants run against one store; the joint model is Task 11. - **Placeholders.** Tasks 1 to 7: whole files or exact `git apply` patches, all dry-run. Tasks 8a to 10: complete code against named external contracts, each name listed in its "Gated on" line, or a numbered request. - **Type consistency.** `hostclaim.Release`, `RecoveryPin`, `PreparedAuthority`, `LocalInstall`, `Probe`, `Target`, `AutomaticLoss` (Task 1) are used unchanged by Tasks 5 to 8c; `hostrelocate.Effects` (Task 6) is implemented by `*host.RelocationEffects` (Task 7, `var _ hostrelocate.Effects`); `hostclaim.ReleaseAuthority` by `host.StopReceiptReleases` (`var _`); `hostrelocate.Lease` by `*lease.Held` (Task 10); `hostops.ExecutorChooser` (Task 2) by Task 10's worker adapters and the console constructor; `hostbox.RelocationRow` (Task 3) by Task 9; `hostrelocate.PinObject`, `ReadPin` and `CopyPinnedManifest` (Task 5) by Tasks 8b and 8c; `ReleaseFlushFiles`, `ReleasePositionsFile`, `RelocationControllers` (Task 7) by Task 8a. - **Review Focus.** Each of the five has its test in the owning task. The dry run removed these guards one at a time and watched a test fail: `released`'s predecessor box; P's single opening; the failover pin delay (claim and worker); the pin reaching the release; the replacement bound; the nested failover after P; the receipt bound to `op.exec`; create's no-substitution; the start guard's live-source chore and region `park-replica`; the row case's closed fence and `lost`; the chooser's fence exclusion; discharge only when qualified; the writers-stopped thread-table check; the release receipt's obligation; the refusal of a replayed receipt; and a failed files flush stopping the release. Nineteen in all. - **Rev 2 guards**, each removed in turn and its test watched to fail: the pin key bound to the claim generation; `ToInstalled`'s files manifest; the empty pin's evidence (claim); `stuck` before the phase table; the retained automatic loss; the lease lost before the pin CAS; the release bounds before the write (two); the empty pin's evidence (pinner); the pinned manifest's digest; the copied entries' equality; the target's eligibility at start; the session-0 drain refusal; `stuckOn` from `probe-failed`; the release deadline; the participant's readiness gate; `StopWriters` first in the drain; the replayed receipt without a record; the positions' session binding; the controller under the agent lifetime; the wake state in the flush; `Permits`; today's create without a selection; the `reconcile` action's participant and lost checks; the console's chooser off with HA off. Twenty-five in all. - **Rev 3 guards**, each removed in turn and its test watched to fail: the probe attempt bound; discharge through `CompleteObligation`; the former writer's last admission; the per-process descriptor scan; exhaustion before a spare is chosen; the retained manifest's digest; the attempt counted before the probe; the sweep's running-operation check; the recorded identity after a restart; the positions' box binding; `Finish` forgetting the controller. Eleven in all. - **Rev 4 guards**, each removed in turn and its test watched to fail: the acknowledgement barrier on `releasing → pinning`; `FenceExhaustedTarget`'s exhaustion check; the worker's use of it over `MarkStuck`; the serving branch's obsolete-hold clearance; `RelocationPolicy`'s executor check; `AdmissionFenced`'s active fence; `Finish` keeping its entry until `Close` succeeds; the recorded directory revision after a restart. Eight in all. - **Rev 5 guards**, each removed in turn and its test watched to fail: the lock-free fence observer (re-locking deadlocks the bounded test); the `probe` action in `Policy.Check`; the HA gate on the obsolete-hold clearance; `Finish`'s durable clear; the journal's retry after a restart; the hold-only path for a fenced executor; the observer error keeping rows. Seven in all. - **Rev 6 guards**, each removed in turn and its test watched to fail: `Finish`'s repeated scoped cleanup before every clear (the real hostexec test's sweep after a failed cleanup clears); the participation journaled before the open; the journal listing's `.tmp` exclusion; a damaged committed entry keeping the readable ones; `Sweep` reporting `Retained`'s error; the env file's staged deletion; `replaceTargetSlot`'s `park-local` refusal; the worker's `park-local` exclusion; the chooser's `park-local` exclusion; `fenceBox`'s regional reassignment; Task 9a's serving-only check with HA off. Eleven in all. - **Rev 7 guards**, each removed in turn and its test watched to fail: the `KCMP_FILES` shared-table check (the real kernel's unshared thread with the copied descriptor passes as the leader's); the `KCMP_FILE` description check (the fixture's shared table with another description passes); an unavailable kcmp failing closed (ENOSYS and EPERM pass instead of `ErrIncomplete`). Three in all. - **Rev 8 guards**, each removed in turn and its test watched to fail: `Release` marking a same-controller retry with recorded positions as resumed; the resumed drain skipping straight to the producers' drain. Two in all. - **Rev 9 guards**, each removed in turn and its test watched to fail: `FenceExhaustedTarget` keeping a live source's `op.exec` and phase (the `starting` and `releasing` cases revoke it); `Release` keeping the recorded directory revision whenever positions are recorded (the same-controller retry asks at the new revision). Two in all. - **Rev 10 guard**, removed and its test watched to fail: `Attempt` creating with the selection it made (`TestCreateIsTodaysWithoutASelection`: the completion substitutes box-a). The in-order dry run is the check for finding 1. One in all. - **Rev 11 guards**, each removed in turn and its test watched to fail: the console chooser's live-edges reader (an empty snapshot fails on the box identities); its claim-loads reader (an empty map fails on the counts); `ProductionAuthority.SelectExecutor` consulting its chooser (a fixed box-a fails); `regionHAEnabled` honouring the parsed value (any parseable value as on fails on `0`). Four in all. - **Rev 12 guards**, each removed in turn and its test watched to fail: the pin's referenced origins (the copy refuses g1's object with `ErrOrigin`); the refusal of a reference outside the incarnation's files (a foreign reference is pinned); `preparedOpened` reading `prepared_opened` (P is forgotten with its admission); the opening admission journaling it. Four in all. - **Rev 13 guards**, each removed in turn and its test watched to fail: `Acknowledge`'s handover (B's acknowledgement waits out the pass); `openBounded` (the open under the agent's lifetime never returns: the test's own timeout); the superseded case for hostexec's `ErrInactive` (the handover fails); removing the superseded journal entry once B opens (A is still retained). Four in all. - **Rev 14 guards**, removed one at a time under `TestRelocationControllerLifecycleProperties` with 200 seeds (the table in the Rev 14 dispositions): ten killed, each with its seed and invariant. One, the extra scoped cleanup before a recovery reopen, survives as defence in depth, because `hostexec.Open` runs the same cleanup and refuses on its failure. Rev 13's guard "removing the superseded journal entry once B opens" is gone with that code (C3). - **Rev 15 guards**, removed one at a time (the table in the Rev 15 dispositions): twelve killed. The survivors are defence in depth: `EnsureLocal` before entering probing is covered by the attempt-time check, and `EnsureLocal`'s own short-circuit for an opened P is covered because the participant never calls it after P opens. - **Rev 16 guards** (the table in the Rev 16 dispositions): the stuck-on-lost-receipt step and the exact receipt comparison are killed by their tests; both `EnsureLocal` calls removed together are killed by the property, seed 182. Rev 14's lifecycle guards are killed again under the crash event. - **Rev 17 guards**, each removed in turn and its test watched to fail: the abandoned-P journal in `install`; the serving CAS's park; the opened-P exclusion; `filesPin`'s refusal; the release-evidence exception; the worker's `stuck` on `ErrFilesUnevidenced`; the `probe-failed` bound; the wait before it. Eight in all. - **Rev 18 guards**, each removed in turn and its test watched to fail: the keyless-manifest refusal; the key required only for a nonempty database; the dead-target-without-spare bound; that bound consuming the fence; `ReplaceTarget` recording the unopened P it drops. Five in all. - **Rev 19 guards**, removed and their tests watched to fail: `Sweep` finishing an excluded participation (the property, seed 4); the record's host as the chooser's hash input (`TestExecutorChooserHashesTheRecordsHostOnEqualLoads`). Two in all.