# Instance moves: two-phase, data-following (issue #331) Status: DRAFT for review — 2026-08-28 Companion issues: #329 (region on create — shipped separately), #330 (replica_regions — shape only, untouched here), titogo residence design §5/§8.3 (the consumer contract this serves). ## Decisions taken in conversation (Paul, 2026-08-28) - Cross-bucket IAM is **push-only into a staging prefix** (`moves/`), never into `instances/` — a box can stage state toward a sibling region but can never touch another region's live replica lineage. - The platform moves **the instance DB and record only**. The uploads bulk-copy stays with the tenant (titogo's router), which owns the bucket, the credentials, and the knowledge of which keys belong to which instance. The move state machine exposes the phases so the tenant can sequence its own copy inside the frozen window. If a per-instance uploads prefix ever becomes a declared platform concept (a record field), the move widens to carry it without unwinding this design. ## The invariant *Residence is a property of a route; the record's `region` field is the residence carrier* (ingress spec §Anticipated #2). This spec never adds a second arbiter: at every instant exactly one region's box is in scope for the record, and the flip of `region` is the LAST act of the source. Never two writers falls out of that plus the freeze the tenant holds across the cut. ## Wire shape The record gains one additive, omitempty object — present only while a move is in flight: ```json "move": { "to": "uk", "state": "preparing" | "ready" | "cutting" | "aborting", "requested_at": "", "ready_at": "= ready>" } ``` Records written before this field parse unchanged (wire-shape rule). `move` is written by the console (the two verbs below) and by the SOURCE box (the `preparing → ready` promotion and the terminal clear-and-flip); each write is the narrow CAS `SetRecordRegion` already uses. The record's `region` stays the SOURCE region until the cut completes — every reader that knows nothing about moves keeps a consistent answer throughout. ## The two verbs Console API, on one instance, gated exactly like every other instance door (app bearer; the tenant's router holds the token — never the instance itself): - `POST .../instances/{host}/move` body `{"region":"uk"}` — validated against the armed-placement set (#329's gate, mandatory here: a move to an unarmed region is always a 400, dormant never applies because a move NEEDS a live target box). Refused 409 if a move is already in flight, 400 if `to` equals the current region. Writes `move:{to,state:"preparing"}`. - `POST .../instances/{host}/move/cut` — refused 409 unless `move.state == "ready"`. Writes `move.state:"cutting"`. The tenant calls this only after freezing writes; the platform does not verify the freeze (it cannot — the freeze is app-level), it documents it. - `DELETE .../instances/{host}/move` — abort, legal in `preparing` or `ready`. TWO-PHASE (review round 1): the door marks `aborting`; the SOURCE box deletes the staged copy from the target bucket and then clears `move` — a bare clear would leave staging behind to win a later landing on the target (staging-wins below). Refused 409 in `cutting` (completes forward) and in `aborting` (already under way). - State is read, not pushed: the existing single-record GET already returns the record verbatim, so `move.state` is pollable today with zero new read surface. - Every move write bumps `control/serial` (the config fast-convergence nudge the env door already uses), so the source box notices in seconds, not on the hourly floor. Between states the source's own move worker self-paces its delta rounds. ## What the source box does The reconcile/activator pass on the box whose `CARLOS_REGION` matches `region` (the source — the record still points at it): 1. **preparing** (AMENDED during planning, 2026-08-28: the draft said "a second litestream replica of the live DB" — two litestream processes cannot tail one WAL, both want to own checkpointing. The fallback IS the design, and it is better): the move copies the REPLICA, never the database. The activator already keeps `instances//` in the source's own bucket continuously synced (running child) or final-checkpointed (parked child) — the move worker mirrors that prefix into `s3:///moves//` with server-side CopyObject (its IAM: read own bucket, push sibling staging), in delta rounds every ~30s — List both sides, copy new/changed keys, delete keys the source pruned (a true mirror, so litestream retention holds in staging too). Uniform for running and parked instances; no wake, no litestream internals touched. Target bucket name and AWS region are derived, not configured per-move: each box publishes its data bucket coordinates as two new additive fields in its edges file (`data_bucket`, `data_bucket_region` — same additive rule as `placement`), so the source reads the target's coordinates from the same file set everything else reads. 2. **ready promotion**: once a delta round completes with zero differences against a fresh listing, the source CASes `move.state: "preparing" → "ready"`, stamping `ready_at`. Rounds keep running — ready is a level, not an edge; writes after ready keep flowing into staging on the next round. 3. **cutting** (order amended, review round 1): on seeing `state:"cutting"`, the source (in order): set the route's Status to "moving" (a non-ready status: the edge stops serving it, the edges file drops it, and the activator's RetireExcept sweep stops the child through the ordinary stop path — which final-checkpoints to the source's OWN bucket — but SWALLOWS a checkpoint failure, safe only while the warm cache stood behind it) → wait for the child to be down → **re-checkpoint the local DB with the lease and wait** (proving the own replica carries the tail — review round 1, the failed-final-checkpoint data-loss hole) → one final delta round into staging (now byte-identical to the own replica) → **delete the local DB, WAL/SHM, and litestream meta dirs** (the warm-cache purge; without it a later move back would resurrect a stale local file — the activator's warm-cache trust is exactly right for crashes and exactly wrong here) → one CAS that sets `region: ` and clears `move` → remove its own route row on its next pass (the record has left its scope, the existing retirement path does this unprompted). If the box dies mid-cut, the record still says `cutting` with the old region: the source resumes the sequence on restart (every step is idempotent — sync again, stop a stopped child, delete absent files, CAS already-flipped is a no-op conflict it re-reads). ## What the target box does Nothing move-specific until the flip — then its ordinary pass sees a record newly in scope and mints the route. One addition in the activator's `Restore`: when the local DB is absent AND `instances//` in its own bucket has no replica, it checks `moves//` (own bucket, own IAM); if present, server-side copy `moves// → instances//`, restore from there, `CheckDB` as always, and delete `moves//` on success. A corrupt staged replica quarantines exactly like a corrupt own one — and the abort story still stands, because the source's last own-bucket checkpoint is intact and `region` can be pointed back by hand (`SetRecordRegion` — the operator recovery door that already exists). ## IAM (the infra half — platform-infrastructure + tito repo applies) `region-host` (and `region-remote-host`) gain an optional `sibling_staging_arns` list; when set, the role gains `s3:PutObject` on `/moves/*` — and nothing else: no Get, no List, no Delete, never `instances/*`. Residency-at-rest holds (cross-region reads remain impossible at IAM); the write blast radius is confined to staging keys no box ever trusts as live state without its own restore-time `CheckDB`. The mirror needs three actions on the sibling, all `moves/*`-scoped: `s3:PutObject` and `s3:DeleteObject` on the object ARNs, and `s3:ListBucket` on the bucket ARN with an `s3:prefix` condition of `moves/*` — still no reads of sibling data, still nothing outside staging. ## Instance env Already shipped with #329: every exec child gets `CARLOS_REGION` (the box's placement code), agent-owned in both directions. An app that lands on the target box after a move reads the new value at boot — titogo's boot-clears-freeze rule needs exactly this. ## Out of scope - Uploads (tenant-side, by decision above). - `replica_regions` behavior (#330 stays shape-only). - Moving unit-backed, addr-backed, static, or everywhere-class routes: the verbs refuse anything but an exec-backed `kind:"instance"` route (400 naming the kind) — the only class whose state the platform owns. - Fleet placements (`fleet/`): refused; fleets keep D6 state-does-not-follow semantics on purpose. - Pooled records (`pool != ""`): refused in v1 — their data lives on the pool's box, not the platform region box that would drive the move. Non-hibernating instances (`hibernate:false`): refused — they run outside the activator, so nothing manages (or could quiesce) their data. - The worker acts only on routes its OWN registry serves: region equality alone would let a standby or second same-region box drive a cut whose child runs on its sibling (per-box registries make its Quiesced trivially true). - Automatic freeze verification; region-aware unit economics; the member-facing residence door (titogo builds on these verbs). ## Testing - Record CAS: move-field transitions (prepare/ready/cut/abort/conflict) against the fake store, including the 409 matrix. - Source pass: fake-store + fake-litestream harness proving the cut ordering (final sync BEFORE stop BEFORE local purge BEFORE flip), the parked-instance prepare (no wake), resume-from-`cutting` idempotence, and that abort leaves the source serving with its own replication intact. - Target restore: staging fallback (copy, restore, CheckDB, cleanup), corrupt-staging quarantine, and precedence (own `instances/` replica beats staging when both exist — a re-landed instance must not regress to an older staged copy). - Console: verb gates (bearer, kind, armed region, state machine), poll-through-GET. - The never-two-writers argument pinned by an ordering test: a request hitting the source during `cutting` after stop is a 503 from the edge (route gone on next pass) while the target has not yet minted. The length of that window is the target's reconcile latency, NOT assumed: measure it during build (the adopt path/timer cadence is the floor) and record it in the runbook — the tenant's freeze must be budgeted to cover it, and if it measures in minutes rather than seconds, a serial-bump nudge on the record write is the follow-up, not a redesign.