# πŸ€– Multi-edge β€” region as a property, relocation, and replicated spaces Status: **direction agreed 2026-07-18** (Paul). This is the plan of record for how a space is *placed* (which box serves it), *relocated* (moved between regions), and eventually *replicated across edges* (served from several boxes at once). It builds on `docs/planet-scale.md` (the running single-owner system) and supersedes the "country in the hostname" scheme the dev mesh trialled (`docs/dev-mesh.md`, PR #1043; described today by the `docs/planet-scale-guide/` proving-ground page); see [What this supersedes](#what-this-supersedes). The north star is **multi-edge**: a space live on several edges at once, so every member reaches a nearby one. Two interim steps stand on their own and lay the groundwork: **stable-address relocation** and **multi-edge within a single region**. ## The one idea: region is a property, not an identity A space's **address is its identity** β€” the name members type, the host its invite links carry, the origin its per-thread encryption keys are bound to. Identity must never change, because changing it re-keys every member. Where a space *lives* β€” its **region** β€” must be free to change: auto-assigned at creation, movable later by the space's admin, and (the endgame) spread across several places at once. So region **cannot be encoded in the address**. It is a separate, mutable property of the space β€” a `home_region` field β€” and the address stays region-neutral and permanent (`.11ch.at`, one wildcard, no country label). This is the `home_region` model `docs/planet-scale.md` always pointed at: placement and residency are properties the infrastructure moves a space *to*, never facts baked into its name. ## The enabling property: append-only + content-blind is a CRDT Everything ambitious below is tractable for one reason, and it is worth stating plainly because it is load-bearing: **our data model is an append-only log of immutable, opaque, id'd events.** Messages, edits, deletes, and reactions are all *events* β€” a delete is a new event, never an in-place row mutation; a reaction rides inside an event. The server stores and relays only ciphertext it cannot read. An append-only set of id'd events is, formally, a **CRDT** (a grow-only set). Merging two boxes' logs is *union β†’ dedup by id β†’ sort by (timestamp, id)*. There is no conflict to resolve, no data to lose, no coordination to run, and it converges regardless of the order events arrive in. The server was never going to do content-based merging β€” it cannot see the content β€” and it does not need to. This is what removes the **hardest categories** of distributed-systems work from the table entirely: - no consensus / leader election for writes, - no write-conflict resolution, - no content merges or last-writer-wins data loss. What is left is *eventual-consistency plumbing* β€” a much lower tier of difficulty. That is the whole reason multi-edge is a road we can walk rather than a moonshot. ## Phase 1 β€” stable-address relocation An admin moves a space to another region; its address, invite links, and keys do not change. Members notice nothing. The mechanism reuses parts that already exist: 1. **Copy** the space's store to the destination region's object store. (In the dev mesh there is one shared bucket, so this is free; in production each country has its own store, so this is a prefix-to-prefix copy β€” done while the space sleeps, per `docs/planet-scale.md`.) 2. **Flip** `home_region`, which replicates to every box the same way routing rows already do (`meshreplicate.go`). 3. **Update DNS** β€” the space's A record now points at the destination box (see [Dynamic DNS](#dynamic-dns-and-the-propagation-window)). ### Dynamic DNS and the propagation window Keeping the *name* fixed while the *serving box* changes is exactly what API-driven dynamic DNS is for (we already use it for DNS-01 wildcard certs): the host stays `.11ch.at`, only its A record's target moves. The DNS TTL means a propagation window where some clients still resolve to the old box. **No live replication is needed to cover it** β€” the [single-writer lease](planet-scale.md) plus the mesh's proxy-to-owner already do. A request that lands on the old box finds `home_region` now points elsewhere and **proxies to the new owner**, exactly as it already does for any wrong-box request. The lease guarantees only the new box ever writes, so there is no dual-writer and no split-brain β€” the window costs at most a proxy hop, not correctness. So Phase 1 is mostly existing machinery (hibernation + lease + proxy + the route-replication plumbing) plus the `home_region` field, region-neutral addressing, and placement-by-region. It is the smallest change that delivers the stable-address promise. ### The `lchat relocate` command The relocate action is a subcommand of the one binary β€” `lchat relocate --to ` (`relocate.go`, a reusable `relocate()` core a future controld RPC or member-facing "change region" button can call). It runs the three steps above in order: 1. **Store copy (or the dev no-op).** A pure decision helper, `relocationStorePlan(src, dst)`, returns `skip=true` exactly when source and destination name the same bytes β€” same bucket, same per-host prefix, same endpoint. That is the **dev mesh's one-shared-bucket case**, so the copy is a no-op there. In production each region has its own store, so it is a real prefix-to-prefix copy of `s3:////…` (`copyS3Prefix`), run while the space sleeps. The copy deliberately **excludes the lease object** (`lock.json`): a copied, still-valid lease would make the destination's first `AcquireLease` see a held lock and 503 until the TTL expired β€” the destination must take its own fresh lease. 2. **Flip `home_region` + re-fan-out.** `setHomeRegion` writes the replicated routes column and `fanOutMeshRoute` pushes the updated route to peers, so every box's `candidatesFor` (`mesh.go`) re-homes the space onto a box in the destination region. Route *edits* aren't auto-replicated by the normal path, so this is the deliberate targeted re-fan-out of the single updated route. The local router needs **no reload** β€” its mesh reads `home_region` live from the shared registry on each placement decision. 3. **Optional A-record move.** `--move-dns --dns-target ` repoints the host's A record at the destination box to avoid the proxy hop. **Off by default** β€” the `*.` wildcard + proxy-to-owner already route any request to the new owner through the DNS-TTL window, so relocation is correct with zero DNS write perms. Only the Route53 provider implements the A upsert today (`route53dns.go`); DNSimple refuses `--move-dns` with a clear message. **Preconditions the command enforces:** the space must be an in-scope, `ready`, **hibernating** instance (a live systemd instance is box-pinned β€” the message tells you to `lchat hibernate --on` first), and a **live mesh box must advertise the target region** (else the flip re-homes onto nothing). Run it on a box whose registry has the route (the source region's box) with the mesh env sourced (`/data/router.env`) β€” `sudo` strips `LCHAT_MESH_*`, the same gotcha the dev-cutover runbook's Step 5 calls out. **Why the single-writer lease makes it data-safe.** Relocation never split-brains because the S3 lease (`internal/activator`, `lock.json` under the space's prefix) is the one truth of who may write; `home_region` and DNS are only placement hints: - *Shared bucket (dev):* both regions contend on the **same** `lock.json`, so the lease is globally singular. After the flip the destination is the HRW owner; if the source is momentarily still awake it holds the lease, the destination's `AcquireLease` fails loudly (`LeaseExistsError` β†’ 503 + Retry-After), and the client's retry lands once the source sleeps and releases. Never two writers. - *Per-region buckets (prod):* the copy runs while the source is **asleep** (lease released, replica fully flushed by the sleep-time `SyncAndWait`), so the destination restores a complete copy and takes a fresh lease in its own bucket. Once `home_region` is flipped the source's `owns()` returns false, so it never wakes there again β€” a stray request proxies to the new owner instead. **Idempotency + the copy-clobber guard.** Once `home_region` already equals the target, the space's authoritative copy is the **destination's**, so a re-run must **not** re-copy sourceβ†’dest (that would overwrite newer destination writes with stale source bytes). The copy + flip therefore run **only on the forward move** (`current != target`); an already-home re-run just heals peers by re-fanning-out the route. A crash between copy and flip converges safely β€” the source is still authoritative (not yet flipped), so the next run re-copies (an idempotent overwrite) and flips. **Tested vs. needs live proof.** Unit tests cover the pure store-plan decision (skip vs copy), `copyS3Prefix` (key layout preserved, `lock.json` excluded), the `home_region` flip + idempotency guard, target-region validation, and the Route53 A upsert. The **dev shared-bucket path (skip) is what a live dev relocation exercises**; the **prod cross-bucket copy is unproven in production** β€” it needs a per-region-bucket dev/prod trial, and two design points remain open: regionβ†’bucket auto-discovery (today the operator names the destination with `--dst-bucket`) and cross-*account*/cross-*endpoint* (R2) copies, which need a download+reupload variant rather than server-side `CopyObject`. Both are follow-ons. ## Phase 2 β€” multi-edge within a region Replicate a space's log across the boxes of *one* region. Same-jurisdiction, so **no residency promise is affected** β€” this is pure resilience and latency, and it is where the merge/gossip layer gets validated on the easy case (all edges in one country, no cross-border concerns). Each edge accepts appends locally and propagates them to its peers; the CRDT merge means every edge converges without coordination. The single-writer lease β€” which today exists to stop two processes corrupting one SQLite file β€” relaxes into **per-edge logs that merge**, rather than one file one writer. A box failing no longer strands its spaces mid-write; its peers already hold the log. ## Phase 3 β€” cross-region multi-edge (the goal) The same replication, now spanning regions: a space served from edges in several countries, each member reaching the nearest. This is the north star β€” and the one place a real trade-off surfaces, below. ## What append-only does *not* solve Honest remaining work β€” the eventual-consistency plumbing the CRDT leaves us: - **Ordering / causal consistency.** Union+sort converges, but "a reply appears after its parent" and "read-your-writes" want causal metadata (a per-sender sequence or a parent ref) carried *inside* the encrypted event. Today there is a real server-assigned monotonic sequence (the subscription watermark); multi-edge moves that to a partial order (timestamp+id, or Lamport clocks) instead of one counter. Chat tolerates weak ordering well, so this is manageable, not scary. - **Ephemeral fan-out.** Typing, presence, and relay-only frames are *not* log entries β€” they are live signals, and the WebSocket hub is per-box today. Multi-edge needs a cross-edge pub/sub for them. New, but small. - **Real-time delivery.** A message appended at edge A must reach a subscriber on edge B *now*, not at anti-entropy time β€” so edges want fast push to each other, not only periodic reconciliation. - **Server-side watermarks.** Read-marks are already forward-only max-wins (CRDT-friendly). Scheduled future-messages (the sweep goroutine, `miniapps.go`) need one edge to own the timer or idempotent delivery. A handful of these, each solvable. - **Cost.** N copies of the data and gossip traffic β€” a real but bounded price. ## Residency βŸ‚ multi-edge β€” two modes, not both Multi-edge and data residency pull in opposite directions: multi-edge **spreads** the ciphertext across edges (and countries) for reach and resilience; residency **pins** it to one country. They cannot both hold for the same space at once. So they are **modes a space's owner chooses**, not a global setting: - **Residency** β€” one region, data stays put; the guarantee a country subdomain can make. - **Multi-edge within a jurisdiction** β€” the sweet spot: resilience and lower latency with *no* residency promise broken. - **Cross-region multi-edge** β€” maximum reach, explicitly trading the residency guarantee for it. Relocation (Phase 1) is available in every mode β€” it is just "change where the one copy lives" in the residency/single-edge case. ## What this supersedes The dev mesh trialled **country in the hostname** (`..dev.11ch.at`, per-country wildcards, placement scoped by the host's country label β€” the domain-scoped mesh in `mesh.go`/`heartbeat.go`/`meshreplicate.go`). That made region an *immutable identity*, which is precisely what the stable-address requirement rules out. It was a useful stepping stone; this model replaces it: - **Addresses go region-neutral** (`.11ch.at`, one wildcard). The country leaves the name. - **Placement keys on `home_region`, not the host label.** Each box advertises its region in its mesh heartbeat body; every box learns its peers' regions and scopes placement to the box(es) whose region matches a space's `home_region` (`mesh.go` `candidatesFor`, keying on the `home_region` the registry already replicates onto every box). The country-in-host trial's domain-scoping lived in an unmerged branch, so there was no prior per-box advertisement to reuse β€” the region rides the heartbeat every box already reads for membership, and the fetch is skipped entirely on a box that declares no region (zero cost when a mesh has not opted into regions). - `docs/dev-mesh.md` and the `docs/planet-scale-guide/` "proving ground" page describe the country-in-host scheme that runs on the dev mesh *today*; they stay accurate until the phases below land, then are reworked in step. ## Status and build order Phase 1 is now largely built; the pieces it reuses were always running. - The **append-only, content-blind data model** it rests on: *running today* β€” it is how every space already works. - **Hibernation + single-writer lease + proxy-to-owner** (the machinery Phase 1 reuses): *running today* (`docs/planet-scale.md`). - **Region-neutral addressing + `home_region` placement** (Phase 1): *built + proven live on the dev mesh* β€” one shared `dev.11ch.at` namespace, a shared wildcard, cross-region serve returning 200 (the dev cutover runbook, `docs/multi-edge-dev-cutover.md`, lands with the cutover PR). - **The `lchat relocate` action** (Phase 1's last slice): *built; the dev shared-bucket path (skip-copy) lands on the proven cross-region serve; the prod per-region-bucket copy is implemented + unit-tested but unproven in production* (see [The `lchat relocate` command](#the-lchat-relocate-command)). - **Multi-edge within a region** (Phase 2) and **cross-region multi-edge** (Phase 3): *designed, not built*. Build order, each its own reviewable slice: 1. **Phase 1 β€” stable-address relocation.** `home_region` field, region-neutral addressing, mesh places by region, relocation via lease + proxy + cross-region copy + dynamic DNS. Mostly existing parts; the foundation everything else sits on. 2. **Phase 2 β€” multi-edge within a region.** The log-merge / gossip layer, validated where there are no residency concerns. 3. **Phase 3 β€” cross-region multi-edge.** The same, spanning regions, as an explicit owner mode.