# 🤖 Multi-edge Phase 1 — dev cutover runbook (region-neutral addressing)

The ordered, hand-runnable steps to flip the two-node dev mesh onto
**region-neutral addressing**: one shared `dev.11ch.at` namespace, a shared
`*.dev.11ch.at` wildcard cert over Route53 DNS-01, and cross-region serving that
returns **200** (not the 503 the country-scoped mesh gave). Design + rationale:
`docs/multi-edge.md`. The tofu it drives: `infra/dev-mesh/`.

This is a **dev-only, reversible** change. Prod (`11ch.at` on DNSimple) is
untouched except for one scoped `dev` NS delegation record.

> 🤖 **Historical record (2026-07) — not a procedure.** This page documents
> the cutover as it was performed and the gotchas learned doing it; it is
> kept as design history. **Do not operate from it.** The dev mesh is
> operated by `./deploy/ops dev plan|apply|deploy|rebuild` — `rebuild` now
> writes the env files below, pairs the mesh token, and seeds routes itself.
> Anything the verbs can't do gets **added to them** (extend ops, then run
> it), never hand-run from this page.

---

## The end state (what "done" looks like)

- New spaces mint as `<sqid>.dev.11ch.at` — no country label — their ids drawn
  from the shared global ledger (`internal/idalloc`), so two region clouds never
  collide.
- **Both** boxes serve the single mesh domain `dev.11ch.at`, hold the shared
  `*.dev.11ch.at` wildcard cert, and place every space by its replicated
  `home_region` (mesh.go). Either box can receive any region-neutral host and
  either serve it (its region owns it) or reverse-proxy to the box that does.
- Cross-region serve returns 200: a `home_region=fi` host hit at Dublin is
  proxied to Helsinki, which terminates TLS with the shared wildcard, finds the
  replicated route, and serves.

Why this fixes the 503: today the two boxes serve **different**
`LCHAT_MESH_DOMAINS` (`ie.dev.11ch.at` vs `fi.dev.11ch.at`). A cross-region host
is in-scope on the sender but out-of-scope on the receiver, so the receiver
serves locally with no route and no cert → "acme: host not in registry". One
shared domain makes it in-scope on both, plain domain-scoped route replication
copies every `*.dev.11ch.at` route to both boxes, and the shared wildcard lets
the receiver terminate the proxied HTTPS hop. (Region-*aware* replication —
branch `multi-edge-p3-repl` — is only a scale optimisation for many boxes on
different domains; two boxes on one domain don't need it.)

---

## Boxes, as found (2026-07-18)

| | Dublin (ie) | Helsinki (fi) |
|---|---|---|
| reach | `ubuntu@54.154.95.205 -i ~/.ssh/eleven-dev-box` (passwordless sudo) | `root@157.180.123.183` (1Password hetzner agent key) |
| arch | arm64 (t4g.small) | amd64 |
| AWS creds | EC2 **instance role** `eleven-dev-mesh-ie` (no keys on box) | scoped IAM **user** `eleven-dev-mesh-fi` (`AWS_ACCESS_KEY_ID`/`_SECRET` in router.env) |
| router svc | `lchat-router` → `/opt/lchat/lchat` | `lchat-router` → `/opt/lchat/lchat` |
| cloud/control | `lchat-cloud`/`lchat-control` → `/opt/lchat/lchat-banner` (**separate binary!**) | → `/opt/lchat/lchat` (unified) |

⚠️ **Both boxes' `/opt/lchat/lchat` were rebuilt by hand on Jul 18** with
ambiguous stamps (Dublin's reports version `dev`). Don't trust the in-place
binaries — **Step 0 rebuilds both from a known main SHA** so the running code is
certain. On Dublin, `lchat-cloud`/`lchat-control` exec a *different* binary
(`lchat-banner`, `f3fa623b`); this cutover unifies them onto the fresh binary so
the cloud has the id-ledger + region-neutral minting code (that binary predates
`#1103`/`#1109`). Backups already on each box: `lchat.bak-mesh` (dcc83d0a),
`lchat.bak-multiedge` (90a6dee0).

All of `#1093 #1094 #1103 #1106 #1109` (placement, provisioner stamp, id ledger,
Route53 provider, region-neutral minting) are on **main** — a plain build of
current main has everything.

---

## Step 0 — build fresh binaries from main (both arches)

From a clean `main` checkout (record the SHA — it stamps the build):

```sh
# Dublin is arm64, Helsinki amd64 — build both.
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -trimpath \
  -ldflags "-X main.buildVersion=$(git rev-parse --short HEAD)" -o lchat-linux-arm64 .
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath \
  -ldflags "-X main.buildVersion=$(git rev-parse --short HEAD)" -o lchat-linux-amd64 .
# (deploy/build-linux.sh builds amd64 the same way; arm64 needs the explicit GOARCH.)
```

Ship + stage (do NOT overwrite the live binary yet — stage beside it):

```sh
scp lchat-linux-arm64 ubuntu@54.154.95.205:/opt/lchat/incoming/lchat-new
scp lchat-linux-amd64 root@157.180.123.183:/opt/lchat/incoming/lchat-new   # (mkdir -p /opt/lchat/incoming first if absent)
```

The binary swap itself happens in Step 3 (after the tofu + DNS are ready), so
router and cloud restart once, onto both the new binary *and* the new env.

---

## Step 1 — apply the Route53 tofu (from your Mac, with the vault token)

`infra/dev-mesh/` now creates the `dev.11ch.at` Route53 zone, its three
wildcards (`*.dev`, `*.ie.dev`, `*.fi.dev`), the `dev` NS delegation into the
prod DNSimple zone, and the DNS-01 IAM on both boxes' role/user. The DNSimple
176940 token comes from the vault at apply time — **never a box**.

```sh
# `ops dev plan`/`apply` sort the credentials out themselves (AWS_* +
# DNSIMPLE_* from Bitwarden — the sops vault is retired):
./deploy/ops dev plan                # expect: 10 to add, 0 to change, 2 to destroy
./deploy/ops dev apply
```

The plan (validated 2026-07-18): **+** zone, `*.dev`/`*.ie.dev`/`*.fi.dev`
records, 4×`dev` NS, 2× DNS-01 IAM policies; **−** the two old DNSimple
`*.ie.dev`/`*.fi.dev` A records (superseded by the ones now inside the delegated
zone). Nothing else changes — the box, bucket, existing IAM, and the
`home.dev.elevenmessenger.com` record are untouched.

Note the zone nameservers for the next check:

```sh
AWS_PROFILE=eleven-dev tofu output dev_zone_nameservers
```

---

## Step 2 — verify delegation propagates

Delegation TTL is short but resolvers cache the parent NS; give it a few minutes.

```sh
dig +short NS dev.11ch.at            # must return the 4 Route53 nameservers
dig +short one.ie.dev.11ch.at        # must still be 54.154.95.205 (now via Route53)
dig +short one.fi.dev.11ch.at        # must still be 157.180.123.183
dig +short anything.dev.11ch.at      # must be 54.154.95.205 (the *.dev wildcard → Dublin)
```

Do NOT proceed to the wildcard cert (Step 3) until `dig NS dev.11ch.at` shows
Route53 — Let's Encrypt validates the DNS-01 TXT against the authoritative
nameservers, which are Route53 only after delegation lands.

---

## Step 3 — reconfigure + swap the boxes

Env deltas (add/modify; everything else stays). Edit `/data/router.env` and, on
Dublin, `/data/cloud.env`; then swap the binary and restart.

### Dublin (ie) — `/data/router.env`
```
LCHAT_MESH_DOMAINS=dev.11ch.at            # was ie.dev.11ch.at
LCHAT_DNS_PROVIDER=route53                # new — the AWS DNS-01 off-ramp
LCHAT_WILDCARD_DOMAINS=dev.11ch.at        # new — obtain *.dev.11ch.at
# unchanged: AWS_REGION=eu-west-1, LCHAT_HIBERNATE_BUCKET, LCHAT_MESH_REGION=ie,
#            LCHAT_MESH_SELF=https://one.ie.dev.11ch.at, LCHAT_MESH_TOKEN
```

### Dublin (ie) — `/data/cloud.env`
```
LCHAT_CLOUD_BASE_DOMAIN=dev.11ch.at       # was ie.dev.11ch.at — mint <sqid>.dev.11ch.at
LCHAT_CLOUD_ID_LEDGER=1                    # new — draw ids from the global ledger
LCHAT_HIBERNATE_BUCKET=eleven-dev-mesh-427880590669  # new — where the ledger object lives
LCHAT_HIBERNATE_REGION=eu-west-1          # new — so idalloc's S3 client resolves the region
# unchanged: LCHAT_CLOUD_REGION=ie, LCHAT_ADMIN_TOKEN, demo/pool vars
```
> `dev.11ch.at` is a **fresh** namespace (no spaces minted under it), so the
> ledger counter starts at 0 → first id 1 → no collision. **No counter seed
> needed.** (Seeding is only required when flipping the ledger on for a
> namespace that already has spaces — see `newCloudLedger`'s warning.)

### Helsinki (fi) — `/data/router.env`
```
LCHAT_MESH_DOMAINS=dev.11ch.at            # was fi.dev.11ch.at
LCHAT_DNS_PROVIDER=route53                # new
LCHAT_WILDCARD_DOMAINS=dev.11ch.at        # new
# unchanged: AWS_ACCESS_KEY_ID/_SECRET (scoped user, now also Route53-DNS01-scoped),
#            AWS_REGION=eu-west-1, LCHAT_HIBERNATE_BUCKET, LCHAT_MESH_REGION=fi,
#            LCHAT_MESH_SELF=https://one.fi.dev.11ch.at, LCHAT_MESH_TOKEN
# (no cloud on Helsinki — nothing else to change.)
```

### Swap + restart (each box)
```sh
# back up the current live binary, move the fresh one in:
sudo cp /opt/lchat/lchat /opt/lchat/lchat.bak-preroute53
sudo mv /opt/lchat/incoming/lchat-new /opt/lchat/lchat
sudo chmod +x /opt/lchat/lchat
# Dublin only: point cloud/control at the fresh unified binary (drop lchat-banner)
#   — edit the units' ExecStart to /opt/lchat/lchat, or symlink lchat-banner→lchat,
#     then `sudo systemctl daemon-reload`.
sudo systemctl restart lchat-router
sudo systemctl restart lchat-cloud lchat-control   # Dublin (Helsinki has none)
```

The router obtains the `*.dev.11ch.at` wildcard on startup (its `run()` loop
checks immediately). Watch for it:

```sh
journalctl -u lchat-router -n 60 --no-pager | grep -iE "wildcard|route53|acme"
# want: "router: wildcard *.dev.11ch.at obtained (valid until …)"
```

---

## Step 4 — verify the wildcard cert (both boxes)

```sh
# The cert both boxes present for a dev host must be the *.dev.11ch.at wildcard:
echo | openssl s_client -connect 54.154.95.205:443  -servername probe.dev.11ch.at 2>/dev/null \
  | openssl x509 -noout -subject           # want: subject=CN=*.dev.11ch.at
echo | openssl s_client -connect 157.180.123.183:443 -servername probe.dev.11ch.at 2>/dev/null \
  | openssl x509 -noout -subject           # same on Helsinki
```

If a box still serves a per-host cert, DNS-01 didn't complete — check its IAM
(the tofu attaches the Route53 policy to Dublin's role / Helsinki's user) and the
`journalctl` line above; per-host issuance keeps the box serving meanwhile, so
this degrades safely.

---

## Step 5 — the proof (cross-region 200)

Two synthetic hosts under the shared domain, one homed to each region, plus one
real minted space to exercise the full ledger path.

⚠️ **Two things that WILL bite (both learned the hard way on the first run):**

1. **The cross-region host MUST be `--hibernate`.** Route replication (the
   fan-out that gives the *other* box a row to proxy to) is gated by
   `route.Hibernate` in `control.go` — a plain `add-instance` makes a live
   systemd instance pinned to one box, which deliberately does NOT replicate,
   so the receiving box 404s ("no route"). Cross-region serving is inherently
   hibernation-backed: the state lives in S3 so the owning box restores it, and
   only then does the route fan out.
2. **The CLI needs the mesh env sourced.** `sudo` strips the environment, so
   `sudo /opt/lchat/lchat add-instance …` has no `LCHAT_MESH_*` /
   `LCHAT_HIBERNATE_BUCKET` and silently skips the fan-out ("mesh: skipping route
   replication"). Wrap it so it reads `/data/router.env`.

Create the fi-homed host **on Helsinki** (its home region — so the box that owns
it also holds the instance to wake), hibernating, with the env sourced:

```sh
# on Helsinki (root):
sudo bash -c 'set -a; . /data/router.env; set +a; \
  /opt/lchat/lchat add-instance --home-region fi --hibernate mxproof-fi.dev.11ch.at'
# ie-homed control on Dublin (served locally — a live instance is fine here):
sudo /opt/lchat/lchat add-instance --home-region ie mxproof-ie.dev.11ch.at
```

Both resolve to Dublin via `*.dev.11ch.at`. Hit them from anywhere:

```sh
curl -s -o /dev/null -w "%{http_code}\n" https://mxproof-fi.dev.11ch.at/api/version   # want 200
curl -s -o /dev/null -w "%{http_code}\n" https://mxproof-ie.dev.11ch.at/api/version   # want 200
```

Confirm it truly served cross-region (a successful proxy does NOT log — only
errors do, so don't look for a proxy line): check that **Dublin's** registry has
the fanned-out `fi` route, and that **Helsinki's** log shows it restoring the DB
from S3 for that host:

```sh
# Dublin: the fi route arrived (home_region=fi, hibernate=1) → Dublin proxies, never owns it:
sudo sqlite3 -column /data/registry.db "SELECT host,home_region,hibernate FROM routes WHERE host LIKE 'mxproof%';"
# Helsinki: it woke the instance from S3 (this is the cross-region serve):
journalctl -u lchat-router --since -3min | grep 'replica sync .*mxproof-fi'
```

Then the real ledger path — mint a space through the ie cloud (uses home's caller
token) and confirm the hostname is region-neutral and ledger-issued:

```sh
# on Dublin:
TOK=$(sudo grep '^LCHAT_CLOUD_TOKEN=' /data/home.env | cut -d= -f2-)
curl -s --unix-socket /run/lchat/cloud.sock -H "Authorization: Bearer $TOK" \
  -H 'Content-Type: application/json' -d '{"requester":"ledger-proof"}' \
  http://cloud/api/instances
# → {"host":"<sqid>.dev.11ch.at", …}  (NOT .ie.dev.11ch.at), serves 200 on first hit.
# The S3 counter mesh/id-sequence now holds the block high-water (e.g. 100 after
# the first mint reserves a block of 100 — ids are gap-tolerant).
```

Tear the throwaways down when done — delete the cloud-minted space via the cloud
API (`DELETE /api/instances/<host>?requester=<the-requester>`, so cloud.db's quota
ledger is freed) and the synthetic hosts with `lchat remove … --stop`. Route
**deletes do NOT fan out**, so also clear the replicated row from the *other*
box's registry by hand (`DELETE FROM routes WHERE host=…` + `systemctl reload
lchat-router`), and purge each hibernating host's `s3://<bucket>/<host>/` prefix.

Tear the synthetic hosts down when done:
```sh
sudo /opt/lchat/lchat remove mxproof-fi.dev.11ch.at --stop && sudo rm -f /data/mxproof-fi.dev.11ch.at.*
sudo /opt/lchat/lchat remove mxproof-ie.dev.11ch.at --stop && sudo rm -f /data/mxproof-ie.dev.11ch.at.*
```

---

## Rollback

- **Binary**: `sudo mv /opt/lchat/lchat.bak-preroute53 /opt/lchat/lchat && sudo systemctl restart lchat-router lchat-cloud lchat-control`.
- **Env**: revert `LCHAT_MESH_DOMAINS` to the per-country value, drop the two
  wildcard/DNS lines, revert cloud.env — restart. The boxes are back to
  per-country placement immediately.
- **DNS/tofu**: `tofu destroy` targets only the new resources; but the fast
  revert is to re-add the DNSimple `*.ie.dev`/`*.fi.dev` A records and remove the
  `dev` NS delegation (DNSimple becomes authoritative again). The Route53 zone can
  linger harmlessly. Because the delegation is the switch, removing it reverts
  resolution wholesale.
- Nothing here touches instance DBs or member data — the dev instances keep their
  `/data/<host>.db`, and the change is DNS + router/cloud config only.

---

## Follow-on (not in this cutover)

- **Slice 5 — the relocate action**: an admin "change region" that flips a live
  space's `home_region`, copies its store to the destination region, and moves
  its `<sqid>.dev.11ch.at` A record — the proxy covers the DNS-TTL window. This
  cutover's synthetic fi/ie hosts prove the *serving* half relocation lands on.
- **Fleet guide**: once this is stable, document `LCHAT_MESH_REGION` /
  `LCHAT_CLOUD_REGION` / `LCHAT_CLOUD_ID_LEDGER` / `LCHAT_DNS_PROVIDER` /
  `add-instance --home-region` in `docs/planet-scale-guide/`.
- **Prod `11ch.at` → Route53** is a separate, deliberate, deferred call — DNSimple
  serves prod fine; consolidate later with a low-TTL dual-run if wanted.