# 🤖 Migrating an instance from one server to another

> 🤖 **Not yet encoded — do not run from this page.** There is no
> `ops` verb for instance migration yet. When a migration is next needed,
> the move is to build one from this record (`ops migrate <host> <box>` or
> similar, per `deploy/README.md § The operator surface`'s no-cookbooks
> rule), then run the verb — and mark this page historical.

Status: **design record** (the shape below was validated once in practice). Moving a running messenger
instance to a new box is **data movement + a DNS flip**, done so it's reversible:
copy the DB, keep the old instance running as a fallback, and only cut DNS over
once the new box is verified. The flip itself is reversible too.

The key property: **keep the hostname the same**. Browser keys (and passkeys, for
the home server) are per-origin, so an unchanged origin means members' keys,
sessions, and history are all unaffected — only the DNS record moves.

## Before you start

- SSH to both the source and destination boxes.
- The destination is already bootstrapped (`lchat bootstrap`; router running).
- Know the instance's **admin token** and any per-instance config (see step 1).

## Steps

### 1. Recover the instance's config from the source

The instance's DB stores only the **hash** of the admin token — the plaintext
lives in the source's deploy env / secret store. Recover it, plus any per-instance
config the instance reads (`LCHAT_PUSH_RELAY_URL`/`_TOKEN`, `LCHAT_HOME_ORIGIN`,
`LCHAT_APP_STORE_URL`, APNs vars, …). You'll write these into the destination's
EnvironmentFile in step 4. **Preserve the admin token** — minting a new one
invalidates every saved admin link.

### 2. Snapshot the DB consistently

Use SQLite's `VACUUM INTO` — it produces a single, compacted, transaction-consistent
file and is **safe on a live DB** (no need to stop the instance):

```sh
sqlite3 /path/to/lchat.db "VACUUM INTO '/tmp/snapshot.db'"
sqlite3 /tmp/snapshot.db "PRAGMA integrity_check"   # expect: ok
```

If the source hides the DB behind an opaque store (many identically-named DB files),
identify the right one by **content** — e.g. grep the files for the instance's
`admin_token_hash` (`hex(sha256(admin token))`), which is unique per instance.

`VACUUM INTO` snapshots the **DB only**. Attachment/photo/video bytes live in a
**sibling directory** `<db>-blobs/` (e.g. `/data/<host>-blobs/`), not in the DB —
so it must be copied separately (step 3), or migrated messages lose their images
and videos. The files are immutable and content-addressed by a random id, so a
live copy is safe. (An older instance that predates disk-backed blobs has no such
directory — its blobs are still inside the DB and need nothing extra.)

### 3. Copy the snapshot to the destination

Copy `/tmp/snapshot.db` to the destination as `/data/<host>.db` and verify it
landed intact (`integrity_check` + a couple of row counts against the source).
**Also copy the blob directory** if the source has one:

```sh
rsync -a /data/<host>-blobs/ <dest>:/data/<host>-blobs/
```

These are the disk-backed attachment bytes the DB references by id; copy them at
or after the DB snapshot (blobs uploaded to the source afterward are subject to
the same gap check as messages — see step 5).

**Also copy the VAPID keyfile** (`/data/<host>.vapid`, another DB sibling —
kept out of the DB so S3 replicas never carry a private key). Without it the
destination re-mints the pair on first boot and every member's web-push
subscription silently breaks until their next visit re-subscribes them.

### 4. Import on the destination (preserve token + config)

With the migrated DB already at `/data/<host>.db`, run `add-instance` with
`--admin-token` so it **imports** the existing token instead of minting a new one
(a fresh token would rewrite `admin_token_hash` on boot and orphan every saved
admin link). It writes the EnvironmentFile, registers the route, starts the unit
against the pre-placed DB, health-checks it, and reloads the router:

```sh
lchat add-instance <host> \
  --admin-token "<existing token>" \
  --push-relay-url https://push.example.com --push-relay-token "<existing relay token>"
```

Pass whatever per-instance config the source had (`--push-relay-url`/`-token`,
`--home-origin`, `--app-store-url`). Verify **before any DNS change**, over the
instance's unix socket: `GET /` → 200, an admin-token request → 200, and the row
counts match the source.

### 5. Gap check

Confirm nothing was written to the source between the snapshot and the cutover
(compare a row count, e.g. `messages`, source-now vs snapshot). For a busy
instance, freeze writes or take the snapshot at cutover time — writes that land on
the source after the snapshot are **not** on the destination.

### 6. Cut over DNS

Flip the host's A record to the destination's IP. **Lower the record's TTL ahead of
time** (to ~60s, ideally a TTL-worth before the cutover) so the flip propagates
fast; otherwise resolvers that cached the old IP keep hitting the old box until the
old TTL expires (harmless while the old box is still up as a fallback).

The TLS cert **can't be pre-issued**: autocert's HTTP-01 validates by resolving the
host over public DNS, so the cert only issues once DNS points at the destination.
Authoritative DNS updates in seconds (only resolver *caches* lag), so the cert
issues on the first HTTPS hit right after the flip.

### 7. Verify + keep the fallback

`GET https://<host>/` → 200 with a valid (Let's Encrypt) cert, serving the migrated
data. **Leave the source instance running** until you're confident — the migration
is a copy, so flipping DNS back is an instant rollback. Retire the source only
afterward.

## Bots (sidecars)

The instance's **bot agent-users migrate with the DB** (they're rows), but the
**sidecar** — the Node process and its session files (the bots' identities) — is
separate and does *not* come across in the DB copy. Do **not** re-run `add-bot` on
the destination: that mints a *new* set of agent identities (duplicate bots in the
Helpers catalog). Instead either copy the sidecar's session files to the
destination and start `lchat-bot@<host>` there, or point the existing sidecar at
the destination (it follows the host over DNS and reconnects with its persisted
token). Plan the bot move as its own step.

## What makes this safe

- **Reversible**: the source is copied, never moved; DNS flip is reversible; the
  cert issues automatically post-flip.
- **No key churn**: same origin ⇒ per-origin browser keys/passkeys untouched.
- **Consistent snapshot**: `VACUUM INTO` captures a clean point-in-time DB; the
  sibling `<db>-blobs/` dir (immutable, content-addressed files) copies alongside.
