# 🤖 Deploying instances — the mental model

> **Not a procedure.** Day-to-day operation goes through **`./deploy/ops`**
> (releases: `publish`/`promote`/`ship`; branch canaries:
> `canary create|deploy|destroy`; preflight: `doctor`). Where this page shows
> command sequences, they document what the verbs do — run the verb, and if
> no verb covers your task, **extend ops** rather than following prose here.

What a box that uses the `lchat` router is made of, and why the pieces are
shaped this way (`docs/instance-router.md` is the design; `deploy/README.md`
is the one-page fresh-box bootstrap). Everything here is generic — hosts are
`chat.example.com`, the box is `<box>`.

## The mental model

One static Go binary (`lchat`) plays three roles on the box:

- **The router** (`lchat-router.service`) — binds `:80`/`:443`, terminates TLS
  (Let's Encrypt via autocert HTTP-01), and reverse-proxies each hostname to a
  unix socket. The route table lives in a small SQLite registry
  (`/data/registry.db`).
- **Instances** (`lchat@<host>.service`, one per chat) — each is its own process,
  socket (`/run/lchat/<host>.sock`), and database (`/data/<host>.db`). Fully
  isolated: its own admin token, its own crash/memory blast radius, its own
  binary version.
- **The control CLI** (`/opt/lchat/lchat add-instance | deploy | instances | …`)
  — run on the box over SSH; it edits the registry, writes env files, and drives
  systemd.

Two directories carry the versioning scheme:

```
/opt/lchat/versions/<ver>/lchat     # the content store: every staged build, keyed by build id
/opt/lchat/live/<host>              # per-instance symlink into the store; the unit execs this
```

Each instance's unit execs its *own* `live/<host>` symlink, so two instances can
run two different builds at once. That's what makes canarying a one-liner.

Per-instance config is an **EnvironmentFile** at `/data/<host>.env` (mode 0600),
written by `add-instance`. It holds the one secret that matters — the plaintext
`LCHAT_ADMIN_TOKEN` (the DB stores only its hash, so this file is the only
recoverable copy) — plus optional per-instance vars (`LCHAT_PUSH_RELAY_URL`/
`_TOKEN`, `LCHAT_HOME_ORIGIN`, `LCHAT_APP_STORE_URL`).

Nothing here weakens the E2E model: the router and the instance only ever see
ciphertext, and the admin token authorises roster management, not decryption.

## Prerequisites

- **A bootstrapped box.** `scp` the binary to `/opt/lchat/lchat`, then
  `lchat bootstrap` (idempotent) — it creates the `lchat` service user, the
  dirs, and the systemd units, and starts the router. See `deploy/README.md`.
- **DNS** for the new host pointing at the box — either a wildcard
  (`*.example.com A <box-ip>`) so new instances need no DNS step, or one A
  record per host. Low TTL (60s) makes future moves painless.
- **Inbound `:80` and `:443` open** (both firewall on the box and any cloud
  firewall). Port 80 is required — autocert's HTTP-01 challenge answers there.
- Certificates issue **lazily on the host's first HTTPS hit**, in a second or
  two. There's no pre-issuance step; if DNS resolves and :80 is reachable, the
  first `curl https://<host>/` just works.

## Adding an instance

One command, run on the box:

```sh
ssh root@<box> /opt/lchat/lchat add-instance chat.example.com
```

It mints an admin token, writes `/data/chat.example.com.env`, registers the
route, stages the current binary and points `live/<host>` at it, starts
`lchat@chat.example.com`, waits for the socket to answer, reloads the router,
and prints a one-time setup link (open it and you're the admin) plus the
admin-token link:

```
added chat.example.com
  get started (one-time link, makes you the admin):
  https://chat.example.com/join/setup#…
  admin token link: https://chat.example.com/admin#token=…
  (recover later: lchat admin-link chat.example.com / lchat setup-invite chat.example.com)
```

Optional flags (all become vars in the instance's env file):

| flag | what it sets |
|---|---|
| `--admin-token <tok>` | import an existing token instead of minting — for migrations, so saved admin links keep working (see `docs/instance-migration.md`) |
| `--push-relay-url` / `--push-relay-token` | iOS push via a shared APNs relay (`docs/push-relay.md`); mint the token on the relay first |
| `--home-origin` | the passkey-backup home server this instance offers members |
| `--app-store-url` | where the welcome page's "Get the app" button points |

All of these are optional — an instance without them is fully functional for
web + Web Push (VAPID is self-generated into the instance DB; no config).

Then, immediately:

- **Store the admin token somewhere durable** (a password manager). It exists
  in exactly one recoverable place — `/data/<host>.env` on the box —  and
  `lchat admin-link <host>` reads it from there, so losing the box means losing
  the link.
- **Verify:**

```sh
ssh root@<box> /opt/lchat/lchat instances     # HOST / KIND / STATUS / VERSION / SOCKET
curl -s https://chat.example.com/api/version  # {"build":"<ver>"} over a fresh LE cert
```

Open the printed admin link in a browser to name the group and mint member
invites.

If `add-instance` fails partway, the route is left in status `failed` with a
`last error` line shown by `lchat instances`; fix the cause and re-run — a
failed/creating route is overwritten on retry.

## Adding bots (optional)

Each instance can have a **sidecar** hosting the default agents (Claude /
Eleven). The sidecar is Node (it reuses `web/static/crypto.js`, so
its E2E crypto can't drift from the browser's), so the box needs a one-time Node
setup — see `deploy/README.md § Bots` for the `apt`/`rsync`/`npm ci` recipe.
Then, per instance:

```sh
ssh root@<box> 'ANTHROPIC_API_KEY=sk-ant-… /opt/lchat/lchat add-bot chat.example.com'
```

`add-bot` mints one agent invite per identity via the instance's admin API,
writes `/data/<host>-bot.env` (0600 — invites, session paths, the API key), and
starts `lchat-bot@<host>`. Notes:

- **Instance first, then sidecar** — Eleven baselines "who already exists" from
  the roster on its first tick; booting the sidecar against a not-yet-ready
  instance can make it welcome-DM long-standing members later.
- Omit `ANTHROPIC_API_KEY` for deterministic-only bots (Eleven still welcomes
  members and answers slash commands; no AI replies). Add the key to
  `/data/<host>-bot.env` and `systemctl restart lchat-bot@<host>` later to give
  it a brain.
- The API key is read from the **environment**, deliberately not a flag — a
  flag would leak it into the process table and shell history.
- The sidecar connects to its instance over the **public URL**. If the sidecar
  runs on the same box as the router, pin every hosted hostname to loopback in
  `/etc/hosts` (`127.0.0.1 chat.example.com …`) so on-box clients reach the
  local router instead of depending on external DNS — and disable cloud-init's
  `manage_etc_hosts` so the pins survive reboots. See
  `docs/instance-router.md § Bots`.
- Sidecar logs: `journalctl -u lchat-bot@chat.example.com -f`.
- Updating sidecar *code* is separate from binary deploys: re-`rsync` `agent/` +
  `web/static/` to `/opt/lchat/` and restart the `lchat-bot@` units (they
  backfill on reconnect, so a restart is safe).

## Shipping a new build (canary → promote)

Deploys are **data-safe by construction**: they never touch `/data` — DB
migrations are additive (`CREATE TABLE IF NOT EXISTS`), so new code over old
data is always fine. The flow is: build locally, stage on the box, prove the
build on one instance, then promote the fleet.

```sh
# 1. Build linux/amd64, stamped with the git short sha as its build id
./deploy/build-linux.sh                       # → lchat-linux-amd64, prints <ver>

# 2. Copy it up and register it in the version store
scp lchat-linux-amd64 root@<box>:/opt/lchat/incoming
ssh root@<box> /opt/lchat/lchat stage --file /opt/lchat/incoming   # → prints <ver>

# 3. Canary ONE instance
ssh root@<box> /opt/lchat/lchat deploy canary.example.com --version <ver>
# …verify: open the app, send a message, check /api/version reports <ver>…

# 4. Promote everyone (one at a time, health-checked, stops on first failure)
ssh root@<box> /opt/lchat/lchat deploy --all --version <ver>

# 5. Who runs what
ssh root@<box> /opt/lchat/lchat instances     # VERSION column shows per-instance skew
```

What `deploy --version` actually does: atomically repoint
`/opt/lchat/live/<host>` at the staged build, `systemctl restart` the instance
(SIGTERM drains in-flight HTTP and closes WebSockets so clients reconnect; the
router retries dials across the sub-second gap, so requests queue rather than
fail), then poll `/api/version` until the instance reports the **expected build
id** — not merely a 200. On mismatch it **auto-rolls-back** to the previous
version and errors out. `deploy --all` walks the fleet one instance at a time
and stops at the first failure, leaving already-promoted instances on the new
build.

Stepping back is explicit too:

```sh
ssh root@<box> /opt/lchat/lchat rollback chat.example.com   # → its recorded previous version
```

Other notes:

- `deploy <host>` with **no** `--version` just gracefully restarts the instance
  on its current build (useful after editing its env file).
- Build ids are the git short sha by default; `./deploy/build-linux.sh
  my-label` stamps a custom id (handy for branch builds, below). `lchat stage`
  reads the id out of the binary itself, so the store can't mislabel a build.
- The build script targets linux/amd64; on an arm64 box set
  `GOARCH=arm64` (edit or inline-override the script).
- **⚠️ `deploy --all` moves *every* instance**, including one you've pinned to
  a branch build. Before promoting a fleet, check the VERSION column of
  `lchat instances` — if a canary instance is meant to stay on its branch
  build, deploy the others individually instead of using `--all`.
- To also refresh the **control/router** binary (rarely needed — instance
  builds don't require it): atomically replace `/opt/lchat/lchat` (`cp` to a
  temp name + `mv` over it) and `systemctl restart lchat-router`.

### Verifying a canary

"Verify" should be more than a 200. A reasonable checklist for a few minutes on
the canary instance before promoting:

- `curl -s https://<host>/api/version` reports the new build id.
- Open the app in a browser: history decrypts, sending a message round-trips,
  and a WebSocket reconnect happened cleanly (no stuck "connecting").
- If the change touched uploads/push/bots, exercise that path specifically.
- `journalctl -u lchat@<host> -e` shows no new errors.

## Standing up a canary *instance* (branch builds)

Sometimes the thing to prove isn't a build of `main` on a real instance, but an
**unmerged branch** on a **throwaway instance** — real TLS, real devices, zero
real users, and the fleet (and the control binary) untouched. Because versions
are per-instance, this needs no special machinery:

```sh
# on your machine, on the feature branch:
./deploy/build-linux.sh                       # id = the branch's git short sha
scp lchat-linux-amd64 root@<box>:/opt/lchat/incoming

# on the box:
/opt/lchat/lchat stage --file /opt/lchat/incoming        # → <branch-ver>
/opt/lchat/lchat add-instance canary-feature.example.com # starts on the CURRENT fleet build
/opt/lchat/lchat deploy canary-feature.example.com --version <branch-ver>   # pin it to the branch
```

The branch binary exists only in the version store; nothing else runs it.
`lchat instances` shows the skew explicitly. When the feature merges and ships:

```sh
/opt/lchat/lchat deploy canary-feature.example.com --version <new-main-ver>  # rejoin the fleet
# …or tear the whole instance down (below) if it was a throwaway.
```

Two cautions:

- Data written by a branch build persists in `/data/<host>.db`. Migrations are
  additive so *moving forward* is safe, but a branch that created tables/rows a
  later design abandons leaves them behind. For feature trials, prefer a
  **throwaway** instance you'll delete over pinning a real one.
- Remember the `deploy --all` warning above — a fleet promotion will stomp the
  pin.

## Removing an instance

`remove` is deliberately data-safe by default — it only stops *routing*:

```sh
/opt/lchat/lchat remove chat.example.com          # route deleted; instance still running, DB/env kept
/opt/lchat/lchat remove chat.example.com --stop   # also stop+disable lchat@<host> and lchat-bot@<host>
```

Even with `--stop`, the DB and env file stay on disk, so the instance can be
resurrected: re-run `add-instance` passing `--admin-token` with the token still
in `/data/<host>.env` (a plain `add-instance` would mint a fresh token and
invalidate saved admin links). **Full teardown** — only for an instance you truly mean to destroy — is the
manual part, on the box:

```sh
rm /data/chat.example.com.db /data/chat.example.com.db-wal /data/chat.example.com.db-shm  # the DB
rm /data/chat.example.com.env                                # the admin token
rm -rf /data/chat.example.com-blobs                          # disk-backed attachments (if present)
rm -rf /data/chat.example.com-bot /data/chat.example.com-bot.env  # sidecar sessions + env (if it had bots)
rm /opt/lchat/live/chat.example.com                          # the version pointer
```

…then delete the host's DNS record (if it wasn't wildcard-covered) and the
stored admin token. There is no undo after the DB is gone.

## Sibling services (home server, push relay, third-party backends)

The router can front processes it doesn't manage — anything already listening
on a local unix socket:

```sh
/opt/lchat/lchat add-service home.example.com --socket /run/lchat/home.sock
```

Unlike an instance, the router only *proxies* a service: no unit is created, no
version is tracked (`instances` shows `-`), and the backend must already be up
when you add it. This is how the home server (`home/`) and push relay (`push/`)
run on the same box — each is its own binary + systemd unit
(`lchat-home.service` / `lchat-push.service`, installed by `bootstrap`, inert
until their `/data/{home,push}.env` exists), deployed by replacing its binary
and restarting its unit.

## Migrating an instance in from elsewhere

Moving an existing instance (from another box, or another hosting setup) onto
this one is a separate, validated procedure — `VACUUM INTO` snapshot, copy the
DB **and its `<host>-blobs/` directory**, `add-instance --admin-token` to
preserve the token, DNS flip with the old deployment kept as rollback. Follow
`docs/instance-migration.md`; the key property is that the **hostname must not
change** (browser keys are per-origin — a new origin means every member re-keys
via add-device links).

## Troubleshooting

- **`lchat instances`** is the first stop: per-route status
  (`creating`/`ready`/`failed`), the running VERSION, and a `last error` line
  when something failed.
- **Logs:** `journalctl -u lchat@<host> -f` (instance),
  `-u lchat-router -f` (TLS/routing), `-u lchat-bot@<host> -f` (sidecar).
- **Cert won't issue:** check DNS actually resolves to the box, and `:80` is
  reachable from the internet (HTTP-01 answers there). The router only issues
  for hosts in its route table — an unknown host is refused by `HostPolicy`
  before ACME is attempted.
- **On-box clients (sidecars) can't reach an instance / reach a stale server:**
  the `/etc/hosts` loopback pins are missing or were regenerated on boot — see
  the bots section above.
- **Health check by hand:** `curl -s https://<host>/api/version` (build id), or
  on the box against the socket:
  `curl -s --unix-socket /run/lchat/<host>.sock http://localhost/api/version`.
- **Recover an admin link:** `lchat admin-link <host>`.
- **A deploy "failed" but the fleet is half-promoted:** that's by design —
  `deploy --all` stops at the first unhealthy instance and leaves earlier ones
  on the new build. Fix the failing instance (its own deploy auto-rolled it
  back), then re-run.