# πŸ€– The instance router β€” a single-machine bootstrap Status: **shipped β€” this page is the design rationale, not a procedure.** The machinery described here is live (`lchat bootstrap`/`router`/`add-instance`…), and day-to-day operation of it goes through **`./deploy/ops`** β€” if something here reads like steps to run and no ops verb covers it (e.g. hardening a brand new box), the move is to add the verb, not to follow the prose. ## The problem The messenger is a single static, no-cgo binary with SQLite compiled in, and it runs as **one fully-isolated instance per chat** β€” each with its own disk, admin token, and subdomain. The obvious way to run several of those on one machine is to reach for an orchestrator (Kamal, a PaaS, Kubernetes): it gives you TLS, subdomain routing, a per-app disk, env injection, and process supervision. But that's a lot of machinery around a single file you could `scp`. This doc describes a lighter alternative for the common case β€” many instances on one machine β€” that keeps the two properties process-per-instance gives you and that are worth keeping: - **Per-instance versioning / canary.** Deploy one instance on a new build while the others stay on the old one. A single shared process can't do that. - **Isolation.** A crash or a resource hog in one instance must not touch the others. …and adds **zero-downtime deploys**. Those reinforce each other: zero-downtime *means* "run the new version alongside the old and hand over," so per-instance versioning falls out of doing it right. No container platform required β€” just the binary, a small router, and systemd. ## One binary, many ways to deploy The messenger makes no assumptions about how it's run. Put it behind nginx or Caddy, on a PaaS, in Kubernetes, under Kamal, or one-container-per-instance on any platform β€” all of those work, and nothing here precludes them. The instance binary stands alone. **What this doc describes is a batteries-included path for the common case:** bootstrapping a single machine from zero to a running, auto-TLS'd, multi-instance box with zero-downtime deploys and no external orchestrator. It's the *simplest* way, not the *only* way β€” the router below is optional convenience. ## The shape ``` :443 / :80 internet ──▢ lchat router ──(reverse proxy over unix sockets)──▢ lchat instance (chat-a) ── chat-a.db (autocert HTTP-01, lchat instance (chat-b) ── chat-b.db HostPolicy = route table, lchat instance (chat-c) ── chat-c.db atomic upstream map) … β”‚ home server (home/) ── home.db ┐ service β”‚ push relay (push/) ── push.db β”˜ routes └── control API on a localhost-only socket (add / deploy / list) systemd supervises everything: router + lchat@ (instance) + lchat-bot@ (sidecar) + home/push units ``` One binary, three roles (subcommands): - `lchat router` β€” the front door (TLS + routing + control plane). - `lchat instance` β€” one messenger instance (today's `main()` minus the TCP listener and minus the process-global env reads). - control subcommands β€” `add-instance`, `deploy`, `instances`, `admin-link`. ## The router - **`:443`** β€” TLS via `golang.org/x/crypto/acme/autocert` (stdlib-adjacent, no DNS-API credentials). `HostPolicy` returns nil only for hosts in the route table, so the route list *is* the cert allowlist. `Cache: autocert.DirCache` on the persistent disk. HTTP-01 (well-known) is the challenge; a brand-new host is issued on first hit β€” no wildcard, no DNS-provider step. - **`:80`** β€” `m.HTTPHandler(nil)`: serves `/.well-known/acme-challenge/…`, 301-redirects everything else to https. - **Routing** β€” the route table maps **host β†’ backend** (a unix socket at a version); `httputil.ReverseProxy` dials it. The upstream map is an `atomic.Pointer` so reloads and deploys swap it without dropping a connection. WebSocket upgrades pass through (Go β‰₯1.12). A backend is usually one instance β€” see *Backends*, below. - **Unknown hosts** β€” 404 by default; with `--fallback-url` (env `LCHAT_FALLBACK_URL`, e.g. the apex marketing site) they 302 there instead, so under a wildcard DNS record a typo'd or retired subdomain lands somewhere useful. Scope caveat: on `:80` this always works, but over HTTPS it only helps hosts that still *have* a cert (e.g. a removed instance) β€” a never-served host fails the TLS handshake before any redirect, because `HostPolicy` refuses to mint certs for unknown hosts (anything else would let strangers burn the ACME rate limit). Full coverage needs the wildcard-cert option below. - **Control API** β€” shipped as `lchat controld` (`lchat-control.service`): a local-only unix socket (`/run/lchat/control.sock`), gated by a root-only bearer token, never exposed on a served hostname. This is the god surface (it creates instances), so it stays off the public interface; its one consumer is the cloud provisioner (`lchat cloud`), the public, unprivileged policy layer for self-serve space creation. Full design: `cloud-provisioner.md`. Threat model is unchanged: TLS terminates at the router, but message bodies are already E2E ciphertext β€” the router sees exactly what the app sees, never plaintext. ### πŸ€– Wildcard TLS (DNS-01) β€” implemented HTTP-01 issues one cert per host on first hit β€” simple, needs no DNS credentials, but rides the Let's Encrypt per-registered-domain rate limit (~50 new certs/week, a hard ceiling on space creation) and can't cover a host before it's requested. So the router can also hold a **wildcard** (`*.example.com`) obtained over ACME **DNS-01** (`wildcard.go`): set `LCHAT_WILDCARD_DOMAINS=example.com` (comma list) and a DNS-01 provider (below) in `/data/router.env` and restart. The two coexist β€” the wildcard serves any host exactly one label under a configured domain; everything else (other domains, the apex, deeper levels) takes the per-host HTTP-01 path, which also keeps covering the wildcard domains until the first issuance lands, so a DNS-provider/ACME outage degrades to today's behaviour rather than a dead router. **DNS-01 provider (pluggable; `LCHAT_DNS_PROVIDER`).** The solver that publishes the challenge TXT is a seam (`dnsProvider`), so the wildcard works against more than one DNS API without changing anything else: - **DNSimple β€” the default.** Leave `LCHAT_DNS_PROVIDER` unset (or set it to `dnsimple`) and provide `LCHAT_DNSIMPLE_TOKEN`, a DNSimple **account** API token that can manage the zone. This is the fully-supported path for self-hosters and production; nothing about it changed when the seam landed. A missing token leaves the wildcard off (per-host issuance continues). - **Route53 β€” the additive off-ramp** (`docs/multi-edge.md`). Set `LCHAT_DNS_PROVIDER=route53`; the solver authenticates with the **standard AWS credential chain** (env vars or an EC2/container **IAM role**), so there is *no DNS token on disk*. The role is scoped to just the wildcard domain's hosted zone β€” this is how the dev boxes obtain a shared `*.dev.11ch.at` cert from their instance role. Route53 is global, so no region is required (the solver defaults the signing region to `us-east-1`; an explicit `AWS_REGION` wins). Mechanics: a renewal loop checks at startup and every 12h (hourly after a failure), re-issuing when the cert is missing or <30 days from expiry. Issuance publishes a `_acme-challenge.` TXT via the selected provider (TTL 60, always cleaned up) β€” DNSimple via its records API, Route53 via `ChangeResourceRecordSets` (UPSERT, then a wait for the change to go INSYNC) β€” waits briefly for propagation, and finalizes with a fresh ECDSA key. The cert+key persist as `wildcard-.pem` (0600) in the acme cache dir β€” same backup story as the autocert cache β€” and load synchronously on restart; the ACME account key is `wildcard-account.key` alongside. `HostPolicy` still gates which hosts are actually *served* a backend; the wildcard only changes how the handshake gets a cert. Deliberate side effect: an unknown subdomain of a wildcard domain now completes TLS and gets the router's fallback/404 instead of a handshake error. ## The instance process `lchat instance --db /data/.db --socket /run/lchat/.sock --config ` It's today's server with two changes: it listens on a **unix socket** instead of a TCP port, and it takes per-instance config from flags/a file the router writes rather than from process-global env (see below). Everything else is unchanged β€” its own SQLite (WAL), its own admin token (`ensureAdminToken`, in the DB), its own VAPID (`ensureVAPID`, in the DB). `handler()` is untouched, so `newTestServer` and the existing handler tests keep working as-is. ## Backends: one per instance (default), one per version (option) The router maps **host β†’ backend** (a socket at a version) and doesn't care how many hosts a backend serves. That leaves two ways to slice processes: **One backend per instance (the default).** Each instance is its own process, socket, and DB β€” full crash and resource isolation, and per-version deploys come *for free*: each unit execs a per-instance `live` symlink into a version store, so moving one instance to a new build is one `lchat deploy --version ` while the rest stay put (a `trunk` instance next to `stable` ones), no coordination. Canary is just "which binary does this unit run." See *Versioned, side-by-side deploys* below. **One backend per version (an option, later).** Alternatively a single process serves *many* hosts at one version (an in-process host map, one DB per host), and the router sends each host to the process running its assigned version: ``` stable-host ─┐ other-host ─┼─▢ backend @ stable (one process, many DBs) another-host β”€β”˜ trunk-host ──▢ backend @ trunk (one process) ``` This trades isolation for fewer processes β€” fewer Go runtimes (less RAM), fewer deploy targets β€” and the costs are real: a crash or OOM takes down *every* host on that version, there's no per-host resource cap, and deploying a version restarts all its hosts together (one big reconnect, not a staggered roll). Moving a host between versions also needs a clean SQLite hand-off (two processes must never write one DB file), so it suits hosts that live permanently on a channel (a dedicated trunk instance) more than ad-hoc per-instance canaries. **Recommendation:** default to one-backend-per-instance β€” isolation is cheap to keep, per-version is already free, and a dozen processes cost a few hundred MB (RAM isn't the binding resource; disk is). Keep the route table as host β†’ backend so consolidating onto per-version backends stays a config change, not a rewrite, and reach for it only if process count ever actually bites. It needs the in-process host map, so it's the *second* thing to build, not the first. ## Beyond messenger instances: sibling services A deployment usually runs sibling services next to the messenger that want the same TLS + host routing β€” for example the other two binaries in this repo: - **the home server (`home/`)** β€” the passkey-backup store, its own binary + DB. Its host *is* the WebAuthn RP origin, so it must keep the same hostname across any move (only the DNS record changes) or it orphans every passkey. - **the push relay (`push/`)** β€” a shared APNs relay that lets instances (yours *and* third parties') send iOS push without holding Apple's signing key (see `push-relay.md`). To the router these are just another **host β†’ upstream** mapping: a `service`-kind route to a local unix socket, TLS via autocert, no lifecycle management (they deploy on their own cadence). That's the only difference from a messenger instance, which the router *also* create/deploy/version-manages. Because a route is nothing but host β†’ upstream, **anyone running their own home server or push relay is first-class** β€” point a host at their backend the same way. This is deliberately the simple version (a flat mapping + a health check); it can be hardened β€” readiness gating, retries, per-service resource isolation β€” later, if it earns it. ## Bots (sidecars) An instance can have a **sidecar** hosting the default agents. It connects to its instance as a **WebSocket client** over the instance's public host β€” authenticating with a one-time agent invite and persisting its session to disk. It serves no inbound traffic, so it never needs a socket or a cert, and the router never routes *to* it. **The sidecar is the same `lchat` binary** β€” `lchat bot run` (`botcmd.go`, `internal/bot/`). No separate runtime: the Go port mirrors `crypto.js`'s wire format byte-for-byte (`internal/e2ee`, pinned by cross-language vectors generated from `crypto.js` itself), so its ciphertext can't drift from what clients expect, and the box that runs the server can run its bots with **no extra install**. The Node reference (`agent/`) stays in the repo as the functional example for self-hosters who prefer it, but the default deployment is Go. **Why still a separate process?** Because a bot is a full E2E client β€” it holds its own keypair, and that private key must never live in the server process (the core rule). So a bot is a *member that runs on a server*, not a server feature β€” run wherever the operator wants the keys to live (same box, off-box, a Raspberry Pi), since it only ever dials out. **One process, the whole pair.** `lchat bot run -base https:// .session.json .session.json` runs both identities as goroutines in one process, each its own connection. The standard pair is **zero-config**: with no recipe file an identity gets the conversational default (command floor + model; badges AI when `ANTHROPIC_API_KEY` is set). A custom bot drops a `.recipe.json` alongside its session (`docs/bot-recipes.md`) β€” behaviour as data, no code. (A legacy session named `Announcer` is skipped with a log line β€” that identity is retired; the server announces its own changelog now. See docs/bots.md.) Supervise it as **one `lchat-bot@.service` per instance**. The unit execs the *same* per-instance live symlink as the instance itself (`ExecStart=/opt/lchat/live/%i bot run`), so the bot runs the **same version as its app by construction** β€” and `deploy`/`update` restart the bot right after they repoint that symlink and health-check the instance (`deployHostVersion β†’ restartBotSidecar`), so the sidecar's code never lags the server. `bot run` with no positional args reads `LCHAT_BASE` + the `LCHAT_*_SESSION` paths from the unit's EnvironmentFile, so the template is fully generic β€” a full trio or an Eleven-only box both work with no per-host args. Because the sidecar backfills on reconnect, that restart is a plain reconnect β€” no socket-swap, no drain. One wrinkle: it reaches its instance at the public URL through the router, so the box must resolve that public hostname **to itself**. Pin every hostname the box hosts to loopback in `/etc/hosts` (`127.0.0.1 chat.example.com …`) so on-box clients β€” sidecars, and an instance's own calls to a sibling service like the push relay β€” reach the local router directly (still over TLS+SNI, so the right cert is served) instead of hair-pinning out to the public IP. This also sidesteps a migration trap: after you move onto this box and flip DNS, an external resolver may still cache the *old* box's IP for these hosts, so an on-box client that resolves them externally can silently talk to the old box β€” or fail once it's gone. (On cloud images that regenerate `/etc/hosts` on boot, disable that so the pins persist.) Its secrets (an API key, the persisted agent sessions) live in its own env + disk, the same way the messenger's admin token lives in the instance DB. ### Sidecars on hibernating instances: co-spawn A **hibernating** instance (activator-managed, `lchat hibernate --on`) doesn't keep its sidecar on the always-on unit β€” a persistent WebSocket client would either hold the sleeping instance awake forever or turn its reconnect loop into a wake loop. Instead the activator **co-spawns** the sidecar: when a wake brings the instance up (socket healthy, replication running), it also execs `live/ bot run` under `/data/-bot.env` β€” the same file the unit reads; "has a sidecar" *is* that file existing β€” and at sleep it SIGTERMs the sidecar *first*, then stops the instance. The sidecar exists exactly while its instance is awake. If it dies in between, the activator restarts it with capped exponential backoff (`Restart=always`, cheaply); if it won't start at all, the instance still serves β€” members before bots. Because the sidecar dials the instance's *public URL through the router* (the loopback pin above), its traffic marks itself with a header (`X-Lchat-Sidecar`, `internal/bot/sidecarmark.go`) that the activator excludes from the idle clock and refuses to be woken by β€” otherwise the pair could never sleep. The marker is spoofable-but-harmless: it only downgrades the sender's own requests (no activity credit; 503 while asleep) and the router strips it before proxying. The Node reference sidecar (`agent/`) does **not** send the marker, so it isn't suitable for a hibernating instance β€” use the Go sidecar there. The flip commands hand the unit over automatically: `hibernate --on` disables `lchat-bot@` (co-spawn takes over), `hibernate --off` re-enables it once the instance unit is healthy. One honest gap: member-bot charters run in the sidecar, so a sleeping instance's charter schedules don't fire until something else wakes it β€” the due-time index covers server-side schedules (future messages, reminders) only. ### Migrating a Node sidecar to Go The session files are **byte-compatible** (`internal/bot/session.go` matches the Node `session.json` shape), so a running Node sidecar swaps to Go in place with no re-provisioning and no lost identity: 1. The shipped `lchat-bot@.service` template already execs the versioned Go sidecar β€” `ExecStart=/opt/lchat/live/%i bot run` β€” sourcing `LCHAT_BASE` and the `LCHAT_*_SESSION` paths (and `ANTHROPIC_API_KEY`, and `LCHAT_CONNECT_BROKER_URL`/`_TOKEN` for connected-services) from `/data/%i-bot.env`. Nothing per-host to write; `add-bot` just enables it. 2. `systemctl daemon-reload && systemctl restart lchat-bot@`. The bot reconnects, re-declares its kind, and backfills β€” members see no gap. 3. The Go sidecar keeps the Node sidecar's `.state.json` (announce memory) for any custom changelog-recipe bot, so nothing replays. Once **no** unit on the box still `ExecStart`s Node, the Node runtime and the `agent/node_modules` install can be removed entirely β€” the box becomes a pure Go deployment. (`add-bot` itself no longer needs Node either: since #856 it mints agent invites in Go via `internal/e2ee`, not `agent/mint-invite.mjs`.) ### Upgrading an existing box to version-tracked bots Older boxes may carry a **hand-written drop-in** pinning the bot to a fixed binary β€” `/etc/systemd/system/lchat-bot@.service.d/override.conf` with `ExecStart=/opt/lchat/lchat bot run …` (or an even older canary path). A drop-in `ExecStart` **wins over the template**, so those bots keep running one frozen build and never track their instance. Migrate each once: ```sh # drop the pin so the versioned template (live/%i) takes effect, and reinstall # the current template in case the box predates it rm -f /etc/systemd/system/lchat-bot@.service.d/override.conf scp deploy/systemd/lchat-bot@.service root@:/etc/systemd/system/ # or re-run `lchat bootstrap` systemctl daemon-reload systemctl restart lchat-bot@ ``` After this the bot execs `/opt/lchat/live/` and every subsequent `deploy`/`update` keeps it in lock-step with the app automatically. ## The registry (a small control DB) The router owns one SQLite table. Status lives here (not a flat file) so an add-in-flight survives a router restart, and it's where per-instance config that used to be env now lives: ```sql CREATE TABLE routes ( host TEXT PRIMARY KEY, kind TEXT NOT NULL, -- 'instance' (messenger, lifecycle-managed) | 'service' (home, push, …) -- | 'hooks' (an instance's in. webhook-ingest sidecar, add-hooks) socket TEXT NOT NULL, -- unix socket the router proxies to status TEXT NOT NULL, -- pending|creating|starting|issuing|ready|failed last_error TEXT, -- instance-only fields (null for registered services): db_path TEXT, version TEXT, -- running binary version (per-instance skew is normal) app_store_url TEXT, -- was LCHAT_APP_STORE_URL home_origin TEXT, -- was LCHAT_HOME_ORIGIN push_relay_url TEXT, -- was LCHAT_PUSH_RELAY_URL push_relay_token TEXT, -- was LCHAT_PUSH_RELAY_TOKEN (per-instance bearer) created_at INTEGER ); ``` πŸ€– The `hooks` kind is how incoming webhooks reach an instance (issue #610, `docs/webhooks.md`): `lchat add-hooks ` registers `in.` β†’ `/run/lchat/-hooks.sock`, where `lchat-hooks@` (the webhook-ingest sidecar) listens. No router code special-cases it β€” it's an ordinary route. TLS rides the per-host autocert path (the `*.domain` wildcard cert covers one label only; wildcard DNS resolves two labels down with no zone change), so an 11ch.at space needs no DNS work; a custom-domain space's owner adds an `in.` record. `lchat remove ` deletes the `in.` row (and, with `--stop`, the hooks unit) along with the instance's. ### Per-instance config: a per-instance EnvironmentFile Config stays env-shaped β€” the instance reads the same `LCHAT_*` vars it always has β€” but instead of process-global env it comes from a per-instance systemd **EnvironmentFile**, `/data/.env` (mode `0600`), written by `add-instance`. No instance code changes: `EnvironmentFile=-/data/%i.env` in the unit, and the existing `os.Getenv` calls just work. | Var | Source | |---|---| | `LCHAT_ADMIN_TOKEN` | `/data/.env` β€” the DB stores only its *hash*, so this file is the one recoverable copy (`admin-link` reads it) | | VAPID keypair | the instance DB (`ensureVAPID`) β€” self-generated, no config needed | | `LCHAT_APP_STORE_URL`, `LCHAT_HOME_ORIGIN`, `LCHAT_PUSH_RELAY_URL`/`_TOKEN` | `/data/.env` (from `add-instance` flags) | | `LCHAT_DB` / socket path | derived from `` by the unit | | APNs `.p8` | **not on instances at all** β€” the push relay holds it (see `push-relay.md`) | The `routes` table tracks routing + status; the EnvironmentFile is the source of per-instance config and the admin token. The push-relay work already did the hardest part β€” the one secret that can't sit safely in a per-host file (the Apple signer) is out of instances entirely, so each carries only a revocable bearer. ## Provisioning the box Because the server is a static, no-cgo binary, a fresh box needs **no build toolchain and no language runtime** for the messenger, the router, or the home/push services β€” provisioning is just "copy binaries + unit files + make directories": 1. `scp` the `lchat` binary (plus `home`/`push` and `litestream`) to the box β€” or run `deploy/install.sh`, which downloads + verifies the latest stable release and does step 2 for you. 2. `lchat bootstrap` β€” writes the systemd units (router + `@` templates), creates `/data`, `/run/lchat`, and the autocert cache dir, then enables and starts the router and the auto-update timer. 3. Open inbound `:80` / `:443`. That's the whole dependency story for the Go side β€” nothing to `apt install`, no version to keep patched, no runtime to manage. A bare box with *no* Go on it is not a concern; it's the point. πŸ€– **The public status collectors ride the binary too.** `bootstrap` also lays down the SSH-access-ledger and fleet-hardening collector scripts, their four systemd units, and the static viewer pages (`web/status/*.html` + `status.css`, into `/opt/lchat/status-pages/`), and enables their 5-minute timers β€” all embedded, source-of-truth `deploy/`. This closes the gap the fleet hardening opened: with operator SSH on WireGuard-only and `/opt/lchat` + `/etc/systemd/system` root-owned, there's no root shell to scp collectors onto a box, so the binary must place them. To refresh just the collectors + pages on a live box β€” no root shell, no router restart, no member-facing blip β€” run `sudo -n /opt/lchat/lchat bootstrap --collectors-only` (allowlisted by `deploy/sudoers-operators`'s `/opt/lchat/lchat *`). Only the `.html` + `.css` viewer assets are embedded; the collector-generated JSON stays mutable on the box. Details: `docs/status.md`. **There is no runtime dependency β€” not even for bots.** The sidecar is now the same `lchat` binary (`lchat bot run`; see *Bots*), so a box that runs the server can run its bots with nothing else installed. The Node reference (`agent/`) remains for self-hosters who want it, but it's optional; a stock box is pure static Go. (A bot can still run **off-box** β€” it's just a WebSocket client to the public URL β€” if you'd rather keep its keys on a different machine.) ## add-instance `lchat add-instance ` runs the steps an operator would otherwise do by hand: ``` mint token β†’ write /data/.env (0600, the unit's EnvironmentFile) β†’ register route (status: creating) β†’ systemctl enable --now lchat@ β†’ wait for the socket β†’ status: ready β†’ systemctl reload lchat-router β†’ print the admin link ``` The route's `status` (`creating`/`ready`/`failed` + `last_error`) is recorded in the registry and shown by `lchat instances`. The cert issues lazily on the host's first HTTPS hit (proven fine in practice); pre-warming it (`m.GetCertificate(&tls.ClientHelloInfo{ServerName: host})`) and a fully async, retrying state machine are refinements the schema already allows. `lchat admin-link ` recovers the admin link from the EnvironmentFile (the DB holds only the token's hash). ## Versioned, side-by-side deploys Per-version deploys are the point of this box, so they're first-class rather than "point a unit at a different file by hand." Two pieces make one instance's version independent of every other's: - **A content store**: built binaries live at `/opt/lchat/versions//lchat`, keyed by build id (the git short sha, stamped in at build via `-ldflags "-X main.buildVersion="` β€” see `deploy/build-linux.sh`). - **A per-instance pointer**: each instance's unit execs `/opt/lchat/live/`, a symlink into the store. The registry records each host's current + previous version (`lchat instances` shows it). ``` /opt/lchat/live/test.example.com ─▢ versions//lchat (one instance moved) /opt/lchat/live/family.example.com ─▢ versions//lchat /opt/lchat/live/paulca… ─▢ versions//lchat ``` **Flow (canary β†’ promote):** ``` ./deploy/build-linux.sh # build linux/amd64, stamped with the git sha scp lchat-linux-amd64 root@box:/opt/lchat/incoming ssh box lchat stage --file /opt/lchat/incoming # register it β†’ prints ssh box lchat deploy test.example.com --version # canary ONE instance # …verify test… ssh box lchat deploy --all --version # promote the rest ssh box lchat rollback test.example.com # or step back to its previous version ``` `stage` copies a binary into the store (reading its own build id via `lchat version`). `deploy --version ` atomically repoints `live/`, restarts, and health-checks β€” see below. `deploy ` with **no** `--version` just restarts on the current version (the binary's already in place). `rollback` repoints to the recorded previous version and restarts. `init-versions` is the one-time adoption step (stage the running binary, create a `live/` link for every existing instance) when moving a box onto this scheme. **The restart is a graceful drain + a retrying router** (rather than dual-socket blue/green β€” simpler, and enough since clients auto-reconnect): 1. `systemctl restart lchat@`. On **SIGTERM** the instance drains in-flight HTTP (`http.Server.Shutdown`) and closes WebSockets (so browsers reconnect), then exits; systemd starts the new process, which rebinds the socket. 2. Across the brief gap (old exited, new not yet bound β†’ dials fail `ENOENT`/`ECONNREFUSED`) the **router retries** the dial for a short window, so requests queue a beat instead of failing. 3. For a `--version` deploy, `deploy` polls `/api/version` until the instance reports the **expected `build`** (not merely "answers 200") β€” so a stale socket or a restart that didn't take fails loudly. On failure it **auto-rolls-back**: repoint the symlink to the previous version, restart, and error out. `--all` stops on the first failure (already-promoted hosts stay on the new version). Blast radius is **one instance**: its WebSocket clients reconnect once; HTTP requests see at most a little added latency. Canary = deploy one instance alone. Verified in practice: 120 requests against an instance during a live deploy all returned 200. The router itself changes rarely; when it does, its own front listener gets zero-downtime via `SO_REUSEPORT` / systemd socket activation. ## πŸ€– Auto-updating (release channels) Instances can keep themselves current without an operator driving `deploy` β€” built on the versioned-deploy machinery above plus the signed release feed described in `docs/server-releases.md`: - **`lchat update run`** (fired hourly by `lchat-update.timer`, installed + enabled by `bootstrap`; root, like `lchat-control`) fetches and Ed25519-verifies the channel manifest, asks each *ready* instance what its admin chose (`GET /api/update-prefs` over its unix socket β€” a pre-feature binary 404s, which reads as the defaults: auto-update on, stable channel), then per instance: downloads + sha256-checks + stages the channel's binary and applies it with the exact `deploy --version` dance (repoint β†’ restart β†’ health-check β†’ auto-rollback β†’ **restart the host's bot sidecar** so it tracks the new version too). A version that fails its health check on one host is not tried on the rest that pass ("poisoned" for the run); other per-host failures log and continue, so one unreachable instance never strands the fleet. - **Prefs are per-instance admin settings** (`update_auto`, `update_download`, `update_channel` in the instance's own config table, set from the admin consoles). Download-only means "staged, apply manually with `lchat deploy --version `". - **Pinning (the canary story).** In addition to a channel, an instance can be pinned to one exact version (`update_pin`) β€” the updater skips a pinned instance entirely and its build survives GC, so an hourly pass never stomps a deliberately-deployed canary back to the channel head. Pin beats channel. The canary flow is one command: `lchat deploy test.example.com --version --pin` (or `lchat update pin []`, defaulting to what it runs now). The pin is *instance state* β€” written through the instance's own admin API (token from its EnvironmentFile), reported by `/api/update-prefs`, and shown in its admin panel as a "Pinned to " entry in the channel picker; the admin resumes updates by picking a channel (one write that sets the channel and clears the pin), or the operator runs `lchat update unpin `. - **A deploy flock** (`/opt/lchat/.deploy.lock`) serialises `deploy`, `rollback`, and `update run` across processes; the loser errors immediately ("another deploy/rollback/update is running") instead of interleaving the swap dance. A timer pass that loses simply retries next tick. - **Anti-replay**: the newest accepted `generated_at` persists in `/opt/lchat/manifest-stamp`; older (validly signed but stale) manifests are refused, so the feed can be pulled backward only by *re-signing* β€” a deliberate act with the key. - **The bake window (supply-chain hold)**: beta/stable heads aren't installed until they've sat at the channel head for 24 h/72 h (+ per-box jitter) by the box's *own* clock (`/opt/lchat/channel-first-seen.json`) β€” a compromised release caught and pulled back inside the window never reaches subscribers, and a pull-back to a previously-seen version applies immediately. Local override: `lchat update run --ignore-bake` (root); design + numbers in `docs/server-releases.md Β§ The bake window`. - **Major version pinning**: `lchat update major ` (or the `update_major` admin setting) keeps an instance on one major β€” minors and patches keep flowing, a head whose `semver` crosses the pin is held (halt-and-notify; a semver-less release also holds, fail closed). Clear with `lchat update major clear`. See `docs/server-releases.md Β§ Major version pinning`. - **GC**: after a pass, version-store dirs nothing references (no route's current/previous, no channel head, no live symlink, not the control binary's own build) are pruned, and crashed `dl-*` downloads in `incoming/` are swept β€” hourly edge updates stay disk-bounded. - **`lchat update status`** prints HOST / CHANNEL / AUTO / CURRENT / AVAILABLE across the fleet; `lchat update run --dry-run` shows what a pass would do. ## Supervision (systemd) - `lchat-router.service` β€” `Restart=always`, socket-activated (its own zero-downtime). - `lchat@.service` β€” a template unit, one per instance. `MemoryMax=` + `CPUWeight=` give resource isolation; `Restart=always` gives crash isolation. Bump `LimitNOFILE` (instances Γ— WS connections Γ— SQLite files). - `lchat-bot@.service` β€” the optional per-instance sidecar (`lchat bot run`; see *Bots*). `Restart=always`; execs the same `live/%i` binary as the instance, so a deploy/update restarts it onto the matching version (it backfills on reconnect). ## Hardening Everything on one box means the box is the blast radius, so treat it as the security boundary. The shipped units and `bootstrap` already apply the service-level items below; the OS-level ones are a few one-time commands. **Run unprivileged.** `bootstrap` creates a locked system user (`lchat`) and owns `/data` + `/run/lchat` to it; every unit runs as that user. Only the router touches privileged ports, so it alone keeps `CAP_NET_BIND_SERVICE` (`AmbientCapabilities` + a matching `CapabilityBoundingSet`) to bind `:80`/`:443` as non-root; everything else binds a unix socket and needs no capabilities (`CapabilityBoundingSet=`). **Sandbox each unit.** The units set the usual systemd confinement: `NoNewPrivileges`, `ProtectSystem=strict` with `ReadWritePaths=/data /run/lchat` (the whole FS is read-only otherwise), `ProtectHome`, `PrivateTmp`, `PrivateDevices`, `ProtectKernel*`, `ProtectControlGroups`, `ProtectProc=invisible`, `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `RestrictNamespaces`, `SystemCallFilter=@system-service`, and `MemoryMax`. `systemd-analyze security ` should score each in the low single digits. > **`MemoryDenyWriteExecute` note.** Every unit β€” including the Go sidecar β€” > can set `MemoryDenyWriteExecute=yes`, since the static Go binary never maps > W+X memory. (The legacy Node sidecar was the one exception β€” V8's JIT maps > writable+executable pages, so it crashed on boot with that directive; a Go > sidecar has no such caveat and gets the same hardening as the router and > instances.) πŸ€– **OS level** (one-time): - **Firewall** β€” allow only `80`, `443`, and the WireGuard port (`51820/udp`) inbound; default-deny the rest. SSH gets **no public allow** β€” it is reachable only over the WireGuard interface (next bullet): ```sh ufw allow 80/tcp && ufw allow 443/tcp && ufw allow 51820/udp ufw allow in on wg0 to any port 22 proto tcp ufw enable # add the allows *before* enabling so you don't lock yourself out ``` Instances and services talk over unix sockets, so nothing else needs a public port. `sshd` itself is left unchanged (still listening on 22, key-only: `PasswordAuthentication no`) β€” the *firewall* does the gating, so there's no boot-order coupling between sshd and the tunnel. - **SSH over WireGuard** β€” operator SSH rides a WireGuard tunnel instead of a public port. WireGuard is silent to unauthenticated packets, so to a scanner the port doesn't exist: the constant background brute-forcing every public 22 attracts drops to zero, and the tunnel key becomes the hard boundary for box access. The box runs one `wg0` interface with a peer per operator; each operator's config points at the box and keeps the tunnel warm: ```ini # box: /etc/wireguard/wg0.conf β€” pick any private subnet for the tunnel [Interface] Address = .1/24 ListenPort = 51820 PrivateKey = [Peer] # one block per operator PublicKey = AllowedIPs = .2/32 # that operator's fixed tunnel address ``` ```ini # operator machine: the client side of the same tunnel [Interface] Address = .2/32 PrivateKey = [Peer] PublicKey = Endpoint = :51820 AllowedIPs = .0/24 # route only the tunnel subnet, not all traffic PersistentKeepalive = 25 # keep NAT mappings alive so ssh works instantly ``` Operator machines run the tunnel persistently (launchd/systemd) and add an `~/.ssh/config` alias whose `HostName` is the box's WireGuard address, so every existing `ssh ` command keeps working unchanged. - **Root SSH is bootstrap-only.** A box's *first* root session runs `lchat bootstrap`, then the operator-setup stage: create a named account per operator (groups `lchat-operators` + `systemd-journal`, the latter so operators can read journals), install the committed sudo allowlist `deploy/sudoers-operators` at `/etc/sudoers.d/lchat-operators` (mode `0440`) and the env-placement helper `deploy/lchat-env-install` at `/usr/local/sbin/lchat-env-install` (mode `0755`), add the WireGuard peers β€” and then disable root logins: `lchat bootstrap --ssh-noroot` writes the drop-in `/etc/ssh/sshd_config.d/10-lchat-noroot.conf` (`PermitRootLogin no`), validates with `sshd -t`, reloads sshd, and then **confirms the effective config** (`sshd -T`) actually reports `no` β€” failing loudly when it doesn't. That last check exists because sshd_config is first-match-wins and a box hardened before this verb existed can carry a drop-in that sorts *before* `10-` (kepler's `00-hardening.conf`, found 2026-07-23, silently kept `prohibit-password`); the fix is renaming the offender to sort after `10-` (keeping its other directives, e.g. `PasswordAuthentication no`, effective) and re-running the verb. The verb refuses until the `lchat-operators` group has members (a fresh box can't shut root out before anyone else can get in), and every full `bootstrap` run converges the same drop-in once operators exist β€” so a box that missed the step picks it up on the next provision pass, via the allowlisted `sudo -n /opt/lchat/lchat`, with no root shell needed. Thereafter operators are named users over the tunnel; root capability exists only through the allowlist, and anything new that needs root gets a line added **via a reviewed PR + redeploy**, never a root shell or a hand-edit on the box. **Operators who join after the lockdown** are added the same no-root-shell way: an existing operator runs `ops provision operator (--github | --keys-file )`, which drives the allowlisted `bootstrap --add-operator ` with the keys piped on stdin β€” account + groups + `authorized_keys` (replaced, so re-running is also key rotation). The account is only half of access: the joiner's WireGuard peer must be on the box's `wg0` too, or nothing routes. (Encoded after kepler was found carrying an operator's peer but no account, 2026-07-29.) Two small helpers back the allowlist, because a couple of routine privileged ops don't fit a fixed command line β€” install both alongside the sudoers file (`/usr/local/sbin/…`, mode `0755`): - **`deploy/lchat-env-install`** places a secret: `sops -d file.env | ssh sudo lchat-env-install `. - **`deploy/lchat-svc`** runs `systemctl` scoped to `lchat*` units. This indirection is *required* on Ubuntu 26.04, whose default sudo is **sudo-rs** (the Rust rewrite): sudo-rs rejects argument wildcards, so a sudoers line like `systemctl restart lchat-*` is a syntax error β€” only a whole-argument `*` is allowed. The helper takes the wildcard (`lchat-svc *`) and enforces the `lchat*` unit prefix itself. Operators run `sudo lchat-svc restart lchat-bot@`, `sudo lchat-svc daemon-reload`, etc. (Keep every other command spec in the sudoers file either exact-args or ` *` for the same reason.) Every operator SSH session is published to the public status ledger. Be honest about what this buys: the allowlist includes deploying binaries, so it is accident-proofing, attribution, and a reviewable ops surface β€” **not** a hard security boundary; the hard boundary is the WireGuard key. - **Break-glass** β€” with no public SSH, a lost tunnel key or a broken `wg0` is recovered through the hosting provider's out-of-band path: the web console, or rescue mode (boot the rescue system, mount the disk, repair the config). Rare true-root operations (OS surgery, filesystem repair) go the same way β€” they're what the console is for, not a reason to keep a root login open. - **The deadman pattern** β€” when flipping any remotely-reachable config (firewall rules, sshd drop-ins, the WireGuard interface), schedule the automatic revert *first*, verify from outside over a *fresh* connection, and only then cancel the revert β€” so a mistake un-does itself instead of locking you out: ```sh systemd-run --on-active=10min --unit=sshflip-revert ufw disable # the safety net # …apply the change, verify from a NEW connection from outside… systemctl stop sshflip-revert.timer # all good β€” disarm ``` - **Automatic security updates** β€” enable `unattended-upgrades`. - If the box hosts its own services, pin their hostnames to loopback in `/etc/hosts` (see *Bots*). ## Backups Collapsing onto one box means every instance's data is on one disk β€” a single failure loses everything, and there's no platform to restore from. - **Litestream per instance DB** β†’ object storage, point-in-time restore. - **Back up the autocert cache dir** too β€” it holds issued certs *and their private keys*. Treat as key material (`0600`); losing it forces re-issue. - Encrypted attachments are stored as blobs **inside** SQLite (the `blobs` table), so the DB backup covers them β€” but they're also what drives disk growth (see sizing). ## Sizing Server-side memory is light **by design**: crypto and image downscaling happen in the browser; the server only moves ciphertext. A modest box goes a long way. - **RAM:** each instance is ~20–40 MB idle (Go runtime + modernc SQLite + embedded assets); the router ~20 MB. A dozen instances is a few hundred MB. Set `MemoryMax` per instance so one can't OOM the box. - **CPU:** no server-side crypto or image work β€” just HTTP, SQLite, WebSocket fan-out, and TLS termination. Two vCPUs handle a small fleet comfortably. - **Disk is the binding resource, not RAM.** Attachments are blobs in SQLite, so DBs grow with media. Watch disk, and let Litestream offload history. If you outgrow anything first, it'll be disk. ## Migrating an existing deployment onto the box Moving an instance from wherever it runs now onto this setup is data movement plus a DNS flip: 1. **Snapshot the DB cleanly.** `VACUUM INTO` a fresh copy (folds the WAL into one consistent file); briefly freeze writes for a clean cut. 2. **Copy it over** to the box as `/data/.db`. 3. **Register it** (`add-instance ` against the copied DB), which warms the cert. 4. **Flip DNS** for the host to the box's IP; verify; retire the old deployment. Two things make this safe: - **Keep the hostname the same and there's no key work.** Browser keys (and passkeys, for the home server) are per-origin, so an unchanged origin leaves members' keys and history untouched β€” only the DNS record moves. - **A host on a domain not covered by a wildcard cert** simply exercises the on-demand HTTP-01 path β€” no special handling. **Rehearse the whole flow with a throwaway instance first** β€” under a temp host, run the router, `add-instance`, cert issuance, a socket-swap deploy, a crash-restart, and a Litestream restore, all at zero real-user risk before touching anything real. ## What you're trading away (honest costs) 1. **The router is a new SPOF** for all instances. But it's tiny, holds no per-request state, is `Restart=always` (sub-second recovery), and is smaller and more stable than a single-process design's whole-app SPOF. Its own deploys are socket-activated. 2. **You own backups + supervision now.** Litestream + systemd cover what a platform did, but it's your operational surface β€” it has to actually be set up and monitored. 3. **One box.** Multi-node failover is exactly where an orchestrator earns its keep; this is deliberately single-box. Revisit if that changes. 4. **Migration is a real one-time project** β€” snapshot DBs, flip DNS, retire the old deployment, per instance. ## Testing - Router: hostβ†’socket routing, unknown-host β†’ 404, `HostPolicy` allowlist, atomic upstream swap under concurrent load, WebSocket pass-through. - `add-instance` state machine: creating / issuing / failed+retry transitions. - Instance: unchanged β€” `handler()` tests via `newTestServer` still exercise the real router. - Deploy: a socket swap drains without dropping an in-flight request (integration). ## Deploy tooling (replaces deploy/miren.sh) - `lchat bootstrap` β€” one-time box setup: write systemd units, make dirs, start the router, and install the status collectors + viewer pages + their timers (no toolchain needed β€” just the copied binary). `--collectors-only` refreshes just the collectors/pages on a live box (no root shell, no router restart). - `lchat add-instance ` β€” create + start + warm cert (async; `instances` shows status). Stages the running binary + points the host's `live` link first. - `lchat stage --file ` β€” register a built binary in the version store. - `lchat deploy [--version ]` β€” zero-downtime restart; with `--version`, repoint the host at that staged version and health-check the `build` (auto-rollback on mismatch). - `lchat deploy --all [--version ]` β€” roll the fleet, one instance at a time. A `--version` sweep **skips pinned instances** (a pin is someone's live canary; the pin used to gate only the auto-updater, and a manual promote could stomp it) β€” `--include-pinned` overrides deliberately, and naming a host explicitly always deploys it. - `lchat rollback [--all]` β€” step back to the recorded previous version. - `lchat init-versions` β€” one-time adoption: seed the store + create `live` links for existing instances. - `lchat version` β€” print the binary's build id (also at `/api/version`). - `lchat admin-link ` β€” recover the admin link from the instance DB (no platform env lookup needed). - `lchat instances` β€” the status table (now with a VERSION column). - `lchat controld` / `lchat cloud` β€” the local control daemon + the public self-serve provisioner built on it (see `cloud-provisioner.md`). Operator specifics β€” box addresses, DNS/record ids, the concrete migration runbook β€” live in `CLAUDE.local.md`, not here.