# 🤖 The operator surface: `./deploy/ops`

One command surface for deploying and managing the fleet, so operating doesn't
mean re-gathering knowledge from `CLAUDE.md`, `docs/server-releases.md`,
`docs/multi-edge-dev-cutover.md`, and the vault's `RUNBOOK.md` every time.
Bare `./deploy/ops` lists the verbs:

```sh
./deploy/ops doctor              # can I deploy right now? (read-only preflight)
./deploy/ops validate            # run every local check (go|web|shell|infra)
./deploy/ops provision units     # re-run bootstrap so NEW systemd units land on the fleet
./deploy/ops provision apex 11ch.at   # declare a box the short-link apex
./deploy/ops provision status    # read-only: which box runs the short-link redirector
./deploy/ops publish "notes"     # build main -> release -> point edge at it
./deploy/ops promote <ver> beta  # move a channel pointer (edge|beta|stable)
./deploy/ops ship                # every fleet box catches up NOW
./deploy/ops canary create x     # per-session test instance (test-x.11ch.at)
./deploy/ops canary deploy x     # build THIS branch onto it — always pinned
./deploy/ops canary reset x      # wipe back to first-run (iterate onboarding)
./deploy/ops canary destroy x    # reset + remove, data archived to trash
./deploy/ops dev plan            # dev-mesh tofu plan, AMI-replacement guarded
./deploy/ops dev apply           # tofu apply behind the same guard
./deploy/ops dev deploy          # dev mesh: build -> scp -> swap -> restart
./deploy/ops dev provision       # dev mesh: refresh systemd units + route short links
./deploy/ops dev rebuild         # destroy + recreate the Dublin box (new keys)
./deploy/ops ledger "why..."     # draft the public SSH-access why-entry
./deploy/ops pages push          # committed web/status data -> the status box
```

Verbs that need credentials fetch them themselves (the `release-server.sh`
shape): `publish`/`promote` pull the signing key from Bitwarden, and `ops dev`
pulls `AWS_*` from your per-operator dev agent item and `DNSIMPLE_*` from the
DNSimple token item — unlocking interactively when the vault is locked,
degrading to instructions (never a hang) without a tty, and syncing bw's
local cache when a lookup looks stale. Values already in the environment
always win, and no secret is ever printed. Which items, and where each value
lives inside them, is the **vault contract** — `deploy/vault-items.sh`, the
one place scripts and vault agree: scripts read exactly what it says and fail
crisply when the vault doesn't match (never a fuzzy search), and renaming a
vault item is a reviewed PR there plus the vault edit, in step.

Each verb is a readable standalone script beside the dispatcher
(`deploy/ops-*.sh`, `deploy/doctor.sh`) — the outlive-AI rule: no runtime, no
machinery only an LLM can drive. The verbs *encode the rules* that used to
live only in prose: publish refuses a HEAD that isn't landed main; there is no
`dev` channel; canary deploys always `--pin`; canary destroy wipes data the
way `lchat remove` deliberately doesn't (via the canary-gated `lchat reset` —
the sudoers allowlist has no `rm`); fleet sweeps leave `AUTO=pinned` rows
alone; a `tofu plan` that would replace the Dublin box refuses to look green.
When a verb's advice goes stale, **fix it in the script, not in prose** — a
script that runs drifts loudly, where a document drifts silently.

**The standing rule — no cookbooks.** Operational procedure lives in these
scripts and nowhere else. When a task has no verb, or a verb falls short,
**extend ops** (a new verb, subcommand, or flag in a small PR) and then run
it — never hand-run a sequence reconstructed from `docs/`, `CLAUDE.md`,
`CLAUDE.local.md`, or an old runbook. The docs are design rationale and
history; each time one is caught carrying live procedure, encode the
procedure here and banner the doc. This is what keeps operations safe and
repeatable for humans and agents alike.

Start with the preflight. `ops doctor` checks local tooling, the Bitwarden
signing key (unlocking to verify by default — `--no-unlock` prevents it; the
session token stays in that one process and is never printed), `gh` write
access to the releases repo, WireGuard + SSH + `lchat-operators` sudo on every
fleet box, and the dev mesh's key/profile/firewall/SSH — printing the fix
under each failure. It is read-only, never prints a secret, and exits non-zero
on hard failures only, so it works as a gate in other scripts.

Two obligations the surface reminds you of, because they are easy to forget:
box-SSH verbs (`ship`, `canary`, `dev deploy`, doctor's `fleet` section — use
`--no-ssh` to avoid it) incur a **why-entry in the public access ledger**
(`web/status/ssh/`), which `ops ledger` drafts; and box aliases come from your
`~/.ssh/config` (defaults in the scripts, overrides via `LCHAT_BOXES` /
`LCHAT_CANARY_BOX` / `LCHAT_DEV_BOX_*`, environment specifics in
`CLAUDE.local.md`).

# 🤖 Single-machine bootstrap

Running the messenger behind the `lchat` router on one box (see
`docs/instance-router.md`). The Go side is a single static binary — no runtime.

```sh
# 1. copy the binary, then let the binary provision the box
scp lchat root@<box>:/opt/lchat/lchat
ssh root@<box> '/opt/lchat/lchat bootstrap'   # writes units + dirs, starts the router

# 2. add an instance (mints token + env file, starts it, registers + reloads)
ssh root@<box> '/opt/lchat/lchat add-instance chat.example.com'

# 3. operator setup — the last thing the root session does before retiring itself:
#    create a named account per operator (groups lchat-operators + systemd-journal),
#    install deploy/sudoers-operators -> /etc/sudoers.d/lchat-operators (0440)
#    and deploy/lchat-env-install + deploy/lchat-svc -> /usr/local/sbin/ (0755),
#    add each operator's WireGuard peer, close public 22, and run
#    `lchat bootstrap --ssh-noroot` — writes the PermitRootLogin-no drop-in
#    (/etc/ssh/sshd_config.d/10-lchat-noroot.conf), validates, reloads sshd,
#    and confirms the EFFECTIVE config reports no (sshd is first-match-wins;
#    a pre-existing drop-in sorting before 10- silently wins otherwise).
#    It refuses until lchat-operators has members, and every later full
#    bootstrap converges the same drop-in — a box that missed the step picks
#    it up on the next provision pass, root shell not required.
```

`lchat bootstrap` embeds the unit files (source `deploy/systemd/`) and writes
them, creates `/data` + `/run/lchat`, and starts the router. It also installs
the **public status collectors** — the SSH access ledger + fleet hardening
posture scripts (`deploy/{ssh-ledger-collect,hardening-posture}.sh`), their four
systemd units, and the static viewer pages (`web/status/*.html` + `status.css`,
into `/opt/lchat/status-pages/`) — and enables their 5-minute timers. These ride
the binary because a hardened box has no root shell to scp them onto (operator
SSH is WireGuard-only; `/opt/lchat` + `/etc/systemd/system` are root-owned). To
refresh just the collectors + pages on a live box without touching the router,
run `sudo -n /opt/lchat/lchat bootstrap --collectors-only` (allowlisted by
`deploy/sudoers-operators`) — no member-facing blip. The collector-generated
JSON (fleet/ssh/hardening/… data) stays mutable on the box; only the `.html` +
`.css` viewer assets are embedded. To install by hand instead: `scp
deploy/systemd/*.service` to `/etc/systemd/system/`, drop a
`/etc/tmpfiles.d/lchat.conf` (`d /run/lchat 0710 root root -`), `daemon-reload`,
`systemctl enable --now lchat-router`.

**Root SSH is for first bootstrap only.** After step 3, operators are named
users over the WireGuard tunnel, and root capability exists only through the
committed sudoers allowlist (`deploy/sudoers-operators`) — anything new that
needs root gets an allowlist line via a reviewed PR, never a root shell. The
full recipe (firewall, WireGuard peer configs, break-glass via the provider's
rescue console, the deadman revert pattern): `docs/instance-router.md
§ Hardening`.

- `lchat-router.service` — the front door (binds :80/:443, autocert HTTP-01).
- `lchat@.service` — an instance template; runs one instance on
  `/run/lchat/<host>.sock`, reading its token + config from `/data/<host>.env`.
- `lchat-bot@.service` — the optional per-instance Node sidecar (see Bots).
- Runs as root here so it can bind :80/:443; for a non-root user add
  `AmbientCapabilities=CAP_NET_BIND_SERVICE` to the router unit and a dedicated
  `User=`/`Group=` with matching ownership on `/run/lchat` + `/data`.

## Bots (sidecars)

The sidecar is the **same Go binary** as the instance — `lchat bot run` — so a box
that runs the server runs its bots with **no extra install** (no Node runtime). The
`lchat-bot@.service` template execs the instance's own versioned live symlink
(`ExecStart=/opt/lchat/live/%i bot run`), so a bot always runs the **same version
as its app**, and `deploy`/`update` restart it automatically when they move the
instance (`docs/instance-router.md § Bots`, § Auto-updating).

```sh
# per instance: mint the three agent invites + start the sidecar
# (API key via env, not a flag, so it doesn't leak into ps/shell history)
ssh root@<box> 'ANTHROPIC_API_KEY=sk-ant-... /opt/lchat/lchat add-bot chat.example.com'
```

`add-bot` mints an agent invite per identity (Claude / Eleven) via the
instance's admin API, writes `/data/<host>-bot.env` (invites + `LCHAT_BASE` +
`LCHAT_*_SESSION` paths + optional `ANTHROPIC_API_KEY`), and enables
`lchat-bot@<host>` — which reads all of that from the env file, so there's nothing
per-host to configure in the unit. Omit `ANTHROPIC_API_KEY` for a deterministic-only
bot (Eleven still welcomes new members + answers slash commands). Start the
**instance before** the sidecar so Eleven baselines the roster correctly.

> The Node reference sidecar (`agent/`) is kept on purpose as a runnable example;
> self-hosters who prefer it still need a Node runtime + `npm ci` in `/opt/lchat/agent`
> and point the unit's `ExecStart` at `node /opt/lchat/agent/agent.mjs`. The default
> deployment is Go.