🤖 Running the fleet · Part 10 of 10

Complete reference: commands, files, variables

Everything in one place, for looking up rather than reading through: every operator command with its flags, every environment variable with its default, every file path, every systemd unit, and the glossary. Each entry links back to the page that explains it.

The map one last time, unhighlighted. If a term below is unfamiliar, the glossary at the bottom of this page defines it.

Command reference: the lchat control commands

All of these run on a box (over ssh, which rides the WireGuard tunnel — part 08), as a named operator invoking /opt/lchat/lchat through the committed sudo allowlist (root SSH is for first bootstrap only). Common conventions: <host> is always a full domain name; flags may come before or after the host; most commands accept --registry <path> to point at a non-default registry (you will essentially never need it — the default is /data/registry.db).

Box and instance lifecycle

lchat bootstrap [--links-apex <host>] [--no-update-timer] [--no-start] [--collectors-only]

One-time box setup: writes every systemd unit file, creates the lchat system user and the /data, /run/lchat, and certificate-cache directories, mints the root-only control token, and enables and starts the router and the hourly update timer. Idempotent — safe to re-run to refresh unit files, and re-running is the only thing that installs a newly-added unit on an existing box (deploy restarts only units that already exist). --links-apex <host> makes this box the short-link apex for that host; without it, the box converges to whatever the registry already says. --no-update-timer disables the hourly auto-updater rather than enabling it, for boxes that must never track a release channel — the dev mesh runs unmerged code, so an updater pass would replace it with the current stable build. It disables rather than merely skips, so the end state follows the flag rather than the box’s history. Explained in part 04.

lchat add-instance <host> [--admin-token <tok>] [--hibernate] [--home-region <cc>] [--app-store-url <url>] [--home-origin <url>] [--push-relay-url <url>] [--push-relay-token <tok>]

Creates a space: mints (or, with --admin-token, imports) the admin token, writes /data/<host>.env, registers the route, points the version link, starts and health-checks the unit (skipped with --hibernate), reloads the router, warms the certificate, and prints the one-time setup link and the admin link. Explained in part 05.

lchat add-service <host> --socket <path>

Registers a service-kind route (home server, push relay, cloud provisioner, or any third-party backend) after verifying the socket answers. No unit is created and no version is tracked — the router only forwards. Explained in part 03.

lchat add-bot <host> [--model <model>]

Provisions the bot sidecar for an instance: mints one agent invite per identity, writes /data/<host>-bot.env and the session directory, and enables lchat-bot@<host> (on a hibernating host the unit is skipped; the activator co-spawns instead). Reads ANTHROPIC_API_KEY from the calling environment if AI behaviour is wanted. Explained in part 03.

lchat add-hooks <host> built, not yet deployed

Provisions the webhook-ingest sidecar for an instance (incoming webhooks, issue #610): mints the service identity’s agent invite, writes /data/<host>-hooks.env and the session directory, registers the in.<host> registry route (kind hooks) pointing at /run/lchat/<host>-hooks.sock, and enables lchat-hooks@<host>. Admins then create webhooks from the app’s Webhooks panel. Design and trust model: docs/webhooks.md.

lchat remove <host> [--stop]

Deletes the host’s registry row and reloads the router (traffic stops); with --stop also stops and disables the instance and sidecar units. Never touches data. Explained in part 05.

lchat hibernate <host> --on | --off

Moves one host between systemd management and activator management, in either direction, safely ordered so the host is never unserveable. Requires the router to be running with a hibernation bucket for --on. Explained in part 06.

lchat replication <host> [stream | batched | daily]

Reads (no argument) or sets a hibernating host’s replication class — how often awake writes sync to the bucket. A set reloads the router and applies at the host’s next wake; on provisioned fleets billing drives this instead. Explained in part 06.

lchat instances

Prints the registry: HOST, KIND, STATUS, VERSION, HIBERNATE, REPLICATION, SOCKET, plus a last error line for any failed row. The first stop for “what does this box think it is serving?”.

lchat mesh-seed [--region <cc>] <peer-url>

Run on an existing mesh box, pointed at a newly joined one: bulk-replicates every in-scope hibernating route to the peer in one pass, authenticated by LCHAT_MESH_TOKEN. The one-time catch-up step of adding a box to the mesh. On a region-scoped fleet, pass --region with the joining box’s LCHAT_MESH_REGION so routes it owns by home_region are seeded too, even ones outside this seed’s LCHAT_MESH_DOMAINS (the multi-edge region-owner union); omitting it seeds by domain exactly as before. Part 07.

lchat relocate <host> --to <region> [--dst-bucket <b>] [--move-dns --dns-target <ipv4>]

Move a hibernating space to another region without changing its address: copies its store to the destination region (a no-op on a shared bucket), flips home_region, and re-fans-out the route so the mesh re-homes it. The lease keeps a single writer throughout. Run on a mesh box with /data/router.env sourced. Part 07.

lchat admin-link <host> / lchat setup-invite <host>

Recover the admin link (from the environment file — the only plaintext copy of the token) / mint a fresh one-time setup invite.

Versions and deploys

lchat stage --file <binary> [--version <ver>]

Copies a built binary into the version store under its build identifier (asked of the binary itself unless overridden). Inert and always safe. Part 04.

lchat deploy <host>|--all [--version <ver>] [--pin] [--include-pinned]

With --version: repoint the host’s live link, restart, poll /api/version for the exact build, auto-roll-back on mismatch, restart the bot sidecar. Without: restart on the current build. --all rolls one host at a time, stops on first failure, and skips pinned hosts unless --include-pinned. --pin (single host + version only) freezes the host there. Hibernating hosts: pure repoint, effective at next wake. Part 04.

lchat rollback <host> | --all

Repoints to the recorded previous version, restarts, health-checks, and swaps current/previous (so rollback twice returns forward). Part 04.

lchat init-versions [--from <binary>]

One-time adoption when moving an existing box onto versioned deploys: stages a seed build and creates missing live links. Idempotent; never repoints an existing link.

lchat update run [--dry-run] / status / pin <host> [<ver>] / unpin <host>

One updater pass (what the hourly timer fires) / the fleet update table (HOST, CHANNEL, AUTO, CURRENT, AVAILABLE) / freeze a host on a version / resume its channel. Part 04.

lchat version

Prints the binary’s own build identifier (also served by every instance at GET /api/version as build).

Daemons and utilities

lchat router

The router process itself — run by lchat-router.service, not by hand. Flags --registry, --acme-cache, --https, --http, --fallback-url, --wildcard-domains. Part 02.

lchat controld

The root-only control daemon — run by lchat-control.service. Flags/env for socket, token file, and capacity policy are under environment variables. Part 05.

lchat cloud --socket <path> --db <path>

The self-serve provisioner — run by lchat-cloud.service. Part 05.

lchat links run --socket <path>

The short-link redirector behind 11ch.at (message permalinks, issue #905) — run by lchat-links.service. Exactly one box in a fleet is the short-link apex, and only that box runs the redirector; everywhere else the unit is written and left disabled. You declare the apex with lchat bootstrap --links-apex <host>, which starts the redirector and registers <host> as a service route pointing at /run/lchat/links.sock.

That route — not the flag — is then the source of truth: later plain bootstrap runs read the registry and converge to it, so re-running bootstrap on the apex box keeps short links up, and running it on any other box keeps the redirector off. (A flag-keyed design would silently switch the apex off the first time someone re-bootstrapped it without remembering the flag.)

Note the apex needs its own DNS A record and certificate: a *.example.com wildcard does not cover bare example.com. The router issues the certificate automatically once the route exists — an unrouted host gets no certificate at all, which is how a missing route surfaces: a TLS handshake failure rather than a 404.

Ours to serve, in practice. Two values in linkscmd.go are Eleven-specific and not configurable: the apex bounces to https://elevenmessenger.com, and the apple-app-site-association advertises our iOS app id. Pointing your own apex here would send your visitors to our marketing site and publish our app id under your domain. Making those configurable is what self-hostable short links would take.

lchat bot run [--base <url>] / lchat bot provision

Run a bot sidecar (with no arguments, configuration comes from the unit’s environment file) / claim an agent invite into a session file. Part 03.

lchat release keygen|sign|verify|payload

The release-feed signing plumbing — used by the publishing pipeline, not in box operation. Part 04.

lchat help

Prints every command on this page, one line each — the fastest way to check a verb’s exact name on a box. Note that an unrecognised command (a typo like lchat routes) now prints this same list to stderr and exits 2. It used to fall through and start a messenger server instead, which either died on a port clash or, if the port was free, kept running unnoticed.

Environment variable reference

Grouped by the file each is read from. All files are plain KEY=value lines, mode 0600, under /data/ unless noted.

The router — /data/router.env

Each instance — /data/<host>.env

The control daemon — /data/control.env

The cloud provisioner — /data/cloud.env

Other files

File path reference

systemd unit reference

Covered in narrative form in part 03; the lookup table:

Health and inspection endpoints

Glossary

space / instance / host

One chat group; its server-side unit (one process + one SQLite database + one subdomain); and that subdomain itself, which names everything belonging to the instance. See the vocabulary table.

box

One Linux server in the fleet, running the router and everything in part 03.

systemd

The standard Linux service manager: it starts processes at boot, restarts them when they crash, applies sandboxing, and keeps their logs (the “journal”). Controlled with systemctl, read with journalctl.

unit / template unit

A systemd service definition; a template unit (name containing @) is stamped out per instance name, which is how one file defines the service for every chat space.

unix socket

A file (here under /run/lchat/) that local processes connect to like a network address. Invisible to the network — the reason instances need no firewall rules of their own.

the registry

The SQLite file /data/registry.db: one row per hostname a box serves. The router’s routing table and the fleet commands’ ledger. Part 02.

DNS / wildcard DNS

The system translating names to network addresses. A wildcard record (*.example.com) points every subdomain at once, which is why creating a space needs no DNS step.

Let’s Encrypt / ACME / HTTP-01 / DNS-01

The free certificate authority and its automated issuance protocol. HTTP-01 proves control of a hostname by answering a challenge on port 80 (used per host); DNS-01 proves control of a whole domain via a temporary DNS record (used for wildcard certificates). Part 02.

object storage / S3 / bucket / prefix

Storage addressed as named objects in a “bucket” rather than files on a disk — durable, replicated, and shared. A “prefix” is the leading part of object names, used like a directory. Amazon S3 is the implementation used here.

Litestream

A tool that continuously streams a SQLite database’s changes into object storage, allowing restore to any recent moment. Used twice in this system, in two forms: a stock copy for fleet backups, and embedded inside the router for hibernation.

the lease

The per-space lock object (lock.json) in the hibernation bucket, held while a process serves the space and renewed every few minutes. The system’s single source of truth about who may write a space’s data. Part 06.

replication class

A hibernating host’s durability rung (stream / batched / daily): how often its awake writes sync to the bucket, set per plan by billing or per host by lchat replication. Graceful sleeps and deploys always flush in full regardless. Part 06.

the activator

The component inside the router that wakes sleeping spaces on demand and puts idle ones to sleep. Exists only when LCHAT_HIBERNATE_BUCKET is set. Part 06.

rendezvous hashing

The rule mesh boxes use to agree which of them owns each space without communicating: every box scores every (box, space) pair the same way, and the highest score wins. Adding or removing a box only moves the spaces whose winner changed. Part 07.

bot sidecar

The separate process hosting an instance’s bot members. It holds the bots’ encryption keys (which must never live in the server process) and connects outward as an ordinary client. Part 03.

canary / pin

Deploying a new build to one instance to watch it before promoting it everywhere; and the mark that freezes an instance on a version so the hourly auto-updater leaves the experiment alone. Part 04.

web push / VAPID key

Browser push notifications, signed per instance with a key pair (the .vapid file). Distinct from Apple device push, which goes through the push relay.