Day-to-day operations and checks

You are here: everywhere. Operations cut across every component, which is why this page leans on the earlier ones for depth.

The routine health check, in ninety seconds

# 1. Are all supervised services up, and has anything been crash-looping?
systemctl list-units 'lchat*' --failed
systemctl status lchat-router lchat-control lchat-litestream --no-pager | grep -E 'Active|since'

# 2. Does the registry agree — is every intended host ready?
lchat instances                       # STATUS=failed rows need their last error read; CANARY=canary rows are throwaway test hosts, safe to reset/remove

# 3. Is the fleet current on its release channels, and whose canaries are pinned?
lchat update status                   # AUTO=pinned rows are someone’s live experiment — leave them be
                                      # (a throwaway test host is also flagged: lchat add-instance --canary, or lchat canary <host> --on; shows as CANARY in lchat instances)

# Reset a canary to a fresh first-run (the operator-clean wipe — no `rm` in /data):
lchat reset <host> --yes             # CANARY-ONLY: refuses any instance not flagged (the data-safety gate).
                                      # Stops the unit, archives its DB to /data/trash (recoverable; env/config kept),
                                      # restarts fresh, and prints the onboarding link. A real instance can never be reset.

# 4. Is disk comfortable? (disk, not memory, is the binding resource on these boxes)
df -h /data

# 5. Does one real space answer end to end, through DNS, TLS, router, and instance?
curl -s https://chat-a.example.com/api/version

For the fleet-wide view without logging into boxes, the public status dashboard (below) publishes instance counts, awake/asleep split, capacity bands, version spread, and replication health every five minutes.

Reading the logs: the lines worth knowing

Everything logs to the systemd journal. The single most informative journal is the router’s, because the activator and mesh live inside it (part 06):

journalctl -u lchat-router -f                     # follow everything: routing, certificates, wakes, sleeps
journalctl -u lchat@chat-a.example.com -n 200     # one systemd-managed instance
journalctl -u lchat-update.service -n 100         # what the last hourly update pass did
journalctl -u lchat-litestream -n 50              # backup replication health

Backups: continuous replication, and the restore you should rehearse

Every database on the box — instance databases, the registry, the home, push, and cloud databases — is continuously replicated to a dedicated backup bucket by a stock Litestream process (lchat-litestream.service, configuration /etc/litestream.yml). This is replication as a property of normal operation, not nightly snapshots: recovery-point loss is measured in seconds, and point-in-time restore is available. Three properties of the arrangement matter operationally:

• The configuration is generated, not written. deploy/fleet-backup.sh regenerates /etc/litestream.yml from what is actually in /data; run it after adding instances so new databases get picked up. It deliberately skips hibernating hosts — their replication belongs to the activator, and two replicators on one database corrupt each other (part 06).
• The bucket is the ransomware posture. It keeps old versions of every object for fourteen days, and the box’s credentials can write objects but can never change versioning, lifecycle, or the bucket itself — so even an attacker with the box’s own keys cannot destroy the history they overwrite.
• Blobs and the certificate cache ride separately. Attachment files (/data/<host>-blobs/) are immutable and content-addressed; the certificate cache (/data/acme/) is key material. Both belong in whatever file-level backup wraps the box.

Restoring a database (rehearse this before you need it)

1. Always restore to a scratch path — never over a live file.

   litestream restore -config /etc/litestream.yml \
     -o /tmp/restore-check/chat-a.example.com.db /data/chat-a.example.com.db
   sqlite3 /tmp/restore-check/chat-a.example.com.db "PRAGMA integrity_check"
   ok

2. Only after verifying, swap it into place with the instance stopped.

   systemctl stop lchat@chat-a.example.com
   mv /data/chat-a.example.com.db /data/chat-a.example.com.db.broken
   mv /tmp/restore-check/chat-a.example.com.db /data/chat-a.example.com.db
   chown lchat:lchat /data/chat-a.example.com.db
   systemctl start lchat@chat-a.example.com

For a hibernating host the parked copy in the hibernation bucket is usually the freshest state; a cold wake restores it automatically. The backup bucket is your answer when the hibernation copy itself is what was damaged.

Moving an instance between boxes

Migration is data movement plus a DNS change — nothing about members’ keys or history is affected, provided the hostname never changes (browser encryption keys and passkeys are bound to the origin). The procedure, condensed from the repository’s validated runbook (docs/instance-migration.md):

1. Recover the admin token and per-instance configuration from the source box.

   Both live in /data/<host>.env. The token must be preserved — importing it on the destination (rather than minting fresh) is what keeps every saved admin link working.

2. Snapshot the database cleanly, on the live system.

   sqlite3 /data/chat-a.example.com.db "VACUUM INTO '/tmp/chat-a-snapshot.db'"
   sqlite3 /tmp/chat-a-snapshot.db "PRAGMA integrity_check"
   ok

   VACUUM INTO produces a single, transaction-consistent, compacted copy and is safe against a running instance — no stop required for the snapshot itself.

3. Copy three things, not one.

   The snapshot to /data/<host>.db on the destination; the blob directory (rsync -a /data/<host>-blobs/ … — immutable files, safe to copy live); and the web-push key file /data/<host>.vapid, without which every browser push subscription silently breaks.

4. Import on the destination, preserving the token.

   lchat add-instance chat-a.example.com \
     --admin-token "<the existing token>" \
     --push-relay-url https://push.example.com --push-relay-token "<existing relay token>"

   add-instance adopts the pre-placed database, starts the unit against it, health-checks, and reloads the router (part 05). Verify over the local socket before touching DNS.

5. Check the gap, then flip DNS.

   Compare message counts between source-now and the snapshot; if the space is busy, freeze writes and re-snapshot at cutover. Lower the DNS record’s TTL to about sixty seconds a while before the flip. The TLS certificate cannot be pre-issued (issuance validates over public DNS), so it issues on the first request after the flip — a few seconds, once.

6. Verify, and keep the source as the instant rollback.

   Leave the source instance running until you are satisfied: flipping DNS back is a complete rollback. Bot sidecars are their own step — their identity files (/data/<host>-bot/) must be copied, never re-provisioned with add-bot (that would mint duplicate bot members).

The hardening checklist (one-time per box)

• Firewall: allow inbound 80, 443, and the WireGuard port (51820, UDP) only; default-deny the rest. SSH gets no public allow — it is permitted only on the WireGuard interface (ufw allow in on wg0 to any port 22 proto tcp). Everything internal is unix sockets, so nothing else needs a port. (Add the allows before enabling the firewall, and verify the tunnel from outside before removing any public SSH rule, so you do not lock out your own session.) running today
• SSH rides WireGuard: each operator is a peer on the box’s wg0 interface and connects to the box’s tunnel address; the tunnel runs persistently on the operator’s machine with an ssh configuration alias, so every existing command works unchanged. WireGuard does not answer unauthenticated packets, so the SSH port is invisible to scanners and brute-force noise drops to zero. Key-only remains in force (PasswordAuthentication no). The full recipe, including the break-glass path through the hosting provider’s rescue console, is in docs/instance-router.md under Hardening. running today
• Root SSH is bootstrap-only: the first root session bootstraps the box and creates named operator accounts (group lchat-operators), installs the committed sudo allowlist (deploy/sudoers-operators → /etc/sudoers.d/lchat-operators) and the lchat-env-install helper for placing secrets, then sets PermitRootLogin no with lchat bootstrap --ssh-noroot (refused until operator accounts exist, so a fresh box cannot shut root out before anyone else can get in; every full bootstrap run converges the same drop-in once they do, so a box that missed the step picks it up on the next provision pass — no root shell needed). The verb confirms the change actually took effect (sshd -T) and fails loudly when an earlier-sorting sshd_config.d file overrides the drop-in, rather than reporting a lockdown that is not real. From then on operators are named users whose root capability is exactly the allowlist — anything new that needs root gets an allowlist line through a reviewed pull request, never a root shell. This is accident-proofing and attribution with a reviewable surface, not a hard security boundary (the allowlist can deploy binaries); the hard boundary is the WireGuard key. Operators who join after the lockdown are added the same no-root-shell way: ops provision operator <name> <box> drives the allowlisted bootstrap --add-operator over the tunnel, piping the joiner’s public keys on stdin (account, groups and authorized_keys in one idempotent verb; re-running it with fresh keys is also the rotation path). The account is half of access — the joiner’s WireGuard peer must be on the box’s wg0 too. running today
• Automatic security updates for the operating system (unattended-upgrades) — the application updates itself (part 04), but the kernel and OpenSSL do not.
• Pin the box’s own hostnames to loopback in /etc/hosts, so on-box clients (bot sidecars, calls to the push relay) reach the local router instead of the public internet — and disable any cloud-image behaviour that regenerates that file on boot. The subtle failure this prevents is described in part 03.
• The unit files already sandbox every service (part 03) — nothing to do, but worth knowing when something cannot write where you expected: ProtectSystem=strict means the whole filesystem is read-only to services except /data and /run/lchat.

What you can observe, and what you cannot yet

A status publisher on one box regenerates a set of public dashboards every five minutes, including a fleet page whose data file (fleet.json) reports: total, systemd-managed, and hibernating instance counts; how many hibernators are awake versus asleep fleet-wide at this moment; the software version spread; up/down booleans for the sibling services; backup-replication health with its lag in seconds; and a coarse capacity band. The capacity band — plenty, getting-full, or at-capacity — is the worst of several internal signals (instance count against the cap, registered rows against the row ceiling, disk against the floor, memory), and deliberately only the band word is published: never counts of slots, gigabytes, percentages, hostnames, or addresses. The same discipline runs through everything public: aggregate numbers and band words, nothing that identifies a machine or a member.

In a mesh, the dashboard aggregates across every live box, not just the one running the publisher. Each mesh member — including that one — publishes its own capacity snapshot as a sibling object next to its membership heartbeat, on a slower (one-minute) cadence; the publisher reads every live peer’s snapshot back over the same S3 substrate the mesh already uses, no new box-to-box port involved. Each box’s own reported awake/asleep count is filtered to the hosts it currently owns by the mesh’s placement hash — so a host that has been re-homed to a peer is never reported as “asleep” by more than one box at once, and the fleet page’s hosts array lists every box exactly once, each with its own whimsical, hash-derived label (never a hostname or address) and, optionally, an operator-set coarse region (LCHAT_BOX_REGION, “EU · Finland”-shaped — continent and country only).

Common problems and their diagnoses