Installing and updating the software

You are here: the auto-updater node — though this page equally concerns the version store under /opt/lchat/ and the per-instance links that decide which build every instance runs.

The mental model: a store of builds and a pointer per instance

Two filesystem facts, introduced in part 03, carry this entire page:

• The version store. Every build of the software that has ever been staged on the box lives at /opt/lchat/versions/<ver>/lchat, where <ver> is the build’s identifier — a short git commit hash such as 1b4bd808, stamped into the binary when it was compiled. Builds sit side by side and never overwrite each other.
• The per-instance pointer. Each instance’s systemd unit runs /opt/lchat/live/<host>, which is a symbolic link into the version store. Which build an instance runs is exactly where its link points, nothing more.

So “deploy version X to instance Y” decomposes into: repoint live/Y at versions/X/lchat, restart Y’s process, and verify Y now reports build X. That is genuinely all a deploy is, and every command below is a wrapper around those three steps with safety checks. It also makes the two most valuable properties fall out for free:

• Canarying: point one instance at a new build while the rest stay put, watch it, then promote the rest.
• Rollback: the previous build is still sitting in the store, and the registry remembers which one it was — stepping back is repointing the link again.

Setting up a brand-new box: lchat bootstrap

Because the software is one static binary, provisioning a fresh Linux server needs no compiler, no language runtime, and no packages. The one-time sequence is:

1. Copy the binaries to the box (or use the installer).

   Either copy a built lchat binary to /opt/lchat/lchat yourself, or run deploy/install.sh, which downloads the latest stable release, verifies its checksum, and performs the next step for you.

2. Run lchat bootstrap, as root, once.

   This writes all the systemd unit files described in part 03, creates the lchat system user, creates /data, /run/lchat, and the certificate cache directory, mints the root-only control token at /etc/lchat/control.token, then enables and starts the router and the hourly update timer. It is idempotent: re-running it refreshes the unit files without touching data.

3. Open ports 80 and 443 (and 22 for yourself) in the firewall.

   Nothing else needs a public port — see part 08.

A manual deploy, end to end: build, stage, canary, promote

This is the full ceremony for shipping a build you compiled yourself. It is the procedure behind every “deploy the new version” task, and each step exists for a reason given alongside it. running today The repository’s operator surface wraps the whole ceremony: ./deploy/ops canary create|deploy|destroy <slug> runs build → stage → pinned-deploy for a per-session test instance (and the release path is ./deploy/ops publish / promote / ship) — the steps below are what those verbs do, worth understanding even when a verb runs them for you. Dev-mesh canaries are created pointing at the dev home server (--home-origin https://home.dev.elevenmessenger.com, override via LCHAT_CANARY_HOME_ORIGIN); the deliberate prod exception keeps the compiled production default.

1. Build the Linux binary on your own machine, stamped with its version.

   ./deploy/build-linux.sh
   # produces lchat-linux-amd64, with the current git commit hash compiled in

   The build script passes the git commit hash into the binary itself, and the running server reports it at GET /api/version as the build field. This self-identification is what makes health-checked deploys possible: the machinery can ask a process which build it actually is, rather than trusting that a restart took.

2. Copy the binary to the box and stage it into the version store.

   scp lchat-linux-amd64 box-1:/opt/lchat/incoming
   ssh box-1 sudo /opt/lchat/lchat stage --file /opt/lchat/incoming
   staged 4f0a2d19

   stage asks the binary for its own build identifier and copies it to /opt/lchat/versions/4f0a2d19/lchat. Nothing is running it yet; staging is inert and always safe.

3. Deploy it to one canary instance first.

   lchat deploy test.example.com --version 4f0a2d19

   This repoints live/test.example.com, restarts the unit, and then polls /api/version until the instance reports build 4f0a2d19 — not merely “answers 200”. If the report never matches (the new build crashes on start, say), the command automatically repoints the link back to the previous version, restarts again, and exits with an error. A failed deploy therefore leaves the instance running the old build, not down.

4. Verify the canary by using it, then promote the whole box.

   lchat deploy --all --version 4f0a2d19

   --all rolls instance by instance, health-checking each, and stops at the first failure (instances already promoted stay promoted). It also skips pinned instances — a pin marks somebody’s deliberate canary, and a fleet-wide promote must not stomp it; --include-pinned overrides that deliberately, and naming a host explicitly always deploys it. Two more useful forms: lchat deploy <host> with no --version simply restarts the instance on its current build, and --pin combined with --version deploys and pins in one step (the standard way to start a canary that the auto-updater must leave alone).

5. If anything looks wrong later, step back.

   lchat rollback test.example.com
   rolled back test.example.com: 4f0a2d19 -> 9c21e77e

   rollback repoints the link at the registry’s recorded previous version, restarts, health-checks, and swaps the “current” and “previous” records — so a second rollback returns you forward again. rollback --all steps every instance back, skipping any that have no recorded previous version.

Three background details that protect you during all of this: a deploy lock (/opt/lchat/.deploy.lock) makes concurrent deploys, rollbacks, and updater passes take strict turns — the loser fails immediately with “another deploy/rollback/update is running” rather than interleaving. The router’s dial retry (part 02) bridges each instance’s restart gap so members see at most a moment of added latency. And each deploy restarts the instance’s bot sidecar right after the instance passes its health check, because the sidecar execs the same versioned link and must track it.

The automatic path: release channels and the hourly updater

Most of the time nobody drives deploys by hand: boxes keep themselves current. Three pieces cooperate:

The signed release feed

Published releases live in a public feed: a manifest file listing, for each channel, the newest version, its download URL, and its checksum. There are exactly three channels — stable (the default), beta, and edge (tracks the main branch). The manifest is signed with an Ed25519 key whose public half is compiled into every binary, and every box remembers the newest manifest timestamp it has ever accepted (/opt/lchat/manifest-stamp) and refuses anything older. Together those two checks mean a compromised or stale download server cannot push the fleet backwards or substitute binaries: only the holder of the signing key can move a channel. There is also an optional mirror of the feed in object storage (LCHAT_RELEASE_MIRROR_URL) that boxes fall back to when the primary fails or serves something stale — the mirror adds availability, never trust, because the signature checks are identical on both paths.

What one updater pass does

The timer lchat-update.timer fires lchat update run hourly (with a random spread of up to ten minutes, so a fleet of boxes does not hit the feed simultaneously). One pass, in order: fetch and verify the manifest; ask each instance for its own admin’s preferences (channel choice, auto-update on or off, download-only) over its socket; then, per instance that wants updating: download, checksum, stage, and apply through exactly the same deploy-with-health-check-and-auto-rollback dance as a manual deploy. A build that fails its health check on one instance is poisoned for the rest of the pass — the updater will not try it on instances that still pass. One unreachable instance logs and is skipped; it never strands the rest of the fleet. After the pass, builds in the version store that nothing references any more (no instance’s current or previous version, no channel head, no pin) are garbage-collected, so hourly updates on the edge channel stay disk-bounded.

Pinning: how a canary survives the hourly pass

A pin freezes one instance on one exact version. The updater skips a pinned instance entirely, and its pinned build survives garbage collection. Pins are how a deliberately deployed canary is protected from being “corrected” back to the channel head an hour later. The pin is stored as instance state (the instance’s admin can also see and clear it in the admin panel), and the fleet commands read and write it:

lchat deploy test.example.com --version 4f0a2d19 --pin   # deploy a canary and freeze it, one command
lchat update pin test.example.com                        # freeze on whatever it runs right now
lchat update unpin test.example.com                      # resume following its channel (next hourly pass)
lchat update status                                      # the fleet update table:
HOST                 CHANNEL  AUTO     CURRENT   AVAILABLE
chat-a.example.com   stable   dl+upd   9c21e77e  9c21e77e
test.example.com     stable   pinned   4f0a2d19  9c21e77e

lchat update run --dry-run prints what a pass would do without doing it — useful before enabling anything new. Hibernating instances get updates too, by a quieter route: the updater reads their preferences straight from their on-disk database (read-only) and simply repoints their live link, so they come up on the new build at their next wake, with no health check (a bad build on a hibernator is recovered with rollback).

Deploying the sibling services and the control binary

The home server and push relay are not versioned instances — they show - in the version column. running today On a hardened box (root SSH bootstrap-only) they deploy through the same version store as everything else, via the allowlisted install verb:

scp ./home box-1:/opt/lchat/incoming/home-<sha>
ssh box-1 sudo -n /opt/lchat/lchat stage --file /opt/lchat/incoming/home-<sha>   # prints <ver>
ssh box-1 sudo -n /opt/lchat/lchat install home --version <ver>
ssh box-1 sudo -n /usr/local/sbin/lchat-svc restart lchat-home

install knows the sibling set (home, push, connect), backs the outgoing binary up beside itself as <name>.prev-<date> (the one-copy rollback), and deliberately does not restart the unit — the restart is the operator's visible act, so an install can be staged ahead of a maintenance window. The version store doesn't care what a binary is: stage --file records any build under its own version id, so sibling builds sit beside messenger builds. Extending the sibling set is a reviewed PR, never a box edit.

running today Routine sibling deploys don't need even that: every publish also uploads lchat-{connect,push,home}-linux-<arch> with their sha256s pinned inside the signed manifest entry, and the hourly pass converges the sibling binaries automatically right after the control binary's own self-update — sha-comparing each loose /opt/lchat/<name> against the channel head, swapping any that drifted (a .prev kept beside it), bouncing the unit only if it was running, and rolling back + quarantining a build whose unit doesn't stay up. Same signature, same bake window, same channel (LCHAT_CONTROL_CHANNEL). A box that doesn't run a service has no file at its path and is skipped — nothing installs uninvited. The manual install verb stays as the ad-hoc/canary path.

running today The control/router binary /opt/lchat/lchat self-updates on the same hourly pass: after the instances, the updater installs its channel's head over its own path — same signed feed, same sha check, same bake window — keeping the outgoing build beside it as lchat.prev-<build> (the one-rename rollback). The channel is LCHAT_CONTROL_CHANNEL in /data/update.env (stable by default — the trust plane takes new builds last; off disables self-update). After a swap the pass restarts lchat-control and lchat-cloud (backend sockets, invisible to members) but never the router — its front door must not blip on a timer's schedule, so lchat-router keeps running the old build until an operator restarts it, and the pass says so in its log. This is also how root-less operator boxes stay current: the manual root cp this replaced stopped being possible when root SSH went bootstrap-only.

Verifying what is deployed where

# Which build does each instance run? (VERSION column)
lchat instances

# Which build is a specific instance really running, asked of the process itself?
curl -s https://chat-a.example.com/api/version
{"version":"a41c9f2","build":"9c21e77e"}

# Which build would the unit start next time? (where the link points)
readlink /opt/lchat/live/chat-a.example.com

# What builds are on the shelf?
ls /opt/lchat/versions/

When the registry, the link, and the process disagree, believe the process (/api/version) about the present and the link about the future — and expect the next restart to reconcile them.