# 🤖 Server releases: channels, publishing, and the signed feed

How server builds get released, how the three channels work, and the trust
model behind instance auto-updating. The operator entry point is
`./deploy/ops publish|promote|ship`; the machinery those verbs drive is
`deploy/release-server.sh`, and the format/signing code is `release.go`
(shared by the publisher and the box updater so they can't drift). The box-side updater
that consumes this feed is described in `docs/instance-router.md`
(§ Auto-updating).

## The shape

The source repo is private, so a box can't read its GitHub releases without a
credential we don't want to hand out. Everything therefore publishes to public
repos, split by audience (2026-07-22 — before the split, both lived on the
releases repo, and the per-build wall made its page unreadable):

- the **releases repo** (`elevenmessenger/releases`) is the HUMAN surface: the
  Mac app's releases + Sparkle appcast (see `ios/release-mac.sh`), the rolling
  `server` feed release, enterprise. GitHub's "Latest" badge belongs to the
  newest Mac release — Sparkle's feed URL rides `latest/download/appcast.xml`,
  and `release-server.sh` warns loudly when that invariant breaks.
- the **builds repo** (`elevenmessenger/builds`) takes the per-commit
  `server-<ver>` build artifacts — machine consumers only. Builds published
  before the split stay on the releases repo; promote resolves per version,
  so old pins and rollbacks keep working.

- **One release per build**: tag `server-<ver>` (ver = the git short sha that
  `deploy/build-linux.sh` stamps into the binary as `main.buildVersion`),
  carrying **both linux architectures** — `lchat-linux-amd64` and
  `lchat-linux-arm64` — plus `lchat-darwin-{arm64,amd64}` (the website's
  `lchat try` download for Macs; never referenced by the manifest) and the
  **sibling-service binaries** `lchat-{connect,push,home}-linux-{amd64,arm64}`,
  each with its `.sha256`.

  The service binaries are how connect/push/home deploy at all now that the
  box has no root SSH: their sha256s ride *inside* the signed manifest entry
  (`services`), and the box's root update pass (`updaterservices.go`)
  sha-compares each loose `/opt/lchat/<name>` against the channel head and
  swaps + bounces any that drifted — same signature, same bake window, same
  rollback-and-quarantine shape as the control binary's own self-update. A box
  without a service's file is skipped, so nothing installs uninvited. (A
  promote whose entry had to be rebuilt from release assets carries no
  `services` block, and the converge simply skips until the next publish.)
- **One rolling release**: tag `server`, whose assets live at fixed URLs
  (re-uploaded with `--clobber`, so the URLs never change):
  - `manifest.json` — the **feed**, updated on every publish/promote:

    ```
    https://github.com/elevenmessenger/releases/releases/download/server/manifest.json
    ```
  - `lchat-linux-{amd64,arm64}` and `lchat-darwin-{arm64,amd64}`
    (+ `.sha256`) — copies of the current **stable** binaries, refreshed by
    `promote … stable`. These JSON-free URLs are what `deploy/install.sh`
    (linux) and the website's try snippet (any OS) download.
  - `install.sh` — the self-hoster installer, refreshed on every publish.
- **The releases repo README** is the public self-hosting guide — synced from
  `docs/self-hosting.md` on every publish (this source repo is private, so the
  downloads page doubles as the documentation).

> Operator gotcha (learned live): GitHub's asset CDN can serve a
> freshly-clobbered asset **stale for a few minutes**, and brand-new assets can
> briefly 404 — different clients may even hit different cache nodes. An update
> pass right after a publish may quietly no-op; the next hourly pass absorbs
> it. Don't chase ghosts during that window.

## Channels

| channel | meaning | moves when |
|---------|---------|------------|
| `edge`   | bleeding edge — tracks `main` | every `publish` |
| `beta`   | near-stable, for people who want to test | `promote <ver> beta` |
| `stable` | well tested, should work in all cases — the default channel | `promote <ver> stable` |

A channel is just a pointer in the manifest. Promotion re-signs the manifest;
no rebuild, no new artifacts. A channel may lag (or be unset early on) — that's
normal. Pointing a channel **back** at an older version is allowed and is how a
bad release gets pulled; updaters follow the channel wherever it points.

## The manifest

A signed wrapper — the Ed25519 signature covers the exact base64 payload bytes
(verify first, decode after), so there is no JSON canonicalisation to get wrong:

```json
{ "payload": "<base64 of the payload JSON>", "sig": "<base64 Ed25519 signature>" }
```

Payload:

```json
{
  "schema": 1,
  "generated_at": "2026-07-08T12:00:00Z",
  "channels": {
    "edge":   { "version": "a1b2c3d",
                "url": "https://github.com/elevenmessenger/releases/releases/download/server-a1b2c3d/lchat-linux-amd64",
                "sha256": "…64 hex…", "size": 123456,
                "arm64_url": "…/server-a1b2c3d/lchat-linux-arm64",
                "arm64_sha256": "…64 hex…", "arm64_size": 123456,
                "notes": "one-liner", "published_at": "…" },
    "beta":   { "…": "…" },
    "stable": { "…": "…" }
  }
}
```

The flat `url`/`sha256` fields are the **amd64** binary — kept flat so every
updater ever shipped can read them; `arm64_*` are additive and optional (an
arm64 box facing a release without them skips it rather than installing the
wrong architecture — `releaseEntry.forArch` in `release.go`).

## Trust model

The updater downloads and *executes* native code, so the chain matters:

- The **public key** is baked into the binary (`releasePubKeyB64` in
  `release.go`). The **private key** (an Ed25519 seed) exists only in the
  password manager — Bitwarden (legacy copy in 1Password), item *"Eleven
  Messenger server release signing key"* — and in the signing process's
  memory while signing (`LCHAT_RELEASE_KEY`, the seed value in the
  environment; it is never written to disk). The older file mode
  (`LCHAT_RELEASE_KEY_FILE`) remains as a fallback.
- The signed payload carries each binary's **sha256**, so the binaries
  themselves need no separate signature: a downloaded binary is hashed and
  checked against the signed manifest before it is ever staged. Compromising
  GitHub (or the download path) without the key yields nothing installable.
- **`generated_at` is stamped at signing time** and consumers persist the last
  value they accepted, refusing anything older — so replaying a stale (but
  validly signed) manifest can't downgrade a fleet. An intentional pull-back is
  a *re-signed* manifest with a newer timestamp, which still applies.

### The bake window (supply-chain hold)

Everything above assumes the signing key is honest. The bake window is the
defence for the day it isn't — a stolen seed, or a compromised operator
machine signing a malicious build. beta and stable subscribers don't install
a release the moment it appears; the box waits until the release has **sat at
the channel head for a while**:

| channel | hold |
|---------|------|
| `edge`   | none — it *is* the operator's own deploy path |
| `beta`   | 24 h |
| `stable` | 72 h |

Real supply-chain incidents are bimodal: noisy ones (malicious npm/PyPI
releases, stolen-token publishes) are overwhelmingly detected and yanked
within hours to ~3 days; stealthy targeted ones take weeks and no tolerable
hold defends against them (that's what key hygiene is for). 24 h absorbs the
caught-within-a-day mass; 72 h additionally **spans a weekend** — a
Friday-night malicious publish is seen by a human on Monday with margin to
pull the channel back, and a pull-back to a previously-seen version applies
*immediately* (evacuation never bakes). This is also why stable-channel lag
saved every distro from xz-utils.

The mechanics (`bake.go`):

- **The clock is the box's own first sighting** — `lchat update run` records
  "first saw `<ver>` at the head of `<chan>`" in
  `/opt/lchat/channel-first-seen.json` and installs only once that record is
  old enough. Never a manifest timestamp: an attacker holding the key
  controls every timestamp in the feed and would simply backdate one, but
  cannot backdate what the box already saw. A missing/corrupt clock file
  fails **closed** (everything bakes from now).
- **Jitter**: each box waits `hold × (1 + j)`, `j ∈ [0, 0.25)` minted once
  per box — a fleet of self-hosted boxes doesn't install in the same hour,
  so the earliest are canaries for the rest.
- **There is deliberately NO publisher-side bypass** ("critical fix, skip the
  hold") — that flag is the first thing an attacker would set. The bypass is
  local and explicit, requiring root on the box:
  `lchat update run --ignore-bake`, or a plain
  `lchat deploy <host> --version <ver>`. A held release shows in
  `lchat update status` as `(held until <time>)` and in the admin consoles as
  "auto-installs after a N-hour safety hold".
- **Detection inside the window** is what makes the hold worth anything: each
  instance's own 6-hourly feed check pushes the space's admins when its
  channel head *changes* (`releasealert.go`) — the alert fires on observation
  of the feed, out-of-band from the signing key, so a rogue publish can't be
  quiet. edge sightings stay silent (its riders are the operators doing the
  publishing; the post-install changelog banner covers arrival).
  Belt-and-braces for operators: also GitHub-watch the releases repo
  (Watch → Custom → Releases), so a publish you didn't make lands in your
  inbox through a channel the release key never touches.

### Critical security releases — the emergency runbook

A critical fix wants to reach the fleet in minutes; the bake window makes
beta/stable wait days. Both are security properties, and the resolution is
**not** a faster feed: the clock stays untouched, and expediting is done from
the box side. The rule that keeps every current and future override honest:

> **An override is safe exactly when it cannot be exercised with the secret
> the delay defends against.** The bake window defends against a compromised
> *release signing key*; every expedite path below rides *box root* or an
> *instance admin's deliberate act* — independent credentials. A same-key
> override (a `critical: true` field in the manifest) is the one thing never
> to build: it is the first flag an attacker with the key would set, and it
> converts the whole defence into decoration.

The procedure, for a fix that can't wait out the hold:

1. **Land the fix on `main`** (normal PR flow — an emergency is when review
   discipline pays, not when it lapses).
2. **Publish and converge the edge fleet** — edge has no hold, so this is
   already the whole story for every edge-riding space:

   ```sh
   ./deploy/release-server.sh publish "security: <one line>" && \
   ./deploy/release-server.sh ship
   ```

3. **Promote to beta and stable.** The channel heads move immediately; only
   *installation* is held. This also fires the release-alert push to every
   subscribed space's admins — for self-hosted boxes, that push (and the
   admin console's "available" line) *is* the expedite channel: their
   operator applies it with their own root, on their own judgement.
4. **Expedite boxes you operate** (requires root on each box — that's the
   point):

   ```sh
   lchat update run --ignore-bake        # whole box: install held heads now
   lchat deploy <host> --version <ver>   # or surgically, one instance
   ```

   `--ignore-bake` skips only the bake clock; pins, major pins, and
   updates-off are still honoured.
5. **Say what happened.** The release notes and the changelog entry should
   name the fix plainly (members see the changelog banner after their space
   updates). If the vuln is sensitive, "security fix — update promptly" is
   enough; silent security releases train nobody to act on the next alert.

What this costs, honestly: a self-hosted box whose operator ignores the push
installs the fix only when the hold elapses (up to ~3.75 days on stable with
maximum jitter). That is the deliberate trade — the window pays for itself
the day a *malicious* release ships, and a malicious binary on an E2EE
messenger (key exfiltration) is unrecoverable in a way a bounded patch lag
is not. Expect social engineering in both directions: an attacker will label
malware "critical security fix" precisely to make humans expedite it, which
is why the expedite paths end at a human with an independent credential
rather than at a feed flag.

Two designed-but-unbuilt rungs, for when self-hoster numbers make
"wait for each operator" unacceptable (build in this order):

- **Consent-based expedite**: make the release alert actionable — an
  "Install now" button in the admin console writes a one-shot, per-version
  consent the box updater honours. The override power distributes to each
  space's human; an attacker with the signing key would have to socially
  engineer every admin individually. No new keys, no publisher power.
- **Second-key co-signature**: a separate *emergency key*, stored apart from
  the release key (offline / second operator — the value is that no single
  machine compromise yields both), whose co-signature shortens the hold.
  Three rules keep it from becoming the attack: the co-sig **binds the exact
  sha and expires within hours** (no hoarding, no replay onto a later build);
  the reduced hold is a **floor, never zero** (2–6h + jitter — the earliest
  boxes stay canaries even in an emergency); and expedited releases alert
  **louder, not quieter**, so abuse of the mechanism is the most visible
  event in the system.

### Major version pinning

Releases carry two names: the **sha** (`version`, the artifact id everything
deploys by) and a **human semver** (`semver`, stamped at publish time from
`deploy/SERVER_SEMVER` — bump that file as part of cutting a release; plain
`MAJOR.MINOR.PATCH`, no suffixes). The semver exists for one consumer: an
instance that wants minors and patches but not to be auto-updated across a
breaking major.

- **Setting the pin**: `update_major` is an instance setting like the version
  pin — `lchat update major <host> <n>` / `lchat update major <host> clear`
  on the box, or `update_major` through the admin settings API.
- **Semantics are halt-and-notify, not LTS.** A channel head whose major is
  ≤ the pin installs as normal; one that crosses it is *held* — the pass says
  so, `update status` shows `(held by major pin N)`, and the admin console
  explains it on the available line. Nothing auto-installs until the admin
  raises or clears the pin. There are no per-major channel heads: we don't
  maintain old-major branches, so a pinned instance's updates simply stop at
  the boundary (that's the honest promise; per-major heads can be added if an
  LTS branch ever actually exists).
- **Fail closed**: a release with no semver at all also holds a major-pinned
  instance — the pin is a promise not to cross a boundary, so an unlabelled
  release must never sneak past it. (`release-server.sh` warns when it
  publishes without `deploy/SERVER_SEMVER`, and when a `promote` has to
  rebuild an entry from release assets, which loses the semver.)
- **Pull-backs always pass**: an older major is never held.

## The S3 mirror

GitHub's asset CDN is the feed's single point of failure — and it has served
freshly-clobbered manifests stale (and brand-new binaries 404) for minutes
after a publish (the gotcha above). The feed therefore also mirrors to S3
(bucket: `infra/prod/releases.tf` — public-read GETs so boxes fetch
anonymously, IAM-only writes, versioned for manifest rollback):

- **Publish side:** set `LCHAT_RELEASE_MIRROR_BUCKET` and
  `deploy/release-server.sh` copies everything it uploads to GitHub into
  `s3://<bucket>/releases/…` too — `manifest.json` at the top (refreshed on
  publish *and* promote), binaries under `server-<ver>/`. Unset, the script
  is unchanged. Mirror writes are best-effort: a failure warns rather than
  aborting a publish whose GitHub half already landed — a lagging mirror is
  harmless because updaters use whichever verified feed is fresher.
- **Fetch side:** set `LCHAT_RELEASE_MIRROR_URL` (a plain HTTPS base, e.g.
  `https://<bucket>.s3.<region>.amazonaws.com/releases`) and the updater
  falls back to the mirror when the primary fetch fails — or when it verifies
  but is *stale*: generated_at older than the newest feed the box ever
  accepted (the anti-replay stamp), which is exactly what the post-publish
  CDN window looks like; a fresher mirror then wins. Binary downloads fall
  back the same way. Every fallback logs loudly; a feed merely *equal* to the
  stamp is indistinguishable from "no new release", so steady-state passes
  never touch the mirror.

The mirror adds availability, never trust: both paths run the identical
Ed25519 manifest verification and sha256 binary check, so a compromised
mirror can only cause a refusal, not an install.

## Publishing

```sh
# cut a release from clean, landed main and point edge at it — the signing
# key comes from Bitwarden automatically (deploy/vault-items.sh), unlocking
# interactively if the vault is locked; no key export, nothing on disk
./deploy/ops publish "what changed, one line"

# make every fleet box catch up NOW instead of on the hourly timer
./deploy/ops ship

# promote when it's proven
./deploy/ops promote a1b2c3d beta
./deploy/ops promote a1b2c3d stable
```

`ship` loops the box's `lchat update run` until every un-pinned instance is
current on its channel — absorbing GitHub's stale-CDN window on a fresh
manifest — and each pass restarts updated instances' bot sidecars in
lock-step. Pinned instances (canaries) are reported and left alone. With the
whole fleet riding `edge`, `publish … && ship` is a complete manual deploy.

**Hibernating hosts** (docs/planet-scale.md) have no unit to restart and are
never woken just to update. The pass instead reads their prefs from the
parked DB (read-only, fail-closed — a host whose prefs can't be read is
skipped, never moved) and **repoints the live symlink** — the next wake
execs the new binary. They're processed *after* every unit-managed host, so
a boot-broken build that fails a health check on an awake instance poisons
the version before any sleeper is moved. Two stated trades vs the unit
path: there is no per-host health check (a bad build surfaces at wake —
recover with `lchat rollback <host>`, which repoints back without waking;
`lchat deploy <host> --version <ver>` likewise repoints), and a
currently-awake child keeps its old binary until it next sleeps. A
sidecar-bearing space that never sleeps therefore still needs the flip-off
flow (`lchat hibernate <host> --off`, update, `--on`). `lchat update status`
reads a sleeping host's prefs the same parked-DB way, so its CHANNEL/AUTO
columns (including a pin) show real values instead of `?` — an invisible pin
is how a fleet operation stomps a canary.

Same rule as deploys and TestFlight: **publish from landed `main`**, never a
feature branch (the script refuses a dirty tree). Publishing is manual for now;
a CI job cutting `edge` on every push to `main` is the natural next step.

`lchat release` subcommands (`keygen` / `sign` / `verify` / `payload`) are the
plumbing the script drives — `verify` checks a manifest exactly the way a box
will, and `keygen` prints a new public key for baking in (only needed if the
key is ever rotated; rotation requires shipping the new public key via the
*old* key's channel first).