# 🤖 Awake-instance convergence: diagnosis + fix **Status: IMPLEMENTED (sweep-driven).** This documents a production gap — awake hibernate-managed instances kept serving their old binary after a release ships — and the fix that closed it. The fix landed as a fourth staleness case in the activator's existing idle sweep (`activator.go`, `sweepOnce`); it needed **no new router control channel**, no updater changes, and no change to the deploy path. The sections below keep the original diagnosis, then describe what was built and why it differs from the first proposal. ## The symptom After `release-server.sh ship` (or the hourly `lchat update run`) repoints an instance's `live/` symlink to a new binary: - `lchat instances` / `lchat update status` show the **new** version. - But `/api/version` over the instance's socket returns the **old** build. Hibernating instances converge on their next wake; **awake** ones (anything with an open WebSocket or recent traffic — notably every bot-sidecar-bearing space, which never sleeps) serve the stale binary indefinitely. The reporting surfaces all read the symlink target, so the fleet *looks* converged while it isn't. (Confirmed live 2026-07-13: after promoting a release, `family`, `test`, and `team` each served a different older build than the symlink named.) ## Why — the exact mechanism An awake hibernate-flagged instance is a **child process of the router**, not a `systemd` unit (`activator.go`: `activatorSvc.entries[host].proc`, an `*exec.Cmd` via `childProc`). It's spawned once by `s3Backend.start`, which execs `liveLink(host)` = `/opt/lchat/live/` → `versions//lchat`. **A running child never re-reads the symlink** — that's the root of the gap. The update path (`updater.go updatePass`) has two branches: - **Unit-managed hosts** repoint the symlink *and* `systemctl restart` the unit (`deployHostVersion` in `control.go`) — so they converge immediately. - **Hibernating hosts** (`updater.go` phase 2) do **only** `pointLive` — a pure symlink repoint, no signal to the running process. The code comment is explicit: *"a currently-AWAKE child keeps its old binary until it next sleeps (a running process doesn't follow the symlink)."* `update status` (`runUpdateStatus`) and `lchat instances` both build their "current" column from `liveVersion(host)` (`os.Readlink`), never from the child's `/api/version` — hence the false "converged" reading. ### What already exists (and what doesn't) - **#778** added the whole hibernating-repoint phase, `hibernatingPrefs` (parked-DB read), WAL-ownership repair, and the narrow `setVersions`. - **#774** added parked-DB pref reads to `update status`. - The "compare symlinks, never restart awake children" behaviour is `updater.go`'s symlink comparison + `deploy`'s hibernating branch — not a separate function. **What does NOT exist anywhere:** a primitive that cycles an *awake* activator child onto the new symlink. `lchat deploy --version` also only repoints for hibernating hosts (`control.go`, the `if rt.Hibernate` branch is a pure `pointLive`). The only graceful cycle today is the heavyweight manual `hibernateOff` → update → `hibernateOn` flip documented in `docs/server-releases.md`. ## Why awake children aren't restarted (the design tension) This is deliberate, for two real reasons: 1. **Dropped connections.** An instance is kept awake *by* in-flight work, including open WebSockets (idle = zero in-flight AND elapsed idle window, `activator.go`). Cycling an awake child cuts live WS/requests. 2. **The single-writer S3 lease.** Each wake acquires a per-host Litestream lease; a naive external `pkill` races the lease and can 502 within the router's dial-retry window (the sweep only reaps every `Idle/4`, ≤30s). So "never converges" is the current cost of "never drop a live connection." ## The safe primitive that already exists The activator's own stop **is** graceful: `childProc.stop` sends `SIGTERM` (the instance's 10s drain), `SIGKILL` only on overstay; `beginStopLocked` → `stop` takes the final S3 checkpoint, releases the lease, stops the sidecar first, and deletes the entry — leaving the host **asleep**. The next request or due-wake re-execs `liveLink(host)`, which now resolves to the **new** binary. So a respawn-onto-new-symlink path exists; it's just **lazy (next request), not triggered by a ship.** ## The fix as built (smallest safe change) The convergence check is a **fourth staleness case in the sweep** the activator already runs (`sweepOnce`, every `Idle/4`, capped 30s). The sweep already reaps awake children for three reasons — lost lease, dead child, idle — each via the graceful `beginStopLocked`. Convergence is the fourth: 1. **Record the spawned build.** The backend's `start` reads `liveVersion(host)` **immediately before `cmd.Start()`** and returns it, so `spawnVersion` names the build the kernel actually resolved the symlink to — not a value re-read later in `wake()`, which a ship landing during the up-to-30s health wait could make disagree with the running binary. A read error leaves it empty, which disables the check for that entry (fail-safe: never cycle on a version we couldn't read). 2. **Notice the drift, then wait out a grace.** Each sweep compares the live symlink against `spawnVersion` (the pure `convergeDecision` helper holds this state machine, so it's unit-tested directly). On first divergence it stamps `staleSince`; only once the drift has persisted `convergeGrace` (90s) does it cycle. The grace means a symlink that moved then rolled straight back (a bad promote reverted) never drops a connection, and a briefly-busy space gets a chance to idle-hibernate — which converges for free — first. A transient symlink-read error is treated as "unknown" — it neither cycles nor resets the accrued grace. 3. **Cycle gracefully, one per tick.** Past the grace it calls the same `beginStopLocked` the other three cases use: sidecar stopped first, SIGTERM drain, final S3 checkpoint, lease released, entry deleted → **asleep**. A subsequent **member request** — or the **due-time index** for a space with a scheduled charter — then wakes it, re-execing `liveLink(host)` = the new binary. (The sidecar itself is SIGKILLed during the stop and can't self-wake; a bot space with no members present and no scheduled work simply stays asleep until someone arrives, which is correct hibernation.) At most `maxConvergeCyclesPerSweep` (1) per tick, so a fleet-wide ship staggers across sweeps instead of blipping every space at once. Unlike idle-hibernation (which only fires at `inflight == 0`), this cycle can fire with a request in flight — that's deliberate, since the always-awake spaces it targets never reach `inflight == 0`. The graceful stop drains in-flight HTTP just as `systemctl restart` does, so the only thing cut is a request still running past the instance's ~10s drain (a large upload) — identical to a unit restart landing mid-request, not a new failure mode. No repoint-site call, no control channel: the updater/deploy still just `pointLive`, and the router's own sweep picks up the drift within ≤30s. This **reuses** the graceful stop and the lazy wake-re-execs-the-symlink path — it extends the existing convergence machinery rather than adding a new one. ### Exclusions (all automatic — no special-casing) - **Pinned canaries.** The updater never repoints a pinned host's symlink, so `spawnVersion` keeps matching and the check never fires. Honouring the pin needs no code here — it falls out of "cycle only when the symlink moved." - **Unit-managed (non-hibernating) hosts.** Not activator children — they have no `entries[host]`, so the sweep never sees them; they converge via `systemctl restart` as before. - **Bot-sidecar spaces are the PRIMARY target.** They never idle, so the idle case never converges them; this cycle does, and the sidecar is stopped-first and re-spawned on wake, so it's sidecar-safe. This automates exactly what the manual `hibernate --off`/`--on` flip did by hand. ## Why this shape, not the first proposal The original proposal (kept in git history) added a per-host `cycle` verb to a new router control socket and gated on `inflight == 0`. Both were dropped: - **No control channel.** A push from the repoint site would have added the first per-host *command* into the live router — new attack surface on the process fronting every space. The sweep already wakes hosts on a timer, so a ≤30s lazy pull needs none of that. The convergence latency (seconds to a couple of minutes) is irrelevant given the 72h supply-chain bake window. - **Grace window, not `inflight == 0`.** An `inflight == 0` gate would leave the *primary* target — always-awake bot-sidecar spaces, whose persistent WebSocket keeps `inflight > 0` forever — stale indefinitely, defeating the point. Cycling them requires a brief connection drop, which the graceful SIGTERM drain + the router's `actStopping` connection-shed makes clean: clients reconnect onto the new binary, exactly as they do across a normal unit `systemctl restart`. The grace window (plus one-per-tick) is what keeps it gentle — a space that goes quiet converges via idle-hibernate first, and only a persistently-awake space is force-cycled, once, per ship. This is a *latency/observability* fix, not a correctness emergency: the bake window holds releases 72h and the lazy next-wake path already converged sleeping spaces. The gap was the always-awake spaces and the dishonest `status` column — both now closed (see below). ## Companion: honest `status` (already shipped) `lchat update status --probe` reads each instance's `/api/version` over its socket and flags an awake-stale instance (⚠) where the old symlink-only column read "current". That shipped first as a small, safe, independent step; this sweep fix removes the underlying staleness the probe surfaces. The manual `lchat hibernate --off` → update → `--on` flip still works for forcing an immediate converge, but is no longer needed for routine ships.