Hibernation: how idle spaces sleep in object storage

You are here: the activator lives inside the router process, and its durable state lives in the hibernation bucket. Nothing new runs on the box — hibernation is a behaviour of the router.

Why parking chat data in a bucket is safe at all

This whole design is only possible because of the product’s encryption rule: an instance’s database contains end-to-end ciphertext plus routing metadata the server already sees, and never a usable decryption key. Copying that file into an object-storage bucket therefore changes nothing about who can read messages — nobody new can. The operational risk of hibernation is availability and consistency, not confidentiality, and the rest of this page is about how consistency is guaranteed.

Enabling hibernation: one bucket variable, then one flag per host

Hibernation is enabled in two deliberately separate steps, so that turning the machinery on is fleet-safe and adopting it is gradual:

Step one: give the router a bucket (per box)

The activator — the component that wakes and sleeps instances — is part of the router process, and it exists only when the router’s environment names a bucket. In /data/router.env:

LCHAT_HIBERNATE_BUCKET=my-hibernation-bucket       # the master switch; unset = the activator does not exist
LCHAT_HIBERNATE_REGION=eu-west-1                   # optional; the AWS default chain applies otherwise
LCHAT_HIBERNATE_PREFIX=prod                        # optional key prefix inside the bucket
LCHAT_HIBERNATE_IDLE=15m                           # optional idle window before sleep; default 15m

The router also needs AWS credentials able to read and write that bucket, supplied the standard AWS ways (environment variables in the same file, or an instance role). Setting the bucket alone changes nothing for any instance: it only makes the activator available.

Step two: flip individual hosts (per host)

Each host opts in via the hibernate flag on its registry row, flipped by one command in each direction. This per-host granularity is what made the feature safe to prove in production: one canary space migrated first, everything else untouched.

lchat hibernate chat-b.example.com --on      # hand the host to the activator
lchat hibernate chat-b.example.com --off     # hand it back to systemd
lchat instances                              # the HIBERNATE column shows who is activator-managed

What --on actually does, in order: fixes the ownership of the host’s environment file (the activator reads it as the service user, whereas systemd read it as root — a real wake failed on exactly this before it was baked in); sets the registry flag; refuses to proceed if the box-wide backup replicator still tracks this database (two replicators on one database corrupt each other’s state — re-run deploy/fleet-backup.sh to regenerate the backup configuration without this host, then retry); stops and disables the host’s systemd unit (and its bot sidecar unit, if present); reloads the router; and finally makes one warm request so the host is immediately woken by the activator rather than left to the first member. The ordering means the host is never unserveable: requests arriving in the brief socket-dead gap ride the router’s thirteen-second dial retry.

--off is the mirror image: clear the flag, reload the router (which makes the activator checkpoint the space to the bucket, release its lock, and stop the child), wait for the child to actually release the socket, then re-enable the systemd unit and health-check it, then re-enable the sidecar unit. The wait matters — an instance refuses to start over a socket that still has a listener.

The lease: one writer, enforced by the bucket itself

The central safety problem of hibernation is: what guarantees that two processes never write the same space’s database at once? (Two boxes might both think they should wake a space; a wedged old process might still be running when a new one starts.) The answer is the lease — a small lock object in the bucket, at <prefix>/<host>/lock.json, acquired using S3’s conditional writes (an operation that atomically fails if the object already exists). Whoever holds a space’s lease is the one writer; everyone else fails loudly and immediately.

• The lease has a fifteen-minute lifetime and its holder renews it roughly every five minutes (stretched from the original 30s/10s in #836: each renewal is one S3 Tier-1 PUT, and at the old cadence the lease heartbeat alone was ~70% of the fleet’s entire S3 bill). A process that stops renewing — crashed, wedged, unplugged — loses the lease within about fifteen minutes, and a healthy process elsewhere can then claim it. That ~15 minutes is the hard-crash failover window of the design; every graceful path (sleep, deploy, shutdown) releases the lease explicitly and is unaffected.
• A failed renewal is terminal on purpose: the holder stands down (checkpoints nothing further, stops serving) rather than retrying, because a retry could collide with a new holder that legitimately took over after expiry.
• Restoring a database from the bucket happens only while holding the lease. Restoring outside it could resurrect a stale copy that shadows the true holder’s newer state.

Replication itself is performed by Litestream — a well-known tool that continuously streams SQLite changes to object storage — embedded as a library inside the router. The embedding is not an implementation detail but a hard-won correction: the Litestream command-line tool, run twice against one database, silently absorbs the second copy and orphans one side’s writes with no error at all; only the library interface exposes the lease that turns a second writer into a loud, typed failure. The build chose the loud failure.

The life of a hibernating space

Waking: what happens when a request arrives for a sleeping space

1. The router sees a request for a host flagged as hibernating with no live process, and hands it to the activator. Many simultaneous requests coalesce into one wake — the rest simply wait for it.
2. The activator claims the lease at <prefix>/<host>/lock.json. If another holder has it (for example, another box in a mesh), the request is answered 503 with a Retry-After header — deliberately: “sleeping spaces are slow to wake” must degrade to a polite retry, never a conflict.
3. It restores the database from the bucket only if the local file is missing. A database already on disk is trusted as-is — on a single box, under the lease, the local copy can only be this box’s own last checkpoint, so most wakes skip the download entirely. This is the “warm cache”, and it is why routine wakes are fast.
4. It starts the instance as a direct child process of the router — the same versioned binary its systemd unit would run (/opt/lchat/live/<host>), with the same environment file (/data/<host>.env) — and waits up to 30 seconds for the socket to answer.
5. It starts replication under the held lease — at the host’s replication class’s cadence: within about a second on the default stream class, every five minutes on batched, daily on daily — and finally spawns the bot sidecar if one is staged (below). Then the waiting requests proceed.

The entire wake has a five-minute budget; a queued request waits at most 30 seconds before giving up with a 503. A failed wake discards all bookkeeping so the next request retries from scratch — one bad wake never becomes a stuck host.

Sleeping: what happens when a space goes idle

A background sweep (every few seconds) puts an instance to sleep when it has had zero requests in flight for the whole idle window (default fifteen minutes, set by LCHAT_HIBERNATE_IDLE). Both conditions matter: an open WebSocket counts as an in-flight request for its entire life, so a space with anyone connected never sleeps, no matter how quiet the conversation. The stop sequence, in order:

1. Stop the bot sidecar first (an orphaned sidecar trying to reconnect would immediately re-wake the space).
2. Ask the instance for its next scheduled delivery (the earliest future message or reminder) and record it in the registry’s next_due column — the hook for scheduled wakes.
3. Terminate the child gracefully (its own drain is the final write), with a 20-second budget before a hard kill.
4. Take the final checkpoint to the bucket and release the lease, each under its own time budget so a wedged upload can never strand the lease until it expires on its own.
5. Forget the entry. The local database file stays on disk as the warm cache; the bucket copy is the durable truth, guarded by the (now released) lease.

Failure is just the wake path again

Every failure mode collapses into “stand down; the next request re-wakes”: a child that dies unexpectedly, a lease lost to a renewal failure, a checkpoint that errors. And an S3 outage while spaces are awake is survivable by design — the local SQLite file is authoritative while the lease is held, replication queues and drains when the bucket returns, and only cold wakes (no local file) actually require the bucket to answer.

Replication classes: the per-plan durability ladder

While a space is awake, how often its writes reach the bucket is a per-host class on the registry row (routes.replication), because the change-driven sync is the one S3 cost that scales with how busy a space is — pennies a year on a quiet space, potentially ~€145/year on one that writes every second around the clock. Three rungs:

Two invariants the class can never weaken, worth internalising before touching it: every graceful path still flushes in full — idle sleep, deploys, restarts and shutdowns all checkpoint the space to the bucket regardless of class — and therefore a sleeping space’s parked copy is always exactly current. The class bounds only what a sudden box loss can take, and only from the current awake window. (Corollary: a space with a bot sidecar never sleeps, so on the daily rung its real cadence is the daily tick — the one profile where “up to a day” is the everyday exposure, not a rare worst case.)

lchat replication chat-b.example.com            # read the current class
lchat replication chat-b.example.com daily      # set it (stream | batched | daily) + reload the router
lchat instances                                 # the REPLICATION column shows every host’s rung
lchat add-instance test-x.11ch.at --hibernate --replication daily   # born on a rung

On provisioned fleets the class is billing-driven, not hand-set: with LCHAT_CLOUD_REPLICATION_CLASSES configured on the cloud service (free=daily,family=daily,community=batched,business=stream), spaces are born on the free rung, Stripe webhook events move them on every upgrade or lapse, and the meter sweep reconciles drift — the same event-driven-plus-sweep shape as storage caps. controld records the class and reloads the router only on an actual change; nothing restarts, no sleeping space wakes, and an awake instance keeps its current cadence until it next sleeps. A misconfigured paid tier deliberately maps to stream — config typos fail toward durability, never away from what a member paid for. See docs/cloud-provisioner.md for the config and docs/tier-economics.md for the cost table behind the rungs.

Scheduled deliveries wake sleeping spaces on time

Spaces can hold scheduled events — a reminder due at 9:00, a message scheduled for tomorrow. A naive hibernation design would sleep through them. Instead, at sleep time the space reports its earliest pending delivery, the activator persists that timestamp in the registry (so it survives router restarts), and the same background sweep wakes any sleeping host whose time has arrived — no member request needed. Worst-case lateness is about one sweep interval (thirty seconds) plus a wake. Bot schedules created by members (“send me a news summary every morning”) live encrypted in the sidecar, which the server cannot read — so the sidecar itself reports its earliest scheduled run to the instance (one bare timestamp crosses; the schedule stays ciphertext), and the instance folds it into the same answer.

Bot sidecars on hibernating spaces: co-spawned, and invisible to the idle clock

A persistent sidecar service would defeat hibernation twice over — its always-open WebSocket would hold the space awake forever, and if the space did sleep, the sidecar’s reconnection attempts would wake it right back. So on a hibernating host the sidecar’s systemd unit is disabled (the hibernate --on flip does this) and the activator co-spawns the sidecar as its own second child: started right after the instance turns healthy, terminated first at sleep. The sidecar exists exactly while its space is awake. Two details make the arrangement sound:

• The sidecar marks every request it makes with a header (X-Lchat-Sidecar), and the activator makes that traffic activity-invisible: it earns no idle-clock credit while the space is awake and receives a 503 (never a wake) while it sleeps. Forging the marker is harmless — it only downgrades the forger’s own requests, and the router strips the header from genuine outside traffic anyway.
• If the co-spawned sidecar dies, the activator restarts it with capped exponential backoff; if it will not start at all, the space still serves — members before bots. The old Node-based reference sidecar does not send the marker and is therefore unsuitable for hibernating hosts; the Go sidecar (the default) is required there.

Observing hibernating spaces (they are not in systemctl)

This is the operational surprise worth internalising from this page: an awake hibernating instance has no systemd unit. It is a child process of the router. Your usual reflexes translate as follows:

journalctl -u lchat-router -f &
curl -s https://chat-b.example.com/api/version
activator: chat-b.example.com waking (lease acquired)
activator: chat-b.example.com restored: local db present, skipping restore
activator: chat-b.example.com awake (2.1s)
{"version":"a41c9f2","build":"1b4bd808"}

The sharp edges, learned in production

• Never let two replicators track one database. The box-wide fleet backup (part 08) and the activator’s embedded replication must divide the databases between them: the hibernate --on flip refuses while the backup configuration still lists the host, and the backup script skips hibernating hosts when regenerating. If you ever edit /etc/litestream.yml by hand, preserve that invariant — the failure mode is mutual corruption of replication state.
• Environment file ownership follows the reader. Systemd read /data/<host>.env as root; the activator reads it as the service user. The flip commands and add-instance fix ownership automatically now, but a hand-copied environment file with root-only permissions will fail its first wake with a permission error.
• The 503s are a feature. A lease conflict or a wake timeout answers 503 with Retry-After. Clients treat it as “try again in a moment”, which is the honest truth. A burst of 503s on one host is worth investigating; scattered occasional ones during wakes are the design working.