The mesh: running more than one box

You are here: the peer boxes, and the bucket they coordinate through. Note that peers never need to discover each other directly — everything flows through the bucket they already share for hibernation.

The idea: ownership as a rule, correctness as a lock

Hibernation (part 06) already moved every space’s durable state into the bucket, and made the per-space lease the single arbiter of who may write it. That does most of the multi-box work by itself: any box could wake any space, and the lease already prevents two from doing so at once. What remains is efficiency and manners — spaces should not ping-pong between boxes, and a request that lands on the “wrong” box should still be answered. The mesh adds exactly three small mechanisms on top of the bucket:

1. Membership: each box periodically writes a tiny heartbeat object; listing them yields the set of live boxes.
2. Placement: a deterministic hashing rule (“rendezvous hashing”) maps each hostname to one preferred owner among the live boxes.
3. Forwarding: a box that receives a request for a space it does not own transparently proxies the request to the owner.

Enabling the mesh on a box

Three environment variables in /data/router.env, on top of an already-configured hibernation bucket:

LCHAT_HIBERNATE_BUCKET=my-hibernation-bucket   # prerequisite: the mesh rides on the hibernation bucket
LCHAT_MESH_SELF=https://box-1.example.net      # this box’s identity: a base URL its PEERS can reach it at
LCHAT_MESH_DOMAINS=11ch.example.com            # comma-separated; ONLY hosts under these domains participate
LCHAT_MESH_TOKEN=<a shared secret, identical on every box>  # authenticates route replication

LCHAT_MESH_SELF — the master switch and the box’s identity

A URL with a scheme (https or http) and a host, no path. Its presence turns the mesh on; absent means a single box and zero mesh code. It must be reachable by the other boxes (it is what they proxy to), which may be a private network address (http://10.0.0.5) or a public one.

LCHAT_MESH_DOMAINS — the scope, and it is required

The router refuses to start with LCHAT_MESH_SELF set but no domains. Only hostnames equal to or directly under a listed domain participate in mesh placement; every other host is always served locally, exactly as without a mesh. Two reasons for the scoping, both practical: a routing entry for a host only exists on the boxes it has been replicated to (see route replication), so placement must be confined to hosts every member can serve; and every mesh member must be able to obtain a TLS certificate for any host that might land on it, which per-host issuance cannot promise (it ties a certificate to whichever box the public DNS points at) but a shared wildcard certificate can. In practice: mesh domains should be wildcard-certificate domains.

LCHAT_MESH_TOKEN — the shared secret, and it is also required

The router refuses to start with LCHAT_MESH_SELF set but no token. A bearer secret, identical across every mesh member, that authenticates the route-replication endpoint (below) — without it, any box that could reach another's mesh port could hand it arbitrary routing entries. Generate it once and copy it to every box’s /data/router.env; it is not derived from anything else and is not printed anywhere at runtime.

Membership: heartbeats in the bucket

Every fifteen seconds, each mesh box writes one object under <prefix>/mesh/members/ in the hibernation bucket and lists the others. The scheme has three properties worth appreciating:

• The object’s key is the record — it is an encoding of the box’s own LCHAT_MESH_SELF URL — so a single cheap LIST operation yields the entire membership, with no per-member reads.
• Liveness is judged by the bucket’s clock, not the boxes’ clocks: freshness is the object’s last-modified time as recorded by S3 itself, so a box with a wrong clock cannot poison anyone’s view. A heartbeat older than forty-five seconds (three missed beats) marks its box dead.
• Failures are soft. A box that fails to write its beat is simply taken over by its peers (the lease arbitrates any overlap); a box that fails to read keeps its last known view and at worst misroutes — which, again, the lease absorbs. Neither failure is fatal, and both are logged.

aws s3 ls s3://my-hibernation-bucket/prod/mesh/members/
2026-07-13 14:02:11         38 aHR0cHM6Ly9ib3gtMS5leGFtcGxlLm5ldA
2026-07-13 14:02:09         38 aHR0cHM6Ly9ib3gtMi5leGFtcGxlLm5ldA
# the timestamps are the liveness signal; the key decodes to the peer URL:
echo aHR0cHM6Ly9ib3gtMS5leGFtcGxlLm5ldA | base64 -d
https://box-1.example.net

Placement: rendezvous hashing, and why the lease still decides

For each in-scope hostname, every box independently computes a score for every live member (a hash of the pair “this box, this host”) and the highest score wins ownership. This is rendezvous hashing, and its virtue is minimal disruption: because each (box, host) pair’s score never changes, adding or removing one box only reassigns the hosts whose winning box appeared or vanished — roughly 1/N of them — while everything else stays put. There is no table to distribute, no leader to elect: ownership is arithmetic on the guest book.

But membership views can momentarily disagree (one box has seen a fresh heartbeat the other has not). The design’s answer is that hashing is only a hint, and the lease is the truth: a box that wrongly believes it owns a space simply loses the lease race at wake time, loudly, and the requester retries onto the right answer. Transient disagreement structurally cannot corrupt anything — it can only cost a retry.

Two refinements keep behaviour calm: a space that is currently awake on some box stays there until its natural sleep, even if the hash now prefers another box — its state is already in the bucket, so “migration” is nothing more than waking on the new owner next time. And scheduled wakes fire only on the box that owns the host, so a due reminder wakes one box, not all of them.

Forwarding: wrong-box requests proxy to the owner, once

When a request arrives at a box that is not the computed owner, the box reverse-proxies the entire request to the owner’s LCHAT_MESH_SELF URL — preserving the original hostname, verifying the owner’s TLS certificate against the instance’s hostname (this is why mesh domains want the shared wildcard certificate), and pinning the underlying connection to the peer’s address. There is no certificate-verification bypass anywhere. A forwarded request is marked with an internal header (X-Lchat-Mesh-Forwarded, stripped from all outside traffic), and a marked request is never forwarded again: if two boxes disagree about ownership, the second box serves the request locally — a wrong-box wake the lease then resolves — instead of the pair bouncing the request forever.

Route replication: every box learns about every host

The router looks a request’s hostname up in its own local registry before any mesh logic runs at all, and answers 404 immediately on a miss. So placement (hashing, forwarding) is only half the story: a box can only ever serve — or even correctly forward — a host it has a registry row for. Two mechanisms keep every mesh member’s registry in sync, both authenticated with LCHAT_MESH_TOKEN:

• Automatic fan-out on creation. The moment a new in-scope hibernating host is registered anywhere, that box replicates the row to every currently live mesh peer (found the same way placement does — the membership heartbeats). Only a narrow, purpose-built payload crosses the wire — host, kind, version, hibernate flag, a handful of routing-relevant fields, and the instance’s own env values (admin token, entitlement, seeded name) so a peer-woken instance boots with the token its creator minted — never the receiving box’s own socket path or database path, which each box always derives locally. A peer that is down at that moment simply misses the row; fan-out is best-effort, logged, and never blocks the create. On a region-scoped fleet the target set is a union: the boxes that cover the host by domain, plus any box that owns it by home_region — so a space placed into a region whose box sits outside the creating box’s domains still gets its row there. (Presenting a valid TLS certificate for such an out-of-domain host is a still-unbuilt follow-on; on the single-region production fleet the two sets coincide and nothing changes.) The daemon doing the creating must see the mesh env: lchat-control.service reads /data/router.env for exactly this (a controld without LCHAT_MESH_SELF silently skips fan-out, and any create HRW-placed on a peer is born dark — the peer answers 404 unknown host for it; a live production incident, 2026-07-24). controld logs its replication posture at startup.
• lchat mesh-seed [--region <cc>] <peer-url> — the one-time catch-up. Run on an existing box and pointed at a newly joined one, this bulk-replicates every in-scope hibernating route the existing box already knows about, in one pass. This is the command that makes joining a box a single step instead of hand-registering every host (see adding a box, below).

Adding, draining, and losing a box

Adding a box to the mesh

1. Provision the box normally.

   Install the binary and run lchat bootstrap, exactly as in part 04. Give it the same hibernation bucket, prefix, and credentials as its peers in /data/router.env — the shared bucket is the entire coordination fabric.

2. Give it the shared wildcard certificate capability.

   Configure LCHAT_WILDCARD_DOMAINS (and the DNS API token) for the mesh domains, so this box can present valid certificates for any space that lands on it.

3. Install the control-token drop-in.

   Copy deploy/systemd/lchat-router-control-token.conf in alongside lchat-router.service so the router can materialize a replicated route’s first wake through controld — see the warning above. Skipping this is the single most common way a fresh mesh box looks fine (heartbeats, membership) until the first real request lands on it.

4. Set its mesh identity and restart the router.

   LCHAT_MESH_SELF=https://box-2.example.net
   LCHAT_MESH_DOMAINS=11ch.example.com
   LCHAT_MESH_TOKEN=<the same shared secret as every other box>
   # then:  systemctl restart lchat-router

   Within one heartbeat interval the peers see the new member.

5. Seed its registry from an existing box.

   lchat mesh-seed --env-file /data/router.env https://box-2.example.net
   # region-scoped fleet — also seed routes the new box owns by home_region:
   lchat mesh-seed --env-file /data/router.env --region fi https://box-2.example.net

   (--env-file borrows the router’s LCHAT_MESH_* values — the seed can run under a sudo’d operator session that has no mesh env of its own. Add --prune to also tombstone peer rows this registry no longer has — the reconciliation half of deletion replication. The encoded operator path is ./deploy/ops mesh seed [--prune] <peer-url> [box].)

   Bulk-replicates every in-scope hibernating route in one pass, so the new box can immediately answer for (or correctly proxy) any of them. After this, roughly 1/N of the in-scope spaces re-home to the new box at their next sleep/wake boundary — nothing needs announcing beyond this one command; the guest book and the registry rows do the rest. On a region-scoped fleet, pass --region with the new box’s LCHAT_MESH_REGION so any space pinned to that region — even one addressed outside the seed’s domains — is seeded to it as well.

Draining a box (deploys, maintenance, scale-down)

Stopping the router is the drain: its shutdown sequence deletes this box’s heartbeat first — so peers take ownership immediately — and then checkpoints every awake space to the bucket and releases its leases as usual. Deploying a new router build, retiring a box, and reacting to failure are deliberately the same code path.

systemctl stop lchat-router   # heartbeat deleted, spaces checkpointed, leases released

The recovery numbers, verified by live tests in both directions: after a graceful drain, the survivor is serving the drained box’s spaces about two seconds later; after a hard kill (no drain, heartbeat left to rot), within about the lease lifetime — thirty seconds when those tests ran, fifteen minutes since #836 stretched the lease to tame its S3 request bill — with data intact in both cases (up to the space’s replication class’s window), because writes had been syncing to the bucket all along.

Region-scoped placement: pinning a space to a place, and moving it

The mesh so far spreads spaces across boxes for capacity and failover, treating every box as interchangeable. Region placement adds a second axis: a space can declare where it wants to live, and the mesh honours it. This is the machinery behind data residency (“this space stays in the EU”) and, eventually, latency (“serve this space near its members”) — and it is running on the dev mesh today, proven across a two-region (Dublin + Helsinki) fleet. It is on main but not yet deployed to production, which is still a single box.

Two fields carry it, one on each side of the pairing:

LCHAT_MESH_REGION — a box’s region

An opaque label (e.g. ie, fi) each box advertises in its heartbeat body (not the key, which stays the peer URL). Absent means the box is region-agnostic and takes part in placement exactly as before; setting it costs nothing until a space actually names a region.

home_region — a space’s desired region

A replicated registry column on the route. When set, rendezvous hashing is scoped to only the live boxes whose LCHAT_MESH_REGION matches it — so the space is owned by a box in its region, and a box in the wrong region forwards to the right one. Empty (every space today) means the whole fleet is eligible, byte-for-byte the pre-region behaviour. If a space names a region no live box serves, placement falls back to the whole fleet rather than stranding it — availability wins over residency.

The address stays put: region is a property, not a place in the name

Because a space’s address is its identity (invite links and per-origin browser keys bind to it), region cannot live in the hostname — changing region would change the name and break every member. So a region-placed fleet mints spaces at a single region-neutral domain (<id>.example.com, one wildcard) and the region rides in the home_region field instead. Three variables make that safe at fleet scale:

LCHAT_CLOUD_REGION + add-instance --home-region <cc> — stamping the region at birth

The provisioner stamps each new space’s home_region from LCHAT_CLOUD_REGION (or the flag, for a hand-run create), so a space is placed in its region from its first wake.

LCHAT_CLOUD_ID_LEDGER — one id space across regions

With a single region-neutral namespace, two regions minting at once must never collide on a hostname. The ledger is a small, shared, S3-backed monotonic counter (block-allocated, so roughly one S3 write per hundred creates) that every provisioner draws from — <id> is globally unique regardless of which region minted it. Off by default (a lone provisioner keeps its local ids); an explicit opt-in, because switching it on for a namespace that already has spaces needs the counter seeded above the existing maximum first, or fresh ids would re-mint live hostnames.

LCHAT_DNS_PROVIDER=route53 — a wildcard cert without a token on the box

Every region’s box must terminate TLS for any region-neutral host (it may own the host, or forward to the box that does), which needs a shared wildcard certificate — and a wildcard is DNS-01 only. The DNS-01 solver is pluggable: DNSimple stays the default, and Route53 is an additive off-ramp that authenticates with the box’s own AWS role/user (scoped to one hosted zone) so no DNS token sits on disk. It never replaces DNSimple — it is chosen only by this variable.

Relocating a space between regions

lchat relocate <host> --to <region> is the deliberate move — an admin changing where a space lives, without changing its name. It runs three steps, and the single-writer lease is what makes it safe:

lchat relocate 8oe.example.com --to fi
relocated 8oe.example.com: ie → fi
  store: no copy — source and destination share the same bucket (the shared-bucket case)
  home_region flipped + route re-fanned-out to peers — the mesh now places 8oe.example.com on a fi box

1. Store copy. The destination box wakes the space by restoring its DB from its region’s object store, so the replica must be there first. When both regions share one bucket the bytes are already visible and this is a no-op; with per-region buckets it is a real prefix-to-prefix copy (deliberately excluding the lease object, so the destination takes a fresh lease).
2. Flip home_region + re-fan-out. The registry column is updated and the route re-replicated to peers, so every box now hashes the space onto a destination-region box.
3. (Optional) move the A record. Repointing DNS at the destination avoids the forwarding hop, but is not required for correctness — the wildcard + the mesh’s forward-to-owner already route through the DNS-TTL window. Off unless --move-dns (Route53 only).

The honest limits: what stands between here and a true multi-box fleet

Two gaps remain. Neither is speculative — each corresponds to a concrete piece of code that is reserved or not yet written. (Two things this section used to list — registry replication and fleet-wide status aggregation — have since shipped and are covered above and in part 08.)

• partially built Route mutations: deletions now replicate; hibernate flips still don’t. Since #1346 both delete paths (controld’s deleteInstance and lchat remove) fan a tombstone to every live peer (DELETE /api/mesh/route/{host} — idempotent; removes the row, the in.<host> ingest row, and the replicated env file, never member-data-shaped files), and mesh-seed --prune reconciles whatever a best-effort fan-out missed by diffing the peer’s in-scope listing (GET /api/mesh/routes) against the source registry — with a zero-source guard so a misread registry can never mass-delete a peer. Without this, a destroyed space lived on — and could wake, from a stale replica — on any peer holding its row (six such zombies were swept off a production box by hand on 2026-07-25). Still open: a hibernate flip doesn’t replicate, and must not naively — a peer holding Hibernate=false would proxy to a local socket that doesn’t exist instead of forwarding; a flipped-to-systemd host needs a “lives elsewhere” route shape peers don’t have yet. Renames don’t exist (the host is the key).
• built, not yet proven in production The cross-region store copy is exercised only as a no-op. Relocation’s per-region-bucket copy path is written and unit-tested, but the dev mesh shares one bucket, so live proof only covers the no-op skip path. Before a real cross-region move in production it needs: the source quiesced (asleep) before the copy so a live-writing source can’t be copied mid-write; the copy client signed for the destination bucket’s region; and a download+reupload variant for cross-account or R2 destinations (CopyObject covers only same-account, same-provider). All three are noted in docs/multi-edge.md.
• partially built The certificate cache is per box. The wildcard certificate makes mesh domains workable (each box independently obtains its own copy), but per-host certificates for domains outside the wildcard cannot follow a space between boxes; the S3-backed shared cache that would fix this is reserved (LCHAT_CERT_CACHE=s3) and deliberately refuses to start. See part 02.