Creating and removing chat spaces

You are here: the control daemon and cloud provisioner. The manual commands on this page do the same work the control daemon does, from your terminal instead of over its socket.

Adding a space by hand: lchat add-instance

One command creates a fully working space. Before running it, the only prerequisite is DNS: the hostname must resolve to the box. With a wildcard DNS record (*.example.com → the box’s address) every subdomain already resolves, so there is no per-space DNS step at all.

lchat add-instance chat-c.example.com
setup link:  https://chat-c.example.com/#setup=Xk3…   (one-time; give this to the space’s owner)
admin link:  https://chat-c.example.com/admin#token=9f2…

What just happened, in order — this sequence is worth reading once carefully, because remove, hibernation, and migration all make sense in terms of it:

1. Validate and register. The hostname is checked (it becomes a filename, a unit name, and a socket path, so it must be a clean domain name), an admin token is minted, and a row is written to the registry with status creating.
2. Write the environment file. /data/chat-c.example.com.env (mode 0600) receives LCHAT_ADMIN_TOKEN=… plus any per-instance settings passed as flags. Remember from part 03: the database will store only a hash of the admin token, so this file is the token’s only recoverable copy.
3. Point the version link. The currently running build is staged into the version store if it is not already there, and /opt/lchat/live/chat-c.example.com is created pointing at it — so the new instance starts on a known build (see part 04).
4. Start and health-check the process. systemctl enable --now lchat@chat-c.example.com, then wait up to fifteen seconds for the socket to answer. On failure the row is marked failed with the error recorded (visible in lchat instances), and the files are left in place for diagnosis.
5. Publish the route and warm the certificate. The registry row flips to ready, the router reloads, and the command makes one HTTPS request through the router with the real hostname — which triggers the Let’s Encrypt issuance (part 02) so the first member does not pay that wait.
6. Mint the setup invite. A one-time setup link is created (this also pre-seeds the space’s admin member). The setup link is what you hand to the person who asked for the space; the admin link is the operator-level console.

Useful flags, all optional: --admin-token <tok> imports an existing token instead of minting one (essential when migrating a space between boxes, because a new token would invalidate every saved admin link); --push-relay-url / --push-relay-token, --home-origin, --app-store-url record per-instance service configuration into the environment file; and --hibernate registers the space as activator-managed from birth (below).

Removing a space: lchat remove, and why deletion archives

Removal is deliberately layered, because “stop serving this hostname” and “destroy this data” are different decisions:

1. Take the space off the internet (data untouched).

   lchat remove chat-c.example.com --stop

   remove deletes the registry row and reloads the router, so the hostname stops being served; --stop also stops and disables the instance unit and its bot sidecar unit. Without --stop, the process keeps running but receives no traffic. Either way, the database, blobs, and environment file remain on disk — re-running add-instance for the same host (with --admin-token from the old environment file) brings the space straight back.

2. Only if the data should go: archive it into the trash.

   Full deletion — the path the self-serve pipeline uses — moves the database, environment file, write-ahead log, push key, and blob directory into /data/trash/<host>-<timestamp>/, and removes the certificate and version link. Nothing in this system unlinks user data directly; the trash directory is the safety net, and a mistaken deletion is recovered by moving the files back and re-running add-instance.

3. Eventually, the trash is purged (only if configured).

   The control daemon sweeps /data/trash daily and deletes archives older than LCHAT_TRASH_KEEP_DAYS days. The default is 0, which means keep forever — purging is opt-in, and the retention window should comfortably exceed the reaper’s thresholds below so that any automated reclaim stays reversible for weeks.

One deliberate exception: for a hibernating space, the parked copy of its database in the S3 bucket is not deleted by the deletion path — the box’s control daemon holds no bucket credentials on purpose. Cleaning old parked prefixes is left to the bucket’s lifecycle rules.

The self-serve pipeline: how a member creates a space with no operator

Members of the product can create new spaces themselves (“Create a space” in the app). No human runs add-instance for those — instead a request flows through three services, each with a narrow job, and understanding the chain makes the failure modes obvious:

The control daemon (lchat controld): the only process allowed to change the box

The control daemon is the “god surface”: it is the one process permitted to run systemctl and write the software store, which is why it runs as root, listens only on a local unix socket (/run/lchat/control.sock), and is never given a public hostname. Its API is four endpoints — create an instance, delete an instance, report a space’s statistics, set a space’s storage quota — and every call must present a bearer token whose only copy lives in the root-only file /etc/lchat/control.token. (Socket file permissions alone cannot be the gate, because every sandboxed service on the box shares the same lchat user; the root-only token is what distinguishes the cloud provisioner, which is handed the token by systemd’s LoadCredential mechanism, from a hypothetically compromised instance, which cannot read it.)

Its create endpoint enforces capacity before doing anything: at most LCHAT_MAX_INSTANCES running instances (default 40), and at least LCHAT_MIN_FREE_GB of free disk (default 5). At capacity it answers 503 at_capacity and the refusal propagates up the chain as a polite “try again later” to the member. Configure these in /data/control.env and restart lchat-control.

The cloud provisioner (lchat cloud): policy, in public

The cloud provisioner is the internet-facing half, running unprivileged behind the router like any other service (registered with lchat add-service). It holds everything that is policy rather than power:

• Caller tokens. Services that may request creations (in practice: the home server) authenticate with tokens the operator mints at runtime — POST /api/admin/tokens with the provisioner’s admin token; tokens are stored hashed and are revocable individually.
• Quotas. Each requesting user may hold at most LCHAT_CLOUD_QUOTA (default 3) free spaces at a time; a deleted space frees its slot, and a paid space does not count. There is also an optional fleet-wide circuit breaker, LCHAT_CLOUD_MAX_CREATES_PER_DAY, whose count deliberately survives create-and-delete games.
• Names. New spaces get short subdomains like k3f.example.com — an encoding (sqids) of the space’s ledger row number, so names are unique by construction, never reused, and carry no information. The base domain comes from LCHAT_CLOUD_BASE_DOMAIN and needs the wildcard DNS record.
• The ledger. Its own small database (/data/cloud.db) records who created what, plus the dormant billing rails: an opaque Stripe customer pointer and an entitlement flag per space — deliberately no email, name, or card data in the ledger itself (identity lives in Stripe, where the operator can see it). partially built (dormant until the Stripe environment variables are set).

Upstream of both sits the home server’s eligibility gate: it forwards a creation request only for a signed-in account that already holds at least one space backup, which keeps space creation invitation-shaped without the server reading anything encrypted. The member’s identity crosses to the provisioner only as an opaque random identifier — the quota is per that identifier, and no personal information flows down the chain.

Enabling the pipeline on a box, once

1. Start the control daemon.

   systemctl enable --now lchat-control     # the token at /etc/lchat/control.token was minted by bootstrap

2. Configure and start the cloud provisioner.

   cat > /data/cloud.env <<'EOF'
   LCHAT_ADMIN_TOKEN=<a fresh secret for the provisioner’s own admin API>
   LCHAT_CLOUD_BASE_DOMAIN=example.com
   LCHAT_CLOUD_QUOTA=3
   EOF
   chown lchat:lchat /data/cloud.env && chmod 600 /data/cloud.env
   systemctl enable --now lchat-cloud

3. Give the provisioner a public hostname through the router.

   lchat add-service cloud.example.com --socket /run/lchat/cloud.sock

4. Mint a caller token and hand it to the home server.

   curl -s -X POST https://cloud.example.com/api/admin/tokens \
     -H "Authorization: Bearer <the provisioner admin token>" -d '{"label":"home"}'
   {"id":1,"token":"<shown exactly once — copy it now>"}
   # then add to /data/home.env:  LCHAT_CLOUD_URL=https://cloud.example.com
   #                              LCHAT_CLOUD_TOKEN=<the minted token>
   systemctl restart lchat-home

The reaper: reclaiming spaces nobody ever claimed

Self-serve creation produces abandonment: someone creates a space, never finishes setting it up, and its slot counts against capacity forever. The reaper is the cloud provisioner’s answer, and it is engineered to be terrified of false positives:

• Only ledger spaces are candidates. It considers only spaces the self-serve pipeline itself created (rows in /data/cloud.db) — operator-created instances, the home server, and the push relay are structurally not candidates. Paid spaces are excluded at the query level.
• “Never claimed” requires three signals to agree. A space is reclaimed only if its setup was never completed, and it has at most its one pre-seeded admin member, and it contains no messages or threads at all. Any error reading a space’s statistics is a skip, never a reap. (A hibernating space that has never even woken has no database file; it is answered definitionally from its registry row rather than being treated as an error — otherwise register-only spaces could never be reclaimed.)
• It is off by default and bounded when on. Enable by setting LCHAT_CLOUD_REAP_AFTER_DAYS (choose a value comfortably past the seven-day life of the setup invite — fourteen is sensible); it sweeps every LCHAT_CLOUD_REAP_INTERVAL_HOURS (default 6), reclaims at most LCHAT_CLOUD_REAP_MAX_PER_SWEEP (default 5) spaces per sweep, and honours LCHAT_CLOUD_REAP_DRY_RUN=1, which logs what it would reclaim. Run the dry run for a few days before trusting it.
• Reaping is the archiving deletion described above — into /data/trash, recoverable until the (separately configured, default-never) trash purge.

Spaces born asleep

On a box with hibernation enabled, creation can skip starting a process at all: add-instance --hibernate (or, for the whole self-serve pipeline, LCHAT_CONTROL_HIBERNATE_CREATES=1 in /data/control.env) registers the space as activator-managed from birth. No systemd unit is created and no process runs; the first request to the hostname performs the first wake, which finds no parked database in the bucket and simply starts fresh. Capacity accounting changes accordingly: hibernating registrations are bounded by a row ceiling (LCHAT_MAX_REGISTERED, default ten times LCHAT_MAX_INSTANCES) rather than the running-process cap, because a sleeping space costs disk, not memory.