# πŸ€– The cloud provisioner β€” self-serve space creation Status: **shipped** (server side). The app UI ("+ β†’ Sign in β†’ new space") is deliberately not built yet; this doc covers everything behind it. ## The idea Creating an instance used to be an operator SSHing to the box and running `lchat add-instance`. The end state we want: a member signed in to their home server taps **+** in the app and gets a fresh, fully-isolated Eleven space β€” address, TLS, admin-granting setup link β€” in a few seconds. Two pieces make that possible: - **The cloud provisioner** (`lchat cloud`, e.g. at `cloud.elevenmessenger.com`) can create instances on its box, gated by caller tokens, per-user quotas, and a box capacity limit. It's a subcommand of the same binary that runs the router, so **any host running Eleven can enable it** and get multi-instance auto-provisioning β€” it isn't infrastructure only we can run. - **The home server** gains one endpoint (`POST /api/spaces`): a signed-in user asks *their* home server for a space; home forwards the request to the cloud provisioner it's paired with and relays the result. ``` app ── link token ──▢ home ── caller token ──▢ lchat cloud ── control token ──▢ lchat controld ──▢ systemctl… (lchat-home unit) (lchat user, sandboxed, (root; local unix socket public via the router) /run/lchat/control.sock) ``` The client then opens the returned **setup link** (`/join/setup#`), claims it, becomes the space's admin, and runs the normal setup conversation. Adding the new space to the user's encrypted backup is the client's move, exactly like any other server β€” the blob stays content-blind. ## The privilege split Creating an instance needs root (systemctl, writes to `/opt/lchat`), but everything network-facing on the box runs as the unprivileged `lchat` user under a strict systemd sandbox. So provisioning is two processes: - **`lchat controld`** (`lchat-control.service`, **root**) β€” the god surface `instance-router.md` reserved: `POST /instances` on a local unix socket, never routed publicly. It executes the same creation core as `lchat add-instance` (`addInstance` in `control.go`), serialized, and is the single authority on **capacity** (below). - **`lchat cloud`** (`lchat-cloud.service`, `lchat` user, sandboxed) β€” the public policy layer: caller tokens, per-user quotas, friendly slug generation. It holds no root powers; everything it can do to the box, it does by asking controld. ### Who holds which token | Token | Lives | Grants | |---|---|---| | control token | `/etc/lchat/control.token`, **root-only 0600**; handed to the cloud unit alone via systemd `LoadCredential` | create instances on the box | | cloud admin token (`LCHAT_ADMIN_TOKEN`) | `/data/cloud.env` | mint/list/revoke caller tokens | | caller token | the paired home server's env (`LCHAT_CLOUD_TOKEN`) | provision spaces, bounded by quota | | device link token | the user's device | ask *their* home server for a space | The root-only control token matters because **every service on the box shares the `lchat` uid** β€” socket permissions can't distinguish the cloud service from a compromised instance. A compromised instance can read `/data` (a known, accepted box-level exposure), but it cannot read the control token, so it cannot mint instances or reach the creation path. `lchat bootstrap` mints the token idempotently. ## Capacity: the box says no before it falls over controld enforces two limits, checked atomically with each create (`LCHAT_MAX_INSTANCES`, default **40**; `LCHAT_MIN_FREE_GB`, default **5**): - **Instance ceiling.** The educated guess for a 4 GB box: the OS, router, push, home, and the existing fleet use ~1 GB; an idle instance is ~25–40 MB (no server-side crypto or media work, by design), so 40 spaces β‰ˆ 1.4 GB with ~1.5 GB headroom for spikes. When Go bot sidecars land (~25 MB each, vs ~120 MB for today's Node sidecar), 40 spaces *with* bots still fits. - **Disk floor.** Disk, not RAM, is the long-run binding resource (attachments are blobs in SQLite), so creation also refuses when `/data` has less than the floor free. Past either limit the API answers `503 {error: "at_capacity"}` and the cloud layer relays the member-facing message: *"We can't create a new Eleven space right now β€” please try again later."* Clients show it verbatim and stay calm. ### Hibernate-by-default creates (`LCHAT_CONTROL_HIBERNATE_CREATES=1`) With this env set on `lchat-control` (off by default β€” the flip is a per-box config decision, not a code deploy), every provisioner-created space is born **hibernating** (#550): the create registers the route, writes the env file, stages the binary β€” but starts **no systemd unit and no process**. The create-path readiness probe is the space's one *first wake* (it mints the setup link and warms the cert through the router's activator); after the idle window the activator puts it to sleep, and a never-claimed space costs **zero RAM** from then on. It requires the **router** to be running with `LCHAT_HIBERNATE_BUCKET`, or created spaces never answer. Direct `lchat add-instance` keeps its explicit `--hibernate` flag either way. Under the flag the capacity policy changes shape: - **`LCHAT_MAX_INSTANCES` counts *materialized* instances only** β€” the systemd-managed, always-on ones that actually hold RAM. Hibernating rows don't consume it, so the cap stops being the constraint on self-serve creation. - **`LCHAT_MAX_REGISTERED`** (default **10Γ— `LCHAT_MAX_INSTANCES`**) is a sanity ceiling on *total* instance rows, hibernating included β€” registry rows, env files, and staged symlinks aren't free forever. - The **disk floor** applies unchanged; disk stays the real long-run bound. ## Addresses Spaces get generated friendly slugs β€” `amber-fox-42.` β€” from small embedded wordlists (collisions re-roll; a reserved blocklist keeps sibling names like `home`/`push`/`cloud` unmintable). The display name of the group is set in the setup conversation; the hostname is just an address, and hostnames are public in certificate-transparency logs regardless. User-chosen slugs (or custom domains) are a possible later feature, deliberately not v1. ### Addressing (sqids) A space's subdomain label is **`sqids(id)`** β€” the sqid of its integer primary key in the `spaces` table. So `a4f.` both *is* short and *encodes* the id reversibly: `spaceID("a4f")` recovers the id, i.e. "which server this is." SQLite supplies the id (`INTEGER PRIMARY KEY AUTOINCREMENT`); Go's `sqids-go` does the encoding (lowercase base36, min length 3 so labels grow only as ids do, default blocklist so nothing rude is minted, `AUTOINCREMENT` so a torn-down space's id β€” and thus its label β€” is never reused). Creation reserves the id first (quota-checked), turns it into the hostname, then creates the instance; there's no wordlist and no collision re-roll, because distinct ids give distinct labels by construction. ### Certs + readiness The base domain (`LCHAT_CLOUD_BASE_DOMAIN`) needs **wildcard DNS** pointing at the box. Certs are per-host (autocert HTTP-01) unless the box is configured for a wildcard cert (DNS-01 β€” see `instance-router.md`). Either way, provisioning **confirms readiness through the public path before returning the setup link**: after the instance passes its own socket health check, `add-instance` issues an HTTPS request *through the local router* (dialed at `:443` with the new host's SNI) and waits for a non-5xx reply. That guarantees the moment a human clicks the setup link they reach a ready admin sign-on β€” never a mid-provision error β€” and, on a per-host-cert box, warms the cert as a side effect (the handshake triggers issuance) so there's no first-hit "not secure" race. Best-effort: on timeout it returns anyway (the instance already health-checked; the public path is a beat behind at most). The home UI keeps a spinner up for the whole call, so the "Create a new space" button only reveals the link once the space is confirmed up. A wildcard record also means typo'd or retired subdomains resolve to the router. Set the router's `LCHAT_FALLBACK_URL` (e.g. the apex marketing site) so those redirect somewhere useful instead of 404ing. On a per-host-cert box an unknown host fails TLS before any redirect can happen (no cert is ever minted for it), so the redirect only covers `:80` and hosts whose cert outlived their route β€” but with the **wildcard cert** (DNS-01, `instance-router.md Β§ Wildcard TLS`) any subdomain of the base domain completes the handshake, so garbage subdomains land on the fallback over https too. ## API ### cloud (`lchat cloud`) ``` POST /api/instances bearer: caller token {requester: } 201 {host, url, setup_link} 409 {error: quota_exceeded} per (caller, requester) FREE-space quota, default 1 (LCHAT_CLOUD_QUOTA; paid spaces never count, and upgrading frees the slot) 503 {error: at_capacity, message} the box is full GET /api/instances?requester=… this requester's spaces (+ entitlement/tier, and a top-level billing price book) POST /api/instances/{host}/checkout bearer: caller token {requester, tier, success_url, cancel_url} 200 {url} the hosted Stripe Checkout page 400 unknown tier | 404 not yours | 409 already_upgraded | 501 billing off POST /api/instances/{host}/portal bearer: caller token {requester, return_url} 200 {url} the hosted Billing Portal (cancel/card/invoices) 404 not yours | 409 not_subscribed | 501 billing off POST /api/stripe/webhook signature-gated (Stripe calls it; see Billing) POST /api/admin/tokens bearer: admin token {label} β†’ {id, token} (shown once) GET /api/admin/tokens list (never the token values) DELETE /api/admin/tokens/{id} revoke ``` ### controld (`lchat controld`, local socket only) ``` POST /instances bearer: control token {host, admin_token?, …env fields} 201 {host, url, setup_link, admin_link} 400 invalid_host | 409 host_exists | 503 at_capacity | 500 create_failed ``` ### home ``` POST /api/spaces bearer: device link token β†’ relays the cloud response (201 / 409 / 503 shapes above) 403 {error: not_eligible} caller has no saved space yet (see gate below) 501 {error: unavailable} this home server has no provisioner configured GET /api/spaces bearer: device link token β†’ the hosts this user provisioned (drives the "Delete space" affordance) DELETE /api/spaces/{host} X-Ceremony-Proof: fresh passkey ceremony ONLY β†’ relays cloud's 200 / 404 POST /api/spaces/{host}/checkout bearer: device link token {tier} β†’ relays cloud's 200 {url} / 400 / 404 / 409 / 501; return URLs point back at home POST /api/spaces/{host}/portal bearer: device link token β†’ relays cloud's 200 {url} / 404 / 409 / 501 (Manage on a paid space's row) ``` πŸ€– **Per-country provisioning (optional, `LCHAT_CLOUD_REGIONS`).** By default home talks to one provisioner (`LCHAT_CLOUD_URL` + `LCHAT_CLOUD_TOKEN`). Setting `LCHAT_CLOUD_REGIONS` β€” a JSON array of `{code,name,url,token}` β€” plus `LCHAT_CLOUD_DEFAULT_REGION` lets a signed-in user's space be placed in one of several country provisioners (the dev mesh runs `ie` + `fi`; see `docs/dev-mesh.md`). It supersedes the single `LCHAT_CLOUD_URL`; the default region's provisioner also backs demo and every host-agnostic fallback. Routing: `POST /api/spaces` takes an optional **`region`** (empty = auto-assign to the default; an unknown code quietly auto-assigns rather than failing); `GET /api/spaces` fans out across every region and merges, so a user sees spaces they hold anywhere; host-specific calls (delete, checkout, portal) route by the region label the host encodes (`..…`). `GET /api/features` advertises the regions β€” `code` + `name` only, never a url or token β€” so the create UI can offer the choice; single-region home omits the key. (The create-page region control is a follow-on; today the choice rides the `region` field.) Creation and listing are link-token-authed (additive/reads) and creation is rate-limited per IP; the real abuse bound is the provisioner's quota. **With `LCHAT_POW_BITS` set on home (default off), creation also demands hashcash**: `POST /api/spaces/challenge` mints a sealed, expiring, single-use challenge bound to the account; the client grinds SHA-256(challenge:nonce) to that many leading zero bits during the fader's build phase and submits it with the create. Verification is one hash; a consumed challenge never buys a second space. This prices the Sybil loop (invite yourself β†’ fresh account β†’ 3 more free spaces) in CPU. Above all of that sits the **fleet-wide creation budget** (`LCHAT_CLOUD_MAX_CREATES_PER_DAY`, 0/unset = uncapped): a trailing-24h circuit breaker counted against an append-only log, so create-and-delete can't reset it. Past the budget, creation answers the same member-facing at-capacity message. This is the pre-production insurance dial β€” the per-account gates do the fine-grained work; this bounds a bad day. On bot detection generally, the position is deliberate: **passkeys are the bot check**. Creating an account requires a platform-authenticator WebAuthn ceremony; there is no fingerprinting, UA sniffing, or behavioral scoring anywhere, and there won't be β€” the layers are cryptographic (passkeys), economic (hashcash, storage caps), and quantitative (quotas, rate limits, the budget, the reaper). **Creation is also gated to existing members:** `POST /api/spaces` returns `403 not_eligible` unless the caller already has a backup blob. A blob only exists once you've enrolled β‰₯1 space, and enrolling needs a space you were *invited* into β€” so this keeps creation invite-gated (not open to the world) while it's early, without the server ever reading the E2E-encrypted server list. The client hides the "+" when the list is empty; the 403 is the boundary behind that. Lift it later by removing the check (the invite-only funnel becomes self-serve). **Deletion is ceremony-gated** β€” same rule as revoking devices or deleting the backup: a stolen link token must never be able to destroy a space. The UI adds its own stern step (type the space's address to arm the button) before the ceremony, and the cloud layer re-checks ownership against its ledger regardless: only the exact (caller, requester) that provisioned a host can delete it, so spaces added to a list by URL are never deletable through this surface. "Remove from list" (vault-only, the space keeps running) is a separate, lighter client-side action. On the box, deletion **archives** the instance's DB/env to `/data/trash/-/` rather than destroying them β€” the operator safety net until the trash is purged. ## What this changes about who-sees-what Provisioning is the one place the home server (and the provisioner's ledger) learns a relationship between a user and an instance hostname β€” `user X provisioned host Y`. That's **operational metadata, held outside the blob**, and it's deliberate: quotas need a ledger. The requester id is the home server's random opaque uid (no email, no name), the blob stays content-blind, and a user who never provisions never appears in the ledger. Members who join by invite are invisible here, unchanged. ## Reaping never-claimed spaces Self-serve creation hands out a capacity slot the instant a space is made. A space whose admin never finishes setup β€” the setup invite expires after 7 days β€” would otherwise sit there forever: empty, but holding a slot and a disk. The **reaper** (in `cloud`, `reaper.go`) reclaims them: - Every `LCHAT_CLOUD_REAP_INTERVAL_HOURS` (default 6) it sweeps the ledger for finalized spaces older than `LCHAT_CLOUD_REAP_AFTER_DAYS` (**opt-in β€” 0/unset disables it**; a value comfortably past the 7-day invite expiry, e.g. 14, is the intent). - For each candidate it asks the control daemon `GET /instances/{host}/stats`, which reads the instance DB **read-only** and returns `setup_complete`, member count, and `has_content`. A candidate is reaped only when **all three guards** agree it's an abandoned shell: 1. **`setup_complete` absent** β€” the admin never finished the setup conversation. (The signal is `setup_complete`, **not** member count: `add-instance` pre-seeds the admin user so the setup invite has an identity, and claiming *becomes* that user β€” so a space has exactly one member whether or not it was ever set up.) 2. **`members <= 1`** β€” nobody beyond the seeded admin joined. 3. **`has_content` false** β€” zero messages and zero threads. The instant anyone posts anything, the space is protected regardless of the other signals; this is the decisive guard against a false positive on a used-but-not-marked space. **Anything set up, joined, with content, or whose status can't be read is left untouched** β€” a stats error is always a skip, never a delete. - A reaped candidate is deprovisioned through the normal controld delete, which **archives** the DB/env to `/data/trash` (never destroys), so a false positive is recoverable. Its ledger row is cleared, freeing the quota slot. - **Per-sweep cap:** `LCHAT_CLOUD_REAP_MAX_PER_SWEEP` (default 5) bounds how many a single sweep reclaims; the rest defer to the next sweep. This caps the blast radius of any future mis-classification to a handful of recoverable archives before an operator would notice. - `LCHAT_CLOUD_REAP_DRY_RUN=1` logs exactly what it *would* reclaim without deleting β€” run it once before arming the real sweep. Because the reaper only ever considers hosts **in the cloud ledger**, operator-run instances (family.…, home, push, …) are never candidates. ### Self-reclaim: a quota-blocked create frees the account's own dead weight With the free quota at 1, a fumbled create (made a space, lost the tab, never set it up) must not lock the account out until the reaper's 14-day sweep. So when a create hits `quota_exceeded`, the cloud first tries to reclaim **one of the requester's own abandoned allocations** and, if that frees a slot, proceeds with the create instead of failing: - Candidates are only the same (caller, requester)'s free-entitlement rows **older than one hour** β€” setup takes minutes, so the age gate cleanly separates "abandoned, retrying" from "setting it up in another tab right now" (the home page keeps the setup link for continuing inside the window). - A **pending row** (a create that died between allocate and finalize β€” it would otherwise hold the slot forever) is cleared ledger-only. - A **provisioned space** must clear the reaper's exact three guards above (never set up, nobody joined, zero content); it archives to `/data/trash` through the same controld delete. A stats error is a skip, never a reclaim. - Anything in real use stays untouched and the 409 stands β€” the way out then is upgrading a space (paid spaces don't count) or deleting one from home. **Hibernating spaces never wake for the reaper.** Stats always come from local state, not from the instance: an *asleep* space is read from its on-disk DB (the warm cache the activator left at its last sleep β€” on one box, the newest truth), and a space that **never woke** has no DB at all, so its stats are definitional β€” `members 0, setup_complete false, has_content false`, answered from the registry row alone. A sweep over thousands of dormant spaces stays a pure metadata read. Reaping a hibernating space follows the same delete path (unit-stop becomes best-effort β€” there is no unit; the route delete + router reload retires any awake child via the activator's reconcile) and archives to `/data/trash` as usual. The space's **S3 replica prefix is deliberately not deleted**: controld holds no bucket credentials, the prefix is opaque ciphertext of a DB whose authoritative copy just moved to trash, and the versioned bucket's noncurrent-version expiry means it ages rather than grows β€” orphan-prefix cleanup is deferred to the S3 lifecycle work. A companion **trash purge** in `controld` reclaims the archive after a longer window: `LCHAT_TRASH_KEEP_DAYS` (opt-in β€” 0/unset keeps trash forever) removes `/data/trash/-` entries older than that, swept daily. Keep it well above the reap threshold so a mistaken reap stays recoverable for a while. ## Demo spaces (docs/demo-spaces.md slice 1) β€” 24h, discarded, no backups With `LCHAT_DEMO_TTL_HOURS` set (unset = the shipped default, demos off), a create may carry `kind: "demo"`: an `entitlement='demo'` ledger row that skips the free-space quota and is bounded instead by `LCHAT_DEMO_MAX_LIVE` (default 10, pending rows included β€” refusals answer `demo_busy`) plus the shared creates-per-day breaker. The create response carries `expires_at`. Three deliberate inversions of the free-space rules: - **Reaped unconditionally** at `created_at + TTL` by an hourly demo sweep β€” no stats call, no guards: content protects a free space, never a demo (the 24h death is the disclosed contract). `LCHAT_CLOUD_REAP_DRY_RUN` is honoured. The main reaper never sees demos (its candidate query is free-only). - **Discarded, not archived**: the reap deletes via controld `DELETE /instances/{host}?discard=1`, which removes the files outright (blobs included) instead of renaming into `/data/trash` β€” the one exception to archive-everything. - **Never activator-managed**: controld forces `Hibernate=false` for a demo create even on a hibernate-by-default box, so a demo never touches S3 β€” no lease, no litestream replica, no parked prefix. It holds a materialized slot for its 24h life; the live-demo cap bounds how many. Demos take a `demo` key in `LCHAT_CLOUD_STORAGE_CAPS` (falling back to the `free` cap when absent). **The home-side gates** (`home/demo.go`, on with `LCHAT_HOME_DEMO=1` β€” default off): `POST /api/demo/challenge` + `POST /api/demo`, the one account-free creation path, so the graduated-friction ladder is the whole boundary β€” **mandatory adaptive hashcash** (base `LCHAT_DEMO_POW_BITS`, default 18; +1 bit at 5 creates in the trailing hour and another per doubling, with the difficulty sealed inside each challenge so escalation never voids honest work in flight), and **one demo per IP bucket per cycle** (IPv6 at /56; blocked from creation until expiry + one full TTL again, so serial recreation converges to one rolling slot). Buckets are HMACs under a per-process random key β€” no raw IP is ever stored, nothing links across a restart, and a failed create never burns the bucket. The client IP comes from the router's `X-Forwarded-For` stamp. `GET /api/features` advertises `demo` so clients can show the button only where it works. **The home surface**: the sign-in card offers "Try a demo space" (feature-gated, with the honest one-liner about the 24 hours), the client grinds the hashcash inline β€” the burn is the wait β€” and lands in the fresh space's setup link same-tab. `POST /api/spaces/claim` (link-token) relays a claim to the cloud so a signed-in account can adopt a demo as its free space; folding the claimed space into the encrypted backup is the client's move, like a create. Still to come: the in-space countdown/claim banner (an instance-side surface). **Claiming** (`POST /api/instances/{host}/claim`, caller-token): the happy exit β€” a signed-in account adopts a demo, transferring ownership and flipping `entitlement` to `free` in one guarded UPDATE (a racing claim or reaper loses cleanly). The claim consumes the account's free-quota slot (same rules as a create, including the courtesy self-reclaim), refuses expired demos (`410 demo_expired` β€” past-TTL is dead by contract even before the sweep), and re-pushes the instance's entitlement env + free storage cap + replication class. Claims are never pool-gated: the space's spend was already counted while it was a demo. Flipping the claimed space onto the activator (hibernation + S3 backups) is an operator/follow-up step β€” the claim itself never restarts anything. ## The storage meter (#668 slice 1) Every sweep interval (`LCHAT_CLOUD_METER_INTERVAL_HOURS`, default 6, 0 disables), the cloud service asks controld for each ledger space's on-disk footprint β€” the SQLite files plus the attachment blob dir, **pure filesystem metadata: nothing is read, and a hibernating space is never woken**. The reading lands on the ledger row (`storage_bytes`, `metered_at`), rides the space listing, and home shows it to the member ("2.1 GB stored"). Bytes of ciphertext are exactly the kind of thing this ledger is allowed to know (#562: price only what ciphertext exposes). **Storage caps (#561, insurance-grade).** With `LCHAT_CLOUD_STORAGE_CAPS` set ("free=2GB,family=5GB,…" β€” tier keys from the price book plus `free`), the cloud pushes each ledger space's cap into its instance env (`LCHAT_BLOB_QUOTA_BYTES`, via controld `POST /instances/{host}/quota`), where the instance's own upload guard enforces it: uploads past the cap answer 507, nothing is ever deleted, and text messaging is never gated. Pushes happen on tier changes (checkout/subscription webhooks), at creation, and the meter sweep reconciles drift β€” self-healing, best-effort at every step. A unit-managed instance is bounced (`try-restart`, seconds) so the cap applies now; a hibernating one picks it up at next wake β€” never wake a space to limit it. Unset = no enforcement, and operator-run instances (not in this ledger) are never touched. Caps follow the 80/20 margin rule in `tier-economics.md` (an allowance is what 20% of a tier's net buys in storage, forever) and always match the published allowance table. Business is per-seat (`business=11GB/seat` β€” cap = 11 GB Γ— the space's live head count, pooled); a big team's pool exceeds the box's disk β€” honored on demand (attach a Hetzner volume) until attachments move off-box (#550). ## The usage pipe (#857, #668 slices 2–3) β€” AI usage into the ledger With `LCHAT_CLOUD_USAGE_URL` set (the cloud's own public `/api/usage` URL), the same meter sweep mints each ledger space a **meter token** and pushes it into the instance env (`LCHAT_METER_URL` + `LCHAT_METER_TOKEN`, via controld `POST /instances/{host}/meter`, hash-only in the ledger, same self-healing best-effort shape as the quota push). The flow is sidecar β†’ instance β†’ cloud: - The **sidecar** counts model spend β€” it's the only place spend is visible β€” as (day, model, dimension) quantities: `llm_requests`, the four token counters (input / output / cache-write / cache-read, priced differently), and `charter_runs`. It flushes every few minutes to its instance (`POST /api/bot-usage`, bot-token-gated), keeping its egress surface at the two destinations `docs/bots.md` promises. - The **instance** is a stateless relay: validate, forward with the meter token, hand the response's budget state back to the sidecar. Unconfigured (self-hosted, BYO key) it accepts and drops β€” answer `"unmetered"`, nothing recorded anywhere. - The **cloud** upserts into `usage(space_id, day, model, dimension, quantity)` β€” quantities only, never content, never per-member attribution (#668's rule) β€” and answers a budget state, `"ok"` unconditionally until the phase-3 allowance/budget math lands (docs/ai-provisioning.md). Unset `LCHAT_CLOUD_USAGE_URL` = the pipe is off fleet-wide: no tokens minted, no env pushes, instances answer sidecars `"unmetered"`. **Replication classes (the durability ladder).** With `LCHAT_CLOUD_REPLICATION_CLASSES` set (`free=daily,family=daily,community=batched,business=stream` β€” same key convention as storage caps), the cloud maps each ledger space's billing state to a replication class on the activator's S3 replica and pushes it via controld (`POST /instances/{host}/replication` β†’ the registry route + a router reload, only on actual change). Same event-driven-plus-sweep shape as caps: spaces are **born classed** (the create request carries the free class), Stripe webhooks re-push on every entitlement move (a lapse drops the space back to the free rung), and the meter sweep reconciles drift. The class is wake-time config β€” nothing is restarted, no sleeping space is woken, and an awake instance keeps its current sync tick until it next sleeps. An unknown **paid** tier maps to `stream` on purpose: a config typo must fail toward durability, never away from what the member paid for. Unset = no pushes, every space streams (the pre-ladder status quo). What the classes mean β€” sync cadence, loss windows, and the graceful-flush guarantee β€” lives with the mechanism in `registry.go` and the ops guide. ## The free-pool gauge (docs/free-pool.md slice 1) The free pool runs the free tier as a budget: free + demo spend must never exceed a configured share of total cost, funded by paid revenue with an operator floor. **Slice 1 is a gauge only** β€” it computes and records, gates nothing, and ships off by default. At the end of each meter sweep the cloud service estimates every ledger space's daily cost from a declared cost model (`LCHAT_POOL_COSTS`, defaults = the docs/free-pool.md snapshot), derives ``` budget_day = min( X% Γ— total_cost_day, max(floor, Y% Γ— net_revenue_day) ) pressure = free_cost_day / budget_day ``` and records one `pool_meter` row (pruned past ~400 days) plus a log line. Knobs: `LCHAT_POOL_MAX_SHARE` (X β€” **0/unset = gauge off**), `LCHAT_POOL_REVENUE_SHARE` (Y, default 30), `LCHAT_POOL_BASE_EUR_DAY` (default 0.20), `LCHAT_POOL_DEMO_SHARE` (default 20, reserved for the demo tier). Revenue comes from an additive `net_eur_year` field per tier in `LCHAT_STRIPE_PRICES` (entries without it fund the pool as €0 and log a startup warning). `GET /api/pool` (admin token) serves the latest reading β€” labelled an estimate, because it is one: derived from declared constants, not metered. **Slice 2 β€” admission control.** At `pressure β‰₯ 1` on the latest reading, new FREE creates answer `503 free_pool_full` with the honest member-facing message (home's 503 passthrough relays it verbatim). The gate **fails open** on any uncertainty β€” pool off, no reading yet, ledger error β€” and never touches anything else: demos are exempt (their carve-out + TTL bound them), paid creates are never gated (they fund the pool), existing spaces are never touched, and deleting your own space still works at red. `LCHAT_POOL_DRY_RUN=1` computes and logs every would-refuse without refusing β€” arm it the reaper's way: gauge first, dry-run second, live third. Budget + pressure now use revenue **smoothed over the trailing ~90 days** of stored readings (one big churn shrinks the pool gradually); each `pool_meter` row still records the instantaneous figure. ## Billing (Stripe) β€” tiers, quota exemption, no metering Optional space upgrades (`billing.go`): a person who created a space can pay a subscription to upgrade it at one of a small set of **tiers** (Family / Friends / Business β€” config, not code). Per-tier powers (storage, member counts, bots) and metering are #561/#562 and layer on later; today a paid space of ANY tier gets the same two things. **The flow.** The home page shows a quiet *Upgrade* on spaces this account created (only when the provisioner advertises billing). It opens a plan picker (the price book from the listing response); the chosen tier key goes home β†’ cloud, which mints a Stripe **Checkout Session** for that tier's price; the browser goes to Stripe's hosted page and comes back to home (`?upgraded=`). Stripe then confirms through the **webhook** β€” `checkout.session.completed` flips the space's ledger row to `paid` and records the tier, and `customer.subscription.updated/deleted` track it for the life of the subscription (`past_due` keeps access through Stripe's dunning window; deletion drops the space back to `free`). The row's numeric ledger id rides in the session's `client_reference_id` and the subscription's metadata, the tier key in both metadata blobs β€” that's how events route back to a space. Creating a space itself is **never pay-gated** (#1374): the create step is just the name, and "who is this space for?" is asked by Eleven inside the space, priced from the cloud's `GET /api/pricebook` (caller-token'd; home republishes it unauthenticated and CORS-open at its own `/api/pricebook`, cached, for space origins to read). The create screen's tier cards appear only when a create bounces off the free quota or the pool gate β€” the two errors paying genuinely fixes; picking one there runs the pay-first checkout-create path below. **What paying does today:** - a paid space **doesn't count toward the creation quota** (`allocateSpace` counts only free spaces β€” upgrading a space frees its slot, and every extra space carries itself); - a paid space is **never a reap candidate** (excluded in the ledger query itself, the strongest of the guards above); - the entitlement + tier come back on space listings so clients render the badge. Per-space limits read the same columns when #561 lands. **Privacy (the point):** the ledger stores two opaque Stripe ids (`stripe_customer_id`, `stripe_subscription_id`), the `entitlement` flag, and `paid_until`. **Never an email, name, address, or card detail** β€” Stripe holds those on its side, and the operator can see them in Stripe's dashboard. Paying is not anonymous; the invariant (#562's documented residual) is that the *ledger* carries no billing identity. Checkout Sessions are created with no customer fields at all. The webhook is the one public unauthenticated-by-token endpoint; its gate is Stripe's HMAC signature over the raw payload (Β±5 min replay window). **Config** (`/data/cloud.env`; all-or-nothing β€” partial config refuses to start): ``` LCHAT_STRIPE_SECRET_KEY restricted API key (write: Checkout Sessions; read: nothing else needed) LCHAT_STRIPE_WEBHOOK_SECRET the webhook endpoint's signing secret LCHAT_STRIPE_PRICES the price book: a JSON array of tiers, e.g. [{"key":"family","name":"Family","label":"$5/year","price":"price_…"}, …] (display strings live here verbatim β€” the server never asks Stripe what a price costs, so a Stripe outage can't block a cloud restart; the checkout charges the real price) Optional per-tier: "eur" (numeric sticker, feeds the picker's β‰ˆ local-currency display), "business": true (ex-VAT tier: checkout collects a VAT ID), and "bands" β€” the honor prices of docs/pricing-bands.md: {"supporter"|"reduced"|"solidarity": {"label","price","eur","net_eur_year"}}. A tier without bands behaves exactly as before; a price id may appear only once in the whole book. LCHAT_STRIPE_API_BASE default https://api.stripe.com (tests point it at a stub) LCHAT_STRIPE_AUTOMATIC_TAX "1" opts Checkout Sessions into Stripe Tax (docs/pricing-bands.md Β§ VAT). Flip it WITH the band-price rollout β€” every live Price must carry a tax_behavior (or an account default) first. ``` Unset = billing off: checkout answers 501, listings say `billing: false`, the UI shows nothing. The Stripe-side webhook endpoint must subscribe to `checkout.session.completed` and `customer.subscription.*`, pointed at `https:///api/stripe/webhook`. **Managing a subscription** is Stripe's hosted **Billing Portal**: Manage on a paid space's row β†’ home β†’ cloud mints a portal session for that space's own Stripe customer (same ownership gate as checkout). The portal also offers **plan switching** (upgrade/downgrade, monthly↔annual, prorated by Stripe); the webhook derives the tier from the subscription's current PRICE (metadata only seeds the first value), so the badge and the pushed storage cap follow a switch automatically. **Cancelling** is at-period-end: the space keeps its tier until the paid time runs out, then the deletion event drops it to free β€” the space survives, its storage cap returns to the free allowance (existing content is never deleted; only new uploads pause if over), and it counts toward the creation quota again. Portals are per-space by construction β€” every space is a separate Stripe customer, deliberately (no cross-space identity exists to merge). `LCHAT_STRIPE_PORTAL_CONFIG` optionally pins a portal configuration id; unset uses the account's default (which must exist β€” save the portal settings once in the Stripe dashboard, or create a configuration via the API). With bands, `LCHAT_STRIPE_PORTAL_CONFIGS` (JSON `{band: configuration id}`) picks a per-band configuration β€” each band's portal offers plan switches inside that band only; band changes go through the home surface, where they can be messaged. Bands without an entry fall back to `LCHAT_STRIPE_PORTAL_CONFIG`. ## Enabling it on a box ```sh lchat bootstrap # installs units, mints /etc/lchat/control.token systemctl enable --now lchat-control cat > /data/cloud.env < LCHAT_CLOUD_BASE_DOMAIN= LCHAT_CLOUD_REAP_AFTER_DAYS=14 # optional: reclaim never-claimed spaces (0/unset = off) EOF systemctl enable --now lchat-cloud # optional trash purge (controld): set LCHAT_TRASH_KEEP_DAYS in /data/control.env, restart lchat-control lchat add-service cloud. --socket /run/lchat/cloud.sock # mint the home server's caller token: curl -sX POST https://cloud./api/admin/tokens \ -H "Authorization: Bearer $ADMIN" -d '{"label":"home."}' # hand it to home: LCHAT_CLOUD_URL + LCHAT_CLOUD_TOKEN in /data/home.env, restart lchat-home ``` ## Phase 2 (designed for, not built) - **App UI** β€” "+ β†’ Sign in (home.elevenmessenger.com) β†’ new space", and the server list syncing/sorting through the existing encrypted blob. - ~~**Bot sidecars on provisioned spaces**~~ β€” SHIPPED (#858): controld's `POST /instances/{host}/bot` stages an **Eleven-only** sidecar (idempotent; the model key comes from controld's own env, never the instance's). The trigger is the admin's AI opt-in: setup's dedicated `ai-optin` step (or any later flip of the AI toggle) makes the instance signal `POST /api/ai-optin` on the cloud over its meter token, and the cloud calls controld. Hibernating hosts stage the env only β€” the activator co-spawns at wake, and its sweep adopts an already-awake child within seconds. Meter tokens are now also minted eagerly at create, so a fresh space can opt in during onboarding without waiting for the sweep. - **Multi-box placement** β€” a cloud front choosing among several router boxes.