# πŸ€– Incoming webhooks: the ingest sidecar (`lchat hooks run`) Issue #610. External services (GitHub, Stripe, a greenhouse sensor) can't encrypt to a thread key β€” but their events belong in the product. This design lets an admin hand an outside service a URL such that every HTTP request it sends becomes an **encrypted, typed event** (`webhooks.`) in **one chosen thread** β€” with the server never seeing plaintext content and the receiving process never able to read anything back. ## The shape: an inverted bot sidecar The bot sidecar (`lchat bot run`, docs/bots.md) is a **pure outbound client**: it dials out, holds keys, and must never listen. The hooks sidecar (`lchat hooks run`, `internal/hooks/`) is its inverse β€” it exists to accept inbound connections, and is **strictly write-only** against the instance: - it never opens a websocket and never calls `GET /api/messages` β€” nothing in the process can read a conversation; - it never holds the instance key, so it can't decrypt profile blobs, member apps, charters, or anything else instance-key-sealed; - per webhook it holds exactly that webhook's private key and device token, which unwrap exactly **one** thread's key (to encrypt, and only encrypt). The two invariants stay unconditional by staying in separate processes: the bot sidecar remains "no inbound ports, relocatable anywhere"; the hooks sidecar is inbound-by-purpose and lives behind the router. Compromising the hooks process yields the threads its webhooks feed β€” never the rest of the space, never a bot's keys, never an Anthropic key (it has none). ## Reaching it: `in.`, a plain registry route Each instance's ingest surface is a sibling host β€” `in.family.11ch.at` for `family.11ch.at` β€” which is just a router registry row (`kind: "hooks"`, socket `/run/lchat/-hooks.sock`). No router code special-cases it: the router proxies any registered host to its socket. TLS/DNS: the `*.11ch.at` **wildcard certificate** covers only one label, so `in..11ch.at` takes the router's **per-host autocert** path instead β€” the cert is minted via HTTP-01 on first contact (HostPolicy allows it because the host is in the registry). Wildcard **DNS** does resolve two labels down when no closer record exists, so 11ch.at spaces need no zone change. (This is why the earlier `/hooks/*` path-split design β€” PR #617 β€” is superseded: the subdomain shape needs no router changes at all. The one real cost: a **custom-domain** space (`chat.paulca.com`) needs its owner to add an `in.chat.paulca.com` DNS record pointing at the same box.) The URL an outside service gets is `https://in./h/` β€” the secret is the capability, exactly the GitHub/Slack convention. The sidecar alone knows which secrets are live; the router forwards blindly and unknown secrets 404 **at the sidecar**, so the server can't even enumerate webhooks. ## One webhook = one agent identity = one thread A webhook is an ordinary **agent member** (roster-visible, labelled with the webhook's name, badged `webhook`), created by the admin's browser: 1. The admin opens the **Webhooks** app (admin-only, `/webhooks` or the Apps picker) and names a webhook + its **event namespace** (`github` β†’ events of type `webhooks.github`) + its **target thread** (a group they're in, or the space thread). 2. Their browser mints the webhook's **keypair**, wraps the target thread's key to it, and calls `POST /api/hooks` (admin-gated) β€” the server creates the agent user, mints its device token (returned once), and writes the membership row carrying the wrapped key. For the **space thread** this is the explicit-membership opt-in `threadMember` was built to honour: agents are fenced out of implicit membership and `open_key`, and a webhook gets the space key only because an admin deliberately wrapped it to that one identity. 3. The browser generates the URL **secret**, then seals `{id, user_id, name, ns, thread_id, secret, token, priv}` to the **hooks service identity's pubkey** (ECIES `sealForPub`) β†’ `def_blob`, and encrypts `{name, ns, thread_id, secret}` under the **instance key** β†’ `meta_blob` (the app's list view). `PUT /api/hooks/{id}/blobs` stores both. The server holds only ciphertext: no key, no token plaintext (tokens are stored hashed), no secret, no target service. The sidecar polls `GET /api/hooks/defs` (~10s, sidecar-marked), opens each definition with its service-identity private key, materializes its route table, and acks new webhooks via `POST /api/hooks/{id}/deployed` β€” a plaintext timestamp, the only thing the server learns, which the app shows as "Deploying… β†’ Active". Because the client minted the secret, the URL is displayed **instantly** at creation; the ack just confirms the route is serving. (#610's discussion had the sidecar minting the URL and feeding it back; client-minting is the deliberate simplification β€” a sidecar-minted URL would have to travel back encrypted, which needs the sidecar to hold the instance key, the very thing this design keeps away from it.) **Removal** (`DELETE /api/hooks/{id}`) deletes the row and sweeps the whole identity (`removeUser`: token, memberships, roster row); the sidecar's next poll drops the route and the URL dies. The **service identity** is provisioned once per instance by `lchat add-hooks ` (box-side, mirrors `add-bot`): it mints an agent invite flagged `hooks_service` β€” recorded in config as `hooks_service_user`, served to members at `GET /api/hooks/service` as the seal target β€” writes `/data/-hooks.env`, registers the `in.` route, and enables `lchat-hooks@`. The sidecar self-provisions from the invite on first boot (`bot.EnsureSession`, the #806 zero-touch path). It's exempt from the bots gate: webhooks are integrations with their own admin-only gate, not AI. ## The event: dumb ingest, smart render On each inbound request the sidecar builds (pinned by `test/event-vectors/webhooks.json`): ```json { "type": "webhooks.", "name": "", "uid": "", "ts": 1752710400000, "hook": { "id": "wh-…", "ns": "" }, "req": { "method": "POST", "path": "/", "query": { "site": ["7"] }, "headers": { "Content-Type": "…", "X-Github-Event": "…" }, "body": "", "body_b64": false } } ``` encrypts it to the thread key, and `POST /api/messages` as the webhook β€” the Announcer's outbound-only walk. The sidecar **never transforms**: the raw request rides inside the event forever, so rendering can improve later without re-ingesting. - **No default rendering.** Webhook events are consumed by *apps* β€” a CI dashboard app claims exactly `webhooks.github` in its `events` map and folds from there. Until an app claims a type, every client silently ignores it: web via `foldEvent`/`isAppType` (no owner β†’ dropped), native via `MessagePayload.isRenderableChat` (typed non-chat β†’ not rendered, not notified), the service worker via its typed-events-stay-silent rule. Old clients degrade to exactly this. The management app itself claims **no** types. - **Headers are allowlisted** (`Content-Type`, `User-Agent`, provider event/signature headers). Full headers routinely carry sender credentials (`Authorization`, cookies) that must never enter the log β€” even encrypted, every thread member could read them. `req.path` is the sub-path *after* the secret segment, so the capability never rides inside an event. - **Notifications:** space-thread deliveries never push at all (push.go skips `kind='space'`); group deliveries push ciphertext but render nothing (typed, unclaimed) β€” so a webhook can't buzz a phone until a consumer app decides events are worth words. ## Ingest hygiene - Unknown secret β†’ **404**; nothing distinguishes "never existed" from "removed". - Body over ~40 KB β†’ **413**, never truncated (the encrypted event must fit the server's 64 KB message-blob cap). - Per-webhook token bucket (60 burst, 1/s sustained) β†’ **429**; the server's content-blind message brake backstops it independently. - Non-UTF-8 bodies ride base64url with `body_b64: true`. - Delivery failure upstream β†’ **502**, so senders with retry logic retry. - Hibernation-aware: the recurring definition poll is sidecar-marked (never holds a sleeping instance awake; the last-known route table keeps serving), while an actual delivery is unmarked β€” a webhook arrival is real external activity and **wakes** the instance to land its event. ## Dev story `./tools/dev up` runs the whole loop locally, no router: it mints the service invite, spawns `lchat hooks run` on a sticky local TCP port (the sidecar's identity persists in `.lchat-dev-hooks.json`, exactly like the dev bot's), and passes the server `LCHAT_HOOKS_BASE=http://127.0.0.1:` β€” which the service endpoint echoes as `ingest_base`, so the Webhooks app displays URLs that actually work on your machine. Dev-you is seeded as an admin member, so the app is right there under Apps β†’ Webhooks; create one, wait for the "Active" chip (~10s poll), and `curl` the URL β€” the event lands encrypted in the chosen thread. `--no-hooks` opts out. Hand-run equivalent (any server): `lchat hooks run --addr :8090 --base http://localhost:1747` with `LCHAT_INVITE` from an agent invite minted with `hooks_service: true`, and `LCHAT_HOOKS_BASE=http://127.0.0.1:8090` on the server so displayed URLs match. In production leave `LCHAT_HOOKS_BASE` unset β€” clients derive `https://in.`. ## Later (deliberately not in v1) - **Member-mintable webhooks** with quotas (creation is admin-only today). - **Sender signature enforcement** (e.g. verify `X-Hub-Signature-256` at the sidecar with a per-webhook shared secret; reject unsigned). - **Consumer apps** β€” the point of the exercise: e.g. a GitHub Actions dashboard claiming `webhooks.github`, parsing `workflow_run` payloads into a live runs board. - **Multi-thread webhooks / re-targeting** β€” v1 binds one webhook to one thread at creation; re-point by remove + recreate.