# 🤖 Tamper-evident hosting **Status: design.** This document settles the design questions left open in issue #573 (spun out of #550's provider-marketplace discussion) and defines the build order. The first slice — chained, signed checkpoints over an append-only event journal — is implemented; everything else here is committed design, not code. The one rule says: *if the server is compromised, the attacker gets nothing.* Today that holds for **confidentiality** — message content, names, and files are ciphertext under keys the server never sees. This work extends the rule to **integrity**: a malicious host (a compromised us, or any third-party provider we later delegate spaces to) must not be able to silently rewrite membership or history. "Silently" is the load-bearing word — a host can always destroy or withhold data; the goal is that *every* such act leaves evidence a client or monitor can catch, and where possible **prove** to a third party. ## What a malicious host can do today 1. **Attack membership** (the key-distribution MITM). A joiner's new public key travels through the server unauthenticated: the invite secret `T` authenticates the *server → joiner* direction (the wrapped key blob only opens under a `T`-derived key), but nothing in the *joiner → members* direction binds the published public key to `T`. The host can substitute its own key as "the new member," receive wrapped thread keys from honest members, and re-wrap to the real joiner — a full read of every thread the fake member is wrapped into. 2. **Roll back.** Serve an old DB snapshot. Clients that were online notice their own messages missing; clients that were offline can't tell. 3. **Withhold or reorder** (bounded nuisance): drop or delay individual events. Clients notice gaps against their own state but can't prove misbehaviour. Three mechanisms close these, in dependency order: | Mechanism | Closes | Depends on | |---|---|---| | Signed checkpoint chain (this slice) | 2 becomes provable | nothing | | Key transparency log | 1 | checkpoint chain, commons log | | Client sampling verification | 3 detectable; escrow conformance | checkpoint chain, restore-read | --- ## Part 1 — the event journal and the checkpoint chain (built) ### Why a journal: the message table is not append-only, on purpose Edits are in-place `UPDATE`s and unsend is a hard `DELETE` — the ciphertext is really gone, which is the product promise. A Merkle tree over the *current* `messages` rows would therefore change under legitimate edits and unsends, and "legitimate mutation" would be indistinguishable from "host rewrote history." So the unit of commitment is not table state but an **append-only event journal**: every insert, edit, and delete appends one journal row in the same transaction. The journal stores **hashes of ciphertext, never ciphertext** — unsend still destroys the blob; what the journal retains is that *an* event happened, not what it said. ``` event_journal( seq INTEGER PRIMARY KEY AUTOINCREMENT, -- total order, never reused ts INTEGER, -- unix seconds, server clock kind TEXT, -- 'msg' | 'edit' | 'delete' (roster kinds join later) msg_id INTEGER, -- the messages.id this event concerns thread_id TEXT, blob_hash TEXT -- sha256 hex of the ciphertext ('' for delete) ) ``` **Privacy accounting.** Every journal column is metadata the server already holds (it stores the blobs and serves them by thread and id). The one *new* retention is temporal: after an unsend, the server previously forgot the message ever existed; now a hash-shaped fossil (`msg_id`, `thread_id`, `ts`, `blob_hash`) remains. That is inherent to tamper evidence — you cannot make deletion verifiable while also forgetting it happened — and it reveals nothing about content: a SHA-256 of AES-GCM ciphertext is doubly opaque. This is the same bargain Certificate Transparency makes, and it is why the journal must never grow a plaintext column. ### Checkpoints A background sweep (same shape as the mini-app scheduler: a `time.Tick` loop delegating to a pure, test-callable unit of work) periodically commits to all new journal rows: - **Leaf hash**: `SHA-256("lchat-ckpt-leaf\n" ‖ canonical encoding of the journal row)`. Domain-separated so leaves can't be confused with interior nodes or other protocol hashes. - **Batch root**: an RFC 6962-style Merkle tree over the leaves `[prev.seq_to+1 … seq_to]`. Batching means each checkpoint is O(new events), not O(history), and inclusion proofs (a later slice) are proof-in-batch plus the chain to the head. - **Chain hash**: `SHA-256("lchat-ckpt-chain\n" ‖ prev_chain_hex ‖ root_hex ‖ seq_to)`. Each checkpoint commits to its predecessor, so two checkpoints presenting different histories for the same range are *cryptographic proof* of equivocation, and serving a state older than a head a client has seen is *provable* rollback — not just an anomaly. - **Signature**: Ed25519 over the base64 JSON payload string, exactly the `release.go` signed-manifest pattern (verify the signature over the encoded string first, decode after — no canonicalization to get wrong). Checkpoints live in a `checkpoints` table and are served, member-token-gated, at `GET /api/checkpoints` (cursorable, newest known to the caller onward). Member-gated rather than public because the chain leaks activity volume and timing; monitors hold member credentials anyway (see distribution, below). ### The signing key is not the security The per-instance Ed25519 seed lives **in the DB config by default**, minted on first use. That placement is deliberate and operational: every path that moves instance state — hibernation wake, `VACUUM INTO` migration, escrow restore — carries *only* the DB, and a key that silently changes on routine ops would make every legitimate restore read as forgery, poisoning the system with false alarms. The trade is that anyone holding a DB copy (including the escrow) can sign as the instance; we accept it, because no key location survives "the host is malicious" anyway — the host runs the process and holds the seed. **The security property is chain continuity plus external witnessing of heads** — a host (or a DB-copy holder) can forge a fresh chain, but it cannot make one that both (a) extends the head a client or the escrow already observed and (b) omits what that head committed to. The signature's job is modest: it makes a checkpoint attributable *given* a witnessed head, and stops anyone without the seed from fabricating checkpoints in the instance's name. Two supporting choices: every checkpoint payload **embeds its own public key**, so a key change is an explicit, investigable event rather than old checkpoints suddenly failing verification; and `LCHAT_CHECKPOINT_KEY_FILE` lets an operator keep the seed in a file outside the DB instead, owning the carry-it-across-restores burden themselves. ### What this slice deliberately does not do No inclusion proofs endpoint yet, no client-side verification yet, no roster events in the journal yet (that's key transparency's job, below), and no UI. It is the evidence *substrate*: from the moment it deploys, every instance is accumulating a committed history that later verifiers can hold it to. Deploying this early matters — evidence only covers events after the journal starts. Messages that predate the journal are **baselined** at migration (one `msg` row each, committing to the ciphertext as it stood at upgrade): the baseline can't retroactively prove the past, but without it, editing or unsending an old message would journal an event for a message the journal never saw — a built-in false alarm. --- ## Part 2 — the three open design questions, settled ### Q1: Log granularity — one commons log with sealed space identifiers **Decision: one global, commons-operated transparency log; spaces appear under sealed (HKDF-derived) identifiers. Per-space logs are rejected.** A per-space log kept by the space's own host adds nothing against that host — the adversary operates the log. The trust win comes precisely from the log living in a *different* trust domain than the space's host, which is the whole point once spaces can be hosted by third-party providers (#575): a provider can't equivocate about a roster without the commons log catching it, and a compromised commons log can't read anything — it holds only commitments. One global log also amortizes what actually costs effort: monitoring. CONIKS died and WhatsApp KT shipped largely on this axis — auditing one log is a commons-scale job; auditing ten thousand per-space logs is nobody's job. **Sealed identifiers.** A space appears in the log as `space_tag = HKDF(instance_secret, "lchat-kt-space-tag")` — stable within a space (members and monitors derive it; they hold the instance secret, which already exists as key material wrapped to every member) but meaningless to the log operator and unlinkable across spaces. Roster entries under a tag are the sealed shape of "key K joined / rotated / left," carrying no names — screen names are E2EE and stay that way. The log learns: some space, some volume of membership churn. It must never learn which space, or who. **What the instance's own chain (Part 1) is, then:** the per-space *history* commitment stays local-with-witnessing (it commits every message event — far too voluminous and too revealing for a global log). The global log carries only (a) roster/key events under sealed tags and (b) periodic instance chain heads, which is what makes local chains fork-evident. Two layers, one trust story: the instance chain says *what this host claims happened*; the commons log pins *which claim the host is stuck with*. ### Q2: The invite `#fragment` bootstraps the log entry — extends, never replaced **Decision: `T` stays the root of trust for admission; the log makes what `T` authorized auditable. Concretely, a join produces a log entry that carries a `T`-derived binding of the joiner's public key, verified by the inviter's client.** The mechanism mirrors the existing `deriveInvite` derivations (id, wrap key, claim secret — all HKDF domain-separated from `T`). Add one more: - `bind_key = HKDF(T, "lchat-invite-bind")`, held by whoever minted the link (their client already had `T`) and derivable by the joiner. - At claim time the joiner publishes, alongside its new public key, `bind_mac = HMAC(bind_key, joiner_pubkey)`. - The **inviter's client** — the one party besides the joiner who ever held `T` — verifies `bind_mac` against the *published roster key* before it (or anyone it vouches to) wraps a single thread key to the new member, and then signs the join's log entry as attester. This closes the substitution attack at its exact seam: a host that swaps the published key can't produce a valid `bind_mac` for its own key (it never held `T` — it stores only `claim_hash`), so the inviter's client refuses to wrap and the join fails loudly instead of succeeding silently. Subsequent lifecycle events are self-certifying without `T`: - **Key rotation / add-device**: signed by the key being replaced (continuity chain rooted in the `T`-bound original). - **Removal**: signed by an admin identity, itself a `T`-rooted chain (the setup invite is the genesis entry). - **Wrap discipline**: clients wrap thread keys only to keys whose lineage appears in the log — "verify before wrap" is the client-side rule that makes the log load-bearing rather than advisory. Bots and the eleven CLI follow the same path — they are ordinary members whose `T` arrives via the agent-invite flow; nothing special-cases them. **Why not replace the fragment with the log?** Because the fragment is the only *out-of-band* channel in the system — the one path the host never sees. A log can prove consistency (everyone sees the same roster) but cannot prove *legitimacy* (that a roster entry was authorized by a human who held the link). Consistency without an anchor just means the host lies to everyone identically. `T` supplies legitimacy; the log makes it globally consistent and equivocation provable. They compose; neither substitutes for the other. ### Q3: Checkpoint distribution — escrow first, member gossip second, commons log third **Decision: three witnessing channels, shipped in this order, each valuable alone.** 1. **Escrow replica (near-free, ships with Part 1).** The `checkpoints` table rides the existing Litestream replication to S3, and S3 versioning timestamps every generation of it — a target the *provider* doesn't control once escrow is commons-held (#550 Part 10). Honest history: monotone chain growth across replica generations. Rollback: a later generation whose head is an ancestor of an earlier generation's. The host could stop replicating, but going dark is itself the detectable event the escrow lease already surfaces. Limit: a fully self-hosted instance with no escrow gets nothing from this channel — hence channels 2 and 3. 2. **Member gossip inside E2EE traffic (the novel one, and ours to use: members already share an authenticated, host-opaque channel — the chat itself).** Clients periodically tuck the head they last verified (`seq_to`, `chain_hash`) inside their normal encrypted payloads; peers compare against their own view. A host that forks — serving member A one history and member B another — is betrayed by its own transport: it can't read the gossip to filter it, and dropping messages wholesale is exactly the withholding that members notice. Split-view detection with zero new infrastructure, working even for self-hosted spaces with no escrow and no commons. This is the distribution answer the issue was looking for. 3. **Commons log witnessing (with Part 3/KT).** Instances push heads under their sealed tag on a slow cadence; the commons log's own consistency proofs then pin every instance's chain globally. Strongest guarantee, biggest dependency — which is why it's last, not first. Ordering rationale: 1 is a table already replicating; 2 is a small client change with outsized reach; 3 requires operating a new commons service and should be built once, for KT, not twice. --- ## Part 3 — key transparency (design sketch, next after this slice) With Q1–Q3 settled, KT's shape is: roster/key events become journal kinds (`join`, `rotate`, `remove`) carrying the Q2 evidence (bind MACs, continuity signatures); the instance chain commits to them like any event; instances publish roster events + chain heads to the commons log under their sealed tag; clients enforce verify-before-wrap; monitors (any client, or the commons) audit for equivocation. The commons log itself is CT-shaped: append-only, Merkle, publicly auditable, dumb — it stores sealed commitments and serves proofs, nothing else. Detailed protocol (proof formats, monitor cadence, anti-flap for offline devices) is its own design pass **after** the journal substrate has soaked in production. It is deliberately not specified further here: the failure mode of KT designs is specifying the whole cathedral before the foundation has carried load. **Part 3a — roster/key journal events (built).** The first, safe increment: membership changes are journaled like message events, so they're committed to the signed checkpoint chain and a monitor catches a server rewriting the roster. A member's first key publish appends a `join` event; removal appends a `remove`. They reuse the journal's generic columns as the kind's detail record (`thread_id` = the member's user id, `blob_hash` = sha256 of the public key for `join` / empty for `remove`, `msg_id` unused) — the leaf encoding already covers these fields, so no wire-format change. Re-registering the same key (add-device) is not a new join. This touches no crypto and no join-path decision; it just makes the roster tamper-evident on the instance chain. **Part 3b — the bind MAC + verify-before-wrap (next, review-gated).** The security-critical increment that closes the key-substitution attack (Q2): `bind_key = HKDF(T, "lchat-invite-bind")`; the joiner publishes `bind_mac = HMAC(bind_key, joiner_pubkey)` at claim; the inviter's client verifies it against the published roster key before wrapping any thread key, and clients enforce verify-before-wrap generally. Because it changes the join/key-distribution path — the crown jewel of the one rule — it ships as its own PR with an adversarial review pass and a canary before it reaches any real space. ## Part 4 — client sampling verification (design sketch) Because clients hold plaintext truth, whole-DB verification needs no proofs-of-retrievability: each client periodically samples k message ids *from its own local store*, fetches those rows through a **restore-read endpoint over the escrow replica** (not the live host — the point is to check what escrow would restore), checks each blob decrypts and matches, and — once inclusion proofs land — proves each against a committed root. Independent sampling across members covers the DB probabilistically at trivial per-client cost, and the pass/fail signal doubles as the client-visible end of provider conformance, which #575's payment gating needs. Depends on: checkpoint chain (shipped), inclusion-proof endpoint, restore-read endpoint. Build after KT's journal extension, before the marketplace. ## Out of scope (unchanged from #573) Provider-served **web assets** remain the weakest trust point: a malicious host can serve JavaScript that exfiltrates keys, and no log catches that. Signed releases + native apps carry the answer for now; a Code Verify-style extension is the eventual one. Separate work, tracked in #550 Part 10. ## Build order 1. **Event journal + signed checkpoint chain** — this slice. Server-only, additive migration, zero client changes. Starts the evidence clock. 2. **Member gossip of chain heads** (Q3 channel 2) — small client change, makes forks member-detectable everywhere, no new services. 3. **Inclusion proofs endpoint** + verify-on-load in clients. 4. **KT journal kinds + bind MAC at claim + verify-before-wrap** (Q2) — the membership fix, still commons-free. 5. **Commons log + sealed tags + monitors** (Q1, Q3 channel 3) — the global equivocation pin. Gates #575. 6. **Restore-read + client sampling** (Part 4). Gates #575's payment conformance.