# 🤖 Sealed search index — plan for message search at business scale

**Status: designed 2026-07-14, not built.** This documents the agreed upgrade
path for client-side message search before large (Business-tier, ~1 TB
allowance) spaces make today's design hurt. Nothing here is urgent yet — the
triggers below say when to build it. Companion decisions this plan leans on:
the storage/durability ladder (`tier-economics.md § Durability per tier`) and
the R2 attachment offload (#550), which keeps media out of the search path
entirely.

## Where search stands today

Search is **local on every platform** — the query never leaves the device,
and the server never gains a search surface. That's a consequence of the E2EE
rule, not a feature toggle, and this plan does not bend it.

- **Web** (#586 + the #228 "time machine"): the local store (`store.js`)
  holds raw **ciphertext** rows in IndexedDB; `search.js`'s `SearchIndex`
  decrypts each thread **once per session into an in-memory index** (never
  persisted), and queries are substring scans over it. "Search earlier
  messages" walks incomplete threads' history into the store (`runBackfill`)
  and streams new hits in. The design's own comments accept the limits and
  say "revisit if a real space ever makes this hurt".
- **iOS / Mac**: the App-Group `MessageStore` keeps **decrypted payloads** in
  SQLite with an **FTS5** twin table (`messages_fts`); `MessageStore.search`
  is a `MATCH` join, newest-first, limit 50. At-rest protection is the OS's
  (file-level Data Protection), which is the platform-normal posture —
  Signal and WhatsApp do the same.

Two honest asymmetries fall out of that, worth naming because this plan
resolves one and keeps the other:

1. **Semantics differ**: web matches *substrings* (`matchText` uses
   `includes`), native matches FTS5 *word/prefix terms*. Searching "ana"
   finds "banana" on web and not on iOS. This plan aligns both on
   word-prefix (below).
2. **At-rest posture differs**: native stores plaintext under OS file
   encryption; web deliberately stores only ciphertext. This plan **keeps**
   web's stricter posture — that's what "sealed" means.

## Why web search will hurt, and when

The in-memory index costs one full-store decrypt per tab session plus RAM
proportional to the whole corpus's text. Media never enters the path (the
store caches message rows only; attachment bytes fetch on demand), so the
problem scales with **history length, not storage bytes**:

| Space | Rows on device | First search (est.) | Index RAM (est.) |
|---|---|---|---|
| Family, years | ~50k | ~1–2 s | ~10 MB |
| Community, years | ~300k | ~10 s | ~50 MB |
| Business, a decade | ~2M | minutes | 300+ MB |

A phone browser will evict or kill the tab well before the last row. The
scan is fine to roughly **100–200k rows per device**; beyond that every
session pays the full decrypt again.

**Triggers to build this** (client-side heuristics, no analytics — measure
locally and log to console): a device's store crosses ~150k rows, or a
measured first-search ensure pass exceeds ~3 s. Either observed in a real
space = schedule the work.

## Design: a sealed, disposable index beside the ciphertext store

One sentence: **persist the derived index the session index already builds,
encrypted at rest under a device-local key, and query it via postings so a
search decrypts a few shards and a handful of candidate rows instead of the
whole store.**

### Invariants (the non-negotiables)

- **Never plaintext at rest in web storage.** Index shards are AES-GCM
  ciphertext in IndexedDB, sealed under a **device-local, non-extractable
  WebCrypto key** stored alongside (structured-clone of a `CryptoKey`).
  `store.js`'s ciphertext-only rule survives intact.
- **The index is a cache, never truth.** It is derived from the ciphertext
  store, deletable at any time, rebuilt lazily from the store. Schema-version
  mismatch → drop and rebuild. Sign-out → delete with everything else.
- **Postings are hints.** Every hit re-decrypts its row from the store and
  re-verifies the match before display — so stale postings from edits,
  deletes, and tombstones fall out silently rather than needing synchronous
  index maintenance.
- **Per-device coverage, honestly stated.** The index covers what this
  device's store holds; the existing backfill narration ("searching back to
  2024…") already sets that expectation and stays.
- **No dependencies, no build step.** Tokenization via the built-in
  `Intl.Segmenter`; everything else is plain ES modules. (Running the
  indexer in a module Worker to keep the UI responsive is an implementation
  freedom, not a requirement.)

### Shape

- **Stores** (new object stores in the existing `lchat-store` DB): 
  `idx-shards` (sealed term-shard blobs), `idx-meta` (per-thread indexed
  interval + schema version), `idx-key` (the CryptoKey).
- **Tokenization**: `Intl.Segmenter` word segmentation, lowercased; message
  text plus sender name. **Query semantics become word-prefix AND** — each
  query term must prefix-match an indexed term. This matches native FTS5,
  fixing asymmetry #1, and is what every mainstream chat app does. (The
  substring-parity alternative — a trigram index — was rejected: ~3× the
  size for a semantics nobody expects.)
- **Shards**: terms bucketed by normalized prefix, target ~128–512 KB per
  shard sealed; postings are `(threadId, messageId)` pairs, delta-encoded.
  Sizing: ~2M messages × ~12 tokens ≈ 24M postings ≈ well under 100 MB
  sealed — comfortable IndexedDB territory.
- **Write path — piggyback on decryption that already happens**: live
  receive, thread render, `SearchIndex.ensureThread`, and `runBackfill`'s
  `onPage` hook (whose docstring already reserves it "for incremental
  indexing"). A buffered writer batches term→posting appends and
  read-modify-writes sealed shards. Per-thread coverage is tracked as an
  **interval** (`[minIndexed, maxIndexed]`) rather than the session index's
  seen-`Set` — backfill extends it downward, live traffic upward. (Same
  known gap-above caveat as the store's backfill watermark; same acceptance.)
- **Query path**: segment the query → decrypt the few shards its term
  prefixes land in → intersect postings (AND) → decrypt candidate rows only,
  newest-first, limit 50 → verify with the real matcher → render through the
  existing `#msg-results` scaffold and jump. First search on a warm index:
  milliseconds of shard decrypt plus tens of row decrypts.
- **Compaction**: lazy and occasional — rewrite shards dropping postings the
  verify step has seen dead (deleted/edited/left threads). Nothing correct
  depends on it; it only reclaims space, because hints are re-verified.

### What sealing buys, and what it doesn't

Sealing keeps derived plaintext (a term list is readable content!) out of
casual reach: disk images, sync'd browser-profile backups, and forensic
strolls through IndexedDB see only ciphertext — the same at-rest level as
the message store beside it. It does **not** defend against an attacker who
controls the same origin at runtime (XSS) or who lifts the whole profile and
runs the browser — but neither does anything else in the client: the user's
private key and wrapped thread keys live in the same origin storage. The
sealed index adds **no new exposure class**; that's the bar it has to meet,
and the reason a plaintext web index (native-style) was rejected — browser
storage lacks the OS file-protection story App-Group SQLite gets.

## Phases

1. **P0 — semantics + measurement** (small, anytime): align web `matchText`
   on word-prefix AND (shared test vectors web ↔ native so the tokenizer/
   match contract is pinned the way event vectors pin folds), and add the
   local first-search timing log that arms the trigger.
2. **P1 — the sealed index** (the feature): stores, writer, query path,
   rebuild-on-version-mismatch. Scan remains the fallback for anything the
   index doesn't cover yet.
3. **P2 — compaction + polish**: lazy shard compaction, leave-thread
   cleanup, coverage surfacing in the results footer.
4. **P3 — richer queries** (optional, later): attachment-filename terms
   (filenames ride inside encrypted payloads, so they index client-side like
   any other text), `from:` / `in:` filters (the entry fields already
   exist), quoted phrases (positional postings — only if wanted).

## Rejected

- **Any server-side search**, including searchable-encryption schemes —
  they leak query and access patterns to exactly the party the product
  promises can see nothing. Off the table permanently.
- **Trigram/substring index** — 3× size for unfamiliar semantics.
- **Cross-device index sync** — each device derives its own from the store
  it already syncs; shipping index blobs around adds a surface for zero
  user-visible gain.
- **Plaintext index on web** — see the sealing analysis above.