# 🤖 Federated blob storage — providers contribute buckets, not trust

**Status: designed 2026-07-15, Phase A ready to build.** The plan of record
for taking attachment storage from one bucket (#550 Phase 1, the s3BlobStore
merged as #987) to many providers — the storage twin of the co-op compute
model (`docs/planet-scale.md` Part 10). Companion decisions this leans on:
the durability ladder (`tier-economics.md § Durability per tier`) and the
E2EE rule (`README.md` / `CLAUDE.md` — the whole design only works because
of it).

## Why storage is the easy half of planet-scale

Hibernation needed leases, heartbeats, and a mesh because a SQLite database
is **mutable with exactly one legitimate writer** — every hard problem in
that design flows from that fact. Attachment blobs are the opposite:
**write-once ciphertext under server-minted ids**. Nothing is ever rewritten,
so there are no conflicts, no single-writer lock, no consensus, and no
ordering. Mirroring immutable objects is order-free and idempotent — a copy
sweep can crash anywhere and rerun from the top. The distributed-systems hard
part evaporates; what remains is sweeps and bookkeeping, which is exactly the
shape this codebase already trusts (the meter sweep, the quota push, the
reaper).

## The trust inversion E2EE buys

A storage provider holds ciphertext under opaque ids. They can read
*nothing* — confidentiality is not a trust axis at all. What a provider is
actually trusted with:

- **Availability** — they might vanish. Answer: a second copy on a second
  provider, plus a sweep that notices and re-mirrors. A vanished provider
  costs a re-copy, never data.
- **Integrity** — they might corrupt or withhold. Answer: every blob's
  ciphertext hash lives in the instance's `blobs` table (recorded at
  upload); reads can verify, and an **audit sweep** samples ranged reads per
  provider and checks them. A cheating provider is *caught*, published, and
  evicted — trust-but-verify with teeth.

Two leaks remain, and the design closes both:

- **Access patterns + member IPs**: a provider serving members directly
  (presigned URLs) would learn who fetches what, when, from where. So
  co-op-provider blobs are served **proxied through the box, never
  presigned** — providers see the fleet's IP and nothing about members.
  (First-party stores may use presigned URLs later; that's a separate,
  deliberate choice per store.)
- **Sizes and timing**: unavoidable for any object store, identical to what
  the server itself already sees. Nothing new is exposed.

## The model, kept boring

**One wire dialect: S3.** It is the lingua franca — R2, Hetzner Object
Storage, and AWS speak it natively, and MinIO/Garage turn any box with disks
into it. A storage contribution is therefore four values: endpoint, bucket,
key pair, region. No custom protocol, no agent, no inbound port on the
provider's side. The s3BlobStore (#987) already consumes exactly these.

**A store registry**, beside the route registry: one row per store — name,
endpoint, bucket, region, provider, declared capacity, standing
(`primary-eligible` / `mirror-only` / `draining`). Tens of rows, not
thousands; a table a human reads whole.

**Placement per space**: each space's row records which store holds its
primary and which its mirror — the same per-space shape as `Route.Provider`
(the co-op seam that has been waiting in the schema since #550 Part 10).
Placement policy stays legible: region affinity first, capacity second,
never more clever than a sentence can state.

**Two sweeps**:

- **Mirror sweep**: the `blobs` table enumerates every id a space owns; copy
  whatever the mirror store lacks. Idempotent, restartable, journal-free —
  the table *is* the journal.
- **Audit sweep**: per store, sample a few blobs, ranged-GET a window,
  verify against the stored hash. Probabilistic coverage over time at
  trivial request cost — the same philosophy as tamper-evidence inclusion
  sampling (#573). Results feed the status pages: provider standing is
  public, like everything else about how this fleet runs.

**Reads**: primary first, then mirror, then the local-disk fallback the
s3BlobStore already has. A store outage degrades to slower reads, never
errors, and the audit sweep turns persistent failures into an eviction.

**Migration = copy + flip.** Because stores are dumb ciphertext buckets
keyed by id, moving a space between providers is: mirror to the new store,
flip the placement row, drain the old. The same no-lock-in shape as instance
migration — a provider leaving the co-op is an afternoon's sweep, not an
event.

## One ladder, two knobs

The durability ladder (`stream`/`batched`/`daily`) governs how fast a
space's *messages* reach the replica. **Copy count is the same product shape
for media**: how many providers hold a space's blobs. One member-facing
story — *your plan's durability covers your messages and your media* — and
plausible rungs (final numbers belong to a pricing session, not this doc):

| Tier | Message replication | Blob copies |
|---|---|---|
| Free | daily | 1 |
| Family / Community | daily / batched | 2 |
| Business | stream | 2, region-diverse |

## Contribution accounting

A provider's contribution is meterable with nothing but the registry and the
audit sweep: **bytes-months held × audited availability**. Both numbers are
publishable (the books, `tier-economics.md`, status pages) — the storage twin
of a co-op member contributing a compute box, credited in whatever form the
Part 10 economics settle on. No usage-secret accounting: the fleet publishes
what it measures, providers can dispute with their own logs.

**Onboarding ratchet**: a new provider's store joins as `mirror-only` and
carries second copies through an audit burn-in (weeks, not hours). Only
stores that survive it become primary-eligible — a hobbyist NAS strengthens
the pool without ever being load-bearing alone. Standing is a ratchet the
audit sweep can lower but only time can raise.

## Phases

- **Phase A — two stores, one owner (ready to build).** Multi-store under
  fleet control: a `mirror` store config beside the primary, the ciphertext
  hash column + recording at upload, the mirror sweep, read fallback
  primary→mirror→disk, and the audit sweep's first form (verify-on-read
  counters). Ships dark behind env like #987. This alone closes the
  vendor-concentration exposure (Hetzner primary + the existing AWS bucket
  as mirror) and exercises every rail federation needs.
- **Phase B — the registry.** Store rows + per-space placement + placement
  at provision (cloud push, like the replication class), migration sweep
  (copy + flip + drain), audit results to the status pages.
- **Phase C — contribution.** Provider onboarding (four values + burn-in),
  standing lifecycle, contribution metering into the books, proxy-only
  serving enforced for co-op stores.

## Non-goals, deliberately

- **DB replicas stay first-party.** Small, critical, mutable,
  lease-guarded — federating them buys pennies and costs the crown jewels'
  simplicity. Media is where the volume, cost, and federation win live.
- **No erasure coding.** Two full copies are legible, cheap at object-store
  prices, and degrade gracefully. Erasure across heterogeneous providers is
  clever; the second rule (`CLAUDE.md`) says legibility wins.
- **No DHT, no consensus, no gossip protocol for placement.** Placement is
  rows in a registry a human reads. The fleet is tens of boxes and tens of
  stores for the foreseeable future; designs for millions can wait for
  millions.
- **No custom provider daemon.** If a provider must run something, it's
  MinIO/Garage — maintained by people whose whole job that is. Contribution
  stays four values.

## Decisions needed before Phase A flips live

1. **Enable Hetzner Object Storage** on the fleet account (primary store;
   ~€4.9/TB-mo, egress €1/TB, same DC as the box) — one console action +
   key pair into the vault.
2. **Confirm the mirror target**: the existing AWS account (a dedicated
   bucket; storage-only cost, writes incur no egress) is the default
   assumption.
3. Pack pricing and pay-flips-paid (already on Paul's desk) — independent,
   but the same pricing session should set the blob copy-count rungs above.