# Request references: one ID a customer can read out, and we can find

**Date:** 2026-09-24
**Status:** approved (Bill, 2026-09-24). Nothing here is built. Every
choice is marked **(decision)** and can be overturned cheaply; the four in
§8 are Bill's **(ruling)**s of 2026-09-24. Builds on
`2026-09-07-errors-and-analytics-design.md` (§3.2 and §11, Availability).
Line numbers cite `main` at `38639c1e`. Reviewed by Fable
(approve-with-amendments); §10 records what it changed.

## 1. The idea

When something goes wrong for a customer, the support call today starts
with "what time was it, roughly, and what were you doing?" The Errors and
Availability tabs group failures well, but nothing links one customer's
failed request to one row.

The fix most services converge on is a request reference: a short ID
minted where the request enters, handed to the app, shown on every error
page, and stored with the error. The customer reads it out, support types
`carlos errors --app titogo --ref R-0K7M-3QXA-9F2C`, and the answer is the
group, the occurrence, and the log lines around it.

CARLOS is well placed to do this, because the edge is where every request
enters. The app does not need to mint anything. It only reads a header and
prints it on its own error page.

## 2. What is already right, and must stay right

- **The edge's wrapper order is load-bearing** (AGENTS.md). Minting a
  reference is a header write near the top of `ServeHTTP`. It moves no
  wrapper.
- **Nothing the platform does for itself may keep an instance awake.**
  Minting is CPU only, on a path every request already takes.
- **Hop trust is proven, never assumed.** `acceptHop`
  (`internal/edge/homemap.go:767`) verifies a backhaul hop before the
  receiving edge believes anything it carries, and `stripForwardedHeaders`
  discards the claims of an unproven one. A reference arriving from outside
  is a claim like any other.
- **Coalescing keeps one sample per group.** The errors sink folds every
  occurrence of `(fp, host, release)` between two ship ticks into one
  record carrying the *first* occurrence's details (§3.2 of the errors
  spec). The tick is 10 s (`errshipTick`, `internal/host/errship.go:22`);
  the errors spec's "five minutes" describes a design the ship loop never
  used. That is what keeps an error storm cheap. It also means a
  reference cannot simply be "the occurrence's": most occurrences have no
  record of their own. §3.4 is the answer.
- **Cached responses replay stored headers.** `writeCached`
  (`internal/edge/edge.go:1897`) copies the stored header map onto the
  response wholesale. A reference captured into the cache would be
  replayed to every later visitor, so the reference must never be stored
  there (§3.2).
- **Response headers set early do not survive.** The gzip writer keeps a
  private header map and, at its final `WriteHeader`, deletes every key on
  the real writer's map before copying its own down
  (`internal/edge/compress.go:211-245`). `httputil.ReverseProxy` clears
  the handler-visible map after a 103 Early Hints, and copies upstream
  headers with `Add`, not `Set`. A response header set at the top of
  `ServeHTTP` is therefore wiped on most responses, or duplicated when the
  app echoes its own (§3.2).

## 3. The change

### 3.1 The reference

**Format (decision, amended in review).** `R-` followed by twelve Crockford
base32 characters, shown in three groups of four: `R-0K7M-3QXA-9F2C`. The
60 bits are:

- 30 bits: seconds since 2026-01-01T00:00:00Z, which lasts until 2060;
- 30 bits: random.

Crockford base32 has no I, L, O or U, and decoding is case-insensitive
and maps those look-alikes to 1 and 0. That makes the reference safe to
read aloud, type from a screenshot, or copy with the wrong case.
Uniqueness only has to hold within one second, and the honest measure is
the birthday bound, not the size of the space. With n requests in one
second the chance of any collision is about n²/2^31. At 100 requests a
second that is about one pair every two or three days. At 1,000 a second
it is about 40 a day. The first draft's 28 random bits gave four times that. A collision
only matters when both requests fail and one is looked up; lookup then
shows both occurrences rather than guessing (§3.5). The random half comes
from `math/rand/v2`. The reference is an identifier, not a secret, and
nothing authorises on it.

**Why the time is inside it (decision).** Lookup must not need an index.
The reference decodes to the second it was minted, so the console reads
one narrow window of chunks and never scans the day (§3.5). The reference
does not encode the box or the app. The window across every box of one
app is small, and a box label in a customer-facing string would be
infrastructure detail leaking into support conversations.

### 3.2 Where it is minted, and who may carry one in

**Minted once, at the first CARLOS edge the request reaches (decision).**
In `ServeHTTP`, immediately after `acceptHop` and before the route
lookup, so the backhaul and tunnel branches below carry it on:

- a request on a **verified hop** (`hopVerified`) keeps the reference the
  sending edge minted;
- a request that arrived **over this box's own tunnel client** keeps the
  front box's reference. Today nothing marks such a request: the tunnel
  client serves the edge as a plain handler
  (`internal/host/tunnel_client.go:53`). The flag is set by a wrapper that
  lives in `internal/edge` (an exported `TunnelArrived(h)`), so the edge's
  own tunnel e2e test drives the same handler production runs. The
  channel is one this box dialled with TLS verification
  (`internal/tunnelclient/tunnelclient.go:207`), so the hub is the only
  possible sender. The two trusted marks never coexist: the front box
  deletes hop headers before it proxies over the tunnel;
- **on both trusted paths the inbound value is kept only if it parses as
  a reference** (`R-` and twelve Crockford characters); otherwise a new
  one is minted. During a rollout the front box may be on an older build
  that neither mints nor strips, and without this check a client's own
  `X-Request-Id` would pass through it and be trusted by the new box
  behind it;
- **every other request gets a new reference**, and any `X-Request-Id` the
  client sent is deleted first. A client-supplied value would let anyone
  plant a reference that collides with a real one or points support at the
  wrong occurrence.

**Carried as `X-Request-Id` on the inbound request (decision).** The mint
site sets it on `r.Header`, and that one value is the carrier everywhere:
`httputil.ReverseProxy` clones inbound headers to the outbound request,
so edges and the app receive it with no Rewrite change; the activator's
wake 503s, the static handler and the pending page read it from the same
request. It is the name Rails' `ActionDispatch::RequestId`, most Go
middleware, and most log shippers already read, so a Rails app gets it
into its tagged logs and into `request.request_id` with no change.

**Returned on every response (ruling, §8.1).** The edge sets
`X-Request-Id` on the response too, so it shows in browser dev tools and
in any `curl -I`. The value is the edge's reference, replacing anything
the upstream set. An app echoing its own ID back does not get to change
the one support will search for.

Where it is set matters (§2). It is set with `Header().Set` in
`countingWriter.WriteHeader` (`internal/edge/usagemeter.go:80`) for any
status of 200 or more. The counting writer is the bottom wrapper, so this
runs after the gzip writer has synced its private map and after
`ReverseProxy` has added the upstream's headers: the value replaces
rather than duplicates. **No new wrapper is added** and none moves. Two
cases fall outside it:

- **WebSocket upgrades.** `ReverseProxy` writes a 101's headers straight
  to the hijacked connection and never calls `WriteHeader`. For an
  upgrade request the edge sets the header on `w.Header()` just before
  proxying. If the app also sets one, the 101 carries both; this is
  accepted and documented rather than worked around.
- **Paths that answer before the counting writer exists**: the steering
  probe, the tunnel hub and store door, the stale-hop refusal, the
  backhaul sender's own 502/503, the unknown-host answer and the pending
  page. No gzip writer wraps these, so each sets the header directly.

**Never cached.** `maybeStore` deletes `X-Request-Id` from the header map
it clones (`internal/edge/edge.go:1720`). That covers both shapes: with
gzip on the clone is the gzip writer's private map, with it off the real
one. `writeCached` needs no change, because the counting writer's
`WriteHeader` sets the current request's reference after the replay. A
test pins that two requests served from one cache entry carry different
references.

### 3.3 Shown on every error page the edge writes

Every status the edge produces itself says the reference:

- the 502s written by the proxy error handlers;
- the activator's wake-path 503s;
- the fleet-dark and no-healthy-upstream 503s;
- the static-site 502;
- the pending-host retry page (`internal/edge/pending.go`).

The plain-text bodies gain one line, `Reference: R-0K7M-3QXA-9F2C`. The
HTML retry page shows it in small type under its message. That page is
rendered once today (`e.pendingBody`, `pending.go:137,186`) and becomes
per-request; the plan checks `pending_script_test.go` for anything that
pins its bytes or its script hash. The format is
fixed so a support person can recognise it on a screenshot.

The app's own error pages are the app's. The platform's part is the
header. The getting-started skill and `docs/site` get a paragraph on
showing `X-Request-Id` on a 500 page, with Rails and Go examples.

### 3.4 Stored with the error

**`refs` on the errors record (decision).** `apperr.Record` gains
`Refs []string` (`omitempty`, additive). The sink appends the reference of
each occurrence it folds into a record, up to `MaxRefs = 20` per record
per ship tick. `Refs` holds the first twenty, in order. `Count` already
says how many there were. The sink clears `Refs` on the `_overflow` row
it builds from the first excess record (`sink.go:104-110` copies the
whole record), so the overflow row never carries any.

**A ref names the group, not the occurrence's page.** A folded occurrence
only increments `Count` and moves `Last` (`sink.go:96-100`). Host and
release are exact, because they are part of the group's key. Path,
status and fields are the group's *first* sample. On Availability, where
groups are fingerprinted by cause and not by path, the fifteenth ref's
page may differ from the sample's. Lookup says so (§3.5). Storing a path
beside each ref is an additive later step if support finds it wants one.

Twenty is enough for the support case, a person who hit an error and
called. It keeps the record's size bounded in an error storm: 20 × 16
bytes, against a stack that is already capped at 8 KB. A storm is not
the case where one reference matters: a thousand customers hit the same
thing, and the group itself is the answer.

**A reference past the cap is still findable, less precisely.** Lookup
(§3.5) falls back to what the reference's time says: the groups on that
app with an occurrence overlapping that second, labelled as such.

**Every source carries it.**

- `edge`: the observing defer already has the request, so it reads the
  reference from the request.
- `log`: an app that logs its `X-Request-Id` has the reference in the line
  itself. Lookup greps the window for it.
- `server` (rollout step 4 of the errors spec, not built): the verb gains a
  `ref` field; the app passes the header's value.
- `browser` (step 3, not built): out of scope here. A page cannot read its
  own response headers. The snippet would need the reference rendered
  into the page, which is the app's to do.

### 3.5 Looking one up

**`carlos errors --app <a> --ref <R-…>` (decision)**, and a Reference box
on both the Errors and Availability tabs, answered by one function
through the API as `?ref=` on both paths:

1. Decode the reference to its second `t`. A malformed reference is a 400
   naming the expected shape. A reference from the future, or from before
   the app existed, says so rather than searching nothing.
2. Read both families over `[t − 1 min, t + 30 min]`. The failure is
   filed when the request *ends*, not when it was minted: a dead socket is
   redialled for 13 s, a backhaul waits up to 30 s for headers, and an
   abandoned wait on a wedged instance can run for minutes. The listing
   prunes by hour (`internal/applog/read.go`), so a wide window inside
   one hour costs nothing extra.
3. **Exact:** records whose `Refs` contains the reference. The answer is
   the group, with that occurrence's host and release, and the group's
   first sample's path and status, labelled as the sample (§3.4). It says
   which tab the group lives on. Two exact matches (a collision, §3.1)
   are both shown.
4. **Near:** when nothing matched exactly, the groups with an occurrence
   overlapping `[t, t + 60 s]` on that app, plainly labelled "this
   reference was not stored individually; these failed within a minute
   of it". The minute covers the dial retry and the abandoned grace.
5. **Logs:** lines in `[t − 5 s, t + 60 s]` whose message or any field
   value contains the reference. `applog.Filter.Grep` matches the message
   only, and `Filter.Fields` is exact equality on a named key, so this
   needs a new any-field substring match; JSON log lines put
   `request_id` in fields, tagged plain-text lines put it in the message.
   The read decodes every chunk in the hour, so this costs what
   `carlos logs --since 1h` costs for the app.
6. **Not shipped yet:** when `t` is within about a minute of now, or the
   app's newest chunk is older than `t`, the answer is "this reference is
   too recent to have been shipped; try again in a minute". Boxes ship
   errors every 10 s and hold a failed upload for the next tick.
7. **Nothing:** "no error or log line carries this reference". This is
   also the honest answer for a request that succeeded but that the
   customer thought was slow, and support should hear it plainly.

App-scoped in v1: support knows which app the customer was using. A
cross-app lookup needs an account-level door with its own gate (§6).

## 4. Nothing here wakes an instance

Minting and header writes happen on paths every request already takes.
Lookup is a console read over chunks the agent already ships. Nothing
new touches the activator ahead of the cache check. The tunnel
flag is one context value.

## 5. Wire shapes

- `X-Request-Id` on requests to apps is new traffic, not a changed shape.
  An app that already sets its own ID from a missing header keeps doing
  so; an app that trusted a client-supplied `X-Request-Id` now sees the
  edge's instead.
- `refs` on the errors record is `omitempty` and additive.
- A box on an older build mints nothing and stores no refs. Lookup falls
  back to "near" for its occurrences, and the answer says so.

## 6. What is not built

- **A cross-app lookup.** An account-level `carlos errors --ref` with no
  `--app` means reading every app's window, and needs its own gate.
- **Honouring a customer's own proxy's ID** (for example a CDN's ray ID).
  Ruled out for now (§8.2).
- **An edge access log.** `applog.StreamEdge` is reserved and unbuilt. With
  it, every request, successful or not, would be findable by reference.
  That is the natural next step, and it is a different cost decision
  (storage per request, not per failure).

## 7. Rollout

1. **Mint and carry.** The edge mints, strips, carries across hops and the
   tunnel, sets the request and response headers, and keeps the reference
   out of the cache. Its own error pages show it. Useful on its own:
   support can already ask for the reference and grep logs for it.
2. **Store and look up.** `Refs` on the record, `?ref=` on both APIs,
   `--ref` on `carlos errors` and `carlos availability`, the Reference box
   on both tabs. Docs.
3. **Apps show it.** titogo prints `X-Request-Id` on its 500 page. That
   change is in titogo, not here; the docs paragraph in step 2 is the
   recipe.

Each step is its own branch. Step 1 touches the edge serving path and gets
the Opus-then-Fable review AGENTS.md asks for.

## 8. Rulings (Bill, 2026-09-24)

Four questions were open in the draft; Bill answered all four.

1. **The reference is returned on every response (ruling)**, not only on
   errors. It costs one header, and it answers "which request was the slow
   one" from dev tools, including for a request that succeeded.
2. **A client-supplied `X-Request-Id` is ignored (ruling).** It is deleted
   before the edge mints its own, and not kept. Keeping it as a
   `client_id` field, so a customer behind their own CDN can correlate the
   two IDs, stays an additive later step (§6).
3. **`MaxRefs = 20` (ruling).**
4. **The `R-` prefix and the 4-4-4 grouping stand (ruling).**

## 9. Testing

- **Format:** round-trip encode and decode; case-insensitive decode;
  look-alike characters (I→1, L→1, O→0); rejection of wrong lengths and
  prefixes; the time half survives the round trip to the second.
- **Minting:** a client-supplied `X-Request-Id` never reaches the app; a
  verified hop's reference survives to the home box; an unverified hop's
  does not; a tunnel-arrived request keeps the front box's; every other
  request gets a fresh one.
- **Cache:** two hits on one entry carry different references, and the
  stored entry holds none.
- **Error pages:** every edge-written 502/503/504 body names the reference
  the response header carries.
- **Storage:** twenty-one occurrences in one tick store twenty refs and a
  count of twenty-one; the overflow row carries none.
- **Response header:** present and single-valued with gzip on and off,
  after a 103 Early Hints, when the app echoes its own `X-Request-Id`,
  and on each early-return path; on a 101 upgrade, present.
- **Trust:** a non-reference value on a verified hop or over the tunnel is
  replaced, not kept.
- **Lookup:** exact (including two matches), near, logs, not-shipped-yet
  and nothing, each against seeded chunks; a failure filed 13 s and 10
  minutes after its reference's second is still an exact match; the
  malformed and future cases.

## 10. Review round 1 (Fable, 2026-09-24)

Approve-with-amendments. Every amendment was taken:

- **Response header site.** The first draft set it at the mint site. The
  gzip writer, 103 handling and `Add` copying would have lost or
  duplicated it on most responses. It now lives in the counting writer's
  `WriteHeader`, with the upgrade and early-return cases named (§3.2).
- **One carrier.** `X-Request-Id` on the inbound request, not a
  `setForwarded` change (§3.2).
- **Mixed-version trust.** Inbound values on trusted paths must parse as
  references (§3.2).
- **Tunnel flag placement** in `internal/edge`, so the e2e test runs the
  production handler (§3.2).
- **Birthday bound.** Stated honestly, and the split moved from 32/28 to
  30/30. This is within Bill's format ruling, which fixed the prefix and
  grouping, not the bit split. Lookup shows collisions rather than
  guessing (§3.1, §3.5).
- **The ship tick is 10 s, not five minutes.** The window reasoning was
  rebuilt on request duration instead, which is what actually moves a
  record past its reference's second (§2, §3.5).
- **A ref names the group, not the page**, and lookup labels the sample
  as a sample (§3.4, §3.5).
- **Overflow row clears refs** (§3.4).
- **"Not shipped yet"** before "nothing" (§3.5).
- **Log matching** needs an any-field substring match, and its cost is
  stated (§3.5).
- **The cache drop** lives in `maybeStore` (§3.2); **the pending page**
  becomes per-request (§3.3).

Declined: nothing.