# Stage 4a: a platform STUN and TURN relay

**Date:** 2026-09-07, revision 6
**Status:** rulings taken. Paul ruled on the six decisions in §9 on
2026-09-07, each marked **(ruling)**; the roadmap's stage 4a text and
gate are amended so the two documents agree. Reviewed five times by
GPT-6 via Codex: revision 1's 20 findings are in
`CODEX-FINDINGS-RELAY.md`, revision 2's 12 in
`CODEX-FINDINGS-RELAY-R2.md`, revision 3's 8 in
`CODEX-FINDINGS-RELAY-R3.md`, revision 4's 3 in
`CODEX-FINDINGS-RELAY-R4.md`, revision 5's 1 in
`CODEX-FINDINGS-RELAY-R5.md`, unedited; every one is taken, marked
**(codex R1-N)** through **(codex R5-N)**. **Round 6 returned accept**
with no findings (`CODEX-FINDINGS-RELAY-R6.md`). Stage 4a of the media roadmap
(`2026-09-07-media-platform-roadmap-design.md`), built under rulings 8,
10 and 11 there.

**Why now.** Oficina's Meet is peer-to-peer WebRTC and cannot name a
third-party STUN server, because suite ruling 6 forbids egress that is a
property of the product. Without STUN, browsers gather only LAN
candidates and calls connect on the same network only. A platform STUN
listener is the smallest thing that makes Meet work across the
internet; TURN on the same listener is what makes the one call in ten
that cannot hole-punch connect anyway. Meet is blocked on nothing else
(Meet session, 2026-09-07), and it has reviewed the app-facing contract
in §2.

## 1. What is built

One new platform unit, `carlos-relay.service`, on every region box the
operator arms, running `carlos relay serve` from the platform binary,
built on `pion/turn` v4.1.4, pure Go, which fetches and cross-compiles
clean for the CI shards (checked 2026-09-07). It answers STUN binding
requests to anyone and TURN allocations to holders of a credential an
enabled app minted. It learns which apps are enabled from a record the
console writes, binds every allocation to the identity that made it,
ends allocations when their credential, app, region, master or its own
authority lapses, meters what it forwards per app, and refuses peers on
private networks.

Three console doors and a CLI verb, `carlos relay enable|status|disable`,
that turn the relay on for an app and deliver its keys as sealed config,
plus an operator verb `carlos relay rotate`. An operator-side master key
per region, staged in a file on the region's boxes and, as a console
secret, in the console. A config-adoption acknowledgement the host's
config sync writes, so status can say "adopted" truthfully. A
security-group change opening the relay's ports. A DNS name per region
relay.

Not built, by ruling: TURN over TLS, an inter-region byte meter. Not
built, by scope: an SFU, recording, an issuance API, client-side relay
selection, commercial quotas. §8 lists them with reasons.

## 2. The app-facing contract

This is what Meet codes against. It is fixed by this spec; the rest of
the design may change without touching it.

**Environment.** One variable reaches the instance through the sealed
`config.json` path, the way store credentials do. Because the config
renderer refuses quotes, backslashes and newlines in any value
(`internal/appconfig/materialize.go:23`), the value is the **standard
base64 encoding, with padding, of a JSON document**; the app decodes
the base64, then parses the JSON. **(codex R1-1)**

- `CARLOS_RELAY`: base64 of a JSON array, one element per region the
  app may use, in the console's order (§5). Absent or empty means no
  relay: the app shows its floor.

      [{"region":"eu-west-1",
        "urls":["stun:relay.eu-west-1.oncarlos.com:3478",
                "turn:relay.eu-west-1.oncarlos.com:3478?transport=udp",
                "turn:relay.eu-west-1.oncarlos.com:3478?transport=tcp"],
        "key":"<standard base64, with padding, of 32 raw bytes>",
        "username_suffix":"abc:meet:0123456789abcdef:k1"}]

  `region` is the AWS region name, which is also the vocabulary of the
  relay hostnames. `username_suffix` is `<account>:<app>:<incarnation>:
  <kid>`, supplied by the platform so the app never has to know its own
  canonical account sqid or app name. **(codex R1-11, R1-12)** There is
  no separate incarnation variable; the suffix carries it. The app
  builds its ICE server list from whatever `urls` are present and
  assumes nothing about their number or schemes.

**Minting, per call.** For the region it wants, the app computes:

    key        = base64_decode( element.key )          // 32 raw bytes
    username   = "<expiry>:" + element.username_suffix  // ASCII
    credential = base64_std( HMAC-SHA256( key, username ) )

where `expiry` is a decimal Unix time in seconds, at most four hours
ahead **(ruling 1)**. The username has exactly five colon-separated
fields: `expiry` (`[0-9]+`), `account` (`[a-z0-9]+`), `app`
(`^[a-z][a-z0-9-]{0,31}$`), `incarnation` (`[0-9a-f]{16}`) and `kid`
(`[a-z0-9]{1,8}`). No field can contain a colon. The app hands the
browser an `RTCIceServer` list: `stun:` entries bare, `turn:` entries
carrying `username` and `credential`. STUN needs no credential.

**Expiry.** The relay checks the expiry on every authenticated request
the browser's ICE agent sends (allocate, refresh, permission, channel
bind) **and** ends the allocation at the expiry regardless of what the
client sends (§3, "termination"), so a credential's deadline is a hard
deadline on relayed traffic. An app that needs a call to outlive its
credential re-mints and performs an ICE restart before the deadline.
Four hours is the maximum, not a promise about meeting lengths.
**(codex R1-4, R1-20)**

**What changes the incarnation.** Only `carlos relay disable` followed by
`carlos relay enable`. A repeated enable, a region change, a deploy, a
restart and a master rotation all keep it. **(codex R1-13)**

**What changes the key.** A master rotation in a region changes that
region's key and `kid`; a new incarnation changes every region's key.
Either re-renders the app's config within about a minute, as any config
change does today, and what happens next depends on the route's shape
(`internal/appconfig/sync.go:30`, `syncexec.go:1`): a unit-backed route
and a socket-backed exec route are restarted onto the new file; a
TCP-address exec route whose bundle opts into file delivery receives
the file and is **not** restarted, because its process belongs to an
external supervisor CARLOS cannot bounce, so the app must re-read it
and then tell the platform it has (`carlos relay adopted`, §5); a
TCP-address route without file delivery receives nothing, and **an app
with such a route cannot enable the relay** until it opts that route
into file delivery or moves it to a unit. `carlos relay status` names
which of these each route is. **(codex R1-10, R1-14, R2-6, R3-4, R5-1)**

**A credential keeps working for its whole life** provided the master
that signed its key is not force-retired before its expiry. The
rotation procedure in §5 guarantees that when followed: the old master
is retired no earlier than four hours after the last instance stopped
minting from it, as observed by the console, and no earlier than 28
hours after rotation regardless. **(codex R2-4)**

**Test vector.** Master is `SHA-256("carlos-relay-test-master")`,
account `abc`, app `meet`, incarnation `0123456789abcdef`, kid `k1`,
expiry `1900000000`:

    master     841d374df509340d6ab18a139a27ed0d9c5d61d0bdf306e4124ff913a7a92fb5 (hex)
    key        uMhwW4P/9O5dQUIs91t2CCHGhwJbeiM0xQMQ2G0CJLU=
    username   1900000000:abc:meet:0123456789abcdef:k1
    credential jRU479Y3GO45nrRHqxBf3BMeBHE6Fip1cwAccNv28MQ=

The key derivation is in §4; an app never needs it, only the key it is
given. The vector will be checked into `internal/relay/vector_test.go`
by the plan, and Meet's minting test uses the same numbers.

**Logging.** The app logs which region it used and nothing else about
the credential. The relay logs the account, app and outcome of an
authentication, never the credential or a key.

## 3. The relay unit

`carlos-relay.service` is a media service in the roadmap's §3 terms: a
network-facing platform unit like the edge, not a codec helper.

- **Process.** `carlos relay serve`, from the platform binary, so it
  rides the same release and promote path as the edge. A new file in
  `internal/host/units/`, embedded and written by bootstrap like every
  unit, with `ConditionPathExists=/etc/carlos/relay-masters` so a box
  without a master file never listens. Added to `enabledUnits` and to
  the list of units the systemctl plan **starts**, not only enables,
  and to the adopt restart path beside `restartEdge`
  (`internal/host/systemapply.go:568`), so a platform update restarts
  the relay onto the new binary and a fresh box runs it without a
  reboot. The staging bootstrap plan prints the same steps without
  running them, as it does for the edge. **(codex R1-16)**
- **Hardening and memory.** From `carlos-edge.service` and then cut
  down: no `CAP_NET_BIND_SERVICE` (3478 is unprivileged),
  `User=carlos`, `ProtectSystem=strict`, `ReadWritePaths=/data/relay
  /run/carlos`, `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`.
  Memory: `MemoryMax=256M`, `MemoryHigh=160M`, `OOMScoreAdjust=1000`.
  What that guarantees, stated exactly: the relay can never take more
  than 256 MiB of the box; above 160 MiB it refuses new allocations and
  ends its oldest until it is under; and it is the kernel's preferred
  victim among processes of similar footprint, since 1000 is the
  largest adjustment there is. What it does not guarantee: that the
  relay dies before a tenant under every host-wide pressure. The
  kernel's score counts swap and page tables as well as RSS, tenants
  may swap up to their memory cap (`internal/cgroup/cgroup.go:313`),
  and a kill inside the edge's own 80% cgroup considers only the edge's
  tree. So the relay does not wait for the kernel: it watches the
  box's memory pressure (`/proc/pressure/memory`, the `some` ten-second
  average) and sheds allocations, oldest first, whenever it exceeds
  20%, whatever caused the pressure, resuming admission when it falls
  under 10%. The gate in §7 tests that shedding, not a kill order.
  `Nice=5` and `CPUWeight=50` for CPU. **(codex R1-19, R2-8, R3-7)**
- **Listeners.** UDP 3478 and TCP 3478 on the box's primary address.
  No TLS listener **(ruling 4)**. The relay allocation range is UDP
  49152 to 65535, bound through the relay's own address generator
  (below) with the box's Elastic IP as the advertised relay address,
  learned as the edge learns it, from IMDS (`readPublicIP`,
  `internal/host/agent.go:1808`), with `CARLOS_RELAY_PUBLIC_IP` as the
  override for a box without IMDS.
- **STUN.** Binding requests are answered on every listener without
  authentication, which is pion's default and the reason this stage
  exists.
- **The relay owns every socket pion uses.** pion has no public call to
  end an allocation; its allocation manager is internal and `Server`
  exposes only `AllocationCount` and `Close`. What it does expose is the
  `RelayAddressGenerator` that creates each allocation's relay socket
  (`server_config.go:20`, returning a `net.PacketConn`), the listener
  sockets the relay passes in, and an `OnAllocationCreated` callback
  that supplies the relay address before pion starts reading from it
  (`internal/allocation/allocation_manager.go:140`). pion's read loop
  deletes the allocation when the relay socket returns an error
  (`internal/allocation/allocation.go:301`). So the relay implements
  the generator and wraps both kinds of socket: every relay socket is
  recorded against its allocation at `OnAllocationCreated` and can be
  closed by the relay at any time, which ends that allocation inside
  pion within one read; and both relay and listener sockets count what
  they forward and apply the token buckets below. That one mechanism is
  how every deadline is enforced, how metering is attributed, and how
  bandwidth is bounded. **(codex R1-4, R1-17)**
- **Owner binding.** pion locates a refresh, permission or channel bind
  by the client's five-tuple and does not compare the authenticating
  username to the one that created the allocation
  (`internal/server/turn.go:34`); it also tags every allocation's tuple
  as UDP even for TCP clients, so the event's protocol string cannot
  distinguish transports. The relay keys its binding on the **typed
  source address**: the Go type of the `net.Addr` it receives
  (`*net.UDPAddr` or `*net.TCPAddr`) plus its string, which pion passes
  unchanged to `AuthHandler` and to the events. On
  `OnAllocationCreated` it records that key to the identity fields
  (account, app, incarnation); its `AuthHandler` refuses a username
  whose identity differs from the one bound to a live allocation from
  that key. Cleanup belongs to the wrapped relay socket, not to
  `OnAllocationDeleted`, which carries no relay address
  (`internal/allocation/event_handler.go:20`): each wrapper knows the
  binding it belongs to, and on `Close`, whether pion closed it or the
  relay did, it removes that binding only if the map still points at
  itself, so a replacement allocation bound while the old one was
  closing (pion removes from its manager before closing,
  `allocation_manager.go:150`) is never erased. **(codex R1-3, R2-3,
  R3-3)**
- **Termination.** An allocation is ended, by closing its relay socket,
  at the earliest of: its credential's expiry (a timer set at creation
  from the username's expiry field); the app being disabled, its
  incarnation changing, or this region leaving its list (on the next
  authority read); the master named by its `kid` reaching `retire_at`
  (on the next masters-file read or timer); the relay's authority lease
  expiring (below); the app or account crossing a byte budget (below);
  the relay shedding under memory pressure. Ending one allocation ends
  its permissions and channels with it. pion's own maximum requested
  lifetime is one hour; the relay does not rely on it. **(codex R1-4,
  R2-4)**
- **Authentication.** A custom `AuthHandler` (§4), called by pion on
  every authenticated request, which is where the username's expiry and
  the authority record are checked; pion verifies the message integrity
  against the key the handler returns.
- **Peer policy.** A `PermissionHandler` refuses `CreatePermission` and
  `ChannelBind` to loopback, link-local (including the IMDS address),
  RFC 1918 and unique-local ranges, the box's own private address and
  VPC CIDR (from `CARLOS_RELAY_DENY_CIDRS`, set by the operator from
  tofu output), and any other relay's private address. The box's own
  Elastic IP is allowed by the permission handler, because two clients
  that both fall back to this relay hold relay candidates on that
  address and must reach each other, **but only live relay allocation
  ports are reachable there**: the permission handler sees only an IP,
  so the wrapped relay socket enforces the port on `WriteTo`, dropping
  any packet to the EIP whose destination port is not a currently
  allocated relay port. Nothing else on the box is reachable through
  the relay, whatever the security group says about source addresses.
  Refusals and drops are counted per app. **(codex R1-8, R2-9)**
- **Limits (ruling 6)**, all admitted under one mutex across every
  listener so counts cannot race, with counters reset on process start
  and the rolling-hour byte windows rebuilt on start from every hourly
  metrics object that intersects the window, normally the previous
  hour's and the current one **(codex R3-6)**:
  - per app per box: 64 live allocations, 30 new a minute, 20 GB a
    rolling hour;
  - per account per box: 256 live, 60 new a minute, 60 GB a rolling
    hour, whatever the app count;
  - per box: 2048 live, 500 new a minute;
  - instantaneous: a token bucket of 8 Mbps per allocation and 200 Mbps
    per box applied in the wrapped sockets on both forwarding legs,
    dropping packets beyond the bucket, so a burst is bounded before it
    reaches the edge's share of the NIC;
  - crossing a byte budget refuses new allocations with 486 for the
    app or account and **ends its live allocations**, because a budget
    that only stops new ones bounds nothing. Failed allocations count
    against the per-minute rates.
  These are operational limits with those defaults, tunable by env;
  commercial quotas are out of scope. **(codex R1-9, R2-7)**
- **Metering.** Two meters per app and one gauge, measured where pion
  forwards and only for what it forwards: **(codex R1-17, R1-18, R2-5;
  ruling 5)**
  - `in`: payload bytes the relay accepted from the client and forwarded
    to a peer, counted at the relay socket's `WriteTo`;
  - `out`: payload bytes the relay forwarded from a peer to the client,
    counted at the listener socket's `WriteTo` to the client's address,
    attributed through the owner binding;
  - `allocs`: live allocations at sample time.
  Both meters are host egress and are what the bill is made of.
  Peer bytes that arrive at a relay socket and are dropped for lack of
  permission are counted separately as `rejected` and never against a
  budget. Every sixty seconds the relay appends one sample
  `{at, apps: {"<account>/<app>": {in, out, allocs, rejected}}}`, with
  `in`, `out` and `rejected` as **deltas since the previous sample**,
  to its own hourly object `metrics/relay/<box>/<hour>.json`, which it
  alone writes: read, append, CAS-write, retry on conflict. On restart
  it reads the current hour's object back, so earlier samples survive
  and its rolling-hour budgets resume from them. The raw readers and
  rollups learn the new key in the plan. An inter-region meter is not
  built (§8).
- **Authority record and lease.** The relay reads `control/relay.json`
  (§5) from the deployment bucket **every sixty seconds**, and
  additionally whenever `control/serial` changes as seen by its own
  poll of that object (the host agent's `fastConvergeLoop`,
  `internal/host/agent.go:1437`, is a separate process; the relay runs
  the same shape of poll itself). The lease clock is the time of the
  last **successful** read; a failed read renews nothing. If that time
  is more than fifteen minutes ago the relay refuses new allocations
  and ends the ones it carries, per the roadmap's serving lease, and
  resumes on the next successful read. A relay with no record yet
  serves STUN only. Each read also writes a heartbeat,
  `control/relay-ack/<box>.json`, carrying the record's `written_at`,
  the read time, which listeners are up, the `kid`s currently loaded
  from the masters file, and the last masters-file reload error if
  any. **(codex R1-5, R1-14, R2-4)**

## 4. Keys and verification

**Region master.** Thirty-two random bytes per region, generated by the
operator with `carlos relay keygen`, which prints a `kid` and the key.
On each of the region's boxes they live in `/etc/carlos/relay-masters`,
root-owned, mode 0640, group `carlos`, one per line:

    <kid> <base64 key> <retire_at RFC 3339 or "-">

The relay reads the file at start and re-reads it when it changes, so
a rotation never restarts the process. A master past its `retire_at` is
ignored for new requests and its live allocations are ended (§3).
**(codex R1-10, R2-4)**

**Where the console's copy lives, and what that exposes (ruling 3).**
The console must hold every region's masters to derive app keys at
enable. It holds them as console secrets, `CARLOS_RELAY_MASTERS_<REGION>`
(region name uppercased with hyphens as underscores, since variable
names refuse hyphens, `internal/appconfig/appconfig.go:103`), one or
more `kid:key` pairs per region, of which the one named by the issuer
pointer `control/relay-issuer.json` is the one keys are derived from
(§5, rotation); and console secrets are sealed config like any app's. Sealed config is
decryptable on every host (roadmap §1). So the honest statement is: **a
compromised host can read every region's master**, not only the derived
keys the roadmap already conceded, and with a master it can mint
credentials for any app in that region, present and future, until the
operator rotates. Per-region masters still mean a relay box itself
holds only its own region's, and rotation of one region recovers that
region. The operator docs say this in those words. A design where only
a region's boxes ever hold its master, the region's relay deriving app
keys and writing them into config itself, is filed as the tightening
and is not version one, because it makes the relay a writer of app
config with all the CAS and ordering that implies. **(codex R1-7,
R2-10)**

**Derived key.** For an app, region and incarnation:

    key = HKDF-SHA256( ikm = master,
                       salt = "carlos-relay-v1",
                       info = account || 0x00 || app || 0x00 || incarnation,
                       length = 32 )

**Verification**, in the relay's `AuthHandler(username, realm, src)`,
which pion calls once per authenticated request and which returns one
key, so the master is chosen from the username, not by trial:
**(codex R1-2)**

1. Split the username on `:` into exactly five fields; anything else is
   refused.
2. Parse `expiry`; refuse if in the past or more than four hours ahead.
3. If the typed source address has a live allocation bound to a
   different identity (§3, owner binding), refuse.
4. Look up `account/app` in the authority record; refuse if absent, if
   the incarnation differs, or if this relay's region is not in the
   app's list.
5. Look up `kid` in the masters file; refuse if absent or retired.
6. Derive the key, compute the expected credential, and return
   `turn.GenerateAuthKey(username, realm, credential)`. pion verifies
   `MESSAGE-INTEGRITY` against that key, so a wrong credential fails
   inside pion; the handler never sees what the client sent.
7. Count the refusal reason per app: bad shape, expired, bound
   elsewhere, unknown app, wrong incarnation, wrong region, unknown
   kid.

The MD5 inside `GenerateAuthKey` is the long-term credential key of RFC
8489's compatibility mechanism, which pion implements; RFC 8489 also
defines a SHA-256 variant that pion does not. The HMAC-SHA256 above is
ours. **(codex R1-20)**

## 5. Enable, disable, rotate

**Records.** Two, both additive. **(codex R1-6, R2-1, R2-2)**

- `apps/<acct>/<app>/relay.json`, CAS-guarded like `store.json`, is the
  **source of truth** for the app: `{enabled, incarnation, regions,
  op, phase, issued_digest, issued_suffixes, cutovers, enabled_at,
  disabled_at}`. `issued_digest` is the sha256 of the sealed envelope
  that was delivered, which is what `Record.IssuedDigest` records and
  what reconcile compares; an earlier draft of this spec called it
  `config_generation` and no implementation ever carried that name.
  `issued_suffixes` (region to the username suffix last issued) and
  `cutovers` (the per-route member acknowledgements) are the other two
  fields `relay.Record` has always carried and this listing has always
  omitted. **(discussion #120)** `op` is a
  counter bumped by every operation; `phase` is one of `config`,
  `aggregate`, `done` and records how far the operation got. The
  incarnation is sixteen hex characters drawn at random at first
  enable and at each re-enable after a disable, never a counter, so
  reuse across delete and name reuse is not a question.
- `control/relay.json`, the authority record the relays read:
  `{apps: [{account, app, incarnation, regions}], written_at}`. It is
  **derived**, never patched. Publishing it is one fixed sequence: read
  the aggregate's current etag; list every `relay.json`; build the
  aggregate from records whose `enabled` is true, whatever their
  `phase`; write with a CAS on the etag read **first**; on conflict
  start the whole sequence again. Because the etag is read before the
  listing, a disable that lands during the listing makes the write
  fail rather than restore the app. Followed by a `control/serial` bump
  so relays notice within seconds; the bump is a latency hint, not the
  lease.

**Every operation is a walk the reconcile can finish, and the
reconcile converges everything.** An operation writes its intent to
`relay.json` first (`op` bumped, `phase` set), then does the config
write, then the aggregate publish, advancing `phase` after each step
with a CAS that carries the `op`. **One writer, enforced.** Every relay
operation, handler or reconcile, runs only in a console process that
holds the relay-writer lease, `control/relay-writer.json`: a CAS-written
record naming the holder with a sixty-second expiry, renewed every
twenty seconds, taken over by another process only after it lapses. A
process that fails a renewal stops every relay write in flight before
its next bucket write and abandons the operation, which the next
holder's reconcile finishes. Inside the holder, operations on one app
are serialised behind a per-app lock. So two console processes,
during a deploy overlap or otherwise, never both write relay config,
and a superseded key selection cannot reach config after a migration:
the writer that selected it either holds the lease, in which case it
is the only writer and no migration happened under it, or lost it and
stopped. **(codex R3-2, R4-1)** On top of that fence, the **reconcile**
does not only finish unfinished records, it **converges both derived
artefacts from every record, every five minutes and at console
start**: for each enabled
record it derives the expected `CARLOS_RELAY` value from the record and
the active masters and re-applies it if the config bundle holds
anything else; for each disabled record it removes the entry if
present; and it always rebuilds and publishes the aggregate whether or
not any record is unfinished. So a stale config write that lands after
a disable is undone within five minutes, an interrupted enable
delivers its config within five minutes, an interrupted disable
removes it within five minutes, a repeated `enable` while a previous
one is unfinished resumes it rather than skipping the config write, and
a deletion that removed the record but died before publishing is
repaired by the next unconditional rebuild. **(codex R2-2, R3-1, R3-2)**

**When a disable is complete.** The app is out of every relay's
authority when the aggregate has been published without it **and**
every armed relay's heartbeat carries a `written_at` at or after that
publish. `carlos relay status` shows "disable pending on <regions>"
until then, and "revoked" after. Bounds: the publish is immediate on the
happy path and within five minutes after a crash; each relay reads
within sixty seconds of the publish; a relay that cannot read stops
serving on its own fifteen-minute lease. **(codex R2-1)**

**State transitions** for `relay.json`: **(codex R1-13)**

| From | Operation | Incarnation | Keys re-derived | Allocations |
|---|---|---|---|---|
| none | enable | new | yes | none |
| enabled, done | enable (same regions) | kept | no, no config write | untouched |
| enabled, not done | enable | kept | resumes the unfinished operation | untouched |
| enabled | enable `--regions` (change) | kept | only for added regions; removed regions' entries dropped | ended in removed regions on next authority read |
| enabled | disable | kept in record, app removed from aggregate | entry removed | ended everywhere on next authority read |
| disabled | enable | **new** | yes, all regions | none |
| any | app deleted | record removed with the app prefix; the deletion walk publishes the aggregate | | ended on next authority read |
| any | rotate `--region r` | kept | region `r` only, new kid | untouched; old kid accepted until retirement |

**`carlos relay enable [--regions r1,r2]`** (member, API-mode only, on
the store model in `store.go` and `internal/console/apistore.go`;
ruling 2):

1. `apiAccountApp` gate; unknown and foreign are one byte-identical 404.
2. Refuse if the deployment has no relay regions (`CARLOS_RELAY_REGIONS`
   in console config is empty) with a message naming the operator.
   Refuse if any of the app's routes has no config delivery (a
   TCP-address route whose bundle does not opt into file delivery),
   naming the route, because such a route could never adopt a rotated
   key and would block retirement forever. **(codex R5-1)**
3. Regions are every region in `CARLOS_RELAY_REGIONS`, **in the
   console's order**, or `--regions` narrows them keeping that order.
   There is no placement-first promise: placement names (`ie`,
   `fleet/x`) are not AWS regions and an app may have several, so the
   order is one fixed list for the app and every browser tries every
   listed relay. **(codex R1-12, R2-11)**
4. Apply the transition above to `relay.json` (CAS, `op` bumped,
   `phase: config`).
5. For each region needing keys, derive from that region's current
   master and its kid, and write `CARLOS_RELAY` as one `kind:"secret"`
   entry through `appconfig.Apply` (the `writeEnv` shape in
   `internal/objstore/walk.go:602`). `Apply` touches the app's config
   generation, a nonce plus timestamp (`internal/appconfig/
   generation.go:40`); record that generation string in `relay.json`
   and advance to `phase: aggregate`. **(codex R2-6, R2-12)**
6. Publish `control/relay.json`, bump `control/serial`, advance to
   `phase: done`.
7. Answer with the record. The CLI prints regions, incarnation, and
   "config generation <g> written; delivery is reported by `carlos
   relay status`", never "delivered".

**Config adoption, observed per route.** The host's config sync, after
it renders a route's env file and does whatever that route's shape
calls for, writes an additive acknowledgement
`control/config-ack/<box>/<route host>.json`: the route's shape (unit,
socket exec, TCP exec with file delivery), the time, whether a restart
happened and succeeded, and **the relay key identities it rendered**:
the `username_suffix` per region parsed out of the `CARLOS_RELAY` value
it just wrote (account, app, incarnation and kid together), or none. A
kid alone would not do: a disable and re-enable between two host syncs
changes the incarnation and the key while keeping the kid, and an ack
naming only kids would report the old credentials as adopted.
**(codex R4-2)** The ack is keyed on the route, not the app, because an
app can have several routes on one box and one can adopt while another
does not; and it names identities rather than a generation string,
because the app-wide generation object is a change notification that
can coalesce or fail after the manifest write
(`internal/appconfig/generation.go:28`) and is not an identity of what
was rendered. Nothing reads the ack today but `carlos relay status`;
it is the first per-route statement of "this is what the process has"
the platform has, and the plan adds it to `internal/appconfig/sync.go`
as its own small step. **(codex R1-14, R2-6, R3-4)**

**`carlos relay status`** reports, separately: desired state from
`relay.json` with its `op` and `phase`; the identities (username
suffixes) the console last issued per region; delivery per route,
"adopted at <time>, restarted" when a config-ack names exactly the
issued identities and a successful restart,
"written, awaiting delivery" when no ack names them yet, "file
delivered, awaiting `carlos relay adopted`" for a TCP exec route with
file delivery until a member acknowledges it, "adopted by member at
<time>" after, "restart failed" when the ack says so, and
"hibernating, delivered on wake" when the route is asleep (a route
added without delivery after enable shows "no config delivery: relay
will not reach this route" and blocks retirement until fixed); disable
progress as above; and per region, from
`control/relay-ack/<box>.json`, the heartbeat age, whether the record
it holds is current, whether its lease is live, which listeners are
up, and which `kid`s it has loaded. A heartbeat older than five minutes
is shown as "stale", so a dead relay cannot look current.

**`carlos relay disable`** applies the transition (`phase: config`),
removes the `CARLOS_RELAY` entry through `appconfig.Apply`, publishes
the aggregate, marks `done`.

**Rotation** (`carlos relay rotate --region r`, operator): **(codex
R1-10, R2-4)**

1. Operator runs `carlos relay keygen`, which prints a new `kid` and key.
2. Operator adds the line to `/etc/carlos/relay-masters` on each of the
   region's boxes with `retire_at` `-`. The relay re-reads the file;
   no restart, no allocation touched. Its next heartbeat lists the new
   `kid`.
3. Operator **adds** the new `kid:key` to the console secret for the
   region, which holds every master the console may need,
   `CARLOS_RELAY_MASTERS_<REGION>="<kid>:<key>[,<kid>:<key>]"`. Adding
   it issues nothing: **which master the console issues from is a
   separate, unsealed pointer**, `control/relay-issuer.json`
   `{"<region>": "<kid>"}`, and every path that derives a key for an
   app (enable, region change, re-enable, reconcile) reads that
   pointer and uses the master it names. A staged master is inert
   until the pointer moves. **(codex R3-5)**
4. `carlos relay rotate --region r` **refuses until every armed relay
   in the region has heartbeated the new `kid`**, then moves the
   issuer pointer to it, re-derives and re-delivers keys for every app
   enabled in that region as an operation on each record, and prints
   the earliest permitted retirement time for the old `kid`: 28 hours
   from now.
5. `carlos relay status --region r` shows, per route of every enabled
   app, which identities its config-ack names and when. **Each route's
   cutover** is established one of two ways, by route shape. A
   unit-backed or socket-exec route: the config sync's ack naming the
   issued identities, since the platform restarted it. A TCP-address
   exec route with file delivery: its process belongs to an external
   supervisor the platform cannot restart, so the cutover is a
   **member acknowledgement**, `carlos relay adopted --host <route>`,
   given after the member has restarted the process; the console
   records it in `relay.json` against that route together with the
   identities issued at that moment, so an acknowledgement given
   before a later rotation does not count for it. A route in neither
   category cannot exist under an enabled relay (enable refuses it),
   and a route added later without delivery blocks retirement until
   it is fixed. The old master's `retire_at` may be set no earlier
   than `max(rotation + 28h, latest cutover + 4h)` across every route
   of every enabled app in the region, and `status` prints that time
   once every route has a cutover, or names the routes still without
   one. Setting it earlier is refused by nothing but the docs, and ends
   any allocation still on the old `kid`, which is the force-retirement
   case §2 excludes from its guarantee. **(codex R3-4, R4-3, R5-1)**
6. After `retire_at` the operator removes the line from the boxes and
   the console secret at leisure.

**App deletion** removes `relay.json` with the rest of the app prefix,
and the deletion walk publishes the aggregate so the app is gone from
authority at once. If the walk dies between the two, the reconcile's
unconditional rebuild, which lists records rather than looking for
unfinished ones, publishes without the app within five minutes.
**(codex R3-1)**

## 6. Infrastructure

- `infra/modules/region-host/main.tf`: a `relay` variable, default off,
  that adds ingress for UDP 3478, TCP 3478 and UDP 49152 to 65535 from
  `0.0.0.0/0`, and outputs the VPC CIDR for `CARLOS_RELAY_DENY_CIDRS`.
  Flagship turns it on per region.
- DNS: an A record `relay.<aws region>.<platform domain>` to the region's
  Elastic IP, next to the existing `aws_eip.host` resource.
- Each armed box gains `/etc/carlos/relay-masters` (staged by the
  operator, like `host.env`) and `CARLOS_RELAY_DENY_CIDRS` and
  optionally `CARLOS_RELAY_PUBLIC_IP` in `host.env`.
- Console config gains `CARLOS_RELAY_REGIONS` and one
  `CARLOS_RELAY_MASTERS_<REGION>` secret per region.
- Egress cost: every forwarded byte leaves the box once. The
  unit-economics spec gets a line for relay egress per region, and the
  two meters in §3 are what fill it.

## 7. Testing

- `internal/relay`: derivation and credential against the §2 vector;
  the `AuthHandler` against a fixture authority record and masters file
  covering every refusal reason in §4, including a valid credential for
  app B presented from a typed source bound to app A, and a UDP and a
  TCP client with the same address string holding separate bindings;
  the permission handler against each denied range and the allowed
  EIP; the relay socket dropping a packet to the EIP on a non-allocated
  port; limits against a fake clock, including an account spreading
  allocations across apps, an allocation ended by the byte budget, and
  a token bucket dropping a burst; the meter against wrapped in-memory
  sockets, checking that rejected peer ingress is not counted as `in`
  and that both forwarding legs are counted; the hourly object surviving
  a restart, including a restart across an hour boundary that rebuilds
  the rolling window from both objects; a wrapper's close removing only
  its own binding while a replacement binding for the same typed
  source survives; shedding triggered by a fake pressure reading.
- End to end, on loopback with a fixture master: a pion TURN client
  allocates, refreshes, and keeps relaying until the credential's
  expiry, at which point traffic stops with no further client request;
  the same after a disable written to the fixture authority record;
  two clients on one relay reaching each other's relay candidates; a
  rotation, by rewriting the masters file, during which an allocation
  on the old kid carries traffic uninterrupted, then a `retire_at`
  ending it; a lease lapse ending allocations and a successful read
  restoring service.
- `internal/host`: the unit is embedded, written and started by the
  bootstrap plan when the staging root carries a masters file, and not
  started when it does not; the adopt restart path names the relay; the
  config sync writes a config-ack.
- `internal/console`: the doors through `rawReq` and a `relayURL`
  fixture: the byte-identical 404, the no-regions refusal, each row of
  the transition table, the config entry written and removed, the
  aggregate after concurrent enable and disable of two apps with the
  etag-first ordering, an enable interrupted after `phase: config`
  finished by the reconcile, a superseded writer refused by its CAS, a
  stale config write landing after a disable and undone by the next
  reconcile, a deletion that removed the record without publishing
  repaired by the next rebuild, an enable during a staged-but-unmoved
  rotation issuing from the old kid, a process that lost the writer
  lease making no further relay write, a delayed old-key publisher
  refused after a migration, a disable and re-enable between two host
  syncs not reported as adopted by the old ack, and `rotate` refusing
  until the fixture heartbeats list the kid; the config sync writing a
  per-route ack naming the identities it rendered; the retirement time
  computed with a manual cutover after hour 28; enable refused for an
  app with a route lacking config delivery; a `carlos relay adopted`
  given before a later rotation not counting as that rotation's
  cutover; an externally supervised TCP route with file delivery
  blocking retirement until acknowledged.
- `docs/site/cli.md` gains `### carlos relay` with a fenced example, or
  the docsite gate fails.
- Flagship gate, as amended in the roadmap: a Meet call between two
  symmetric-NAT networks connecting through the Ireland relay; both
  browsers forced to relay and connecting through it **on the real
  EIP**, and a probe to a non-allocated port on the EIP dropped; a
  credential minted for one app refused for another, including on an
  existing allocation's tuple; a credential from before a disable
  refused after re-enable and its allocation ended; traffic stopping at
  a credential's expiry with the browser doing nothing; a relay with a
  lease older than fifteen minutes refusing new allocations and ending
  carried ones; a permission to a private address refused; the relay
  meter object showing the call's bytes on both legs; a rotation
  during a call with the call unaffected; the relay shedding
  allocations under host memory pressure caused by a tenant, with the
  edge still serving.

## 8. Not built

- **TURN over TLS (5349 or 443)** **(ruling 4)**. Certificate issuance
  for a non-HTTP listener needs the relay hostname registered with the
  deployment's ACME issuer, which is elected and may be another box
  (`internal/edge/certsync.go`), and its warmer excludes such names
  (`internal/edge/certwarm.go`); a listener that cannot start until a
  certificate exists cannot trigger issuance by being visited. That is
  its own small design, to be written when a network that needs it is
  reported. 443 additionally needs a second address per box. The
  roadmap's stage 4a text is amended to match. **(codex R1-15, R2-10)**
- **An inter-region byte meter** **(ruling 5)**. Classifying a peer
  address by AWS region needs the published IP ranges on the box and a
  rule for non-AWS peers; version one meters what is forwarded in each
  direction, which is what the bill is made of. Filed for stage 5 with
  the usage view; the roadmap's meters and gate are amended.
  **(codex R1-18, R2-10)**
- An issuance API: apps mint offline from delivered keys.
- Client-aware relay selection: the list is in console order; ICE picks.
- Commercial quotas and an app-page usage view: the meters exist, the
  page is stage 5.
- An SFU: stage 4b, after this ships.
- Any change to how apps signal: SSE plus POST stays Meet's.

## 9. Rulings

Paul ruled on each of these on 2026-09-07. **(codex R2-10)**

1. **(ruling)** Maximum credential life is four hours. The roadmap's
   one-hour figure is amended.
2. **(ruling)** Enable is member-level; the operator's consent is the
   region list and the limits.
3. **(ruling)** The console holds every region's master as sealed
   config; the exposure is stated in §4 and the operator docs; the
   relay-derives-and-writes design is filed as the tightening.
4. **(ruling)** No TURN over TLS in version one; the roadmap's stage 4a
   ingress text is amended.
5. **(ruling)** Two meters in version one, in and out; the inter-region
   meter and the usage line move to stage 5; the roadmap's metering
   text and gate are amended.
6. **(ruling)** The limits in §3 with their stated defaults, including
   the token buckets.