# Fleet-internal engines: a doc engine nobody on the internet can call

**Date:** 2026-10-21
**Revision:** 14. Approved by Paul, 2026-10-21.
**Discussion:** amadan carlos/platform #335. Related: #328 (per-app egress allow).

- **Rev 14** applies Astra round 13's two P1s (`CODEX-FINDINGS-ENGINES-R13.md`)
  as small fixes within R3, per R4 with no further round. The ingress
  precondition covers private sources too: another box reaching the
  engine's backend directly (R13-1). The fence on an engine's box is
  box-wide, cancelling every non-engine proxy request admitted before it,
  including one held by a row since deleted or replaced (R13-2). Marked
  **(codex R13-n)**.
- **Rev 13** answers Astra round 12 (`CODEX-FINDINGS-ENGINES-R12.md`,
  reject, two P1s: an engine binding its own high port, and a separate
  public route row pointing at the engine's backend) with Paul's ruling R3:
  an engine's home box is dedicated, and the platform enforces it (§5.9).
  Ruling R4 sets the review end: one more round, then Paul's read. Marked
  **(codex R12-n)**.
- **Rev 12** answers Astra round 11 (`CODEX-FINDINGS-ENGINES-R11.md`,
  reject, one P1: a declaring release that also declares `ports` keeps a
  granted TCP listener the edge never sees). `doc_engine` and `ports` are
  refused together at ship, as `doc_engine` and `host_role` already are, and
  a box renders no port listener for an internal group (R11-1). §5.1 now
  also says why systemd's own starts (socket activation, `Restart=always`)
  keep I1. Marked **(codex R11-1)**.
- **Rev 11** answers Astra round 10 (`CODEX-FINDINGS-ENGINES-R10.md`,
  reject, one P1: a mark written but never fenced, after a failed reload
  and a timed-out adoption, let a recovery launch start a declaring build
  while a public request was admitted). I1 now speaks of the **fenced**
  mark, not the registry's, and every declaring launch requires the fence
  whether or not it finds `1` already written (R10-1). Marked
  **(codex R10-1)**.
- **Rev 10** answers Astra round 9 (`CODEX-FINDINGS-ENGINES-R9.md`, reject,
  one P1: after `--public`, a failed first wake falls back to a declaring
  `GoodVersion`). Instead of a fourth path-specific rule, §5.1 now states
  the invariant every path must keep, **I1**, and enforces it at the one
  place every path passes through: the launch. A launch of a declaring
  build sets the mark and completes the barrier first, cancelling the
  public request that caused it (R9-1). Marked **(codex R9-1)**.
- **Rev 9** answers Astra round 8 (`CODEX-FINDINGS-ENGINES-R8.md`, reject,
  a P1 and a P2, both accepted) by changing approach rather than adding a
  third proof: a route never becomes public automatically. Clearing the
  mark is an operator verb that stops the group under the adoption lock and
  checks its cgroups are empty (R7-1, R8-1, R8-2). Upgrade classification
  also marks a route internal when any staged version of its app declares
  `doc_engine`. Marked **(codex R8-n)**.
- **Rev 8** answers Astra round 7 (`CODEX-FINDINGS-ENGINES-R7.md`, reject,
  a P1 and a P2, both accepted). Clearing the mark waits for proof that no
  process of a declaring build is left, not for adoption's return, which
  under generation rollouts only schedules the swap (R7-1). A minted row
  records its owner explicitly; the fallback match needs a unique non-empty
  backend, TCP included (R7-2). Marked **(codex R7-n)**.
- **Rev 7** answers Astra round 6 (`CODEX-FINDINGS-ENGINES-R6.md`, reject,
  a P1 and a P2 in rev 6's minted-row rule, both accepted). Inheritance is
  scoped to minted `instance` rows, static and service platform rows keep
  `0` (R6-2), and a `doc-engine` grant on a minted row marks its execution
  owner, so the whole group goes internal together (R6-1). Marked
  **(codex R6-n)**.
- **Rev 6** answers Astra round 5 (`CODEX-FINDINGS-ENGINES-R5.md`, accept
  with fixes, three P2s, all accepted). Platform-minted rows take their
  execution owner's mark and barrier (R5-1). Arming judges exec-backed
  routes by layout and admits unit-backed ones (R5-2). The watch ticket
  signs the probe path the watcher uses (R5-3). Marked **(codex R5-n)**.
- **Rev 5** answers Astra round 4 (`CODEX-FINDINGS-ENGINES-R4.md`, reject,
  two P1s and a P2, all accepted). "No flat route" becomes a positive,
  operator-armed box state that also excludes `foreign` instances and
  workers, and refuses unisolated launches while armed (R4-1). An engine
  call is dispatched only to the engine hostname's own route, never through
  a path map (R4-2). Watch tickets are renewed for as long as the watch runs
  (R4-3). Marked **(codex R4-n)**.
- **Rev 4** answers Astra round 3 (`CODEX-FINDINGS-ENGINES-R3.md`, reject,
  two P1s and a P2, all accepted). The barrier's acknowledgement now also
  covers public requests admitted before the mark: the edge cancels them and
  waits for them to finish before it acknowledges (R3-1). Engine sockets
  open only on a box with no flat instance at all, because a flat tenant can
  write a nested sibling's files and run code inside its slice (R3-2). Every
  deploy target gets a watch ticket, not only declaring releases (R3-3).
  Marked **(codex R3-n)**.
- **Rev 3** answers Astra round 2 (`CODEX-FINDINGS-ENGINES-R2.md`, reject,
  three P1s and two P2s, all accepted). The mark gains an "unclassified"
  state and a serving barrier: set before the release switch, cleared after
  it, and observed by the edge before adoption proceeds (R2-1). Every route
  is classified before the upgraded edge opens a listener (R2-2). The door
  carries engine calls only for callers with their own uid, so flat
  instances cannot call engines at all (R2-3). `carlos deploy` watches an
  internal engine with a console-signed watch ticket (R2-4). The refusal is
  a plain 404 that never redirects, and the no-redirect promise on :80 is
  scoped to the home box (R2-5). Marked **(codex R2-n)**.

- **Rev 2** answers Astra round 1 (`CODEX-FINDINGS-ENGINES-R1.md`, reject,
  four P1s, all accepted) and the builder check, and records Paul's two
  rulings of 2026-10-21 (§2.1). What makes a route internal moves from a
  console record to the route itself (R1), which closes R1-3 (a new box
  starts public) and R1-2 (aliases and path routes) by construction. The
  socket's caller is proved by the peer's cgroup, not the socket's path
  (R1-1). The grant table keeps a durable revision floor (R1-4). The door's
  owner exception is dropped and the door-to-engine transition is defined
  (R1-5). The socket gets an instance-level owner (R1-6) and engine calls
  their own transport (R1-7). Each change is marked **(codex R1-n)**.

## 1. Intent

The `doc_engine` capability (`internal/caps/render.go`, README "App
capabilities") fences a conversion engine's **egress** away from private
networks. It says nothing about who can **call** the engine. A doc-engine
route is a route like any other: its box's public edge serves it on :443,
any other box backhauls a public request for it, and the only gate is
whatever auth the engine applies itself (Gotenberg: basic auth, often
none). An operator who skips that auth, or leaks its password, publishes a
Chromium and LibreOffice renderer to the internet: free compute for anyone,
and an SSRF engine whose remaining reach is the public internet and its own
box's loopback.

This spec makes every doc engine **fleet-internal**. Its routes answer only
calls the platform carried from an app an operator granted, and an isolated
tenant reaches it through a socket inside its own instance, with no private
address opened in its fence.

## 2. Goals, non-goals, anti-goals

**Goals**

- G1. A route whose live release declares `doc_engine`, or which holds the
  `doc-engine` grant, is refused to every public request, through every
  hostname, alias, wildcard or path that resolves to it, on every listener,
  except the ACME challenge.
- G2. An operator grants an app the use of a named engine, per app. Every
  nested instance of the app, on any box and in every unit kind its code
  runs as (run, generation, seed, `carlos run` command, legacy sidecar),
  can then call it, and so can the app's own host processes that hold a
  door key and run under their own uid (§5.4). A revoke takes effect within
  §6.4's bounds.
- G3. A nested instance calls an engine through a unix socket in its own run
  directory. Its `IPAddressDeny=` fence is unchanged.
- G4. The engine box knows which app made each call and refuses a call from
  an app without a grant, even if the calling box admitted it.
- G5. One app cannot starve the others: the engine box caps each app's calls
  in flight.
- G6. Provider-neutral (§10).

**Non-goals**

- **Unit-backed callers** (`carlos-app@`, DynamicUser). Ruling R2: later, if
  a caller appears. They have no instance run directory and cannot be
  enrolled for the door. Their routes can still *be* engines.
- **Flat-layout instances as callers** (§6.2). They share uid carlos with
  every other flat tenant, so nothing they hold can prove which app they
  are. An app that needs an engine moves to the nested layout.
- Metering or billing engine use. Member-requested grants.
- A general internal-route product for arbitrary services.
- Reaching an engine over TCP from a nested instance: its fence denies
  loopback and every private range, and `IPAddressAllow=` cannot name a
  port (#328's limit 1).
- Replacing #328. An engine outside the platform still needs #328's
  egress allow.

**Anti-goals**

- A window in which an engine route is public: a new or replacement box, a
  restarted edge, a stale mirror or a box that cannot reach the bucket must
  never serve one to the public.
- A caller identity a tenant can forge or borrow: through a header, another
  instance's socket, a sibling's process or the door.
- Opening any private address in an isolated instance's fence.

### 2.1 Rulings (Paul, 2026-10-21; do not reopen)

- **R1. Declaring `doc_engine` makes a route internal.** No opt-out and no
  separate registration step for it. A doc-engine route in public use today
  goes dark when its home box takes this build; its callers move to the
  socket or door first (§7).
- **R2. Unit-backed apps are not callers in this spec.** Later, if a caller
  appears.
- **R3. An engine's home box is dedicated, and the platform enforces it**
  (§5.9). Public ingress beyond the edge's own ports is a stated box
  precondition, not a platform-installed firewall.
- **R4. Review end:** one Astra round on rev 13, small fixes, then Paul's
  read. The plan's first task is a simulation of I1 and §5.9 across every
  launch and ingress path.
- **R5. Grant records go stale ten minutes after signing (Paul,
  2026-10-21).** An edge treats an `engines.json` record whose `signed_at`
  is more than 10 minutes old as stale, so its grant table is empty; the
  console re-signs every 2 minutes and on every change (§5.2, §5.3) (sim,
  plan 1, final review).

## 3. What exists today, and what this builds on

Cited from main at `3d444a1b`.

- **The backhaul hop** (`internal/edge/edge.go` `backhaul`/`backhaulTo`,
  `homemap.go`): one hop, ever, from the box a request reached to the box
  that homes the host. It dials the home box's `private_ip` when region HA
  is on and both boxes share a region and placement, its public address
  otherwise (`homemap.go` `dialAddress`), on :443 with public-CA TLS and SNI
  set to the real hostname, with a 30 s response-header timeout and a
  transport shared per destination (`edge.go:2825`, `:2938`). It is
  authenticated by `X-Carlos-Hop` and `X-Carlos-Hop-Auth`, an HMAC-SHA256
  over host and timestamp (±30 s) under the deployment's mesh key
  (`mesh.go`). **The hop carries no caller** (`acceptHop`).
- **The edge-local door** (#175 P7, `internal/edge/door.go`): a unix socket
  (`CARLOS_EDGE_DOOR`) through which an app's own processes reach the app's
  hosts on any box with an app-scoped door key (`internal/doorkey`): a
  console-minted, root-signed, revisioned, expiring record under
  `control/door-keys/`, mirrored by every edge every 30 s with a ten-minute
  stale bound and an in-memory revision floor (`doorkeys.go:22`). The door's
  mark is a context value; it does not cross a hop.
- **The per-instance control socket** (`internal/edge/purge.go`
  `ListenControl`), opened per generation for the run role only
  (`prepare.go:111`) and closed with the generation (`unitproc.go:203`).
  Every nested unit binds its whole run directory
  `/run/carlos/instances/<iid>` (`carlos-instance@.service:53`; generations
  through `instancectl/gen.go:48`). Every nested unit kind runs in
  `Slice=carlos-instances-<iid>.slice` (`instancectl.SliceName`; the four
  templates and the generation drop-in). The instance ID is at most 40
  bytes (`provision.go:2918`).
- **Peer checks on a unix socket:** the root control service accepts a
  connection only from SO_PEERCRED uid carlos and the edge's MainPID
  (`instancectl/server.go` `PeerOK`).
- **Capabilities:** the release's manifest declares them; adoption stages
  the manifest beside the live version (`release/adopt.go`
  `stageAuthorized`, `capability_pass` reads it via
  `stagedManifestOfLiveVersion`); grants are box-local in `routes.caps`
  (`registry.SetCaps`), set with `carlos route --host X --grant doc-engine`.
- **Flat tenants share uid carlos** with nested ones and can write their
  siblings' directories (`layout/preflight.go:181`).

## 4. The shape, in one paragraph

A route becomes **internal** on the box that serves it when its live
release declares `doc_engine` or it holds the `doc-engine` grant; the box
records that in its own registry and refuses the route to every public
request. An operator **names** an engine (`gotenberg` → its hostname) and
**grants** it to apps; the console publishes the set as a root-signed
`control/engines.json`. A granted nested instance's code sends plain HTTP
to `$CARLOS_ENGINES_SOCKET` with `Host: gotenberg`. Its box's edge proves
the caller from the connecting process's cgroup, checks the grant, and
relays the request to the engine's home box over the backhaul hop, now
signed with the caller's account, app and engine. The engine box verifies
the signature, checks the grant again, applies the per-app cap and serves
the route. A door caller does the same with `Host: gotenberg`.

## 5. The parts

### 5.1 The internal mark (every box, on its own routes)

**(codex R1-2, R1-3, R2-1, R2-2)** Internal is a property of a route on the
box that serves it, never of a hostname and never of a bucket record.

**The column.** `routes.internal` has three states: `NULL` (unclassified),
`0` (public) and `1` (internal). Routes of a kind that serves no release
(`static`, `service`, the status host, sidecars) are `0` by construction.
An `instance` route is classified from the manifest of its live release:
`1` if it declares `doc_engine`, `0` otherwise. The `doc-engine` grant also
sets `1`, whatever the manifest says. The edge treats `NULL` like `1`.

**Platform-minted rows (codex R5-1, R6-1, R6-2).** A platform domain is
its own registry row (`Platform: true`, `domains/reconcile.go:881`),
copying the source's kind and backend but no version, and adoption skips it
(`release/adopt.go:1210`).

- A minted `static` or `service` row (including a static canary on its own
  channel, `reconcile.go:368`) is `0` by construction, like any row of
  those kinds. The rest of this paragraph does not apply to it.
- A minted `instance` row's **execution owner** is recorded on the row: the
  domains reconciler writes the source's host into a new `routes.owner`
  column when it mints (`mintedRoute` has the source in hand) and backfills
  it on its next pass for rows minted earlier **(codex R7-2)**. Until a row
  has an owner recorded, the edge falls back to matching the non-platform
  `instance` row with the same account and app and the same **non-empty**
  backend: `Socket`, `Addr` (TCP-backed rows, `cli.go:172`,
  `domains/reconcile.go:835`), or a fleet-placed `Region`. Empty values never
  match, and a match that is not unique counts as no match. The edge
  resolves owners at `Reload`.
- The owner and every minted row that shares it form one **group** with
  one mark and one admission epoch. The group is internal when the owner's
  own classification is `1` **or any row in the group holds the
  `doc-engine` grant**. The grant CLI already permits a grant on a plain
  minted row (`cli.go:1432`); such a grant marks the whole group, and
  `--grant` and `--revoke` on any row of the group run §5.1's barrier for
  the group.
- A minted `instance` row whose owner the edge cannot find, or finds
  ambiguously, is treated as `NULL` and refused, and logged by name, until
  the reconciler records its owner. `--grant doc-engine` on a minted row
  whose owner is unresolved or ambiguous is refused; the row stays
  unclassified and refused (sim, plan 1).

**The invariant (codex R9-1).**

> **I1.** A process of a build whose manifest declares `doc_engine` starts
> only after the edge has **fenced** its group as internal: the edge's
> fenced counter (below) has reached a registry change that set the
> group's mark to `1`, and no later change has cleared it.

What protects the public is the edge's serving map, not the registry row
(codex R10-1): a `1` written while the edge's `Reload` fails
(`edge.go:925`, `:993`) refuses nobody. So the registry mark is the
intent, and the fence is the fact I1 is about. A group closed as
unclassified (`NULL`) counts as fenced only once the edge has loaded that
`NULL`: the change that closed it is recorded for the group, and the reload
that reached it drained every older public request on the box (the
upgrade's restart does both). A newly created or unseen row never counts as
fenced, so a first declaring launch writes `1` and waits for the fence like
any other (sim, plan 1, final review).

Rounds 2 to 9 each found one more path by which a declaring build could
run under a public mark: adoption's order, requests admitted early, a
pending generation swap, an un-cycled hibernating route, a sleeping unit,
and wake recovery falling back to `GoodVersion`
(`rollout/wakeplan.go:76`, `:111`; `activator/wakeplan.go:448`). So I1 is
enforced where every one of those paths ends: **the launch gate.** Every
activator launch path (an exec child, a nested run or generation unit
through the control service, a unit-backed `systemctl start`, a seed, a
`carlos run` command, a swap's incoming generation, a wake plan's
fallback or recovery build) reads the manifest of the build it is about to
start. If that build declares `doc_engine`, the launch writes `1` if it is
not already written, and then **always** waits for the edge's fence to
cover the group's current mark (below), which cancels every public request
admitted before it, including the one whose wake triggered the launch.
Only then does it start the process. A launch that finds `1` written but
not yet fenced (a timed-out adoption, a reload that keeps failing) waits
the same way; if the fence does not arrive within the bound, the launch is
refused, the wake fails, and the request gets the activator's ordinary
wake-failure answer. A launch whose build manifest it cannot read is
treated as declaring. Adoption's
set-before is the same gate, reached earlier. The gate holds a per-group
lock from its check through process creation, and `--public` holds the same
lock for its whole run, so no launch can pass the gate between `--public`'s
empty check and its write of `0` (sim, plan 1, final review).

`--public` (below) is the only writer of `0`, and it checks I1 directly:
the group is stopped and its cgroups empty. Any later declaring launch
re-marks the group through the gate. The one place I1 is not enforced is
stated as a residual under "Classification before serving".

**Starts systemd makes on its own.** A unit-backed route's unit is also
started by socket activation (`carlos-app@.socket`) and restarted by
`Restart=always` (`carlos-app@.service:50`), without the activator. Both
execute `/opt/carlos/live/<host>` (`:42`), the live version. Adoption
passes the launch gate before it repoints the live link to a declaring
build, and `--public` clears only when the live build does not declare, so
whatever systemd starts is either fenced or non-declaring. The builder
keeps that true by construction: nothing but adoption (through the gate)
and `--public` (through its checks) may move a live link.

**The edge is the only ingress (codex R11-1).** I1 protects the public only
if every way in goes through the edge. A granted `ports` capability is a
`ListenStream=<port>` socket the edge never sees (`caps/render.go:286`,
`:319`), and a hibernating route's port starts the unit without the
activator (`caps/converge.go:241`). So:

- **At ship**, `doc_engine` and `ports` are refused together, beside the
  existing `host_role` refusal (`caps/caps.go:382`): a conversion engine is
  reached over HTTP through the platform, never on a port of its own.
- **On the box**, the capability pass renders no `ListenStream` for any
  route of an internal group, whatever its grants, and removes an existing
  one (stopping the socket unit) before the launch gate's fence completes
  for that group; it logs the grant it is ignoring with the revoke command.
  A `ports` listener is removed on every write of `1` or `NULL` (a grant,
  the upgrade, the launch gate), not only at the launch gate (sim, plan 1).

**Setting the mark: the serving barrier (codex R2-1).** Adoption
repoints the live symlink (`release/adopt.go:1751`), restarts
(`adopt.go:1831`) and records the version afterwards (`adopt.go:2051`),
and the edge serves an in-memory map refreshed on a 2 s registry poll
(`edge.go:2298`). So the order is explicit:

- **Becoming internal** (the incoming release declares `doc_engine`, the
  route is `0`): adoption writes `1`, then waits for the edge's
  acknowledgement, then repoints and restarts. The edge publishes the
  registry change it has **fenced** (a counter in a file under
  `/run/carlos`), and it advances that counter only after both:
  1. `Reload` has swapped in a map carrying the mark (a failed `Reload`
     advances nothing); and
  2. **every request admitted to that route as a non-engine call before the
     swap has finished (codex R3-1).** Each such request holds a reference
     on the route's public-admission epoch from the moment it passes the
     internal check until its response ends, including while it waits in
     the activator's queue for a wake (`activator.go:1554`, `:5404`). Loading
     a `0`→`1` change bumps the epoch and cancels the contexts of every
     request holding the old one, which also drops it from the activator's
     queue; the counter advances when the last of them has released. A
     request admitted after the swap sees `1` and is refused.

  The wait is
  bounded (30 s); on timeout adoption does not repoint this pass, the mark
  stays `1`, and the next pass retries. The old, non-engine release is
  refused publicly for that interval, which costs availability, never
  exposure.
- **Becoming public is never automatic (codex R7-1, R8-1, R8-2).** No
  adoption, revoke, rollback or pass ever writes `0` over `1`. Rounds 6 to
  8 each found a new interleaving in which an automatic clear raced a
  process of the declaring build that was still running (a generation
  swap still pending, an awake hibernating unit adopted without a restart,
  a sleeping unit with no main process to vouch for). So clearing is an
  operator act whose preconditions are checked, not inferred:

      carlos route --host <host> --public

  On the box, holding the group's lock for its whole run (the adoption
  lock, so the live version cannot move under it, and the launch gate's,
  so nothing starts between step 3 and step 4; sim, plan 1, final review),
  it:
  1. refuses unless the live release's manifest is readable and does not
     declare `doc_engine`, and no row of the group holds the grant;
  2. stops the whole group (every unit and generation of the owner, through
     the activator and the control service);
  3. confirms every cgroup of the group is empty (`carlos-instances-<iid>.slice`,
     `carlos-app@<host>.service`, `carlos-exec@<host>`), and refuses if it
     cannot;
  4. writes `0` and releases the lock.

  Whatever starts afterwards passes the launch gate: the live version,
  which step 1 proved non-declaring, starts under `0`; a declaring build
  chosen by wake recovery (`GoodVersion`) re-marks the group first and the
  triggering public request is cancelled (I1). The route's next request
  wakes it. The cost is a short stop on a rare operation; a log line on the
  capability pass names every internal route whose live release no longer
  declares `doc_engine` and holds no grant, with this command.
- **Rollback** to a declaring release is an adoption like any other: set
  before. Rollback away from one leaves the mark set.
- The same barrier applies to a grant: `carlos route --grant doc-engine`
  writes `1` and returns once the edge has loaded it (or says it timed out).

**Classification before serving (codex R2-2, R8-1).** The new column arrives
`NULL` for every existing route. Before the upgraded agent opens any edge
listener, it classifies every `NULL` instance route (minted rows follow their
owner):

- `1` if any row of the group holds the grant, or the live version's
  manifest declares `doc_engine`, or **the manifest of any other version of
  the app staged on the box declares it**. The last clause covers a box
  whose live symlink has moved off a declaring build that is still running
  (an awake hibernating route is repointed without a restart,
  `release/adopt.go:1174`).
- `0` only if the live manifest is readable, none of the above holds.
- Otherwise (the live manifest is unreadable from the staged sibling and
  from the bucket, as adoption's back-fill reads it) the route stays `NULL`,
  is refused publicly, and is named in a log line every pass until the
  capability pass classifies it. That is the fail-closed choice.

A route that turns out `1` only because an old staged version declared is
cleared with `--public` like any other. **Residual, stated:** a declaring
version whose staged manifest failed to write, still running across the
upgrade after the live symlink moved to a non-declaring version, would be
classified `0`; it needs a sibling write failure on a doc-engine release
and a pending, un-cycled switch away from it at the moment of the upgrade.
A route the agent creates later (provisioning, a move) is classified by its
first adoption, which runs before the route is served.

**What the edge does with it** (`ServeHTTP`):

- The check runs on the **final resolved upstream's route**: after the
  route lookup and after the login POST's re-resolution, and before the
  import gate, the password gate, the meter, the cache, an upgrade and the
  activator's wake. Aliases (`edge.go:929`) and path-mapped hosts
  (`edge.go:1364`) copy the target's upstream, so every name and path that
  resolves to an internal route is covered, in any spelling the lookup
  accepts.
- An internal route admits only an engine call (§5.4). Everything else gets
  the **internal refusal**: a plain `404 not found`, `Cache-Control:
  no-store`, never the pending page and never the fallback redirect
  (`unknownHost` does both, `edge.go:2205`, so it is not reused).
- There is no owner exception: the engine app's own processes reach it by
  granting the engine to their own app.
- **:80 on the home box** serves the ACME challenge for an internal route's
  hostnames and the internal refusal for everything else: no redirect.
  **:80 on another box** knows only the home map, so it redirects to https
  as it does for any host homed elsewhere (`edge.go:2262`), and the https
  request is then backhauled and refused (codex R2-5). The redirect names
  nothing DNS and certificate transparency do not already publish.
- `Allowed` (certificate issuance) is unchanged: the hop's TLS needs a
  publicly trusted certificate on the home box.
- A box that does not home the host backhauls a public request as an
  ordinary hop with no caller, and the home box refuses it. A home box on
  an older build is the one gap, closed by deploy order (§7).

### 5.2 Engine names and grants (console)

Operator-only CLI, gated like `domains attach --platform` (an owner of the
ops account or an address on `CARLOS_SYSTEM_OPERATORS`,
`Server.systemOperator`):

    carlos engines add <name> --host <hostname> [--per-app <n>]
    carlos engines remove <name>
    carlos engines grant <name> --app <account>/<app>
    carlos engines revoke <name> --app <account>/<app>
    carlos engines list

- `<name>` matches `^[a-z][a-z0-9-]{0,31}$`, unique in the deployment, never
  contains a dot. It is what apps put in `Host:`.
- `--host` is an exact hostname the engine route serves on its home box.
  The console cannot check it (it holds no route registry); §5.7 reports
  mismatches from the box.
- `--per-app` is the engine box's in-flight cap per calling app. Default 4,
  range 1 to 64.
- `remove` refuses while grants remain. Removing a name changes nothing
  about whether the route is internal (§5.1).
- Every change is audited with `store.RecordEvent`, like `store grant`.
- `list` shows each engine's name, host, cap and grantees to an operator. A
  member sees only the engine names granted to apps in their accounts.

The console's database is the authority. It publishes the whole set as one
record, `control/engines.json`, signed with the root release key under its
own domain (`carlos engines v1\n`), with a strictly increasing revision and
`expires_at` 72 hours out. It re-signs every 2 minutes and on every change,
and an edge treats a record whose `signed_at` is more than 10 minutes old as
stale (ruling R5; sim, plan 1, final review). It
publishes an empty set (revision 1) the first time the feature is on, so a
box can tell "no engines" from "not read yet".

    {
      "v": 1, "revision": 17, "signed_at": "...", "expires_at": "...",
      "engines": [
        { "name": "gotenberg", "host": "gotenberg.example.com", "per_app": 4,
          "grants": [ { "account": "acct1", "app": "shop" } ] }
      ]
    }

### 5.3 The grant mirror (every edge)

**(codex R1-4)** The mirror verifies signature, shape, domain and expiry as
the door-key mirror does, and adds a **durable revision floor**:

- The edge persists the highest revision it has admitted, with the record
  itself, to its data directory (write to a temporary file, fsync, rename).
  It loads both at start, before any engine socket opens. The mirror also
  persists the time of its last good read, so a restart is not a fresh
  read (sim, plan 1).
- A record whose revision is below the floor is refused, at runtime and
  after a restart. An equal revision must be byte-identical.
- Stale (no good read for ten minutes, or a record signed more than ten
  minutes ago), expired, or never read: the grant table is **empty**, so
  every engine call is refused. A persisted record that has since expired
  or gone stale counts as expired. The mirror refuses a stale record like
  an expired one (ruling R5; sim, plan 1, final review).
- If the floor cannot be persisted, the edge keeps refusing engine calls
  until it can, and logs it once per minute.
- **The freshness bound (ruling R5; sim, plan 1, final review).** A bucket
  writer can replay an older signed record: before any edge has read a
  revoke, or to a box with no persisted floor (new or rebuilt). Every
  record that grants a revoked key was signed before the revoke, and goes
  stale ten minutes after its signing, so a replay keeps a revoked grant at
  most until the replayed record's `signed_at` plus ten minutes. The
  worst-case revoke delay is therefore about ten minutes plus one poll,
  replay or not; a box that has read the revoke refuses the replay at once
  (its floor). The cost: engine calls fail once the console has been down
  for ten minutes.

### 5.4 The call path

**The engine socket (nested instances; codex R1-1, R1-6).** An
instance-level listener manager in the edge owns one socket per nested
instance: `/run/carlos/instances/<iid>/engines.sock` (at most 75 bytes).

- **Lifetime:** opened when the instance's run directory is prepared for
  its first unit of any kind (a seed, a run, a command), kept across
  generations and restarts of the instance's units, closed only when the
  route leaves the box (removal, a move's commit, a reset). It is not tied
  to a generation and never closed by one. It exists whether or not the
  app holds a grant, so a later grant needs no restart. Answering on a
  sleeping instance's socket does not wake the instance. All of this holds
  only on an armed box (§6.2); elsewhere no engine socket is
  open, and `CARLOS_ENGINES_SOCKET` still names the path, so an app finds
  the socket missing rather than the variable.
- **Mode 0666**, because the connecting process's identity is checked
  below, not by file permission.
- **Who is calling:** on every accepted connection, before a byte is read,
  the edge takes the peer's pidfd (SO_PEERPIDFD), reads the process's
  cgroup, and confirms through the pidfd that the process is still the
  same one. The connection is admitted only if the cgroup is inside
  `carlos-instances-<iid>.slice` for the instance this socket belongs to.
  A flat tenant (in `carlos-edge.service`'s cgroup), a sibling instance, or
  anything else is closed with no response. The socket's path proves
  nothing; the cgroup is the identity, and every nested unit kind runs in
  that slice (§3). A box whose kernel lacks SO_PEERPIDFD (before 6.5) opens
  no engine sockets and says so once.
- **Environment:** `CARLOS_ENGINES_SOCKET` is set for every nested unit kind
  the way `CARLOS_CONTROL_SOCKET` is for the run role (tenant config cannot
  set it): the run launch file, each generation's launch file, the seed's
  variables and the command unit's environment.

**The door (codex R1-5, R2-3).** A door caller whose `Host:` has no dot and
names an engine is an engine call only if the connecting process runs under
a uid other than carlos (SO_PEERCRED): an enrolled host-app user, whose door
key sits in files only that user can read. A caller running as carlos (a
flat instance, or anything else sharing the edge's uid) gets 404 `engine
not granted`, because every flat tenant can read every other's config
(`layout/preflight.go:181`, `activator/paths.go:35`), so a door key in a
flat app's hands proves nothing. When the uid passes and `doorRequest`
admits the key, the door mark is replaced by an engine mark for the key's account and app, and the
door's own scope check (`doorOwnsRoute`) does not run for it; the engine
grant check (step 3 below) replaces it. A door caller whose `Host:` is an
engine's real hostname reaches the internal route without an engine mark,
so it gets the internal refusal (§5.1) like any other non-engine call.

**On the calling box.**

1. The listener strips every client-address header and every `X-Carlos-*`
   header, as `doorRequest` does, and presents the box's own address.
2. The caller is the instance's route (`release.RouteAccount`, `Route.App`),
   or the door key's account and app.
3. `Host:` names the engine. Unknown name, ungranted app, or an empty grant
   table: 404 `engine not granted`, one body for all three.
4. The edge puts an engine mark (account, app, engine) on the request
   context, sets `Host:` to the engine's hostname, and refuses `Upgrade`
   requests with 400.
5. If this box homes the hostname it dispatches locally; otherwise it
   backhauls to the home box (§5.5's transport). A home reached through a
   customer-fleet tunnel is not supported: 502 `engine unreachable`.

**The hop.** An engine call's hop adds `X-Carlos-Hop-Caller:
<account>/<app>/<engine>`, and its MAC covers it: a second MAC form over
`"v2\n" + host + "\n" + ts + "\n" + caller`, verified with the same
dual-key rule. The receiver deletes the header with the other hop headers
on every request, verified or not, and turns a verified caller into the
same engine mark. A hop without a caller is an ordinary hop, unchanged.
The door, the engine socket and `acceptHop` all delete the header on the
way in, so only an edge can speak it.

**On the engine box.** An engine-marked request reaches a route only if the
route is internal, the mark's engine names the request's hostname in the
grant table, the table grants that engine to the mark's app, **and the
route is the engine hostname's own route (codex R4-2)**. An engine call is
resolved by exact hostname only, never through a path map or a wildcard
(`edge.go:1364`; path targets may cross apps within an account,
`domains/aliases.go:395`), so a grant for one engine can never dispatch to
another. An alias of the engine's hostname resolves to the same route and
is fine. Otherwise the internal refusal (§5.1). Then:

- the per-app cap (§5.5);
- the import gate, unchanged;
- **no password gate** (the platform authenticated the caller; a login page
  means nothing to a program);
- the meter and analytics, unchanged: calls count against the engine's own
  app, as any request to it does today;
- **no cache**;
- the activator's dispatch, which may wake a hibernating engine.

### 5.5 Limits and transport (codex R1-7)

- **Per-app in flight**, on the engine box: at most `per_app` concurrent
  calls per calling app. Over the cap: 429, `Retry-After: 1`, and a
  rate-limited log line naming the app. The cap is per serving box: an
  engine served by more than one box (replica placements) allows `per_app`
  on each.
- **Transport:** engine hops use their own transport, never the backhaul
  transport shared per destination: a 5-minute response-header timeout,
  and one 5-minute deadline over the whole call (upload, wake, conversion,
  response). The socket's server side carries the same deadline.
- **Body:** 64 MiB. A declared `Content-Length` over it is refused with 413
  before anything is forwarded. A body of unknown length is streamed and
  counted; at 64 MiB the edge aborts the upstream request and answers 413,
  so the engine may receive a truncated request it then discards.
- **Async work:** longer conversions use the engine's own async mode, whose
  webhook reaches the caller's public URL over the internet.

### 5.6 What the engine sees

The request as the app sent it, with `Host:` set to the engine's hostname,
the client-address headers replaced by the calling box's address, and the
app's own `Authorization` passed through, so an engine that keeps basic auth
keeps it as a second lock. No caller header reaches the engine; the engine
box's access log records the caller app for each call.

### 5.7 Box-side advisories

The hourly capability pass on each box logs once per pass:

- an internal route whose hostname is not named by any engine in the grant
  table (nobody can call it);
- an engine whose hostname this box serves on a route that is not internal
  (the engine is still public; calls to it are refused by §5.4's engine-box
  rule, so the name is useless until the route declares `doc_engine`);
- an internal route that declares `doc_engine` without the grant (no JIT
  or memory band; it will not render).

### 5.8 Watching an engine deploy (codex R2-4)

`carlos deploy` verifies a release by polling its public URL
(`deploy.go:1426`, `:1536`), which an internal route now refuses. Whether a
route is internal is box-local (a grant can make it so whatever the release
declares), so **every deploy target's watch carries a watch ticket (codex
R3-3)**. The edge honours it only on an internal route and strips it
everywhere else, where the watch proceeds exactly as today:

- The console mints it on the deploy request it already authorizes: signed
  with the root release key under its own domain (`carlos watch v1\n`),
  bound to one hostname, one version and **the probe path the watcher will
  poll (codex R5-3)**, expiring 5 minutes after minting. The probe comes
  from the same app setting the console already returns with the ship
  response (`console/apicli.go:2114`, `:2285`; used at `deploy.go:404`);
  no probe means `/` only.
  It is returned to the CLI with the promote response.
- In bucket mode, where no console is involved, the CLI signs the ticket
  itself with the root release key it already holds to sign the promote,
  over the probe path it is about to poll.
- **Renewal (codex R4-3):** the CLI asks for a fresh ticket for the same
  hostname and version every 4 minutes for as long as the watch runs
  (`--timeout` may be longer than one ticket's life, `deploy.go:75`). The
  console re-authorizes the request with the same deploy credential and
  refuses a hostname or version the deploy did not name; in bucket mode the
  CLI re-signs.
- The CLI sends it as `X-Carlos-Watch-Ticket` on each poll. The home box's
  edge admits a request carrying a valid ticket to the internal route only
  for `GET` or `HEAD` with no body, on `/` or the ticket's signed probe
  path, strips the header, and dispatches as usual (no cache, as the watch
  already busts it). Anything else with a ticket gets the internal refusal.
- The ticket crosses a backhaul as a header and is verified only on the home
  box. It names no app and grants no engine call.
- A ticket is a credential for reading a version, not for converting a
  document: the probe paths are the app's own choice, and Gotenberg's
  `/health` is a fine probe.

### 5.9 The engine box is dedicated (codex R12-1, R12-2; ruling R3)

I1 keeps a declaring build from starting before the edge refuses the
public, but the edge is not the only way to a process on its box. Round 12
found two others: the engine binding a port of its own on every interface
(the template allows it, `carlos-app@.service:21`, `:66`; the capability
validator leaves high ports to the app, `caps/caps.go:401`), and another
route row whose upstream is the engine's backend (`carlos add` with a
`service` kind and an explicit address, `cli.go:126`, `:276`; the edge
proxies by that row's own address, `edge.go:760`, `:889`, `:1790`). The
README already tells operators to run a doc engine on a box of its own.
This spec makes that a rule the box enforces:

- **One internal group per box, and nothing else that proxies.** On a box
  that homes an internal group, every route row whose upstream is a local
  process (any `instance`, `service` or sidecar row: a socket path or an
  address) must belong to that group. Rows the edge answers without a
  local process (`static`, the status host, the tunnel and steering
  handlers) are unaffected. An unclassified (`NULL`) route homes its box
  like an internal one for this rule, since otherwise a declaring `carlos
  run` on a shared box is publicly reachable; the cost is that an
  unclassifiable route on a shared box refuses its proxying neighbours
  until it is classified (sim, plan 1).
- **The fence is box-wide on an engine's box (codex R13-2).** The fence a
  declaring launch waits for (§5.1) cancels every non-engine proxy request
  the box's edge admitted before it, on any row, not only the group's. That
  includes a request admitted through a row since deleted or replaced, which
  still holds its old proxy (`registry.go:988`, `edge.go:1790`): the
  epoch is the box's, not the row's. On a dedicated box that is cheap.
- **The launch gate checks it.** Before it waits for the fence, a declaring
  launch confirms the rule holds on the box. If any other proxying row
  exists, the launch is refused (the wake fails), and one line per pass
  names each offending row.
- **The edge keeps it.** While an internal group is homed on the box, the
  edge refuses to serve any proxying row outside the group: it gets the
  internal refusal, and the pass logs it with the move or remove command.
  A row placed on the box later (provisioning, a move, `carlos add`) is
  therefore never served there; the engine keeps running.
- **Ingress precondition, stated (ruling R3; codex R13-1).** The engine's
  box admits connections **from anywhere, public or private, including the
  deployment's other boxes**, only on the edge's ports (80 and 443, plus
  whatever the operator keeps for administration), by its security group or
  host firewall. Engine traffic between boxes enters through the home
  edge's backhaul, never a direct connection to the engine's port: a
  `service` row on another box aimed at the engine box's address is a
  violation of this precondition, and the survey looks for such rows. The platform cannot see that boundary from inside the box. The
  rollout survey (§7) checks it for every engine box, and the README's
  doc-engine preconditions gain it as a third, beside the loopback resolver
  and the dedicated box.

With the box dedicated, a port the engine binds on its own can be reached
only through that firewall, and no other row on the box can point at it.

## 6. Correctness rules

### 6.1 Identity

On the calling box the caller is the kernel's answer about the connecting
process's cgroup, or a verified door key. On the engine box it is a verified
hop caller, or the local engine mark. No header the app wrote names a
caller. Every box holds the mesh key, so a compromised box can call any
engine as any app; that is the trust the hop already places in boxes, stated
and not solved.

### 6.2 Flat layout

**Engine sockets need an armed box (codex R3-2, R4-1).** Any tenant code
running as uid carlos outside an instance slice can write its nested
siblings' directories (`layout/preflight.go:181`), including the
environment file a nested launch reads (`activator/backend_exec.go:2701`),
plant a library and an `LD_PRELOAD`, and so run code inside a granted
sibling's slice, which the cgroup check then accepts as that sibling. Three
launch paths run tenant code that way today: a `flat` instance, a `foreign`
one (`activator/layout.go:103`; both go through the exec-child path,
`backend_exec.go:1826`), and an app's worker, which spawns into
`carlos-exec@<host>` whatever its layout (`StartWorker`, `opset.go`). The
predicate is therefore positive: **the only tenant code on the box runs in
instance slices or as unit-backed DynamicUser services** (which cannot write
carlos-owned directories).

- **Arming is an operator act on the box:** `carlos engines arm`. It
  refuses unless the box's layout intent is nested, every exec-backed
  `instance` route classifies `nested`, no route on the box spawns workers,
  and every `carlos-exec@` cgroup on the box is empty **(codex R5-2)**.
  Unit-backed routes (`backing=unit`, `carlos-app@`, DynamicUser) are
  admitted: their socket is `/run/carlos/apps/<host>.sock`, which
  `Classify` reports as `foreign` (`activator/layout.go:103`), so they are
  judged by backing, not layout. Platform-minted rows are judged through
  their execution owner. An exec-backed route that classifies `foreign`
  still blocks arming. It records the armed state in
  the box registry. The edge opens engine sockets only on an armed box.
- **While armed, the activator refuses every unisolated launch**: a flat or
  foreign start and a worker spawn are refused with one log line naming the
  route and `carlos engines disarm`. The check sits in the launch path, so
  no unisolated process can start on an armed box, rather than being
  noticed afterwards. Provisioning a route that would classify flat or
  foreign on an armed box leaves it unlaunched until the box is disarmed.
- **Disarming** (`carlos engines disarm`) closes every engine socket first,
  then clears the state; launches follow. Re-arming is the same operator act
  as arming, with the same checks.
- **Residual, stated:** arming vouches for the present, not the past. A
  nested instance's files on a box that ran unisolated tenants before it was
  armed are only as trustworthy as that period, which is true of isolation
  as a whole. `carlos engines arm` says so when it succeeds; an operator who
  doubts a box's past rebuilds its instances from the bucket first.
- **Consequence, stated:** an app that uses workers cannot run on an armed
  box until workers move into instance slices. That is a separate change.

**(codex R2-3)** A flat instance cannot call an engine. It has no engine
socket, it cannot use a nested sibling's (its process is not in that
sibling's slice), and the door refuses engine calls from uid carlos. Flat
tenants share one uid and can read each other's files, so no credential a
flat app holds can say which app it is. An app that needs an engine moves to
the nested layout, the default since v0.25.0. A host-app process under its
own uid uses the door; enabling the door on a box (`CARLOS_EDGE_DOOR`),
enrolling the user (`carlos bootstrap --socket-consumer`) and minting the
key are operator steps in the runbook, not part of this build.

### 6.3 The public never reaches an internal route

- The mark lives in the box's own registry and survives restarts. It is set
  before a declaring release can serve, with the edge's fenced state as the
  barrier, and cleared only by the operator's `--public`, which stops the
  group first; an unclassified route is refused (§5.1). No bucket read
  happens at serve time.
- The check runs on the final upstream's route, after every re-resolution,
  before every dispatch branch (§5.1).
- Old non-home boxes backhaul public requests as ordinary hops; the home box
  refuses every hop without a verified caller for an internal route.

### 6.4 Freshness

A grant or revoke reaches every edge within one 30 s interval of the
record. A box that cannot read the bucket drops every grant ten minutes
after its last good read; an expired record (a console down for 72 hours)
drops every grant. The calling box and the engine box check independently,
so a revoke is in force once either has read it.

## 7. Deploy order and rollout

- **Engine boxes.** For each box that will home an engine: confirm its
  public ingress is the edge's ports only (security group or firewall), and
  move every other app off it. The launch gate refuses a declaring build on
  a box that still has one.
- **Survey first.** On dev and the flagship, list every route whose live
  release declares `doc_engine` or holds the grant, and who calls it. A
  route with public callers is cut off the moment its home box takes this
  build (R1), and its callers can only switch once the console and their own
  boxes have it too, so that route's cut is scheduled with its owner. A
  route with no public callers needs nothing.
- **Boxes, then the console**, in the usual order: dev boxes, dev console,
  check, flagship boxes, flagship console. Engine calls need the calling
  box, the home box and the console all on this build.
- **Arm the calling boxes.** On each box whose instances will call an
  engine, the operator runs `carlos engines arm` once the box is fully
  nested and runs no workers. The survey lists, per box, what blocks
  arming.
- **Then names and grants.** With no `engines.json` the grant table is empty
  and every engine socket answers 404; nothing else changes. `carlos engines
  add` and `grant` follow the roll, and callers switch to the socket.
- A calling box on an older build cannot make engine calls (no socket, no
  caller on the hop). A home box on an older build serves its engine
  publicly, which is today's behaviour and why boxes roll before anything is
  named.
- A box whose kernel lacks SO_PEERPIDFD opens no engine sockets; the survey
  checks kernels (Debian 13's 6.12 has it).

## 8. Tests that prove it

Production-wired: real listeners, real hops between two edges in one test
process, real unix sockets and a real cgroup check where the kernel allows
(the cgroup reader takes a root path for fixtures, as `instancectl` does).

- Public requests to an internal route through its host, an alias, a path
  map, a wildcard, a mixed-case spelling and a login POST whose `next`
  re-resolves to it: all the internal refusal (a plain 404, never a redirect,
  even with a fallback URL set), on :443 and on the home box's :80; the ACME
  challenge still answers.
- A public request reaching a non-home box is backhauled and refused by the
  home box.
- A route adopted from a release declaring `doc_engine` is internal before
  its first request, on a fresh registry, with the bucket unreachable after
  adoption.
- Becoming internal: with the edge's reload held back, adoption of a
  declaring release does not repoint; once the edge loads the mark, it does,
  and no public request reaches the new release at any point. Becoming
  public: the mark clears only after the non-declaring release is live; a
  failed adoption leaves it set.
- An upgraded box with an unclassified declaring route and a missing staged
  manifest refuses it publicly before its first request, and classifies it
  from the bucket.
- A door engine call from uid carlos is refused with a valid, granted key;
  the same key from an enrolled host-app uid succeeds.
- A deploy watch with a valid ticket reads the version on `/` and the probe
  path; the ticket on a POST, another path, an expired ticket or one for
  another host gets the internal refusal.
- A public request admitted before a `0`→`1` mark and held (mid-proxy, and
  queued behind a wake) is cancelled before the edge acknowledges, and never
  reaches the declaring release.
- `carlos engines arm` refuses with a flat route, a foreign route, a route
  with workers, or a non-empty `carlos-exec@` cgroup on the box; on an
  armed box a flat start, a foreign start and a worker spawn are refused
  before any process starts; `disarm` closes every engine socket before
  launches resume.
- An engine call with `Host: A` whose path maps to engine B's route is
  refused, though the caller holds A's grant.
- A platform-minted hostname of an internal route is refused publicly and
  admits engine calls exactly as the owner does; a minted hostname of a
  public route keeps working across the upgrade; a static platform domain
  and a static canary keep serving.
- Deploying a non-declaring release over a declaring one, with and without
  generation rollouts, leaves the route internal; `--public` refuses while
  the live manifest declares or a grant remains, stops the group, refuses if
  a cgroup stays non-empty, and otherwise clears; the next request wakes the
  non-declaring build.
- I1 per launch path: for each of an exec child, a nested run unit, a
  generation swap, a seed, a command, a unit-backed start and a wake plan's
  fallback to a declaring `GoodVersion` after a failed first wake, a launch
  of a declaring build under `0` sets `1`, completes the fence, and cancels
  the public request that caused it before the process starts.
- With the edge's reload held failing: an adoption of a declaring build
  times out with `1` written; a later recovery wake that selects a
  declaring `GoodVersion` is refused, no declaring process starts, and the
  public request gets the wake-failure answer.
- `carlos ship` refuses a capability block with both `doc_engine` and
  `ports`; a box with an existing port grant on a route that becomes
  internal removes the listener before the fence completes, and a direct
  connection to the port is refused afterwards.
- A public request admitted through a `service` row that is then deleted,
  held before dispatch, is cancelled by the next declaring launch's fence
  and never reaches the engine.
- On a box homing an internal group, a `service` row pointing at the
  engine's address, or any other app's instance, gets the internal refusal;
  a declaring launch on a box that has one is refused; static and status
  rows keep serving.
- Upgrade classification marks a route internal when its live version is
  non-declaring but another staged version of the app declares.
- A TCP-backed instance's minted hostname keeps serving across the upgrade;
  two TCP instances of one app with an unrecorded owner are refused until
  the reconciler records it.
- A `doc-engine` grant on a minted row of a non-declaring release makes the
  owner and every minted row internal, through the barrier.
- `carlos engines arm` succeeds on a box with nested callers and a
  unit-backed engine.
- A watch with a probe of `/healthz` reads it with a ticket; the same
  ticket on `/other` gets the internal refusal.
- A watch with a 10-minute timeout whose target turns healthy at minute 6
  succeeds on a renewed ticket.
- A watch ticket on a public route is stripped and the watch behaves as
  today; on a grant-only internal route it reads the version.
- A socket call from a process in the instance's slice reaches the engine on
  another edge; from a process outside the slice (a flat child, a sibling
  slice) the connection is closed unread.
- A forged `X-Carlos-Hop-Caller` from a client, a door caller or an app
  through the socket never reaches the engine box's decision.
- A hop signed with a caller the engine box's table does not grant is
  refused even though the calling box admitted it.
- A restart with a persisted floor refuses a replayed older record.
- The per-app cap: the n+1th concurrent call from one app gets 429 while a
  second app's call proceeds.
- The socket survives a generation swap and is up before a seed runs; every
  nested unit kind sees `CARLOS_ENGINES_SOCKET`; tenant config cannot set it.
- A 45-second conversion completes; a declared 65 MiB body gets 413 before
  the engine sees a byte.

### Simulation (#335 plan 1)

`internal/sim/engines` is a test-only model of this spec: 40 named rules,
each with a mutation that turns it off, checked against the invariants
INV-P (no public request is delivered to a declaring build), INV-I1,
INV-AV (at the end, every row of a `0` group answers a public request),
INV-G (an engine call reaches only the engine's exact route; no box admits
one on a table that was never read, has expired, has had no good read for
ten minutes or holds a record signed ten minutes ago; none is granted more
than ten minutes plus one poll after a revoke, replay or not, nor by a box
that has read the revoke), INV-ID (the caller the engine box records is the
app whose code made the call) and INV-ARM (no unisolated tenant process on
a box while an engine socket is open there).
`internal/sim/engines/mutate.sh 3000` runs each
mutation through a 3000-seed random walk and through the rule's
hand-scheduled witnesses; `expected-outcomes.txt` lists the rules caught
only in combination.

Sweep after the final review, 2026-10-21, on the tree committed as
"enginesim: clean baseline env, immediate listener checks,
completed-transition coverage" (a337fb28). It adds three rules
(`new-group-fenced`, `clear-launch-lock`, `grant-freshness-bound`), the
freshness bound of ruling R5 (the walk now replays any older record at any
time) and the oracle changes of the final review, so its counts are not
comparable with the first sweep's (parent 4648305e). The unmutated sweep:
0 of 3000 seeds violated, 0 residuals. Result: 36 rules CAUGHT, 4
EXPECTED-MISS, none MISSED, ERROR or STALE (sim, plan 1, final review).

| Rule | Verdict | Sweep catches (of 3000) | First seed | Witness |
|---|---|---|---|---|
| `set-before-repoint` | CAUGHT | 2448 | 2 | violated |
| `fence-needs-reload` | CAUGHT | 412 | 6 | violated |
| `fence-drains-admitted` | CAUGHT | 217 | 31 | violated |
| `fence-box-wide` | CAUGHT | 13 | 481 | violated |
| `gate-adopt` | EXPECTED-MISS | 0 | - | held |
| `gate-swap` | CAUGHT | 2294 | 2 | violated |
| `gate-fallback` | CAUGHT | 265 | 18 | violated |
| `gate-recovery` | CAUGHT | 262 | 5 | violated |
| `gate-seed` | CAUGHT | 1711 | 2 | violated |
| `gate-cmd` | CAUGHT | 1734 | 2 | violated |
| `gate-waits-when-written` | CAUGHT | 287 | 6 | violated |
| `unreadable-declares` | CAUGHT | 2012 | 3 | violated |
| `live-link-only-adoption` | CAUGHT | 857 | 2 | violated |
| `mark-on-final-upstream` | CAUGHT | 2832 | 1 | violated |
| `classify-null-refused` | CAUGHT | 476 | 13 | violated |
| `classify-staged` | CAUGHT | 66 | 8 | violated |
| `minted-follow-owner` | CAUGHT | 802 | 1 | violated |
| `grant-marks-group` | CAUGHT | 484 | 549 | violated |
| `static-minted-public` | CAUGHT | 2981 | 1 | violated |
| `owner-nonempty-unique` | CAUGHT | 42 | 2359 | held |
| `public-checks-live` | CAUGHT | 851 | 2 | violated |
| `public-stops-group` | EXPECTED-MISS | 0 | - | held |
| `public-holds-lock` | CAUGHT | 62 | 37 | violated |
| `public-confirms-empty` | CAUGHT | 49 | 8 | violated |
| `ports-excluded` | CAUGHT | 1995 | 1 | violated |
| `dedicated-launch-check` | EXPECTED-MISS | 0 | - | held |
| `dedicated-edge-refusal` | CAUGHT | 1138 | 2 | violated |
| `ingress-precondition` | CAUGHT | 1248 | 4 | violated |
| `cgroup-identity` | CAUGHT | 171 | 32 | violated |
| `door-uid` | CAUGHT | 85 | 43 | violated |
| `hop-caller-signed` | CAUGHT | 137 | 41 | violated |
| `engine-box-recheck` | CAUGHT | 44 | 79 | violated |
| `exact-route-engine` | CAUGHT | 206 | 13 | violated |
| `floor-persisted` | CAUGHT | 4 | 13 | violated |
| `stale-clears-grants` | EXPECTED-MISS | 0 | - | held |
| `armed-refuses-unisolated` | CAUGHT | 2738 | 2 | violated |
| `arm-checks-box` | CAUGHT | 2321 | 1 | violated |
| `new-group-fenced` | CAUGHT | 0 | - | violated |
| `clear-launch-lock` | CAUGHT | 65 | 42 | violated |
| `grant-freshness-bound` | CAUGHT | 455 | 2 | violated |

The four EXPECTED-MISS rules are each caught in combination, by their
witness and by the sweep (200 seeds: 187, 12, 125 and 59 catches):
`gate-adopt` with `set-before-repoint` (adoption's set-before is the same
gate, reached earlier), `public-stops-group` with `public-confirms-empty`
(step 3 refuses while the declaring generation still runs),
`dedicated-launch-check` with `dedicated-edge-refusal` (the edge refuses
the proxying row once the group is internal), and `stale-clears-grants`
with `grant-freshness-bound` (a table's record was signed no later than
its last good read, so the `signed_at` bound empties a stale table first).
`owner-nonempty-unique`'s witness needs `dedicated-edge-refusal` gone too,
so it holds alone; the sweep catches the rule alone through INV-I1 (a
misgrouped `0` row makes the owner's group read public).
`new-group-fenced` is caught by its witness only: the walk's dedicated
launch check refuses every declaring launch in a new group beside the
starting routes, so only a box that homes nothing reaches it.

## 9. Docs

`docs/site/cli.md`: a `carlos engines` section. The README's doc-engine
passage gains the public refusal and the deploy order. A short "Calling an
engine from your app" passage with a Go example (a unix-socket transport,
`http://gotenberg/forms/chromium/convert/html`). All user-facing strings go
through copy review before they ship.

## 10. Provider boundary

Core only. The mirror reads a `release.Store` (S3 or a directory); signing
uses the root release key; the hop is the existing mesh; the peer check is
Linux, as the whole nested layout already is. On AWS with region HA a hop
within one region uses private addresses; everywhere else it crosses the
public internet under TLS and the hop MAC, as backhaul does today. A non-AWS
deployment loses nothing. "Fleet-internal" means only the platform can call
it, not only over a private network.

## 11. Calls I made (for review)

- Engine selection by `Host:` name on one socket, so a client's base URL is
  `http://<name>`.
- Limits: 64 MiB body, 5 minutes end to end, per-app cap default 4, per
  serving box.
- Engine calls skip the password gate and the cache; the import gate, the
  meter and analytics run as for any request.
- No owner exception on internal routes: the engine app grants itself.
- Tunnel-homed engines are refused (502), not supported.
- Flat instances cannot call engines at all (§6.2); host-app processes
  under their own uid use the door, enabled by the operator.
- Engine sockets only on boxes an operator armed; an armed box refuses
  flat, foreign and worker launches rather than closing sockets afterwards.
- A route never becomes public automatically; `carlos route --public` is
  the only way back, and it stops the group.
- One internal group per box, enforced: a second engine needs a second
  box.
- Unclassified routes are refused publicly (fail closed) even though that
  can make a route with no readable manifest unreachable while the bucket
  is down.
- The deploy watch ticket rather than skipping verification for engines.
- The engine box re-checks the grant (G4) rather than trusting the calling
  box.

## 12. Disposition of review findings

| Finding | Disposition |
| --- | --- |
| R1-1 flat tenant borrows a nested socket | Accepted: peer cgroup check (§5.4) |
| R1-2 aliases and path routes stay public | Accepted: mark on the route, checked on the final upstream (§5.1) |
| R1-3 new box starts public | Accepted: mark from adoption, no bucket at serve time (§5.1, R1) |
| R1-4 revoked grant resurrected after restart | Accepted: durable floor; fresh-box residual stated (§5.3) |
| R1-5 door exceptions | Accepted: owner exception dropped, door-to-engine transition defined (§5.4) |
| R1-6 socket lifecycle | Accepted: instance-level owner (§5.4) |
| R1-7 transport and body | Accepted: engine transport, streamed body with abort (§5.5) |
| Builder Q1 unit-backed callers | Ruling R2: out of scope |
| Builder Q2 door for flat apps | Superseded by R2-3: flat instances cannot call engines; host-app door is an operator step (§6.2) |
| Builder Q3 engine owner's own processes | Grant itself (§5.1) |
| Builder Q4 gate, meter | §5.4 |
| Builder Q5 socket lifetime | §5.4 |
| Builder Q6 several homes, tunnels | Per-box cap; tunnels refused (§5.4, §5.5) |
| R2-1 stamp does not precede serving | Accepted: set before, clear after, edge-loaded barrier (§5.1) |
| R2-2 upgrade serves unclassified engines | Accepted: NULL state, classification before listeners, NULL refused (§5.1) |
| R2-3 flat callers borrow a door key | Accepted: door engine calls need a non-carlos uid; flat cannot call (§5.4, §6.2) |
| R2-4 deploy watch cannot verify | Accepted: watch ticket (§5.8) |
| R2-5 :80 on non-home boxes; unknownHost redirects | Accepted: plain 404 refusal; no-redirect scoped to the home box (§5.1) |
| R3-1 requests admitted before the mark | Accepted: the acknowledgement waits for cancelled pre-mark requests to finish (§5.1) |
| R3-2 flat tenant injects code into a nested caller | Accepted, then widened by R4-1: armed boxes (§6.2) |
| R3-3 grant-only routes lack a watch ticket | Accepted: every deploy target carries one (§5.8) |
| R4-1 foreign instances and workers are unisolated too | Accepted: positive armed state, unisolated launches refused while armed (§6.2) |
| R4-2 a path map widens a grant to another engine | Accepted: engine calls resolve by exact hostname to the engine's own route (§5.4) |
| R4-3 watch tickets expire mid-watch | Accepted: renewal every 4 minutes (§5.8) |
| R5-1 platform-minted rows never classify | Accepted: execution owner's mark and epoch (§5.1) |
| R5-2 arming rejects unit-backed routes | Accepted: exec-backed routes by layout, unit-backed by backing (§6.2) |
| R5-3 no source for the probe path | Accepted: the ticket signs the watcher's probe path (§5.8) |
| R6-1 grant on a minted row ignored | Accepted: grants anywhere in the group mark the group (§5.1) |
| R6-2 static platform domains refused | Accepted: inheritance only for minted instance rows (§5.1) |
| R7-1 clear-after reopens a still-serving engine under rollouts | Accepted in rev 8, superseded in rev 9 by the operator clear (§5.1) |
| R7-2 TCP-backed minted rows have no owner | Accepted: owner recorded on the row; unique non-empty fallback, TCP included (§5.1) |
| R8-1 upgrade classifies a running declaring build public | Accepted: staged declaring versions classify `1`; residual stated (§5.1) |
| R8-2 a sleeping unit-backed route cannot clear | Accepted, with R7-1: no automatic clear; `--public` stops the group and checks cgroups (§5.1) |
| R9-1 `--public` then a fallback to a declaring `GoodVersion` | Accepted: invariant I1 enforced at every launch (§5.1) |
| R10-1 a written but unfenced `1` lets a declaring launch through | Accepted: I1 is about the fenced mark; every declaring launch waits for the fence (§5.1) |
| R11-1 a granted port bypasses the edge | Accepted: `doc_engine` + `ports` refused at ship; no listener for internal groups (§5.1) |
| R12-1 an engine binds its own public port | Ruling R3: dedicated box; ingress beyond the edge's ports is a stated, surveyed precondition (§5.9) |
| R12-2 a separate service row proxies to the engine | Ruling R3: no proxying row outside the group on an engine's box, enforced by the launch gate and the edge (§5.9) |
| R13-1 another box proxies privately to the engine | Accepted within R3: ingress precondition covers private sources; survey looks for such rows (§5.9) |
| R13-2 a request admitted via a deleted row | Accepted: the fence is box-wide on an engine's box (§5.9) |