# `carlos run`: a member runs an app command against one instance (#259)

**Status:** rev 4, 2026-10-08. Rev 3 answered round 2 (§12); rev 4 answers round 3 (§13). Rev 1 (c66792a3) was approved by Paul.
Rev 2 answers Astra's round 1 (`CODEX-FINDINGS-RUN-AND-SEEDED-R1.md`;
findings 1 to 7 are this spec's). Each change is marked **(codex N)**, and
§11 lists the dispositions.

**Discussion:** amadan carlos/platform #259. Companion spec:
`2026-10-08-seeded-instances-design.md` (#260, #261).

## 1. The problem

Every one-off command against a live instance goes over
`aws ssm send-command` as root on the box today. On 2026-10-06 a titogo
session imported 22,462 check-ins that way (titogo PR #4048). It:

- ran the app's own binary against the instance's database;
- read a credential from the app's config;
- staged the input file through presigned URLs.

The canary recipe (`titogo add`, `auth grant`, `seed`) has the same shape.

Two members of Tito's team can sign in to the console as members of `tito`,
but they can't do any of this without root on a box every tenant shares.
The platform has no door for "run this app's command against that
instance".

## 2. Rulings (Paul, 2026-10-08)

1. **Who may:** any person who is a member of the account, the same rule as
   `carlos db pull`. Service credentials are refused. A member can already
   read every instance's database and publish to non-stable channels, so
   this grants nothing new.
   *Amended 2026-10-24: a `run` service credential may, for the commands it
   was granted. See `2026-10-24-service-runs-design.md`.*
2. **Output transport:** the run's own chunks in the deployment bucket,
   polled by the CLI through the console. There is no new network path; the
   console still never connects to a box.
3. **Platform-run instances only.** controld-run (`titogo@`) instances are
   converted (#261) rather than served.

## 3. What it does

```
carlos run --app <app> --instance <host> [--file <path>]... [--timeout 30m] -- <arg>...
carlos run --follow <id>
```

- It runs the release the instance follows, as
  `/opt/carlos/live/<host> <arg>...`. The caller never names the binary.
- It runs as the instance: the same uid, namespace, bind mounts, slice and
  environment. Never root, and never the agent.
- It streams stdout and stderr to the terminal, about one second behind,
  and exits with the command's own status.
- It records who ran what, where, and how it ended.

**Defaults:**

| | |
|---|---|
| hibernated instance | woken first, so its database is restored and replicating |
| live instance | left running; the command opens the same database beside it (WAL) |
| stdin | none; argv only |
| Ctrl-C | cancels the run on the box |
| `--timeout` | 30 minutes by default, 6 hours at most |
| `--file` | repeatable; 64 MB in total |
| concurrency | one run at a time per instance |
| layout | refused unless the instance is nested |
| host lifecycle claim | refused while the box's claim control is armed (§3.1) |

**Not in this version:**

- `--version`. The trampoline execs only `/opt/carlos/live/<host>`
  (`internal/instanceexec/exec.go:72`), and the control service binds that
  link to the instance's own build.
- `--stopped` (stop, run, start).
- `--record` (keep the output).
- An interactive shell.

### 3.1 With the host lifecycle claim armed (codex 2)

The claim code is on main but not active anywhere. Once a box arms claim
control, two things stop working:

- instancectl answers 503 to every legacy start, seed and stop route
  (`claimAwareLegacy`, `internal/instancectl/server.go:840`);
- `RoleLaunch.Start` refuses every launch until the isolated root protocol
  carries reservations (`internal/activator/rolelaunch_linux.go:23-28`).

So on such a box `carlos run` refuses with `refused: "claim"`, the same way
it refuses on a flat box. Supporting it there is part of the claim's
activation work (#250), not this spec. That work must add `cmd` to the
reservation protocol beside `main`, `seed` and `sidecar`, and must settle
the cmd unit's producer before the generation is published. #250 gets a
note saying so.

## 4. The command's contract

This is what the app's binary sees. It goes into the `docs/site/cli.md`
section "What an instance is started with".

- **argv:** exactly what followed `--`, after the binary. The platform
  appends nothing, unlike a seed's `--db` (`seedArgv`).
- **Environment:** a new composer, `cmdVars` (codex 7). It takes the
  instance's file environment (the config bundle with `<host>.env` on
  top), then:
  - removes `CARLOS_CONTROL_SOCKET` and `CARLOS_PURGE`, as `seedVars` does
    (`backend_exec.go:1359`);
  - sets or removes `CARLOS_REGION` from the backend's region, exactly as
    the serving composer does (`backend_exec.go:1340-1344`);
  - sets `CARLOS_HOST`, keeps `CARLOS_ADMIN_TOKEN`, and adds the three
    variables below.

| Variable | Value |
|---|---|
| `CARLOS_DB` | the instance's database path, the one `--db` names when the instance starts |
| `CARLOS_RUN_ID` | this run's identifier |
| `CARLOS_RUN_DIR` | a directory holding the `--file` uploads under their base names. Always present, even with no files. Deleted after the run |

- **Working directory:** the instance's data directory, as for every
  template (#256 item D).
- **Signals:** SIGTERM on cancel, timeout or an instance stop (§5.3), then
  SIGKILL 20 seconds later (`KillMode=mixed`, `TimeoutStopSec=20`).

Secrets reach the command only through its environment. The platform never
puts them in argv and never prints the environment.

## 5. The flow

### 5.1 Request (CLI to console)

`POST /api/cli/apps/{account}/{app}/instances/{host}/runs`, multipart: a
`meta` part (`argv`, `timeout`, file names and sizes), then one part per
file. The console streams the file parts straight to the bucket (the F8
rule: bytes go through the console, never through a presigned URL). The
whole request is capped by `http.MaxBytesReader` at 64 MB plus the meta
allowance.

The console refuses, in this order, each with its own status and message:

1. A service credential (`cliRoute` with the zero `svcrole.Set`).
2. Not a member of the account: the byte-identical 404 `apiAccountApp`
   gives.
3. Over budget (429), on the instance verbs' per-token window.
4. No record for the host, or a record of another app: 404.
5. The record isn't platform-run exec, or is pooled, or is placed on a
   customer's fleet: 409.
6. A move is in flight (`move` state other than none): 409.
7. A run is already open on the instance: 409, naming its id and caller.
8. Bad input, 400:
   - argv empty, more than 256 arguments, over 64 KB in total, or
     containing a NUL byte;
   - a timeout outside 1 second to 6 hours;
   - files over the cap, or a file name that isn't a plain base name.

9. The instance is an import that isn't committed yet (seeded spec §5.4):
   409, "finish or abort the import first" (codex R2-2).

If none apply, it writes these under the app's own subtree
(`release.AppPrefix`, so the owner gate maps onto them and app trash and
purge sweep them):

- `runs/<id>/in/<name>` for each file;
- `runs/<id>/request.json`: host, the record's incarnation
  (`hostops.Record.Incarnation`: account, app, host and `CreatedAt`), argv,
  timeout, the file list with sizes and sha256, the caller, and the time
  requested.

Then it writes the run onto the **instance record** itself, in one CAS on
the record's ETag (codex 1, R2-6):

```go
ActiveRun *RunState `json:"run,omitempty"`

type RunState struct {
    ID       string    `json:"id"`
    State    string    `json:"state"`    // queued | picked | cancel | done | cancelled | expired
    Box      string    `json:"box,omitempty"`
    BootID   string    `json:"boot,omitempty"`
    Deadline time.Time `json:"deadline"` // pickup deadline, then run deadline
}
```

Ordinary record writers carry `ActiveRun` through unchanged, the way they
carry `Maintenance` today (`provision.go:40-42`).

It then bumps `control/serial`, writes the `run-start` audit row (§6), and
answers `202` with the run id.

**One record, one CAS.** Runs, moves and imports all change the same
object, so they serialize on the record's ETag:

- a run is admitted only when there is no non-terminal `ActiveRun`, no
  move in flight, and no uncommitted import;
- a move request (`provision.go:1129`) is refused while `ActiveRun` is
  non-terminal;
- an import's abort is refused while `ActiveRun` is non-terminal, although
  run admission already refuses uncommitted imports.

Whichever write lands first wins. The loser's CAS fails, it re-reads the
record, and it refuses.

Deleting and recreating the host makes a new record with no `ActiveRun`,
so an old request can never run against the replacement. The box also
checks that `request.json`'s incarnation matches the record before it
launches (codex 3).

### 5.2 Pickup on the box (codex 3)

Restart's pattern (`internal/host/restart.go`) records the first sight of
an object and never acts on it (`:114`, `:165`). That's right for restart
and wrong here: it would drop runs queued before a box started, or before
it came to own a host. So pickup works like this instead:

- **When the box looks.** Reconcile already reads the instance records in
  its scope on every serial bump and at agent start, so the box reads
  `ActiveRun` from those same reads. A record the box has just come to own
  is covered by the same pass, and an idle box pays nothing extra.
- **Which runs it takes.** It acts on a `queued` run only for a host whose
  exec row it owns. Every other box ignores it.
- **Taking a run.** The box moves the run from `queued` to `picked`, with
  its box name, agent boot id and the run's deadline, in one CAS on the
  record.
  - If the CAS fails, the box re-reads. If the run is no longer `queued`
    (taken, cancelled or expired), the box leaves it alone.
  - Only the box whose CAS succeeded launches, and it checks the deadline
    and cancel state again right before launch.
- **A run this box picked but never finished.** At agent start, a run
  marked `picked` by this box under an earlier boot id has lost its unit:
  `BindsTo=carlos-edge.service` stopped it with the edge. The box writes
  its `result.json` as `interrupted: "agent restart"` and CASes the run to
  `done`. **A picked run is never executed again.**
- **No pickup** (codex R2-6). Sixty seconds after the request, the CLI asks
  the console to expire the run. The console CASes it from `queued` to
  `expired`, which is terminal.
  - If that CAS succeeds, the CLI reports "not picked up" and exits 3. An
    expired run can never be picked, so a box that turns up late can't
    run it.
  - If the CAS fails because a box picked the run in the meantime, the CLI
    goes on following it.

Once the box has taken the run:

1. **The layout check.** It refuses unless `activator.Classify` answers
   `LayoutNested` for the row and `nestedGate` passes. A refusal is a
   `result.json` with `refused: "flat"`, never a fallback to running the
   command as the agent. With claim control armed it is
   `refused: "claim"` (§3.1).
2. **The wake.** It wakes the instance if it is hibernated, through the
   ordinary wake path, so the database is restored and replication is
   running. A wake failure is `refused: "wake"`, with the wake's error.
3. **The files.** It downloads the files into
   `provision.InstanceDir(dataDir, host)/.carlos-run/<id>/` (codex 6:
   long hosts hash, `provision.go:2032-2045`). Each one is checked against
   its size and sha256 in `request.json`, written mode 0700 by `carlos`
   through `instancefs`, and no symlink is followed.
4. **The launch.** It attaches the run to the instance's activator entry
   (§5.3), then starts the unit.

### 5.3 The run is part of the instance's writer lifecycle (codex 1)

The command writes the instance's database, so it has to be inside every
rule that protects those writes, not beside them. The activator entry for
the host gains a `cmd` field: the run id, the unit's InvocationID, and a
done channel. Then:

- **Every way the instance stops** first stops the cmd unit and waits for
  it to settle, before replication is closed or the entry is forgotten.
  That covers:
  - hibernation (`stop`, `activator.go:2108`);
  - `Bounce` (`:1425`), which config sync and `carlos restart` use;
  - convergence that stops an instance;
  - a serving process that fails;
  - the move freeze;
  - agent shutdown.

  The ordering lives in `finishStop` (`:2355`), so a new stop path can't
  skip it. A run stopped this way ends as
  `interrupted: "<reason>"`, and everything it committed before the stop
  was replicated before replication closed.
- **Hibernation's idle timer doesn't fire** while `cmd` is set. That's
  the run hold. The other stops above aren't held off; they interrupt the
  run.
- **A restore never runs under the command.** The cmd unit starts only for
  an entry that is awake and replicating. While the entry is waking or
  stopping it isn't started, and the box waits up to 60 seconds for the
  wake.
- **A wake failure, or a stop before launch,** ends the run as
  `refused: "wake"` or `interrupted`. It is never queued for later.

### 5.4 The unit (codex 4)

A fourth template, `carlos-instance-cmd@.service`, is copied from
`carlos-instance-seed@.service`. Every sandbox line stays the same, and a
test fails if the two ever differ outside these lines:

- `Description=carlos exec-backed instance %I (cmd)`
- `ExecStart=/opt/carlos/carlos instance-exec cmd %I` and
  `ExecStopPost=/opt/carlos/carlos instance-exit cmd %I`
- `TimeoutStartSec=6h5m`, a backstop past the 6-hour cap. The agent
  enforces the run's own timeout.
- `StandardOutput=file:/run/carlos/logs/%I.cmd.sock` and
  `StandardError=file:/run/carlos/logs/%I.cmd-err.sock`, two listeners so
  the streams stay separate.

`Type=oneshot` and `RemainAfterExit=yes` stay, so the result is kept until
the agent reads it, as for the seed.

`instancectl` gains `RoleCmd = "cmd"` and
`UnitPrefixCmd = "carlos-instance-cmd@"` (`internal/instancectl/names.go`;
`run` is already the serving role). The control service starts, stops and
sweeps the new unit like the other three, through two routes:

- **`POST /cmd`** starts the unit and answers at once with its
  InvocationID. (`/seed` blocks for up to 150 seconds; this one doesn't.)
- **`GET /cmd/{host}?invocation=<id>`** is a bounded long-poll, at most 30
  seconds. It answers `{state, InvocationID, Result, ExecMainStatus,
  Code}`, where `state` is `running`, or `done` once that invocation's
  main process has exited. `done` covers both "active (exited)" and
  "failed".

When `GET` answers `done`, the agent records the result, then calls the
existing `/stop` for the host's cmd role and waits for the unit to settle,
exactly as `/seed` does (`server.go:431-448`). An invocation id that
doesn't match is `gone`, and the run ends `interrupted`.

The agent writes `launch.cmd.json` through `writeLaunch`, with argv
`[/opt/carlos/live/<host>, <arg>...]` and the environment from §4.

### 5.5 Output (codex 5)

The agent reads the two sockets and writes `runs/<id>/out/<seq>-<stream>`,
where stream is `o` or `e`. The seq is zero-padded and shared across both
streams, so the CLI can interleave them in the order the agent read them.
It writes a chunk every second or every 256 KB, whichever comes first.

Total output is capped at 64 MB. Past the cap the agent stops writing
chunks and records `truncated`; the command keeps running.

`result.json` is written **only after every chunk is durably written**,
and it names the last seq:

```json
{"exit": 0, "signal": "", "timed_out": false, "cancelled": false,
 "interrupted": "", "refused": "", "started": "...", "ended": "...",
 "box": "...", "last_seq": 17, "truncated": false}
```

The agent then deletes the run directory and clears the entry's `cmd`.

### 5.6 Following (CLI and console) (codex 5)

`GET …/runs/{id}?after=<seq>` answers the chunks after `seq` (a bounded
page) and `result.json` once it exists. The CLI:

- polls every second;
- writes `o` chunks to stdout and `e` chunks to stderr;
- keeps reading pages until it has every chunk up to `result.last_seq`;
- then sends `POST …/runs/{id}/ack` with `last_seq`.

The console refuses an ack whose `last_seq` doesn't match the result. The
CLI exits with:

| case | exit |
|---|---|
| command exited | its own status |
| command killed by a signal | 128 + the signal number |
| timed out | 124 (`timeout(1)`'s) |
| interrupted by an instance stop or agent restart | 125 |
| refused by the box (flat, claim, wake) or not picked up in 60 s | 3 |
| refused by the console | 2 |

### 5.7 Cancel and timeout

When you press Ctrl-C:

1. The CLI sends `POST …/runs/{id}/cancel`.
2. The console changes the run on the record by CAS:
   - `queued` becomes `cancelled`, which is terminal, so no box ever
     launches it (codex R2-6);
   - `picked` becomes `cancel`, and the console bumps the serial.
3. On `cancel`, the owning box stops the unit. If it hasn't launched yet,
   it doesn't launch at all.

A second Ctrl-C exits the CLI without waiting; the cancel still happens.
At the run's timeout the agent stops the unit itself. All of these end in
an ordinary `result.json`. Once the box has written `result.json`, it CASes
the run to `done`.

If the CLI goes away without cancelling, the run carries on to its end or
its timeout. `carlos run --follow <id>` attaches to an open or finished run
and prints every chunk not yet acknowledged.

### 5.8 Cleanup

- **The ack.** After a valid ack (§5.6), the console deletes `runs/<id>/`
  and clears `ActiveRun` from the record by CAS, but only if the record
  still names this run in a terminal state.
- **The console sweep.** It runs in the existing deadline-sweep loop and:
  - deletes any run whose result is more than 24 hours old, whether or not
    it was acknowledged;
  - expires any run still `queued` 60 seconds past its request (codex
    R3-3). It CASes `queued` to `expired` (losing to a pickup is fine),
    writes the `run-end` audit row as `expired`, and clears it. So a run
    whose caller disconnected never blocks the instance;
  - deals with any run still `picked` or `cancel` an hour past its
    deadline: it marks the run `done` with the audit end `lost`, and
    clears it.

  A run left in a terminal state is cleared the same way, so the next run
  is never blocked by one nobody acknowledged.

  The 24-hour bucket lifecycle rule from chat is replaced by this sweep.
  The runs live under each app's subtree, which a prefix rule can't match,
  and the sweep needs no infrastructure change.
- **The box sweep.** At agent start, the box deletes any `.carlos-run/`
  directory with no open run.

## 6. The record

Two audit rows are written through `Server.auditActivity`, with the
instance as the target:

- `run-start`: the caller, app, host, argv, file names and sizes, the
  release version the instance follows, and the run id.
- `run-end`: the run id, exit, signal, timed out, cancelled, interrupted,
  refused, started, ended and box; or `lost`. The console writes it the
  first time it reads `result.json`, whether from the CLI's poll or from
  the sweep.

Output never goes in the audit log, but argv does. A member who puts a
secret in argv has put it in the log, and `docs/site/cli.md` says so and
says to pass secrets through `--file`.

## 7. Security

- **Reach.** The command has exactly the instance's reach. That's why it
  is refused on a flat box, where it would have the agent's.
- **No widening for members.** A member can already pull the database and
  publish a build to a non-stable channel that the instance runs. `run`
  only executes the build the instance follows.
- **Box-written objects can be forged.** Any box can write anywhere in the
  deployment bucket today (#171), so a compromised box could forge another
  box's run state, output or result. That is the existing
  record-authority boundary, not a new one, and `run` adds no path from a
  forged object to any action.
- **File names** are plain base names, checked at the console and again by
  the agent.

## 8. Errors the member sees

Each has its own message, and every one names the instance:

- not a member, or no such instance;
- not platform-run, pooled, or on a customer's fleet;
- moving, or (for a move) a run is open;
- a run is already open (with its id and who started it);
- the box isn't nested yet;
- the box has claim control armed;
- not picked up within 60 seconds;
- the wake failed (with the wake's error);
- interrupted (with the reason);
- timed out;
- a file is too large, or the output was truncated.

Every CLI and console string goes through copy review before it is written
into the code.

## 9. Testing

- **Console door:**
  - each refusal in §5.1, in order;
  - the member-only rule;
  - the audit rows;
  - the record CAS under two concurrent run requests;
  - the run/move serialization in both directions;
  - cancel, ack (including a wrong `last_seq`) and the sweep.
- **Pickup:**
  - only the owning box acts;
  - a run queued before agent start is picked up;
  - a run queued before the box owned the host is picked up;
  - the pickup CAS with two boxes racing;
  - expiry winning over a late box, and a late pickup winning over expiry;
  - the sweep expiring a queued run whose CLI went away;
  - cancel before pickup never launching;
  - a request whose incarnation doesn't match the record being refused;
  - run, move and import abort racing on the record's ETag;
  - an uncommitted import refused at admission and again at launch;
  - a run picked before an agent restart ends `interrupted` and is never
    run again.
- **Writer lifecycle:**
  - each stop path in §5.3 stops and settles cmd before replication
    closes, shown by a write the command commits just before the stop
    reaching the replica;
  - the idle timer is held while a run is open;
  - no start while the entry is waking or stopping.
- **Unit and control:**
  - the cmd template equals the seed template outside the named lines;
  - `RoleCmd` parsing;
  - `/cmd` start, the long-poll for `running`, `done` and `gone`;
  - stop after done;
  - the sweep;
  - refusal while claim control is armed.
- **Output:**
  - chunk ordering across both streams;
  - the cap;
  - `result.json` only after the last chunk;
  - several pages of output after the result exists, all delivered before
    the ack.
- **CLI:**
  - interleaving;
  - every exit code;
  - Ctrl-C once and twice;
  - `--follow`.
- **On the dev box once built:**
  - a real run against a nested instance, including a write the instance
    then serves;
  - a run across a hibernation timer;
  - a run interrupted by `carlos restart`.
- **Cross-compile:** `GOOS=darwin` and `windows` builds stay green. The
  unit and control code is linux-only, behind build tags with stubs.

## 10. Rollout

It ships in the next release. It does nothing on a box until a member runs
it, and it refuses on flat boxes and on claim-armed boxes, so it needs no
switch. Tito's boxes need the nested layout before Doc and Vicky can use it
(#247). The claim's activation (#250) carries the work in §3.1.

## 11. Round 1 dispositions

| Finding | Disposition |
|---|---|
| codex 1 (P1) the hold misses stop paths | **Accepted.** §5.3: the run joins the instance's writer lifecycle, and every stop settles cmd before replication closes; run and move are serialized at the console (§5.1) |
| codex 2 (P2) claim-armed launch is missing | **Accepted.** §3.1: refused while claim control is armed; the work moves to #250 |
| codex 3 (P2) the ETag pattern drops or replays runs | **Accepted.** §5.2: pickup from reconcile's record reads (at start and on ownership); a CAS from queued to picked; a picked run is never run again |
| codex 4 (P2) oneshot success never becomes inactive | **Accepted.** §5.4: completion is the invocation's main process exiting; the agent records the result, then stops and settles the unit |
| codex 5 (P2) ack can delete unread output | **Accepted.** §5.5 and §5.6: the result comes after the last chunk and names `last_seq`; the ack must match |
| codex 6 (P2) the long-host path | **Accepted.** §5.2 step 3: `provision.InstanceDir` |
| codex 7 (P3) `CARLOS_REGION` | **Accepted.** §4: `cmdVars` |

## 12. Round 2 dispositions

| Finding | Disposition |
|---|---|
| R1-1, still open: run and move don't share a reservation | **Accepted.** §5.1: the run state lives on the instance record, so run, move and import abort all CAS the same ETag |
| R1-3, still open: no incarnation | **Accepted.** §5.1: the request carries the record's incarnation, and recreating the host drops `ActiveRun` |
| R2-2 (P1) run gets past the import gate | **Accepted.** §5.1 step 9: refused while an import is uncommitted, checked under the record CAS and again at launch |
| R2-6 (P2) "not picked up" leaves a runnable request | **Accepted.** §5.2 and §5.7: queued, picked, cancel, done, cancelled and expired, all moved by CAS on the record; expired and cancelled are terminal before launch |

## 13. Round 3 dispositions

| Finding | Disposition |
|---|---|
| R3-3 (P2) a disconnected caller leaves the run queued | **Accepted.** §5.8: the console sweep expires queued runs 60 seconds past the request, by CAS against pickup |