# Runs started by a service credential

**Date:** 2026-10-24
**Status:** Design, awaiting approval
**Owner:** Paul
**Asked for by:** Tito (Linear PRO-6214). Fleet's Classic import and recurring
sync need to run an app command against an instance without a person at a
terminal.
**Amends:** `2026-10-08-carlos-run-design.md`, § 2 ruling 1.

## Why

`carlos run` lets a member of an account run the app's own command against an
instance, as the instance. Fleet, Tito's back office, needs exactly that for two
jobs:

- **The Classic import.** A person presses a button and Fleet copies one
  account's data in.
- **The recurring sync.** It keeps an imported account up to date on a
  schedule.

Fleet holds a server credential and no person's token. Today every run route
refuses a service principal: `runGate` and `handleAPIRunCreate` check for
`PrincipalUser` (`internal/console/apiruns.go:156`,
`internal/console/apirunsfollow.go:36`). The work-arounds on the table are both
unacceptable:

- **A personal operator token in Tito's router.** A person's whole membership
  would sit in a service, and nobody could tell that from the audit.
- **Root or systemd jobs on the box.** That goes around the platform, which is
  what `carlos run` exists to end.

## Decisions (Paul, 2026-10-24)

- **Service credentials may start and follow runs.** This reverses run spec
  § 2 ruling 1 ("service credentials are refused"), for the one role below
  only.
- **A `run` credential names the commands it may run** (option A). It does
  not get any command the binary has. A leaked credential can then run only
  those, not arbitrary commands against every instance of the app.
- **The person a run is "for" is recorded on the service's word.** When Fleet
  starts a run because someone pressed a button, the run records the name
  Fleet gives, labelled as stated by that service. It grants nothing and is
  not verified. A scheduled run records no person.

## Goals

- A service can create, read (progress and output), cancel, expire and ack
  runs on its own app's instances, and nothing else, once it has been granted
  that on purpose.
- An ungranted service, a service from the wrong app, and a revoked service
  are all refused.
- Deleting and recreating an account, or moving it, can never redirect a run.
- A caller that loses an HTTP response can find out what happened and carry
  on without starting a second run.
- Everything above goes through the supported console API, with no box access.

## Non-goals

- Host-claim consoles. They refuse runs for everyone until the reservation
  protocol (#250) adds `cmd` (see § 6).
- Verifying who the person was. That is a later upgrade (§ 4).
- A scheduler. The caller decides when to run, and Fleet owns its schedule.
- Anything about what Fleet's command does: the Classic DB credentials, how
  snapshots are cut. See § 7 for the caller's side.

## Anti-goals

- **A role that quietly widens.** No existing role, `Admin` included, gains
  runs. A service that can provision instances cannot run commands on them
  unless it is separately minted for that.
- **A second run from a retry.** A lost response retried must never start the
  command twice.
- **An audit line that misleads.** A service's run must never read as if a
  person ran it.

## 1. The capability: a new role, `run`

- **`svcrole.Run` joins the closed catalogue** (`internal/svcrole/svcrole.go`).
  It is app-scoped, so `App` is required, refused account-wide the way
  `Service` is (`ErrAccountScopedService`). It carries no rungs.
- **It grants exactly the five run routes** for instances of its own app:
  create, get (with `?after=`), cancel, expire and ack
  (`authcli.go:736-740`). Those routes are registered with `svcrole.Set{Run}`
  instead of the empty set. People keep the routes on the existing member
  rule, unchanged.
- **No other role implies it, Admin included** (my call, holding the
  anti-goal). Today `svcrole.Allow` always adds Admin to a route's set
  (`internal/svcrole/svcrole.go:126`, pinned by
  `TestAdminPermitsEveryNonEmptySet`), so this IS new code. Add a constructor
  for an exact set (e.g. `svcrole.Exactly(Run)`) that Admin does not join,
  and amend that test and the Admin doc to name the one exception. A test
  asserts that every role except `Run`, Admin included, is refused on all five
  routes.
- **Minted, rotated and revoked like every other service:**
  `carlos services create --role run --app <app> --commands "<prefix>,<prefix>" <name>`,
  by an account owner, through `handleAPIServiceCreate`. Revocation is the
  existing soft revoke, and a revoked token is refused at the door.
- **The command list (Paul, 2026-10-24, option A).** A `run` credential
  carries a non-empty list of NON-EMPTY allowed command prefixes. A
  zero-word prefix would match every argv, so it is refused at mint, refused
  when the credential is read back, and never matches in the matcher. The
  list is stored on its
  `cli_tokens` row the way `Deploy` carries rungs: an additive column, parsed
  only for this role, and refused at mint when it is empty
  (`ErrRunNeedsCommands`).
  - Each prefix is a sequence of argv words, written space-separated on the
    CLI. `import classic` is the words `import`, `classic`.
  - A run is allowed only when its argv starts with every word of at least
    one prefix, compared word for word and exactly: no globbing, no case
    folding, no substring.
  - Create refuses anything else with 403 `command_not_granted`, before any
    object is written, and with no key state either: an unkeyed refusal.
  - `carlos services list` shows the list. Changing it means minting a new
    credential (rotation keeps the list, and there is no edit door; my call).
  - Prefixes contain no empty word, and each word is at most 128 bytes, with
    at most 32 prefixes (the builder may tune these).
  - People are not affected: members keep running any command.
- **A service sees and acts on only the runs it started** (my call). The run
  records the CREDENTIAL's id (its `cli_tokens` row), not its name, because a
  revoked name can be minted again (`authstore.go:250`) and must not inherit
  the old credential's runs. Rotation keeps the row, so a rotated credential
  keeps its runs. Get, cancel, expire and ack check that id. A run it didn't
  start answers 404, the same as an unknown id, so its existence doesn't leak.
  Members still see every run on their account, the service's included.
- **Create's "already has a run open" refusal is redacted for a service**
  (my call). Today it names the open run's id and the member's email
  (`apiruns.go:32`, `:202`). A service gets the refusal and its code without
  either, unless the open run is its own.
- **Rate budgets are unchanged:** create spends `s.cli.instances` and the
  follow routes spend `s.cli.runs`, both keyed on the token hash
  (`apiruns.go:168`, `apirunsfollow.go:48`). A service's runs therefore draw on
  that credential's own budget, which is separate from Fleet's provisioning
  credential.

## 2. What a service's run records

`runs.Request` (`internal/runs/runs.go:166`) gains fields, additively, so old
readers ignore them:

| Field | For a person's run | For a service's run |
|---|---|---|
| `by` | the member's email (today) | what `auditActor` writes for a service today, `service:<account sqid>/<name>` (`apicli.go:105`) |
| `by_service` | empty | the service name, for display |
| `by_credential` | empty | the credential's id, which the own-runs check uses |
| `on_behalf_of` | empty | the email the service gave, or empty for a scheduled run |
| `idempotency_key` | the key's scoped hash (§ 3), when one was sent | the key's scoped hash, expected from Fleet |

Every place the run is shown or audited (the CLI's follow output, the
console's run history, the account's audit log) shows a service's run as
"run by service `<name>` for `<email>` (as stated by the service)", or "run by
service `<name>` (scheduled)". It is never shown as if the email ran it.

`on_behalf_of` is validated for shape only: it must be an email, at most 254
bytes. It is not checked against membership, because it grants nothing.

## 3. Lost responses: idempotent create

Run ids are minted by the console (`runs.MintID`), so today a caller who loses
the create response can't tell "started" from "never arrived". So:

- **Create accepts an `Idempotency-Key` header**: 1–128 bytes of
  `[A-Za-z0-9._:-]`, chosen by the caller.
- **A key is scoped to its caller** (my call): to the credential id for a
  service, to the user for a person, and within that to the instance. One
  caller can never find, probe or collide with another's key.
- **The proof that a key was admitted lives in the instance record, written
  by the same compare-and-swap that admits the run** (my call, after Astra's
  review). The record gains `run_keys`: a short list of recent admissions,
  each with a hash of (caller scope, key), the run id, the request digest, a
  hash of the caller scope (which the per-caller cap counts by) and the time. `provision.RunAdmit` refuses to admit when the list already holds
  this caller's key, and returns that run instead.
  - **Admission and dedupe are therefore one atomic write.** Two identical
    creates at once can both write their objects, but only one is admitted.
    The other finds its key already in the record, deletes its own objects
    and answers with the winner's run.
  - **There is no separate key object,** so there is no pending or abandoned
    state, no sweep, and no crash window between "admitted" and "key
    recorded".
  - **Ack, the run sweep and the next run replacing a finished one all keep
    `run_keys`,** so a retry after any of them still finds the run.
- **The digest covers the meta part only:** argv, timeout, `on_behalf_of`,
  and each file's name, size and sha256. A keyed create must declare each
  file's sha256 in the meta, and the console verifies the uploaded bytes
  against it. The meta is the first multipart part, so the key is checked
  against the record straight after parsing it, before the open-run, moving
  and import refusals and before any file byte is read. A retry of an
  admitted key is answered without reading the files.
- **The same key with a different digest** is `409` `idempotency_key_reused`,
  and nothing is written.
- **Answers to a keyed create:**
  - a fresh admission and a replay of an admitted key both return `202`, the
    status create gives today, with the same body, plus `"replayed": true` on
    a replay;
  - `GET .../instances/{host}/runs?key=<key>` returns the run id from the
    record, or 404.
- **Refusals are not remembered** (my call). Only an admitted run is.
  - A 4xx means THIS attempt admitted nothing. An earlier attempt with the
    same key may already have run: the shared gates (401, the rate budget's
    429) come before the key lookup, so the caller confirms with
    `GET ...?key=` before treating a 4xx on a retry as "never ran".
  - A retry with the same key is evaluated afresh: a `run_open` refusal can
    succeed once the other run has finished, which is what the caller asked
    for.
  - A 5xx, a timeout or a dropped connection means unknown, and the caller
    retries with the same key.
- **How long a key lasts:** each admission prunes `run_keys` to entries younger
  than 24 hours (`runs.KeyTTL`), then keeps the newest 32 entries of each
  caller (`runs.MaxRunKeysPerCaller`). The cap is per caller, so one caller's
  admissions can never evict another's key. A key is honoured for 24 hours,
  or until that same caller has made 32 newer keyed admissions on this
  instance, whichever comes first. A caller must not retry a key past that.
  Fleet retries within minutes.
- **`request.json` records the key** as `idempotency_key`: its scoped hash,
  never the caller's own words. It links the run to its key after the record
  has pruned the entry.
- **Mixed versions:** an older box binary rewriting the record would drop
  `run_keys`. Keyed creates stay off (`503` `keys_not_ready`) until the console
  is told, by config, that every record writer preserves the field.
- **A known limit:** deleting an instance deletes its record, `run_keys` with
  it. A retry that arrives after the host was deleted and created again is a
  fresh run on the new incarnation. It is still only a command the credential
  was granted, on its own app.
- **Refusal codes:** the run handlers' own refusals carry a stable `code`
  beside `error`, and the API docs list them. The shared gates before the
  handler keep their bodies exactly as today, deliberately identical across
  doors (AGENTS.md): not signed in, role mismatch, and the canonical
  membership 404. Those already distinguish themselves by status.

People may send the key too, and the CLI gains `--idempotency-key`. Nothing
changes for a person who doesn't.

## 4. Upgrading "on the service's word" later

If the person ever needs to be verified, the request gains a signed assertion
from the caller's identity system, and the console adds `on_behalf_verified:
true` when it checks out. The fields above stay, so nothing already recorded
changes meaning. Not built now.

## 5. What stays exactly as it is

These were already built in the run series, and must be shown to hold for a
service's run too, each by a test that starts the run with a `Run` service:

- **Incarnation binding.** The incarnation is taken at create
  (`apiruns.go:250`) and checked by the box before launch. A run created
  before an account is deleted and recreated, or moved, never launches
  against the new one.
- **One run per instance**, through the `ActiveRun` compare-and-swap. This
  covers a service's run and a person's run on the same instance too.
- **The 64 MB file cap** (`runs.MaxFileBytes`), with the same 413.
- **Runs execute beside a live instance** (the same database, WAL). Live sync
  relies on this.
- **Cancel, expire, timeout, poll-and-ack, and the 24-hour sweep.**

## 6. Host-claim consoles

A console with `HostClaim` on refuses every run route with
`claimRunUnsupportedMsg` (`apiruns.go:26`, `:164`; `apirunsfollow.go:44`).
That stays true for services. A granted `Run` service gets the same refusal
a person does. An ungranted service gets 403 first, because the role check
runs in `cliRoute` (`authcli.go:455`) before any handler, and that is fine. The API docs say so: a service run works only
on a console without host-claim until #250 adds `cmd` to the reservation
protocol. Nothing here bypasses or pre-empts #250.

Tito's console (`carlos.tito.io`) serves runs today: a follow of an unknown
run there answers "no such run", not the host-claim refusal (checked
2026-10-24). So Fleet is not blocked by #250.

## 7. The caller's side (Tito; recorded here, built there)

- **Fleet keeps the run id and key in its own store before calling,** so a
  deploy or restart mid-run resumes by key, then follows by id, until the
  result is acked.
- **The Classic DB credentials stay on the source reader**, Fleet's side. They
  never go in argv (argv is audited) or in uploaded files, and never into a
  customer instance. Fleet reads Classic and uploads an account-specific
  snapshot.
- **A snapshot over 64 MB** is Fleet's problem to split into several runs, or
  to refuse with a clear message. The platform's cap does not change.

## Tests and fences

- The command list:
  - an allowed prefix runs;
  - a longer argv under it runs;
  - an argv that only shares a substring of a word is refused;
  - a missing word is refused;
  - an empty list is refused at mint;
  - rotation keeps the list;
  - the refusal writes nothing.
- A role-by-route matrix: `Run` is allowed on the five routes and refused
  everywhere else. Every other role is refused on all five.
- Wrong app, wrong account, revoked, and a run that isn't this service's own:
  all refused or 404. So is a revoked name minted again: the new credential
  cannot see the old one's runs.
- Idempotency:
  - the same key and the same request give the same run with no second
    launch, after ack, after the sweep, and after a later run has replaced it
    in the record;
  - a different request is a 409;
  - two concurrent identical creates make one run, and the loser's objects
    are deleted;
  - another caller's key is invisible;
  - an admitted key is answered without reading the files;
  - a refusal is re-evaluated on retry;
  - `run_keys` is pruned by age and count.
- Incarnation, one-run and move tests repeated with a service principal.
- Audit: a service's run renders as the service's, never as the email's.
- `docs/site/cli.md` gains `--idempotency-key`, and the services docs gain
  the `run` role (AGENTS.md: a command isn't done until the docs name it).

## Left to the builder

- The exact shape of `run_keys` entries.
- The 4xx code list.
- The 24-hour and 32-per-caller bounds, if the builder finds a reason to change
  them, stated in the docs.