# Runs started by a scoped service credential: implementation plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** A service credential minted with the new `run` role and a command list can create, follow, cancel, expire and ack runs of those commands on its own app's instances, and can recover from a lost response without starting a second run. **Architecture:** There are three layers: - the role and the credential's stored scope (`svcrole`, `cli_tokens`, `Principal`); - the run doors admitting that principal with its narrower checks (`apiruns.go`, `apirunsfollow.go`); - an idempotency key, recorded in the instance record by the same compare-and-swap that admits the run, so admission and dedupe are one atomic write. **Tech Stack:** Go. The console's SQLite store (`internal/console/authstore.go`). The deployment bucket through `release.Store` (`Put(ctx, key, body, ifMatch)`: an empty `ifMatch` means create-only, and `release.ErrCASConflict` is returned on a 412). **Spec:** `docs/superpowers/specs/2026-10-24-service-runs-design.md`, which amends `docs/superpowers/specs/2026-10-08-carlos-run-design.md` § 2 ruling 1. ## Global Constraints - **People's behaviour on every run door is unchanged.** Every existing run test passes unmodified, apart from new fields in JSON bodies. - **Wire and storage changes are additive** (AGENTS.md): - new `cli_tokens` columns go in the ALTER list (`authstore.go:217-224`), with defaults equal to today's value; - new `runs.Request` and `runs.Meta` fields are `omitempty`. - **`Run` is reachable only through `svcrole.Exactly(svcrole.Run)`.** No other role reaches the run doors, and `Admin` does not. - **Command prefixes:** - at most 32 per credential; - each prefix has at least one word; - each word is 1–128 bytes; - matched word for word, exactly: no globbing, no case folding, no substring; - a zero-word prefix never matches. - **`Idempotency-Key`:** 1–128 bytes of `[A-Za-z0-9._:-]`. The record's `run_keys` keeps entries for 24 hours, and the newest 128 at most. - **`on_behalf_of`:** an email of at most 254 bytes, set only by a `run` service. A person who sends it gets a 400. - **Every refusal written by a run HANDLER carries a stable `code`** in its JSON body, alongside today's `error` message. The shared gates keep their bodies exactly as today, deliberately identical across doors: `cliNotSignedIn`, the role-mismatch 403 in `cliRoute`, and the canonical membership `http.NotFound` in `apiAccountApp`. - **A new `carlos` flag isn't done until `docs/site/cli.md` names it** (AGENTS.md). - **Branch rules:** the branch lands through amadan (`amadan branch describe`, then squash, then green CI, then `amadan branch merge … -expect `), never GitHub. ## Review Focus 1. **A retry of an admitted key while the first run is still open.** Today the "already has a run open" refusal (`apiruns.go:201`) fires before the body is read. A keyed retry must get its run back, not that refusal. The pin is `TestRunKeyedRetryWhileOpenReturnsTheRun` in Task 4. 2. **A service listing or following a person's run by guessing its id.** It must get a 404 identical to an unknown id. The pin is `TestRunServiceCannotSeeAPersonsRun` in Task 3. 3. **A revoked credential name minted again.** The new credential must not see the old one's runs. The pin is `TestRunRemintedNameDoesNotInheritRuns` in Task 3. 4. **Argv that matches a prefix only as a substring of a word.** For example, `importer` under the prefix `import`. It is refused. The pin is in Task 2's `TestCommandGranted` table. 5. **A keyed create whose meta declares a sha256 that doesn't match the uploaded bytes.** It gets a 400 `file_hash_mismatch`, and nothing is written. The pin is `TestRunKeyedHashMismatchWritesNothing` in Task 4. --- ### Task 1: The `run` role, which Admin does not reach **Files:** - Modify: `internal/svcrole/svcrole.go`. Add `Run`, add it to `catalogue`, add `Exactly`, and update the Admin doc comment. - Test: `internal/svcrole/svcrole_test.go` **Interfaces:** - Produces: - `const Run Role = "run"` - `func Exactly(roles ...Role) Set`. Like `Allow`, it panics on an unknown role, but it does NOT add Admin. - The doc on `Admin` changes from "the union of the five above" to say it is the union of every role except `Run`, which is reachable only through `Exactly` (spec § 1). - [ ] **Step 1: Write the failing tests** - `TestRunIsInTheCatalogue`: `Parse("run")` returns `(Run, true)`, and `All()` contains `Run`. - `TestExactlyDoesNotAddAdmin`: for `s := Exactly(Run)`, `s.Permits(Run)` is true and `s.Permits(Admin)` is false, as is every other role. - `TestExactlyPanicsOnAnUnknownRole`: `Exactly(Role("rnu"))` panics. - Amend `TestAdminPermitsEveryNonEmptySet` so it still covers every `Allow` set, then add: "an `Exactly` set admits only its listed roles." - [ ] **Step 2: Run them and watch them fail.** `go test ./internal/svcrole/` fails: `Run` and `Exactly` are undefined. - [ ] **Step 3: Implement** in `svcrole.go`. - [ ] **Step 4: Watch them pass.** `go test ./internal/svcrole/` passes. Then run `go test ./internal/console/ -run 'Role|Service'` and fix any role-list expectations that enumerate the catalogue. That includes `roleNames()` users in `services.go`. Every one of them must now include `run`. - [ ] **Step 5: Commit**: `svcrole: a run role that only an exact set reaches, never Admin (PRO-6214)` --- ### Task 2: A `run` credential's id and command list **Files:** - Modify: `internal/console/authstore.go`. Add to the ALTER list: `ALTER TABLE cli_tokens ADD COLUMN credential_id TEXT NOT NULL DEFAULT ''` and `ALTER TABLE cli_tokens ADD COLUMN commands TEXT NOT NULL DEFAULT ''`. Then a one-time backfill: every row with `kind` a service and an empty `credential_id` gets a fresh random id. It is idempotent, and only empty rows are written. - Modify: `internal/console/svcprincipal.go`. Covers `Principal`, `CLIPrincipal`, minting, rotation and listing. - Modify: `internal/console/apiservices.go`. Create accepts `commands`, and create and list echo it. - Modify: `services.go`. Adds the `--commands` flag. - Create: `internal/runs/commands.go`, holding the matcher. It stays a pure function so the console and its tests share it. - Test: `internal/runs/commands_test.go`, `internal/console/svcprincipal_run_test.go` and `services_test.go` (extended). **Interfaces:** - Produces in `internal/runs`: - `func CommandGranted(argv []string, prefixes [][]string) bool` - `func CheckCommandPrefixes(prefixes [][]string) error`. It enforces the limits in Global Constraints and returns `ErrNoCommands` for an empty list. - Produces in `internal/console`: - `Principal.CredentialID string`. It is set for every service principal, and empty for a person. - `Principal.Commands [][]string`. It is non-nil only for `Run`. - `ErrRunNeedsCommands`, `ErrCommandsNeedRun`. - `NewServicePrincipalScoped(accountID, name, app, role, rungs, commands [][]string, mintedBy, now)`. The signature gains `commands`, and every caller is updated. - The stored form of `commands` is canonical JSON: an array of arrays of strings, in the order given. Rotation keeps both `credential_id` and `commands`. A new mint always gets a new `credential_id`. - `CLIPrincipal` refuses a `run` row whose `commands` can't be parsed or is empty, the same way it refuses a deploy row without rungs (`svcprincipal.go:136-147`). - `--commands` is a comma-separated list of prefixes, and each prefix's words are separated by spaces: `--commands "import classic,import sync"`. It is required with `--role run`, and refused with any other role. Create also requires `--app` for `run`, using the same refusal path as `Service`. - [ ] **Step 1: Write the failing tests** - `TestCommandGranted`, a table test: - `{"import","classic","--slug","x"}` under `[["import","classic"]]` is true; - `{"import"}` under `[["import","classic"]]` is false; - `{"importer","classic"}` under `[["import","classic"]]` is false; - `{"Import","classic"}` is false; - `{}` under anything is false; - with two prefixes, matching the second is true. - a zero-word prefix (`[[]]`, and `[nil]`) matches nothing. - `TestCheckCommandPrefixes`: an empty list gives `ErrNoCommands`. A zero-word prefix, an empty word, a 129-byte word and 33 prefixes are each refused. `CLIPrincipal` also refuses a stored `run` row whose commands hold a zero-word prefix (`TestRunCredentialRefusedWithAnEmptyPrefix`). 32 prefixes of 128-byte words are accepted. - `TestRunCredentialCarriesItsCommands`: mint a `Run` principal with `[["import","classic"]]`, and `CLIPrincipal` returns those `Commands` and a non-empty `CredentialID`. - `TestRunCredentialNeedsCommands`: minting `Run` with nil commands gives `ErrRunNeedsCommands`. Minting `Instance` with commands gives `ErrCommandsNeedRun`. Minting `Run` with no app gives the account-scoped refusal. - `TestRotationKeepsCredentialIDAndCommands`, and `TestRemintGetsANewCredentialID` (revoke, then mint the same name). - `TestCredentialIDBackfillIsIdempotent`: reopening the store keeps existing ids. - A services CLI test: `--commands` without `--role run` fails, and `--role run` without `--commands` fails. - [ ] **Step 2: Run them and watch them fail.** `go test ./internal/runs/ ./internal/console/ -run 'Command|RunCredential|Rotation|Remint|CredentialID'` fails to compile. - [ ] **Step 3: Implement.** Follow how rungs are minted, stored, parsed and echoed, which is the pattern for every piece. - [ ] **Step 4: Watch them pass**, plus `go test ./internal/console/ -run Service` and the services CLI tests. - [ ] **Step 5: Commit**: `console: a run credential carries its own id and the commands it may run (PRO-6214)` --- ### Task 3: The run doors admit a `run` service, for its own runs only **Files:** - Modify: `internal/console/authcli.go:736-740`. The five run routes go from `svcrole.Set{}` to `svcrole.Exactly(svcrole.Run)`. - Modify: `internal/console/apiruns.go`. On create: - the principal check becomes "a person, or a `Run` service"; - the command check comes after the meta parse and before any write: 403, code `command_not_granted`; - `on_behalf_of` is validated; - the request fields are set; - the open-run refusal is redacted for a service. - Modify: `internal/console/apirunsfollow.go`. `runGate` admits a `Run` service. `runCall` gains the principal. Every read and write of a run checks `request.json`'s `ByCredential` against a service principal's `CredentialID`, and a mismatch answers exactly what an unknown id does. - Modify: `internal/runs/runs.go`. - `Request` gains `ByService string`, `ByCredential string` and `OnBehalfOf string` (all `omitempty`). - `Meta` gains `OnBehalfOf string` (`omitempty`). - Modify: `internal/console/authcli.go:1098`. Add `func jsonErrorCode(w http.ResponseWriter, status int, code, msg string)`, which writes `{"error": msg, "code": code}`. Every refusal on the five run routes switches to it. Name the codes for what they refuse (for example `not_found`, `host_claim`, `rate_limited`, `run_open`, `moving`, `import_uncommitted`, `files_too_large`, `malformed`, `command_not_granted` and `on_behalf_of_invalid`), and list them in one place: `var runRefusalCodes = []string{...}`. - Modify: wherever a run's requester is shown. Find these by grepping for `.By` on `runs.Request` and the run audit rows (CLI follow output, the console run history and the audit log). A service's run reads `run by service for (as stated by the service)`, or `run by service (scheduled)` when `OnBehalfOf` is empty. - Test: `internal/console/apiruns_service_test.go` **Interfaces:** - Consumes: `svcrole.Exactly`, `svcrole.Run` (Task 1); `Principal.CredentialID`, `Principal.Commands` and `runs.CommandGranted` (Task 2). - Produces: - `runs.Request{ByService, ByCredential, OnBehalfOf}`; - `jsonErrorCode`; - `runRefusalCodes`; - `func (s *Server) runVisibleTo(p Principal, req runs.Request) bool`, which is true for a person (membership was already checked) and, for a service, true only when `req.ByCredential == p.CredentialID`. - [ ] **Step 1: Write the failing tests** (use `runFixture` and `serviceToken`, and mint `Run` with `[["import","classic"]]`): - `TestRunRoleMatrix`: on all five routes, a `Run` service passes the role gate. `Publish`, `Deploy`, `Instance`, `Operate`, `Service` and `Admin` each get `cliRoute`'s role-mismatch 403 (`authcli.go:467`), not the closed-door message, because the set is no longer empty. Update the tests that pin the closed-door registrations and the route-role table (`svcroutes_test.go:226` and the fixed door count there). Keep explicit coverage that Admin is refused on the run doors. - `TestRunServiceStartsAGrantedCommand`: argv `import classic --slug x` gets a 202, as create gives today (`apiruns.go:317`), and request.json has `ByService`, `ByCredential` and `OnBehalfOf`. - `TestRunServiceRefusedAnUngrantedCommand`: argv `db shell` gets 403 `command_not_granted`, and no run objects exist. - `TestRunServiceWrongAppRefused`: a `Run` credential for app `other` on `shop` is refused. - `TestRunRevokedServiceRefused`. - `TestRunServiceCannotSeeAPersonsRun`: the person starts a run, and the service's GET, cancel, expire and ack on its id each give a body identical to an unknown id's. - `TestRunRemintedNameDoesNotInheritRuns`: the service starts a run, the credential is revoked, the same name is minted again, and the new credential's GET is the unknown-id answer. - `TestRunOpenRefusalRedactedForServices`: a person's run is open, and the service's create gets `run_open` with neither that run's id nor the person's email anywhere in the body. - `TestRunPersonMaySetNoOnBehalfOf`: a person's meta with `on_behalf_of` gets 400 `on_behalf_of_invalid`. - `TestRunOnBehalfOfShape`: not an email, or 255 bytes, gets 400. - `TestRunRefusalsCarryCodes`: every refusal a run HANDLER writes has a `code` that is in `runRefusalCodes`. The shared-gate refusals (401, the role 403, the membership 404) are byte-identical to today, and `apiruns_test.go:195`'s canonical-404 test is unchanged. - `TestServiceRunRendersAsTheServices`: the follow and history rendering of a service's run contains `run by service ci for paul@example.com (as stated by the service)` and never presents the email as the actor. - Repeat the existing incarnation, one-run and moving tests with a `Run` service as the creator: delete and recreate, a move in progress, and a person's run plus a service's run on one instance. - [ ] **Step 2: Run them and watch them fail.** `go test ./internal/console/ -run 'TestRun'` fails. - [ ] **Step 3: Implement.** - [ ] **Step 4: Watch them pass**, and run all of `go test ./internal/console/ -run 'Run|Service'`, with every pre-existing run test unmodified. - [ ] **Step 5: Commit**: `console: a run service starts and follows its own runs of the commands it was granted (PRO-6214)` --- ### Task 4: Idempotent create, settled by the record **Files:** - Modify: `internal/provision/provision.go`. `Record` gains `RunKeys []RunKey \`json:"run_keys,omitempty"\``. - Modify: `internal/provision/run.go`. - `RunAdmission` gains `Key string` (the scoped key hash) and `Digest string`. - `RunAdmit` handles the key inside the same mutation. - Every record mutation that replaces or clears `ActiveRun` (the next admission, ack, expire, the sweep's clear) keeps `RunKeys`. - Create: `internal/runs/keys.go` - Modify: `internal/runs/runs.go`. `MetaFile` gains `SHA256 string` (`omitempty`). Convert the positional `MetaFile{...}` literals in existing tests (`internal/runs/runs_test.go:74-89` and any others the compiler finds) to named fields. - Modify: `internal/console/apiruns.go` - Test: `internal/runs/keys_test.go`, `internal/provision/run_keys_test.go`, `internal/console/apiruns_key_test.go` **Interfaces:** - Produces in `internal/runs`: - `func ValidKey(k string) bool` - `func ScopedKey(scope, key string) string`. It returns `hex(sha256(scope + "\x00" + key))`. `scope` is `"svc:" + CredentialID` for a service and `"user:" + address` for a person. - `func MetaDigest(m Meta) string`: sha256 over a canonical encoding of argv, timeout_s, on_behalf_of and every file's name, size and sha256. - `const KeyTTL = 24 * time.Hour`, `const MaxRunKeys = 128`. - Produces in `internal/provision`: - `type RunKey struct { Key, RunID, Digest string; At time.Time }` - `type RunKeyReplay struct { RunID string }`, which implements `error`, returned by `RunAdmit` when the key is already there with the same digest; - `var ErrRunKeyReused`, for when the key is already there with a different digest. - `RunAdmit`'s mutation, in this order inside the one CAS: 1. prune `RunKeys` to entries younger than `KeyTTL`; 2. if `p.Key != ""` and an entry has that key: return `ErrRunKeyReused` if its digest differs, else `&RunKeyReplay{RunID}`. Both run BEFORE today's open-run, moving and import refusals, so a replay is answered even while its own run is still open; 3. today's admission; 4. on success, when `p.Key != ""`, append `{Key, RunID, Digest, now}` and trim to the newest `MaxRunKeys`. - **Create's order, but only for a keyed request** (an unkeyed request keeps today's exact order): 1. principal, membership, host-claim and rate budget; 2. read the record (a 404 stays a 404); 3. parse the meta; 4. **look up the key in that record read.** On the same digest, answer `202` with the run (`"replayed": true`) and stop reading the body. On a different digest, `409` `idempotency_key_reused`; 5. today's class, move, open-run and import refusals; 6. the command check (Task 3); 7. spool the files and verify each against its declared sha256 (`400` `file_hash_mismatch`, nothing written); 8. mint the id, write the objects, and admit with `Key` and `Digest`; 9. if admission returns `*RunKeyReplay`, compare its `RunID` with the id this request minted: - **equal:** this request's own admission committed, and the CAS retry inside `updateRecord` found its own key. That is success: answer `202` as a fresh admission, and delete nothing; - **different:** a concurrent twin won. Delete ONLY this request's candidate prefix (`runs.Prefix(acct, app, mintedID)`), then answer `202` with the winner's run, replayed; `ErrRunKeyReused` answers `409`; 10. today's lost-admission settling (`apiruns.go:279-291`) is unchanged, and is now also correct for keys, because the key commits with the admission or not at all. - **The rollout gate (mixed versions).** A box running an older binary rewrites the record through a typed `Record` that doesn't know `run_keys`, and drops it. So keyed admission is OFF until every record writer preserves the field: - the console config gains `RunKeysEnabled bool`, from `CARLOS_RUN_KEYS=on`; - while it is off, a keyed create is refused `503` `keys_not_ready` before anything is written, never silently treated as unkeyed; - it is turned on only after the box fleet has rolled to a build that includes this task. The branch description and the landing note say so. - `GET .../instances/{host}/runs?key=`: a new route on the run set (`Exactly(Run)`, plus people). It returns `{"run_id": ...}` from the record's `RunKeys` for the caller's scoped key, or 404. - [ ] **Step 1: Write the failing tests** - `TestValidKey`: a table of good keys, plus an empty key, a 129-byte key and one containing `/` or a space. - `TestScopedKeySeparatesCallers`. - `TestMetaDigestCoversEveryField`: changing argv, timeout, on_behalf_of, a file name, a size or a sha256 each changes the digest. - `TestRunAdmitRecordsAndReplaysAKey`, at the provision level: - admit with K and D, then admit again with K and D, which gives `RunKeyReplay` with the first id; - K with D2 gives `ErrRunKeyReused`; - after the run is acked or cleared, K and D still replay; - after a second, unkeyed run replaces it, K and D still replay. - `TestRunKeysPrunedByAgeAndCount`: entries older than 24h are dropped, and at most 128 are kept, newest first. - `TestRunKeyedCreateThenRetryReturnsTheSameRun`: same id, one request.json, and the retry carries `replayed: true`. - `TestRunKeyedRetryWhileOpenReturnsTheRun` - `TestRunKeyedRetryAfterAckStillReturnsTheRun` - `TestRunKeyReusedWithADifferentRequest`: 409, with nothing written. - `TestRunConcurrentIdenticalKeyedCreatesMakeOneRun`: two goroutines with the same key and request. Exactly one admission happens, both answers name the same id, and the loser's objects are deleted. - `TestRunKeyOfAnotherCallerIsInvisible`: the person's K used by the service is fresh, and `?key=K` from the service is a 404. - `TestRunKeyedHashMismatchWritesNothing` - `TestRunKnownKeyAnsweredWithoutReadingFiles`: retry with a body whose file part errors on read, and still get a 202 with the id. - `TestRunRefusedKeyIsReevaluated`: a keyed create refused with `run_open` starts the run once the other run is acked. - `TestRunKeyLookup` - `TestRunUnkeyedOrderUnchanged`: an unkeyed create against an open run is still refused before the body is read. - `TestAPIRunCreateAdmissionWhoseAnswerWasLost`: extend the existing test to a keyed request. The store's conflict-after-commit gives a `RunKeyReplay` with this request's OWN id, the answer is a fresh `202`, and the run's objects still exist. - `TestRunKeysSurviveEveryRecordMutation`: a record carrying `run_keys`, passed through each of the six `ActiveRun` mutations in `internal/provision/run.go` and the move and import mutations, still carries them. - `TestKeyedCreateRefusedUntilKeysAreOn`: with `RunKeysEnabled` false, a keyed create gets `503` `keys_not_ready` and nothing is written. An unkeyed create is unaffected. - [ ] **Step 2: Run them and watch them fail.** `go test ./internal/runs/ ./internal/provision/ ./internal/console/ -run 'Key|Keyed|Digest'` fails. - [ ] **Step 3: Implement.** - [ ] **Step 4: Watch them pass**, plus every pre-existing test under `go test ./internal/provision/ ./internal/console/ -run Run`. - [ ] **Step 5: Commit**: `console: a keyed run create is admitted once, settled by the instance record, and a retry gets the same run (PRO-6214)` --- ### Task 5: The CLI flag, the docs and the run-spec amendment **Files:** - Modify: `run.go`. `carlos run` gains `--idempotency-key `, sends it as the header, and sends each file's sha256 in the meta when a key is given. It accepts the `202` replay (`"replayed": true`) the same way as a fresh `202` (`run.go:223`), saying "already started: run " before following it. It prints the run id before streaming, as it does today. - Modify: `docs/site/cli.md`, covering `carlos run --idempotency-key` and `carlos services create --role run --commands`. - Modify: the services and run docs pages under `docs/site/`, wherever the role catalogue and the run API are described. Add the `run` role, the command-list rule, `on_behalf_of`, the refusal codes (from `runRefusalCodes`) and the key lifetime. - Modify: `docs/superpowers/specs/2026-10-08-carlos-run-design.md` § 2 ruling 1. Add one line: "Amended 2026-10-24: a `run` service credential may, for the commands it was granted. See `2026-10-24-service-runs-design.md`." - Test: `run_test.go` (extended), plus the docs test that checks `docs/site/cli.md` names every command or flag, if one exists. Find it with `grep -rn "cli.md" *_test.go`. **Interfaces:** - Consumes: `runs.ValidKey` (Task 4). - [ ] **Step 1: Write the failing tests** - `TestRunIdempotencyKeyFlag`: the request carries `Idempotency-Key`, and the meta's files carry `sha256`. - `TestRunCLIRetryFollowsTheReplayedRun`: against a test console, a second `carlos run` with the same key prints "already started", follows the same id, and starts no second run. - An invalid key is refused by the CLI before any network call. - `TestRunRefusalCodesAreDocumented`: every entry in `runRefusalCodes` appears in the run API doc page. - [ ] **Step 2: Run them and watch them fail.** - [ ] **Step 3: Implement.** - [ ] **Step 4: Watch them pass**, and run `make test` once, before the branch is described. - [ ] **Step 5: Commit**: `cli, docs: carlos run takes an idempotency key, and the run role is documented (PRO-6214)` --- ## Landing One branch, `pro-6214-service-runs`, landed through amadan as AGENTS.md describes. Describe it with `amadan branch describe carlos/platform `, then squash to one commit, wait for CI to go green on the tip, and merge with `-expect `. Before landing, an Opus or Fable review covers the whole branch, because it grants authority (AGENTS.md delegation policy). Astra reviews too, since the session is Opus. After it lands, roll the console AND the boxes, confirm every box runs a build that includes Task 4, and only then set `CARLOS_RUN_KEYS=on` on the console. Then Tito mints `fleet-runs` with `--role run --app titogo --commands ""`. The exact commands are settled with the titogo side of PRO-6214, outside this plan.