> **2026-10-05:** steps 2 and 4 are superseded by docs/superpowers/specs/2026-10-04-scanner-host-rollout-design.md. clamd runs on its own box, the switch is carlos system antivirus enable, and CARLOS_STORE_SCANNER is deprecated. Steps 5 to 8 and the watch list still stand. # Rolling object-storage scanning out to Carloku **Not TDD tasks — an ordered operational sequence.** Do not start it until the implementation plan (`2026-09-01-object-storage-virus-scanning.md`) is merged and on `main`. Each step states how to know it worked. **Spec:** `docs/superpowers/specs/2026-09-01-object-storage-virus-scanning-design.md` **Tito first throughout**, for the reason the console-as-an-app rollout gives: smaller blast radius, and Tito's console going down does not take Carloku with it. ## What is actually being rolled out An opt-in scanner that is **inert until three separate things are true**: `CARLOS_STORE_SCANNER` is set on the deployment, a member has requested scanning for their app, and an operator has granted it. Shipping the binary changes nothing on its own, which is what makes this rollout low-risk and why step 1 is safe to do first and sit on. The failure mode worth naming up front: the ClamAV engine **reads tenant object bytes**. Nothing reads anything until an operator grants a specific app, but the capability arrives with the binary, and the sequence below puts the disclosure before the first grant rather than after it. --- ### 1. Ship the binary with the scanner unset Deploy `main` to Tito, then Carloku, with `CARLOS_STORE_SCANNER` **unset**. **How to know it worked:** `carlos store scan status` on any app answers `engine: none` and the operator doors 503 naming `CARLOS_STORE_SCANNER`. No scan pass appears in the console log. `carlos store status` is unchanged for every existing app, and no `scan.json` exists anywhere: ``` aws s3 ls s3:///apps/ --recursive | grep scan.json ``` should print nothing. If it prints anything, stop — a sweep ran when nothing should have. ### 2. Stand up clamd on the Tito console box This is the `tito/carlos-infrastructure` half. Per spec §10: - a `clamav-daemon` systemd unit, socket at `/run/clamav/clamd.ctl` - `freshclam` on a timer — **a stale signature database is the silent failure mode of this whole feature**; a clean verdict from three-month-old signatures reads exactly like a fresh one - `StreamMaxLength` matched to whatever `CARLOS_CLAMD_MAX_BYTES` will be - the console unit's user added to the `clamav` group - roughly 1–1.5 GB of RAM for resident signatures, held whether or not anything is being scanned — check the box has it rather than assuming - **resource control on the unit**, which matters more than the headroom: ```ini [Service] MemoryMax=2G # the kernel kills clamd, never the console MemoryHigh=1600M CPUQuota=50% Nice=10 ``` This runs on the CONSOLE box. If clamd drives that into the OOM killer we do not lose scanning, we lose deploys, routing and config for every app on the deployment. With `MemoryMax` on clamd's own cgroup the kernel kills clamd and the console carries on; without it the two compete for the same memory and the wrong one can lose. - **the reload spike, which is the number that catches people.** Current ClamAV defaults to `ConcurrentDatabaseReload yes`, holding the old and new databases in memory at once so scanning never pauses during a `freshclam` update — briefly around doubling it. Either size for ~3 GB of headroom, or set `ConcurrentDatabaseReload no` and size for steady state. The second trade is nearly free: it blocks scanning during a reload rather than doubling memory, and a pass that fails a tick resumes on the next one **How to know it worked:** ``` systemctl is-active clamav-daemon sudo -u test -w /run/clamav/clamd.ctl && echo reachable echo -n 'zVERSION' | sudo -u socat - UNIX-CONNECT:/run/clamav/clamd.ctl ``` The last prints `ClamAV //`. Check the date is today or yesterday. If `freshclam` has never run, it will not be. ### 3. Grant the console its S3 read and tag permissions Add `s3:GetObject`, `s3:GetObjectTagging` and `s3:PutObjectTagging` on `arn:aws:s3:::*` to the Tito console's role. It already lists those buckets for the economics storage pass; this adds read and tag. **How to know it worked:** step 5's first sweep completes. There is no useful intermediate check that does not involve reading a tenant object, and doing that by hand before a member has opted in is exactly what this feature is supposed to require consent for. ### 4. Turn the engine on, still with nothing opted in Set `CARLOS_STORE_SCANNER=clamav`, `CARLOS_CLAMD_SOCKET=/run/clamav/clamd.ctl` and `CARLOS_CLAMD_MAX_BYTES` on the Tito console. Restart. The other three have defaults and only need setting deliberately: `CARLOS_CLAMD_MAX_SIG_AGE` (168h — when `scan status` starts warning), `CARLOS_STORE_SCAN_BUDGET` (60s of scanning per app per hourly tick) and `CARLOS_STORE_SCAN_INTERVAL` (24h floor before a *completed* sweep re-runs; a sweep still in progress ignores it, which is how a bucket bigger than one budget ever finishes). Setting `CARLOS_STORE_SCANNER` without `CARLOS_STORE_BACKEND` is refused at boot, so a typo here fails loudly rather than leaving a scanner with nothing to scan. **How to know it worked:** the console log shows the scan pass starting and then finding nothing to do. `carlos store scan status` on any app now answers `engine: clamav`, `requested: no`. **No bucket has been read** — the pass skips every app without an `enabled` block, and there are none. This is the state to sit in for a day. A scan pass that wakes hourly and correctly does nothing is the cheapest possible proof that the enumeration and the gating work. ### 5. First real store: one Tito app you own Pick an app whose objects are **plaintext** and whose data you own — not a member's. Then: ``` carlos store scan request --plaintext carlos store scan grant carlos store scan sweep carlos store scan status ``` **How to know it worked:** `status` reports a completed sweep with a `last complete` timestamp and a signature version matching step 2. Read the arithmetic carefully, because it is deliberately not the obvious one. `scanned` and `unscanned` are **not complements of each other against `objects`** — they are two independent facts: ``` objects = scanned + skipped + denied + failed unscanned = skipped + denied + failed (NOT objects - scanned) ``` An object too large to stream was never judged, so it counts as both `skipped` and `unscanned`. `unscanned: 0` is therefore only the right expectation on a store where nothing was skipped, denied or failed — likely on a small plaintext app, but do not treat a non-zero number as a bug. It is the feature: the second sentence of the audit answer. `status` also splits coverage into verdicts reached during this sweep and verdicts carried forward from an earlier one. On a first sweep everything is fresh; on a re-sweep of an unchanged bucket everything is carried forward, which is correct and is why the engine database version is printed as what it is *now* rather than as a claim about every object beneath it. Then verify the two properties that are easy to assume and worth proving: - **Verdicts are on the objects.** `aws s3api get-object-tagging` on any scanned key shows `carlos-scan`, `carlos-scan-version` and `carlos-scan-at`, and the etag matches the object's current one. - **A rewrite invalidates the verdict.** Overwrite one object, then run `carlos store scan sweep` — that object must be re-scanned rather than reading back its old verdict, which shows up as the fresh count going up. If it is carried forward instead, the version binding is not working and **stop the rollout**: verdicts that outlive the bytes they judged are worse than no verdicts, because they are trusted. The tag records the object's ETag *and* its last-modified time. Both halves matter: a single-part upload's ETag is the content MD5, and chosen-prefix MD5 collisions are practical, so an attacker who controls two uploads could otherwise hand a payload the ETag of the clean object it replaced. ### 6. Prove the destructive paths before any member sees this On a throwaway app in Tito: - `carlos store scan grant`, then destroy the app. Confirm no `scan.json` remains under its prefix and **the app name is immediately claimable again** — an orphan sidecar makes a name permanently unusable and is the failure this feature's teardown path exists to prevent. - `carlos store scan enforce on`, then unset `CARLOS_STORE_SCANNER` and restart. Confirm `carlos store scan enforce off` still works with no engine built. This is the deadlock draft 3 of the spec would have shipped; prove it is gone on a real deployment, not only in a unit test. ### 7. Disclose before the first member grant Before granting scanning on any store CARLOS does not own, the member has to know what they are consenting to: - the deployment will **read their object bytes** to scan them - a clean verdict on a **client-side-encrypted** store means nothing, and they should request `--client-encrypted` if that is their shape - verdicts are **eventually consistent** — an object is readable before it is scanned, and there is no upload hook The `scan request` verb prints this, which is the point of making the attestation an explicit flag. But a member who has not run the verb has not read it, so an operator granting on someone's behalf must have had the conversation first. ### 8. Carloku, same sequence Repeat steps 2 through 6 on Carloku. Then stop and wait: on Carloku the first grant should be **driven by an actual member request**, not by an operator enabling it speculatively. This feature exists to answer a compliance question, and the person with the compliance question is the one who should ask for it. ## Watch after each grant - **Signature age.** `scan status` warns past `CARLOS_CLAMD_MAX_SIG_AGE`. Treat the warning as an incident, not a notice — every clean verdict written while it was stale is worth less than it looks. - **Sweep duration against the budget.** If a sweep spans many ticks, the audit answer's T is as old as the sweep is long. Raise `CARLOS_STORE_SCAN_BUDGET` or shorten the interval, and say the real number when anyone quotes coverage. - **`skipped` climbing.** Over-ceiling objects are not scanned and are not clean. A store that is mostly skips has not been meaningfully scanned and the compliance answer must say so. - **`denied` appearing at all.** It means the console could not read those objects — an SSE-KMS key policy that does not cover it, or an object owned by another account in an adopted bucket. It is re-issued every sweep rather than cached, so a permissions fix repairs it on the next tick with nothing to clean up. A `denied` count that stays flat after a fix means the fix did not work. - **Console box memory**, now carrying clamd's resident signatures — and the reload spike, not just the steady state (step 2). ## Rollback Unset `CARLOS_STORE_SCANNER` and restart. The pass stops, the doors 503, and every existing verdict tag stays on its object — harmless, and still readable if the engine comes back. Nothing needs undoing in a bucket. The one thing that does **not** roll back by unsetting the variable is an active `enforce` deny, which lives in the app's IAM policy. Clear it first with `carlos store scan enforce off` (which works with no engine built, by design) before unsetting, or an app keeps a Deny statement referring to a tag nothing will ever write again. Harmless in practice — the tag will not appear, so nothing is denied — but it is a stale statement in a live policy and it should not be left behind.