# No-inbound box runbook (private-subnet posture) > **STATUS: written for Paul's applies. Nothing in this document has been > applied, no tofu in it has been written into a `.tf` file, and no live box > was touched by the branch that added it.** It documents a network shape > for `carlos sidecars` design §8's `CARLOS_INBOUND=none` boxes; the module > that would produce it does not exist in this repo yet — authoring it is > Step 1 below, not a precondition already met. A `CARLOS_INBOUND=none` box accepts sidecar work only: no routed instance traffic, no certs, no tunnel-served upstream, no public status entry. This runbook covers the **hardest** version of that box — one that sits in a private subnet with no route to the internet at all, egress only through a NAT gateway, a security group with zero inbound rules, and no SSM agent, so the box has no **inbound** surface at all, from any direction, by any channel — no SSH, no SSM, nothing dialing in. It still has the platform's own outbound surface: the fleet tunnel and the sidecar's own agent-socket protocol keep working exactly as they do on any other fleet box (see "The box env contract" below), and those two are what every verification in this document ends up reading, because they are the only channels that exist. ## Read this first: do you actually need this? **Probably not yet.** Fleet boxes in this estate are *already* outbound-only dial — `infra/eleven-fleet.md` and `infra/remote-host.md` describe the shape every existing fleet box runs today: a security group with no inbound rules beyond what a normal host needs, admin reachability through SSM (or none, off-cloud), and all platform traffic — heartbeats, release pulls, the fleet tunnel — dialed **out**. Eleven's boxes in `titogo` are that shape right now. Design §8 is explicit about this: the private-subnet posture below is *available when wanted*, not a prerequisite for running a sidecar. Eleven's first sidecar box needs **no new network shape at all** — enrol it on the fleet box it already has, per `infra/eleven-fleet.md`, and stop reading here. Reach for this runbook only when outbound-only-at-the-security-group-layer isn't enough — when the requirement is "this workload must be provably unreachable even if every SG rule and every credential on the box were compromised," because there is, at the network layer, no path in to reach it with. That is a stronger claim than "nothing dials it today," and it costs a NAT gateway (an hourly + per-GB charge) and the loss of SSM break-glass access — weigh that before applying anything below. ## Preconditions (must be true before Step 1) - [ ] **Sidecar members merged and deployed.** The `CARLOS_INBOUND` env read (`internal/host/agent.go`), the placement half (`internal/provision/sidecar_reconcile.go`, `provision.PlacementNoInbound`), and the mint-time refusal on no-inbound boxes (`internal/provision/reconcile.go`) all need to be live on the box's `carlos` build — built fresh, not a stale checkout (recorded trap, `carlos-cli-releases.md`). - [ ] **The workload already exists as a fleet box**, following `infra/eleven-fleet.md`'s Ceremony through Step 3 (fleet created, box attached, `host.env` flipped and verified clean) — this runbook adds network hardening around a box that already dials the right console with the right credentials, it does not replace that ceremony. - [ ] **A decision on which existing VPC/subnet plan hosts this.** The sketch below assumes an existing VPC (e.g. `titogo`'s) with spare CIDR room for one new private subnet and either a shared or new NAT gateway. A from-scratch VPC is a bigger tofu change than this runbook covers. - [ ] **The sidecar's own record targets `placement: "no-inbound"`** (or its app's manifest does) — `carlos sidecars enable -app -placement no-inbound` (or `carlos sidecars create ... ` under a manifest already carrying it). Without this the record mints on any box in the pool, including ones with inbound service, and the box built below sits idle. - [ ] **Nobody is depending on SSM Session Manager reaching this box.** There is no admin channel to it once this ships except the fleet tunnel and the sidecar's own agent-socket protocol (design §7.1) — confirm that is acceptable for this workload before Step 3. ## The tofu shape Four pieces, applied in order because each one's Verify depends on the last existing. **No module in this repo produces this shape yet** — the HCL sketches below name the resources it needs, in this repo's existing naming conventions (`infra/modules/region-host/main.tf`, `infra/modules/region-remote-host/main.tf`), for whoever authors the module; they are not a file to copy-paste and apply as-is. **1. NAT gateway (skip if one already serves this VPC)** A NAT gateway must sit in an existing **public** subnet — one that already has a route to an internet gateway — even though nothing routes to it from outside. It is the private subnet's only way out. resource "aws_eip" "nat" { domain = "vpc" } resource "aws_nat_gateway" "sidecar" { allocation_id = aws_eip.nat.id subnet_id = } *Verify:* `aws ec2 describe-nat-gateways` (or the tofu plan/apply output) shows state `available`. This step alone costs money and touches no box — safe to apply and sit on. *Abort:* `tofu destroy -target aws_nat_gateway.sidecar -target aws_eip.nat` before anything below references it. Nothing yet depends on it. **2. Private subnet + route table (no IGW route)** The defining property: this subnet's route table has **no** `0.0.0.0/0 -> igw-...` route. Its only default route is to the NAT gateway from Step 1. resource "aws_subnet" "sidecar_private" { vpc_id = cidr_block = availability_zone = } resource "aws_route_table" "sidecar_private" { vpc_id = route { cidr_block = "0.0.0.0/0" nat_gateway_id = aws_nat_gateway.sidecar.id } # No route to an aws_internet_gateway, ever — that absence IS the # posture. Do not add one "temporarily" to debug something; that # defeats this runbook's entire point for as long as it's there. } resource "aws_route_table_association" "sidecar_private" { subnet_id = aws_subnet.sidecar_private.id route_table_id = aws_route_table.sidecar_private.id } *Verify:* `aws ec2 describe-route-tables --route-table-ids ` — confirm the route list contains exactly one `0.0.0.0/0` entry and its target is the NAT gateway, not `igw-...`. This is the single fact this whole runbook exists to guarantee; check it by reading the table, not by trusting the plan. *Abort:* the subnet and route table have nothing in them yet (Step 3 hasn't placed an instance) — `tofu destroy -target` both, no downstream cleanup. **3. Security group — zero inbound** Stricter than `region-host`'s SG (which still opens 80/443): this box serves no certs and no tunnel-terminated traffic, so it has nothing to listen on at all. resource "aws_security_group" "sidecar_no_inbound" { name = "-no-inbound" description = "sidecar-only box: no ingress by design" vpc_id = # No ingress block. Not "ingress restricted" — no ingress rule # exists to restrict. Do not add var.extra_ingress-style exceptions # to this SG; a box that needs an inbound rule is not this class of # box. egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } } *Verify:* `aws ec2 describe-security-groups --group-ids ` shows an empty `IpPermissions` list (ingress) and the single egress rule. *Abort:* unattached SG, `tofu destroy -target` it directly. **4. The instance — no SSM** Same AMI/instance-type shape as `infra/modules/region-host`, placed in the Step 2 subnet with the Step 3 SG, **no public IP**, and — the deliberate omission — no `aws_iam_role_policy_attachment` for `AmazonSSMManagedInstanceCore`. Without a route to the internet or a VPC interface endpoint for SSM (which this runbook does not add — an endpoint is itself a reachable surface), the SSM agent could not dial its service even if installed, so leaving the IAM grant off is not extra caution, it is removing a credential that would do nothing but sit there. **This means there is no break-glass path onto this box, ever, by anyone, other than what the sidecar and fleet-tunnel protocols carry.** That is the point, not an oversight — confirm the last Precondition checkbox again before applying this step. resource "aws_instance" "sidecar" { ami = instance_type = subnet_id = aws_subnet.sidecar_private.id vpc_security_group_ids = [aws_security_group.sidecar_no_inbound.id] associate_public_ip_address = false iam_instance_profile = user_data = } *Verify:* the instance reaches `running` in `describe-instances`, and — the important negative check — `aws ssm describe-instance-information` does **not** list it, ever, even minutes later. If it appears there, something above is wrong (a stray endpoint, a stray IGW route, or the SSM agent running with a grant it shouldn't have) — treat that as a failed apply, not a bonus. *Abort:* terminate the instance (`tofu destroy -target aws_instance.sidecar`); nothing else in Steps 1-3 depended on it existing. ## The box env contract `/etc/carlos/host.env` on this box carries the same fleet contract `infra/eleven-fleet.md` Step 3 stages, **plus** the one new variable that makes this a no-inbound box: AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials CARLOS_HIBERNATE_BUCKET= CARLOS_FLEET= CARLOS_FLEET_BOX= CARLOS_REGION=fleet/ CARLOS_STATUS_PREFIX=fleets// CARLOS_RELEASE_PUBKEY= CARLOS_TUNNEL_URL=https:// CARLOS_TUNNEL_NAME=/ CARLOS_TUNNEL_TOKEN= CARLOS_INBOUND=none `CARLOS_INBOUND=none` is read wherever `CARLOS_REGION`/`CARLOS_POOLS` are (`internal/host/agent.go`), folded case/whitespace-insensitively (`provision.NormalizeInbound`), and — deliberately — any value that is neither empty, `any`, nor `none` is treated as `none`: an unrecognised setting is a typo, and the restrictive reading is the safe one. Getting this var wrong fails closed (a box that should serve nothing accidentally stays inert) rather than open. The NAT gateway makes `CARLOS_TUNNEL_URL`'s dial-out (and the release/ heartbeat traffic to `CARLOS_HIBERNATE_BUCKET`) work exactly the way it does on any other fleet box — this box's whole platform surface is still outbound, it just has no *possible* inbound path underneath it, where an ordinary fleet box merely has no *configured* one. ## Verify the box came up and adopted its sidecars This box has no shell, no SSM, and — deliberately — no path into the platform's own log stream either: `internal/activator/logsink.go`'s `Event` calls are the only thing that ever writes to `CARLOS_STATUS_PREFIX`'s log sink, and nothing in `internal/host/agent.go`, `internal/tunnelclient` (the source of the "tunnel connected" line), or `internal/provision` (the source of a mint-time log line) calls it. So "go read a log line" is not an available verb for this box, on-box or off — every verification below uses a surface the platform already exposes with no box access of any kind: 1. **`carlos sidecars ls -app -long`** — the strongest single check. `HOST` names a route, `PLACEMENT` reads `no-inbound`, and once the child has called `/v1/ready` at least once, `VERSION` reads the running build instead of `unknown` (design §9.2's honesty rule: an absent version beats an overclaimed one). A reported version proves the whole chain end to end in one command — the box booted, dialed out, adopted the record, spawned the child, and the child called back over the agent socket. This is the check that matters most; do it first. 2. **`carlos fleets ls -account `** proves the box is **registered**, nothing more. Its `STATUS` column is `registry.FleetBox.Status` — an `active`/`detached` membership flag set at `add-box` time (`internal/console/fleets.go`), not a heartbeat. `infra/eleven-fleet.md`'s own Step 2 shows this plainly: it reads `active` immediately after `add-box`, before the box has ever booted. Reading `active` here tells you the token exists and hasn't been detached; it tells you nothing about whether the box has ever dialed in. Do not stop at this step and conclude the box came up. 3. **The console fleet page** (`/a//fleets/`) is the one surface that actually renders heartbeat freshness — `internal/console/fleets.go` joins each box's registry row to its own heartbeat by label and shows it Reachable (a fresh heartbeat joined) or gives its Age (a heartbeat joined but has gone stale) or neither (no heartbeat has ever joined at all). This is a browser-only check — no CLI verb surfaces it as of this branch — but it needs no box access, only a signed-in session, so it still fits this runbook's "no box commands" constraint. 4. **`aws ec2 get-console-output --instance-id `** if 1 and 3 both stay stale past a reasonable wait and you need to see the box actually trying to boot. This is the one channel that needs no reachable surface on the box at all — it reads whatever the instance wrote to its serial console (cloud-init/systemd boot log) from the AWS API side, not the box's. It is the sole operator-side breadcrumb for a box with neither SSH nor SSM, and the reason to leave the AMI's normal verbose boot logging on rather than quieting it for this box class. There is no honest check for "did the provisioning pass log a mint-time refusal or success" from off the box — if 1 and 3 both look right but the record still isn't where you expect, the *Abort* below is the next step, not a log line to go find. *Abort (this section):* if the sidecar never mints here, check placement before network: `carlos sidecars ls -app -long` showing `PLACEMENT any` means the record itself never asked for `no-inbound` — fix the manifest/record, not the box. If placement is right and it still doesn't mint, there is no way to inspect the running process's actual environment on this box (no shell, no SSM) — treat `CARLOS_INBOUND` as unfalsifiable in place. The fix is the same one Rollback summary's last entry describes for this box class: there is no in-place restage, so terminate the instance and re-apply Step 4 with the value double-checked in its user-data, then re-run checks 1 and 3 above against the replacement. ## Flipping an EXISTING box to `CARLOS_INBOUND=none` **This does not evacuate the instances the box already serves.** Setting `CARLOS_INBOUND=none` on a box that currently runs routed instances (or was reachable and had inbound work placed on it) only refuses **new** routed mints from then on — it is a mint-time check (`internal/provision/reconcile.go`'s `case !ok:` branch), not a converge- time or removal-time one. Every route already `Provisioned` on that box keeps running, keeps converging, keeps being served, exactly as before. This is deliberate, not an oversight: making the flag evacuate the box would mean a single typo in `host.env` — a fat-fingered `CARLOS_INBOUND=none` meant for a different box, or a stray copy-paste — could take down every app the box was serving, with no confirmation step and no way to see it coming. A box that stops *accepting new work* loudly logs every refusal; a box that *stops serving existing work* on the same typo is a production incident with the blast radius of whatever happened to be running there. **So: drain the box before flipping it, not after.** Move its existing routed instances off (`carlos apps place` to another target, or whatever this box's own placement mechanism is) and confirm they've actually stopped being served here — a `curl -sI --resolve` against each affected host showing it answered from elsewhere, not just that placement changed — before staging the new `host.env`. Only once the box carries no routed work should `CARLOS_INBOUND=none` land on it; at that point it is a statement of fact about the box, not an instruction to make it one. ## Rollback summary Every step above has its own Abort; in order of how far you got: - Only Steps 1-3 applied, no instance yet: `tofu destroy -target` each resource in reverse order (SG, route table + association, subnet, NAT gateway + EIP) — nothing else references them. - Instance applied, box never adopted anything: `tofu destroy -target aws_instance.sidecar`, then the network resources as above. - Box adopted a sidecar, now going away: detach it from the fleet first (`carlos fleets detach-box -account ` — the hard cut, per `infra/eleven-fleet.md`'s own Abort for that step) so the console stops expecting heartbeats from it, then terminate the instance and tear down the network resources. - Flipped an existing box and want to revert to accepting inbound work again: restore the prior `host.env` (drop `CARLOS_INBOUND=none`, or set it back to `any`), restart the agent unit — this is a converge-time read, so the very next pass allows new mints again. **This step assumes the ordinary shell/SSM access an existing fleet box normally has.** It does NOT apply to a box built via this runbook's own Step 4, which has neither: there is no in-place way to edit that box's `host.env` at all. Reverting one of those means replacing it — terminate the instance and re-apply Step 4 with the desired `host.env` baked into its user-data — not restaging a file on a running box. It does not retro- actively place anything that was moved off during the drain; that's a separate `carlos apps place` back, a deliberate decision, not a side effect of the env flip.