# region-host: bringing the box up

> 🤖 Written by an LLM (Claude), per factor X.

`tofu apply` gives you a machine with an IP, an IAM role and no CARLOS on
it. This is the other half: what to put on the box and what to tell it, so
that it becomes a region of your deployment.

It lives beside the module on purpose. The knowledge here used to sit only
in the private infrastructure repo, which meant a deployment could triple
its fleet with no committed way to bring a box up — and did (discussion #7).

## The four steps

Run them over SSM Session Manager. There is no SSH ingress and there is not
meant to be.

1. **Put the `linux/arm64` binary at `/opt/carlos/carlos`**, mode 755. How it
   gets there is your deployment's business — the host role can read your
   deployment bucket, so pulling it from wherever your build publishes it is
   the usual answer. The binary is static and has no runtime dependencies.

2. **Write `/etc/carlos/host.env`**, mode 0600, root-owned. The minimum:

   ```sh
   CARLOS_DEPLOYMENT_BUCKET=carlos-myco-platform-123456789012
   CARLOS_RELEASE_PUBKEY=<the deployment's public release key>
   CARLOS_REGION=ie
   AWS_REGION=eu-west-1
   ```

   That is enough to make the box an **edge**: it will terminate TLS, adopt
   releases, converge routes and serve static sites. It is not enough to run
   an app. If this region is to **host instances**, it also needs its own
   data bucket — the one `tofu apply` already made and granted the host role:

   ```sh
   CARLOS_HIBERNATE_BUCKET=carlos-myco-ie-123456789012
   CARLOS_HIBERNATE_REGION=eu-west-1
   CARLOS_HIBERNATE_PREFIX=instances
   ```

   Without those, the hibernation activator never arms, and nothing on the
   box ever starts an instance's process. The failure is quiet from the
   box's side — the record is accepted, the route is minted and the binary
   is staged, and only the request finds out — so the box now publishes
   what it was told as `hibernation_configured` in its edges file, and the
   console refuses instance creates into a region whose boxes report false
   rather than accepting one that would strand. Add the variables and
   restart `carlos-edge` and the region arms itself within a minute or two.

   `CARLOS_HIBERNATE_REGION` is the AWS region of the **data bucket**, and
   like `AWS_REGION` above it is a real AWS region name, not a placement
   code.

   `AWS_REGION` is the one that bites. It must name the region the
   **deployment bucket** lives in, not the region this box is in. The AWS Go
   SDK does not follow S3's cross-region redirects, so a box in Sydney
   pointed at an Irish deployment bucket with `AWS_REGION=ap-southeast-2`
   fails every read with `PermanentRedirect: 301` — a day was lost to this in
   one deployment before anyone wrote it down.

   `CARLOS_REGION` is the region **code** this box serves (`ie`, `au`, `pi`),
   not an AWS region name. It is what makes the box claim its own region's
   work.

3. **Run `carlos bootstrap`** (as root). It creates the service user and the
   consumers group, writes the data and runtime dirs, installs the systemd
   units from inside the binary, and arms the timers and path units. It is
   idempotent: re-running it is how you refresh unit files after a binary
   upgrade, and `carlos bootstrap -root <dir>` stages the same files into a
   directory tree without touching the live system, which is what an
   automated upgrade path should use.

4. **Start the edge**: `systemctl start carlos-edge`. From here the box
   polls `control/serial`, adopts releases, and serves.

## There is no per-region secret

Worth stating plainly, because at least one deployment's region files
asserted the opposite and the wrong comment was copy-pasted into three more
before anyone caught it. Following it would have put the root signing key in
six regions.

A host **verifies** releases. It does that with `CARLOS_RELEASE_PUBKEY`,
which is public — the whole point of the signing scheme is that the private
half never leaves your secret store and never touches a box. There is
nothing region-specific to create, copy or rotate when you add a region.

What a region does need is in the four steps above and nothing more: the
binary, `host.env`, `carlos bootstrap`, start the edge. The only thing that
is easy to leave out is the hibernation block in step 2 — and leaving it out
is what makes a region an edge that cannot host.

## Arming the root passes

Two path units let the unprivileged agent ask a root pass to do something it
cannot do itself — stage a release into `/opt/carlos` (`carlos-stage.path`)
and cycle a unit-backed route (`carlos-restart.path`). `carlos bootstrap`
arms both. If you upgrade the binary by hand rather than through your own
platform-update script, re-run `carlos bootstrap` so any new unit is
installed and armed; a stage relay that is not armed fails silently, which
looks exactly like a deploy that did nothing.

## Object-storage scanning

`store_scanning = true` lets this host's role read and tag objects under `store_bucket_prefix` (`s3:GetObject`, `s3:GetObjectTagging` and `s3:PutObjectTagging`), which is what the console's virus scanner needs. It is off by default, so a deployment that has not switched scanning on plans no change and its console still never reads tenant data. Set it on the console's box only, and only on a deployment that runs a [scanner-host](../scanner-host/README.md). It requires `store_bucket_prefix`.

## Checking it worked

```sh
systemctl status carlos-edge
curl -sI --resolve <host>:443:<eip> https://<host>/ | grep -i x-carlos-version
```

Use `--resolve` against the box's own EIP rather than trusting DNS: a stale
answer from a resolver has produced more than one false "verified" report.

## Termination evidence

The module creates a queue, `<iam_name>-terminations`, that receives an
event whenever an EC2 instance in this account and region reaches
`terminated`. Region HA reads it to prove a dead box is gone before its
instances run elsewhere. Only the module's own EventBridge rule can write
to the queue, and the box role can only read and delete from it.

After the first apply, check that nobody but the rule can write to it.
From an administrator's session:

    aws sqs send-message --queue-url <termination_queue_url> --message-body '{"forged":true}'

must fail with `AccessDenied`. Then terminate a throwaway instance in the
region and confirm its event arrives.

The root volume is deleted when the instance is terminated. Region HA
depends on that: a terminated box must leave no disk behind.

Open item for region HA plan 1b: the box role's `ec2:DescribeInstances`
is unconditioned, because the infrastructure inventory reads every
region. The HA spec asks for an `ec2:Region` condition on the call the
placer makes. Decide there whether that is a second, narrower statement.
