# Remote host runbook (software half)

A CARLOS host is anything that can reach the deployment bucket. A remote
host is OUTBOUND-ONLY: its whole network surface is outbound 443 to S3 and
the edge tunnel. No inbound port, no SSH path required by the platform.

Prereqs: systemd Debian-family OS, 64-bit (the artifact key is
GOOS-GOARCH — linux-arm64 or linux-amd64; a 32-bit OS resolves nothing).

## Steps

1. Provision the IAM principal + region data bucket
   (`infra/modules/region-remote-host`). Scope: deployment bucket + this
   host's own region bucket, and NOTHING else — wrong-region-fails-at-IAM
   holds off-cloud too. A scoped access key is acceptable to start; IAM
   Roles Anywhere is the long-term, hq-grade alternative (no long-lived
   key on the box at all).

2. Stage `/etc/carlos/credentials`, owned by root, group `carlos`, mode 0640:

       [default]
       aws_access_key_id=...
       aws_secret_access_key=...

   Static keys only. `carlos bootstrap -offcloud` refuses to proceed if this file is missing or is not mode 0600 or 0640. It will name this path in the error. The agent will not use a file that holds anything else, such as a `credential_process` line or a second profile.

3. Stage `/etc/carlos/host.env` — the usual agent env, plus the
   credentials pointer and the three tunnel variables. All three tunnel
   variables are required together; if any is missing the dial-out loop
   stays inert (no crash, no retry storm — it just never dials):

       AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials
       CARLOS_TUNNEL_URL=https://<edge's reserved tunnel hostname>
       CARLOS_TUNNEL_TOKEN=<this host's token>
       CARLOS_TUNNEL_NAME=<this host's channel name>

   Do not set `AWS_PROFILE` or `AWS_DEFAULT_PROFILE` in `host.env`. The agent ignores both and logs an alert.

4. Stage the binary at `/opt/carlos/carlos` (matching GOOS-GOARCH).

5. Run: `carlos bootstrap -offcloud`

   This does everything the normal `carlos bootstrap` does (units, dirs, service user, timers) and, before any of it, checks the credentials precondition from step 2. A failed check leaves the box untouched. On a live root, bootstrap and every agent start rewrite the credentials file as a new file owned by root, group `carlos`, mode 0640, keeping only the static keys. The service user can read it but not change it. A file that is already right is left alone.

6. `systemctl start carlos-edge.service`

7. Verify:
   - `journalctl -u carlos-edge` shows "tunnel connected".
   - The box appears in `carlos status -live` within one heartbeat
     (5 minutes + jitter).

## Edge side (once per edge box, not per remote host)

Set in the edge box's `host.env`:

    CARLOS_TUNNEL_HOST=<reserved hostname>
    CARLOS_TUNNEL_CLIENTS=name=token,name2=token2,...

`CARLOS_TUNNEL_HOST` must be a bare hostname — no port, no scheme. Each
name in `CARLOS_TUNNEL_CLIENTS` must match `^[a-z0-9-]{1,32}$` or the edge
refuses to boot.

`carlos route` repoints an EXISTING route — it does not create one. Add the
route first, then back it with the tunnel channel:

    carlos add <h>
    carlos route -host <h> -tunnel <name>

## Fleet boxes (customer fleets)

Fleet boxes use the same agent with the fleet env contract specified in the
customer-fleets design spec (private `platform-infrastructure` repo, which
also holds the worked Pi example). The `carlos add`
and `carlos route -tunnel` ceremony documented above applies only to platform
remote boxes (tito hq), NOT fleet boxes — fleet instances are placed at app
create time and boxes auto-join fleet registries.

## Security note

In `-dev` (HTTP) edge mode the registration bearer token travels in clear
text. Tunnel hosts must only ever dial a real-TLS edge in production —
never point `CARLOS_TUNNEL_URL` at an HTTP edge outside a dev sandbox.

## Known limitations

- **Apps see the edge's IP, not the real client's.** A tunnel-backed request
  crosses two reverse-proxy hops: client → edge (over the internet), then
  edge → app (over the tunnel, on the remote host's own local edge). The
  second hop re-stamps `X-Forwarded-For`/`X-Forwarded-Proto` from its own
  immediate peer, so an app on a tunnel-backed host sees the edge's IP as
  client, and `X-Forwarded-Proto: http` even though the original request was
  TLS. This fails safe — client identity is dropped, never spoofed — but any
  app on a tunnel-backed host that logs or gates on client IP will be wrong
  today.
- **No WebSockets to a tunnel-backed host.** The tunnel transport can't carry
  a `Connection: Upgrade` handshake; every WebSocket attempt to an app on a
  tunnel-backed host fails with a 502 (loud, not a hang). Plain HTTP request/
  response is unaffected. RFC 8441 extended CONNECT is the likely future fix.

## Related

The hq PHYSICAL half (racking, power, network uplink for a region-hq box)
lands separately in `infra/tito/region-hq.md`. For a concrete Pi
instantiation of this runbook (values, seeding), see the private
`platform-infrastructure` repo.