# moveprobe moveprobe is a small app for rehearsing instance moves. It keeps state in every place a real app can, so that after a move you can check that each kind of state arrived. It is not part of the platform release. You build it and ship it like any other app. ## What it keeps - **Database rows.** A background loop writes a row every half second. A separate worker process writes one every second. Each timer fire writes one. Every row records the region it was written in. - **A key file.** On first boot it writes `instance.key` into the instance's files directory and records the same key in the database. Every boot checks that the two still match. - **The platform's instance key.** On first boot it records a fingerprint of `CARLOS_INSTANCE_KEY`. Every boot checks that the key is still set and unchanged. - **An object in its store.** On every boot it writes `probe/.txt` to the store bound with the prefix `MOVEPROBE`, and reads it back on every readiness check. - **A sealed secret.** It needs `MOVEPROBE_SECRET` to be set. - **Timers.** A once-timer you arm through the app, and a schedule you set with `carlos schedule set`. ## Build The rehearsal boxes are arm64: ```sh CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o moveprobe ./cmd/moveprobe ``` ## Deploy Run these signed in to the ops account, unless the rehearsal names another. Use the deployment's instance domain where these say `example.com`, and pick a host name no earlier rehearsal has used. ```sh carlos ship --app moveprobe --label "moveprobe" ./moveprobe carlos instances enable --app moveprobe carlos store create --app moveprobe --env-prefix MOVEPROBE carlos secrets set --app moveprobe MOVEPROBE_SECRET= carlos schedule set --app moveprobe --name every-minute --every 1m --path /tick carlos instances create --app moveprobe --host mp1.example.com --region ie ``` Leave private storage off when you enable instances, and create the store without `--environment`. A move only carries the app's own store. It refuses an instance whose store belongs to an environment or to the instance itself. The account must be approved for object storage before `store create` provisions anything. Keep the value of `MOVEPROBE_SECRET` in your shell. It is also the password for `/arm-once` and `/report`. ## Use it ```sh curl https://mp1.example.com/ready curl -X POST -H "Authorization: Bearer $MOVEPROBE_SECRET" "https://mp1.example.com/arm-once?in=600" curl -H "Authorization: Bearer $MOVEPROBE_SECRET" https://mp1.example.com/report ``` | path | who can call it | what it does | |---|---|---| | `/` and `/ready` | anyone | 200 when the key file, the platform key, the stored object and the secret all check out, and the worker's writes are landing. Otherwise 500 with every failed check. The store check runs at most once every 5 seconds, and requests in between share its result. A move probes `/`. | | `GET /report` | `MOVEPROBE_SECRET` | JSON: row counts by kind and region with their lowest and highest sequence numbers, the timer fires by name and due time, both key fingerprints, the stored object, whether the secret is set, the worker's pid and failed writes, and the region the app saw. Any failed database read makes it a 500, never a partial report. | | `POST /arm-once?in=` | `MOVEPROBE_SECRET` | Arms a once-timer named `once-` that calls `/once` after that many seconds. Answers 503 when the instance has no control socket. | | `POST /tick`, `POST /once` | the platform | Timer fires. Each one writes a row with the timer's name and due time. | The background rows are numbered in one sequence that carries on across boots and regions. If no row is missing, the count of `bg` rows equals the highest `bg` sequence number. The region comes from `CARLOS_REGION`. On a box that has no region, the app uses `home`, and `/report` says so. ## Make readiness fail on purpose `MOVEPROBE_FAIL_READY_IN=` makes the app fail its readiness check in that region. The app reads it when it starts, so set it before you request the move, and remove it once the run's checks are recorded: ```sh carlos env set --app moveprobe MOVEPROBE_FAIL_READY_IN=ie carlos env unset --app moveprobe MOVEPROBE_FAIL_READY_IN ``` There are two modes: - `MOVEPROBE_FAIL_READY_IN=ie` writes nothing at all in `ie`. It does not open the database, touch the files directory, start the worker or call the control socket. Every path answers 503. A rollback from here should be clean. - `MOVEPROBE_FAIL_READY_IN=ie,after-write` writes one background row in `ie`, starts the worker, and then answers 503 on `/` and `/ready`. A rollback from here is dirty, and the source should end up holding that row. In any other region the variable has no effect. ## The worker The app starts a worker process in its own session, with no connection to the app's output. Stopping the app does not stop the worker. That is the point: it shows whether a move's stop really stops every process the instance started. Once the app has gone, the worker keeps writing for 60 seconds and then exits by itself. The worker tells the app whether each write landed. `/report` shows how many committed and how many failed since the app started, how many of the latest failed in a row, the last error, when it last heard from the worker, and why the worker is gone if it is. `/ready` fails until a worker row has committed since the app started, whenever the worker's latest write failed, if the worker has sent nothing for 3 seconds, and once the worker has exited. ## Strings Every body the app can send, and when. Text in angle brackets is filled in when the app answers. `/ready` joins its failed checks with `; `. On `/` and `/ready`: - `ready`: every check passed (200). - `before-write mode: this box writes nothing`: the failure switch names this region with no suffix. Every path gives this answer (503). - `after-write mode: not ready here`: the failure switch names this region with `,after-write` (503). - `key mismatch: instance.key exists but the database has no key`: the files directory has a key file, but the database never recorded one (500). - `key mismatch: instance.key is missing`: the database has a key, but the file is gone (500). - `key mismatch: instance.key differs from the database`: the file and the database hold different keys (500). - `key mismatch: instance.key: `: the file is there but cannot be read (500). - `CARLOS_INSTANCE_KEY is not set`: the platform key is missing (500). - `CARLOS_INSTANCE_KEY differs from the one seen at first boot`: the platform key changed (500). - `MOVEPROBE_SECRET is not set`: the sealed secret is missing (500). - `store: not set`: one or more of the store variables is missing (500). - `store: put : `: the boot's object could not be written (500). - `store: get : `: the boot's object could not be read back (500). - `store: reads back different bytes`: the object came back changed (500). - `store: `: the request gave up while waiting for a store check that another request had started (500). - `worker: exited ()`: the worker has exited. `` is how it ended, such as `exit status 1` or `signal: killed`, or `its status pipe closed` when the app saw it go before learning how (500). - `worker: did not start: `: the app could not start the worker (500). - `worker: no status for s`: the worker has not reported a write for 3 seconds or more, for example because it is stuck (500). - `worker: no row committed since start`: the worker has not written a row since the app started, for example in its first moments or if it did not start (500). - `worker: no row committed since start; last error: `: as above, and its writes are failing (500). - `worker: the last write failed: `: the worker's latest write failed (500). On `/report`: - The JSON report (200). Its fields are `host`, `region`, `region_from`, `mode`, `boot_id`, `rows` (each with `kind`, `region`, `count`, `min_seq`, `max_seq`), `fired` (each with `kind`, `name`, `due_at`, `count`), `file_key_sha256`, `platform_key_sha256`, `object` (with `key`, `body` and, when it could not be read, `error`), `has_secret`, `worker_pid`, `worker_writes_committed`, `worker_writes_failed`, `worker_failed_in_a_row`, `worker_last_error`, `worker_exit` and `worker_last_status_at` (unix seconds, 0 before the first). - `report: `: a database read failed (500). The values the app writes into the report itself: - `region`: the value of `CARLOS_REGION`, or `home` when the box has no region. - `region_from`: `CARLOS_REGION` when the variable is set, and `unset, treated as home` when it is not. - `mode`: `normal`, or `after-write` when the failure switch names this region with `,after-write`. Before-write mode sends no report. - `kind` in `rows`: `bg` for the background loop, `worker` for the worker, `tick` and `once` for timer fires. `kind` in `fired`: `tick` or `once`. - `object.key`: `probe/.txt`, the object this boot wrote. - `object.body`: the object's contents as read back. The app writes `moveprobe boot host region at