# Platform GeoIP, plan 2: the feed and the `geoip` alert

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (this plan is executed natively, as plan 1 was). Steps use checkbox (`- [ ]`) syntax.

**Goal:** The console keeps the published database current by itself, fetching DB-IP Lite or GeoLite2. Every box raises a `geoip` alert, through the existing box-alert pipeline, when its database is stale, expiring, expired or not loaded, or (on the console's box only) when the feed keeps failing.

**Architecture:**
- **The fetcher** lives in the console (`internal/console/geoipfeed.go`). It is the one control-plane process, so there is only one fetcher. It reuses plan 1's publish slot (`geoipPublisher.Stage`/`Commit`), adding two feed-only rules: the type must match the configured feed, and an edition must be newer than the last one from the same feed.
- **Archives** are opened by pure, capped readers in `internal/geoip/archive.go`.
- **The feed's state** goes to `control/geoip/feed.json` after every check, naming the console's box.
- **The alert** is a fifth box rule beside disk, mem, cpu and lag. Each mirror computes a reason code. `metrics.Evaluate` applies its sustain, and the host's alert sink and the console's alert watcher deliver it, unchanged.

**Tech Stack:** Go stdlib (`compress/gzip`, `archive/tar`, `net/http`), plan 1's packages.

**Spec:** `docs/superpowers/specs/2026-10-06-platform-geoip-design.md`, §10 (rev 11 or later). Plan 1 is built on this branch; its code is the base.

## Global Constraints

- **Console env:**
  - `CONSOLE_GEOIP_FEED`: one of `dbip-city-lite`, `dbip-country-lite`, `geolite2-city`, `geolite2-country`;
  - `CONSOLE_GEOIP_MAXMIND_ACCOUNT` and `CONSOLE_GEOIP_MAXMIND_KEY`.
  - An unknown name, or GeoLite2 without both credentials, is logged once at startup, and the feed stays off.
- **Feed name to type:**
  - `dbip-city-lite` → `DBIP-City-Lite`
  - `dbip-country-lite` → `DBIP-Country-Lite`
  - `geolite2-city` → `GeoLite2-City`
  - `geolite2-country` → `GeoLite2-Country`
- **URLs:**
  - DB-IP: `https://download.db-ip.com/free/dbip-<city|country>-lite-YYYY-MM.mmdb.gz`
  - MaxMind: `https://download.maxmind.com/geoip/databases/GeoLite2-<City|Country>/download?suffix=tar.gz`, with `suffix=tar.gz.sha256` for the checksum.
- **Caps:**
  - compressed download 512 MiB, enforced while copying;
  - all expanded tar bytes 1.5 GiB, skipped members included;
  - the `.mmdb` itself 1 GiB (plan 1's `copyCapped`);
  - checksum file 4 KiB;
  - `feed.json` 64 KiB;
  - each check under the publisher's 15-minute deadline.
- **Redirects:** at most 3, `https` only. The Basic `Authorization` header is dropped on a cross-host redirect. The key never appears in a URL or a log line.
- **Schedule:** a first check 5 minutes after start, then daily with up to an hour of jitter.
- **Seven** consecutive 404/410 answers raise the alert's feed reason.
- **Alert codes:** the rule's `Value` is the reason code; 0 = none.

  | Code | Reason |
  |---|---|
  | 1 | loaded database older than 45 days |
  | 2 | GeoLite2 within 9 days of D1 expiry (21 days old) |
  | 3 | the pointer is expired, marked or by the clock |
  | 4 | a live pointer has not been loaded for over an hour |
  | 5 | the feed has failed 7 times in a row (console box only) |

  When several hold at once, the most severe wins: 3, then 4, then 2, then 1, then 5.
- **The rule** fires after the reason has held for 30 minutes (`Rules.GeoIPSustain`). It clears at once when the code returns to 0.
- **Copy:** every new alert sentence and log line follows the house voice (no em dashes). The alert sentences join plan 1's copy review.

## Review Focus

1. **A feed that serves a different product, an older edition, or a truncated file** must leave the published database untouched.
2. **A redirect** to another host must never carry the MaxMind credentials.
3. **A decompression bomb, or a tar padded with junk members,** must stop at the caps without filling the disk.
4. **The feed reason** must fire on exactly one box, the console's, and must still fire on a fresh deployment where the feed has never succeeded.
5. **`off`** must clear every pointer-related reason; expiry must not.

---

### Task 1: `internal/geoip/archive.go` and `feed.go`

**Files:** Create `internal/geoip/archive.go`, `internal/geoip/feed.go`, `internal/geoip/archive_test.go`, `internal/geoip/feed_test.go`.

**Produces:**

```go
// feed.go
type FeedSpec struct{ Name, Type, Provider string } // Provider: "dbip" | "maxmind"
func ParseFeed(name string) (FeedSpec, bool)
func DBIPURL(spec FeedSpec, month time.Time) string
func MaxMindURL(spec FeedSpec, suffix string) string
type FeedStatus struct {
	V                   int       `json:"v"`
	Feed                string    `json:"feed"`
	Box                 string    `json:"box"`          // the console's box label: the one box that alerts on feed failure
	LastCheck           time.Time `json:"last_check"`
	LastSuccess         time.Time `json:"last_success,omitempty"`
	LastEdition         string    `json:"last_edition,omitempty"`
	ConsecutiveFailures int       `json:"consecutive_failures"`
	LastFailure         string    `json:"last_failure,omitempty"` // a fixed category, never an error text
	ConsecutiveNotFound int       `json:"consecutive_not_found"`
}
const FeedNotFoundAlert = 7
func ParseFeedStatus(raw []byte) (FeedStatus, error)
func (s FeedStatus) Failing() bool // ConsecutiveFailures >= 7 || ConsecutiveNotFound >= 7

// archive.go
var ErrTooLarge = errors.New("geoip archive: larger than its cap")
func CappedReader(r io.Reader, max int64) io.Reader        // returns ErrTooLarge once more than max bytes are read
func GunzipReader(r io.Reader, maxExpanded int64) (io.Reader, error)
func TarGzMMDB(r io.Reader, maxExpanded int64) (io.Reader, error) // the first regular *.mmdb member; every expanded byte counts
```

- [ ] **Step 1: Tests.**
  - `ParseFeed` returns the four names with their types and providers, and rejects anything else.
  - `DBIPURL` for `dbip-city-lite` in 2026-10 is `https://download.db-ip.com/free/dbip-city-lite-2026-10.mmdb.gz`.
  - `CappedReader` errors with `ErrTooLarge` at max+1 and passes exactly max.
  - `GunzipReader` on a 10 MiB run of zeros gzipped, with a cap of 1 MiB, gives `ErrTooLarge` after reading about 1 MiB, never the full expansion.
  - `TarGzMMDB` returns the bytes of `db.mmdb` from an archive whose first member is a 2 MiB `README`, and errors with `ErrTooLarge` when the cap is 1 MiB.
  - `TarGzMMDB` on an archive with no `.mmdb` member gives a clear error.
  - `FeedStatus` round-trips through JSON.
  - `Failing` crosses at exactly 7.
  - Build the test archives in memory with `archive/tar` and `compress/gzip`.
- [ ] **Step 2: Implement.**
  - `CappedReader` counts bytes, and once the count exceeds max it returns `ErrTooLarge`.
  - `GunzipReader` is `CappedReader(gzip.NewReader(r), maxExpanded)`.
  - `TarGzMMDB` wraps the gunzipped stream in one `CappedReader(…, maxExpanded)`, so skipped members count. It walks `tar.NewReader` to the first `TypeReg` member whose name ends `.mmdb` (path base) and returns that member's reader.
- [ ] **Step 3:** `go test ./internal/geoip`, then commit.

### Task 2: publish rules for the feed

**Files:** Modify `internal/console/geoippublish.go`, `geoippublish_test.go`.

**Produces:** `geoipPublishOpts` gains `RequireType string` (refuse unless `db.Source()` equals it) and `Monotonic bool`. With `Monotonic` set, the publish is refused when the current pointer has the same `Origin` and the candidate's `BuildEpoch` is not after the current one's. The feed sets both and never sets `Force`.

- [ ] **Step 1: Tests.**
  - `RequireType: "GeoIP2-Country"` refuses the City fixture with the category "the feed returned a GeoIP2-City database, not GeoIP2-Country".
  - With `Monotonic`, `Origin: "feed:x"`, publishing the City fixture twice refuses the second as "not newer than the published edition". The pointer is unchanged.
  - With `Monotonic` against a current pointer whose `Origin` is `manual`, the publish proceeds.
- [ ] **Step 2: Implement it in `Commit`,** after the country probe and before the shrink check.
- [ ] **Step 3:** Run the tests, then commit.

### Task 3: the feed

**Files:** Create `internal/console/geoipfeed.go` and `geoipfeed_test.go`. Modify `internal/console/server.go` (Config `GeoipFeed`, `GeoipMaxMindAccount`, `GeoipMaxMindKey` and `BoxLabel`; start the feed from `StartGeoipJobs`), `cmd/console/main.go` (read the env; `BoxLabel: status.Label(<machine id>)`, reading `/etc/machine-id` the way `internal/host/probes.go` does).

**Produces:**

```go
type geoipFeeder struct {
	pub         *geoipPublisher
	store       release.Store
	spec        geoip.FeedSpec
	account, key string
	box         string
	client      *http.Client // CheckRedirect: geoipFeedRedirect
	now         func() time.Time
	dbipBase, maxmindBase string // test seams; "" = the real hosts
}
func newGeoipFeeder(cfg Config, pub *geoipPublisher) (*geoipFeeder, bool) // false = no feed (unset or misconfigured, logged once)
func (f *geoipFeeder) Check(ctx context.Context) error                   // one check; always writes feed.json
func (f *geoipFeeder) run(ctx context.Context)
func geoipFeedRedirect(req *http.Request, via []*http.Request) error      // ≤3, https only, drops Authorization cross-host
```

`Check`:
1. Read the pointer and `feed.json` (capped).
2. **DB-IP.**
   - The edition is this month's `YYYY-MM`. If the pointer's `Origin` is this feed and its `Edition` is this month, the check is done (a success, nothing to do).
   - Otherwise GET this month's URL, and on a 404 last month's. Only when there is no pointer, or the pointer came from elsewhere, is last month tried, because a pointer from this feed already holds last month.
   - A 404 or 410 on every candidate is a "not out yet" check. `ConsecutiveNotFound` goes up by one; it is not a failure.
3. **MaxMind.**
   - HEAD the archive URL with Basic auth. If `Last-Modified` equals the pointer's `Edition` (same feed origin), the check is done.
   - Otherwise GET `suffix=tar.gz.sha256`: the first 64 hex characters, capped at 4 KiB.
   - Then GET the archive.
4. **Stream the body:**
   - `CappedReader(body, 512 MiB)`, teed into a sha256 hasher;
   - then `GunzipReader` (DB-IP) or `TarGzMMDB` (MaxMind), capped at 1.5 GiB;
   - then `pub.Stage`. `errGeoipBusy` skips this check; it is not a failure.
5. **MaxMind checksum:** drain the rest of the archive through the tee (capped), and compare the archive's sha256 with the published one. On a mismatch, `Abandon` and record the failure "checksum".
6. **Commit** with `geoipPublishOpts{Origin: "feed:" + name, Edition: edition, RequireType: spec.Type, Monotonic: true}`.
7. **Record the outcome.** Success sets `LastSuccess` and `LastEdition`, and zeroes both counters. A refusal or error increments `ConsecutiveFailures` and sets `LastFailure` to a fixed category:
   - `download` / `not found` / `too large` / `checksum` / `invalid` / `wrong type` / `not newer` / `expired` / `store`.
8. **Write `feed.json`** with `Box: f.box`, on every path.

`run`: wait 5 minutes, then check; then wait 24 hours plus up to 1 hour of jitter, and check again. If `CONSOLE_GEOIP_FEED` is unset at startup, delete any `feed.json`, so a switched-off feed alerts nobody.

- [ ] **Step 1: Tests,** against an `httptest.NewTLSServer` standing in for both providers. Its client is the server's client, wrapped with `geoipFeedRedirect`. Use `dbipBase`/`maxmindBase` pointing at it, and the gzip/tar.gz of the GeoIP2 fixtures. Feed types are matched through `RequireType`, so use a `FeedSpec{Type: "GeoIP2-City"}` built by hand:
  - **DB-IP:** a new edition is published, `feed.json` says success, and the edition is this month. A second check makes no GET at all, because it's the same month and the same origin.
  - **404 handling:** this month 404s and last month serves; with no pointer, last month is published. Every candidate 404s and `ConsecutiveNotFound` goes up by one each check; after 7, `FeedStatus.Failing()` is true.
  - **MaxMind:**
    - The HEAD with an unchanged `Last-Modified` makes no GET.
    - A changed one fetches `.sha256` and then the archive, and publishes.
    - A wrong `.sha256` records "checksum", and the pointer is unchanged.
    - The test server asserts that every request carries Basic auth, and that the key appears in no URL.
  - **Redirect:** the archive URL redirects to a second `httptest` TLS server, on another host. That server sees no `Authorization` header. A 4th redirect is refused.
  - **Bomb:** a gzip of 2 GiB of zeros (generate it streamed) is refused as "too large", with the staging directory empty afterwards. For speed, make the expanded cap a field set to 1 MiB in the test.
  - **A feed type mismatch** is refused as "wrong type", even with no pointer at all (the first publish).
  - **A busy slot** (`pub.mu` held) skips the check without counting a failure.
- [ ] **Step 2:** Implement it.
- [ ] **Step 3:** Run the tests, then commit.

### Task 4: the mirror's alert code and `feed.json`

**Files:** Modify `internal/edge/geoipmirror.go`, `internal/edge/geoipmirror_test.go`.

**Produces:**
- `geoMirror` tracks `wantSince` (when the pointer first named a sha that isn't loaded) and `pointerExpired`. It also reads `feed.json` (capped, best effort) on each poll, keeping `feedFailing` and `feedBox`.
- `func (m *geoMirror) AlertCode(now time.Time, box string) float64` implements the priority above.
- `func (e *Edge) GeoIPAlertCode(now time.Time) float64` uses the edge's box label (`WithBoxLabel`). It returns 0 when there is no mirror.

- [ ] **Step 1: Tests:**
  - each code from a constructed state;
  - code 4 only after an hour;
  - code 5 only when `feed.json`'s box equals the edge's label, and with no pointer at all;
  - code 3 for an expired pointer;
  - `off` (no pointer) gives 0 even with a feed failing elsewhere;
  - the priority when 2 and 4 both hold.
- [ ] **Step 2:** Implement it.
- [ ] **Step 3:** Run the tests, then commit.

### Task 5: the `geoip` rule

**Files:** Modify:
- `internal/metrics/alert.go`: `Observation.GeoIP *float64`; `Rules.GeoIPSustain` (default 30 minutes); `evalFrac("geoip", obs.GeoIP, 0.5, r.GeoIPSustain)`;
- `internal/host/metrics_loop.go`: fill `obs.GeoIP` from a published source, the same pattern as `geoIPSource`;
- `internal/host/agent.go`: publish `e.GeoIPAlertCode`;
- `internal/host/alertsink.go`: the `geoip` cases.

Alert sentences, sent to copy review with plan 1's batch:

| Code | Firing | Clearing |
|---|---|---|
| 1 | `the geoip database on %s is more than 45 days old` | `geoip on %s is back to normal` |
| 2 | `the GeoLite2 database on %s expires within 9 days` | the same |
| 3 | `the geoip database on %s has expired and was removed` | the same |
| 4 | `%s has not loaded the published geoip database for over an hour` | the same |
| 5 | `the geoip feed has failed 7 times in a row (seen from %s)` | the same |

`alertLevel` has no level for `geoip`, and `alertSustain` returns `GeoIPSustain`.

- [ ] **Step 1: Tests.**
  - `metrics.Evaluate` fires geoip after 30 minutes at code 1 and not before. It clears on the first 0. A nil reading changes nothing.
  - The alert sink renders each code's sentence.
  - The heartbeat-adjacent test: with no edge published, `obs.GeoIP` is nil.
- [ ] **Step 2:** Implement it.
- [ ] **Step 3:** Run the tests outside the sandbox for `internal/host`, then commit.

### Task 6: feed status in `carlos system geoip status`

**Files:** Modify `internal/console/apisystemgeoip.go` (add `feed` from `feed.json`: name, last check, last success, edition, failures, last failure category), `system_geoip.go` (print one line: `feed: <name>, last success <date>` or `feed: <name>, failing (<n> in a row, last: <category>)`), plus tests.

- [ ] **Steps:** test, implement, then commit.

### Final gate

Run the full gate per memory, check the cross-compile, push, and wait for CI. Then an Astra code review of plan 2's diff. Add the new strings to the copy review.