# Errors and Analytics: two tabs every app gets for free **Date:** 2026-09-07 **Status:** approved (Paul, 2026-09-07: "go ahead with that and plan, build review cycle overnight"). Rollout step 1 (§7) is merged (forge main 1a99060; plan `docs/superpowers/plans/2026-09-07-errors-step1-log-and-edge.md`). Step 2 is implemented on branch `analytics-step2` (plan `docs/superpowers/plans/2026-09-07-analytics-step2-edge-counting.md`). Every choice below is marked **(decision)** and can be overturned cheaply; the four in §8 are Paul's **(ruling)**s of 2026-09-07 and are marked where they land. Drafted by Claude Fable, then reviewed by a second Fable context and by Codex; both returned needs-rework on the first draft and §9 records what they changed. Line numbers cite `origin/main` at `09a22d6`. ## 1. The idea, and why it fits this platform A member who ships an app to CARLOS today gets a URL, a certificate, logs, schedules, and a bill. What they do not get is the two questions every operator asks within a day of launch: *is it broken* and *is anyone using it*. The answers live in AppSignal, Datadog, Sentry, PostHog, Plausible: a second account, a second bill, an SDK to wire in, and for the analytics half a cookie banner to justify. The platform is unusually well placed to answer both questions itself, because it already stands in the two places the data passes through: - **The edge sees every request** to every app, including the ones its cache answers without waking the instance (`writeCached`, `internal/edge/edge.go:1311`). Method, path, status, referer and user agent are in hand at the point where `countingWriter` is installed to bill the bytes (`edge.go:1241`). Server-side page-view counting there costs the app nothing and needs no JavaScript and no cookie. It is the Plausible model, done where Plausible cannot stand. - **The agent already captures every line the app writes** and ships it as hour chunks to the bucket (`internal/applog`, `internal/host/logship.go`). A line the app tagged `level=error` is already an error event; it is just filed under Logs, undifferentiated from a million info lines. And when the app dies, the edge answers a 502 or reports a wake failure: the crash is already observed. So the first version of both tabs is mostly *reading what is already flowing past*. The hooks add the cases the platform cannot see on its own: a public endpoint on the app's own hostname for browser errors and named events, and a loopback verb for server-side errors and events. The member who wants AppSignal can still rent it; the event shape is deliberately close to what those services accept, so a "forward to" setting later is a translator, not a redesign (§5). The proposal is two tabs on the app dashboard, **Errors** and **Analytics**, between Logs and Activity in `appnav.html`, each with a console API and a `carlos` verb behind it (AGENTS.md: every command must complete the job for a user with no infra access). ## 2. What is already right, and must stay right **Logs are the pattern for events.** `applog.Record` (`internal/applog/record.go:72`) is buffered on the box under a token budget (`sink.go:95`), flushed as gzip'd JSONL hour chunks under `apps///logs///`, read back by `applog.ReadWindow` merging chunks across boxes, pruned by `logshipPrune` (`logship.go:161`) on age and byte cap, and served through `GET /api/cli/apps/{account}/{app}/logs` (`authcli.go:595`) with a cursor. Errors that the platform must store are events too; they reuse this machinery rather than growing a second one (§3.2). **Usage is the pattern for counters.** `usage.EdgeDay` (`internal/usage/edge.go`) is a CAS-merged day file, hour-bucketed, one per box, flushed every five minutes by `usageLoop` (`internal/host/usage_loop.go:27`) from `UsageMeter.Drain()`. Page views and visitors are counters; they reuse this shape, with two durability fixes the reviewers asked for (§3.3). **The control socket is the pattern for a loopback API.** `ListenControl` (`internal/edge/purge.go:108`) mounts verbs on a 0600 unix socket the app finds in `CARLOS_CONTROL_SOCKET`; `schedule.Mount` (`internal/schedule/control.go:41`) shows the bearer check against `CARLOS_ADMIN_TOKEN`. Server-side ingestion mounts beside it (§3.5). The socket is not the activator: a call on it never reaches `beginRequest`. **Nothing the platform does for itself may keep an instance awake.** The scheduled-ticks spec (2026-08-30) exists because one line of accounting cost a member fifteen minutes of billed residency per millisecond of work. Every ingest path here is handled in the edge or the agent and none enters `activator.Service.ServeHTTP`; a beacon from a browser to a hibernating app is answered by the edge and the app stays asleep (§4). **The edge's wrapper order is load-bearing** (AGENTS.md): recorder above the compressor, usage counter below it, wrap site after the upgrade branch. This design changes what the usage counter *observes*, never where it sits (§3.3). **Fleet boxes write through the store door**, whose write grant is segment-exact `apps///logs/**` (`aclWriteAppLogs`, `internal/edge/storedoor.go:503`), and usage is rebased under `fleets//` (`usage_loop.go:47`). Two new families need the door to know about them, or a fleet box's writes are refused (§3.2). ## 3. The change ### 3.1 One event shape, four sources Both tabs are fed by events in one JSON shape. **(decision)** Fields: ``` ts RFC3339 nanoseconds, stamped by the platform, never taken from input source "log" | "edge" | "server" | "browser" kind errors: exception class or "http"; analytics: "pageview" | "event" name errors: the message; analytics: the event name host route host that received it (how owners name instances) box the box that recorded it release X-Carlos-Version of the serving build, when known path request path, query and fragment removed, token segments collapsed (§3.3) status HTTP status, when the event came from a request referer host-only, never the full URL ua browser family only, never the raw string stack errors only, capped at 8 KB fields map[string]string of caller-supplied scalars, 32 keys, 256 B each fp errors only: fingerprint (§3.2) fpv errors only: fingerprint algorithm version, 1 ``` Sources and where they enter: | source | errors | analytics | |---------|----------------------------------------------------------|---------------------------------------------| | log | applog record at or above `errors.log_level`, read-derived | none | | edge | upstream 5xx, gateway 502/504, wake failure | every HTML 200 GET becomes a pageview | | server | `POST /errors` on the control socket | `POST /events` on the control socket | | browser | `POST /__carlos/e` on the app's own host | `POST /__carlos/a` on the app's own host | `log` and `edge` need no integration at all. They are why the Errors tab is not empty on day one for any app in the fleet, and why Analytics starts counting the moment an account turns it on (§3.3). **Sanitisation before buffering (decision, Codex).** Errors carry messages, stacks, paths and caller fields, and those carry secrets in practice. Before any record is buffered, on every source: query strings and fragments are removed from `path`; in `name`, `stack` and `fields`, email addresses become ``, `Bearer`/`Basic` credentials and `token=`/`key=`/`secret=`/`password=` values become ``, and hex or base64 runs of 32+ characters become ``. The browser snippet applies the same rules before sending, so nothing leaves the page that the platform would strip. This is a floor, not a guarantee; the Errors tab says so in one line. ### 3.2 Errors: sources, fingerprints, coalescing, storage **The `log` source is derived on read, not stored twice (decision, Fable review 7).** The first draft mirrored `level=error` lines into a second family. `ReadWindow` with `Filter.Level="error"` (`filter.go:22`) over `logs/` yields the same rows with the same retention, and a mirror would have had to choose a side of the token budget at `sink.go:95`. So the Errors tab reads two families and merges: `logs/` filtered to the levels the `errors.log_level` setting names (§3.3; `error` means `error`+`fatal`, `warn` adds `warn`; **a setting, per Paul's ruling of 2026-09-07**), and `errors/` for the other three sources. Whatever the setting, every row carries its level and the tab, API and CLI filter on it, so a member on `warn` can look at errors only and a member on `error` can widen one read to `warn` without changing the setting. A JSON line whose `level` is unrecognised is not an error; `ParseAppLine` never guesses, and neither does this. A plain-text `panic:` on stderr carries no level and is not a `log` error either: the crash reaches the tab through the `edge` source when the next request meets a dead instance, which is the honest signal. **Edge errors (decision).** Where the observing writer (§3.3) sees a final status of 500–599 from the upstream, or the edge itself answers 502/504 because the instance did not come up, it emits `source: edge, kind: http, name: " "`. Wake failures already reach the member through `emitEvent` in `internal/activator/logsink.go`; the same call site sends one record to the errors sink. To keep that exactly once, the activator branch marks the request context when it answered the failure itself, and the observing writer skips a 502/504 that carries the mark; the writer's own emission covers only statuses the upstream returned or the proxy produced without the activator. Client 4xx are not errors. **Fingerprint (decision, amended per both reviews).** `fp` groups occurrences into one row. A caller may supply one, capped at 64 bytes. Otherwise the platform computes `sha256(len-delimited(source, kind, norm(name), norm(top frame)))[:16]` with `fpv: 1`. `norm` splits on whitespace and punctuation; a token that is a digit run of 4+ characters, a hex run of 8+, a UUID, or a quoted string becomes a placeholder; tokens of 1–3 digits and mixed tokens like `v2` or `E404` are kept, so `HTTP 503`, `v2` and `retry 3` still separate groups while `user 4123` and `user 9911` do not. Small ids under 1,000 will split groups; that is the stated trade-off, and `fp` overrides it. A frame is normalised to file and function with line, column and bundle hash removed, so a redeploy does not split every browser group. The rule is versioned so it can change without merging old groups into new ones. It is wrong sometimes; a caller who knows better sets `fp`. **Coalescing at the box (decision, amended per Codex 8).** An error storm is when the tab must work and when a naive stream fails. The box sink keeps, per `(app, fp, host, release)` per flush interval, one record carrying `count`, `first`, `last`, and the first sample's `stack`, `path`, `status` and `fields`. Ten thousand identical panics in a minute become one record with `count: 10000`. Coalesced records are capped per app per hour at 1,000, counting `(fp, host, release)` combinations, not distinct fingerprints, so the memory bound is real (per app, not per box: one tenant's storm must not suppress a neighbour's, Codex 9); beyond that, the overflow folds into a single `_overflow` row whose `count` is occurrences, labelled as such on the tab. A record belongs to a read window when `[first, last]` overlaps it and is counted whole; the flush interval is five minutes, so a window edge is wrong by at most one interval, and the tab's window labels say "to the nearest flush". (Correction, 2026-09-24: the ship loop as built flushes every 10 s, `errshipTick` in `internal/host/errship.go`, so records coalesce over 10 s, not five minutes. Found reviewing the request-reference spec.) **Storage (decision, amended per Fable 6).** `errors/` is a sibling family of `logs/`: `apps///errors///-.jsonl.gz`. The chunk key and hour layout are shared, so `ChunkKey`, `Prefix`, `BoxPrefix`, `HourPrefix`, `ChunkKeySize`, `hourNameOfDir`, `hourNameOf`, `logshipPrune` and `logshipLoop` take a family argument. The record is not shared: `applog.Record` has none of `fp`, `count`, `first`, `last`, `stack`, `source`, `kind`, `status`, `path`, `release`, and `EncodeChunk`, `DecodeChunk`, `ReadWindow`, `Filter.Match` and `Cursor` all take `Record`. The codec becomes generic over the record type, or `Record` is widened with `omitempty` fields; the implementer picks whichever leaves `logs/` chunks byte-identical, and the existing applog tests are the proof. The consumer list for the key shape, per AGENTS.md, is: `ChunkKeySize` (`chunk.go:65`), `hourNameOfDir` (`read.go:91`), `hourNameOf` (`logship.go:216`), `aclWriteAppLogs` (`storedoor.go:503`), `logshipPrune` (`logship.go:161`). Wire shape is additive: nothing under `logs/` changes. **Store door (decision, Fable 3).** `aclWriteAppLogs` becomes a family allow-list: `logs`, `errors`, `analytics`, each with the same "at least one component below" rule, with door tests for every family and for `errors` as an app name. Without this a fleet box's writes are refused and the tabs are empty for exactly the members on fleets. ### 3.3 Analytics: counting at the edge **Where the edge sees it (decision, Fable 2).** The first draft assumed a tap on the recorder. The recorder is built only on the hibernating and cacheable branches (`edge.go:1322, 1333`); a cache hit goes through `writeCached` (`:1311`) and the plain path (`:1339`), every POST included, has no recorder at all. The one wrapper installed at a single site for every response shape is `countingWriter` (`usagemeter.go:64`), which today has no `WriteHeader` and is installed only when a meter is armed (`edge.go:1241`). It becomes an *observing* writer: installed unconditionally with a nil-safe meter, latching final status and `Content-Type` at the first `WriteHeader` of 200 or above (1xx are forwarded and ignored), an implicit 200 at first `Write`, and a 200 with no `Content-Type` when the handler returns having written nothing, which is never a pageview. `Flush` and `Hijack` forward as they do today. The meter wrap today skips `Kind == "status"` routes (`edge.go:1241`); the observing writer is installed there too, but the tap ignores status routes, so the status host never enters any app's analytics. It stays below the compressor; `Content-Type` is visible there because `gzipWriter.syncHeader` copies its private map down before the underlying `WriteHeader` (`compress.go:236`). HEAD skips gzip (`compress.go:197`), so the tap excludes by method itself. The upgrade branch (`:1257`) is untouched; a 101 is never a pageview. The order rule in AGENTS.md is preserved: nothing moves, one wrapper learns to look. **What is counted (decision, amended per Codex 3).** A request is a *pageview* when the method is GET, the final status is 200, and the response media type parses as `text/html`. Redirects are not views (the destination will be); 304s are not views (no media type, and a revalidated page is an undercount the tab states); 204/205 are not; requests with `Sec-Purpose: prefetch` or `Purpose: prefetch` are not; `Sec-Fetch-Dest`, when present, must be `document` (an `empty` dest is `fetch()`, which is how htmx and SPAs load HTML fragments, and a fragment is not a page; an absent header is an old UA or a bot, and the bot list handles the latter). Requests under `/__carlos/` are never counted. A request whose user agent matches the vendored bot list is counted into `bots` and nowhere else; the list is incomplete by nature and the tab shows `bots` so the member can see what was excluded. **Client IP (decision, Fable 5).** There is no client-IP value in the edge today. `clientIP(r)` is defined as: the last element of `X-Forwarded-For` (all header lines joined, as `setForwarded` does at `homemap.go:915`) when `hopVerified(ctx)` (`homemap.go:767` accepts a verified chain verbatim; `:800` strips an unverified one), otherwise the host of `RemoteAddr`. A header the client can set never reaches it. **Visitor (decision, amended per Codex 1 and Fable 10).** A visitor is `sha256(day-salt ‖ route host ‖ clientIP ‖ user agent)[:8]`. The salt is per box per UTC day, generated at first use, written 0600 to the box's state directory beside the registry DB (`/data/`, which the agent already owns) so a restart mid-day does not re-salt and double-count, and deleted on rotation; it is never written to the bucket. That location is Fable's recommendation, confirmed against who owns the directory; Codex's ask was for a restart-stable, app-scoped salt shared by every box serving the app, which the persisted file gives for the first half and which the home-box rule makes unnecessary for the second. No cookie is set, no IP is stored, and the platform holds nothing that joins a hash across days. Because a route is served by its home box (a box without the row backhauls, `edge.go:1216`), a visitor is normally seen by one box; during an instance move they are counted twice for a day. Multi-day ranges on the tab are labelled "daily visitors, summed", because that is what they are; Plausible's are too. **Cardinality (decision, amended per Codex 10).** Unique visitors per day per box are a HyperLogLog sketch with 2¹¹ registers (2 KB, standard error about 2.3%), one per day, encoded as base64 with a one-byte version prefix, merged across boxes by register max. Pageviews per hour are integers. Top pages, top referers and named events are Misra–Gries summaries with 500 counters each: bounded, mergeable, and correct for any app with fewer than 500 distinct keys a day, which is most of them. The reader re-folds after merging so the tab never shows more than 500 rows plus `_other`. Below 500 keys the counts are exact; above, a counter is a lower bound within the summary's error and the table is labelled "approximate: more than 500 distinct keys today". Keys are capped at 256 bytes. **Page paths (decision, Fable 10).** `pages` keys are the sanitised `path` of §3.1: query and fragment removed, and any segment that is a UUID, a hex run of 16+ characters, a base64-looking run of 20+, or an email replaced by `:token`. `/reset/8f3a…` never reaches the bucket; `/blog/2024-review` does. **Storage (decision, amended per Codex 13).** `apps///analytics//days/.json`, written by a sibling of `UpsertEdgeDay` with the same read-etag-merge-conditional-put loop, and two differences from usage. First, buckets are keyed by *event time*, not flush time, so a flush delayed across an hour boundary lands in the right hour. Second, the drain returns deltas, and a failed flush re-queues its deltas into the next drain rather than losing them (`usageLoop` documents that it loses the window; modelling grade is fine for billing estimates, not for a number a member reads as truth). One window survives a failure; a second consecutive failure drops the older one and increments `dropped.flush`. A conditional put whose outcome is unknown (timeout after send) must not double-count on retry: every delta batch carries a box-monotonic `seq`, the day file records `seq` per box, and a merge whose `seq` is not greater than the recorded one is a no-op. A process restart loses the in-memory window, stated on the tab as "counts since the last flush may be missing" only when a restart happened that day. Shape: ```json { "v": 1, "date": "2026-09-07", "hours": {"09": {"pv": 412, "bots": 30, "bytes": 1843201}}, "visitors": "AQ…", "pages": {"/": 300, "/pricing": 80, "_other": 32}, "referers": {"news.ycombinator.com": 200, "direct": 180}, "events": {"signup": 12, "checkout": 3}, "seq": {"box-a": 1187}, "dropped": {"e": {"rate": 0, "size": 0, "origin": 0, "malformed": 0}, "a": {"rate": 0, "size": 0, "origin": 0, "malformed": 0}, "flush": 0}, "countries": {} } ``` `countries` is present and empty in v1 so a GeoIP pass is additive. `dropped` is what the member sees when a cap or a failure bit: the tab shows it, so a number that is not the whole truth says so. **Named events.** `carlos.track("signup")` from the browser, or `POST /events {"name":"signup"}` from the server, increments `events[name]`. Names are lowercased, 64 bytes of `[a-z0-9_.-]`, and bounded by the same 500-counter summary. Event properties are accepted in `fields` and *not stored* in v1 (§5). **Accumulation.** The edge holds deltas in memory; the agent's five-minute `usageLoop` cadence flushes them in the same tick as usage, as a second CAS write (two objects, two writes; the first draft's "one round-trip" was wrong). **The settings object (decision, Fable 9; default is a ruling).** `features.json` is console-only and no box reads it. The switches are one bucket object at two levels with the same shape: ``` accounts//telemetry.json account default apps///telemetry.json per-app override {"analytics": {"count": "edge" | "off", "door": "on" | "off"}, "errors": {"log_level": "error" | "warn"}} ``` Each key resolves app → account → built-in default, so an account can turn analytics on once for every app it has and a single app can still differ. The console writes either object with a serial bump and boxes converge on the next pass like `instances/*.json`; the account object lives beside `account.json`, which stays identity-only. Built-in defaults: **`count: off` (ruling, Paul, 2026-09-07)**, `door: on`, `log_level: error`. Analytics is therefore opt-in: the visitor design is built to be defensible without consent, but whether it is consent-free is the deployment's assessment and the member's, not the platform's, so the platform does not switch it on for anyone. `count: off` stops counting at the observing writer and makes both doors drop analytics events (`/__carlos/a` and the socket's `/events`) with `202`; existing day files stay until retention. `door: off` closes the browser door entirely, errors included, for an owner who wants no public ingest surface at all. `"count": "browser"` is reserved for a later mode (§5) so the shape is additive. Because the default is off, the tab is not empty when it is off: it shows the switch, what turning it on records, and the one-line privacy reading, so a member decides from the tab rather than finding a setting by accident. The `getting-started` skill that `carlos skills` fetches from carlosframework.com (published from the website repo, not here) gains one question in the same step: whether to turn analytics on for the account, with the same one-line reading. That is where an agent setting up a member's first app will ask it. ### 3.4 The browser door **Where in the edge (decision, Fable 1, Codex 6).** The first draft said "before any routing". That is the wrong spot: the app is known only from `up.route`, which exists after `lookupUpstreamPath` (`edge.go:1197`), and a box that has no row for the host must backhaul (`:1216`) so the home box answers. The intercept sits between the lookup (`:1197`) and the meter wrap (`:1241`): after routing, before upgrade (`:1257`), gzip (`:1275`), cache (`:1311`) and the activator (`:1322`). At that spot beacons are unbilled like `status` routes and `beginRequest` is never reached, which is what §4 promises. **Attribution on a shared hostname.** `lookupUpstreamPath` can route different path prefixes on one hostname to different apps (`:1008`). A beacon carries no app id. The snippet puts `location.pathname` in every beacon's `path`, errors and events alike, and the beacon is attributed to the upstream the lookup returns for that *raw* path on that hostname (the match runs before §3.1 sanitisation, which is applied to what is stored), and to the hostname's root app when `path` is absent or matches nothing. Because `Origin` must equal that hostname, the sender is already a page served from it: same origin, same trust boundary the browser already grants those siblings. An unknown hostname takes the existing `!ok` branch and is never answered by this code. The lookup must never accept `__carlos` as a prefix key; a test pins that. **Endpoint (decision).** ``` POST /__carlos/e one error {kind, name, stack, path, fields, fp} POST /__carlos/a one event {name, fields} | {"name":"pageview","path":…} GET /__carlos/carlos.js loader, Cache-Control: max-age=300, must-revalidate GET /__carlos/carlos..js the snippet, immutable ``` Both POSTs answer `202` with an empty body in every case: accepted, rate-limited, rejected, malformed. The endpoint reveals nothing, and the browser never sees a failure worth retrying. Every drop increments the app's `dropped.rate` (or `dropped.origin`, `dropped.size`), which the tab shows, so silence outward is not silence to the owner (Codex 11). `/__carlos/` is reserved on all routes and never proxied upstream from this change on. No app in this repo serves it; §8 asks about the deployment roots. **Who may post (decision, amended per Codex 11, Fable 12).** The request must carry an `Origin` whose scheme is `https` and whose host and port resolve, through the edge's own map (`lookupIn(e.current(), originHost)`, the same lookup the request itself took), to an entry whose `route.Host` equals `up.route.Host`; aliases share the primary's route verbatim, so this covers them with no registry call on the hot path. A bad `Origin` is rejected even if `Referer` passes; `Referer` is consulted only when `Origin` is absent. `Sec-Fetch-Site: cross-site` is rejected outright. This is identification, not authentication: a direct client can forge any of it, as it can against every public beacon on the internet, and a per-app public key would add revocation but not truth. What bounds the damage is that a forged post can only put rows on the owner's own tab, under the owner's own caps, and the owner can turn the door off. Body is capped at 16 KB; `stack` at 8 KB. **Rate (decision, amended per Codex 9).** Token buckets per *app* of 600/min for errors and 600/min for events, separately, and per visitor hash of 60/min, all in edge memory, all dropping silently and counting `dropped.rate`. Visitor-bucket state is bounded to 10,000 entries per app with LRU eviction. Errors are then coalesced by fingerprint at the box as in §3.2, so a page throwing in a loop on ten thousand screens is one row with a large count and, if it exceeds the bucket, an honest `dropped`. **The snippet (decision).** `carlos.js` is vendored into the edge binary the way Lucide icons are vendored into the console: in-code, never a CDN. It installs `window.onerror` and `unhandledrejection` handlers that post to `/__carlos/e` after §3.1 sanitisation, and exposes `carlos.track(name, fields)` which posts to `/__carlos/a` via `navigator.sendBeacon`. **It does not send pageviews in edge mode.** The edge already counted the page when it served the HTML; a JS pageview would double every number. What edge mode misses, stated on the tab: SPA route changes, HTML served by a CDN in front of CARLOS, and back/forward cache restores. The reserved `"count": "browser"` mode (§5) is the answer for those apps, not a same-second dedup heuristic. Until then an SPA may call `carlos.track("route", {path})` and see it as a named event. Members include the snippet with one line: ```html ``` ### 3.5 The server door **Verbs (decision).** `errors.Mount` and `analytics.Mount` follow `schedule.Mount` exactly: same bearer check, same 0600 socket, same `host` binding. ``` POST /errors {kind, name, stack, path, status, fields, fp} POST /events {name, fields} ``` The app reads `CARLOS_CONTROL_SOCKET` and `CARLOS_ADMIN_TOKEN` from its environment as it already does for schedules. A Rails app wires it in an `ActionDispatch` error subscriber; a Go app in an `http.Handler` middleware. The confused-deputy discussion in `purge.go` applies unchanged: a same-uid sibling that can read the socket can post a fake error, and the worst it can do is put a row on the owner's own tab. A call on the control socket does not stamp `lastActive`: `ListenControl` serves its own mux (wired at `backend_exec.go:624`) and never enters `Service.serve`. The same 16 KB body cap, 8 KB stack cap and 600/min per-app buckets as the browser door apply, counted into the same `dropped` keys; the socket answers `202` on a drop and `400` only on a body it cannot parse, since the caller here is the owner's own code and can act on it. Unit-backed and DynamicUser tenants have no control socket today and so have no server door; they have `log` and `edge` sources and the browser door like everyone else. This is the same gap `purge.go` records for schedules, and it closes when that one does. ### 3.6 The tabs, the API, the CLI **Errors tab** (`GET /a/{sqid}/apps/{app}/errors`, `handleAppErrors` through `appPage(w, r, "errors")`, template `app_errors.html`, `ActiveTab: "errors"`). Default window 24 h, selectable 1 h / 24 h / 7 d. One row per fingerprint: source badge, kind, name, count in window, first seen, last seen, hosts affected, releases seen. Rows sort by last seen. A row expands in place to the stored sample: stack, path, status, fields, and a link to Logs at that timestamp. Filters: source, level, host, release. Grouping happens on read over the window. `ReadWindow` fetches the whole window and discards beyond `logsCollectCap` (50,000), which bounds memory, not bucket reads and not totals (Codex 14): when the cap is hit the tab says "showing the most recent 50,000 of a larger window" and the counts are labelled as counts within what was read. A day rollup is not built until a real app shows the read is too slow (§5). **Analytics tab** (`/apps/{app}/analytics`). Range today / 7 d / 30 d. Headline: pageviews, visitors (daily, summed for ranges), bytes served, bots excluded, dropped. One chart, per hour for today and per day otherwise, drawn the way `templates/metrics.html:29` draws sparklines: points computed in Go, an inline `` in the template, no library. Three tables: top pages, top referers, named events. **API (decision, Fable 8).** Two `cliRoute` paths beside the logs read: ``` GET /api/cli/apps/{account}/{app}/errors svcrole.Allow(Instance, Operate) GET /api/cli/apps/{account}/{app}/analytics svcrole.Allow(Instance, Operate) ``` Bearer-authenticated through `apiAccountApp` like `/logs`; an app token may read its own app's errors and analytics. The session → CSRF → owner (byte-identical 404) → fresh-auth ordering applies to the two *pages*, which go through `appPage`; the API paths have no session and no CSRF by construction. The tabs render from the same functions the API answers with. **CLI.** `carlos errors -app X [-since 24h] [-source browser] [-level warn] [-host h] [-release v] [-fp ]` lists groups, `-fp` expands one; `carlos analytics -app X [-range 7d]` prints the headline, chart as a text sparkline, and the three tables. Both get headings in `docs/site/cli.md` in the same branch, or `internal/docsite` fails the gate. **Settings.** The three keys of §3.3 on the app Settings tab and on the account settings page, and via `carlos apps` and `carlos accounts`, writing the two objects. Error *capture* has no switch: an app that is failing is an operational fact the owner needs; `log_level` only widens what the `log` source counts. A member may delete an error group from the tab or the API (`DELETE …/errors/{fp}`, owner on the page, `Operate` token on the API). This writes a tombstone `apps///errors/tombstones/.json` carrying the deletion time; the reader hides records of that `fp` with `last` before that time, from both `errors/` and the log-derived rows, until retention removes the chunks. A later occurrence reappears: this is delete, not mute (§5). Logs themselves are untouched; deleting log lines is a Logs feature and not part of this design. **Retention (decision).** Error chunks follow log retention, both age and byte cap, in `logshipPrune`. Analytics day files are kept 400 days, enough for year-over-year, and are small enough (tens of KB per app-day) that no byte cap is needed; the same prune loop deletes past the age. **Coverage line.** Each tab carries one line saying which sources are live for this app: "log, edge" for an app with nothing wired, "+browser" once a beacon has arrived, "+server" once the socket verb has. A member should never wonder whether silence means no errors or no wiring. ## 4. The consequence to be explicit about: nothing here wakes an instance Every path in §3 is checked against the scheduled-ticks rule: - `/__carlos/*` is answered between route lookup and the meter wrap, before the cache and the activator. A beacon to a hibernating app returns `202` from the edge; `beginRequest` is never called and the idle clock is not stamped. The test reads `a.entries[host].lastActive` directly, as `activator_test.go:1378` does. - Analytics counting is a tap on responses the edge is already sending. - The `log` source is a read over chunks the agent already ships. - The server door is a control-socket verb, which does not stamp. - Flushes ride the existing `usageLoop` and `logshipLoop` tickers in the agent. The one new cost is bucket writes: one analytics day-file CAS per box per five minutes per app with traffic, and error chunks at the cadence of log chunks. Both are bounded by the caps in §3.2–3.3 regardless of traffic. Memory on the edge is bounded per app: 500-counter summaries, one 2 KB sketch, a 10,000-entry visitor bucket table, and the coalescing map capped at 1,000 fingerprints. ## 5. What is not built - **Resolve and mute.** An owner will want to mark a group resolved or mute a known one. It is the first triage feature to add and comes before alerts; the tombstone in §3.6 is its seed. - **Alerts.** "Email me when a new fingerprint appears" belongs with the existing `alertwatch` machinery, not here. - **Browser-counted mode** (`"count": "browser"`): the snippet sends pageviews with SPA route support and prerender activation, and the edge stops counting HTML. For apps behind a CDN or built as SPAs. - **Event properties.** `fields` on analytics events are accepted and dropped; property breakdowns change the day-file shape additively. - **Error day rollups.** Grouping reads the window every time. Fine to 50k records; a heavy app will tell us when it is not. - **Forwarding.** A `forward: sentry|posthog|…` setting that relays events to a rented service. The shape in §3.1 makes this a translator. - **Source maps, breadcrumbs, session replay, funnels, GeoIP, per-country tables.** Rent those. - **Fleet-wide views.** Errors across a member's apps, or across the fleet for ops, are a different page with different gates. ## 6. Testing - **Sanitisation:** table test over messages, stacks, fields and paths containing emails, bearer tokens, `password=`, 32-char hex and base64 runs; each is replaced and nothing else is touched. - **Fingerprint:** table test of `norm`: digits, hex, UUIDs and quoted strings are placeholders; `v2`, `E404`, `HTTP 503` survive; frames lose line, column and bundle hash; a caller-supplied `fp` is taken verbatim and capped; `fpv` is stamped. - **Coalescing:** 10,000 identical errors in one interval produce one record with `count: 10000` and `first < last`; the same `fp` on two hosts produces two records; 1,001 distinct fingerprints for one app produce 1,000 records and one `_overflow`; a second app on the same box is unaffected. - **Log-derived rows:** a `level=error` JSON line appears on the Errors read and in Logs; `level=info`, an unparseable line, and a plain `panic:` line do not appear as `log` errors. - **Edge errors:** an upstream 500 on a POST (the plain path), a 502 from a failed wake, and a 500 on a cacheable GET each produce exactly one edge error; a 404 produces none. - **Chunk family:** `logs/` chunks written and read after the refactor are byte-identical to before; every existing applog test passes unchanged; `errors/` round-trips the new record. - **Store door:** a fleet box may write `apps///errors/x` and `…/analytics/x` and may not write `apps//errors/x`, `apps///errors`, or `apps///other/x`. - **Observing writer:** status and `Content-Type` latch at `WriteHeader` and at implicit 200; visible below gzip; installed with a nil meter; byte counts unchanged for every existing usage test. - **Pageview definition:** `httptest` through `Edge.ServeHTTP` with a fake upstream: HTML 200 GET counts; JSON 200, HTML 404, HTML 302, 304, HEAD, `Sec-Purpose: prefetch`, `/__carlos/carlos.js` do not; a cache hit counts; `Sec-Fetch-Dest: empty` does not; a known bot UA counts only into `bots`; a `status` route counts nothing; `count: off` counts nothing. - **Client IP:** verified hop takes the last XFF element; unverified hop and direct take `RemoteAddr`; a client-set XFF on a direct request is ignored. - **Visitor hash:** same IP+UA on the same day is one visitor; a different day is different; the salt survives a restart and never appears in the day file; the salt file is removed on rotation. - **HLL:** across ten random datasets of 1k, 10k and 100k, the estimate is within 3 standard errors of truth; merge is commutative and idempotent; the encoding round-trips with its version byte. - **Misra–Gries:** with fewer than 500 keys the summary is exact; two summaries merge and re-fold to 500 plus `_other` with the total preserved. - **Day-file CAS:** event-time bucketing places a flush at 10:00:02 for events at 09:59 into `09`; a failed flush re-queues and the next succeeds with the sum; two failures drop the older window and increment `dropped.flush`; two boxes writing the same app-day converge to the sum of pageviews and the union of sketches. - **Browser door:** correct `Origin` counts; missing `Origin` with matching `Referer` counts; foreign `Origin` with matching `Referer` does not; `Sec-Fetch-Site: cross-site` does not; 17 KB body does not; the 601st post in a minute does not; each drop increments the right `dropped` key; every case answers `202`. A hibernating route receives a beacon and `lastActive` is unchanged. On a path-routed hostname the beacon lands on the app owning its `path` prefix. `__carlos` is refused as a prefix-map key. - **`/__carlos/` reservation:** a route whose upstream serves `/__carlos/x` never sees the request; the loader revalidates and the hashed asset is immutable. - **Server door:** a bad bearer is refused; a good one lands a record; `lastActive` is unchanged after the call; a 17 KB body answers `202` and increments `dropped.e.size`; `count: off` drops `/events` and still accepts `/errors`. - **Seq idempotency:** re-merging a batch with an already-recorded `seq` leaves the day file unchanged. - **Tombstone:** after `DELETE`, the group is absent from the read and the API; a new occurrence after the deletion time reappears; Logs still shows the underlying lines. - **Wake failure once:** a failed wake produces one edge error, not two. - **Console gates:** both pages give a non-member the byte-identical 404 the Logs tab gives; both API paths refuse a token without `Instance` or `Operate`. - **Settings resolution:** with no object at either level nothing is counted and `log_level` is `error`; an account object with `count: edge` turns counting on for every app; an app object with `count: off` under that account turns it off for that app alone; writing either and bumping the serial converges on the box within the next pass; the browser door drops `/__carlos/a` and still accepts `/__carlos/e` when off; `door: off` drops both; `log_level: warn` brings warn lines into the Errors read and `-level error` filters them back out. - **Docsite:** the gate passes with both new CLI headings present and fails with either removed. ## 7. Rollout Each step is a separate branch that ships value on its own and needs no member action. Each is canaried on the flagship for a day against edge latency, edge memory, bucket write count and the hibernation invariant before promotion, in the ladder the fleet already runs. 1. **Errors from `log` and `edge` sources, tab, API, CLI.** Every app in the fleet gets a populated Errors tab. Requires the chunk-family refactor, the store-door allow-list, the observing writer (for edge errors), and the coalescing sink. 2. **Analytics counting at the edge, tab, API, CLI, the settings objects at both levels, and the skill question.** An account that turns it on gets pageviews and visitors for every app with no JavaScript. Shares the observing writer with step 1, so it lands second. The `getting-started` skill change is a companion branch in the website repo, landed the same day. 3. **Browser door and `carlos.js`.** Members opt in with one script tag; JS errors and named events appear. 4. **Server door on the control socket.** Exec-backed apps post structured errors and events. Wire shapes are additive at every step: `logs/` is untouched, `errors/` and `analytics/` are new prefixes, `telemetry.json` absent means off, and a box on an older build writes none of them. A mixed-version fleet shows the coverage line per host so a missing source is visible rather than puzzling. Binaries roll as usual: promote → adopt → restart for the console; edge and agent pick up on adoption. ## 8. Rulings (Paul, 2026-09-07) Four questions were open in the draft; Paul answered all four. 1. **Analytics is off by default (ruling).** Settable at the account level so one switch covers every app, overridable per app, and asked about in the `getting-started` skill. §3.3 carries the objects and the resolution order; §7 step 2 carries the skill change. 2. **The reserved prefix is `/__carlos/` (ruling).** Paul asked whether a random per-app identifier would be better for unpredictability. The recommendation is no, and the spec stays static: the path is not a secret and cannot be one, because the snippet URL is in every page's source and the beacon endpoint is public by construction (§3.4). What bounds abuse is the Origin check, the caps, `dropped`, and the `door` switch, none of which a random path would strengthen. A random path would also cost the one-line snippet its one line and would make the reservation rule ("this prefix is never proxied upstream") per-app state instead of a constant. The double underscore buys the collision safety Paul wanted: no app in this repo serves it, and nothing conventional does. 3. **Warn level is a setting, and level is always filterable (ruling).** `errors.log_level` in §3.3 at either level, `error` by default; the tab, API and CLI filter on level regardless (§3.2, §3.6). 4. **The salt lives beside the registry DB, 0600 (Fable's recommendation, taken).** §3.3 records both reviewers' positions: Fable named the location and confirmed the agent owns it; Codex asked for restart-stability and cross-box sharing, and the persisted file gives the first while the home-box rule makes the second unnecessary. The `platform-infrastructure` collision check for `/__carlos/` is still worth one grep before step 3 ships; the double underscore makes a hit unlikely enough that it is a checklist line, not an open question. ## 9. Review round 1 Both reviewers read the first draft against `main` and returned needs-rework. What each changed: **Fable (second context).** Moved the beacon intercept from "before routing" to between `lookupUpstreamPath` and the meter wrap, with the backhaul argument (§3.4). Replaced the recorder tap with the observing `countingWriter`, citing the three response paths that bypass the recorder (§3.3). Found that fleet boxes' writes would be refused by `aclWriteAppLogs` (§3.2). Corrected the §1 citations, which pointed at ACME and attribution comments. Defined `clientIP` from the verified-hop chain (§3.3). Listed the real refactor scope and the key-shape consumer list, and corrected `retentionPass` to `logshipPrune` (§3.2, §3.6). Argued the log mirror away in favour of a read-side filter (§3.2). Named the `cliRoute` API pattern and the gate-ordering scope (§3.6). Found that no box reads `features.json`, so the switch became a bucket object (§3.3). Sanitised page paths (§3.3). Corrected the chart precedent, the Logs-tab grep, and the non-existent test hook (§3.6, §4). Noted the 500-key merge overflow (§3.3). **Codex.** Made the visitor salt restart-stable and the multi-day visitor figure honest (§3.3). Reframed the privacy claim as the deployment's assessment (§8). Tightened the pageview definition to 200 only, excluding redirects, 304s, HEAD and prefetch (§3.3). Reserved a browser-counted mode instead of a same-second dedup (§3.4, §5). Named the path-routed attribution problem independently of Fable (§3.4). Made fingerprints token-boundary, versioned, and frame-normalised (§3.2). Made the coalesced record carry `first`, `last`, host and release (§3.2). Made caps per app and exposed `dropped` everywhere something is dropped (§3.2, §3.3, §3.4). Replaced first-admitted maps with Misra–Gries and stated the HLL error correctly (§3.3). Tightened Origin matching and kept the uniform 202 only alongside owner-visible drop counts (§3.4). Added sanitisation before buffering and owner-triggered deletion (§3.1, §3.6). Made day-file buckets event-time and flushes re-queue on failure (§3.3). Added the collect-cap disclosure and the coverage line (§3.6). Split the snippet into a revalidating loader and a hashed immutable asset (§3.4). Put resolve/mute ahead of alerts in §5. Noted that a plain `panic:` line is not a parsed error, which §3.2 now answers through the `edge` source. Not taken: a per-app public key for the browser door (identification without truth, and the caps already bound the damage; §3.4 says why); storing the raw pageview path (the token-collapsing rule keeps real page names and drops secrets); a mirror of `log` errors into `errors/` (read derivation is cheaper and loses nothing). ## 10. Review round 2 Both reviewers re-read the rewrite. Fable: approve-with-amendments, all three blockers confirmed resolved against the code; amendments taken: the Origin check is a `lookupIn` on the edge's map, not an alias list (`upstream.alias` is a bool); `Sec-Fetch-Dest: empty` is a fetch, not a page; status routes are excluded from the tap; joined `X-Forwarded-For` lines; salt file 0600. Codex: needs-rework on under-specification; taken: 1–3 digit tokens survive normalisation so `HTTP 503` does; the snippet stamps `path` on every beacon so path-routed events attribute; the coalescing cap counts records and the window rule is stated; a `seq` per box makes CAS retries idempotent; a `door` switch closes the browser door independently of counting and `count: off` also drops the socket's `/events`; the server door gets the same size and rate limits; Misra–Gries tables are labelled approximate above 500 keys; the `dropped` schema lists every key it uses, split by errors and events; wake failures are suppressed at the writer when the activator already emitted; tombstones have a scope, an authorisation, and a stated non-effect on Logs. Nothing from round 2 was declined. ## 11. Amendment 2026-09-24: Availability **Why.** A day of the flagship's Errors tab (titogo, 2026-09-23) held 29 groups: 25 edge 502s, 3 edge 503s and one 500. Only the 500 was a failure of what a request asked for. The rest said the instance was down, restarting or unreachable when some page was asked for, and §3.2's `" "` fingerprint split each outage into one row per page that happened to hit it. 17 of the 29 rows were one scanner probing `/.env`, `/phpinfo.php` and so on against a dead canary. The record also could not say *why* the edge had no answer: the proxies logged the error and threw it away. **Availability is the Errors tab's sibling (decision, Bill 2026-09-24).** A sub-tab between Errors and Analytics (`GET /a/{sqid}/apps/{app}/availability`), an API path beside `/errors` with the same gate (`GET /api/cli/apps/{account}/{app}/availability`), and `carlos availability`. It holds: - every edge-observed 502, 503 and 504, whoever wrote it; and - the platform stream's events at `warn` and above (failed wakes, exits, a sidecar that will not restart), whatever `errors.log_level` says, since an exit on a signal is filed at `warn` and is exactly the event an outage row needs beside it. The synthetic rate-cap line (`applog.IsDropNotice`) is about the logging budget and is left out. Errors keeps the rest: the app's own 500, 501 and 505+, and the `log` source's app-stream lines. Nothing is in both. **Grouped by cause, never by path (decision).** An edge 502/503/504 is filed as `kind: unavailable` with `cause` and `name: " · "`, and so fingerprinted by status and cause alone; the path stays on the record as the occurrence's sample. The cause comes from the ErrorHandler that wrote the status, through a per-request `outcome` on the context (`edge/answered.go`, which already carried the activator's answered flag and now rides every observed request, not only hibernating ones): | cause | where | |---------------|--------------------------------------------------------------| | `refused` | ECONNREFUSED / ENOENT on the dial: nothing listening | | `reset` | EOF, unexpected EOF, ECONNRESET, EPIPE: hung up mid-request | | `timeout` | a deadline or a net timeout | | `abandoned` | the client left after waiting 5 s or more (below) | | `no_upstream` | `multiProxy`: every upstream failing its probe | | `fleet_dark` | `fleetProxy`: no box of the fleet connected | | `tunnel` | any other failure through a fleet box's tunnel | | `static` | a static site's files could not be resolved | | `proxy` | any other proxy failure | | `upstream` | no handler noted one: the app wrote the status itself (direct routes only; below) | `unixProxy`, `tcpProxy` and `tunnelProxy` had no ErrorHandler; they get one that is `httputil.ReverseProxy`'s default (same log line, same bare 502) plus the note. **A visitor who left quickly is not an outage; one kept waiting is (decision, amended in review).** When the client's own context is done by the time a proxy fails, `ReverseProxy` still writes a 502, to nobody, and the error is the cancellation whatever the instance was doing. Within 5 s of the request (`clientGoneGrace`) the edge notes `client_gone` and files nothing: a visitor navigating away mid-load is not the instance's fault, and a single-count 502 on a font file is the typical shape of it. Past 5 s it files `abandoned`. The review caught why that half matters: a dead unix socket is redialled for 13 s and a wedged instance is never timed out at all, so a monitor or visitor giving up first is how those outages end — skipping them would hide exactly the worst ones. **Behind a tunnel, the front box files only the hop (decision, Fable review; corrected in its follow-up).** When a box proxies a request over a reverse channel, the upstream is not the app but another box's edge — the agent's, the only tunnel client — which files what it sees serving the app, and deliberately files nothing for a wake 503 its own activator answered. A 5xx that merely passed back through the front box was therefore filed twice, and under this amendment would have been labelled "the app answered it itself" for an app that never saw the request. The front box now files only failures it noted itself (`tunnel`, `fleet_dark`, `abandoned`); everything else is the serving box's record. This also ends the double filing of tunnelled 500s, which predates the amendment. The rule keys on how THIS box served the request (`upstream.viaTunnel`, the build-time decision), never on the route row: the fleet box that serves the app carries the same `fleet/` placement on its own row, has no hub, and serves from the socket — it must file everything. (The first cut keyed on the row, and a fleet-placed app's 500s were filed nowhere; `TestEdgeServingFleetBoxFilesItsOwn` pins the far end.) Two edges remain: a far agent whose status store failed to open has no error sink and files nothing, which the front box no longer covers; and a client that leaves past the grace can be filed `abandoned` on both boxes, since the stream reset reaches the far box's request too. **The overflow row is per family (decision, from review).** The sink's hourly `_overflow` row copied its source, kind and status from the first record that overflowed, so one family's excess could land on the other tab. It is now one row per family (`IsAvailability`), and `AsAvailability` leaves an `_overflow` row alone. **History regroups on read (decision).** Records filed before this carry `kind: http`, no `cause`, and the per-path fingerprint. The read recognises them by status and regroups them as `" · cause not recorded"` (`apperr.AsAvailability`); the FP filter applies after the regroup so the row the tab shows expands. The bucket is not rewritten. **Not changed.** The wire shape is additive (`cause` and a read-only `stream` on `apperr.Record`, both `omitempty`); nothing under `logs/` or `errors/` moves. The sending half of a backhaul still files nothing when its hop fails: it has no route row and so no app to file under, the same reason it bills nothing. Correlating a row with the nearest deploy or restart on that host, and alerting on a new cause, are the natural next steps and are not built here.