# Scheduled ticks must not hold an instance awake

**Date:** 2026-08-30
**Status:** approved in principle (Paul, 2026-08-30, after the flagship
measurement below). Two decisions were taken in the brainstorm and are marked
**(ruling)**; everything else is marked **(decision)** and can be overturned
cheaply.

## 1. The problem, as found

Asked whether one member was hogging resources on the flagship, the answer was
no — and the way the answer was reached turned up a bug underneath it.

`james@keymail.dev` (account `bac`, app `jam`) declared one schedule on
2026-08-29: `sweep`, every 5 minutes, hitting `/jobs/sweep`. The handler
answers `204` in **1 millisecond**. From the moment that schedule landed
(2026-08-29 21:25Z) the instance has been resident **continuously** — present
in every 60-second metrics sample since, 24 hours and counting, where the
comparable tenant `seapointish` sleeps and wakes ten times a day.

It is not the app's fault and it is not the schedule's fault. It is one line
of accounting.

Scheduled delivery is otherwise exactly right. `activatorDeliverer` sends the
tick through the activator into the instance's unix socket, never through the
public edge, and its doc comment states the property it was built for
(`internal/host/schedule_wiring.go:216`):

> a hibernating instance WAKES for its tick, exactly as it would for a
> visitor, and stays awake for the tick's duration because the activator's
> in-flight count holds it open. No separate keep-awake to get wrong.

The tick reaches the instance through `Service.ServeHTTP`
(`internal/activator/activator.go:1141`) — the same door the edge uses — which
calls `beginRequest`, whose release stamps `e.lastActive = a.now()`
(`activator.go:594`). The sweeper hibernates on
`e.inflight == 0 && now.Sub(e.lastActive) >= a.cfg.Idle` (`activator.go:933`).

`CARLOS_HIBERNATE_IDLE` defaults to 15m. The spec is 5m. The clock is reset
three times before it can ever expire, so the sweeper's condition is
unreachable for the life of the schedule. **Every tick buys fifteen minutes of
residency for one millisecond of work.**

### Why it is worth fixing rather than tolerating

Three reasons, in ascending order of weight.

**It bills.** `instance_hours` is a metered, priced quantity, and in
rate-sheet mode it is the one term with no free-tier subtraction applied —
compare `billable(...)` on egress, storage, apps and memory against the raw
`meters[usage.MeterInstanceHours]` in `internal/economics/compute.go:120`, at
`InstanceHourUSD` = $0.01. A 5-minute schedule accrues ~730 instance-hours a
month: **$7.30 billed for roughly four seconds of work**.

**It defeats the capacity model.** The 2026-08-30 sizing measurement put peak
concurrent awake instances at 74 of the top 300 accounts on a 90s window. That
number assumes instances sleep between requests. An app with any schedule
tighter than the idle window never sleeps at all, so as schedules spread the
model degrades toward "everything is always awake" — which is the assumption
the r8g.2xlarge recommendation was chosen against.

**It is invisible.** Nothing tells the member. `carlos schedule ls` reports
what will fire and what fired last; nothing says "and this interval means your
instance never hibernates". James could not have known, and neither could
anyone reading the console.

## 2. What is already right, and must stay right

Worth stating, because two of these were nearly re-engineered before the code
was read properly.

**There are already two wake paths for scheduled work, and they are enough.**
The runner polls `DueTimers` every 15 seconds and delivers, waking the
instance through the activator; separately the sweeper reads `DueInstances`
and wakes any sleeping host whose recorded due time has arrived
(`activator.go:909-914`). Platform timers are covered by the first.

**`NextDue` is therefore not the mechanism to change.** `GET /api/next-due`
over the instance socket (`internal/activator/backend_exec.go:506`) asks the
*app* about work the platform cannot see — its own in-process schedule. It is
not the platform's timer index and should not become one. This spec adds no
column, no `SetNextDue` change and no second waker. **(decision)**

**The sidecar precedent already solves the accounting half.**
`BeginSidecarRequest` (`activator.go:526`) passes traffic to an awake instance
without registering activity — "no in-flight count, no lastActive stamp (the
sidecar's own long-lived WebSocket must not hold its instance awake; that
non-count is the whole reason the pair can sleep)". The shape needed here is
that shape, differing in one respect: a tick must *wake* a sleeping instance,
where a sidecar request is refused.

## 3. The change

Three parts. Together they are perhaps a hundred lines.

### 3.1 A non-stamping, waking entry point

Add `Service.ServeScheduled(w, r, rt, proxy) bool`, sibling to `ServeHTTP`,
and a `beginScheduledRequest` behind it that:

- wakes a sleeping instance, exactly as `beginRequest` does — the tick is the
  wake source, and that must not change;
- takes the in-flight count for the duration, so a tick is never cut off
  mid-delivery by a concurrent sweep;
- on release, decrements **without** stamping `lastActive`.

`activatorDeliverer.Deliver` calls it instead of `ServeHTTP`
(`schedule_wiring.go:317`).

**No wire marker. (decision)** The sidecar identifies itself with
`SidecarHeader` because it has to: it dials its instance's public URL and its
traffic arrives through the edge, so the only place to carry the signal is the
request. A scheduled tick never leaves the process — "the tick never leaves
the box: no DNS, no TLS, no edge auth, nothing an outsider could aim at the
same path" (`schedule_wiring.go:219`) — so the call site itself is the signal.
A dedicated method has no spoofing surface at all, where a header, even one
that is safe by construction, would have to be argued about.

`X-Carlos-Schedule-At` is untouched. It is the app-facing half of the tick
contract and continues to reach the handler.

### 3.2 The guard: don't sleep into an imminent tick

Without a guard, part 3.1 alone converts `jam` from one resident process into
**288 wake/sleep cycles a day**. Each cycle is a full SQLite
checkpoint-and-restore traversal, and the wake path is where
`2026-08-15-sqlite-corruption-defence-design.md` (issue #200) records an
unclosed integrity gap. Multiplying a scheduled app's traversals by ~28× to
save 26 MB is a bad trade.

So: in `sweepOnce`, before hibernating an otherwise-idle host, skip the sleep
when this box holds a timer for that host due sooner than `cfg.Idle` away.

```
hibernate when:  inflight == 0
             &&  now - lastActive >= cfg.Idle
             &&  (no timer for this host, or nextAt - now > cfg.Idle)
```

**The threshold is `cfg.Idle` itself, not a new constant. (ruling — Paul,
2026-08-30)** If the gap to the next tick is shorter than the window we would
have stayed awake for anyway, staying awake *is* the cheaper answer. It adds
no second tuning number to drift out of step with the first, and it composes
with the idle window rather than fighting it:

| Idle window | Schedule | Behaviour |
|---|---|---|
| 15m | every 5m | stays resident (today's behaviour, unchanged) |
| 90s | every 5m | sleeps ~3.5 of every 5 minutes |
| 90s | every 60s | stays resident |
| 15m | every 1h | sleeps ~45 minutes |

Residency is measured from each WAKE, not from each tick: a tick that wakes a
sleeping instance stamps `lastActive` via the wake path's own stamp, so the
instance gets one full idle window of residency starting at that wake before
it becomes eligible to sleep again. That is why the 15m/1h row sleeps ~45
minutes (60 minus the 15-minute window it earns on waking), not 59 (60 minus
the one-minute tick that would apply if the clock started at delivery). Rows
2 and 3 were already computed this way; anyone extending this table — or the
§1 billing arithmetic that reads off it — should keep using the wake, not the
tick, as the start of the window.

**Implementation constraint, from the code as it stands:** `sweepOnce` does
its `DueInstances` read deliberately before taking `a.mu` — "it's a registry
read, and nothing under a.mu should wait on SQLite" (`activator.go:911`). The
earliest-due lookup joins that pre-lock batch as one query returning
`host → earliest next_at`, never a per-entry query inside the loop.

A host with no timers is unaffected: no row, no guard, sleep as today.

**A tick-held host still converges.** The guard only skips the *sleep*; it
must not skip the build-convergence check that runs after it. A host held
awake by the guard never reaches a "next wake" to converge on — that is what
the guard guarantees — so it is now in the same class as an always-awake
sidecar host, and reaches the convergence arm on every sweep exactly as a
sidecar does. Without this, a `carlos ship` behind a tight enough schedule
would be served stale indefinitely. **(ruling — Paul, review round 1,
2026-08-31, overruling an earlier "converges at its next genuine sleep"
draft of this section that assumed a sleep the guard makes impossible.)**

### 3.3 Tell the member

The box is the only place that knows both numbers — it holds the timer rows
and it constructed the activator with `cfg.Idle`. So the box decides and
reports.

Add `HoldsAwake bool \`json:"holds_awake"\`` to `schedule.TimerStatus`
(`internal/schedule/status.go:24`) — true when this row's interval is shorter
than the box's idle window, so the row's instance stays resident between
ticks. Written
wherever the status object is already written (after every fire and every
converge that changed a host's rows). Additive, per the repo's wire-shape
rule: anything written before this change must still parse, and a status
object from an older box simply lacks the field.

`carlos schedule ls` renders it — one line, in the member's own vocabulary:
that this schedule fires more often than the deployment's idle window, so the
instance stays awake between ticks and accrues instance-hours accordingly.
`carlos schedule set` shows the same line once the converger has written the
status.

**Warn, never refuse. (decision)** A tight schedule is a legitimate thing to
want, and the platform serves it correctly today. The defect is that it was
silent, not that it was allowed.

## 4. The consequence to be explicit about

**Not every host is inert under this change — it depends which side of the
idle window the host's schedule sits on.**

A host whose schedule fires *more often* than the idle window — `jam`, 5m
against the flagship's 15m window — is unaffected. The guard holds it
resident for the same reason the accounting bug used to: its next tick is
always inside the window, so the sweeper's answer is "stay awake" today and
after this lands. `jam` will not begin sleeping when this ships.

A host whose schedule fires *less often* than the idle window is not inert,
if that host also takes visitor traffic. Before this branch, every tick —
regardless of how far apart — reset `lastActive`, so a tick landing between
two visits kept pushing hibernation out. After this branch a tick no longer
resets the clock, so that host loses the extra runway its ticks used to buy
it: it can now hibernate up to one full idle window earlier than before,
measured from its last real visitor request rather than from whichever tick
last happened to land. That is the fix working as designed — a tick was
never meant to count as activity — but it is a real, observable behaviour
change on *this* branch, not a case of "provably alters nothing". Whether the
flagship carries any host in this shape today is exactly what §7's rollout
check exists to establish before rolling.

The sequencing that remains deliberate is narrower than "inert, then
behaviour change": this change runs under every tenant instance on every
box, so shipping it and confirming its effect matches §7's expectations
before separately tightening `CARLOS_HIBERNATE_IDLE` 15m → 90s keeps the two
changes distinguishable if something needs reverting. The idle-window
tightening is the change that turns `jam` from "stays resident" into
"sleeps and wakes"; this branch is the change that stops loose-schedule
hosts getting undeserved extra runway from their ticks.

**(ruling — Paul, 2026-08-30)** The two land as separate branches.

## 5. What is not built

- **No `NextDue` rework, no new registry column, no second waker.** §2.
- **No per-route idle override.** One knob per deployment, plus a guard that
  reads it, covers every case found. A per-route window is a product decision
  nobody has asked for.
- **No new tuning constant.** §3.2.
- **No special case for `carlos schedule run`.** A manual fire is
  `FireTimerNow`, which moves `next_at` and nothing else
  (`internal/registry/timers.go:132`); the runner cannot tell it from a
  recurring fire without a new column. A schema change to buy "the instance
  stays up 15 minutes after you press run" is not worth it. Manual runs are
  ticks. **(decision — reversed during the brainstorm, after reading the
  code: the first instinct was to treat a human's press as visitor activity.)**
- **No use of `routes.next_due` in the guard.** It looks like free extra
  coverage — the app's own declared due time, already in the registry, already
  read by `DueInstances` — and it is a trap. That column is written only at
  sleep and is deliberately not cleared on wake: "the registry's stale past
  value while the host is up is harmless" (`activator.go:1007`). A guard
  reading it on an awake host would compare against a timestamp in the past,
  conclude the next delivery is imminent forever, and wedge itself open. The
  value could only be made current by probing the app on every sweep pass,
  which is a socket round trip per idle host per tick. The guard covers
  platform timers; app-declared schedules stay uncovered, and are fine there
  because their intervals (hourly and up, in every case on the fleet today)
  are far wider than any plausible idle window.
- **No change to sidecar accounting.** It is already correct and is the
  precedent this follows.

## 6. Testing

All of it runs without a box.

`internal/activator`:

- a scheduled delivery leaves `lastActive` untouched; a visitor request to the
  same host moves it — the two asserted against each other in one test, since
  the bug is precisely that they were the same path;
- a tick against a *sleeping* host wakes it, and the in-flight count holds it
  through delivery (a sweep racing the tick must not stop it mid-flight);
- the guard: a host idle past the window with a timer due inside `cfg.Idle`
  stays resident; the same host with the timer beyond `cfg.Idle` hibernates;
  a host with no timer rows hibernates as before;
- the guard's query runs before `a.mu` — asserted structurally, the way the
  existing sweep tests pin their ordering.

`internal/host`:

- `activatorDeliverer.Deliver` routes through `ServeScheduled`, not
  `ServeHTTP`, and a delivery to a host that was asleep when the tick came due
  still returns the handler's status;
- `X-Carlos-Schedule-At` still reaches the handler unchanged.

`internal/schedule`:

- a status object written by a box that computed the verdict round-trips, and
  one written without the field still parses.

## 7. Rollout

Ordinary: merge, roll the fleet, confirm the change lands where it should
and nowhere else — see §4, this is not a blanket "confirm inert". Two checks:

- **`jam`-shaped hosts — schedule interval shorter than the box's idle
  window — must not move at all.** `jam` itself must still be resident
  afterwards, and `seapointish` (no schedule) must still be doing its ten
  wake/sleep cycles a day. A change to either here is a bug, not a win.
- **Before rolling, enumerate hosts whose schedule interval is longer than
  the idle window.** Expect their wake/sleep cycle counts to rise slightly —
  this branch stops their ticks buying free runway between real visits, so a
  small rise there is the change working as designed. Only a large or
  otherwise unexplained jump is worth investigating.

The separate idle-window tightening (15m → 90s) is verified on the *next*
branch, against the same apps.