package ops // Is the replica CURRENT, not merely restorable? // // The drill's original question — "does this replica restore clean?" — has a // blind spot that only shows up in the failure it is meant to catch. A // replicator that dies leaves behind a replica that is valid SQLite, restores // perfectly, and passes `integrity_check` forever, while the database it // mirrors moves on without it. The drill stays green; the backup quietly // becomes a snapshot of the day the replicator died. // // This is not hypothetical. On 2026-08-11 the flagship's litestream exited // with "permission denied" writing its shadow WAL into a DynamicUser tenant's // state directory (a root drop-in fixed it the same minute). Had nobody been // watching the journal, nothing in the platform would have said a word: the // next morning's drill would have restored four frozen replicas and reported // them clean. // // The signal used here is SQLite's own: the file-change counter in the // database header, which advances on every write transaction that reaches the // main database file. Compare the source's counter with the restored copy's, // and a replica that has stopped tracking its source shows up as a number that // stopped moving. No S3 layout knowledge, no litestream version dependency, // no clock skew between machines — the comparison is between two files this // process just read. import ( "fmt" "os" "time" "github.com/carlosframework/platform/internal/lease" ) // staleGrace is how long a source must have been QUIET before a replica that // trails it is called stale. It exists to make false alarms impossible in the // ordinary case: a database written seconds before the drill ran is expected // to be a transaction or two ahead of its replica, and reporting that as a // fault would put the drill permanently red on the busiest — most valuable — // databases on the box, which is precisely how a check stops being read. // // 15 minutes is far longer than any replication lag this fleet has (litestream // syncs on change, seconds at most) and far shorter than the gap that matters // (a replicator that died). const staleGrace = 15 * time.Minute type freshness int const ( // freshnessOK: the replica is level with its source, or trailing a source // that was written too recently to judge. freshnessOK freshness = iota // freshnessStale: the source has been quiet for longer than staleGrace and // the replica is STILL behind it. Replication is not happening. freshnessStale // freshnessNoSource: the config names a database that is not on disk. The // replica cannot ever advance again, and nothing is watching the gap. freshnessNoSource // freshnessUnknown: one of the two headers could not be read. Say so; // never let "could not tell" read as "fine". freshnessUnknown ) func (f freshness) String() string { switch f { case freshnessOK: return "ok" case freshnessStale: return "STALE" case freshnessNoSource: return "no-source" default: return "unknown" } } // ChangeCounter is lease.ChangeCounter — one definition of the freshness // signal, shared by both drills. Kept exported here because this package's // tests and callers are its documented door. func ChangeCounter(path string) (uint32, error) { return lease.ChangeCounter(path) } // replicaFreshness compares a restored replica against the live database it // mirrors. srcModTime is the source's last-modified time, passed in so the // caller stats once and tests can drive the grace window. func replicaFreshness(srcPath, restoredPath string, srcModTime time.Time) (freshness, string) { restoredCounter, err := ChangeCounter(restoredPath) if err != nil { return freshnessUnknown, fmt.Sprintf("restored header unreadable: %v", err) } return replicaFreshnessCounter(srcPath, restoredCounter, srcModTime) } // replicaFreshnessCounter is replicaFreshness for a caller that already holds // the restored copy's counter and no longer has the file — which is the // hibernation drill, where lease.VerifyReplica reads the counter and then // sweeps its throwaway copy in the same call. func replicaFreshnessCounter(srcPath string, restoredCounter uint32, srcModTime time.Time) (freshness, string) { srcCounter, err := ChangeCounter(srcPath) if os.IsNotExist(err) { return freshnessNoSource, "source database is not on disk — this replica can never advance again" } if err != nil { return freshnessUnknown, fmt.Sprintf("source header unreadable: %v", err) } if restoredCounter >= srcCounter { return freshnessOK, "" } if quiet := time.Since(srcModTime); quiet <= staleGrace { // Behind, but the source was written within the grace window — this is // ordinary lag, and the next pass will judge it properly. return freshnessOK, "" } return freshnessStale, fmt.Sprintf( "replica is at change %d, the live database at %d, and the database has been quiet since %s — replication has stopped", restoredCounter, srcCounter, srcModTime.UTC().Format(time.RFC3339)) } // sourceModTime is the source database's mtime, or the zero time when it // cannot be stated (which replicaFreshness will treat through its own paths). func sourceModTime(path string) time.Time { fi, err := os.Stat(path) if err != nil { return time.Time{} } return fi.ModTime() }