package edge // The home map, and the RECEIVING half of edge-to-edge backhaul (spec // docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md, Part 3, // Task 3). // // Two things live here because they are read on the same poll and used in the // same breath: // // - homeMap mirrors control/edges/ — the per-box files internal/host's // edgesFilePass publishes (its edgesFile shape, parsed here as // edgesFileView) — and folds them into one deployment-wide host→home // answer, Home(host). It is the same 30s list-then-get, carry-forward, // last-good mirror certsync.go's certMirror is; that shape survived heavy // review and is copied rather than reinvented. It NEVER writes: the // writer map gives control/edges/.json to that box's own loop, and // nothing in this package bumps control/serial. // // - the mesh keys ride the SAME refresh (loadMeshKeys, mesh.go), so a // request path never reads the store: who homes a host and what key // proves a hop are answered from one poll's view of the bucket rather // than two. // // acceptHop is where a hop lands on the request path. The trust reasoning it // rests on — why the MAC binds the concrete host, why ±hopSkew is the replay // bound, and why dual-accept rotation means the request path must call // verifyHopEitherKey and never the single-key verifyHop — is written out in // mesh.go's file header; read that, this code does not re-derive it. Two // consequences of it are worth naming HERE because they are this file's // decisions rather than mesh.go's: // // - The replay envelope is deliberately loose and that is affordable. A // captured X-Carlos-Hop-Auth is good for at most 2×hopSkew against ONE // host, and what it buys a replayer is a request to a route the edge // already serves to the whole internet — no new reachability, no // privilege — PLUS, for that one host and that one window, XFF belief: // verification gates exactly one thing, whether the inbound // X-Forwarded-For is believed, and a valid MAC is what grants that // belief. What keeps the loose envelope affordable is not that XFF // belief is withheld — it is that capturing a valid Hop-Auth in the // first place requires breaking the hop's TLS or holding // control/edge-mesh-key outright, either of which already hands an // attacker strictly more than "who does this one host think the client // was for 2×hopSkew seconds." // // - XFF is trusted as an inbound chain to CARRY, never one to extend. On a // verified hop the chain the sending edge built is passed through // verbatim (setForwarded, below), so the app sees exactly what a // single-region app sees and the LAST element is still the client. The // backhaul leg is deliberately invisible in that header: this // deployment's own consumers take the last element as the caller // (internal/console/authcli.go's clientIP, and every tenant framework's // rightmost-public heuristic), so appending the sibling edge would make // every backhauled request look like it came from infrastructure. On // anything but a verified hop the chain is discarded outright: an // unverified sender is an ordinary client, and an ordinary client has // never been allowed to state its own address here. // // One hop, ever, is STRUCTURAL and not a policy check: a verified hop that // names a host this box does not serve leaves through staleHopRefusal (503 + // Retry-After: 2) from inside the lookupUpstream-miss branch, BEFORE the point // where Task 4 adds the backhaul dial. There is no path from an accepted hop // to Home() at all, so a re-forward cannot be introduced by adding a branch — // only by deleting one. // // Single-box inertness: with one platform box the only edges file is this // box's own, own-box exclusion drops it, and Home() therefore never offers a // target. Nothing signs hops either (Task 4 is the only signer, and it needs a // target), so acceptHop's verified branch is unreachable on a single-box // deployment — this task ships the receiving half, dormant. import ( "context" "encoding/json" "errors" "fmt" "log" "net/http" "net/http/httputil" "os" "slices" "sort" "strconv" "strings" "sync" "time" "github.com/carlosframework/platform/internal/edgesfile" "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/release" ) // edgesPrefix is where every box publishes what it homes — one flat object // per box, control/edges/.json (internal/host's edgesFileKey). const edgesPrefix = "control/edges/" // The hop's two headers, verbatim from the spec's Part 3. Task 2 left them // prose-only; this file is their first consumer, and Task 4's signer imports // these constants rather than spelling the strings a second time. const ( hopHeader = "X-Carlos-Hop" hopAuthHeader = "X-Carlos-Hop-Auth" ) // edgeRoleHome is the only role v1 writes, and the only one that routes. // Unknown roles (the spec's read-local anticipation) are IGNORED rather than // refused: a newer box publishing a role this build has never heard of must // not take its whole file — or the poll — down with it. const edgeRoleHome = "home" // edgeRoleWildcard is the cert-only half of a delegated wildcard parent // (#305): the parent's PEM should be loaded everywhere, but this box does // not serve the catch-all, so the entry is no dial target. const edgeRoleWildcard = "wildcard" // homeMapInterval / homeMapTimeout / homeMapBootTimeout mirror the cert // mirror's cadence and budgets exactly (certsync.go), for the same reasons: a // home move is a placement event that converge heals, the poll runs on its own // goroutine and blocks nothing, and the ONE boot refresh must delay the front // door by seconds rather than minutes. Vars so tests can shorten them. var ( homeMapInterval = 30 * time.Second homeMapTimeout = 2 * time.Minute homeMapBootTimeout = 20 * time.Second ) // edgesFileView is the READER's half of internal/host's edgesFile wire shape // (edgesfile.go). Deliberately a separate declaration rather than a shared // type: internal/host imports this package and never the reverse, and the // fields this side needs are the stable three (box, eip, hosts) plus the // version. Additive growth on the writer's side lands here as fields this // decoder ignores, which is the point. type edgesFileView struct { V int `json:"v"` Box string `json:"box"` EIP string `json:"eip"` Hosts map[string]edgesHostView `json:"hosts"` // PrivateIP, PrivateCIDR, Region and Placement choose the dial address // (region HA spec 2 §6, dialAddress). All are absent from a writer that // predates them, and the dial is then EIP, as before. PrivateIP string `json:"private_ip"` PrivateCIDR string `json:"private_cidr"` Region string `json:"region"` Placement string `json:"placement"` } // dialAddress is where this edge dials f's box (region HA spec 2 §6). With // privateDial (this edge's own region HA switch) on, a box that publishes // the same AWS region and placement as this one is dialled at its private // address, so the dial stays inside the region's network, but only when // that address passes edgesfile.PrivateAddress against the network this // box's own file publishes (census ruling region-ha-2a). The home's // placement code is "", so two home boxes share a placement as two boxes // of one named region do; a home box and a named one never do (Astra 2a // final review, P2). Every other box, and every box with privateDial off, // is dialled at its public address, as before. self is this box's own // file, which selfKnown says was found. "" is nothing to dial. func dialAddress(f, self edgesFileView, selfKnown, privateDial bool) string { if privateDial && selfKnown && f.Region != "" && f.Region == self.Region && f.Placement == self.Placement { if addr, _ := edgesfile.PrivateAddress(f.PrivateIP, []string{self.PrivateCIDR}); addr != "" { return addr } } return f.EIP } // regionHAFromEnv is CARLOS_REGION_HA as strconv.ParseBool reads it, the // reading every region HA switch makes. func regionHAFromEnv(getenv func(string) string) bool { on, err := strconv.ParseBool(strings.TrimSpace(getenv("CARLOS_REGION_HA"))) return err == nil && on } // edgesHostView is the per-host object. An object rather than a bare string is // what makes the role field addable without a wire break — see edgeRoleHome. type edgesHostView struct { Role string `json:"role"` // Platform is the publisher's class mark (internal/host edgesfile.go): // a platform-minted row. Absent on files written before the mark // existed, which reads as "not platform" — fail closed for the one // consumer that cares (Hosts, the wildcard parent set). Platform bool `json:"platform,omitempty"` // Alias marks an attached custom domain (#287). On a name collision // across boxes a route entry beats an alias entry (foldHomes). Alias bool `json:"alias,omitempty"` // Wildcard marks a delegated customer wildcard parent's catch-all entry // (#305): WildcardParents feeds it to the wildcard manager. Wildcard bool `json:"wildcard,omitempty"` // Account and App are the one app that owns every route the publishing // box serves under this name (#175 P7). The publisher leaves both out // when the name is split between apps; so does a publisher that predates // them. The edge-local door refuses a remote host with no owner. Account string `json:"account,omitempty"` App string `json:"app,omitempty"` // Stamp orders two boxes' claims on one host: see foldHomes. Stamp int64 `json:"stamp,omitempty"` } // homeEntry is one folded answer: where to dial, and which box that is. type homeEntry struct { // eip is the address to dial: dialAddress's answer, the home box's // private address within its region and its EIP from anywhere else. eip string box string // platform carries the publisher's class mark through the fold. platform bool // alias marks an attached custom domain's entry (never platform). alias bool // wildcard marks a delegated wildcard parent's catch-all entry. wildcard bool // dial reports whether the entry is a backhaul target (role home). A // cert-only wildcard entry folds with dial false and Home skips it. dial bool // account and app are the owner the dial target's box published. They // travel with eip and box and are never merged across entries. account, app string // stamp is the dial winner's installation stamp, 0 when unstamped. stamp int64 } // homeSnapshot is one poll's result, swapped in whole and immutable once // published, so a reader holds no lock past the pointer read. type homeSnapshot struct { // files is the parsed per-box file, keyed by its BUCKET key. This is the // carry-forward unit: an object that could not be read (or could not be // parsed) this poll keeps its previous parse, per box, so one bad file // never costs the deployment its whole map. files map[string]edgesFileView // hosts is the fold: host → home, own box already excluded. hosts map[string]homeEntry } var emptyHomeSnapshot = &homeSnapshot{files: map[string]edgesFileView{}, hosts: map[string]homeEntry{}} type homeMap struct { s release.Store // self is THIS box's label — the same derivation the box's own edges file // is published under (internal/host's metricsBoxLabel), handed down at // edge construction through WithBoxLabel. "" means this process does not // know which box it is, and Home then offers nothing at all: see its doc. self string // refreshMu serializes Refresh's whole body, for the reason certMirror's // does — two overlapping polls could otherwise publish out of order, the // slower one clobbering the fresher. refreshMu sync.Mutex mu sync.RWMutex snap *homeSnapshot // hostsWake fires after a Refresh that changed the NAME set (setHostsWake). hostsWake func() // partialPolls counts polls that could not read (or parse) every object, // for the first-and-every-tenth log cadence. partialPolls int // staleHopN counts staleHopRefusal occurrences per "box host" key (box is // always m.self — despite the name this is a PER-HOST counter in // practice, not cross-sender isolation; see staleHopRefusal's doc for // why box is still named in the key and the log line), for the SAME // first-and-every-tenth cadence partialPolls drives. Guarded by mu, // following logPartialPoll's own choice to reuse the one lock rather // than mint a second. // // Capped at staleHopKeyCap distinct keys — host here is CLIENT- // CONTROLLED (Home()'s one-label wildcard match means a single // catch-all edges-file entry lets a sending box mint a signed hop for // any invented subdomain, edge.go's backhaul doc), so an unbounded map // would be unbounded memory for the life of the process. Review catch // on this same commit: the identical reasoning already gated the // backhaul-error logThrottled key (edge.go) and is logThrottled's own // documented rule (never key from remote input) — this map just has no // logThrottled to inherit the cap from, so it gets its own. staleHopN map[string]int // The mirrored mesh keys, read on the same refresh. Replaced only on a // SUCCESSFUL load: a bucket blip leaves the last good key verifying hops, // exactly as a blip leaves the last good host map routing them. keyMu sync.RWMutex keyCurrent []byte keyNext []byte // privateDial is this edge's region HA switch: only with it on does // the fold dial a same-region box's private address (dialAddress). privateDial bool // health is the load balancer health state (region HA spec 2 §6), // nil without region HA: Refresh latches the first loaded map and key // and stamps every attempt it completes. health *lbhealth.State // watch and now are certMirror's (#336): the process's change-index // watch, resolved on every poll (nil, or a nil watch, is today's map), // and the poll's clock (nil is time.Now). Only the hosts half is gated // on them; the mesh keys keep their own read on every poll. watch func() *release.ChangeWatch now func() time.Time // watchTimeout bounds the poll of the watch (0 is // release.ChangeIndexPollTimeout; tests shorten it). watchTimeout time.Duration // Guarded by refreshMu: the last hosts rebuild that read every edges // file (lastHosts), the jittered backstop after which the next poll // reads them all again (hostsAge), and the control/edges generation, // captured BEFORE that read, which the fold is known to reflect // (actedOK false = none: the watch could not vouch, or a file went // unread or unparsed and must be retried). lastHosts time.Time hostsAge time.Duration actedGen uint64 actedOK bool } func newHomeMap(s release.Store, self string) *homeMap { return &homeMap{s: s, self: self, snap: emptyHomeSnapshot} } // newHomeMap is the Edge's map, or nil when there is nothing to mirror FROM — // the same no-deployment-store case that leaves the cert mirror nil. A box // with no deployment bucket is not part of any mesh: it offers no backhaul // target AND verifies no inbound hop (no key to verify against), which is the // fail-closed answer either way. func (e *Edge) newHomeMap() *homeMap { if e.staticStore == nil { return nil } m := newHomeMap(e.staticStore, e.box) m.privateDial = regionHAFromEnv(os.Getenv) m.health = e.lbHealth m.watch = e.changeWatch return m } // Refresh rebuilds the host fold and re-reads the mesh keys. The two halves // fail INDEPENDENTLY — a bucket that answered the edges list but not the key // object should still land the fold — so both run and their errors join. // Whatever fails leaves its own last-good in place; the caller logs. // // For the load balancer's health answer (region HA spec 2 §6) each half // latches its first success, and every call that returns stamps an // attempt, failed or not: a bucket outage after arming leaves the box // healthy, and only a refresh stuck inside one attempt goes stale. func (m *homeMap) Refresh(ctx context.Context) error { m.refreshMu.Lock() defer m.refreshMu.Unlock() return m.refresh(ctx, true) } // poll is the 30s loop's refresh (#336). Disarmed, it is Refresh. Armed, it // polls the change index and skips the hosts half when control/edges has not // moved since the generation the fold reflects and the last full hosts read // is younger than the jittered backstop; otherwise (moved, the watch unable // to vouch, no read yet, or at the backstop) the hosts half runs as today. // The mesh keys are read on every poll whatever the index says: their // objects (control/edge-mesh-key and .next) bump no scope, so a rotation // must not wait on control/edges (codex 8). Poll's moved list is advisory // (the watch is shared), so the decision compares generations instead. func (m *homeMap) poll(ctx context.Context) error { w := m.changeWatch() if w == nil { return m.Refresh(ctx) } // Bounded on its own (pollWatch): a hung index GET elsewhere in the // process must not stop this poll from stamping RefreshAttempted. Any // error, giving up on the watch's lock included, reads the hosts. _, perr := pollWatch(ctx, w, m.watchTimeout) m.refreshMu.Lock() defer m.refreshMu.Unlock() gen, ok := w.Gen(edgesScope) current := perr == nil && ok && m.actedOK && gen == m.actedGen && !m.lastHosts.IsZero() && m.clock().Before(m.lastHosts.Add(m.hostsAge)) return m.refresh(ctx, !current) } // refresh is Refresh's body, under refreshMu. readHosts false skips the // hosts half, which then counts as a success: the fold it keeps is one a // clean read landed (poll skips only after one, and that read latched // MapLoaded), so the latch below is a no-op and the health answer is what a // full read would have given. The attempt is stamped either way, so the // five-minute refresh heartbeat holds across skipped polls. func (m *homeMap) refresh(ctx context.Context, readHosts bool) error { var hostsErr error if readHosts { hostsErr = m.refreshHosts(ctx) } keysErr := m.refreshKeys(ctx) if hostsErr == nil { m.health.MapLoaded() } if current, _ := m.meshKeys(); keysErr == nil && len(current) > 0 { m.health.KeyLoaded() } m.health.RefreshAttempted(m.clock()) return errors.Join(hostsErr, keysErr) } // edgesScope is the change-index scope every write under edgesPrefix bumps. const edgesScope = "control/edges" func (m *homeMap) changeWatch() *release.ChangeWatch { if m.watch == nil { return nil } return m.watch() } func (m *homeMap) clock() time.Time { if m.now != nil { return m.now() } return time.Now() } // refreshHosts rebuilds the whole map from one list plus a get per key — no // delta cleverness, for certMirror.Refresh's reason: a full rebuild is what // makes a RETIRED box's hosts actually leave the map. // // An error leaves the last-good snapshot in place. That matters as much here // as it does for certs: an empty map on a bucket blip would stop every // backhaul in the deployment at once, turning a blip into a regional outage // for every host homed elsewhere. func (m *homeMap) refreshHosts(ctx context.Context) error { // The generation this read will reflect, captured BEFORE the list, so a // write landing during the read leaves control/edges moved past it (#336). var gen uint64 var genOK bool if w := m.changeWatch(); w != nil { gen, genOK = w.Gen(edgesScope) } keys, err := m.s.List(ctx, edgesPrefix) if err != nil { return fmt.Errorf("list %s: %w", edgesPrefix, err) } prev := m.snapshot() next := make(map[string]edgesFileView, len(keys)) // failed counts both fault classes; unread counts only the objects the // BUCKET would not hand over, which is the one that says "this is not an // object fault at all" when it covers everything (see below). var tried, failed, unread int var lastErr error for _, key := range keys { // The prefix is flat and every member is .json. Anything else // under it was not written by this deployment's edges loop. name, ok := strings.CutPrefix(key, edgesPrefix) if !ok || !strings.HasSuffix(name, ".json") || strings.Contains(name, "/") { continue } tried++ body, _, err := m.s.Get(ctx, key) if errors.Is(err, os.ErrNotExist) { // Raced a delete between the list and the get — the box was // retired mid-poll. Not a fault; it is simply gone. continue } // ONE bad object is not a bad poll, whichever way it went bad. // Aborting would discard every other box's file that landed this tick, // and a permanently unreadable (or permanently corrupt) key would // freeze the map forever. Carry this box's previous parse forward and // keep going. if err != nil { unread++ } else if f, perr := parseEdgesFile(key, body); perr != nil { err = perr } else { next[key] = f continue } failed++ lastErr = err if old, ok := prev.files[key]; ok { next[key] = old } } if tried > 0 && unread == tried { // Nothing at all could be READ: a bucket fault, not an object fault, // reported like a failed List — last-good stands. Note the bound is // unread and not failed: a deployment whose only edges file is a // corrupt one has a bucket that answered perfectly well, and calling // that a bucket fault would report a blip that is not happening. return fmt.Errorf("get %s*: every one of %d object(s) was unreadable, last: %w", edgesPrefix, tried, lastErr) } if failed > 0 { m.logPartialPoll(failed, tried, lastErr) } hosts, tied := foldHomes(next, m.self, m.privateDial) if len(tied) > 0 { // One claim revision names one executor, so two boxes publishing // one host at the same stamp is a fault upstream. The fold stays // deterministic; say so once a refresh. log.Printf("edge: home map: %d host(s) published by two boxes at the same installation stamp, e.g. %s", len(tied), tied[0]) } snap := &homeSnapshot{files: next, hosts: hosts} m.mu.Lock() m.snap = snap wake := m.hostsWake m.mu.Unlock() // A partial read reflects no generation: the file it could not read or // parse is retried on the next poll, as it always was. m.lastHosts, m.hostsAge = m.clock(), jitteredBackstop() m.actedGen, m.actedOK = gen, genOK && failed == 0 if wake != nil && !samePlatformSet(prev.hosts, snap.hosts) { // The deployment's platform name set moved: a wildcard parent may // have appeared for an account homed elsewhere (#300). Off this // goroutine — the wake re-derives through Hosts() (its own locks) // and may touch the registry, and refreshKeys should not wait on // that. go wake() } return nil } // samePlatformSet reports whether two folds name the same PLATFORM hosts — // the only names that can move a wildcard parent (homes may differ; an // unclassed name may come and go). func samePlatformSet(a, b map[string]homeEntry) bool { na, nb := 0, 0 for h, e := range a { if !parentInput(h, e) && !wildcardInput(h, e) { continue } na++ if eb, ok := b[h]; !ok || (parentInput(h, e) != parentInput(h, eb)) || (wildcardInput(h, e) != wildcardInput(h, eb)) { return false } } for h, e := range b { if parentInput(h, e) || wildcardInput(h, e) { nb++ } } return na == nb } // parentInput is the filter Hosts and samePlatformSet share for ACCOUNT // parents: a marked, exact (non catch-all) name. wildcardInput is its twin // for delegated parents (WildcardParents). Anything else coming or going // cannot move a parent, so it must not wake the manager either. func parentInput(host string, e homeEntry) bool { return e.platform && !strings.HasPrefix(host, "*.") } func wildcardInput(host string, e homeEntry) bool { if !e.wildcard || !strings.HasPrefix(host, "*.") { return false } parent := host[2:] // The shape a local wildcard_domains row passed (validHostShape at // reconcile), applied to a remote one: a real dotted name, no further // wildcard label, nothing a challenge FQDN could not be built from. return parent != "" && strings.Contains(parent, ".") && !strings.ContainsAny(parent, "*/ \t") } // WildcardParents is every delegated customer wildcard parent another box // publishes ("*." entries marked wildcard, #305), as bare parents. // The wildcard manager unions these with its own wildcard_domains rows so // a region edge loads the parent's mirrored PEM and terminates the // one-label hosts Home already dials for it. Sorted, nil-safe, nothing for // an unlabelled box (as Hosts). func (m *homeMap) WildcardParents() []string { if m == nil || m.self == "" { return nil } snap := m.snapshot() out := make([]string, 0) for h, e := range snap.hosts { if wildcardInput(h, e) { out = append(out, strings.TrimPrefix(h, "*.")) } } sort.Strings(out) return out } // Hosts is every exact PLATFORM-class name the deployment's OTHER boxes // home. Sorted, nil-safe. The wildcard manager unions these with this box's // own routes so its parent set is the deployment's, not the box's (#300): // a region edge that adopts only everywhere-class routes would otherwise // never derive the account parent of an exec app homed elsewhere, and so // never load the wildcard PEM the mirror had already written to its disk. // // Two bounds, both fail-closed: the publisher's class mark (a row that is // neither Platform nor Provisioned — an operator's stray `carlos add` // under the apps domain — derives nothing here, exactly as currentHosts // refuses it locally; a file from before the mark existed contributes // nothing until its box republishes), and parentsFor's two-labels-under- // the-apps-domain rule on top. Catch-all "*." keys are excluded: a // remote catch-all is a claim's row, never an account parent. An // unlabelled box (self == "") reports nothing, as Home does — its own // file would otherwise fold in as "another box". func (m *homeMap) Hosts() []string { if m == nil || m.self == "" { return nil } snap := m.snapshot() out := make([]string, 0, len(snap.hosts)) for h, e := range snap.hosts { if !parentInput(h, e) { continue } out = append(out, h) } sort.Strings(out) return out } // ServedHosts lists every EXACT hostname any box in this deployment homes — // routes and attached custom domains alike, with no class filter. // // Deliberately NOT Hosts(): that one applies parentInput, which keeps only // platform-class rows because its consumer derives wildcard PARENTS from // them. The cert warm's question is the opposite one — "what exact names does // this deployment answer for?" — and a custom domain is precisely the // non-platform row Hosts() drops. Filtering by class here would warm every // canary and no customer domain, which is backwards. // // Catch-all "*." keys are excluded: they are covered by a wildcard PEM, and // autocert refuses the wildcard label anyway (Allowed). Only dial-able // entries count — a cert-only wildcard row homes nothing. // // An unlabelled box (self == "") reports nothing, as Home does: an edge that // cannot tell its own rows from another box's must not answer at all. func (m *homeMap) ServedHosts() []string { if m == nil || m.self == "" { return nil } snap := m.snapshot() out := make([]string, 0, len(snap.hosts)) for h, e := range snap.hosts { if !e.dial || strings.HasPrefix(h, "*.") { continue } out = append(out, h) } sort.Strings(out) return out } // setHostsWake installs the callback a changed host set fires (Refresh). // Under the map's own lock: run() is already polling when the wildcard // manager is wired in after it. func (m *homeMap) setHostsWake(fn func()) { if m == nil { return } m.mu.Lock() m.hostsWake = fn m.mu.Unlock() } // parseEdgesFile decodes one box's file. A parse failure gets the same // treatment an unreadable object gets — carry the last good parse forward — // because a half-written body is no more usable than a body that never // arrived, and the fix for both is the publisher's next pass. func parseEdgesFile(key string, body []byte) (edgesFileView, error) { var f edgesFileView if err := json.Unmarshal(body, &f); err != nil { return edgesFileView{}, fmt.Errorf("%s: %w", key, err) } if f.V < 1 { // No version at all: not a file this reader wrote a decoder for. A // HIGHER version is accepted on purpose — the schema grows additively // (the plan's global constraint), so a newer box's file still carries // the box/eip/hosts this reader needs, and refusing it would black out // backhaul to every box that upgraded first. return edgesFileView{}, fmt.Errorf("%s: unversioned edges file", key) } return f, nil } // logPartialPoll reports unreadable or unparseable files on the loop's own // first-and-every-tenth cadence: a permanently bad object would otherwise say // the same thing every 30 seconds forever. func (m *homeMap) logPartialPoll(failed, tried int, err error) { m.mu.Lock() m.partialPolls++ n := m.partialPolls m.mu.Unlock() if n == 1 || n%10 == 0 { log.Printf("edge: %d of %d object(s) under %s unreadable this poll (#%d, last: %v) — the previous copy of each is carried forward", failed, tried, edgesPrefix, n, err) } } // foldHomes turns the per-box files into one host→home map. // // Three exclusions, each of which is a request this box must never make: // // - OWN BOX. A host homed here is SERVED here; offering it as a backhaul // target would have the edge dial itself. Excluded by the file's own box // field rather than by its object name, because the field is what the // publisher stamps and what self is derived to match. // - NO DIAL ADDRESS. Task 1 publishes a box with no discoverable public IP // as eip:"" so the DR runbook still gets its hosts list. There is // nothing to dial, so it is not a target. The address is dialAddress's: // a box in this box's own region is dialled at its private address when // it publishes one (region HA spec 2 §6), every other at its EIP. // - A ROLE THIS BUILD DOES NOT KNOW. Only "home" routes. The role field // exists so read-local roles can be added later; a reader that treated // every role as a home would send traffic to a box that had declared it // serves only reads. // // Iteration is over SORTED file keys so a duplicate claim — two boxes both // saying they home one host, which is a placement bug mid-move — resolves the // same way on every edge in the deployment. Convergent-but-wrong beats // split-and-wrong: the whole fleet aiming at one box is a condition that heals // on the next publish, where half the fleet aiming at each is a condition that // looks fine from any single box. func foldHomes(files map[string]edgesFileView, self string, privateDial bool) (map[string]homeEntry, []string) { keys := make([]string, 0, len(files)) for k := range files { keys = append(keys, k) } slices.Sort(keys) var own edgesFileView ownKnown := false for _, k := range keys { if f := files[k]; self != "" && f.Box == self { own, ownKnown = f, true } } hosts := make(map[string]homeEntry) for _, k := range keys { f := files[k] if f.Box == "" || f.Box == self { continue } addr := dialAddress(f, own, ownKnown, privateDial) for host, h := range f.Hosts { if host == "" { continue } switch h.Role { case edgeRoleHome: // A dial target needs an address. A file with none (a box // with no discoverable public IP) still carries its cert // halves below; its dial halves are simply not dialable. if addr == "" { if !h.Wildcard || !strings.HasPrefix(host, "*.") { continue } h.Role = edgeRoleWildcard } case edgeRoleWildcard: // The cert-only role is defined for marked catch-all keys // alone; on an exact key it would be a KnownHost admission // with no dial behind it — refused. if !h.Wildcard || !strings.HasPrefix(host, "*.") { continue } default: continue } in := homeEntry{eip: addr, box: f.Box, platform: h.Platform && !h.Alias && !h.Wildcard, alias: h.Alias, wildcard: h.Wildcard, dial: h.Role == edgeRoleHome, account: h.Account, app: h.App, stamp: h.Stamp} prev, dup := hosts[host] if !dup { hosts[host] = in continue } // A collision merges CLASSES and picks ONE dial target: the // classes are facts about the name (a delegated parent's PEM // must load everywhere even if some other box also routes the // key), the dial target is a choice. First box in sorted order // wins the dial — except that a ROUTE beats an ALIAS whatever // the order (#287), and any dial target beats none. The owner // is part of the dial choice, not a class: it comes whole from // the winner, so a route that replaces an alias never inherits // the alias's owner, even when it publishes none of its own. merged := prev merged.platform = prev.platform || in.platform merged.wildcard = prev.wildcard || in.wildcard if dialBeats(in, prev) { merged.eip, merged.box, merged.alias, merged.dial = in.eip, in.box, in.alias, true merged.account, merged.app, merged.stamp = in.account, in.app, in.stamp } hosts[host] = merged } } return hosts, tiedStamps(files) } // tiedStamps lists, sorted, the hosts that two boxes publish as home at // the same positive stamp. It looks at every file, this box's own // included, and at every pair, not only at whichever entry won the dial: // in a two-box region each box must be able to see that it and its // neighbour disagree, and a third box's higher stamp must not hide a tie // between two others. // // Unstamped duplicates are not reported. They are normal: a static site // or a public address route runs on every box, and every box publishes it. func tiedStamps(files map[string]edgesFileView) []string { type key struct { host string stamp int64 } boxes := map[key]map[string]bool{} for _, f := range files { if f.Box == "" { continue } for host, h := range f.Hosts { if host == "" || h.Role != edgeRoleHome || h.Stamp <= 0 { continue } k := key{host, h.Stamp} if boxes[k] == nil { boxes[k] = map[string]bool{} } boxes[k][f.Box] = true } } seen := map[string]bool{} var tied []string for k, set := range boxes { if len(set) > 1 && !seen[k.host] { seen[k.host] = true tied = append(tied, k.host) } } slices.Sort(tied) return tied } // dialBeats reports whether a later file's entry takes the dial from the // one already folded. In order: // // - only a dial target can take it, and any dial target beats none; // - a ROUTE beats an ALIAS whatever else is true (#287); // - between two of the same kind, the newer installation wins: a stamped // entry beats an unstamped one, and a higher stamp beats a lower one // (region HA spec 1 HA4). A dead box's file lingers; its successor's // stamp is always higher, so the stale one loses without any liveness // rule; // - otherwise the earlier file keeps it, which is the first box in // sorted order, as before. func dialBeats(in, prev homeEntry) bool { if !in.dial { return false } if !prev.dial { return true } if prev.alias != in.alias { return prev.alias && !in.alias } return in.stamp > prev.stamp } func (m *homeMap) snapshot() *homeSnapshot { m.mu.RLock() defer m.mu.RUnlock() return m.snap } // Home answers "which OTHER box homes this host, and where do I dial it?". // // The lookup is lookupUpstream's, rule for rule, and that is load-bearing // rather than tidy: Task 1 publishes registry row hosts VERBATIM, catch-all // "*." rows included, so an exact match wins and a miss retries with // the leading dot of the FIRST label swapped for "*" — one label, never two, // and never the parent matching its own wildcard. Deliberately NOT // case-folded, for the same reason: a host lookupUpstream would miss on case // must miss here too, or this edge would backhaul a request the home edge // would then refuse with a 503. // // A nil map (no deployment store) and a map with no self label both answer // "nothing". The self case is fail-closed on purpose: without knowing which // box it is, this edge cannot exclude its own hosts, and an edge that cannot // exclude itself must not be handing out dial targets at all. func (m *homeMap) Home(host string) (eip, box string, ok bool) { h, ok := m.lookupHome(host) if !ok { return "", "", false } return h.eip, h.box, true } // ExactHome is Home without the one-label wildcard fallback: the dial target // another box published for exactly host. The edge asks it when its own // answer for host came only through a local "*." key, so that an // exact name served elsewhere still wins (ServeHTTP). Fail-closed without a // self label, exactly as Home is. func (m *homeMap) ExactHome(host string) (eip, box string, ok bool) { if m == nil || m.self == "" { return "", "", false } if h, found := m.snapshot().hosts[host]; found && h.dial { return h.eip, h.box, true } return "", "", false } // lookupHome is Home's lookup, answering the whole winning entry from one // snapshot read. Home and Owner both go through it, so they can never pick // different entries for the same host. func (m *homeMap) lookupHome(host string) (homeEntry, bool) { if m == nil || m.self == "" { return homeEntry{}, false } hosts := m.snapshot().hosts if h, ok := hosts[host]; ok && h.dial { return h, true } if i := strings.IndexByte(host, '.'); i > 0 { if h, ok := hosts["*"+host[i:]]; ok && h.dial { return h, true } } return homeEntry{}, false } // Owner answers, from one read of one snapshot, both whom the edge-local // door must check and where it would dial (#175 P7), so the check and the // dial cannot disagree. The lookup is Home's, one-label wildcard included, // and eip and box are exactly what Home returns for the same host. // // ok is false, and everything else zero, when the winning entry is not a // dial target or published no owner: a hostname split between apps, or a // publisher that predates the field. An ownerless exact entry does not fall // through to a wildcard that names one, because the exact entry is the one // Home dials. A nil map, or one with no self label, answers nothing, as Home // does. func (m *homeMap) Owner(host string) (account, app, eip, box string, ok bool) { h, found := m.lookupHome(host) if !found || h.account == "" || h.app == "" { return "", "", "", "", false } return h.account, h.app, h.eip, h.box, true } // KnownHost reports whether host is an EXACT key in the mirrored home map — a // name some OTHER box in this deployment homes, spelled precisely. // // Deliberately NOT Home()'s one-label wildcard fallback, and refusing "*." // outright: this predicate widens cert issuance and challenge admission // (certsync.go's three gates, through Edge.deploymentHost), where PR #180's // rule stands — attacker-chosen labels under a catch-all must never drive an // ACME order or a bucket read, and their TLS coverage is the wildcard cert's // job (Allowed's own doc says the same thing for this box's own rows). The // widening loses nothing by staying exact: an order only ever exists for a // name that passed the exact gate, so a challenge only ever needs answering // for an exact name. // // Unlike Home it needs no EIP and no dial target — membership is the whole // question — but it reads the same fold, so it inherits both of the fold's // exclusions. Own-box hosts are excluded and that is harmless: they pass // Allowed on the other half of deploymentHost. Boxes with EIP=="" are // excluded and that is accepted: a box with no public IP offers no backhaul // target and appears in no DNS answer, so a validation for its hosts has // nowhere legitimate to land anyway. // // There is deliberately no m.self gate, unlike Home: Home fail-closes without // a self label because it could otherwise offer this box itself as a dial // target, and membership carries no such hazard — an unlabelled box's fold // keeps its own file, whose hosts are its own registry rows and so pass // Allowed anyway, modulo edges-file staleness (a route deleted from the // registry lingers in the published file until the next publish, minutes; // inherent to the design, and the widened set is registry rows either way). // // Nil-receiver-safe, like Home: the map may not be armed yet (or at all, on a // box with no deployment store). func (m *homeMap) KnownHost(host string) bool { if m == nil || strings.HasPrefix(host, "*.") { return false } _, ok := m.snapshot().hosts[host] return ok } // refreshKeys re-reads the mesh key pair. loadMeshKeys is already tolerant of // a malformed .next and fail-closed on the current key (mesh.go); this only // adds the mirror's own posture on top — the pair is replaced ONLY on a clean // read, so a bucket blip leaves the last good key verifying hops. // // A clean read that finds NOTHING does replace the pair with nil, and that is // not a blip: an absent control/edge-mesh-key means no hop can be proven, and // refusing every hop (they degrade to ordinary clients) is the only safe // reading of "the deployment has no mesh key". func (m *homeMap) refreshKeys(ctx context.Context) error { current, next, err := loadMeshKeys(ctx, m.s) if err != nil { return err } m.keyMu.Lock() m.keyCurrent, m.keyNext = current, next m.keyMu.Unlock() return nil } // meshKeys is the request path's cached view of the key pair — never a store // read per request. Both may be nil (nothing minted yet, or nothing mirrored // yet), which verifyHopEitherKey answers as "no". func (m *homeMap) meshKeys() (current, next []byte) { if m == nil { return nil, nil } m.keyMu.RLock() defer m.keyMu.RUnlock() return m.keyCurrent, m.keyNext } // run polls until stop closes — the edge's standing background-loop shape, // the same one certMirror.run uses: fixed ticker, errors counted and logged at // the first and then every tenth, recovery logged once. func (m *homeMap) run(stop <-chan struct{}) { log.Printf("edge: home map armed (%s poll of %s*, this box is %q) — hosts homed on another box are reachable through it", homeMapInterval, edgesPrefix, m.self) tick := time.NewTicker(homeMapInterval) defer tick.Stop() errs := 0 for { select { case <-stop: return case <-tick.C: } ctx, cancel := context.WithTimeout(context.Background(), homeMapTimeout) err := m.poll(ctx) cancel() if err != nil { errs++ if errs == 1 || errs%10 == 0 { log.Printf("edge: home map refresh failed (#%d): %v — serving the last good map and key", errs, err) } continue } if errs > 0 { log.Printf("edge: home map refresh recovered after %d failure(s)", errs) errs = 0 } } } // ---- hop acceptance on the request path ------------------------------------- // hopVerifiedKey marks a request that arrived as a PROVEN hop from another // edge in this deployment's mesh. A context value rather than a header, // because a header is something a client can send: nothing outside this file // can put this mark on a request, which is exactly the property the XFF trust // below rests on. type hopVerifiedKey struct{} // hopVerified reports whether this request arrived as a verified hop. Task 4's // forwarding decision reads it too — a request that ARRIVED as a hop never // consults the home map. func hopVerified(ctx context.Context) bool { v, _ := ctx.Value(hopVerifiedKey{}).(bool) return v } // acceptHop is the whole of hop handling on the request path, and it runs // before the route lookup so that BOTH of its outcomes are decided ahead of // anything that serves: // // no hop header → nothing happens at all (the overwhelming case: one // Header.Get on a request that is not a hop). // verified hop → the mark goes on the context, the inbound // X-Forwarded-For is believed, and the caller serves // the route locally or refuses (staleHopRefusal). // unverified hop → every forwarded header is stripped and the sender is // an ordinary client from here on. // // The hop headers themselves are deleted in ALL hop cases, verified included: // they are the edge's vocabulary between boxes and have no business reaching a // tenant's app, and a stripped header cannot be laundered back out by an app // that reflects what it was sent (scrubInboundPurgeHeader, purge_header.go, // records what that class of bug already cost this package). // // host is the caller's already-port-stripped request host; canonicalization // into the exact bytes the MAC covers is hopMAC's job, in one place, so a // signer and a verifier can never disagree (mesh.go). func (e *Edge) acceptHop(r *http.Request, host string) (*http.Request, bool) { if r.Header.Get(hopHeader) == "" { // A signature with no marker beside it is not a hop — but it is still // this edge's vocabulary, and the invariant above is "deleted in ALL // hop cases" without an exception for the half a client made up. // scrubInboundPurgeHeader deletes both purge headers unconditionally // for exactly this reason (purge_header.go): an app that reflects what // it was sent turns a client-supplied edge header into an // app-DECLARED one, and no tenant's reflection bug should be able to // put hop vocabulary back on the wire. The Get stays the fast path's // single lookup — the Del only happens on a request that actually // carries the stray header, which is never, in the ordinary case. if r.Header.Get(hopAuthHeader) != "" { r.Header.Del(hopAuthHeader) } return r, false } auth := r.Header.Get(hopAuthHeader) r.Header.Del(hopHeader) r.Header.Del(hopAuthHeader) // verifyHopEitherKey, never verifyHop: the single-key form cannot accept a // signer that has already moved to .next, and a rotation is precisely when // refusing hops would hurt. Keys come from the mirror's cache — a hop must // never cost a bucket read. current, next := e.homes.Load().meshKeys() if auth == "" || !verifyHopEitherKey(current, next, host, auth, time.Now()) { stripForwardedHeaders(r) return r, false } return r.WithContext(context.WithValue(r.Context(), hopVerifiedKey{}, true)), true } // stripForwardedHeaders discards every claim a request makes about who sent // it. httputil.ReverseProxy's Rewrite path already deletes these from the // OUTBOUND request, so for a proxied route this is belt and braces — but the // edge serves plenty that is not a ReverseProxy (static trees, the status // host, cached replays, the activator's dispatch), and all of them read this // same inbound request. func stripForwardedHeaders(r *http.Request) { r.Header.Del("Forwarded") r.Header.Del("X-Forwarded-For") r.Header.Del("X-Forwarded-Host") r.Header.Del("X-Forwarded-Proto") } // staleHopRefusal answers a VERIFIED hop that named a host this box does not // serve: the sending edge's home map is stale, mid-move. 503 + Retry-After: 2 // is the same "come back shortly" answer a dark fleet gets (fleetProxy). // Retry-After: 2 is a keep-trying hint, not a heal estimate: there is no 2s // serial-converge path here (edges never bump control/serial), so what // actually heals a stale map is the edges-file cycle: the home boxes // republish within a couple of seconds of their registry changing // (internal/host's edgesFileLoop, since #174; before that only on a // 5min+jitter timer), and the sending box then picks that up on its 30s // mirror poll. So a sender can retry into this refusal for up to about half // a minute during a home move. // // The body says nothing. It is reachable only by a party holding the mesh key // — an unauthenticated stranger asking for the same host gets the ordinary, // unchanged unknownHost answer — so it cannot tell an outsider that a host // exists somewhere else in the deployment. Keeping it contentless holds that // line for the ONE case where it could still leak: a peer whose key is valid // but whose map is wrong learns only "not here", never "here is where it // went". // // Silent until now was leaving a home move that gets stuck mid-way with // nothing on this box's OWN journal to show it — a sender can retry into // this refusal for a while (see above) and the only visibility was on the // SENDING side. This logs first-and-every-tenth per (box, host), the same // cadence logPartialPoll (above) uses for its own standing condition; box is // always m.self (there is exactly one homeMap per process, and it never // learns the sending edge's identity — the hop MAC covers only host and // time, mesh.go), named in the key and the line anyway so a centrally // aggregated journal reads as "box X refused host Y" rather than an // anonymous host with no owner. // // A nil receiver (no deployment store — the no-mesh-at-all case) still // answers the refusal; it just cannot count or log one, the same "answer // first, degrade the bookkeeping" shape Home and meshKeys already have. func (m *homeMap) staleHopRefusal(w http.ResponseWriter, host string) { w.Header().Set("Retry-After", "2") http.Error(w, "not served here", http.StatusServiceUnavailable) if m == nil { return } m.logStaleHop(host) } // staleHopKeyCap bounds staleHopN's key set (review catch, 2026-08-23): // host is client-controlled (staleHopN's doc, above — a wildcard-homed // route lets a sending box mint a signed hop for any invented subdomain), // so an uncapped map would be unbounded memory, and an unbounded run of // distinct first-time log lines, for the life of the process. Once the cap // is reached, every NEW key folds into one " (other)" overflow bucket // instead of growing the map further; keys already tracked keep their own // counts and cadence. const staleHopKeyCap = 1024 // logStaleHop is staleHopRefusal's counting half, split out so the HTTP // response is written before anything touches mu. func (m *homeMap) logStaleHop(host string) { key := m.self + " " + host m.mu.Lock() if m.staleHopN == nil { m.staleHopN = make(map[string]int) } if _, tracked := m.staleHopN[key]; !tracked && len(m.staleHopN) >= staleHopKeyCap { key = m.self + " (other)" } m.staleHopN[key]++ n := m.staleHopN[key] m.mu.Unlock() if n == 1 || n%10 == 0 { log.Printf("edge: stale hop refusal on box %s for %s (#%d) — the sending edge's home map has not caught up yet", m.self, host, n) } } // setForwarded is what every proxy Rewrite in this package calls in place of // pr.SetXForwarded(), and it exists for one narrow case. // // ReverseProxy's Rewrite path deletes Forwarded and X-Forwarded-* from the // outbound request before Rewrite runs, so SetXForwarded always writes a // single address: the peer that dialled us. That is right for a client and // wrong for a verified hop, where the peer that dialled us is a sibling edge // and the address the app wants is the one that edge already recorded. // // So on a verified hop — and only there — SetXForwarded's answer is REPLACED // by the inbound chain, verbatim. Not appended to: the last element has to // stay the client. This deployment's own consumer says so in as many words // (internal/console/authcli.go's clientIP: "the LAST element is the one to // take"), and tenant frameworks' rightmost-public heuristics agree — a chain // of ", " would rate-limit every backhauled console // request as one bucket per region and show tenant apps infrastructure in // place of their caller. Passing the chain through unchanged means a // backhauled request is indistinguishable from a local one at the app, which // is the whole promise of the hop. Multiple inbound X-Forwarded-For lines are // joined, which is what list-valued header semantics already say they mean // (RFC 9110 §5.3). // // SetXForwarded still runs first, and is still what sets X-Forwarded-Host and // X-Forwarded-Proto: the hop carries the original Host header and is itself // TLS, so both are correct as computed here and need no restore. It touches // only pr.Out, so reading the inbound chain after it is the same read as // before it. func setForwarded(pr *httputil.ProxyRequest) { pr.SetXForwarded() // Last, whichever branch below returns: the proof signs the chain as it // finally leaves (forwardproof.go). defer signForwarded(pr) // A door request is a request from this box (door.go): its RemoteAddr is // already the box's public address, so SetXForwarded wrote the right // chain. Only the scheme differs from the public route to the same URL, // which would have arrived over TLS. if _, ok := doorCallerFrom(pr.In.Context()); ok { pr.Out.Header.Set("X-Forwarded-Proto", "https") return } if !hopVerified(pr.In.Context()) { return } if chain := pr.In.Header.Values("X-Forwarded-For"); len(chain) > 0 { pr.Out.Header.Set("X-Forwarded-For", strings.Join(chain, ", ")) } }