// Config adoption, observed per route (relay spec §5). Config sync writes // an additive acknowledgement after it converges a route: the shape it // delivered into, whether a restart happened and succeeded, and — for a // route whose rendered CARLOS_RELAY carries relay config — the identities // it just wrote. Nothing reads the ack today but `carlos relay status` // (internal/console/apirelay.go); it is the first per-route statement of // "this is what the process has" the platform keeps. // // The ack names IDENTITIES, not kids: RelaySuffixes carries each region's // full username_suffix (account:app:incarnation:kid), parsed from the // rendered CARLOS_RELAY value with relay.ParseConfig. A kid alone would not // do — a disable and re-enable between two host syncs changes the // incarnation and the key while keeping the kid, and an ack naming only // kids would report the old credentials as adopted. // // internal/appconfig importing internal/relay is one-directional and safe: // internal/relay does not import internal/appconfig. package appconfig import ( "context" "crypto/ecdh" "encoding/json" "errors" "fmt" "log" "os" "strings" "time" "github.com/carlosframework/platform/internal/registry" "github.com/carlosframework/platform/internal/relay" "github.com/carlosframework/platform/internal/release" ) // ConfigAckSchema is the ack schema THIS binary writes. It exists so a // reader can tell an agent that renders the instance config layer (senders // design §6.2) from one that does not, and the only honest way to do that // is a positive statement: an ack written by an older agent decodes with // Schema 0, because the field was not there to write. // // It is not the manifest schema and is not tied to it. Bump it when an // agent starts stating something a reader may need to depend on. const ConfigAckSchema = 1 // instanceConfigETagSchema is the ack schema at which InstanceConfigETag // first became a field an agent could write — 1, the schema this plan // introduced it at. InstanceConfigDelivery compares against THIS constant, // not against ConfigAckSchema: the two happen to be equal today, but they // answer different questions, and conflating them is a trap the next schema // bump would fall into silently. ConfigAckSchema is "what schema does this // BINARY write" and is meant to climb every time the ack states something // new; instanceConfigETagSchema is "at what schema did THIS field become // trustworthy" and has no reason to climb again unless the field itself is // ever redefined. A bump to ConfigAckSchema for some unrelated addition // (say, schema 2 adds a field about something else entirely) must not // demote every schema-1 ack — which carries a perfectly good // InstanceConfigETag — to InstanceConfigUnknownAgent for a question it can // already answer. const instanceConfigETagSchema = 1 // ConfigAck is one route's config-adoption acknowledgement, written after // config sync converges its env/config file — see ConfigAckKey for where. type ConfigAck struct { Box string `json:"box"` Host string `json:"host"` // Schema is ConfigAckSchema as written by the agent that produced this // ack, and 0 for every ack written before this field existed. A reader // asking "does this box render the instance config layer" reads this and // nothing else: absence of an ack, or an ack at schema 0, both mean the // question cannot be answered from the box's own statements yet. Schema int `json:"schema"` // Shape is one of "unit", "socket-exec" or "tcp-exec-file" — spec §5's // three delivery shapes. Shape string `json:"shape"` At time.Time `json:"at"` // Restarted and RestartError describe the restart/bounce this sync pass // attempted for the route, if it could attempt one at all: a unit route // that try-restarted cleanly sets Restarted true; one whose try-restart // failed sets RestartError instead. syncExecHost (the exec-file and // socket-exec shapes) always leaves both zero — its bounce is // best-effort and fire-and-forget (bounceLogged swallows the error), // and a TCP-exec-file route's process belongs to an external supervisor // this platform cannot restart at all; its cutover is a member // acknowledgement instead (carlos relay adopted). Restarted bool `json:"restarted"` RestartError string `json:"restart_error,omitempty"` // Hibernating is set when the route was asleep at sync time, so its // restart (a systemd try-restart on an inactive unit) was a no-op by // construction — the config landed but nothing woke to run it. `carlos // relay status` reports such a route as "hibernating, delivered on // wake" rather than claiming an adoption that has not happened yet. Hibernating bool `json:"hibernating,omitempty"` // RelaySuffixes is the username_suffix (account:app:incarnation:kid) // this pass rendered into CARLOS_RELAY, per region — nil if the route's // manifest carries no CARLOS_RELAY entry at all. RelaySuffixes map[string]string `json:"relay_suffixes,omitempty"` // InstanceConfigETag is the etag of the instance overlay // (apps///instances-config/.json) this pass rendered // into the file, empty when the route has no overlay at all. The writer // keeps the etag its own Put returned and compares the two // (InstanceConfigDelivery): equal is delivered, different is stale. A // map of delivered var names could not make that distinction — an agent // too old to read the layer and a new agent rendering an overlay whose // vars the reader does not recognise produce the same map. InstanceConfigETag string `json:"instance_config_etag,omitempty"` // MoveOp, FileSHA256 and BundleETags are written only by a move's held // render (RenderHeld, moves design §9.1), never by a sync pass. MoveOp is // the operation the render was made for, FileSHA256 the hex sha256 of the // file as it stands after the render, and BundleETags the etags of the // "default" and "environment" objects merged into it ("" for one that was // absent or not bound). T3 compares them with what it read itself. MoveOp string `json:"move_op,omitempty"` FileSHA256 string `json:"file_sha256,omitempty"` BundleETags map[string]string `json:"bundle_etags,omitempty"` } // ConfigAckPrefix is where every route's config-adoption ack lives, one // object per (box, route host). const ConfigAckPrefix = "control/config-ack/" // ConfigAckKey is where a route's config-adoption ack lives: additive, one // object per (box, route host), rewritten (never appended to) on every sync // that changes what was delivered. func ConfigAckKey(box, host string) string { return ConfigAckPrefix + box + "/" + host + ".json" } // The four answers to "is this instance running the overlay I wrote", named // because three of them are not "no" and a caller that flattened them would // tell an operator to roll a box that is merely one pass behind. const ( // InstanceConfigDelivered: the agent rendered exactly this version of // the overlay. a route with no overlay, whose caller expects none, is // also delivered. there is nothing outstanding. InstanceConfigDelivered = "delivered" // InstanceConfigStale: the agent rendered a different version. one more // pass, or a generation that never got read, is the usual cause. InstanceConfigStale = "stale" // InstanceConfigUnknownAgent: this box's agent predates the instance // config layer, so its ack says nothing about it. not "no". the answer // cannot be had until the box is rolled. InstanceConfigUnknownAgent = "unknown-agent" // InstanceConfigNotYetDelivered: no ack for this host at all. acks are // change-only, so this is the honest answer for a route whose file has // not moved since its agent rolled. it never means "old agent": that // question is the box record's agent version, which step 5 reads // (design §6.5). InstanceConfigNotYetDelivered = "not-yet-delivered" ) // InstanceConfigDelivery compares an ack with the etag a write returned and // says which of the four states the route is in. want is the etag of the // overlay the caller wrote, or "" when the caller wrote none. // // the comparison is on etags rather than on delivered var names because only // etags are decidable: an agent too old to read the layer and a new agent // rendering an overlay whose vars the reader does not recognise deliver the // same set of names (design §6.5). // // The schema test below is against instanceConfigETagSchema, deliberately // NOT ConfigAckSchema — see that constant's own doc comment for the hazard // a future schema bump would otherwise walk into. func InstanceConfigDelivery(a *ConfigAck, want string) string { switch { case a == nil: return InstanceConfigNotYetDelivered case a.Schema < instanceConfigETagSchema: return InstanceConfigUnknownAgent case a.InstanceConfigETag == want: return InstanceConfigDelivered default: return InstanceConfigStale } } // LatestConfigAck reads the freshest ack naming host, across every box that // has written one, and nil (with no error) when nothing has. a route can // legitimately have acks from two boxes at once across a migration, and the // freshest is the one describing what is running now. // // it lists the whole ack prefix, which is the same limit relayConfigAcks // accepts (internal/console/relayops.go): there is no per-host box list to // narrow a listing by. the Gets are narrowed, though. only keys whose // basename is this host are read. func LatestConfigAck(ctx context.Context, s release.Store, host string) (*ConfigAck, error) { keys, err := s.List(ctx, ConfigAckPrefix) if err != nil { return nil, err } var best *ConfigAck for _, key := range keys { if !strings.HasSuffix(key, "/"+host+".json") { continue } body, _, err := s.Get(ctx, key) if err != nil { if errors.Is(err, os.ErrNotExist) { continue } return nil, err } var a ConfigAck if err := json.Unmarshal(body, &a); err != nil { // one unreadable object must not hide the box that answered // properly, and there is nothing in it to report either way. log.Printf("config: config ack %s: %v", key, err) continue } if best == nil || a.At.After(best.At) { latest := a best = &latest } } return best, nil } // WriteConfigAck writes ack, get-then-put with the CAS etag of whatever is // there now — "" (create) only when nothing is. An ack is rewritten on // every sync that changes delivery, unlike release.Store.Put's create-only // contract, so this can't just Put with "": the object almost always // already exists. Retries once on release.ErrCASConflict, the same shape as // relay.WriteHeartbeat (internal/relay/authority.go) — a conflict here can // only be this route's own previous ack, since ConfigAckKey is unique per // (box, host) and nothing else writes it. func WriteConfigAck(ctx context.Context, s release.Store, ack ConfigAck) error { body, err := json.Marshal(ack) if err != nil { return err } key := ConfigAckKey(ack.Box, ack.Host) var lastErr error for attempt := 0; attempt < 2; attempt++ { etag := "" _, curEtag, err := s.Get(ctx, key) switch { case errors.Is(err, os.ErrNotExist): // no existing record: create-only Put below. case err != nil: return err default: etag = curEtag } if _, err := s.Put(ctx, key, body, etag); err != nil { lastErr = err if errors.Is(err, release.ErrCASConflict) { continue } return err } return nil } return lastErr } // carlosRelayVar is the well-known var name relay enable's config write // (spec §5 step 5) uses — appconfig.Apply, same as any other app var. const carlosRelayVar = "CARLOS_RELAY" // relaySuffixesFromRelayVar decrypts m's CARLOS_RELAY var, if it has one, // and parses it with relay.ParseConfig into a region -> username_suffix // map. Reads the decrypted manifest var directly — the same "env" vs // "secret" decoding RenderEnv applies to every var — never the rendered // EnvironmentFile bytes: CARLOS_RELAY is exactly as likely to be a sealed // secret entry as any other var, and re-parsing rendered KEY=value lines // would just repeat, worse, the decryption RenderEnv already performs. // // Returns nil, nil when the manifest carries no CARLOS_RELAY entry at all // — an ordinary route the relay was never enabled for, not an error. func relaySuffixesFromRelayVar(m *Manifest, priv *ecdh.PrivateKey, b Binding) (map[string]string, error) { v, ok := m.Vars[carlosRelayVar] if !ok { return nil, nil } var val string switch v.Kind { case "env": val = v.Value case "secret": if priv == nil { return nil, fmt.Errorf("var %s: secret entry but no decryption key was supplied", carlosRelayVar) } plain, err := Open(priv, b, v.Envelope) if err != nil { return nil, fmt.Errorf("decrypt %s: %w", carlosRelayVar, err) } val = string(plain) default: return nil, fmt.Errorf("var %s: unknown kind %q", carlosRelayVar, v.Kind) } entries, err := relay.ParseConfig(val) if err != nil { return nil, fmt.Errorf("%s: %w", carlosRelayVar, err) } suffixes := make(map[string]string, len(entries)) for _, e := range entries { suffixes[e.Region] = e.UsernameSuffix } return suffixes, nil } // shapeOf names which of spec §5's three delivery shapes rt is: "unit" for // a carlos-app@ unit tenant, "socket-exec" for an activator-spawned // route the platform bounces over a unix socket, "tcp-exec-file" for a // TCP-address route belonging to a process this platform does not own (the // config lands as a file; adoption is a member acknowledgement instead of // a restart). Shared by syncHost and syncExecHost so the two never name a // shape differently for the same route. func shapeOf(rt registry.Route) string { switch { case rt.Backing == registry.BackingUnit: return "unit" case rt.Socket != "": return "socket-exec" default: return "tcp-exec-file" } }