package activator // The instance key (region HA request L21; rastrillo/rastrillo #66). An app // on CARLOS seals its data with a key. Before this file the app minted that // key itself after it started, so a box that died before the key reached a // backup left a database nothing could unseal. The platform now mints the // key on the box, into the instance's own .env as // CARLOS_INSTANCE_KEY, before the app's first start, and never rewrites it. // rastrillo's carlos.InstanceKey reads it verbatim when the app has no key // of its own. // // The rules, each a ruling in // docs/superpowers/plans/2026-10-10-instance-key-provision.md: // - create-only: a key already in the file is never rewritten, whatever // its shape. A malformed one is reported and left in place, as // EnsureAdminToken does with a malformed token; // - the decision is made on the effective env (instancekey.Effective, // the rule mergeEnv applies): a key the rendered configuration // supplies is the app's key and nothing is minted; an empty own value // is no key; // - nothing is minted until the row's configuration is here. The files // say what this box has delivered, not what the configuration sets: a // first wake can land before the config sync's first render, and a // key minted then would shadow the configured one for good once it // arrived. So before it mints, the ensure asks the bucket // (ExecBackend.ConfiguredKey, the replica gate's reader) whether the // configuration sets a key now; if it does, or the bucket cannot say, // the launch is deferred (ErrInstanceKeyDeferred) and retried once the // sync has delivered the file; // - durable every time: before an ensure reports success the file is // 0600 (fchmod on its descriptor), the file is fsynced and so is its // directory, when the key was already there too. A sync that failed // once is redone by the next ensure, never skipped because the bytes // are already there. The known limit: once fsync has returned EIO, // Linux may mark the pages clean, so a later fsync's success does not // prove the bytes reached the disk. The retry narrows the gap; it // cannot close it, and no create-only design can; // - a mint that fails part way through its append is undone: the file // is truncated back to its length before the append and fsynced, so a // short key line never survives to be read as present. Those bytes // were never reported, so removing them rewrites no key; // - the value is never logged, never put in an error, and never leaves // the file except in the launch environment; // - a flat row's own file is opened O_NOFOLLOW on every path, read and // append alike (openOwnEnvNoFollow): a symlink there is refused, never // followed into a file a same-uid sibling chose. A nested row's file // gets the same from instancefs. import ( "context" "crypto/rand" "encoding/base64" "errors" "fmt" "io" "os" "path/filepath" "strings" "time" "github.com/carlosframework/platform/internal/instancefs" "github.com/carlosframework/platform/internal/instancekey" "github.com/carlosframework/platform/internal/registry" ) // InstanceKeyVar is the variable the key is delivered as. const InstanceKeyVar = instancekey.Var // instanceKeyBytes is the key's length before encoding: 32 CSPRNG bytes, // standard base64 with padding, 44 characters. rastrillo uses the exact // bytes the env file yields, so this encoding is fixed for the life of // every key minted with it; changing it is a rotation. const instanceKeyBytes = 32 // InstanceKeyOutcome is what one ensure did. Its zero value is // InstanceKeyFailed, what every ensure answers beside an error, so a // caller that reads the outcome before the error can never count a // refused row as anything. type InstanceKeyOutcome int const ( // InstanceKeyFailed: the ensure returned an error and reports nothing. // It refused the row (fenced, conflicting, a configuration not yet // delivered) or failed part way (an open, a write, a sync); the error // says which. Nothing it may have written is reported as durable. InstanceKeyFailed InstanceKeyOutcome = iota // InstanceKeyPresent: the own env file already carried a key. No byte // was written; the file was made durable. InstanceKeyPresent // InstanceKeyMinted: a key was appended and made durable. InstanceKeyMinted // InstanceKeyFromConfig: the rendered configuration supplies the key. // Nothing was written. InstanceKeyFromConfig // InstanceKeyNoEnvFile: the backfill found no own env file, so the host // has never been started here. Nothing was written; its first start // mints the key. InstanceKeyNoEnvFile // InstanceKeyReplica: a replica row (#162). A replica is never minted // a key: it may hold only its home's (ruling 26, replicaKeyGate). InstanceKeyReplica ) func (o InstanceKeyOutcome) String() string { switch o { case InstanceKeyFailed: return "failed" case InstanceKeyPresent: return "present" case InstanceKeyMinted: return "minted" case InstanceKeyFromConfig: return "from-config" case InstanceKeyNoEnvFile: return "no-env-file" case InstanceKeyReplica: return "replica" } return fmt.Sprintf("outcome(%d)", int(o)) } // instanceKeySync is the durability step's two syncs, a seam so a test can // fail each one. Read and replaced only under adminTokenMu. var instanceKeySync = struct { file func(*os.File) error dir func(string) error }{file: (*os.File).Sync, dir: syncDir} // syncDir fsyncs a directory, so an entry created in it survives a power // cut. func syncDir(dir string) error { d, err := os.Open(dir) if err != nil { return err } err = d.Sync() if cerr := d.Close(); err == nil { err = cerr } return err } // instanceKeyWrite is the mint's append, a seam so a test can make it fail // part way (a short write, ENOSPC). Read and replaced only under // adminTokenMu. var instanceKeyWrite = func(f *os.File, b []byte) (int, error) { return f.Write(b) } // instanceKeyHooks are the three test seams on the launch path's hot path, // all nil in production; the sync and write seams above are what the // durability tests need and stay on their own. beforeOpen runs under // adminTokenMu after the read and the decision and before the ensure // opens the file to append or settle, so a test can change the file // between the two and pin the open's own symlink refusal, not only the // read's. betweenPhases runs after the ensure's first read (and its // bucket read, if any) and before it takes adminTokenMu, so a test can // change the files between the two phases as a concurrent writer can. // replicaGate runs at the start of every replica gate, after the env was // read, so a test can change the env files between the read and the // gate, as a concurrent config sync can. var instanceKeyHooks struct { beforeOpen func(path string) betweenPhases func(path string) replicaGate func() } func mintInstanceKey() (string, error) { raw := make([]byte, instanceKeyBytes) if _, err := rand.Read(raw); err != nil { return "", err } return base64.StdEncoding.EncodeToString(raw), nil } // wellFormedInstanceKey reports whether key has the shape this file mints. // A key that fails it is not necessarily broken (an operator may have // written one by hand), which is why a failure is a warning, never a // rotation. func wellFormedInstanceKey(key string) bool { if len(key) != base64.StdEncoding.EncodedLen(instanceKeyBytes) { return false } raw, err := base64.StdEncoding.Strict().DecodeString(key) return err == nil && len(raw) == instanceKeyBytes } // configEnvName is the rendered configuration's file name beside an own env // file: .env's sibling .config.env, the pair pathsFor names. func configEnvName(envName string) string { return strings.TrimSuffix(envName, ".env") + ".config.env" } // ErrInstanceKeyDeferred is a first start held back because the row's // configuration is not here yet: the bucket's configuration sets // CARLOS_INSTANCE_KEY and this box has not delivered it, or the bucket // could not be read to tell. Nothing is minted (ruling 5: a configured key // is the app's, and one minted while it was on its way would shadow it for // good). The launch fails and is retried: the config sync's next pass // delivers the file, and the next start finds the key configured. var ErrInstanceKeyDeferred = errors.New("activator: instance key not minted: the instance's configuration is not delivered") // configReadTimeout bounds each read of the bucket's configuration, by // the mint and by the replica gate. const configReadTimeout = 10 * time.Second // configuredKeyReader answers, for one row, the SHA-256 of the // CARLOS_INSTANCE_KEY the bucket's configuration renders now, "" for // none. nil is a box with no configuration source. type configuredKeyReader func(context.Context) (string, error) // configuredKeyFor is b.ConfiguredKey for rt, each read bounded by // configReadTimeout; nil when the backend has no reader. func (b *ExecBackend) configuredKeyFor(rt registry.Route) configuredKeyReader { if b.ConfiguredKey == nil { return nil } return func(ctx context.Context) (string, error) { ctx, cancel := context.WithTimeout(ctx, configReadTimeout) defer cancel() return b.ConfiguredKey(ctx, rt) } } // ensureInstanceKeyNoConfig makes sure path (an instance's own .env // on the flat layout) carries CARLOS_INSTANCE_KEY, creating the file if // needed, for a row whose configuration can set no key: there is no // reader, so nothing is asked before the mint. Tests use it; every launch // path goes through ensureInstanceKeyFor, which asks the bucket first. func ensureInstanceKeyNoConfig(path string) (InstanceKeyOutcome, error) { return ensureInstanceKey(context.Background(), filepath.Dir(path), filepath.Base(path), false, true, nil) } // ensureInstanceKeyBeneathNoConfig is ensureInstanceKeyNoConfig for a // nested row's own env file, read and written through instancefs because // the directory is tenant-writable (isolated-exec design §3.2). func ensureInstanceKeyBeneathNoConfig(dir, name string) (InstanceKeyOutcome, error) { return ensureInstanceKey(context.Background(), dir, name, true, true, nil) } // ensureInstanceKeyFor is the launch paths' ensure for rt: none for a // replica row, the nested body for a nested row, the flat one otherwise. // ctx bounds the one bucket read a mint makes; a launch with none passes // context.Background(). // // A fenced or conflicting row is refused and nothing is written, as // BackfillInstanceKey refuses it. Every launch dispatcher (Start, // StartGen, StartStaged, Seed) refuses those rows already, but on its own // classification: the launch body classifies again in pathsFor, and p is // that second answer. A layout journal written between the two (a // migration starting mid-wake) makes p fenced, with the flat directory as // its dataDir, and minting there would write to a half-migrated host // (isolated-exec design §8.2). The refusal is decided on p, the paths the // mint would use. Every caller names the host around the error ("instance // key for : ..."), so the refusal does not. func (b *ExecBackend) ensureInstanceKeyFor(ctx context.Context, rt registry.Route, p instancePaths) (InstanceKeyOutcome, error) { return b.ensureInstanceKeyAt(ctx, rt, p, true, "minted") } // BackfillInstanceKey is the startup backfill's step for one row: a key is // appended only to an own env file that already exists. A host with no file // has never been started on this box, and its first start mints the key; // creating a file here could also occupy the path a move's install expects // to be empty (moveslandinstall.go placeStaged). A row this backend would // refuse to start (fenced, conflict) is an error and nothing is written. // The agent's backfill names the host when it logs the error, so the // refusal does not. func (b *ExecBackend) BackfillInstanceKey(ctx context.Context, rt registry.Route) (InstanceKeyOutcome, error) { return b.ensureInstanceKeyAt(ctx, rt, b.pathsFor(rt), false, "backfilled") } // ensureInstanceKeyAt is the two entry points' one guard: a replica row is // never minted; a fenced or conflicting row is refused on p, the paths the // mint would use, with verb naming which caller refused ("not minted", // "not backfilled"); every other row is ensureInstanceKey on p's layout, // with the bucket reader for rt. func (b *ExecBackend) ensureInstanceKeyAt(ctx context.Context, rt registry.Route, p instancePaths, create bool, verb string) (InstanceKeyOutcome, error) { if rt.Replica { return InstanceKeyReplica, nil } switch p.layout { case LayoutFenced, LayoutConflict: return InstanceKeyFailed, fmt.Errorf("not %s: %s", verb, p.reason) } return ensureInstanceKey(ctx, p.dataDir, rt.Host+".env", p.layout == LayoutNested, create, b.configuredKeyFor(rt)) } // openOwnEnv opens an own env file, .env under dir: through // instancefs when beneath (a nested row's tenant-writable directory), else // through openOwnEnvNoFollow. Both refuse a link or a FIFO in the open. // The one place the two layouts' opens meet. func openOwnEnv(dir, name string, beneath bool, flags int, perm os.FileMode) (*os.File, error) { if beneath { return instancefs.OpenBeneath(dir, name, flags, perm) } return openOwnEnvNoFollow(filepath.Join(dir, name), flags, perm) } // openInstanceEnv opens an own env file for append. Only a create passes // a mode: openat2 refuses a non-zero mode without O_CREAT (EINVAL), and an // existing file's mode is set by the fchmod that follows anyway. func openInstanceEnv(dir, name string, beneath, create bool) (*os.File, error) { flags, perm := os.O_APPEND|os.O_WRONLY, os.FileMode(0) if create { flags, perm = flags|os.O_CREATE, 0o600 } return openOwnEnv(dir, name, beneath, flags, perm) } // errOwnEnvSymlink is a flat row's own env file found to be a symlink. The // ensure refuses it on every path (mint, existing key, backfill, the claim // settle): following it would append to, chmod and fsync a file a same-uid // sibling chose (spec §7.2). var errOwnEnvSymlink = errors.New("activator: the instance env file is a symlink, refused") // errOwnEnvNotRegular is a flat row's own env file, or its rendered // configuration, found to be something other than a regular file: a FIFO, // which a blocking open would wait on for a writer that never comes (the // startup backfill holds adminTokenMu while it reads, so one planted FIFO // would keep every instance on the box from waking), or a device. Refused // on every path, as the link is. var errOwnEnvNotRegular = errors.New("activator: the instance env file is not a regular file, refused") // ownEnvReadLimit caps a flat read of an own env file or its rendered // configuration, as instancefs.ReadBeneathLimit caps the nested one: no // read under adminTokenMu is unbounded. It is instancekey's one cap, the // one relocation's reads of the same files use. const ownEnvReadLimit = instancekey.EnvReadLimit // errOwnEnvTooLarge is a flat env file over ownEnvReadLimit. var errOwnEnvTooLarge = errors.New("activator: the instance env file is larger than the limit, refused") // readOwnEnv reads a flat row's own env file, or its rendered // configuration, through openOwnEnvNoFollow, so the read refuses a link // or a FIFO exactly as the append does, and never reads more than // ownEnvReadLimit. func readOwnEnv(path string) ([]byte, error) { f, err := openOwnEnvNoFollow(path, os.O_RDONLY, 0) if err != nil { return nil, err } defer f.Close() raw, err := io.ReadAll(io.LimitReader(f, ownEnvReadLimit+1)) if err != nil { return nil, err } if len(raw) > ownEnvReadLimit { return nil, errOwnEnvTooLarge } return raw, nil } // settleInstanceEnv is the durability step every successful ensure ends // with: 0600 on the descriptor, then syncInstanceEnv. Caller holds // adminTokenMu. func settleInstanceEnv(f *os.File, dir, path string) error { if err := f.Chmod(0o600); err != nil { return fmt.Errorf("chmod %s: %w", path, err) } return syncInstanceEnv(f, dir, path) } // syncInstanceEnv fsyncs an own env file, then its directory: the two // syncs every durability step ends with, the claim's settle included. // Caller holds adminTokenMu. func syncInstanceEnv(f *os.File, dir, path string) error { if err := instanceKeySync.file(f); err != nil { return fmt.Errorf("sync %s: %w", path, err) } if err := instanceKeySync.dir(dir); err != nil { return fmt.Errorf("sync %s: %w", dir, err) } return nil } // readInstanceEnvPair reads an own env file and its rendered configuration // the same guarded way on both layouts: through instancefs on the nested // one, and on the flat one through readOwnEnv, which refuses a link or a // FIFO at either name and caps the read. exists is false when the own file // is absent (raw nil); an absent configuration is nil cfg and no error. func readInstanceEnvPair(dir, name string, beneath bool) (raw, cfg []byte, exists bool, err error) { read := func(n string) ([]byte, error) { if beneath { return instancefs.ReadBeneathLimit(dir, n, ownEnvReadLimit) } return readOwnEnv(filepath.Join(dir, n)) } raw, err = read(name) switch { case errors.Is(err, os.ErrNotExist): raw = nil case err != nil: return nil, nil, false, fmt.Errorf("read %s: %w", filepath.Join(dir, name), err) default: exists = true } cfg, err = read(configEnvName(name)) if err != nil && !errors.Is(err, os.ErrNotExist) { return nil, nil, false, fmt.Errorf("read %s: %w", filepath.Join(dir, configEnvName(name)), err) } return raw, cfg, exists, nil } // bucketVerdict is what the bucket said before the lock was taken: asked // is whether a mint looked next on that read and the bucket was consulted; // deferral is its answer when the mint must wait (a configured key not yet // delivered, or a read that failed), nil when it may go ahead. type bucketVerdict struct { asked bool deferral error } // ensureInstanceKey is the shared body. It holds adminTokenMu, the lock // every mint into .env takes, so a key append and a token append can // never interleave. create false is the backfill: an absent file is // InstanceKeyNoEnvFile and nothing is created. // // The bucket is read in two phases, so a slow or unreachable bucket for // one row never holds the lock every other row's mint waits on. Outside // the lock: read the files, and if a mint looks next, ask the bucket // (configured; the one read a row ever costs). Under the lock: read the // files again and decide from that read alone. A key that appeared in // either file since is present or configured, and nothing is minted; a // mint goes ahead only when the locked read still finds no key and the // bucket was asked and said none. A mint that looks next under the lock // without a verdict (the first read saw a key that has since gone) defers // rather than mint on nothing, and the next attempt asks. Create-only, // fail-closed and defer-on-unknown hold on every path. func ensureInstanceKey(ctx context.Context, dir, name string, beneath, create bool, configured configuredKeyReader) (InstanceKeyOutcome, error) { path := filepath.Join(dir, name) // Phase one, outside the lock. Nothing delivered here sets a key. The // configuration may still: a first wake can run before the config // sync's first render of .config.env (the fast-converge pass // reconciles the row ready and syncs config last), and a key minted // in that gap would shadow the configured one for good once it // arrived (Effective: the own file wins). So ask the bucket whether // the configuration sets a key now. One it sets is not here yet: // defer, and the next start finds it delivered. A bucket that cannot // be read cannot say, and defers the same way. With no reader there // is no configuration source, and nothing can be on its way. A read // error here is left for the locked read to report. var pre bucketVerdict if configured != nil { if raw, cfg, exists, err := readInstanceEnvPair(dir, name, beneath); err == nil && (exists || create) { if _, layer := instancekey.Effective(parseEnvBytes(cfg), parseEnvBytes(raw)); layer == instancekey.LayerNone { pre.asked = true digest, err := configured(ctx) switch { case err != nil: pre.deferral = fmt.Errorf("%w: its configuration could not be read: %w", ErrInstanceKeyDeferred, err) case digest != "": pre.deferral = fmt.Errorf("%w: its configuration sets %s and this box has not delivered it yet (the next config sync does, and the next start finds it)", ErrInstanceKeyDeferred, InstanceKeyVar) } } } } if instanceKeyHooks.betweenPhases != nil { instanceKeyHooks.betweenPhases(path) } // Phase two, under the lock: the decision, from this read only. adminTokenMu.Lock() defer adminTokenMu.Unlock() raw, cfg, exists, err := readInstanceEnvPair(dir, name, beneath) if err != nil { return InstanceKeyFailed, err } if !exists && !create { return InstanceKeyNoEnvFile, nil } // The decision is the effective env's: the key the launch will hand // the app, by the same rule mergeEnv applies. key, layer := instancekey.Effective(parseEnvBytes(cfg), parseEnvBytes(raw)) switch layer { case instancekey.LayerConfig: logPerFile("key-config", path, fmt.Sprintf("activator: %s is set by the app's configuration for %s: the platform mints none while it is", InstanceKeyVar, path)) return InstanceKeyFromConfig, nil case instancekey.LayerOwn: if !wellFormedInstanceKey(key) { logPerFile("key-shape", path, fmt.Sprintf("activator: %s in %s is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it", InstanceKeyVar, path)) } // Present is not a no-op: a mint whose sync failed left these // bytes behind, and this call is the retry that makes them // durable before anything is started with them. if instanceKeyHooks.beforeOpen != nil { instanceKeyHooks.beforeOpen(path) } f, err := openInstanceEnv(dir, name, beneath, false) if err != nil { return InstanceKeyFailed, fmt.Errorf("open %s: %w", path, err) } defer f.Close() if err := settleInstanceEnv(f, dir, path); err != nil { return InstanceKeyFailed, err } return InstanceKeyPresent, nil } // A mint is next. It goes ahead only on the bucket's word from phase // one; without it (the files changed between the two reads) nothing // is known, and the mint defers rather than go on nothing. if configured != nil { switch { case !pre.asked: return InstanceKeyFailed, fmt.Errorf("%w: its env files changed while the configuration was being read (retried at the next start)", ErrInstanceKeyDeferred) case pre.deferral != nil: return InstanceKeyFailed, pre.deferral } } minted, err := mintInstanceKey() if err != nil { return InstanceKeyFailed, fmt.Errorf("mint instance key for %s: %w", path, err) } if instanceKeyHooks.beforeOpen != nil { instanceKeyHooks.beforeOpen(path) } f, err := openInstanceEnv(dir, name, beneath, create) if errors.Is(err, os.ErrNotExist) && !create { return InstanceKeyNoEnvFile, nil // removed since the read: nothing to backfill } if err != nil { return InstanceKeyFailed, fmt.Errorf("open %s: %w", path, err) } defer f.Close() // The length the append starts from, taken from the open descriptor: // what a failed append is undone to. Under adminTokenMu it is // len(raw); reading it here also keeps any byte another process // appended since the read. st, err := f.Stat() if err != nil { return InstanceKeyFailed, fmt.Errorf("stat %s: %w", path, err) } prefix := "" if len(raw) > 0 && raw[len(raw)-1] != '\n' { prefix = "\n" } line := []byte(prefix + InstanceKeyVar + "=" + minted + "\n") if n, err := instanceKeyWrite(f, line); err != nil || n != len(line) { if err == nil { err = io.ErrShortWrite } return InstanceKeyFailed, undoInstanceKeyAppend(f, st.Size(), path, err) } if err := settleInstanceEnv(f, dir, path); err != nil { return InstanceKeyFailed, err } return InstanceKeyMinted, nil } // undoInstanceKeyAppend removes a failed append's bytes: it truncates f // back to size, the length before the append, and fsyncs, so a short key // line can never survive to be read as present. Nothing past size was // ever reported to anyone. Returns the write error, joined with any // failure to undo it. Caller holds adminTokenMu. func undoInstanceKeyAppend(f *os.File, size int64, path string, werr error) error { err := fmt.Errorf("write %s: %w", path, werr) if terr := f.Truncate(size); terr != nil { return errors.Join(err, fmt.Errorf("undo the partial write to %s: %w", path, terr)) } if serr := instanceKeySync.file(f); serr != nil { return errors.Join(err, fmt.Errorf("sync %s after undoing the partial write: %w", path, serr)) } return err } // errInstanceEnvNeedsWrite is settleClaimInstanceEnv's answer for a file // whose mode is not 0600: making it so is a write, which the claim path // does only under its guard. var errInstanceEnvNeedsWrite = errors.New("activator: the instance env file needs its mode set") // settleClaimInstanceEnv is the claim launch source's durability step for // an env file that already carries what the launch needs: it makes the // file and its directory durable and writes nothing. fsync changes no // byte, mode or name, so it is not a host write and runs without the // claim's guard. A file that is not 0600 is errInstanceEnvNeedsWrite, and // the caller takes the guarded path, whose ensure sets the mode when the // own file holds the key (and leaves it when the key is configured or the // row is a replica: EnsureClaimAdminToken). func (b *ExecBackend) settleClaimInstanceEnv(rt registry.Route) error { p := b.pathsFor(rt) adminTokenMu.Lock() defer adminTokenMu.Unlock() f, err := openOwnEnv(p.dataDir, rt.Host+".env", p.layout == LayoutNested, os.O_RDONLY, 0) if err != nil { return fmt.Errorf("open %s: %w", p.envFile, err) } defer f.Close() st, err := f.Stat() if err != nil { return fmt.Errorf("stat %s: %w", p.envFile, err) } if st.Mode().Perm() != 0o600 { return errInstanceEnvNeedsWrite } return syncInstanceEnv(f, p.dataDir, p.envFile) }