// The NARROWED restart signal (discussion #16): a request to cycle ONE route, // or the routes reading ONE config environment, rather than every instance an // app owns. // // Why a second object rather than a field on restart-gen.json. A box acts on // that object's ETAG MOVING — it never reads the content — so a selector // written INTO it would be invisible to every box running today's binary, and // those boxes would read the movement the only way they know how: restart the // whole app. For titogo that is nineteen production routes cycled to deliver // one credential to one of them. The rule this repo already has for wire // shapes ("anything written before your change must still parse") has a twin // that this file is built on: an old reader that cannot understand a NARROWED // request must do NOTHING, never something wider than was asked for. // // A sibling key gets that for free. Nothing that exists today reads // restart-scope.json, so: // // - OLD box, NEW (narrowed) request — never looks at the key, sees no // movement on restart-gen.json, cycles nothing. The request simply does // not reach it. The console says which routes it addressed and on which // box's registry it resolved them, so an operator whose fleet still has // an old box can see that it is one box's answer. // - NEW box, OLD (app-wide) request — restart-gen.json moves exactly as it // always did and this file takes no part. // - NEW box, FUTURE request kind — an entry whose `kind` this binary does // not know is SKIPPED, not widened (see RestartScope.Selects). A selector // nobody here understands addresses nothing. // // Bucket layout addition: // // apps///restart-scope.json CAS-guarded, schema-1 request log // // It is a LOG rather than a single generation, which is the one place this // shape costs more than restart-gen.json's. Movement is enough to carry "some // restart was asked for"; it is not enough to carry WHICH routes, because two // requests inside one 2s poll window collapse into one etag move and the // earlier selector would be lost — silently, having told its caller // "requested". Recording each request with an id, and remembering the last id // acted on, makes a coalesced pair two requests again. // // The content is TRUSTED here in a way restart-gen.json's is not, so it is // worth saying exactly how far that goes: an entry can only ever NARROW. The // box matches entries against routes it already serves for that account and // app — the account, the app and the route set come from the box's own // registry, never from this object — so the worst a forged or corrupted entry // can do is name a selector matching every route of one app, which is // precisely what the unsigned app-wide object next to it already permits. // Nothing about what runs, or about which account owns what, is decided here. package release import ( "context" "crypto/rand" "encoding/hex" "encoding/json" "errors" "fmt" "io" "os" "strings" "time" "github.com/carlosframework/platform/internal/registry" ) // The selector kinds. Only Host and Environment are ever written into the // object; App names an app-wide request, which is restart-gen.json's job and // travels no further than the HTTP door and the CLI that talks to it. It // lives here so the three of them spell it the same way. const ( RestartScopeHost = "host" RestartScopeEnvironment = "environment" RestartScopeApp = "app" // RestartScopeRetry is `carlos instances retry ` (zero-downtime spec // §8.4): Value is the host whose failure hold to clear. It is NOT a // restart. Selects does not understand it, so a box built before it // steps over it, and a box built after it hands it to the rollout manager // instead of cycling anything (internal/host/restartscope.go). Unlike a // restart, a retry lifts a restriction, so the file header's narrowing // argument does not cover it: it acts only on a root-signed grant in // Signed (HostRetryGrant, spec §11), and a box drops one without. // // It is the kind of every entry in the host-retry log (AppendHostRetry, // hostretrylog.go), and no longer written into this one // (ValidRestartScope): a console built before Signed strips it from every // entry when it re-marshals this log. A box still verifies and applies a // signed retry an earlier console left here, with the same checks. RestartScopeRetry = "retry" ) // restartScopeKeep bounds the log. Entries older than the last few dozen are // dropped on the next append: a box that has been away long enough to miss // them re-baselines (see PendingSince) rather than replaying a week of // requests, and the object stays a small, cheap read. const restartScopeKeep = 32 // RestartScopeNone is the mark a box records for "I have looked and there is // nothing here" — the object is absent, or present with an empty log. The two // collapse deliberately: an object that appears carrying its first request // moves the mark from this sentinel to that request's id, which is movement, // which is exactly what it is. // // The NUL byte is what makes it safe as a sentinel — no request id contains // one, since ids are hex — and it is the same trick the box's own absent // marker uses for etags. const RestartScopeNone = "\x00none" // RestartScope is one selector: what to narrow a restart to. // // Kind is compared, never interpreted — an unknown one addresses nothing. // Value is matched against a registry column and never becomes a bucket key, // a filesystem path or a shell word; the strict shape checks belong at the // door that accepts it from a person, where a typo can still be answered with // what actually exists. type RestartScope struct { Kind string `json:"kind"` Value string `json:"value"` // Signed is a retry's root-signed grant (HostRetryGrant, spec §11): a // box clears a hold only on a grant that verifies against this entry's // account, app and host, so a box that can write this log cannot forge // a retry (Astra impl-6 finding 5). Only a retry carries it, and only in // the host-retry log is one written now (AppendHostRetry); a legacy // entry in the restart-scope log may still carry one. It is additive: // an older box's lax parse steps over it, and an unsigned retry is // dropped by every box that checks. Signed string `json:"signed,omitempty"` } // RestartRequest is one recorded request. ID is what a box remembers; nothing // is derived from it beyond "have I acted on this one". type RestartRequest struct { ID string `json:"id"` RequestedAt string `json:"requested_at"` Scope RestartScope `json:"scope"` } // ScopedRestartGeneration is the object: apps///restart-scope.json. // Requests are OLDEST FIRST, so the newest is the last element and a box's // "what have I not acted on" is a suffix. type ScopedRestartGeneration struct { Schema int `json:"schema"` // 1 Requests []RestartRequest `json:"requests"` } // RestartScopeKey is the bucket key of app's narrowed-request log, derived // from AppPrefix like every other per-app key. func RestartScopeKey(acct, app string) string { return AppPrefix(acct, app) + "restart-scope.json" } // Selects reports whether rt is addressed by this scope, and whether this // binary understands the scope's kind at all. // // The second return is the forward-compatibility hinge, and the reason this // is two bools rather than one. A selector kind added after this binary was // built must address NOTHING here — not everything — so a caller that gets // understood=false skips the whole request rather than falling back to the // app. Widening on ignorance is the failure this whole file exists to avoid. // // The caller has already narrowed to the right account and app: this decides // only which of THOSE routes the selector names. func (sc RestartScope) Selects(rt registry.Route) (selected, understood bool) { switch sc.Kind { case RestartScopeHost: return rt.Host == sc.Value, true case RestartScopeEnvironment: // An environment selector never matches the unbound routes: "" is not // an environment name, it is the absence of one, and `--environment ""` // cannot be typed. A request to cycle what reads the DEFAULT bundle is // an app-wide request, and has the app-wide object. return sc.Value != "" && rt.Environment == sc.Value, true } return false, false } // ValidRestartScope refuses a selector this package would write but no box // could act on. It is the last gate before the bucket, not the first: the // door that took the value from a person has already checked its shape and // answered a typo with what exists. // // A retry, and any signature, is refused: retries ride the host-retry log // (AppendHostRetry), where no console older than it can strip a grant's // signature by re-marshalling the log. func ValidRestartScope(sc RestartScope) error { switch sc.Kind { case RestartScopeHost, RestartScopeEnvironment: case RestartScopeRetry: return fmt.Errorf("restart scope: a retry is written to the host-retry log, not this one") default: return fmt.Errorf("restart scope: unknown kind %q", sc.Kind) } if err := validScopeValue(sc.Kind, sc.Value); err != nil { return err } if sc.Signed != "" { return fmt.Errorf("restart scope: %s carries a signature, which only a retry may", sc.Kind) } return nil } // validScopeValue is the shape check every entry's Value passes: present, // at most a hostname's length, and no NUL or line break. func validScopeValue(kind, value string) error { if value == "" { return fmt.Errorf("restart scope: %s has no value", kind) } if len(value) > 253 || strings.ContainsAny(value, "\x00\n\r") { return fmt.Errorf("restart scope: %s value is not usable", kind) } return nil } // LoadRestartScopes reads the log. found=false means the object is absent, // which is the ordinary state of almost every app and is legal forever. // // A newer schema is an ERROR rather than a best-effort parse: a box that // cannot be sure what a request means must do nothing, and its caller records // nothing, so the request is picked up whenever the box is upgraded rather // than dropped or guessed at. func LoadRestartScopes(ctx context.Context, s Store, acct, app string) (*ScopedRestartGeneration, bool, error) { key := RestartScopeKey(acct, app) raw, _, err := s.Get(ctx, key) if errors.Is(err, os.ErrNotExist) { return &ScopedRestartGeneration{Schema: 1}, false, nil } if err != nil { return nil, false, err } g, err := parseRestartScopes(key, raw) if err != nil { return nil, false, err } return g, true, nil } func parseRestartScopes(key string, raw []byte) (*ScopedRestartGeneration, error) { var g ScopedRestartGeneration if err := json.Unmarshal(raw, &g); err != nil { return nil, fmt.Errorf("restart scopes at %s: %w", key, err) } if g.Schema > 1 { return nil, fmt.Errorf("restart scopes at %s: schema %d is newer than this binary understands — upgrade carlos", key, g.Schema) } return &g, nil } // Newest is the id of the last request in the log, or RestartScopeNone for an // empty one — the value a box records when it has nothing to act on. func (g *ScopedRestartGeneration) Newest() string { if g == nil || len(g.Requests) == 0 { return RestartScopeNone } return g.Requests[len(g.Requests)-1].ID } // PendingSince returns the requests recorded after mark — what a box that has // already acted up to mark still owes — and whether mark could be located at // all. // // found=false means the mark is an id that is no longer in the log: the box // has been away long enough for its position to roll off the end. There is no // honest answer to "what did I miss" then, and the two candidate guesses are // not equal — replaying the whole log would cycle routes for requests served // days ago, while re-baselining drops requests that are recoverable by typing // the verb again. Callers re-baseline; see the box's restart pass, which makes // the same choice everywhere else it is offered. // // RestartScopeNone as the mark is NOT an unknown position: it is a box that // looked and found nothing, so everything now in the log is new to it. func (g *ScopedRestartGeneration) PendingSince(mark string) (pending []RestartRequest, found bool) { if g == nil { return nil, true } if mark == RestartScopeNone { return g.Requests, true } for i, r := range g.Requests { if r.ID == mark { return g.Requests[i+1:], true } } if len(g.Requests) == 0 { // The log emptied under a mark that named a request — only possible if // the object was replaced wholesale. Unknown position, same answer. return nil, false } return nil, false } // AppendRestartScope records one narrowed request and returns it. // // A CAS conflict is RETRIED here, which is the opposite of TouchRestartGen's // treatment of the same error, and the difference is the whole point of this // object. There, any touch means "restart this app", so a racing writer has // already fired the exact signal this call exists to fire and a retry would // only manufacture a second restart. Here the other writer's request names a // DIFFERENT set of routes: dropping ours would answer "requested" for a // restart that will never happen. So we re-read and append again. // // The retry budget is small on purpose. This is a human-paced verb; a // contended object means something is wrong that another lap will not fix, // and the caller gets an error rather than a lie. func claimRawAppendRestartScope(ctx context.Context, s Store, acct, app string, scope RestartScope) (RestartRequest, error) { if !ValidAccount(acct) { return RestartRequest{}, fmt.Errorf("restart: invalid account %q", acct) } if !ValidApp(app) { return RestartRequest{}, fmt.Errorf("restart: invalid app %q", app) } if err := ValidRestartScope(scope); err != nil { return RestartRequest{}, err } return claimRawAppendRequestLog(ctx, s, RestartScopeKey(acct, app), scope, "restart") } // claimRawAppendRequestLog is the CAS append both request logs share (the // restart-scope log and the host-retry log): a fresh random id, the log // re-read and appended to on a conflict, at most restartScopeKeep entries // kept, and a log this binary cannot parse refused rather than // overwritten. verb names the request in the conflict error. The caller // has validated the account, the app and the scope. func claimRawAppendRequestLog(ctx context.Context, s Store, key string, scope RestartScope, verb string) (RestartRequest, error) { id := make([]byte, 16) if _, err := io.ReadFull(rand.Reader, id); err != nil { return RestartRequest{}, err } req := RestartRequest{ ID: hex.EncodeToString(id), RequestedAt: time.Now().UTC().Format(time.RFC3339), Scope: scope, } var lastErr error for attempt := 0; attempt < 3; attempt++ { raw, etag, err := s.Get(ctx, key) if err != nil && !errors.Is(err, os.ErrNotExist) { return RestartRequest{}, err } g := &ScopedRestartGeneration{Schema: 1} if err == nil { if g, err = parseRestartScopes(key, raw); err != nil { // Refusing beats overwriting: a log this binary cannot read is // one whose entries it would silently drop on the next write. return RestartRequest{}, err } } else { etag = "" // create-only } g.Schema = 1 g.Requests = append(g.Requests, req) if len(g.Requests) > restartScopeKeep { g.Requests = g.Requests[len(g.Requests)-restartScopeKeep:] } body, err := json.Marshal(g) if err != nil { return RestartRequest{}, err } if _, err := s.Put(ctx, key, body, etag); err != nil { if errors.Is(err, ErrCASConflict) { lastErr = err continue } return RestartRequest{}, err } return req, nil } return RestartRequest{}, fmt.Errorf("%s: %s is being written by someone else — nothing was requested; try again: %w", verb, key, lastErr) } func AppendRestartScope(ctx context.Context, s Store, acct, app string, scope RestartScope) (claimResult0 RestartRequest, claimResult1 error) { claimResult1 = WithAppMutation(ctx, s, acct, app, "release.AppendRestartScope", func(claimContext context.Context) error { claimResult0, claimResult1 = claimRawAppendRestartScope(claimContext, s, acct, app, scope) return claimResult1 }) return }