3.9 KiB
Restart and crash recovery
What each restart reconciles
The control plane reconciles on every start. Reconciliation is idempotent by design, so a restart storm converges rather than accumulating.
lifecycle incomplete operations roll back from their immutable snapshot, exactly once
migration interrupted cutovers are reported, never auto-resolved
recovery interrupted backups fail closed; interrupted restores need operator review
serving stale leases and residency are reconciled against live node state
observability current alert state is re-derived; history is retained
Migration is deliberately the exception. External alias truth cannot be inferred after a crash, so an interrupted cutover stays in its intermediate stage and is reported as requiring reconciliation through the typed adapter. Guessing here is how a platform ends up serving two embedding spaces.
Restart storms
Four consecutive control-plane restarts under load produced no authoritative drift: models, revisions, artifacts, artifact sets, projects, bindings, capability deployments, lifecycle operations, migration plans and cutovers, compute nodes, node credentials, service clients and backup sets were all identical before and after, and the registry answered on every start.
Audit events are expected to grow — reconciliation is itself an audited action — so the drift check
excludes them and the audit_chain_intact invariant covers their integrity instead.
Node Agent restarts
A restarting agent keeps its persisted identity file, so it re-attaches to the same node with the same accelerators, storage roots and history. It does not re-enrol unless it has lost its credential, and enrolment revokes the previous credential rather than adding a second.
The publication and artifact loops share an enrolment lock. Without it they could enrol concurrently from separate threads and burn a single-use token on two identities for the same hardware.
Database restarts
The control plane fails safely while PostgreSQL is unavailable: liveness stays up because it does not touch the database, and any route that needs persistence returns an error rather than inventing state. The connection pool recovers on its own — no control-plane restart is required — and no duplicate audit sequence appears.
An error raised while the database is down carries no DSN, driver name, SQL or traceback. That is verified directly, because a dependency failure is when a framework is most likely to leak internals.
Redis restarts
Redis holds queues and transient payloads, never authoritative state. An outage leaves the control plane serving, and the return produces no duplicate serving job: idempotency keys are unique in the database, not in the queue.
Runtime Worker crashes
A worker crash during load, inference or unload produces a typed failure for the request, fences the
worker generation so a late result from a dead generation cannot be committed, and reconciles the
lease and residency. A crash loop raises RUNTIME_CRASH_LOOP.
Interrupted backups and restores
A backup interrupted mid-write is moved to FAILED with MANIFEST_INCOMPLETE on the next start and
can never become restore eligible. A restore interrupted mid-phase becomes
MANUAL_INTERVENTION_REQUIRED rather than resuming silently; its journal records the phase it
reached.
A backup also cannot contain a record of itself as complete, since the dump is taken while its own
BackupSet row is still CREATING. A restored control plane therefore consistently reports the
backup it came from as not restore eligible.
Checking after a restart
python scripts/m16_invariants.py --label "after restart"
curl -s localhost:8000/api/v1/health/ready
Fifteen invariants holding, readiness healthy, and no orphaned lease or queued job is what "the
restart reconciled" means. See docs/operations/CHAOS_TESTING.md for the fault matrix.