83 lines
3.9 KiB
Markdown
83 lines
3.9 KiB
Markdown
# Restart and crash recovery
|
|
|
|
## What each restart reconciles
|
|
|
|
The control plane reconciles on every start. Reconciliation is idempotent by design, so a restart
|
|
storm converges rather than accumulating.
|
|
|
|
```text
|
|
lifecycle incomplete operations roll back from their immutable snapshot, exactly once
|
|
migration interrupted cutovers are reported, never auto-resolved
|
|
recovery interrupted backups fail closed; interrupted restores need operator review
|
|
serving stale leases and residency are reconciled against live node state
|
|
observability current alert state is re-derived; history is retained
|
|
```
|
|
|
|
Migration is deliberately the exception. External alias truth cannot be inferred after a crash, so
|
|
an interrupted cutover stays in its intermediate stage and is reported as requiring reconciliation
|
|
through the typed adapter. Guessing here is how a platform ends up serving two embedding spaces.
|
|
|
|
## Restart storms
|
|
|
|
Four consecutive control-plane restarts under load produced no authoritative drift: models,
|
|
revisions, artifacts, artifact sets, projects, bindings, capability deployments, lifecycle
|
|
operations, migration plans and cutovers, compute nodes, node credentials, service clients and
|
|
backup sets were all identical before and after, and the registry answered on every start.
|
|
|
|
Audit events are expected to grow — reconciliation is itself an audited action — so the drift check
|
|
excludes them and the `audit_chain_intact` invariant covers their integrity instead.
|
|
|
|
## Node Agent restarts
|
|
|
|
A restarting agent keeps its persisted identity file, so it re-attaches to the same node with the
|
|
same accelerators, storage roots and history. It does not re-enrol unless it has lost its
|
|
credential, and enrolment revokes the previous credential rather than adding a second.
|
|
|
|
The publication and artifact loops share an enrolment lock. Without it they could enrol
|
|
concurrently from separate threads and burn a single-use token on two identities for the same
|
|
hardware.
|
|
|
|
## Database restarts
|
|
|
|
The control plane fails safely while PostgreSQL is unavailable: liveness stays up because it does
|
|
not touch the database, and any route that needs persistence returns an error rather than inventing
|
|
state. The connection pool recovers on its own — no control-plane restart is required — and no
|
|
duplicate audit sequence appears.
|
|
|
|
An error raised while the database is down carries no DSN, driver name, SQL or traceback. That is
|
|
verified directly, because a dependency failure is when a framework is most likely to leak
|
|
internals.
|
|
|
|
## Redis restarts
|
|
|
|
Redis holds queues and transient payloads, never authoritative state. An outage leaves the control
|
|
plane serving, and the return produces no duplicate serving job: idempotency keys are unique in the
|
|
database, not in the queue.
|
|
|
|
## Runtime Worker crashes
|
|
|
|
A worker crash during load, inference or unload produces a typed failure for the request, fences the
|
|
worker generation so a late result from a dead generation cannot be committed, and reconciles the
|
|
lease and residency. A crash loop raises `RUNTIME_CRASH_LOOP`.
|
|
|
|
## Interrupted backups and restores
|
|
|
|
A backup interrupted mid-write is moved to `FAILED` with `MANIFEST_INCOMPLETE` on the next start and
|
|
can never become restore eligible. A restore interrupted mid-phase becomes
|
|
`MANUAL_INTERVENTION_REQUIRED` rather than resuming silently; its journal records the phase it
|
|
reached.
|
|
|
|
A backup also cannot contain a record of itself as complete, since the dump is taken while its own
|
|
`BackupSet` row is still `CREATING`. A restored control plane therefore consistently reports the
|
|
backup it came from as not restore eligible.
|
|
|
|
## Checking after a restart
|
|
|
|
```sh
|
|
python scripts/m16_invariants.py --label "after restart"
|
|
curl -s localhost:8000/api/v1/health/ready
|
|
```
|
|
|
|
Fifteen invariants holding, readiness healthy, and no orphaned lease or queued job is what "the
|
|
restart reconciled" means. See `docs/operations/CHAOS_TESTING.md` for the fault matrix.
|