Initial public ModelForge release

docs/operations/RESTART_RECOVERY.md

@@ -0,0 +1,82 @@
+# Restart and crash recovery
+
+## What each restart reconciles
+
+The control plane reconciles on every start. Reconciliation is idempotent by design, so a restart
+storm converges rather than accumulating.
+
+```text
+lifecycle     incomplete operations roll back from their immutable snapshot, exactly once
+migration     interrupted cutovers are reported, never auto-resolved
+recovery      interrupted backups fail closed; interrupted restores need operator review
+serving       stale leases and residency are reconciled against live node state
+observability current alert state is re-derived; history is retained
+```
+
+Migration is deliberately the exception. External alias truth cannot be inferred after a crash, so
+an interrupted cutover stays in its intermediate stage and is reported as requiring reconciliation
+through the typed adapter. Guessing here is how a platform ends up serving two embedding spaces.
+
+## Restart storms
+
+Four consecutive control-plane restarts under load produced no authoritative drift: models,
+revisions, artifacts, artifact sets, projects, bindings, capability deployments, lifecycle
+operations, migration plans and cutovers, compute nodes, node credentials, service clients and
+backup sets were all identical before and after, and the registry answered on every start.
+
+Audit events are expected to grow — reconciliation is itself an audited action — so the drift check
+excludes them and the `audit_chain_intact` invariant covers their integrity instead.
+
+## Node Agent restarts
+
+A restarting agent keeps its persisted identity file, so it re-attaches to the same node with the
+same accelerators, storage roots and history. It does not re-enrol unless it has lost its
+credential, and enrolment revokes the previous credential rather than adding a second.
+
+The publication and artifact loops share an enrolment lock. Without it they could enrol
+concurrently from separate threads and burn a single-use token on two identities for the same
+hardware.
+
+## Database restarts
+
+The control plane fails safely while PostgreSQL is unavailable: liveness stays up because it does
+not touch the database, and any route that needs persistence returns an error rather than inventing
+state. The connection pool recovers on its own — no control-plane restart is required — and no
+duplicate audit sequence appears.
+
+An error raised while the database is down carries no DSN, driver name, SQL or traceback. That is
+verified directly, because a dependency failure is when a framework is most likely to leak
+internals.
+
+## Redis restarts
+
+Redis holds queues and transient payloads, never authoritative state. An outage leaves the control
+plane serving, and the return produces no duplicate serving job: idempotency keys are unique in the
+database, not in the queue.
+
+## Runtime Worker crashes
+
+A worker crash during load, inference or unload produces a typed failure for the request, fences the
+worker generation so a late result from a dead generation cannot be committed, and reconciles the
+lease and residency. A crash loop raises `RUNTIME_CRASH_LOOP`.
+
+## Interrupted backups and restores
+
+A backup interrupted mid-write is moved to `FAILED` with `MANIFEST_INCOMPLETE` on the next start and
+can never become restore eligible. A restore interrupted mid-phase becomes
+`MANUAL_INTERVENTION_REQUIRED` rather than resuming silently; its journal records the phase it
+reached.
+
+A backup also cannot contain a record of itself as complete, since the dump is taken while its own
+`BackupSet` row is still `CREATING`. A restored control plane therefore consistently reports the
+backup it came from as not restore eligible.
+
+## Checking after a restart
+
+```sh
+python scripts/m16_invariants.py --label "after restart"
+curl -s localhost:8000/api/v1/health/ready
+```
+
+Fifteen invariants holding, readiness healthy, and no orphaned lease or queued job is what "the
+restart reconciled" means. See `docs/operations/CHAOS_TESTING.md` for the fault matrix.