ModelForge/docs/operations/RUNBOOK_FULL_DR.md

Runbook: full disaster recovery

The complete procedure when ModelForge must be rebuilt from a backup. Each step has its own runbook; this one gives the order and the gates between them.

Before you start

You need a VERIFIED backup set, the backup encryption key, access to the canonical Git repository, a PostgreSQL server of a compatible major version, and the operator API key you intend to use afterwards. Without the encryption key the restore cannot proceed, and no part of ModelForge can supply it — it is deliberately never written into a backup.

Run python -m modelforge_api.cli.dr bundle --backup-id <id> first. Everything below is listed there: the ModelForge commit and repository, the backup identity and manifest hash, the encryption key requirement, the host paths, the artifact recovery plan, the node enrolment requirement and the external dependencies.

Order

1. Verify the backup. See RUNBOOK_BACKUP_FAILURE.md if it does not verify. Do not continue with an unverified backup; nothing downstream can compensate.
2. Rebuild the control plane. RUNBOOK_CONTROL_PLANE_LOSS.md. Exact commit, provisioned database, supplied secrets.
3. Restore the database. RUNBOOK_DATABASE_RESTORE.md. Empty destination, eleven preflight checks, journalled phases, READY gate.
4. Reconcile. The restore clears current truth, suppresses restored firing alerts and reports in-flight lifecycle operations and migration cutovers by id and stage. Read that list. Lifecycle operations are rolled back conservatively from their immutable snapshots on the first start. Migration cutovers are not auto-resolved — external alias truth cannot be inferred after a crash — and must be reconciled through the typed adapter.
5. Recover the node. RUNBOOK_NODE_LOSS.md. Until a live agent reports, the platform has node identities but no telemetry, and that is correct.
6. Recover artifacts. RUNBOOK_ARTIFACT_LOSS.md. Rehydratable sets first; non-rehydratable sets need their payload backup.
7. Validate. Health, OpenAPI, operator auth, registry, lifecycle, migration, observability, scheduler reconciliation, capability smoke.
8. Take a new backup immediately and verify it. A recovered deployment has no recovery point of its own until it does, and the dashboard will say so.

Recovery is complete only when

database validated                     schema at head
API healthy                            scheduler reconciled against re-measured truth
node status understood                 critical artifact coverage known
lifecycle and migration state valid    audit history available
no unknown corruption                  a new verified backup exists

What recovery must not touch

ExampleRAG's Qdrant collections and production alias, ExampleVision's recognizer and VISION_AUTO_ACCEPT_ENABLED, external Ollama, Plex and Tdarr workloads, and any GPU work ModelForge does not own. ModelForge records these as external dependencies and writes to none of them. If a recovery procedure appears to require changing one of them, stop and escalate — it is the wrong procedure.

Declaring completion

Record the measured RPO and RTO with the phase breakdown, the fingerprint comparison and its explained differences, and the artifact and node recovery evidence. An operator declares recovery complete; the platform reports measurements and deliberately refuses to declare it for you.