63 lines
3.4 KiB
Markdown
63 lines
3.4 KiB
Markdown
# Runbook: full disaster recovery
|
|
|
|
The complete procedure when ModelForge must be rebuilt from a backup. Each step has its own runbook;
|
|
this one gives the order and the gates between them.
|
|
|
|
## Before you start
|
|
|
|
You need a `VERIFIED` backup set, the backup encryption key, access to the canonical Git repository,
|
|
a PostgreSQL server of a compatible major version, and the operator API key you intend to use
|
|
afterwards. Without the encryption key the restore cannot proceed, and no part of ModelForge can
|
|
supply it — it is deliberately never written into a backup.
|
|
|
|
Run `python -m modelforge_api.cli.dr bundle --backup-id <id>` first. Everything below is listed
|
|
there: the ModelForge commit and repository, the backup identity and manifest hash, the encryption
|
|
key requirement, the host paths, the artifact recovery plan, the node enrolment requirement and the
|
|
external dependencies.
|
|
|
|
## Order
|
|
|
|
1. **Verify the backup.** See `RUNBOOK_BACKUP_FAILURE.md` if it does not verify. Do not continue
|
|
with an unverified backup; nothing downstream can compensate.
|
|
2. **Rebuild the control plane.** `RUNBOOK_CONTROL_PLANE_LOSS.md`. Exact commit, provisioned
|
|
database, supplied secrets.
|
|
3. **Restore the database.** `RUNBOOK_DATABASE_RESTORE.md`. Empty destination, eleven preflight
|
|
checks, journalled phases, `READY` gate.
|
|
4. **Reconcile.** The restore clears current truth, suppresses restored firing alerts and reports
|
|
in-flight lifecycle operations and migration cutovers by id and stage. Read that list. Lifecycle
|
|
operations are rolled back conservatively from their immutable snapshots on the first start.
|
|
Migration cutovers are *not* auto-resolved — external alias truth cannot be inferred after a
|
|
crash — and must be reconciled through the typed adapter.
|
|
5. **Recover the node.** `RUNBOOK_NODE_LOSS.md`. Until a live agent reports, the platform has node
|
|
identities but no telemetry, and that is correct.
|
|
6. **Recover artifacts.** `RUNBOOK_ARTIFACT_LOSS.md`. Rehydratable sets first; non-rehydratable sets
|
|
need their payload backup.
|
|
7. **Validate.** Health, OpenAPI, operator auth, registry, lifecycle, migration, observability,
|
|
scheduler reconciliation, capability smoke.
|
|
8. **Take a new backup immediately** and verify it. A recovered deployment has no recovery point of
|
|
its own until it does, and the dashboard will say so.
|
|
|
|
## Recovery is complete only when
|
|
|
|
```text
|
|
database validated schema at head
|
|
API healthy scheduler reconciled against re-measured truth
|
|
node status understood critical artifact coverage known
|
|
lifecycle and migration state valid audit history available
|
|
no unknown corruption a new verified backup exists
|
|
```
|
|
|
|
## What recovery must not touch
|
|
|
|
ExampleRAG's Qdrant collections and production alias, ExampleVision's recognizer and
|
|
`VISION_AUTO_ACCEPT_ENABLED`, external Ollama, Plex and Tdarr workloads, and any GPU work ModelForge
|
|
does not own. ModelForge records these as external dependencies and writes to none of them. If a
|
|
recovery procedure appears to require changing one of them, stop and escalate — it is the wrong
|
|
procedure.
|
|
|
|
## Declaring completion
|
|
|
|
Record the measured RPO and RTO with the phase breakdown, the fingerprint comparison and its
|
|
explained differences, and the artifact and node recovery evidence. An operator declares recovery
|
|
complete; the platform reports measurements and deliberately refuses to declare it for you.
|