116 lines
5.3 KiB
Markdown
116 lines
5.3 KiB
Markdown
# Disaster recovery
|
|
|
|
## Canonical restore flow
|
|
|
|
```text
|
|
fresh host / control plane
|
|
→ checkout the exact ModelForge commit recorded in the BackupSet
|
|
→ provision PostgreSQL of a compatible major version
|
|
→ verify the BackupSet (manifest hash, payload hashes, decryption, archive structure)
|
|
→ restore the database into an empty destination
|
|
→ run only the forward migrations this build knows
|
|
→ supply secrets; rotate what cannot be restored
|
|
→ start the control plane against the restored database
|
|
→ reconcile: clear current truth, suppress restored alerts, surface in-flight operations
|
|
→ register or re-enrol the node
|
|
→ rehydrate missing artifacts from their exact revisions
|
|
→ validate, compare fingerprints, measure RPO and RTO
|
|
→ operator declares recovery complete
|
|
```
|
|
|
|
## RestorePlan
|
|
|
|
`RestorePlan` is first class: backup set, mode, target environment and label, redacted database
|
|
destination, artifact strategy, secret strategy, node strategy, expected ModelForge version,
|
|
preflight evidence and validation requirements. Plans are idempotent — the same backup, mode,
|
|
destination and label return the same plan rather than a second one.
|
|
|
|
```text
|
|
VALIDATION isolated rehearsal; never a production environment
|
|
REPLACEMENT an operator-declared maintenance target
|
|
DISASTER_RECOVERY the only mode permitted to target production, and only when explicitly enabled
|
|
```
|
|
|
|
## Preflight
|
|
|
|
Eleven checks run before any byte moves, and again at execution time:
|
|
|
|
```text
|
|
backup_verified payload_hashes schema_compatibility
|
|
encryption_key destination_isolation destination_reachable
|
|
destination_empty postgres_compatibility disk_space
|
|
source_control_revision external_dependencies
|
|
```
|
|
|
|
A failing preflight moves the plan to `PREFLIGHT_FAILED` and `start` is refused with
|
|
`restore_preflight_required`.
|
|
|
|
## RestoreOperation
|
|
|
|
The journal is durable and append-only. Each phase writes a `RestoreOperationEvent` with its
|
|
evidence, and phase durations accumulate into the measured RTO.
|
|
|
|
```text
|
|
PLANNED → PREFLIGHT → RESTORING_DATABASE → RESTORING_CONFIGURATION
|
|
→ REHYDRATING_ARTIFACTS → RECONCILING → VALIDATING → READY
|
|
↘
|
|
FAILED | MANUAL_INTERVENTION_REQUIRED
|
|
```
|
|
|
|
Retry resumes the journalled operation rather than creating a second one. A completed database
|
|
phase is never replayed onto live data: the destination-empty guard turns a retry into
|
|
`DESTINATION_NOT_EMPTY` instead of a merge. A restore interrupted mid-phase is moved to
|
|
`MANUAL_INTERVENTION_REQUIRED` by startup reconciliation, never silently resumed.
|
|
|
|
## Reconciliation
|
|
|
|
Reconciliation runs exactly once per restore operation, keyed by a marker audit event carrying that
|
|
operation's id. Matching on the action alone would be wrong: a backup legitimately contains the
|
|
reconciliation events of *earlier* restores, and an unscoped check made every later restore believe
|
|
it had already reconciled.
|
|
|
|
Reconciliation clears every current-truth table, suppresses restored firing alerts so a stale alert
|
|
is not presented as present-day truth while its history is retained, and reports in-flight lifecycle
|
|
operations and migration cutovers by id and stage with the typed outcome
|
|
`RECOVERY_RECONCILIATION_REQUIRED`. It never guesses external state.
|
|
|
|
## In-flight operations
|
|
|
|
A backup taken while a lifecycle promotion or a migration cutover is mid-flight restores that
|
|
operation intact, with its idempotency key, expected version and generation. On startup the
|
|
lifecycle reconciler conservatively rolls the incomplete operation back from its immutable snapshot
|
|
exactly once; a second start changes nothing. Migration cutovers are deliberately *not* auto-
|
|
resolved, because external alias truth cannot be inferred after a crash — they stay in their
|
|
intermediate stage and are reported as requiring reconciliation.
|
|
|
|
## RPO and RTO
|
|
|
|
Both are measured, never assumed.
|
|
|
|
RPO is the gap between the newest audit event in the restored database and the newest in the source
|
|
at validation time; the restore point is the backup's completion timestamp. RTO is the sum of the
|
|
measured phase durations.
|
|
|
|
## Rehearsal discipline
|
|
|
|
A rehearsal that reuses the live database volume is not a rehearsal. `docker-compose.dr.yml`
|
|
provisions a separate PostgreSQL 17 with its own volume and a second control plane bound to it, and
|
|
every rehearsal starts from a destroyed and recreated volume. The production database, ExampleRAG's
|
|
Qdrant, ExampleVision's recognizer and external GPU workloads are never written to.
|
|
|
|
## Rehearsal catalogue
|
|
|
|
| | Scenario | Outcome |
|
|
| --- | --- | --- |
|
|
| A | Database loss | full restore into a fresh PostgreSQL, measured RTO/RPO |
|
|
| B | Corrupt backup | payload and manifest tampering both fail closed |
|
|
| C | Artifact loss | exact-revision rehydration into a disposable location |
|
|
| D | Node credential loss | revoke, re-enrol, identity preserved |
|
|
| E | Control-plane rebuild | isolated plane serving restored truth |
|
|
| F | In-flight lifecycle | reconciled exactly once |
|
|
| G | In-flight migration | surfaced, never auto-resolved |
|
|
| H | Observability history | history restored, current state re-derived |
|
|
|
|
See `docs/quality/M15_VERIFICATION.md` for the measured results and
|
|
`docs/operations/RUNBOOK_FULL_DR.md` for the operator procedure.
|