Initial public ModelForge release
This commit is contained in:
Jens
@@ -0,0 +1,115 @@
|
||||
# Disaster recovery
|
||||
|
||||
## Canonical restore flow
|
||||
|
||||
```text
|
||||
fresh host / control plane
|
||||
→ checkout the exact ModelForge commit recorded in the BackupSet
|
||||
→ provision PostgreSQL of a compatible major version
|
||||
→ verify the BackupSet (manifest hash, payload hashes, decryption, archive structure)
|
||||
→ restore the database into an empty destination
|
||||
→ run only the forward migrations this build knows
|
||||
→ supply secrets; rotate what cannot be restored
|
||||
→ start the control plane against the restored database
|
||||
→ reconcile: clear current truth, suppress restored alerts, surface in-flight operations
|
||||
→ register or re-enrol the node
|
||||
→ rehydrate missing artifacts from their exact revisions
|
||||
→ validate, compare fingerprints, measure RPO and RTO
|
||||
→ operator declares recovery complete
|
||||
```
|
||||
|
||||
## RestorePlan
|
||||
|
||||
`RestorePlan` is first class: backup set, mode, target environment and label, redacted database
|
||||
destination, artifact strategy, secret strategy, node strategy, expected ModelForge version,
|
||||
preflight evidence and validation requirements. Plans are idempotent — the same backup, mode,
|
||||
destination and label return the same plan rather than a second one.
|
||||
|
||||
```text
|
||||
VALIDATION isolated rehearsal; never a production environment
|
||||
REPLACEMENT an operator-declared maintenance target
|
||||
DISASTER_RECOVERY the only mode permitted to target production, and only when explicitly enabled
|
||||
```
|
||||
|
||||
## Preflight
|
||||
|
||||
Eleven checks run before any byte moves, and again at execution time:
|
||||
|
||||
```text
|
||||
backup_verified payload_hashes schema_compatibility
|
||||
encryption_key destination_isolation destination_reachable
|
||||
destination_empty postgres_compatibility disk_space
|
||||
source_control_revision external_dependencies
|
||||
```
|
||||
|
||||
A failing preflight moves the plan to `PREFLIGHT_FAILED` and `start` is refused with
|
||||
`restore_preflight_required`.
|
||||
|
||||
## RestoreOperation
|
||||
|
||||
The journal is durable and append-only. Each phase writes a `RestoreOperationEvent` with its
|
||||
evidence, and phase durations accumulate into the measured RTO.
|
||||
|
||||
```text
|
||||
PLANNED → PREFLIGHT → RESTORING_DATABASE → RESTORING_CONFIGURATION
|
||||
→ REHYDRATING_ARTIFACTS → RECONCILING → VALIDATING → READY
|
||||
↘
|
||||
FAILED | MANUAL_INTERVENTION_REQUIRED
|
||||
```
|
||||
|
||||
Retry resumes the journalled operation rather than creating a second one. A completed database
|
||||
phase is never replayed onto live data: the destination-empty guard turns a retry into
|
||||
`DESTINATION_NOT_EMPTY` instead of a merge. A restore interrupted mid-phase is moved to
|
||||
`MANUAL_INTERVENTION_REQUIRED` by startup reconciliation, never silently resumed.
|
||||
|
||||
## Reconciliation
|
||||
|
||||
Reconciliation runs exactly once per restore operation, keyed by a marker audit event carrying that
|
||||
operation's id. Matching on the action alone would be wrong: a backup legitimately contains the
|
||||
reconciliation events of *earlier* restores, and an unscoped check made every later restore believe
|
||||
it had already reconciled.
|
||||
|
||||
Reconciliation clears every current-truth table, suppresses restored firing alerts so a stale alert
|
||||
is not presented as present-day truth while its history is retained, and reports in-flight lifecycle
|
||||
operations and migration cutovers by id and stage with the typed outcome
|
||||
`RECOVERY_RECONCILIATION_REQUIRED`. It never guesses external state.
|
||||
|
||||
## In-flight operations
|
||||
|
||||
A backup taken while a lifecycle promotion or a migration cutover is mid-flight restores that
|
||||
operation intact, with its idempotency key, expected version and generation. On startup the
|
||||
lifecycle reconciler conservatively rolls the incomplete operation back from its immutable snapshot
|
||||
exactly once; a second start changes nothing. Migration cutovers are deliberately *not* auto-
|
||||
resolved, because external alias truth cannot be inferred after a crash — they stay in their
|
||||
intermediate stage and are reported as requiring reconciliation.
|
||||
|
||||
## RPO and RTO
|
||||
|
||||
Both are measured, never assumed.
|
||||
|
||||
RPO is the gap between the newest audit event in the restored database and the newest in the source
|
||||
at validation time; the restore point is the backup's completion timestamp. RTO is the sum of the
|
||||
measured phase durations.
|
||||
|
||||
## Rehearsal discipline
|
||||
|
||||
A rehearsal that reuses the live database volume is not a rehearsal. `docker-compose.dr.yml`
|
||||
provisions a separate PostgreSQL 17 with its own volume and a second control plane bound to it, and
|
||||
every rehearsal starts from a destroyed and recreated volume. The production database, ExampleRAG's
|
||||
Qdrant, ExampleVision's recognizer and external GPU workloads are never written to.
|
||||
|
||||
## Rehearsal catalogue
|
||||
|
||||
| | Scenario | Outcome |
|
||||
| --- | --- | --- |
|
||||
| A | Database loss | full restore into a fresh PostgreSQL, measured RTO/RPO |
|
||||
| B | Corrupt backup | payload and manifest tampering both fail closed |
|
||||
| C | Artifact loss | exact-revision rehydration into a disposable location |
|
||||
| D | Node credential loss | revoke, re-enrol, identity preserved |
|
||||
| E | Control-plane rebuild | isolated plane serving restored truth |
|
||||
| F | In-flight lifecycle | reconciled exactly once |
|
||||
| G | In-flight migration | surfaced, never auto-resolved |
|
||||
| H | Observability history | history restored, current state re-derived |
|
||||
|
||||
See `docs/quality/M15_VERIFICATION.md` for the measured results and
|
||||
`docs/operations/RUNBOOK_FULL_DR.md` for the operator procedure.
|
||||
Reference in New Issue
Block a user