52 lines
2.3 KiB
Markdown
52 lines
2.3 KiB
Markdown
# Runbook: control-plane database restore
|
|
|
|
## Trigger
|
|
|
|
Loss, corruption or suspected divergence of the ModelForge PostgreSQL database, or a scheduled
|
|
disaster-recovery rehearsal.
|
|
|
|
## Preconditions
|
|
|
|
A `VERIFIED` backup set, the backup encryption key, a PostgreSQL server of a compatible major
|
|
version, and an empty destination database. A restore may never target the database the running
|
|
control plane is using; that is refused with `DESTINATION_NOT_ISOLATED`.
|
|
|
|
## Act
|
|
|
|
1. Verify the backup again — `POST /api/v1/admin/recovery/backups/{id}/verify`. Never restore from
|
|
a set you have not just verified.
|
|
2. Create a restore plan with the destination, artifact strategy, secret strategy and node
|
|
strategy. Use `VALIDATION` into an isolated environment unless this is a real disaster.
|
|
3. Run the preflight and read all eleven checks. Do not proceed on a failure; each failure code
|
|
names a real precondition.
|
|
4. Start the restore and advance it. The journal records every phase with its evidence and
|
|
duration.
|
|
5. Read the validation result: table, constraint and index counts, the fingerprint diff, the
|
|
residual current-truth rows, the measured RPO and RTO, and the `READY` gate.
|
|
|
|
For a destructive replacement, use the operator CLI rather than the API:
|
|
|
|
```sh
|
|
python -m modelforge_api.cli.dr verify-backup --backup-id <id>
|
|
python -m modelforge_api.cli.dr plan-restore --backup-id <id> --target-url <dsn> \
|
|
--target-label <label> --mode DISASTER_RECOVERY --target-environment PRODUCTION --reason "..."
|
|
python -m modelforge_api.cli.dr validate-restore --plan-id <uuid>
|
|
python -m modelforge_api.cli.dr run-restore --plan-id <uuid> --reason "..."
|
|
```
|
|
|
|
## If it fails
|
|
|
|
`DESTINATION_NOT_EMPTY` on a retry is correct behaviour, not an obstacle: the previous attempt left
|
|
data. Drop and recreate the destination database, then retry. Do not merge into a partially
|
|
restored database.
|
|
|
|
`MANUAL_INTERVENTION_REQUIRED` means the restore reached a state that must not be resolved
|
|
automatically. Read the journal, decide, and record the decision.
|
|
|
|
## Confirm
|
|
|
|
The operation reaches `READY` with every gate true, including `current_truth_reconciled`. Compare
|
|
the fingerprint diff against the expected differences: the recovery reconciliation audit event, and
|
|
whatever the live source wrote after the backup point. Anything else must be explained before the
|
|
restore is accepted.
|