ModelForge/docs/operations/RUNBOOK_DATABASE_RESTORE.md

Runbook: control-plane database restore

Trigger

Loss, corruption or suspected divergence of the ModelForge PostgreSQL database, or a scheduled disaster-recovery rehearsal.

Preconditions

A VERIFIED backup set, the backup encryption key, a PostgreSQL server of a compatible major version, and an empty destination database. A restore may never target the database the running control plane is using; that is refused with DESTINATION_NOT_ISOLATED.

Act

1. Verify the backup again — POST /api/v1/admin/recovery/backups/{id}/verify. Never restore from a set you have not just verified.
2. Create a restore plan with the destination, artifact strategy, secret strategy and node strategy. Use VALIDATION into an isolated environment unless this is a real disaster.
3. Run the preflight and read all eleven checks. Do not proceed on a failure; each failure code names a real precondition.
4. Start the restore and advance it. The journal records every phase with its evidence and duration.
5. Read the validation result: table, constraint and index counts, the fingerprint diff, the residual current-truth rows, the measured RPO and RTO, and the READY gate.

For a destructive replacement, use the operator CLI rather than the API:

python -m modelforge_api.cli.dr verify-backup --backup-id <id>
python -m modelforge_api.cli.dr plan-restore --backup-id <id> --target-url <dsn> \
    --target-label <label> --mode DISASTER_RECOVERY --target-environment PRODUCTION --reason "..."
python -m modelforge_api.cli.dr validate-restore --plan-id <uuid>
python -m modelforge_api.cli.dr run-restore --plan-id <uuid> --reason "..."

If it fails

DESTINATION_NOT_EMPTY on a retry is correct behaviour, not an obstacle: the previous attempt left data. Drop and recreate the destination database, then retry. Do not merge into a partially restored database.

MANUAL_INTERVENTION_REQUIRED means the restore reached a state that must not be resolved automatically. Read the journal, decide, and record the decision.

Confirm

The operation reaches READY with every gate true, including current_truth_reconciled. Compare the fingerprint diff against the expected differences: the recovery reconciliation audit event, and whatever the live source wrote after the backup point. Anything else must be explained before the restore is accepted.