# Runbook: control-plane database restore

## Trigger

Loss, corruption or suspected divergence of the ModelForge PostgreSQL database, or a scheduled
disaster-recovery rehearsal.

## Preconditions

A `VERIFIED` backup set, the backup encryption key, a PostgreSQL server of a compatible major
version, and an empty destination database. A restore may never target the database the running
control plane is using; that is refused with `DESTINATION_NOT_ISOLATED`.

## Act

1. Verify the backup again — `POST /api/v1/admin/recovery/backups/{id}/verify`. Never restore from
   a set you have not just verified.
2. Create a restore plan with the destination, artifact strategy, secret strategy and node
   strategy. Use `VALIDATION` into an isolated environment unless this is a real disaster.
3. Run the preflight and read all eleven checks. Do not proceed on a failure; each failure code
   names a real precondition.
4. Start the restore and advance it. The journal records every phase with its evidence and
   duration.
5. Read the validation result: table, constraint and index counts, the fingerprint diff, the
   residual current-truth rows, the measured RPO and RTO, and the `READY` gate.

For a destructive replacement, use the operator CLI rather than the API:

```sh
python -m modelforge_api.cli.dr verify-backup --backup-id <id>
python -m modelforge_api.cli.dr plan-restore --backup-id <id> --target-url <dsn> \
    --target-label <label> --mode DISASTER_RECOVERY --target-environment PRODUCTION --reason "..."
python -m modelforge_api.cli.dr validate-restore --plan-id <uuid>
python -m modelforge_api.cli.dr run-restore --plan-id <uuid> --reason "..."
```

## If it fails

`DESTINATION_NOT_EMPTY` on a retry is correct behaviour, not an obstacle: the previous attempt left
data. Drop and recreate the destination database, then retry. Do not merge into a partially
restored database.

`MANUAL_INTERVENTION_REQUIRED` means the restore reached a state that must not be resolved
automatically. Read the journal, decide, and record the decision.

## Confirm

The operation reaches `READY` with every gate true, including `current_truth_reconciled`. Compare
the fingerprint diff against the expected differences: the recovery reconciliation audit event, and
whatever the live source wrote after the backup point. Anything else must be explained before the
restore is accepted.