# Runbook: control-plane loss

## Trigger

The ModelForge control plane is unrecoverable: host lost, image lost, configuration lost, or all
three.

## Rebuild

1. Read the DR bundle for the chosen backup:
   `python -m modelforge_api.cli.dr bundle --backup-id <id>`. It names the ModelForge commit,
   repository and reference, the backup identity and manifest hash, the encryption key requirement,
   the host paths, the artifact recovery plan, the node enrolment requirement and the external
   dependencies.
2. Clone the canonical repository and check out the exact commit the backup records. ModelForge does
   not back up its own Git host; that is platform operations.
3. Provision PostgreSQL of a compatible major version and an empty database.
4. Supply configuration and secrets. There is no default administrator password anywhere in
   ModelForge. `MODELFORGE_OPERATOR_API_KEY` and `MODELFORGE_BACKUP_ENCRYPTION_KEY` are operator
   inputs; the encryption key is never written into a backup and a restore cannot proceed without it.
5. Restore the database — see `RUNBOOK_DATABASE_RESTORE.md`.
6. Start the control plane against the restored database. Do not let it run migrations it has not
   restored; the restore already brought the schema to head.
7. Re-enrol or reconnect the node — see `RUNBOOK_NODE_LOSS.md`.
8. Rehydrate missing artifacts — see `RUNBOOK_ARTIFACT_LOSS.md`.

## Break-glass credentials

If the operator API key is lost, the recovered deployment starts with none configured and every
admin route returns 503 rather than falling open. Set a new key in deployment configuration and
restart. Project and node credentials restore as hashes only: consumers that lost their plaintext
need a new credential issued, not a recovered one.

## What the rebuilt plane will and will not know

It knows all restored history: provenance, lifecycle, migrations, approvals, audit, observability
history, project bindings and node identities. It does not know current truth — telemetry, leases,
residency and inventory are empty until a live agent reports. Restored firing alerts are suppressed
so the rebuilt plane re-derives which alerts are actually firing now.

## Confirm

Liveness and readiness are healthy, the OpenAPI document publishes, an unauthenticated admin call
returns 401 and an authenticated one returns 200, the registry and project views serve restored
truth, and the recovery dashboard honestly reports that the *new* deployment has no verified backup
of its own yet. Take one immediately.