ModelForge/docs/operations/RUNBOOK_CONTROL_PLANE_LOSS.md

Runbook: control-plane loss

Trigger

The ModelForge control plane is unrecoverable: host lost, image lost, configuration lost, or all three.

Rebuild

1. Read the DR bundle for the chosen backup: python -m modelforge_api.cli.dr bundle --backup-id <id>. It names the ModelForge commit, repository and reference, the backup identity and manifest hash, the encryption key requirement, the host paths, the artifact recovery plan, the node enrolment requirement and the external dependencies.
2. Clone the canonical repository and check out the exact commit the backup records. ModelForge does not back up its own Git host; that is platform operations.
3. Provision PostgreSQL of a compatible major version and an empty database.
4. Supply configuration and secrets. There is no default administrator password anywhere in ModelForge. MODELFORGE_OPERATOR_API_KEY and MODELFORGE_BACKUP_ENCRYPTION_KEY are operator inputs; the encryption key is never written into a backup and a restore cannot proceed without it.
5. Restore the database — see RUNBOOK_DATABASE_RESTORE.md.
6. Start the control plane against the restored database. Do not let it run migrations it has not restored; the restore already brought the schema to head.
7. Re-enrol or reconnect the node — see RUNBOOK_NODE_LOSS.md.
8. Rehydrate missing artifacts — see RUNBOOK_ARTIFACT_LOSS.md.

Break-glass credentials

If the operator API key is lost, the recovered deployment starts with none configured and every admin route returns 503 rather than falling open. Set a new key in deployment configuration and restart. Project and node credentials restore as hashes only: consumers that lost their plaintext need a new credential issued, not a recovered one.

What the rebuilt plane will and will not know

It knows all restored history: provenance, lifecycle, migrations, approvals, audit, observability history, project bindings and node identities. It does not know current truth — telemetry, leases, residency and inventory are empty until a live agent reports. Restored firing alerts are suppressed so the rebuilt plane re-derives which alerts are actually firing now.

Confirm

Liveness and readiness are healthy, the OpenAPI document publishes, an unauthenticated admin call returns 401 and an authenticated one returns 200, the registry and project views serve restored truth, and the recovery dashboard honestly reports that the new deployment has no verified backup of its own yet. Take one immediately.