47 lines
2.5 KiB
Markdown
47 lines
2.5 KiB
Markdown
# Runbook: control-plane loss
|
|
|
|
## Trigger
|
|
|
|
The ModelForge control plane is unrecoverable: host lost, image lost, configuration lost, or all
|
|
three.
|
|
|
|
## Rebuild
|
|
|
|
1. Read the DR bundle for the chosen backup:
|
|
`python -m modelforge_api.cli.dr bundle --backup-id <id>`. It names the ModelForge commit,
|
|
repository and reference, the backup identity and manifest hash, the encryption key requirement,
|
|
the host paths, the artifact recovery plan, the node enrolment requirement and the external
|
|
dependencies.
|
|
2. Clone the canonical repository and check out the exact commit the backup records. ModelForge does
|
|
not back up its own Git host; that is platform operations.
|
|
3. Provision PostgreSQL of a compatible major version and an empty database.
|
|
4. Supply configuration and secrets. There is no default administrator password anywhere in
|
|
ModelForge. `MODELFORGE_OPERATOR_API_KEY` and `MODELFORGE_BACKUP_ENCRYPTION_KEY` are operator
|
|
inputs; the encryption key is never written into a backup and a restore cannot proceed without it.
|
|
5. Restore the database — see `RUNBOOK_DATABASE_RESTORE.md`.
|
|
6. Start the control plane against the restored database. Do not let it run migrations it has not
|
|
restored; the restore already brought the schema to head.
|
|
7. Re-enrol or reconnect the node — see `RUNBOOK_NODE_LOSS.md`.
|
|
8. Rehydrate missing artifacts — see `RUNBOOK_ARTIFACT_LOSS.md`.
|
|
|
|
## Break-glass credentials
|
|
|
|
If the operator API key is lost, the recovered deployment starts with none configured and every
|
|
admin route returns 503 rather than falling open. Set a new key in deployment configuration and
|
|
restart. Project and node credentials restore as hashes only: consumers that lost their plaintext
|
|
need a new credential issued, not a recovered one.
|
|
|
|
## What the rebuilt plane will and will not know
|
|
|
|
It knows all restored history: provenance, lifecycle, migrations, approvals, audit, observability
|
|
history, project bindings and node identities. It does not know current truth — telemetry, leases,
|
|
residency and inventory are empty until a live agent reports. Restored firing alerts are suppressed
|
|
so the rebuilt plane re-derives which alerts are actually firing now.
|
|
|
|
## Confirm
|
|
|
|
Liveness and readiness are healthy, the OpenAPI document publishes, an unauthenticated admin call
|
|
returns 401 and an authenticated one returns 200, the registry and project views serve restored
|
|
truth, and the recovery dashboard honestly reports that the *new* deployment has no verified backup
|
|
of its own yet. Take one immediately.
|