Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,43 @@
+# Runbook: backup failure or stale backup
+
+## Trigger
+
+`BACKUP_FAILED`, `BACKUP_VERIFICATION_FAILED`, `BACKUP_STALE` or `RECOVERY_READINESS_DEGRADED`.
+
+## Diagnose
+
+Read the backup set's `failure_code` and the `verification` block, which records exactly how far
+verification reached before it stopped.
+
+| Failure code | Meaning |
+| --- | --- |
+| `MANIFEST_HASH_MISMATCH` | the manifest no longer matches its recorded identity |
+| `MANIFEST_INCOMPLETE` | objects and journal disagree, no database payload, or the backup was interrupted |
+| `HASH_MISMATCH` | a payload object no longer matches the manifest |
+| `PAYLOAD_MISSING` | an object is absent from the allowlisted root |
+| `DECRYPTION_FAILED` | the configured key cannot authenticate the payload |
+| `ENCRYPTION_KEY_UNAVAILABLE` | the policy requires encryption and no key is configured |
+| `INSUFFICIENT_CAPACITY` | the destination lacked headroom; nothing was written |
+| `BACKUP_TOOL_UNAVAILABLE` | `pg_dump` is missing from the control-plane image |
+| `CONCURRENT_OPERATION` | another backup or restore is in flight |
+
+`BACKUP_STALE` means no *verified* backup exists inside the policy window. A `CREATED` backup that
+was never verified does not count and never will.
+
+## Act and recover
+
+Do not delete the failed set: it is evidence, and retention protects the last verified backups
+independently of it. Fix the cause — restore storage headroom, supply the encryption key, replace
+damaged media — then take a new backup and verify it. Verification is idempotent and safe to repeat.
+
+If verification fails on a backup that previously verified, treat the destination as suspect. Take a
+fresh backup immediately, verify it, and only then investigate the older one; a platform with one
+damaged backup and no newer verified backup is unprotected.
+
+Never mark a backup restore eligible by editing the database. Only successful verification does that.
+
+## Confirm
+
+`GET /api/v1/admin/recovery/dashboard` shows a `latest_verified_backup_id`, an age inside
+`backup_staleness_threshold_seconds`, `stale_backup: false` and no unprotected assets. The
+`BACKUP_STALE` and `RECOVERY_READINESS_DEGRADED` alerts resolve on the next evaluation.