44 lines
2.1 KiB
Markdown
44 lines
2.1 KiB
Markdown
# Runbook: backup failure or stale backup
|
|
|
|
## Trigger
|
|
|
|
`BACKUP_FAILED`, `BACKUP_VERIFICATION_FAILED`, `BACKUP_STALE` or `RECOVERY_READINESS_DEGRADED`.
|
|
|
|
## Diagnose
|
|
|
|
Read the backup set's `failure_code` and the `verification` block, which records exactly how far
|
|
verification reached before it stopped.
|
|
|
|
| Failure code | Meaning |
|
|
| --- | --- |
|
|
| `MANIFEST_HASH_MISMATCH` | the manifest no longer matches its recorded identity |
|
|
| `MANIFEST_INCOMPLETE` | objects and journal disagree, no database payload, or the backup was interrupted |
|
|
| `HASH_MISMATCH` | a payload object no longer matches the manifest |
|
|
| `PAYLOAD_MISSING` | an object is absent from the allowlisted root |
|
|
| `DECRYPTION_FAILED` | the configured key cannot authenticate the payload |
|
|
| `ENCRYPTION_KEY_UNAVAILABLE` | the policy requires encryption and no key is configured |
|
|
| `INSUFFICIENT_CAPACITY` | the destination lacked headroom; nothing was written |
|
|
| `BACKUP_TOOL_UNAVAILABLE` | `pg_dump` is missing from the control-plane image |
|
|
| `CONCURRENT_OPERATION` | another backup or restore is in flight |
|
|
|
|
`BACKUP_STALE` means no *verified* backup exists inside the policy window. A `CREATED` backup that
|
|
was never verified does not count and never will.
|
|
|
|
## Act and recover
|
|
|
|
Do not delete the failed set: it is evidence, and retention protects the last verified backups
|
|
independently of it. Fix the cause — restore storage headroom, supply the encryption key, replace
|
|
damaged media — then take a new backup and verify it. Verification is idempotent and safe to repeat.
|
|
|
|
If verification fails on a backup that previously verified, treat the destination as suspect. Take a
|
|
fresh backup immediately, verify it, and only then investigate the older one; a platform with one
|
|
damaged backup and no newer verified backup is unprotected.
|
|
|
|
Never mark a backup restore eligible by editing the database. Only successful verification does that.
|
|
|
|
## Confirm
|
|
|
|
`GET /api/v1/admin/recovery/dashboard` shows a `latest_verified_backup_id`, an age inside
|
|
`backup_staleness_threshold_seconds`, `stale_backup: false` and no unprotected assets. The
|
|
`BACKUP_STALE` and `RECOVERY_READINESS_DEGRADED` alerts resolve on the next evaluation.
|