2.1 KiB
Runbook: backup failure or stale backup
Trigger
BACKUP_FAILED, BACKUP_VERIFICATION_FAILED, BACKUP_STALE or RECOVERY_READINESS_DEGRADED.
Diagnose
Read the backup set's failure_code and the verification block, which records exactly how far
verification reached before it stopped.
| Failure code | Meaning |
|---|---|
MANIFEST_HASH_MISMATCH |
the manifest no longer matches its recorded identity |
MANIFEST_INCOMPLETE |
objects and journal disagree, no database payload, or the backup was interrupted |
HASH_MISMATCH |
a payload object no longer matches the manifest |
PAYLOAD_MISSING |
an object is absent from the allowlisted root |
DECRYPTION_FAILED |
the configured key cannot authenticate the payload |
ENCRYPTION_KEY_UNAVAILABLE |
the policy requires encryption and no key is configured |
INSUFFICIENT_CAPACITY |
the destination lacked headroom; nothing was written |
BACKUP_TOOL_UNAVAILABLE |
pg_dump is missing from the control-plane image |
CONCURRENT_OPERATION |
another backup or restore is in flight |
BACKUP_STALE means no verified backup exists inside the policy window. A CREATED backup that
was never verified does not count and never will.
Act and recover
Do not delete the failed set: it is evidence, and retention protects the last verified backups independently of it. Fix the cause — restore storage headroom, supply the encryption key, replace damaged media — then take a new backup and verify it. Verification is idempotent and safe to repeat.
If verification fails on a backup that previously verified, treat the destination as suspect. Take a fresh backup immediately, verify it, and only then investigate the older one; a platform with one damaged backup and no newer verified backup is unprotected.
Never mark a backup restore eligible by editing the database. Only successful verification does that.
Confirm
GET /api/v1/admin/recovery/dashboard shows a latest_verified_backup_id, an age inside
backup_staleness_threshold_seconds, stale_backup: false and no unprotected assets. The
BACKUP_STALE and RECOVERY_READINESS_DEGRADED alerts resolve on the next evaluation.