# Runbook: backup failure or stale backup

## Trigger

`BACKUP_FAILED`, `BACKUP_VERIFICATION_FAILED`, `BACKUP_STALE` or `RECOVERY_READINESS_DEGRADED`.

## Diagnose

Read the backup set's `failure_code` and the `verification` block, which records exactly how far
verification reached before it stopped.

| Failure code | Meaning |
| --- | --- |
| `MANIFEST_HASH_MISMATCH` | the manifest no longer matches its recorded identity |
| `MANIFEST_INCOMPLETE` | objects and journal disagree, no database payload, or the backup was interrupted |
| `HASH_MISMATCH` | a payload object no longer matches the manifest |
| `PAYLOAD_MISSING` | an object is absent from the allowlisted root |
| `DECRYPTION_FAILED` | the configured key cannot authenticate the payload |
| `ENCRYPTION_KEY_UNAVAILABLE` | the policy requires encryption and no key is configured |
| `INSUFFICIENT_CAPACITY` | the destination lacked headroom; nothing was written |
| `BACKUP_TOOL_UNAVAILABLE` | `pg_dump` is missing from the control-plane image |
| `CONCURRENT_OPERATION` | another backup or restore is in flight |

`BACKUP_STALE` means no *verified* backup exists inside the policy window. A `CREATED` backup that
was never verified does not count and never will.

## Act and recover

Do not delete the failed set: it is evidence, and retention protects the last verified backups
independently of it. Fix the cause — restore storage headroom, supply the encryption key, replace
damaged media — then take a new backup and verify it. Verification is idempotent and safe to repeat.

If verification fails on a backup that previously verified, treat the destination as suspect. Take a
fresh backup immediately, verify it, and only then investigate the older one; a platform with one
damaged backup and no newer verified backup is unprotected.

Never mark a backup restore eligible by editing the database. Only successful verification does that.

## Confirm

`GET /api/v1/admin/recovery/dashboard` shows a `latest_verified_backup_id`, an age inside
`backup_staleness_threshold_seconds`, `stale_backup: false` and no unprotected assets. The
`BACKUP_STALE` and `RECOVERY_READINESS_DEGRADED` alerts resolve on the next evaluation.