# Backup and recovery M15 makes recovery a first-class control-plane concern with versioned policy, immutable backup identity and measured outcomes. Nothing in this plane is allowed to look successful without evidence. ## RecoveryPolicy `RecoveryPolicyRevision` is an immutable, fingerprinted revision per asset class carrying backup method, retention, minimum verified backups, RPO and RTO targets, restore verification depth, encryption requirement, external-dependency and rehydration flags, secret class and rationale. Superseding a policy creates revision *n+1* and deactivates its predecessor; recovery rules are never scattered inline. Seven policies ship by default: `control-plane.database`, `artifacts.rehydratable`, `artifacts.non-rehydratable`, `runtime.ephemeral`, `external.projects`, `secrets.credentials` and `configuration.source-controlled`. ## BackupSet A `BackupSet` is immutable recovery identity. Beyond the backup id and state it records the ModelForge version and commit, the canonical repository and reference, the Alembic revision, an environment fingerprint, the PostgreSQL server identity and backup-tool version, the destination root, payload and manifest paths with their SHA-256, the included and excluded asset classes, the payload size, encryption algorithm and key id, verification evidence, milestone and legal hold. Identity fields are sealed by `immutable_at`. A backup is built across several flushes — planned, written, then stamped — so the guard permits exactly the sealing flush and rejects every later change to identity. Lifecycle fields (state, verification, expiry) stay writable. ```text PLANNED → CREATING → CREATED → VERIFYING → VERIFIED ↘ ↘ FAILED FAILED VERIFIED/CREATED/FAILED → EXPIRED → DELETED ``` Only `VERIFIED` is restore eligible. A `CREATED` backup that has never been verified cannot become a restore plan, and neither can a `FAILED` one. ## Manifest Every backup writes `manifest.json` listing each object with its logical asset type, relative path, size, SHA-256, source generation, schema version and dependency references. The manifest itself is hashed into the backup set, so tampering with an entry is caught before any payload is read. The same entries are journaled as `backup_manifest_entries` rows, and verification requires the file and the journal to agree. The manifest contains no secret material. `configuration.json` names the secrets a recovery will need — operator API key, backup encryption key, Hugging Face token — and their recovery class, never their values. ## Verification `verify_backup` is the gate between "a file exists" and "this can be restored": 1. the manifest file hash matches the recorded manifest identity, or `MANIFEST_HASH_MISMATCH`; 2. the manifest objects match the journaled entries and include a control-plane database payload, or `MANIFEST_INCOMPLETE`; 3. every object exists at its allowlisted path with the recorded size and hash, or `HASH_MISMATCH` / `PAYLOAD_MISSING`; 4. the payload decrypts under the configured key and matches its recorded plaintext hash, or `DECRYPTION_FAILED` / `ENCRYPTION_KEY_UNAVAILABLE`; 5. `pg_restore --list` reads the archive structure. Only then does the set become `VERIFIED`. ## Encryption Backups of authoritative control-plane state are protected at rest with AES-256-GCM from `cryptography`, framed in bounded chunks so a multi-gigabyte dump never has to be held in memory. ModelForge designs no cryptography of its own; `BackupCipher` is an adapter boundary so a KMS or HSM provider can replace the local key without touching the recovery service. A wrong key fails closed: decryption raises before anything is renamed into place, and both the partial file and the destination are removed, so no readable plaintext is ever left behind. The key lives only in deployment configuration and is never written into a backup, a manifest, a log or an audit event. ## Consistency under load The dump is `pg_dump --format=custom --serializable-deferrable`, a supported online-consistent logical backup taken in one snapshot, so concurrent control-plane writes cannot tear it. Copying a live data directory is never done. Honesty about the window matters more than a clean-looking number: a semantic fingerprint is taken immediately before and immediately after the dump, and the backup records which subject groups were stable across it and which were being written. A live rehearsal under continuous capacity and SLO writes reported 13 of 15 groups stable, with `identity`, `observability` and `serving` active. ## Retention `apply_retention` expires aged backups but never leaves the platform without a recovery point. The newest *n* verified backups (`minimum_verified_backups`, default 2) are protected regardless of age, milestone and legal-hold backups are never expired, and retention refuses to run at all while a restore is in flight. ## Capacity Backups check headroom before writing rather than discovering it by filling a disk. The estimate separates bytes that must be copied from bytes protected by manifest alone, and an insufficient destination fails the backup with `INSUFFICIENT_CAPACITY` instead of producing a truncated payload. ## Interruption A process that dies mid-backup leaves a `CREATING` row. Startup reconciliation moves it to `FAILED` with `MANIFEST_INCOMPLETE`, so an interrupted backup can never be mistaken for a usable one. This is not theoretical: the first live M15 backup attempt failed mid-write and was reconciled exactly this way on the next start. A backup also cannot contain a record of *itself* as complete — the dump is taken while its own `BackupSet` row is still `CREATING` — so a restored control plane consistently reports the backup it was restored from as not restore eligible. See ADR-0042, `docs/architecture/POSTGRESQL_BACKUP_RESTORE.md` and `docs/architecture/DISASTER_RECOVERY.md`. ## M16 recovery under fault A backup taken while a mixed capability soak is running verifies normally: the dump is a single consistent snapshot, and the write-window report names which subject groups were being written rather than claiming a quiet system. A live run recorded 11 of 15 groups stable with `identity`, `scheduler_history`, `serving` and `traffic_history` active. Restarting the control plane mid-backup leaves the set `FAILED` with `MANIFEST_INCOMPLETE` on the next start; it can never become restore eligible. An unwritable or absent backup destination now fails closed with `DESTINATION_UNAVAILABLE` rather than escaping as an unhandled error. Before M16 it returned an untyped server error and left the set stuck in `CREATING`, so `BACKUP_FAILED` never fired.