ModelForge/docs/architecture/BACKUP_AND_RECOVERY.md

Backup and recovery

M15 makes recovery a first-class control-plane concern with versioned policy, immutable backup identity and measured outcomes. Nothing in this plane is allowed to look successful without evidence.

RecoveryPolicy

RecoveryPolicyRevision is an immutable, fingerprinted revision per asset class carrying backup method, retention, minimum verified backups, RPO and RTO targets, restore verification depth, encryption requirement, external-dependency and rehydration flags, secret class and rationale. Superseding a policy creates revision n+1 and deactivates its predecessor; recovery rules are never scattered inline.

Seven policies ship by default: control-plane.database, artifacts.rehydratable, artifacts.non-rehydratable, runtime.ephemeral, external.projects, secrets.credentials and configuration.source-controlled.

BackupSet

A BackupSet is immutable recovery identity. Beyond the backup id and state it records the ModelForge version and commit, the canonical repository and reference, the Alembic revision, an environment fingerprint, the PostgreSQL server identity and backup-tool version, the destination root, payload and manifest paths with their SHA-256, the included and excluded asset classes, the payload size, encryption algorithm and key id, verification evidence, milestone and legal hold.

Identity fields are sealed by immutable_at. A backup is built across several flushes — planned, written, then stamped — so the guard permits exactly the sealing flush and rejects every later change to identity. Lifecycle fields (state, verification, expiry) stay writable.

PLANNED → CREATING → CREATED → VERIFYING → VERIFIED
                          ↘         ↘
                           FAILED    FAILED
VERIFIED/CREATED/FAILED → EXPIRED → DELETED

Only VERIFIED is restore eligible. A CREATED backup that has never been verified cannot become a restore plan, and neither can a FAILED one.

Manifest

Every backup writes manifest.json listing each object with its logical asset type, relative path, size, SHA-256, source generation, schema version and dependency references. The manifest itself is hashed into the backup set, so tampering with an entry is caught before any payload is read. The same entries are journaled as backup_manifest_entries rows, and verification requires the file and the journal to agree.

The manifest contains no secret material. configuration.json names the secrets a recovery will need — operator API key, backup encryption key, Hugging Face token — and their recovery class, never their values.

Verification

verify_backup is the gate between "a file exists" and "this can be restored":

1. the manifest file hash matches the recorded manifest identity, or MANIFEST_HASH_MISMATCH;
2. the manifest objects match the journaled entries and include a control-plane database payload, or MANIFEST_INCOMPLETE;
3. every object exists at its allowlisted path with the recorded size and hash, or HASH_MISMATCH / PAYLOAD_MISSING;
4. the payload decrypts under the configured key and matches its recorded plaintext hash, or DECRYPTION_FAILED / ENCRYPTION_KEY_UNAVAILABLE;
5. pg_restore --list reads the archive structure.

Only then does the set become VERIFIED.

Encryption

Backups of authoritative control-plane state are protected at rest with AES-256-GCM from cryptography, framed in bounded chunks so a multi-gigabyte dump never has to be held in memory. ModelForge designs no cryptography of its own; BackupCipher is an adapter boundary so a KMS or HSM provider can replace the local key without touching the recovery service.

A wrong key fails closed: decryption raises before anything is renamed into place, and both the partial file and the destination are removed, so no readable plaintext is ever left behind. The key lives only in deployment configuration and is never written into a backup, a manifest, a log or an audit event.

Consistency under load

The dump is pg_dump --format=custom --serializable-deferrable, a supported online-consistent logical backup taken in one snapshot, so concurrent control-plane writes cannot tear it. Copying a live data directory is never done.

Honesty about the window matters more than a clean-looking number: a semantic fingerprint is taken immediately before and immediately after the dump, and the backup records which subject groups were stable across it and which were being written. A live rehearsal under continuous capacity and SLO writes reported 13 of 15 groups stable, with identity, observability and serving active.

Retention

apply_retention expires aged backups but never leaves the platform without a recovery point. The newest n verified backups (minimum_verified_backups, default 2) are protected regardless of age, milestone and legal-hold backups are never expired, and retention refuses to run at all while a restore is in flight.

Capacity

Backups check headroom before writing rather than discovering it by filling a disk. The estimate separates bytes that must be copied from bytes protected by manifest alone, and an insufficient destination fails the backup with INSUFFICIENT_CAPACITY instead of producing a truncated payload.

Interruption

A process that dies mid-backup leaves a CREATING row. Startup reconciliation moves it to FAILED with MANIFEST_INCOMPLETE, so an interrupted backup can never be mistaken for a usable one. This is not theoretical: the first live M15 backup attempt failed mid-write and was reconciled exactly this way on the next start.

A backup also cannot contain a record of itself as complete — the dump is taken while its own BackupSet row is still CREATING — so a restored control plane consistently reports the backup it was restored from as not restore eligible.

See ADR-0042, docs/architecture/POSTGRESQL_BACKUP_RESTORE.md and docs/architecture/DISASTER_RECOVERY.md.

M16 recovery under fault

A backup taken while a mixed capability soak is running verifies normally: the dump is a single consistent snapshot, and the write-window report names which subject groups were being written rather than claiming a quiet system. A live run recorded 11 of 15 groups stable with identity, scheduler_history, serving and traffic_history active.

Restarting the control plane mid-backup leaves the set FAILED with MANIFEST_INCOMPLETE on the next start; it can never become restore eligible.

An unwritable or absent backup destination now fails closed with DESTINATION_UNAVAILABLE rather than escaping as an unhandled error. Before M16 it returned an untyped server error and left the set stuck in CREATING, so BACKUP_FAILED never fired.