132 lines
6.6 KiB
Markdown
132 lines
6.6 KiB
Markdown
# Backup and recovery
|
|
|
|
M15 makes recovery a first-class control-plane concern with versioned policy, immutable backup
|
|
identity and measured outcomes. Nothing in this plane is allowed to look successful without
|
|
evidence.
|
|
|
|
## RecoveryPolicy
|
|
|
|
`RecoveryPolicyRevision` is an immutable, fingerprinted revision per asset class carrying backup
|
|
method, retention, minimum verified backups, RPO and RTO targets, restore verification depth,
|
|
encryption requirement, external-dependency and rehydration flags, secret class and rationale.
|
|
Superseding a policy creates revision *n+1* and deactivates its predecessor; recovery rules are
|
|
never scattered inline.
|
|
|
|
Seven policies ship by default: `control-plane.database`, `artifacts.rehydratable`,
|
|
`artifacts.non-rehydratable`, `runtime.ephemeral`, `external.projects`, `secrets.credentials` and
|
|
`configuration.source-controlled`.
|
|
|
|
## BackupSet
|
|
|
|
A `BackupSet` is immutable recovery identity. Beyond the backup id and state it records the
|
|
ModelForge version and commit, the canonical repository and reference, the Alembic revision, an
|
|
environment fingerprint, the PostgreSQL server identity and backup-tool version, the destination
|
|
root, payload and manifest paths with their SHA-256, the included and excluded asset classes, the
|
|
payload size, encryption algorithm and key id, verification evidence, milestone and legal hold.
|
|
|
|
Identity fields are sealed by `immutable_at`. A backup is built across several flushes — planned,
|
|
written, then stamped — so the guard permits exactly the sealing flush and rejects every later
|
|
change to identity. Lifecycle fields (state, verification, expiry) stay writable.
|
|
|
|
```text
|
|
PLANNED → CREATING → CREATED → VERIFYING → VERIFIED
|
|
↘ ↘
|
|
FAILED FAILED
|
|
VERIFIED/CREATED/FAILED → EXPIRED → DELETED
|
|
```
|
|
|
|
Only `VERIFIED` is restore eligible. A `CREATED` backup that has never been verified cannot become
|
|
a restore plan, and neither can a `FAILED` one.
|
|
|
|
## Manifest
|
|
|
|
Every backup writes `manifest.json` listing each object with its logical asset type, relative path,
|
|
size, SHA-256, source generation, schema version and dependency references. The manifest itself is
|
|
hashed into the backup set, so tampering with an entry is caught before any payload is read. The
|
|
same entries are journaled as `backup_manifest_entries` rows, and verification requires the file
|
|
and the journal to agree.
|
|
|
|
The manifest contains no secret material. `configuration.json` names the secrets a recovery will
|
|
need — operator API key, backup encryption key, Hugging Face token — and their recovery class,
|
|
never their values.
|
|
|
|
## Verification
|
|
|
|
`verify_backup` is the gate between "a file exists" and "this can be restored":
|
|
|
|
1. the manifest file hash matches the recorded manifest identity, or `MANIFEST_HASH_MISMATCH`;
|
|
2. the manifest objects match the journaled entries and include a control-plane database payload,
|
|
or `MANIFEST_INCOMPLETE`;
|
|
3. every object exists at its allowlisted path with the recorded size and hash, or `HASH_MISMATCH`
|
|
/ `PAYLOAD_MISSING`;
|
|
4. the payload decrypts under the configured key and matches its recorded plaintext hash, or
|
|
`DECRYPTION_FAILED` / `ENCRYPTION_KEY_UNAVAILABLE`;
|
|
5. `pg_restore --list` reads the archive structure.
|
|
|
|
Only then does the set become `VERIFIED`.
|
|
|
|
## Encryption
|
|
|
|
Backups of authoritative control-plane state are protected at rest with AES-256-GCM from
|
|
`cryptography`, framed in bounded chunks so a multi-gigabyte dump never has to be held in memory.
|
|
ModelForge designs no cryptography of its own; `BackupCipher` is an adapter boundary so a KMS or
|
|
HSM provider can replace the local key without touching the recovery service.
|
|
|
|
A wrong key fails closed: decryption raises before anything is renamed into place, and both the
|
|
partial file and the destination are removed, so no readable plaintext is ever left behind. The key
|
|
lives only in deployment configuration and is never written into a backup, a manifest, a log or an
|
|
audit event.
|
|
|
|
## Consistency under load
|
|
|
|
The dump is `pg_dump --format=custom --serializable-deferrable`, a supported online-consistent
|
|
logical backup taken in one snapshot, so concurrent control-plane writes cannot tear it. Copying a
|
|
live data directory is never done.
|
|
|
|
Honesty about the window matters more than a clean-looking number: a semantic fingerprint is taken
|
|
immediately before and immediately after the dump, and the backup records which subject groups were
|
|
stable across it and which were being written. A live rehearsal under continuous capacity and SLO
|
|
writes reported 13 of 15 groups stable, with `identity`, `observability` and `serving` active.
|
|
|
|
## Retention
|
|
|
|
`apply_retention` expires aged backups but never leaves the platform without a recovery point. The
|
|
newest *n* verified backups (`minimum_verified_backups`, default 2) are protected regardless of
|
|
age, milestone and legal-hold backups are never expired, and retention refuses to run at all while
|
|
a restore is in flight.
|
|
|
|
## Capacity
|
|
|
|
Backups check headroom before writing rather than discovering it by filling a disk. The estimate
|
|
separates bytes that must be copied from bytes protected by manifest alone, and an insufficient
|
|
destination fails the backup with `INSUFFICIENT_CAPACITY` instead of producing a truncated payload.
|
|
|
|
## Interruption
|
|
|
|
A process that dies mid-backup leaves a `CREATING` row. Startup reconciliation moves it to `FAILED`
|
|
with `MANIFEST_INCOMPLETE`, so an interrupted backup can never be mistaken for a usable one. This
|
|
is not theoretical: the first live M15 backup attempt failed mid-write and was reconciled exactly
|
|
this way on the next start.
|
|
|
|
A backup also cannot contain a record of *itself* as complete — the dump is taken while its own
|
|
`BackupSet` row is still `CREATING` — so a restored control plane consistently reports the backup
|
|
it was restored from as not restore eligible.
|
|
|
|
See ADR-0042, `docs/architecture/POSTGRESQL_BACKUP_RESTORE.md` and
|
|
`docs/architecture/DISASTER_RECOVERY.md`.
|
|
|
|
|
|
## M16 recovery under fault
|
|
|
|
A backup taken while a mixed capability soak is running verifies normally: the dump is a single
|
|
consistent snapshot, and the write-window report names which subject groups were being written
|
|
rather than claiming a quiet system. A live run recorded 11 of 15 groups stable with `identity`,
|
|
`scheduler_history`, `serving` and `traffic_history` active.
|
|
|
|
Restarting the control plane mid-backup leaves the set `FAILED` with `MANIFEST_INCOMPLETE` on the
|
|
next start; it can never become restore eligible.
|
|
|
|
An unwritable or absent backup destination now fails closed with `DESTINATION_UNAVAILABLE` rather
|
|
than escaping as an unhandled error. Before M16 it returned an untyped server error and left the
|
|
set stuck in `CREATING`, so `BACKUP_FAILED` never fired.
|