Initial public ModelForge release
This commit is contained in:
Jens
@@ -0,0 +1,131 @@
|
||||
# Backup and recovery
|
||||
|
||||
M15 makes recovery a first-class control-plane concern with versioned policy, immutable backup
|
||||
identity and measured outcomes. Nothing in this plane is allowed to look successful without
|
||||
evidence.
|
||||
|
||||
## RecoveryPolicy
|
||||
|
||||
`RecoveryPolicyRevision` is an immutable, fingerprinted revision per asset class carrying backup
|
||||
method, retention, minimum verified backups, RPO and RTO targets, restore verification depth,
|
||||
encryption requirement, external-dependency and rehydration flags, secret class and rationale.
|
||||
Superseding a policy creates revision *n+1* and deactivates its predecessor; recovery rules are
|
||||
never scattered inline.
|
||||
|
||||
Seven policies ship by default: `control-plane.database`, `artifacts.rehydratable`,
|
||||
`artifacts.non-rehydratable`, `runtime.ephemeral`, `external.projects`, `secrets.credentials` and
|
||||
`configuration.source-controlled`.
|
||||
|
||||
## BackupSet
|
||||
|
||||
A `BackupSet` is immutable recovery identity. Beyond the backup id and state it records the
|
||||
ModelForge version and commit, the canonical repository and reference, the Alembic revision, an
|
||||
environment fingerprint, the PostgreSQL server identity and backup-tool version, the destination
|
||||
root, payload and manifest paths with their SHA-256, the included and excluded asset classes, the
|
||||
payload size, encryption algorithm and key id, verification evidence, milestone and legal hold.
|
||||
|
||||
Identity fields are sealed by `immutable_at`. A backup is built across several flushes — planned,
|
||||
written, then stamped — so the guard permits exactly the sealing flush and rejects every later
|
||||
change to identity. Lifecycle fields (state, verification, expiry) stay writable.
|
||||
|
||||
```text
|
||||
PLANNED → CREATING → CREATED → VERIFYING → VERIFIED
|
||||
↘ ↘
|
||||
FAILED FAILED
|
||||
VERIFIED/CREATED/FAILED → EXPIRED → DELETED
|
||||
```
|
||||
|
||||
Only `VERIFIED` is restore eligible. A `CREATED` backup that has never been verified cannot become
|
||||
a restore plan, and neither can a `FAILED` one.
|
||||
|
||||
## Manifest
|
||||
|
||||
Every backup writes `manifest.json` listing each object with its logical asset type, relative path,
|
||||
size, SHA-256, source generation, schema version and dependency references. The manifest itself is
|
||||
hashed into the backup set, so tampering with an entry is caught before any payload is read. The
|
||||
same entries are journaled as `backup_manifest_entries` rows, and verification requires the file
|
||||
and the journal to agree.
|
||||
|
||||
The manifest contains no secret material. `configuration.json` names the secrets a recovery will
|
||||
need — operator API key, backup encryption key, Hugging Face token — and their recovery class,
|
||||
never their values.
|
||||
|
||||
## Verification
|
||||
|
||||
`verify_backup` is the gate between "a file exists" and "this can be restored":
|
||||
|
||||
1. the manifest file hash matches the recorded manifest identity, or `MANIFEST_HASH_MISMATCH`;
|
||||
2. the manifest objects match the journaled entries and include a control-plane database payload,
|
||||
or `MANIFEST_INCOMPLETE`;
|
||||
3. every object exists at its allowlisted path with the recorded size and hash, or `HASH_MISMATCH`
|
||||
/ `PAYLOAD_MISSING`;
|
||||
4. the payload decrypts under the configured key and matches its recorded plaintext hash, or
|
||||
`DECRYPTION_FAILED` / `ENCRYPTION_KEY_UNAVAILABLE`;
|
||||
5. `pg_restore --list` reads the archive structure.
|
||||
|
||||
Only then does the set become `VERIFIED`.
|
||||
|
||||
## Encryption
|
||||
|
||||
Backups of authoritative control-plane state are protected at rest with AES-256-GCM from
|
||||
`cryptography`, framed in bounded chunks so a multi-gigabyte dump never has to be held in memory.
|
||||
ModelForge designs no cryptography of its own; `BackupCipher` is an adapter boundary so a KMS or
|
||||
HSM provider can replace the local key without touching the recovery service.
|
||||
|
||||
A wrong key fails closed: decryption raises before anything is renamed into place, and both the
|
||||
partial file and the destination are removed, so no readable plaintext is ever left behind. The key
|
||||
lives only in deployment configuration and is never written into a backup, a manifest, a log or an
|
||||
audit event.
|
||||
|
||||
## Consistency under load
|
||||
|
||||
The dump is `pg_dump --format=custom --serializable-deferrable`, a supported online-consistent
|
||||
logical backup taken in one snapshot, so concurrent control-plane writes cannot tear it. Copying a
|
||||
live data directory is never done.
|
||||
|
||||
Honesty about the window matters more than a clean-looking number: a semantic fingerprint is taken
|
||||
immediately before and immediately after the dump, and the backup records which subject groups were
|
||||
stable across it and which were being written. A live rehearsal under continuous capacity and SLO
|
||||
writes reported 13 of 15 groups stable, with `identity`, `observability` and `serving` active.
|
||||
|
||||
## Retention
|
||||
|
||||
`apply_retention` expires aged backups but never leaves the platform without a recovery point. The
|
||||
newest *n* verified backups (`minimum_verified_backups`, default 2) are protected regardless of
|
||||
age, milestone and legal-hold backups are never expired, and retention refuses to run at all while
|
||||
a restore is in flight.
|
||||
|
||||
## Capacity
|
||||
|
||||
Backups check headroom before writing rather than discovering it by filling a disk. The estimate
|
||||
separates bytes that must be copied from bytes protected by manifest alone, and an insufficient
|
||||
destination fails the backup with `INSUFFICIENT_CAPACITY` instead of producing a truncated payload.
|
||||
|
||||
## Interruption
|
||||
|
||||
A process that dies mid-backup leaves a `CREATING` row. Startup reconciliation moves it to `FAILED`
|
||||
with `MANIFEST_INCOMPLETE`, so an interrupted backup can never be mistaken for a usable one. This
|
||||
is not theoretical: the first live M15 backup attempt failed mid-write and was reconciled exactly
|
||||
this way on the next start.
|
||||
|
||||
A backup also cannot contain a record of *itself* as complete — the dump is taken while its own
|
||||
`BackupSet` row is still `CREATING` — so a restored control plane consistently reports the backup
|
||||
it was restored from as not restore eligible.
|
||||
|
||||
See ADR-0042, `docs/architecture/POSTGRESQL_BACKUP_RESTORE.md` and
|
||||
`docs/architecture/DISASTER_RECOVERY.md`.
|
||||
|
||||
|
||||
## M16 recovery under fault
|
||||
|
||||
A backup taken while a mixed capability soak is running verifies normally: the dump is a single
|
||||
consistent snapshot, and the write-window report names which subject groups were being written
|
||||
rather than claiming a quiet system. A live run recorded 11 of 15 groups stable with `identity`,
|
||||
`scheduler_history`, `serving` and `traffic_history` active.
|
||||
|
||||
Restarting the control plane mid-backup leaves the set `FAILED` with `MANIFEST_INCOMPLETE` on the
|
||||
next start; it can never become restore eligible.
|
||||
|
||||
An unwritable or absent backup destination now fails closed with `DESTINATION_UNAVAILABLE` rather
|
||||
than escaping as an unhandled error. Before M16 it returned an untyped server error and left the
|
||||
set stuck in `CREATING`, so `BACKUP_FAILED` never fired.
|
||||
Reference in New Issue
Block a user