3.8 KiB
Recovery asset classification
Recovery starts with an inventory, not with a backup job. Every persisted state source in ModelForge is classified once, and the classification decides whether the state is copied, rebuilt, discarded, or explicitly disowned.
Classes
AUTHORITATIVE exists nowhere else; must be copied and provably restorable
REBUILDABLE reproducible from an exact identity; a manifest is enough
EPHEMERAL describes the present, not history; must be re-measured after recovery
EXTERNAL owned by another project; ModelForge records the dependency and never writes
SECRET credential material; hashes restore, plaintext does not
A class is not a label on a document: RecoveryPolicyRevision refuses incoherent combinations.
Authoritative state cannot declare NOT_BACKED_UP or MANIFEST_ONLY, and cannot omit an RPO.
Ephemeral state cannot claim a payload backup. External state must be marked as an external
dependency. Secret state must name a secret recovery class. Only rebuildable state may be
rehydrated instead of copied.
The inventory
recovery_asset_records holds one row per state source with its owner, location, backup method,
restore method, rebuild method, RPO, dependencies and readiness. The seeded inventory is:
| Asset | Class | Backup | Recovery |
|---|---|---|---|
postgres.modelforge |
AUTHORITATIVE | pg_dump --format=custom |
pg_restore into a fresh database |
artifacts.derived |
AUTHORITATIVE | file copy | payload restore preserving derived lineage |
artifacts.huggingface |
REBUILDABLE | manifest only | exact-revision redownload, quarantine, verify, promote |
configuration.repository |
REBUILDABLE | source-control reference | checkout of the recorded commit |
backup.destination |
REBUILDABLE | source-control reference | operator-provided storage |
redis.queues |
EPHEMERAL | not backed up | queues start empty; leases reconciled |
runtime.residency |
EPHEMERAL | not backed up | cleared on restore, re-measured from live GPU Node state |
node.gpu_node |
SECRET | hashes only | restored hash stays valid, or revoke and re-enrol |
credentials.project |
SECRET | hashes only | rotate; plaintext is unrecoverable |
credentials.operator |
SECRET | never in a backup | operator-supplied break-glass |
external.examplerag |
EXTERNAL | not backed up | owned by ExampleRAG |
external.examplevision |
EXTERNAL | not backed up | owned by ExampleVision |
external.huggingface |
EXTERNAL | not backed up | upstream availability only |
external.gitea |
EXTERNAL | not backed up | platform operations |
History versus current truth
The same distinction runs through the database itself. recovery_fingerprint.py partitions every
persisted table into fingerprinted history and CURRENT_TRUTH_TABLES, and a test asserts the two
sets together cover the entire schema — a new milestone cannot add a table that is silently
neither.
Current truth is host_telemetry_latest, accelerator_telemetry_latest, storage_volume_states,
hardware_inventory_runs, scheduler_accelerator_states, residency_allocations,
serving_gpu_leases and gpu_leases. These are cleared during recovery reconciliation and
re-derived by a running control plane. Restoring them would present a stale NVML reading or a dead
GPU lease as a present-day fact, and the restore gate refuses to reach READY while any of them
still holds a row.
Readiness
Each asset reports one of PROTECTED, REHYDRATABLE, ROTATION_REQUIRED,
EXTERNAL_DEPENDENCY or UNPROTECTED. Authoritative assets are reported UNPROTECTED whenever no
verified backup exists, regardless of what the policy claims: readiness is measured, not declared.
See docs/architecture/BACKUP_AND_RECOVERY.md for the backup contract and ADR-0041.