66 lines
3.8 KiB
Markdown
66 lines
3.8 KiB
Markdown
# Recovery asset classification
|
|
|
|
Recovery starts with an inventory, not with a backup job. Every persisted state source in
|
|
ModelForge is classified once, and the classification decides whether the state is copied,
|
|
rebuilt, discarded, or explicitly disowned.
|
|
|
|
## Classes
|
|
|
|
```text
|
|
AUTHORITATIVE exists nowhere else; must be copied and provably restorable
|
|
REBUILDABLE reproducible from an exact identity; a manifest is enough
|
|
EPHEMERAL describes the present, not history; must be re-measured after recovery
|
|
EXTERNAL owned by another project; ModelForge records the dependency and never writes
|
|
SECRET credential material; hashes restore, plaintext does not
|
|
```
|
|
|
|
A class is not a label on a document: `RecoveryPolicyRevision` refuses incoherent combinations.
|
|
Authoritative state cannot declare `NOT_BACKED_UP` or `MANIFEST_ONLY`, and cannot omit an RPO.
|
|
Ephemeral state cannot claim a payload backup. External state must be marked as an external
|
|
dependency. Secret state must name a secret recovery class. Only rebuildable state may be
|
|
rehydrated instead of copied.
|
|
|
|
## The inventory
|
|
|
|
`recovery_asset_records` holds one row per state source with its owner, location, backup method,
|
|
restore method, rebuild method, RPO, dependencies and readiness. The seeded inventory is:
|
|
|
|
| Asset | Class | Backup | Recovery |
|
|
| --- | --- | --- | --- |
|
|
| `postgres.modelforge` | AUTHORITATIVE | `pg_dump --format=custom` | `pg_restore` into a fresh database |
|
|
| `artifacts.derived` | AUTHORITATIVE | file copy | payload restore preserving derived lineage |
|
|
| `artifacts.huggingface` | REBUILDABLE | manifest only | exact-revision redownload, quarantine, verify, promote |
|
|
| `configuration.repository` | REBUILDABLE | source-control reference | checkout of the recorded commit |
|
|
| `backup.destination` | REBUILDABLE | source-control reference | operator-provided storage |
|
|
| `redis.queues` | EPHEMERAL | not backed up | queues start empty; leases reconciled |
|
|
| `runtime.residency` | EPHEMERAL | not backed up | cleared on restore, re-measured from live GPU Node state |
|
|
| `node.gpu_node` | SECRET | hashes only | restored hash stays valid, or revoke and re-enrol |
|
|
| `credentials.project` | SECRET | hashes only | rotate; plaintext is unrecoverable |
|
|
| `credentials.operator` | SECRET | never in a backup | operator-supplied break-glass |
|
|
| `external.examplerag` | EXTERNAL | not backed up | owned by ExampleRAG |
|
|
| `external.examplevision` | EXTERNAL | not backed up | owned by ExampleVision |
|
|
| `external.huggingface` | EXTERNAL | not backed up | upstream availability only |
|
|
| `external.gitea` | EXTERNAL | not backed up | platform operations |
|
|
|
|
## History versus current truth
|
|
|
|
The same distinction runs through the database itself. `recovery_fingerprint.py` partitions every
|
|
persisted table into fingerprinted history and `CURRENT_TRUTH_TABLES`, and a test asserts the two
|
|
sets together cover the entire schema — a new milestone cannot add a table that is silently
|
|
neither.
|
|
|
|
Current truth is `host_telemetry_latest`, `accelerator_telemetry_latest`, `storage_volume_states`,
|
|
`hardware_inventory_runs`, `scheduler_accelerator_states`, `residency_allocations`,
|
|
`serving_gpu_leases` and `gpu_leases`. These are cleared during recovery reconciliation and
|
|
re-derived by a running control plane. Restoring them would present a stale NVML reading or a dead
|
|
GPU lease as a present-day fact, and the restore gate refuses to reach `READY` while any of them
|
|
still holds a row.
|
|
|
|
## Readiness
|
|
|
|
Each asset reports one of `PROTECTED`, `REHYDRATABLE`, `ROTATION_REQUIRED`,
|
|
`EXTERNAL_DEPENDENCY` or `UNPROTECTED`. Authoritative assets are reported `UNPROTECTED` whenever no
|
|
verified backup exists, regardless of what the policy claims: readiness is measured, not declared.
|
|
|
|
See `docs/architecture/BACKUP_AND_RECOVERY.md` for the backup contract and ADR-0041.
|