Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,65 @@
+# Recovery asset classification
+
+Recovery starts with an inventory, not with a backup job. Every persisted state source in
+ModelForge is classified once, and the classification decides whether the state is copied,
+rebuilt, discarded, or explicitly disowned.
+
+## Classes
+
+```text
+AUTHORITATIVE   exists nowhere else; must be copied and provably restorable
+REBUILDABLE     reproducible from an exact identity; a manifest is enough
+EPHEMERAL       describes the present, not history; must be re-measured after recovery
+EXTERNAL        owned by another project; ModelForge records the dependency and never writes
+SECRET          credential material; hashes restore, plaintext does not
+```
+
+A class is not a label on a document: `RecoveryPolicyRevision` refuses incoherent combinations.
+Authoritative state cannot declare `NOT_BACKED_UP` or `MANIFEST_ONLY`, and cannot omit an RPO.
+Ephemeral state cannot claim a payload backup. External state must be marked as an external
+dependency. Secret state must name a secret recovery class. Only rebuildable state may be
+rehydrated instead of copied.
+
+## The inventory
+
+`recovery_asset_records` holds one row per state source with its owner, location, backup method,
+restore method, rebuild method, RPO, dependencies and readiness. The seeded inventory is:
+
+| Asset | Class | Backup | Recovery |
+| --- | --- | --- | --- |
+| `postgres.modelforge` | AUTHORITATIVE | `pg_dump --format=custom` | `pg_restore` into a fresh database |
+| `artifacts.derived` | AUTHORITATIVE | file copy | payload restore preserving derived lineage |
+| `artifacts.huggingface` | REBUILDABLE | manifest only | exact-revision redownload, quarantine, verify, promote |
+| `configuration.repository` | REBUILDABLE | source-control reference | checkout of the recorded commit |
+| `backup.destination` | REBUILDABLE | source-control reference | operator-provided storage |
+| `redis.queues` | EPHEMERAL | not backed up | queues start empty; leases reconciled |
+| `runtime.residency` | EPHEMERAL | not backed up | cleared on restore, re-measured from live GPU Node state |
+| `node.gpu_node` | SECRET | hashes only | restored hash stays valid, or revoke and re-enrol |
+| `credentials.project` | SECRET | hashes only | rotate; plaintext is unrecoverable |
+| `credentials.operator` | SECRET | never in a backup | operator-supplied break-glass |
+| `external.examplerag` | EXTERNAL | not backed up | owned by ExampleRAG |
+| `external.examplevision` | EXTERNAL | not backed up | owned by ExampleVision |
+| `external.huggingface` | EXTERNAL | not backed up | upstream availability only |
+| `external.gitea` | EXTERNAL | not backed up | platform operations |
+
+## History versus current truth
+
+The same distinction runs through the database itself. `recovery_fingerprint.py` partitions every
+persisted table into fingerprinted history and `CURRENT_TRUTH_TABLES`, and a test asserts the two
+sets together cover the entire schema — a new milestone cannot add a table that is silently
+neither.
+
+Current truth is `host_telemetry_latest`, `accelerator_telemetry_latest`, `storage_volume_states`,
+`hardware_inventory_runs`, `scheduler_accelerator_states`, `residency_allocations`,
+`serving_gpu_leases` and `gpu_leases`. These are cleared during recovery reconciliation and
+re-derived by a running control plane. Restoring them would present a stale NVML reading or a dead
+GPU lease as a present-day fact, and the restore gate refuses to reach `READY` while any of them
+still holds a row.
+
+## Readiness
+
+Each asset reports one of `PROTECTED`, `REHYDRATABLE`, `ROTATION_REQUIRED`,
+`EXTERNAL_DEPENDENCY` or `UNPROTECTED`. Authoritative assets are reported `UNPROTECTED` whenever no
+verified backup exists, regardless of what the policy claims: readiness is measured, not declared.
+
+See `docs/architecture/BACKUP_AND_RECOVERY.md` for the backup contract and ADR-0041.