Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,72 @@
+# Artifact recovery
+
+ModelForge holds several gigabytes of model weights. Copying them into every backup would be
+expensive and pointless for artifacts that are reproducible from an exact upstream identity, and
+dangerous to assume for artifacts that are not. Artifacts are therefore classified, not treated
+uniformly.
+
+```text
+REHYDRATABLE      exact upstream repository + commit SHA + per-file SHA-256 exist
+NON_REHYDRATABLE  no exact upstream identity; requires a payload backup
+DERIVED           produced locally from exact source artifacts; lineage must survive a rebuild
+LOCAL_ONLY        never had an upstream; payload backup only
+```
+
+## Manifest rather than payload
+
+For rehydratable artifact sets the backup stores `artifacts.json`: per set, the artifact set id,
+revision id, recovery class, upstream repository, exact commit SHA, file count, total bytes and
+every file with its filename, SHA-256, size, type, serialization format and security and licence
+status. A live backup recorded 24 artifact sets covering 8,100,376,532 rehydratable bytes in a
+39,276-byte manifest.
+
+## Rehydration
+
+Recovery reuses the existing acquisition plane rather than inventing a second download path:
+
+```text
+ModelRevision + exact commit SHA + expected file list + expected SHA-256
+  → download plan against a disposable storage root
+  → Node Agent downloads into quarantine
+  → hash verification against provenance
+  → atomic promotion
+```
+
+A floating revision is never used. `_revision_identity` resolves the repository and commit from the
+upstream snapshot recorded for that revision, falling back to the model's upstream source; if
+neither yields both, the set is classified `NON_REHYDRATABLE` and the recovery operation is created
+already `BLOCKED` with `ARTIFACT_NOT_REHYDRATABLE` rather than attempting a download.
+
+## Recovery is not a security bypass
+
+A rehydrated artifact re-enters the supply chain as new material. It is written to quarantine,
+hashed, checked against the security policy and promoted only on success, and it carries
+`static_checks_passed_unapproved` — not the approval its predecessor held. `trust_remote_code`
+stays `false`. A restore never launders an artifact into an approved state.
+
+## Upstream unavailable
+
+If the upstream cannot be reached the operation is recorded `BLOCKED` with
+`ARTIFACT_REHYDRATION_BLOCKED` and zero bytes recovered. Recovery does not report the platform as
+fully recovered while an artifact it depends on is still missing.
+
+## Hash mismatch
+
+`record_artifact_recovery` compares every verified file against the expected digest from
+provenance. A mismatch downgrades the operation to `FAILED` with `ARTIFACT_HASH_MISMATCH` even when
+the caller asserted success — the digest decides, not the reporter.
+
+## Lineage
+
+Each operation stores the artifact set id and variant, the upstream snapshot id, the model and
+revision ids, the upstream revision and resolved commit, and the derived-artifact links with their
+source digests. A rebuilt derived artifact keeps its path back to the exact source artifacts and
+transformation identity; recovery never resets lineage.
+
+## Disposable locations
+
+Rehearsals rehydrate into a disposable storage root, never over the only usable copy. Live evidence
+from disaster rehearsal C: `microsoft/trocr-small-printed` at commit
+`04e994ab854b0089d4929f48c2b4dbe2ce78a340`, 7 files and 247,200,667 bytes recovered into
+`m15-recovery-scratch` in 24.2 seconds, all seven digests matching provenance, while the 75
+production artifact locations remained untouched.