Artifact recovery

ModelForge holds several gigabytes of model weights. Copying them into every backup would be expensive and pointless for artifacts that are reproducible from an exact upstream identity, and dangerous to assume for artifacts that are not. Artifacts are therefore classified, not treated uniformly.

REHYDRATABLE      exact upstream repository + commit SHA + per-file SHA-256 exist
NON_REHYDRATABLE  no exact upstream identity; requires a payload backup
DERIVED           produced locally from exact source artifacts; lineage must survive a rebuild
LOCAL_ONLY        never had an upstream; payload backup only

Manifest rather than payload

For rehydratable artifact sets the backup stores artifacts.json: per set, the artifact set id, revision id, recovery class, upstream repository, exact commit SHA, file count, total bytes and every file with its filename, SHA-256, size, type, serialization format and security and licence status. A live backup recorded 24 artifact sets covering 8,100,376,532 rehydratable bytes in a 39,276-byte manifest.

Rehydration

Recovery reuses the existing acquisition plane rather than inventing a second download path:

ModelRevision + exact commit SHA + expected file list + expected SHA-256
  → download plan against a disposable storage root
  → Node Agent downloads into quarantine
  → hash verification against provenance
  → atomic promotion

A floating revision is never used. _revision_identity resolves the repository and commit from the upstream snapshot recorded for that revision, falling back to the model's upstream source; if neither yields both, the set is classified NON_REHYDRATABLE and the recovery operation is created already BLOCKED with ARTIFACT_NOT_REHYDRATABLE rather than attempting a download.

Recovery is not a security bypass

A rehydrated artifact re-enters the supply chain as new material. It is written to quarantine, hashed, checked against the security policy and promoted only on success, and it carries static_checks_passed_unapproved — not the approval its predecessor held. trust_remote_code stays false. A restore never launders an artifact into an approved state.

Upstream unavailable

If the upstream cannot be reached the operation is recorded BLOCKED with ARTIFACT_REHYDRATION_BLOCKED and zero bytes recovered. Recovery does not report the platform as fully recovered while an artifact it depends on is still missing.

Hash mismatch

record_artifact_recovery compares every verified file against the expected digest from provenance. A mismatch downgrades the operation to FAILED with ARTIFACT_HASH_MISMATCH even when the caller asserted success — the digest decides, not the reporter.

Lineage

Each operation stores the artifact set id and variant, the upstream snapshot id, the model and revision ids, the upstream revision and resolved commit, and the derived-artifact links with their source digests. A rebuilt derived artifact keeps its path back to the exact source artifacts and transformation identity; recovery never resets lineage.

Disposable locations

Rehearsals rehydrate into a disposable storage root, never over the only usable copy. Live evidence from disaster rehearsal C: microsoft/trocr-small-printed at commit 04e994ab854b0089d4929f48c2b4dbe2ce78a340, 7 files and 247,200,667 bytes recovered into m15-recovery-scratch in 24.2 seconds, all seven digests matching provenance, while the 75 production artifact locations remained untouched.

3.5 KiB Raw Permalink Blame History