73 lines
3.5 KiB
Markdown
73 lines
3.5 KiB
Markdown
# Artifact recovery
|
|
|
|
ModelForge holds several gigabytes of model weights. Copying them into every backup would be
|
|
expensive and pointless for artifacts that are reproducible from an exact upstream identity, and
|
|
dangerous to assume for artifacts that are not. Artifacts are therefore classified, not treated
|
|
uniformly.
|
|
|
|
```text
|
|
REHYDRATABLE exact upstream repository + commit SHA + per-file SHA-256 exist
|
|
NON_REHYDRATABLE no exact upstream identity; requires a payload backup
|
|
DERIVED produced locally from exact source artifacts; lineage must survive a rebuild
|
|
LOCAL_ONLY never had an upstream; payload backup only
|
|
```
|
|
|
|
## Manifest rather than payload
|
|
|
|
For rehydratable artifact sets the backup stores `artifacts.json`: per set, the artifact set id,
|
|
revision id, recovery class, upstream repository, exact commit SHA, file count, total bytes and
|
|
every file with its filename, SHA-256, size, type, serialization format and security and licence
|
|
status. A live backup recorded 24 artifact sets covering 8,100,376,532 rehydratable bytes in a
|
|
39,276-byte manifest.
|
|
|
|
## Rehydration
|
|
|
|
Recovery reuses the existing acquisition plane rather than inventing a second download path:
|
|
|
|
```text
|
|
ModelRevision + exact commit SHA + expected file list + expected SHA-256
|
|
→ download plan against a disposable storage root
|
|
→ Node Agent downloads into quarantine
|
|
→ hash verification against provenance
|
|
→ atomic promotion
|
|
```
|
|
|
|
A floating revision is never used. `_revision_identity` resolves the repository and commit from the
|
|
upstream snapshot recorded for that revision, falling back to the model's upstream source; if
|
|
neither yields both, the set is classified `NON_REHYDRATABLE` and the recovery operation is created
|
|
already `BLOCKED` with `ARTIFACT_NOT_REHYDRATABLE` rather than attempting a download.
|
|
|
|
## Recovery is not a security bypass
|
|
|
|
A rehydrated artifact re-enters the supply chain as new material. It is written to quarantine,
|
|
hashed, checked against the security policy and promoted only on success, and it carries
|
|
`static_checks_passed_unapproved` — not the approval its predecessor held. `trust_remote_code`
|
|
stays `false`. A restore never launders an artifact into an approved state.
|
|
|
|
## Upstream unavailable
|
|
|
|
If the upstream cannot be reached the operation is recorded `BLOCKED` with
|
|
`ARTIFACT_REHYDRATION_BLOCKED` and zero bytes recovered. Recovery does not report the platform as
|
|
fully recovered while an artifact it depends on is still missing.
|
|
|
|
## Hash mismatch
|
|
|
|
`record_artifact_recovery` compares every verified file against the expected digest from
|
|
provenance. A mismatch downgrades the operation to `FAILED` with `ARTIFACT_HASH_MISMATCH` even when
|
|
the caller asserted success — the digest decides, not the reporter.
|
|
|
|
## Lineage
|
|
|
|
Each operation stores the artifact set id and variant, the upstream snapshot id, the model and
|
|
revision ids, the upstream revision and resolved commit, and the derived-artifact links with their
|
|
source digests. A rebuilt derived artifact keeps its path back to the exact source artifacts and
|
|
transformation identity; recovery never resets lineage.
|
|
|
|
## Disposable locations
|
|
|
|
Rehearsals rehydrate into a disposable storage root, never over the only usable copy. Live evidence
|
|
from disaster rehearsal C: `microsoft/trocr-small-printed` at commit
|
|
`04e994ab854b0089d4929f48c2b4dbe2ce78a340`, 7 files and 247,200,667 bytes recovered into
|
|
`m15-recovery-scratch` in 24.2 seconds, all seven digests matching provenance, while the 75
|
|
production artifact locations remained untouched.
|