ModelForge/docs/operations/RUNBOOK_ARTIFACT_LOSS.md

Runbook: artifact loss

Trigger

An artifact location is missing, corrupt, or unavailable after a storage failure or a control-plane restore.

Diagnose

Establish the recovery class first, because it decides whether recovery is possible at all:

Class	Recovery
REHYDRATABLE	exact upstream repository and commit exist; redownload
NON_REHYDRATABLE	no upstream identity; only a payload backup can recover it
DERIVED	rebuild from exact source artifacts, preserving lineage
LOCAL_ONLY	payload backup only

artifacts.json inside the backup lists every artifact set with its class, upstream repository, exact commit SHA and per-file SHA-256.

Act and recover

1. Register a disposable storage root for the recovery. Never rehydrate over the only usable copy of a production artifact.
2. Create the recovery operation (POST /api/v1/admin/recovery/artifact-recoveries) and confirm it reports REHYDRATABLE with the exact commit you expect. If it reports NON_REHYDRATABLE, stop: this artifact needs a payload restore, not a download.
3. Create a download plan for that artifact set against the disposable root, approve it and execute it. The Node Agent downloads into quarantine, hashes every file and promotes atomically.
4. Record the observed evidence against the recovery operation.

Rehydration is never done from a floating revision. If the upstream is unreachable, record the operation as BLOCKED with ARTIFACT_REHYDRATION_BLOCKED and say so — do not report the platform as recovered while an artifact it depends on is still missing.

Security

A rehydrated artifact is new material. It re-enters quarantine, is re-hashed against provenance and re-checked against the security policy, and it carries static_checks_passed_unapproved rather than inheriting its predecessor's approval. trust_remote_code stays false. A digest mismatch fails the recovery with ARTIFACT_HASH_MISMATCH even if the download reported success.

Confirm

Every expected file present with a matching digest, byte counts equal to provenance, the artifact locations verified, lineage intact for derived artifacts, and a runtime probe or capability smoke against the recovered set before it is used for anything.