Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,48 @@
+# Runbook: artifact loss
+
+## Trigger
+
+An artifact location is missing, corrupt, or unavailable after a storage failure or a control-plane
+restore.
+
+## Diagnose
+
+Establish the recovery class first, because it decides whether recovery is possible at all:
+
+| Class | Recovery |
+| --- | --- |
+| `REHYDRATABLE` | exact upstream repository and commit exist; redownload |
+| `NON_REHYDRATABLE` | no upstream identity; only a payload backup can recover it |
+| `DERIVED` | rebuild from exact source artifacts, preserving lineage |
+| `LOCAL_ONLY` | payload backup only |
+
+`artifacts.json` inside the backup lists every artifact set with its class, upstream repository,
+exact commit SHA and per-file SHA-256.
+
+## Act and recover
+
+1. Register a disposable storage root for the recovery. Never rehydrate over the only usable copy of
+   a production artifact.
+2. Create the recovery operation (`POST /api/v1/admin/recovery/artifact-recoveries`) and confirm it
+   reports `REHYDRATABLE` with the exact commit you expect. If it reports `NON_REHYDRATABLE`, stop:
+   this artifact needs a payload restore, not a download.
+3. Create a download plan for that artifact set against the disposable root, approve it and execute
+   it. The Node Agent downloads into quarantine, hashes every file and promotes atomically.
+4. Record the observed evidence against the recovery operation.
+
+Rehydration is never done from a floating revision. If the upstream is unreachable, record the
+operation as `BLOCKED` with `ARTIFACT_REHYDRATION_BLOCKED` and say so — do not report the platform
+as recovered while an artifact it depends on is still missing.
+
+## Security
+
+A rehydrated artifact is new material. It re-enters quarantine, is re-hashed against provenance and
+re-checked against the security policy, and it carries `static_checks_passed_unapproved` rather than
+inheriting its predecessor's approval. `trust_remote_code` stays `false`. A digest mismatch fails
+the recovery with `ARTIFACT_HASH_MISMATCH` even if the download reported success.
+
+## Confirm
+
+Every expected file present with a matching digest, byte counts equal to provenance, the artifact
+locations `verified`, lineage intact for derived artifacts, and a runtime probe or capability smoke
+against the recovered set before it is used for anything.