49 lines
2.2 KiB
Markdown
49 lines
2.2 KiB
Markdown
# Runbook: artifact loss
|
|
|
|
## Trigger
|
|
|
|
An artifact location is missing, corrupt, or unavailable after a storage failure or a control-plane
|
|
restore.
|
|
|
|
## Diagnose
|
|
|
|
Establish the recovery class first, because it decides whether recovery is possible at all:
|
|
|
|
| Class | Recovery |
|
|
| --- | --- |
|
|
| `REHYDRATABLE` | exact upstream repository and commit exist; redownload |
|
|
| `NON_REHYDRATABLE` | no upstream identity; only a payload backup can recover it |
|
|
| `DERIVED` | rebuild from exact source artifacts, preserving lineage |
|
|
| `LOCAL_ONLY` | payload backup only |
|
|
|
|
`artifacts.json` inside the backup lists every artifact set with its class, upstream repository,
|
|
exact commit SHA and per-file SHA-256.
|
|
|
|
## Act and recover
|
|
|
|
1. Register a disposable storage root for the recovery. Never rehydrate over the only usable copy of
|
|
a production artifact.
|
|
2. Create the recovery operation (`POST /api/v1/admin/recovery/artifact-recoveries`) and confirm it
|
|
reports `REHYDRATABLE` with the exact commit you expect. If it reports `NON_REHYDRATABLE`, stop:
|
|
this artifact needs a payload restore, not a download.
|
|
3. Create a download plan for that artifact set against the disposable root, approve it and execute
|
|
it. The Node Agent downloads into quarantine, hashes every file and promotes atomically.
|
|
4. Record the observed evidence against the recovery operation.
|
|
|
|
Rehydration is never done from a floating revision. If the upstream is unreachable, record the
|
|
operation as `BLOCKED` with `ARTIFACT_REHYDRATION_BLOCKED` and say so — do not report the platform
|
|
as recovered while an artifact it depends on is still missing.
|
|
|
|
## Security
|
|
|
|
A rehydrated artifact is new material. It re-enters quarantine, is re-hashed against provenance and
|
|
re-checked against the security policy, and it carries `static_checks_passed_unapproved` rather than
|
|
inheriting its predecessor's approval. `trust_remote_code` stays `false`. A digest mismatch fails
|
|
the recovery with `ARTIFACT_HASH_MISMATCH` even if the download reported success.
|
|
|
|
## Confirm
|
|
|
|
Every expected file present with a matching digest, byte counts equal to provenance, the artifact
|
|
locations `verified`, lineage intact for derived artifacts, and a runtime probe or capability smoke
|
|
against the recovered set before it is used for anything.
|