Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,52 @@
+# Runbook: permanently decommission a compute node
+
+Use this only when hardware is permanently leaving ModelForge. For a temporary outage, lost
+credential or rebuild, use `RUNBOOK_NODE_LOSS.md`. Never use GPU Node or another production node for a
+rehearsal; enroll a disposable node identity.
+
+## Prepare
+
+1. Drain and retire every deployment, job, request, probe, lease and residency on the node.
+2. Move any sole artifact copy and finish recovery, lifecycle and migration operations.
+3. Stop the Node Agent and wait until liveness is `offline` or `stale`.
+4. Record the change owner and a durable reason. Obtain the operator API key; project credentials
+   and node credentials are intentionally insufficient.
+
+## Preview
+
+In Console, open Compute Nodes, select the node and use **Preview decommission** in the danger zone.
+Alternatively:
+
+```text
+POST /api/v1/admin/hardware/nodes/{node_id}/decommission/preview
+X-ModelForge-Admin-Token: <operator key>
+```
+
+Resolve every returned blocker. Do not attempt to bypass a blocker in the database. Confirm the
+preview names the intended persisted identity and lists both cleanup and retained history.
+
+## Execute
+
+Type the displayed persisted identity, hostname or display name exactly. Supply the preview's
+`node_generation` and `dependency_digest`, a fresh idempotency key, operator and reason to:
+
+```text
+POST /api/v1/admin/hardware/nodes/{node_id}/decommission
+```
+
+A 409 means state changed or a blocker remains: generate a fresh preview. Repeating a completed
+request is safe and returns the existing operation; it does not create another audit event.
+
+## Confirm
+
+- node status and liveness are `decommissioned`, all eligibility is false and current inventory is
+  empty;
+- all old node credentials return 401 and rotation/ordinary re-enrollment are refused;
+- scheduler/current telemetry rows are absent, accelerators are decommissioned and storage roots
+  are unavailable/read-only;
+- historical inventory, jobs, probes, deployments and audit evidence remain queryable;
+- exactly one `NODE_DECOMMISSIONED` audit event and one operation record exist;
+- the `decommissioned_nodes_are_terminal` invariant holds.
+
+There is no undo button. A later physical host must enroll with a new persisted identity unless a
+future explicit, audited recovery lifecycle is implemented.