ModelForge/docs/operations/RUNBOOK_NODE_DECOMMISSION.md

Runbook: permanently decommission a compute node

Use this only when hardware is permanently leaving ModelForge. For a temporary outage, lost credential or rebuild, use RUNBOOK_NODE_LOSS.md. Never use GPU Node or another production node for a rehearsal; enroll a disposable node identity.

Prepare

1. Drain and retire every deployment, job, request, probe, lease and residency on the node.
2. Move any sole artifact copy and finish recovery, lifecycle and migration operations.
3. Stop the Node Agent and wait until liveness is offline or stale.
4. Record the change owner and a durable reason. Obtain the operator API key; project credentials and node credentials are intentionally insufficient.

Preview

In Console, open Compute Nodes, select the node and use Preview decommission in the danger zone. Alternatively:

POST /api/v1/admin/hardware/nodes/{node_id}/decommission/preview
X-ModelForge-Admin-Token: <operator key>

Resolve every returned blocker. Do not attempt to bypass a blocker in the database. Confirm the preview names the intended persisted identity and lists both cleanup and retained history.

Execute

Type the displayed persisted identity, hostname or display name exactly. Supply the preview's node_generation and dependency_digest, a fresh idempotency key, operator and reason to:

POST /api/v1/admin/hardware/nodes/{node_id}/decommission

A 409 means state changed or a blocker remains: generate a fresh preview. Repeating a completed request is safe and returns the existing operation; it does not create another audit event.

Confirm

• node status and liveness are decommissioned, all eligibility is false and current inventory is empty;
• all old node credentials return 401 and rotation/ordinary re-enrollment are refused;
• scheduler/current telemetry rows are absent, accelerators are decommissioned and storage roots are unavailable/read-only;
• historical inventory, jobs, probes, deployments and audit evidence remain queryable;
• exactly one NODE_DECOMMISSIONED audit event and one operation record exist;
• the decommissioned_nodes_are_terminal invariant holds.

There is no undo button. A later physical host must enroll with a new persisted identity unless a future explicit, audited recovery lifecycle is implemented.