# Runbook: permanently decommission a compute node

Use this only when hardware is permanently leaving ModelForge. For a temporary outage, lost
credential or rebuild, use `RUNBOOK_NODE_LOSS.md`. Never use GPU Node or another production node for a
rehearsal; enroll a disposable node identity.

## Prepare

1. Drain and retire every deployment, job, request, probe, lease and residency on the node.
2. Move any sole artifact copy and finish recovery, lifecycle and migration operations.
3. Stop the Node Agent and wait until liveness is `offline` or `stale`.
4. Record the change owner and a durable reason. Obtain the operator API key; project credentials
   and node credentials are intentionally insufficient.

## Preview

In Console, open Compute Nodes, select the node and use **Preview decommission** in the danger zone.
Alternatively:

```text
POST /api/v1/admin/hardware/nodes/{node_id}/decommission/preview
X-ModelForge-Admin-Token: <operator key>
```

Resolve every returned blocker. Do not attempt to bypass a blocker in the database. Confirm the
preview names the intended persisted identity and lists both cleanup and retained history.

## Execute

Type the displayed persisted identity, hostname or display name exactly. Supply the preview's
`node_generation` and `dependency_digest`, a fresh idempotency key, operator and reason to:

```text
POST /api/v1/admin/hardware/nodes/{node_id}/decommission
```

A 409 means state changed or a blocker remains: generate a fresh preview. Repeating a completed
request is safe and returns the existing operation; it does not create another audit event.

## Confirm

- node status and liveness are `decommissioned`, all eligibility is false and current inventory is
  empty;
- all old node credentials return 401 and rotation/ordinary re-enrollment are refused;
- scheduler/current telemetry rows are absent, accelerators are decommissioned and storage roots
  are unavailable/read-only;
- historical inventory, jobs, probes, deployments and audit evidence remain queryable;
- exactly one `NODE_DECOMMISSIONED` audit event and one operation record exist;
- the `decommissioned_nodes_are_terminal` invariant holds.

There is no undo button. A later physical host must enroll with a new persisted identity unless a
future explicit, audited recovery lifecycle is implemented.