53 lines
2.3 KiB
Markdown
53 lines
2.3 KiB
Markdown
# Runbook: permanently decommission a compute node
|
|
|
|
Use this only when hardware is permanently leaving ModelForge. For a temporary outage, lost
|
|
credential or rebuild, use `RUNBOOK_NODE_LOSS.md`. Never use GPU Node or another production node for a
|
|
rehearsal; enroll a disposable node identity.
|
|
|
|
## Prepare
|
|
|
|
1. Drain and retire every deployment, job, request, probe, lease and residency on the node.
|
|
2. Move any sole artifact copy and finish recovery, lifecycle and migration operations.
|
|
3. Stop the Node Agent and wait until liveness is `offline` or `stale`.
|
|
4. Record the change owner and a durable reason. Obtain the operator API key; project credentials
|
|
and node credentials are intentionally insufficient.
|
|
|
|
## Preview
|
|
|
|
In Console, open Compute Nodes, select the node and use **Preview decommission** in the danger zone.
|
|
Alternatively:
|
|
|
|
```text
|
|
POST /api/v1/admin/hardware/nodes/{node_id}/decommission/preview
|
|
X-ModelForge-Admin-Token: <operator key>
|
|
```
|
|
|
|
Resolve every returned blocker. Do not attempt to bypass a blocker in the database. Confirm the
|
|
preview names the intended persisted identity and lists both cleanup and retained history.
|
|
|
|
## Execute
|
|
|
|
Type the displayed persisted identity, hostname or display name exactly. Supply the preview's
|
|
`node_generation` and `dependency_digest`, a fresh idempotency key, operator and reason to:
|
|
|
|
```text
|
|
POST /api/v1/admin/hardware/nodes/{node_id}/decommission
|
|
```
|
|
|
|
A 409 means state changed or a blocker remains: generate a fresh preview. Repeating a completed
|
|
request is safe and returns the existing operation; it does not create another audit event.
|
|
|
|
## Confirm
|
|
|
|
- node status and liveness are `decommissioned`, all eligibility is false and current inventory is
|
|
empty;
|
|
- all old node credentials return 401 and rotation/ordinary re-enrollment are refused;
|
|
- scheduler/current telemetry rows are absent, accelerators are decommissioned and storage roots
|
|
are unavailable/read-only;
|
|
- historical inventory, jobs, probes, deployments and audit evidence remain queryable;
|
|
- exactly one `NODE_DECOMMISSIONED` audit event and one operation record exist;
|
|
- the `decommissioned_nodes_are_terminal` invariant holds.
|
|
|
|
There is no undo button. A later physical host must enroll with a new persisted identity unless a
|
|
future explicit, audited recovery lifecycle is implemented.
|