55 lines
2.5 KiB
Markdown
55 lines
2.5 KiB
Markdown
# Runbook: node loss or credential loss
|
|
|
|
## Trigger
|
|
|
|
A compute node stops publishing, its credential is lost or believed stolen, or a node must be
|
|
rebuilt after a control-plane restore.
|
|
|
|
This runbook is for recoverable loss. If hardware is permanently leaving service, use
|
|
`RUNBOOK_NODE_DECOMMISSION.md`; do not disable or delete the row by hand.
|
|
|
|
## Diagnose
|
|
|
|
Distinguish the three failures, because they have different recoveries:
|
|
|
|
| Symptom | Meaning |
|
|
| --- | --- |
|
|
| heartbeats 401 | the credential is revoked or does not match a stored hash |
|
|
| heartbeats 403 | the credential authenticates but is not authorised for that node |
|
|
| no requests at all | the agent is down, or cannot reach the control plane |
|
|
|
|
Check `node_credentials` for the node: how many are active, how many revoked, and when the active
|
|
one was last used. Only hashes are stored; the plaintext cannot be read back from anywhere.
|
|
|
|
## Act and recover
|
|
|
|
**The node still has its credential.** Nothing to do. After a database restore the restored hash
|
|
matches and the node reconnects on its own.
|
|
|
|
**The credential is lost.** Issue a fresh enrolment token
|
|
(`POST /api/v1/admin/node-enrollments`) and restart the agent with it. The agent keeps its persisted
|
|
identity file, so it re-enrols onto the same node id with its accelerators, storage roots and
|
|
history intact. Enrolment revokes the previous credential automatically.
|
|
|
|
**The credential is believed stolen.** Revoke it first
|
|
(`DELETE /api/v1/admin/hardware/nodes/{node_id}/credential`), then re-enrol. Revocation is
|
|
permanent: the old credential stays refused after re-enrolment.
|
|
|
|
**The identity file is also lost.** The agent enrols as a new node, because ModelForge genuinely
|
|
cannot tell it apart from different hardware. If the old identity is permanently superseded, use
|
|
the audited decommission preview/execute flow rather than deleting or hand-editing it, and confirm
|
|
only one enabled node claims that hostname.
|
|
|
|
One enrolment token mints exactly one node identity. If two nodes appear from one token, the control
|
|
plane predates the M15 atomic-claim fix.
|
|
|
|
## Confirm
|
|
|
|
Exactly one enabled node for that hardware, one active credential with the old ones revoked,
|
|
heartbeats returning 200, inventory and telemetry re-collected, the scheduler reconciled against
|
|
freshly measured NVML state rather than a restored snapshot, and a capability smoke against the
|
|
recovered node.
|
|
|
|
Never revoke a production node credential merely to rehearse this. Use a disposable agent identity;
|
|
`docker-compose.node-recovery.yml` exists for exactly that.
|