Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,54 @@
+# Runbook: node loss or credential loss
+
+## Trigger
+
+A compute node stops publishing, its credential is lost or believed stolen, or a node must be
+rebuilt after a control-plane restore.
+
+This runbook is for recoverable loss. If hardware is permanently leaving service, use
+`RUNBOOK_NODE_DECOMMISSION.md`; do not disable or delete the row by hand.
+
+## Diagnose
+
+Distinguish the three failures, because they have different recoveries:
+
+| Symptom | Meaning |
+| --- | --- |
+| heartbeats 401 | the credential is revoked or does not match a stored hash |
+| heartbeats 403 | the credential authenticates but is not authorised for that node |
+| no requests at all | the agent is down, or cannot reach the control plane |
+
+Check `node_credentials` for the node: how many are active, how many revoked, and when the active
+one was last used. Only hashes are stored; the plaintext cannot be read back from anywhere.
+
+## Act and recover
+
+**The node still has its credential.** Nothing to do. After a database restore the restored hash
+matches and the node reconnects on its own.
+
+**The credential is lost.** Issue a fresh enrolment token
+(`POST /api/v1/admin/node-enrollments`) and restart the agent with it. The agent keeps its persisted
+identity file, so it re-enrols onto the same node id with its accelerators, storage roots and
+history intact. Enrolment revokes the previous credential automatically.
+
+**The credential is believed stolen.** Revoke it first
+(`DELETE /api/v1/admin/hardware/nodes/{node_id}/credential`), then re-enrol. Revocation is
+permanent: the old credential stays refused after re-enrolment.
+
+**The identity file is also lost.** The agent enrols as a new node, because ModelForge genuinely
+cannot tell it apart from different hardware. If the old identity is permanently superseded, use
+the audited decommission preview/execute flow rather than deleting or hand-editing it, and confirm
+only one enabled node claims that hostname.
+
+One enrolment token mints exactly one node identity. If two nodes appear from one token, the control
+plane predates the M15 atomic-claim fix.
+
+## Confirm
+
+Exactly one enabled node for that hardware, one active credential with the old ones revoked,
+heartbeats returning 200, inventory and telemetry re-collected, the scheduler reconciled against
+freshly measured NVML state rather than a restored snapshot, and a capability smoke against the
+recovered node.
+
+Never revoke a production node credential merely to rehearse this. Use a disposable agent identity;
+`docker-compose.node-recovery.yml` exists for exactly that.