78 lines
3.6 KiB
Markdown
78 lines
3.6 KiB
Markdown
# Node recovery
|
|
|
|
A compute node is two separate things during recovery: a persisted hardware identity, which is
|
|
authoritative history, and a credential, which is secret material that hashes cannot reconstruct.
|
|
|
|
## Identity
|
|
|
|
`ComputeNode.key` is the agent's persisted identity, written to the node's state volume and reused
|
|
across restarts. A recovered node that still has its identity file keeps its node id, its
|
|
accelerators, its storage roots and its history. Losing the credential does not lose the identity.
|
|
|
|
A node whose identity file is also gone enrols as a new node. That is correct — ModelForge cannot
|
|
distinguish it from different hardware — and the operator disables the superseded row rather than
|
|
letting two nodes claim the same machine.
|
|
|
|
## Single-use enrolment
|
|
|
|
One enrolment token mints exactly one node identity. The claim is a conditional update whose row
|
|
count decides:
|
|
|
|
```sql
|
|
update node_enrollments set used_at = now()
|
|
where id = :id and used_at is null
|
|
```
|
|
|
|
This closed a real defect found by the M15 node-recovery rehearsal: the previous check read
|
|
`used_at` and wrote it later, and two agent threads racing into `ensure_enrolled` burned one token
|
|
on two compute-node rows for the same hardware. The Node Agent's publication and artifact loops now
|
|
also share an enrolment lock, so it issues one enrol request rather than two.
|
|
|
|
## Credential recovery
|
|
|
|
```text
|
|
RESTORABLE_SECRET restored from backup and immediately usable
|
|
ROTATABLE_SECRET hash restores as authoritative state; the consumer needs a new secret
|
|
NON_EXPORTABLE_SECRET never written to a backup at all
|
|
```
|
|
|
|
Node credentials are `ROTATABLE_SECRET`. Only `secret_hash` is stored, so a restore reinstates
|
|
which credentials existed and which were revoked, but cannot hand the plaintext back to a node that
|
|
lost it. Scenarios:
|
|
|
|
| Scenario | Recovery |
|
|
| --- | --- |
|
|
| database restored, node still holds its credential | the restored hash matches; the node reconnects |
|
|
| credential lost on the node | operator issues a fresh enrolment token; the node re-enrols and keeps its identity |
|
|
| credential believed stolen | operator revokes it; the node re-enrols |
|
|
| node hardware replaced | new identity, new enrolment; the old row is disabled |
|
|
|
|
Revocation is permanent. A revoked credential stays refused after re-enrolment: the partial unique
|
|
index `uq_active_node_credential` allows one active credential per node, and enrolment revokes the
|
|
previous one rather than deleting it, so the revocation survives in history.
|
|
|
|
## Recovery sequence
|
|
|
|
```text
|
|
node unavailable
|
|
→ liveness reflects offline/stale; capability health degrades honestly
|
|
→ operator issues an enrolment token
|
|
→ agent re-enrols, keeping its persisted identity
|
|
→ inventory and telemetry are re-collected
|
|
→ scheduler reconciles against re-measured NVML truth
|
|
→ capability smoke
|
|
```
|
|
|
|
Nothing about the node's current state is restored from a backup. After a control-plane restore the
|
|
node rows exist with their identities and history, but telemetry, leases and residency are empty
|
|
until a live agent reports again.
|
|
|
|
## Live evidence
|
|
|
|
Disaster rehearsal D used a disposable Node Agent identity; the production GPU Node credential was
|
|
never revoked. The disposable node enrolled once (one enrol request, zero 403s), its credential was
|
|
revoked and refused three times consecutively with 401, its credential file was deleted, and it
|
|
re-enrolled on a fresh token as node `41a7530c-7750-47c9-92d7-78302ee664fd` — the same persisted
|
|
identity, with one active and one revoked credential, one enabled rehearsal node and one distinct
|
|
hostname. The old credential still returned 401 afterwards.
|