50 lines
3.0 KiB
Markdown
50 lines
3.0 KiB
Markdown
# Node decommission architecture
|
|
|
|
## Contract
|
|
|
|
Node decommission is a terminal, audited lifecycle transition for a `ComputeNode`; it is not row
|
|
deletion and not a recovery shortcut. The node UUID, persisted identity, historical inventory,
|
|
artifact jobs, runtime evidence, deployments, serving history, capacity history and audit records
|
|
remain addressable. Only rebuildable current truth is removed or made unschedulable.
|
|
|
|
The operator first calls the preview endpoint. The preview returns the node generation, a
|
|
deterministic SHA-256 dependency digest, blockers, current-state cleanup counts and retained
|
|
provenance counts. Execute requires the same generation and digest, a unique idempotency key, an
|
|
operator identity, a reason of at least ten characters, and an exact typed match of the persisted
|
|
identity, hostname or display name. There is no force or cancel-dependencies flag.
|
|
|
|
## Fail-closed dependency graph
|
|
|
|
Execution refuses an online node and any non-terminal production/LAB deployment, experiment route,
|
|
download plan, artifact or serving job, gateway request, runtime probe, residency, GPU lease,
|
|
production approval, lifecycle approval/operation, migration/cutover, recovery operation or
|
|
identity-referencing enrollment. It also refuses removal of the last known live artifact copy and
|
|
the last production-eligible node while production deployments exist.
|
|
|
|
Unknown status values are treated as active because blocker queries exclude an explicit terminal
|
|
set. PostgreSQL execution locks every dependency and cleanup table in the transaction, re-runs the
|
|
preview, then claims the node generation with a conditional update. A changed generation or digest
|
|
returns 409 and requires a fresh preview.
|
|
|
|
## Terminal mutation
|
|
|
|
One transaction revokes every active node credential, clears host/storage/accelerator latest-state
|
|
telemetry and scheduler accelerator state, removes cold residency truth, marks accelerators
|
|
decommissioned, makes storage roots unavailable and read-only, marks their artifact locations
|
|
unreachable, removes eligibility and current inventory/capabilities, and writes the node tombstone.
|
|
It also writes one immutable `NodeDecommissionOperation` and one `NODE_DECOMMISSIONED` audit event
|
|
containing the reason, operator, timestamp, prior state, cleanup counts and credential result.
|
|
|
|
A uniqueness constraint on node and idempotency key plus the transaction makes retries deterministic
|
|
and the audit exactly once. Normal enrollment, metadata update, credential rotation, authentication,
|
|
agent publication, hardware reconciliation and scheduler placement all reject or exclude a
|
|
decommissioned identity. An explicit future recovery lifecycle would require a separate audited
|
|
design; none exists today.
|
|
|
|
## Invariant
|
|
|
|
`decommissioned_nodes_are_terminal` verifies every tombstone is disabled, in decommissioned
|
|
status/liveness, ineligible, without active credentials, active work, residencies, leases, runtime
|
|
probes, scheduler/current telemetry or current inventory. The invariant is read-only and runs with
|
|
the existing platform invariant suite.
|