Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,49 @@
+# Node decommission architecture
+
+## Contract
+
+Node decommission is a terminal, audited lifecycle transition for a `ComputeNode`; it is not row
+deletion and not a recovery shortcut. The node UUID, persisted identity, historical inventory,
+artifact jobs, runtime evidence, deployments, serving history, capacity history and audit records
+remain addressable. Only rebuildable current truth is removed or made unschedulable.
+
+The operator first calls the preview endpoint. The preview returns the node generation, a
+deterministic SHA-256 dependency digest, blockers, current-state cleanup counts and retained
+provenance counts. Execute requires the same generation and digest, a unique idempotency key, an
+operator identity, a reason of at least ten characters, and an exact typed match of the persisted
+identity, hostname or display name. There is no force or cancel-dependencies flag.
+
+## Fail-closed dependency graph
+
+Execution refuses an online node and any non-terminal production/LAB deployment, experiment route,
+download plan, artifact or serving job, gateway request, runtime probe, residency, GPU lease,
+production approval, lifecycle approval/operation, migration/cutover, recovery operation or
+identity-referencing enrollment. It also refuses removal of the last known live artifact copy and
+the last production-eligible node while production deployments exist.
+
+Unknown status values are treated as active because blocker queries exclude an explicit terminal
+set. PostgreSQL execution locks every dependency and cleanup table in the transaction, re-runs the
+preview, then claims the node generation with a conditional update. A changed generation or digest
+returns 409 and requires a fresh preview.
+
+## Terminal mutation
+
+One transaction revokes every active node credential, clears host/storage/accelerator latest-state
+telemetry and scheduler accelerator state, removes cold residency truth, marks accelerators
+decommissioned, makes storage roots unavailable and read-only, marks their artifact locations
+unreachable, removes eligibility and current inventory/capabilities, and writes the node tombstone.
+It also writes one immutable `NodeDecommissionOperation` and one `NODE_DECOMMISSIONED` audit event
+containing the reason, operator, timestamp, prior state, cleanup counts and credential result.
+
+A uniqueness constraint on node and idempotency key plus the transaction makes retries deterministic
+and the audit exactly once. Normal enrollment, metadata update, credential rotation, authentication,
+agent publication, hardware reconciliation and scheduler placement all reject or exclude a
+decommissioned identity. An explicit future recovery lifecycle would require a separate audited
+design; none exists today.
+
+## Invariant
+
+`decommissioned_nodes_are_terminal` verifies every tombstone is disabled, in decommissioned
+status/liveness, ineligible, without active credentials, active work, residencies, leases, runtime
+probes, scheduler/current telemetry or current inventory. The invariant is read-only and runs with
+the existing platform invariant suite.