Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,81 @@
+# Hardware Plane Architecture
+
+## Boundary
+
+M1 observes hardware through `SystemHostCollector` and `NvidiaNvmlCollector`; M1.5 runs those same
+typed collectors in an outbound compute-node agent and publishes observations to the central API.
+API routes never call NVML or SQLAlchemy directly; `HardwareInventoryService` reconciles typed
+observations through `HardwareRepository`. No shell parsing, Docker socket, privileged mode, arbitrary
+host mount, inbound agent service, scheduler or inference runtime is introduced. The central API host
+is not implicitly a compute node when startup refresh is disabled.
+
+## Identity
+
+A ComputeNode has an internal UUID and a stable identity key. Direct host execution uses a UUIDv5
+derived from the operating-system machine ID. Containers use a UUID stored on the `modelforge-state`
+volume, preventing image/container IDs or hostnames from becoming identity. An explicit configured ID
+has highest priority. Hostname and display name are mutable metadata, never identity.
+
+NVIDIA accelerators reconcile by `(compute_node_id, GPU UUID)`. Device index, PCI bus ID and display
+name are observed facts and can change without creating a duplicate. A successful enumeration that no
+longer includes a known UUID marks it `missing`; an incomplete/unavailable NVML scan never does.
+
+## Availability semantics
+
+Every optional fact is `known`, `unknown`, `unsupported`, `unavailable` or `temporarily_failed`.
+Only `known` observations carry values. Zero is a valid measured value and is never substituted for a
+missing metric. One unsupported metric does not invalidate the accelerator.
+
+## Inventory and telemetry
+
+Inventory contains identity and relatively stable facts: OS/kernel/architecture, CPU topology, total
+RAM, GPU UUID/model/PCI/VRAM/compute capability, driver/CUDA-driver capability and MIG status.
+Latest-state telemetry contains available RAM, relevant storage use, GPU VRAM use, utilization,
+temperature, power, clocks, fan and performance state. Latest telemetry is upserted; M1 deliberately
+creates no unbounded history. A later observability adapter may export samples without schema rebuild.
+
+## Reconciliation and audit
+
+Refresh is process-serialized and idempotent. It creates or updates one node, upserts storage/current
+telemetry, rediscovers accelerators by UUID and preserves missing devices. Meaningful changes emit
+`NODE_DISCOVERED`, `ACCELERATOR_DISCOVERED`, `ACCELERATOR_MISSING` or `HARDWARE_CHANGED` audit events.
+Telemetry polling does not generate audit noise. Failed runs emit `INVENTORY_FAILED`, are persisted,
+and leave the last successful inventory intact.
+
+## Fingerprint
+
+The deterministic SHA-256 fingerprint includes node identity, OS/version/kernel/architecture and the
+sorted accelerator UUID/model/PCI/total VRAM/compute capability/driver/CUDA-driver facts. Driver or
+relevant hardware changes produce a new fingerprint. Available RAM, storage use, utilization,
+temperature, power, clocks and all other transient telemetry are excluded.
+
+## Health
+
+- no inventory: system process healthy, hardware inventory `pending`;
+- host observed with zero NVIDIA devices: node `active`, accelerator capability absent;
+- NVML unavailable with no prior GPU: node stays active, accelerator inventory is `degraded` with an
+  explicit unavailable reason;
+- incomplete NVML scan: last good GPU state remains, inventory is degraded;
+- GPU absent from a successful enumeration: accelerator `missing` and audited;
+- unsupported temperature/power/fan: metric is unsupported, accelerator remains active;
+- host collection failure: run failed, last good inventory retained, refresh returns normalized 503.
+
+## Real GPU validation
+
+Run `POST /api/v1/hardware/refresh` twice, then compare `/nodes` and `/accelerators` identities and
+inspect typed metrics. The optional direct diagnostic is `nvidia-smi --query-gpu=...`; NVML remains
+the implementation source. For containers, start with the GPU overlay and verify the NVIDIA Container
+Toolkit exposes the intended device. Never infer RTX 4080 availability from the host description.
+
+Remote reports retain agent observation time and central receive time. Persistent per-stream
+sequences reject replay/reordering, while liveness uses received heartbeats to tolerate clock skew.
+For enrollment, protocol and remote deployment details, see `NODE_AGENT_PROTOCOL.md` and
+`../operations/SERVER_AGENT_DEPLOYMENT.md`.
+
+## Permanent decommission
+
+A permanently retired node becomes a terminal tombstone; it is never deleted. Preview and execute
+are generation- and dependency-bound, active work blocks the transition, credentials are revoked,
+current scheduler/telemetry truth is cleared, and historical evidence remains. Inventory refresh
+and agent publication cannot revive the identity. See `NODE_DECOMMISSION.md` and
+`../operations/RUNBOOK_NODE_DECOMMISSION.md`.