ModelForge/docs/architecture/HARDWARE_PLANE.md

Hardware Plane Architecture

Boundary

M1 observes hardware through SystemHostCollector and NvidiaNvmlCollector; M1.5 runs those same typed collectors in an outbound compute-node agent and publishes observations to the central API. API routes never call NVML or SQLAlchemy directly; HardwareInventoryService reconciles typed observations through HardwareRepository. No shell parsing, Docker socket, privileged mode, arbitrary host mount, inbound agent service, scheduler or inference runtime is introduced. The central API host is not implicitly a compute node when startup refresh is disabled.

Identity

A ComputeNode has an internal UUID and a stable identity key. Direct host execution uses a UUIDv5 derived from the operating-system machine ID. Containers use a UUID stored on the modelforge-state volume, preventing image/container IDs or hostnames from becoming identity. An explicit configured ID has highest priority. Hostname and display name are mutable metadata, never identity.

NVIDIA accelerators reconcile by (compute_node_id, GPU UUID). Device index, PCI bus ID and display name are observed facts and can change without creating a duplicate. A successful enumeration that no longer includes a known UUID marks it missing; an incomplete/unavailable NVML scan never does.

Availability semantics

Every optional fact is known, unknown, unsupported, unavailable or temporarily_failed. Only known observations carry values. Zero is a valid measured value and is never substituted for a missing metric. One unsupported metric does not invalidate the accelerator.

Inventory and telemetry

Inventory contains identity and relatively stable facts: OS/kernel/architecture, CPU topology, total RAM, GPU UUID/model/PCI/VRAM/compute capability, driver/CUDA-driver capability and MIG status. Latest-state telemetry contains available RAM, relevant storage use, GPU VRAM use, utilization, temperature, power, clocks, fan and performance state. Latest telemetry is upserted; M1 deliberately creates no unbounded history. A later observability adapter may export samples without schema rebuild.

Reconciliation and audit

Refresh is process-serialized and idempotent. It creates or updates one node, upserts storage/current telemetry, rediscovers accelerators by UUID and preserves missing devices. Meaningful changes emit NODE_DISCOVERED, ACCELERATOR_DISCOVERED, ACCELERATOR_MISSING or HARDWARE_CHANGED audit events. Telemetry polling does not generate audit noise. Failed runs emit INVENTORY_FAILED, are persisted, and leave the last successful inventory intact.

Fingerprint

The deterministic SHA-256 fingerprint includes node identity, OS/version/kernel/architecture and the sorted accelerator UUID/model/PCI/total VRAM/compute capability/driver/CUDA-driver facts. Driver or relevant hardware changes produce a new fingerprint. Available RAM, storage use, utilization, temperature, power, clocks and all other transient telemetry are excluded.

Health

• no inventory: system process healthy, hardware inventory pending;
• host observed with zero NVIDIA devices: node active, accelerator capability absent;
• NVML unavailable with no prior GPU: node stays active, accelerator inventory is degraded with an explicit unavailable reason;
• incomplete NVML scan: last good GPU state remains, inventory is degraded;
• GPU absent from a successful enumeration: accelerator missing and audited;
• unsupported temperature/power/fan: metric is unsupported, accelerator remains active;
• host collection failure: run failed, last good inventory retained, refresh returns normalized 503.

Real GPU validation

Run POST /api/v1/hardware/refresh twice, then compare /nodes and /accelerators identities and inspect typed metrics. The optional direct diagnostic is nvidia-smi --query-gpu=...; NVML remains the implementation source. For containers, start with the GPU overlay and verify the NVIDIA Container Toolkit exposes the intended device. Never infer RTX 4080 availability from the host description.

Remote reports retain agent observation time and central receive time. Persistent per-stream sequences reject replay/reordering, while liveness uses received heartbeats to tolerate clock skew. For enrollment, protocol and remote deployment details, see NODE_AGENT_PROTOCOL.md and ../operations/SERVER_AGENT_DEPLOYMENT.md.

Permanent decommission

A permanently retired node becomes a terminal tombstone; it is never deleted. Preview and execute are generation- and dependency-bound, active work blocks the transition, credentials are revoked, current scheduler/telemetry truth is cleared, and historical evidence remains. Inventory refresh and agent publication cannot revive the identity. See NODE_DECOMMISSION.md and ../operations/RUNBOOK_NODE_DECOMMISSION.md.