82 lines
4.7 KiB
Markdown
82 lines
4.7 KiB
Markdown
# Hardware Plane Architecture
|
|
|
|
## Boundary
|
|
|
|
M1 observes hardware through `SystemHostCollector` and `NvidiaNvmlCollector`; M1.5 runs those same
|
|
typed collectors in an outbound compute-node agent and publishes observations to the central API.
|
|
API routes never call NVML or SQLAlchemy directly; `HardwareInventoryService` reconciles typed
|
|
observations through `HardwareRepository`. No shell parsing, Docker socket, privileged mode, arbitrary
|
|
host mount, inbound agent service, scheduler or inference runtime is introduced. The central API host
|
|
is not implicitly a compute node when startup refresh is disabled.
|
|
|
|
## Identity
|
|
|
|
A ComputeNode has an internal UUID and a stable identity key. Direct host execution uses a UUIDv5
|
|
derived from the operating-system machine ID. Containers use a UUID stored on the `modelforge-state`
|
|
volume, preventing image/container IDs or hostnames from becoming identity. An explicit configured ID
|
|
has highest priority. Hostname and display name are mutable metadata, never identity.
|
|
|
|
NVIDIA accelerators reconcile by `(compute_node_id, GPU UUID)`. Device index, PCI bus ID and display
|
|
name are observed facts and can change without creating a duplicate. A successful enumeration that no
|
|
longer includes a known UUID marks it `missing`; an incomplete/unavailable NVML scan never does.
|
|
|
|
## Availability semantics
|
|
|
|
Every optional fact is `known`, `unknown`, `unsupported`, `unavailable` or `temporarily_failed`.
|
|
Only `known` observations carry values. Zero is a valid measured value and is never substituted for a
|
|
missing metric. One unsupported metric does not invalidate the accelerator.
|
|
|
|
## Inventory and telemetry
|
|
|
|
Inventory contains identity and relatively stable facts: OS/kernel/architecture, CPU topology, total
|
|
RAM, GPU UUID/model/PCI/VRAM/compute capability, driver/CUDA-driver capability and MIG status.
|
|
Latest-state telemetry contains available RAM, relevant storage use, GPU VRAM use, utilization,
|
|
temperature, power, clocks, fan and performance state. Latest telemetry is upserted; M1 deliberately
|
|
creates no unbounded history. A later observability adapter may export samples without schema rebuild.
|
|
|
|
## Reconciliation and audit
|
|
|
|
Refresh is process-serialized and idempotent. It creates or updates one node, upserts storage/current
|
|
telemetry, rediscovers accelerators by UUID and preserves missing devices. Meaningful changes emit
|
|
`NODE_DISCOVERED`, `ACCELERATOR_DISCOVERED`, `ACCELERATOR_MISSING` or `HARDWARE_CHANGED` audit events.
|
|
Telemetry polling does not generate audit noise. Failed runs emit `INVENTORY_FAILED`, are persisted,
|
|
and leave the last successful inventory intact.
|
|
|
|
## Fingerprint
|
|
|
|
The deterministic SHA-256 fingerprint includes node identity, OS/version/kernel/architecture and the
|
|
sorted accelerator UUID/model/PCI/total VRAM/compute capability/driver/CUDA-driver facts. Driver or
|
|
relevant hardware changes produce a new fingerprint. Available RAM, storage use, utilization,
|
|
temperature, power, clocks and all other transient telemetry are excluded.
|
|
|
|
## Health
|
|
|
|
- no inventory: system process healthy, hardware inventory `pending`;
|
|
- host observed with zero NVIDIA devices: node `active`, accelerator capability absent;
|
|
- NVML unavailable with no prior GPU: node stays active, accelerator inventory is `degraded` with an
|
|
explicit unavailable reason;
|
|
- incomplete NVML scan: last good GPU state remains, inventory is degraded;
|
|
- GPU absent from a successful enumeration: accelerator `missing` and audited;
|
|
- unsupported temperature/power/fan: metric is unsupported, accelerator remains active;
|
|
- host collection failure: run failed, last good inventory retained, refresh returns normalized 503.
|
|
|
|
## Real GPU validation
|
|
|
|
Run `POST /api/v1/hardware/refresh` twice, then compare `/nodes` and `/accelerators` identities and
|
|
inspect typed metrics. The optional direct diagnostic is `nvidia-smi --query-gpu=...`; NVML remains
|
|
the implementation source. For containers, start with the GPU overlay and verify the NVIDIA Container
|
|
Toolkit exposes the intended device. Never infer RTX 4080 availability from the host description.
|
|
|
|
Remote reports retain agent observation time and central receive time. Persistent per-stream
|
|
sequences reject replay/reordering, while liveness uses received heartbeats to tolerate clock skew.
|
|
For enrollment, protocol and remote deployment details, see `NODE_AGENT_PROTOCOL.md` and
|
|
`../operations/SERVER_AGENT_DEPLOYMENT.md`.
|
|
|
|
## Permanent decommission
|
|
|
|
A permanently retired node becomes a terminal tombstone; it is never deleted. Preview and execute
|
|
are generation- and dependency-bound, active work blocks the transition, credentials are revoked,
|
|
current scheduler/telemetry truth is cleared, and historical evidence remains. Inventory refresh
|
|
and agent publication cannot revive the identity. See `NODE_DECOMMISSION.md` and
|
|
`../operations/RUNBOOK_NODE_DECOMMISSION.md`.
|