1.7 KiB
Capacity snapshots and trends
The M14 minute poll samples existing Node Agent/NVML, scheduler, residency, lease and storage truth. It does not sample raw NVML at high frequency. Each enabled node/accelerator snapshot stores:
- observed and received timestamps, freshness and availability;
- GPU total, observed, external, ModelForge resident, leased, reserve and schedulable bytes;
- scheduler pressure state;
- total/available system RAM;
- total/free storage.
External GPU use is max(0, NVML observed - measured ModelForge residency). Schedulable VRAM uses
the existing scheduler invariant and never grants authority to manage external processes.
Aggregation and retention
Raw minute snapshots are retained for seven days. Older rows are aggregated into hourly buckets and retained for 90 days; cleanup is an application operation, not manual SQL. Each aggregate preserves sample count, schedulable min/max/average and storage-free minimum. Both tiers are bounded.
Trend responses calculate schedulable min/max/average/p50/p95, external min/max, storage min/max
and GPU pressure frequency. Fewer than three snapshots produce INSUFFICIENT_DATA; a latest stale
snapshot produces STALE.
Storage forecast requires at least 12 samples spanning at least one hour. Otherwise forecast status
is INSUFFICIENT_DATA. With sufficient evidence it reports observed bytes/day and, only for a
negative slope, a bounded days-to-current-zero estimate. It is a trend signal, not an automated
cleanup decision.
Capability headroom compares a measured CapabilityResourceEnvelope with known persisted node
capacity. A sustained mismatch becomes a warning; unknown/stale capacity is not treated as a
confident fit and no unlimited-scaling claim is made.