33 lines
1.7 KiB
Markdown
33 lines
1.7 KiB
Markdown
# Capacity snapshots and trends
|
|
|
|
The M14 minute poll samples existing Node Agent/NVML, scheduler, residency, lease and storage truth.
|
|
It does not sample raw NVML at high frequency. Each enabled node/accelerator snapshot stores:
|
|
|
|
- observed and received timestamps, freshness and availability;
|
|
- GPU total, observed, external, ModelForge resident, leased, reserve and schedulable bytes;
|
|
- scheduler pressure state;
|
|
- total/available system RAM;
|
|
- total/free storage.
|
|
|
|
External GPU use is `max(0, NVML observed - measured ModelForge residency)`. Schedulable VRAM uses
|
|
the existing scheduler invariant and never grants authority to manage external processes.
|
|
|
|
## Aggregation and retention
|
|
|
|
Raw minute snapshots are retained for seven days. Older rows are aggregated into hourly buckets and
|
|
retained for 90 days; cleanup is an application operation, not manual SQL. Each aggregate preserves
|
|
sample count, schedulable min/max/average and storage-free minimum. Both tiers are bounded.
|
|
|
|
Trend responses calculate schedulable min/max/average/p50/p95, external min/max, storage min/max
|
|
and GPU pressure frequency. Fewer than three snapshots produce `INSUFFICIENT_DATA`; a latest stale
|
|
snapshot produces `STALE`.
|
|
|
|
Storage forecast requires at least 12 samples spanning at least one hour. Otherwise forecast status
|
|
is `INSUFFICIENT_DATA`. With sufficient evidence it reports observed bytes/day and, only for a
|
|
negative slope, a bounded days-to-current-zero estimate. It is a trend signal, not an automated
|
|
cleanup decision.
|
|
|
|
Capability headroom compares a measured `CapabilityResourceEnvelope` with known persisted node
|
|
capacity. A sustained mismatch becomes a warning; unknown/stale capacity is not treated as a
|
|
confident fit and no unlimited-scaling claim is made.
|