Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,32 @@
+# Capacity snapshots and trends
+
+The M14 minute poll samples existing Node Agent/NVML, scheduler, residency, lease and storage truth.
+It does not sample raw NVML at high frequency. Each enabled node/accelerator snapshot stores:
+
+- observed and received timestamps, freshness and availability;
+- GPU total, observed, external, ModelForge resident, leased, reserve and schedulable bytes;
+- scheduler pressure state;
+- total/available system RAM;
+- total/free storage.
+
+External GPU use is `max(0, NVML observed - measured ModelForge residency)`. Schedulable VRAM uses
+the existing scheduler invariant and never grants authority to manage external processes.
+
+## Aggregation and retention
+
+Raw minute snapshots are retained for seven days. Older rows are aggregated into hourly buckets and
+retained for 90 days; cleanup is an application operation, not manual SQL. Each aggregate preserves
+sample count, schedulable min/max/average and storage-free minimum. Both tiers are bounded.
+
+Trend responses calculate schedulable min/max/average/p50/p95, external min/max, storage min/max
+and GPU pressure frequency. Fewer than three snapshots produce `INSUFFICIENT_DATA`; a latest stale
+snapshot produces `STALE`.
+
+Storage forecast requires at least 12 samples spanning at least one hour. Otherwise forecast status
+is `INSUFFICIENT_DATA`. With sufficient evidence it reports observed bytes/day and, only for a
+negative slope, a bounded days-to-current-zero estimate. It is a trend signal, not an automated
+cleanup decision.
+
+Capability headroom compares a measured `CapabilityResourceEnvelope` with known persisted node
+capacity. A sustained mismatch becomes a warning; unknown/stale capacity is not treated as a
+confident fit and no unlimited-scaling claim is made.