Initial public ModelForge release

docs/architecture/ADVANCED_GPU_SCHEDULER.md

@@ -0,0 +1,61 @@
+# Advanced GPU Scheduler
+
+M10 turns the single-GPU admission boundary into one deterministic planner used by dry-runs and
+real requests. The planner consumes current NVML use, managed residency, only the unmaterialized
+part of active leases, authoritative envelopes, health, queue state, priority and versioned policy.
+
+## Accounting invariant
+
+NVML is the physical truth and already includes loaded ModelForge models:
+
+```text
+physical_used = max(observed_nvml_used, attributed_managed_resident)
+unmanaged_external = max(0, observed_nvml_used - attributed_managed_resident)
+future_lease = max(0, reserved - materialized)
+schedulable = max(0, total - physical_used - future_leases - dynamic_reserve)
+```
+
+This deliberately does not subtract residency twice. Attribution is `KNOWN`, `ESTIMATED` or
+`UNKNOWN`; missing or stale telemetry is `UNKNOWN` and has zero schedulable capacity. The invariant
+delta is exposed for diagnosis without permitting negative external use.
+
+The active `m10-v1` policy uses the maximum of 1 GiB, 5% total VRAM and a 256 MiB runtime margin.
+Deployment admission uses measured peak plus the maximum of 128 MiB or 10% measured peak. These
+values are one versioned policy, not dispersed constants.
+
+## PlacementPlan
+
+Every plan records capability, node and accelerator, eligible state, residents, unmanaged use,
+reserve, requirement, priority, evictions, before/after headroom, reason codes and a deterministic
+fingerprint. Verdicts are `ADMIT`, `ADMIT_AFTER_EVICTION`, `QUEUE`, `REJECT_CAPACITY`,
+`REJECT_HEALTH` and `REJECT_POLICY`. Recent history is bounded to 500 records.
+
+Pressure worsens immediately through `NORMAL`, `ELEVATED`, `HIGH`, `CRITICAL`. Recovery requires a
+separate threshold and 30 seconds of stable evidence. At HIGH/ELEVATED, new background/LAB loads
+wait; at CRITICAL, new admissions fail with `EXTERNAL_GPU_PRESSURE`. ModelForge never changes an
+unmanaged process.
+
+## Operations and security
+
+Operator controls are limited to dry-run, policy revision/LAB pause, managed residency policy,
+drain and idle unload. There is no shell, Docker socket, SSH or GPU-PID control. Ordinary clients
+select only a capability. Scheduler evidence contains bounded shapes and hashes, never content.
+
+M11 validates the existing scheduler rather than adding a project shortcut. ExampleVision reference
+index requests use a background-bound client, while card and speech requests use interactive-bound
+clients. During a 40-reference indexing continuation, ten interactive vision requests all completed.
+Vision and ASR also ran under observed external GPU pressure without OOM; final unload returned
+managed residency and leases to zero.
+
+
+## M16 pressure behaviour
+
+Under GPU pressure the scheduler rejects or queues rather than risking an out-of-memory failure. A
+capacity rejection is a correct outcome, not a failure, and the soak counts those separately from
+internal errors for exactly that reason.
+
+External VRAM is measured as pressure and never reclaimed: Ollama, Plex, Tdarr and any other
+unmanaged process are observed, never stopped, paused, throttled or reconfigured.
+
+Stale telemetry blocks admission rather than being extrapolated, and `no_stale_gpu_lease` asserts
+no expired lease is still presented as a live reservation.