Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,116 @@
+# GPU Scheduler Design
+
+## M6 reindex QoS
+
+The database claim order remains production, interactive, background, benchmark. ExampleRAG shadow
+jobs are stamped `background` from their scoped identity. External Ollama/Tdarr VRAM remains
+observed capacity pressure and is never managed by ModelForge.
+
+## M10 operational objective
+
+Admit capability requests on one finite GPU without letting ModelForge or experimental work disrupt
+production or unmanaged workloads.
+
+## Priority classes
+
+1. production
+2. interactive
+3. background
+4. benchmark
+5. maintenance
+
+Priority is not identical to preemption support. Some runtimes/jobs cannot be safely interrupted and must be drained instead.
+
+## Residency policies
+
+- `always_warm`
+- `keep_warm` with TTL
+- `load_on_demand`
+- `exclusive` and `lab_only` remain future policy values, not M5 serving choices.
+
+## Resource leases
+
+Inference begins only when the scheduler issues a lease for required accelerator resources.
+
+Lease evaluation considers:
+
+- target accelerator compatibility;
+- latest total NVML usage, including unmanaged processes;
+- reserved safety margin;
+- deployment resource envelope;
+- current leases;
+- queue priority;
+- model load/sleep/unload cost;
+- production eligibility and exact measured compatibility.
+
+M5 never silently falls back to CPU, another model, a workstation or cloud inference.
+
+The pre-M10 budget was:
+
+```text
+total VRAM - external usage - ModelForge residency - active leases - 1 GiB reserve
+```
+
+Rows are aggregated once per accelerator. M10's authoritative invariant, dynamic reserve,
+multi-resident placement, hysteresis and typed failure behavior are specified in
+`docs/architecture/ADVANCED_GPU_SCHEDULER.md`. GPU Node is eligible; the DESKTOP workstation is explicitly
+rejected because `production_eligible=false`. The accelerator row is locked while a lease is granted.
+
+## Resource envelope
+
+Memory is not a single static number. Record measured profiles across:
+
+- context length;
+- concurrency;
+- batch size;
+- idle vs peak VRAM;
+- runtime version/configuration.
+
+## M5 embedding decision
+
+```text
+Request: rag.embedding@1
+Deployment: stable and cold
+Envelope: 1,216,348,160 process-reserved bytes
+Policy: KEEP_WARM 901s, concurrency 1, queue 16
+
+Decision: lock GPU Node GPU, account unmanaged VRAM and reserve, grant a lease, load once, run a
+functional health check, invoke, release the request lease and retain residency.
+```
+
+## Failure classes
+
+- MODEL_LOAD_FAILED
+- GPU_OOM
+- RUNTIME_CRASH
+- TIMEOUT
+- HEALTHCHECK_FAILED
+- INVALID_OUTPUT
+- ARTIFACT_CORRUPT
+- CAPABILITY_UNAVAILABLE
+- DRIVER_ERROR
+- SCHEDULER_STATE_STALE
+- INSUFFICIENT_SCHEDULABLE_VRAM
+- EXTERNAL_GPU_PRESSURE
+- RESIDENCY_CONFLICT
+- EVICTION_NOT_ALLOWED
+- DEADLINE_CANNOT_BE_MET
+- LEASE_CONFLICT
+- GPU_MEMORY_NOT_RECLAIMED
+- QUEUE_FULL
+- LEASE_TIMEOUT
+
+Failures must produce structured events. No silent retry loop may hide prolonged capability degradation.
+
+## M14 operational telemetry
+
+The Operations plane samples the existing NVML/scheduler truth once per minute. A retained snapshot
+separates total, NVML-observed, ModelForge resident, active leased, reserve, external and schedulable
+bytes and carries observed/received freshness. `GPU_PRESSURE` uses the existing hysteresis state;
+`CAPABILITY_CAPACITY` compares a measured production ResourceEnvelope with known headroom. Neither
+alert authorizes control of Ollama, Plex, Tdarr or another external GPU process. See
+`docs/architecture/CAPACITY_TRENDS.md` and `docs/operations/RUNBOOK_GPU_PRESSURE.md`.
+
+Runtime adapters own model load, crash and invalid-output detection. The scheduler owns VRAM
+reservation, OOM recovery and lease release. The gateway owns bounded retries and only contract-
+approved fallback/degradation. The control plane blocks corrupt artifacts and records state changes.