117 lines
3.7 KiB
Markdown
117 lines
3.7 KiB
Markdown
# GPU Scheduler Design
|
|
|
|
## M6 reindex QoS
|
|
|
|
The database claim order remains production, interactive, background, benchmark. ExampleRAG shadow
|
|
jobs are stamped `background` from their scoped identity. External Ollama/Tdarr VRAM remains
|
|
observed capacity pressure and is never managed by ModelForge.
|
|
|
|
## M10 operational objective
|
|
|
|
Admit capability requests on one finite GPU without letting ModelForge or experimental work disrupt
|
|
production or unmanaged workloads.
|
|
|
|
## Priority classes
|
|
|
|
1. production
|
|
2. interactive
|
|
3. background
|
|
4. benchmark
|
|
5. maintenance
|
|
|
|
Priority is not identical to preemption support. Some runtimes/jobs cannot be safely interrupted and must be drained instead.
|
|
|
|
## Residency policies
|
|
|
|
- `always_warm`
|
|
- `keep_warm` with TTL
|
|
- `load_on_demand`
|
|
- `exclusive` and `lab_only` remain future policy values, not M5 serving choices.
|
|
|
|
## Resource leases
|
|
|
|
Inference begins only when the scheduler issues a lease for required accelerator resources.
|
|
|
|
Lease evaluation considers:
|
|
|
|
- target accelerator compatibility;
|
|
- latest total NVML usage, including unmanaged processes;
|
|
- reserved safety margin;
|
|
- deployment resource envelope;
|
|
- current leases;
|
|
- queue priority;
|
|
- model load/sleep/unload cost;
|
|
- production eligibility and exact measured compatibility.
|
|
|
|
M5 never silently falls back to CPU, another model, a workstation or cloud inference.
|
|
|
|
The pre-M10 budget was:
|
|
|
|
```text
|
|
total VRAM - external usage - ModelForge residency - active leases - 1 GiB reserve
|
|
```
|
|
|
|
Rows are aggregated once per accelerator. M10's authoritative invariant, dynamic reserve,
|
|
multi-resident placement, hysteresis and typed failure behavior are specified in
|
|
`docs/architecture/ADVANCED_GPU_SCHEDULER.md`. GPU Node is eligible; the DESKTOP workstation is explicitly
|
|
rejected because `production_eligible=false`. The accelerator row is locked while a lease is granted.
|
|
|
|
## Resource envelope
|
|
|
|
Memory is not a single static number. Record measured profiles across:
|
|
|
|
- context length;
|
|
- concurrency;
|
|
- batch size;
|
|
- idle vs peak VRAM;
|
|
- runtime version/configuration.
|
|
|
|
## M5 embedding decision
|
|
|
|
```text
|
|
Request: rag.embedding@1
|
|
Deployment: stable and cold
|
|
Envelope: 1,216,348,160 process-reserved bytes
|
|
Policy: KEEP_WARM 901s, concurrency 1, queue 16
|
|
|
|
Decision: lock GPU Node GPU, account unmanaged VRAM and reserve, grant a lease, load once, run a
|
|
functional health check, invoke, release the request lease and retain residency.
|
|
```
|
|
|
|
## Failure classes
|
|
|
|
- MODEL_LOAD_FAILED
|
|
- GPU_OOM
|
|
- RUNTIME_CRASH
|
|
- TIMEOUT
|
|
- HEALTHCHECK_FAILED
|
|
- INVALID_OUTPUT
|
|
- ARTIFACT_CORRUPT
|
|
- CAPABILITY_UNAVAILABLE
|
|
- DRIVER_ERROR
|
|
- SCHEDULER_STATE_STALE
|
|
- INSUFFICIENT_SCHEDULABLE_VRAM
|
|
- EXTERNAL_GPU_PRESSURE
|
|
- RESIDENCY_CONFLICT
|
|
- EVICTION_NOT_ALLOWED
|
|
- DEADLINE_CANNOT_BE_MET
|
|
- LEASE_CONFLICT
|
|
- GPU_MEMORY_NOT_RECLAIMED
|
|
- QUEUE_FULL
|
|
- LEASE_TIMEOUT
|
|
|
|
Failures must produce structured events. No silent retry loop may hide prolonged capability degradation.
|
|
|
|
## M14 operational telemetry
|
|
|
|
The Operations plane samples the existing NVML/scheduler truth once per minute. A retained snapshot
|
|
separates total, NVML-observed, ModelForge resident, active leased, reserve, external and schedulable
|
|
bytes and carries observed/received freshness. `GPU_PRESSURE` uses the existing hysteresis state;
|
|
`CAPABILITY_CAPACITY` compares a measured production ResourceEnvelope with known headroom. Neither
|
|
alert authorizes control of Ollama, Plex, Tdarr or another external GPU process. See
|
|
`docs/architecture/CAPACITY_TRENDS.md` and `docs/operations/RUNBOOK_GPU_PRESSURE.md`.
|
|
|
|
Runtime adapters own model load, crash and invalid-output detection. The scheduler owns VRAM
|
|
reservation, OOM recovery and lease release. The gateway owns bounded retries and only contract-
|
|
approved fallback/degradation. The control plane blocks corrupt artifacts and records state changes.
|