ModelForge/docs/operations/GPU_SCHEDULER.md

GPU Scheduler Design

M6 reindex QoS

The database claim order remains production, interactive, background, benchmark. ExampleRAG shadow jobs are stamped background from their scoped identity. External Ollama/Tdarr VRAM remains observed capacity pressure and is never managed by ModelForge.

M10 operational objective

Admit capability requests on one finite GPU without letting ModelForge or experimental work disrupt production or unmanaged workloads.

Priority classes

1. production
2. interactive
3. background
4. benchmark
5. maintenance

Priority is not identical to preemption support. Some runtimes/jobs cannot be safely interrupted and must be drained instead.

Residency policies

• always_warm
• keep_warm with TTL
• load_on_demand
• exclusive and lab_only remain future policy values, not M5 serving choices.

Resource leases

Inference begins only when the scheduler issues a lease for required accelerator resources.

Lease evaluation considers:

• target accelerator compatibility;
• latest total NVML usage, including unmanaged processes;
• reserved safety margin;
• deployment resource envelope;
• current leases;
• queue priority;
• model load/sleep/unload cost;
• production eligibility and exact measured compatibility.

M5 never silently falls back to CPU, another model, a workstation or cloud inference.

The pre-M10 budget was:

total VRAM - external usage - ModelForge residency - active leases - 1 GiB reserve

Rows are aggregated once per accelerator. M10's authoritative invariant, dynamic reserve, multi-resident placement, hysteresis and typed failure behavior are specified in docs/architecture/ADVANCED_GPU_SCHEDULER.md. GPU Node is eligible; the DESKTOP workstation is explicitly rejected because production_eligible=false. The accelerator row is locked while a lease is granted.

Resource envelope

Memory is not a single static number. Record measured profiles across:

• context length;
• concurrency;
• batch size;
• idle vs peak VRAM;
• runtime version/configuration.

M5 embedding decision

Request: rag.embedding@1
Deployment: stable and cold
Envelope: 1,216,348,160 process-reserved bytes
Policy: KEEP_WARM 901s, concurrency 1, queue 16

Decision: lock GPU Node GPU, account unmanaged VRAM and reserve, grant a lease, load once, run a
functional health check, invoke, release the request lease and retain residency.

Failure classes

• MODEL_LOAD_FAILED
• GPU_OOM
• RUNTIME_CRASH
• TIMEOUT
• HEALTHCHECK_FAILED
• INVALID_OUTPUT
• ARTIFACT_CORRUPT
• CAPABILITY_UNAVAILABLE
• DRIVER_ERROR
• SCHEDULER_STATE_STALE
• INSUFFICIENT_SCHEDULABLE_VRAM
• EXTERNAL_GPU_PRESSURE
• RESIDENCY_CONFLICT
• EVICTION_NOT_ALLOWED
• DEADLINE_CANNOT_BE_MET
• LEASE_CONFLICT
• GPU_MEMORY_NOT_RECLAIMED
• QUEUE_FULL
• LEASE_TIMEOUT

Failures must produce structured events. No silent retry loop may hide prolonged capability degradation.

M14 operational telemetry

The Operations plane samples the existing NVML/scheduler truth once per minute. A retained snapshot separates total, NVML-observed, ModelForge resident, active leased, reserve, external and schedulable bytes and carries observed/received freshness. GPU_PRESSURE uses the existing hysteresis state; CAPABILITY_CAPACITY compares a measured production ResourceEnvelope with known headroom. Neither alert authorizes control of Ollama, Plex, Tdarr or another external GPU process. See docs/architecture/CAPACITY_TRENDS.md and docs/operations/RUNBOOK_GPU_PRESSURE.md.

Runtime adapters own model load, crash and invalid-output detection. The scheduler owns VRAM reservation, OOM recovery and lease release. The gateway owns bounded retries and only contract- approved fallback/degradation. The control plane blocks corrupt artifacts and records state changes.