ModelForge/docs/operations/RUNBOOK_RUNTIME_FAILURE.md

Runbook: runtime failure or crash loop

Trigger

RUNTIME_CRASH_LOOP, capability-unavailable alerts, failed load/invoke/unload/health jobs or runtime-success SLO degradation.

Diagnose

Inspect bounded failure code, exact deployment/artifact/runtime identities, worker health, queue, leases, residency and NVML freshness. Confirm offline execution and trust_remote_code=false remain in effect. Distinguish model failure from node/GPU pressure.

Act and recover

Do not send shell commands through ModelForge or expose the worker. Use the established isolated runtime probe/LAB path for reproduction. Production rollback requires an approved M12 plan; M14 does not perform it. Recovery requires successful typed health/invoke evidence, no orphan lease and an explicit resolved alert.