# Runbook: runtime failure or crash loop

## Trigger

`RUNTIME_CRASH_LOOP`, capability-unavailable alerts, failed load/invoke/unload/health jobs or
runtime-success SLO degradation.

## Diagnose

Inspect bounded failure code, exact deployment/artifact/runtime identities, worker health, queue,
leases, residency and NVML freshness. Confirm offline execution and `trust_remote_code=false` remain
in effect. Distinguish model failure from node/GPU pressure.

## Act and recover

Do not send shell commands through ModelForge or expose the worker. Use the established isolated
runtime probe/LAB path for reproduction. Production rollback requires an approved M12 plan; M14 does
not perform it. Recovery requires successful typed health/invoke evidence, no orphan lease and an
explicit resolved alert.