20 lines
797 B
Markdown
20 lines
797 B
Markdown
# Runbook: runtime failure or crash loop
|
|
|
|
## Trigger
|
|
|
|
`RUNTIME_CRASH_LOOP`, capability-unavailable alerts, failed load/invoke/unload/health jobs or
|
|
runtime-success SLO degradation.
|
|
|
|
## Diagnose
|
|
|
|
Inspect bounded failure code, exact deployment/artifact/runtime identities, worker health, queue,
|
|
leases, residency and NVML freshness. Confirm offline execution and `trust_remote_code=false` remain
|
|
in effect. Distinguish model failure from node/GPU pressure.
|
|
|
|
## Act and recover
|
|
|
|
Do not send shell commands through ModelForge or expose the worker. Use the established isolated
|
|
runtime probe/LAB path for reproduction. Production rollback requires an approved M12 plan; M14 does
|
|
not perform it. Recovery requires successful typed health/invoke evidence, no orphan lease and an
|
|
explicit resolved alert.
|