Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,19 @@
+# Runbook: runtime failure or crash loop
+
+## Trigger
+
+`RUNTIME_CRASH_LOOP`, capability-unavailable alerts, failed load/invoke/unload/health jobs or
+runtime-success SLO degradation.
+
+## Diagnose
+
+Inspect bounded failure code, exact deployment/artifact/runtime identities, worker health, queue,
+leases, residency and NVML freshness. Confirm offline execution and `trust_remote_code=false` remain
+in effect. Distinguish model failure from node/GPU pressure.
+
+## Act and recover
+
+Do not send shell commands through ModelForge or expose the worker. Use the established isolated
+runtime probe/LAB path for reproduction. Production rollback requires an approved M12 plan; M14 does
+not perform it. Recovery requires successful typed health/invoke evidence, no orphan lease and an
+explicit resolved alert.