Jens/ModelForge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial public ModelForge release

Browse Source
This commit is contained in:
Jens
2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/RUNBOOK_NODE_OFFLINE.md
+21
View File
@@ -0,0 +1,21 @@
# Runbook: production node offline or stale
## Trigger
`NODE_OFFLINE` is pending/firing, or GPU Node freshness is `STALE`/over 90 seconds.
## Diagnose
1. Confirm API and Operations view timestamps; do not infer current health from an old snapshot.
2. Inspect Node Agent liveness, last server-received heartbeat/telemetry and credential state.
3. Check network/TLS reachability from GPU Node outward and the Node Agent service logs.
4. Review the incident for capability alerts correlated as `DOWNSTREAM`.
5. Verify ExampleRAG's external Ollama production baseline independently.
## Act and recover
Acknowledge with an operator reason. Restart the Node Agent only through the established host
runbook and only after checking unrelated workloads; never stop the GPU Node host. Do not promote a
fallback model. Recovery requires new server-received telemetry, online liveness, resolved alerts
and a retained timeline. Escalate if the credential is rejected, inventory identity changed or
freshness oscillates.