4.7 KiB
Operations
The index an operator works from. Everything here already exists in the repository; this page is the map, not a copy.
Daily signals
| Where | What it tells you |
|---|---|
/api/v1/health/live |
The process is up. It touches nothing else, so it stays 200 when a dependency is down. |
/api/v1/health/ready |
The manifest set is loadable. |
/api/v1/version |
Which build is answering, and what it is compatible with. |
| Operations workspace | Capability health, capacity, SLO evaluations, active alerts. |
| Recovery dashboard | Newest verified backup, unprotected assets, recovery readiness. |
The health model
liveness, readiness, OBSERVABILITY_DEGRADED and CAPACITY_CONSTRAINED each mean something
specific and different — see
FAILURE_AND_HEALTH_MODEL.md.
The one that surprises people: a capacity rejection is a correct outcome. The scheduler refusing work it cannot place safely is the system working, not an incident.
Runbooks
| Situation | Runbook |
|---|---|
| A node stops reporting | RUNBOOK_NODE_OFFLINE.md |
| A node is gone for good | RUNBOOK_NODE_LOSS.md |
| GPU pressure | RUNBOOK_GPU_PRESSURE.md |
| Gateway errors | RUNBOOK_GATEWAY_ERRORS.md |
| A runtime fails | RUNBOOK_RUNTIME_FAILURE.md |
| A backup fails | RUNBOOK_BACKUP_FAILURE.md |
| Restoring the database | RUNBOOK_DATABASE_RESTORE.md |
| Losing the control plane | RUNBOOK_CONTROL_PLANE_LOSS.md |
| Full disaster recovery | RUNBOOK_FULL_DR.md |
| An artifact is lost | RUNBOOK_ARTIFACT_LOSS.md |
| A migration cutover is stuck | RUNBOOK_MIGRATION_FAILURE.md |
| Storage is low | RUNBOOK_STORAGE_LOW.md |
Backups and recovery
Backup classification, the immutable manifest, encryption and measured RPO/RTO are in BACKUP_AND_RECOVERY.md.
Three properties worth knowing before you need them:
- a backup interrupted mid-write can never become restore eligible — it is reconciled to
FAILED/MANIFEST_INCOMPLETEon the next start; - an unwritable destination fails closed with
DESTINATION_UNAVAILABLErather than producing a partial recovery point; - point-in-time recovery is
NOT_SUPPORTEDin v1. Verified snapshot restore is what exists.
Losing MODELFORGE_BACKUP_ENCRYPTION_KEY makes every existing backup unrecoverable. Store it
outside the deployment it protects.
Lifecycle and migration
- MODEL_LIFECYCLE.md — discover, verify, evaluate, approve, promote, canary, roll back, deprecate.
- MIGRATION_RECOVERY.md — why an interrupted cutover is reported and never auto-resolved.
Lifecycle resolves after an interruption; migration reports; backups fail closed. Each subsystem declares which of the three it does — see ADR-0045.
Observability
OBSERVABILITY.md. Monitoring is never a safety dependency: when observability persistence is unavailable, serving and recovery continue on their own authoritative state.
Verifying the platform is sound
python scripts/m16_invariants.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge
Fifteen read-only checks over authoritative state. They never write, never call an external system, and deliberately never read the observability tables — so they can answer "is state still sound?" precisely when monitoring is the thing that failed.
Chaos and soak
CHAOS_TESTING.md, SOAK_TESTING.md, FAULT_INJECTION.md, RESTART_RECOVERY.md.
Deployment
- INSTALLATION.md — clean install
- UPGRADE.md — upgrading, and rolling back
- UNRAID_DEPLOYMENT.md — the Unraid/GPU Node deployment specifically
- NODE_AGENT.md — enrolling and running a compute node
- CONFIGURATION.md — every setting, generated from the code
- RELEASE_NOTES_v1.0.0.md — release behavior and operator-facing changes from v1.0.0 production cutover: what was measured, and what was left as a documented limitation