100 lines
4.7 KiB
Markdown
100 lines
4.7 KiB
Markdown
# Operations
|
|
|
|
The index an operator works from. Everything here already exists in the repository; this page is the
|
|
map, not a copy.
|
|
|
|
## Daily signals
|
|
|
|
| Where | What it tells you |
|
|
| --- | --- |
|
|
| `/api/v1/health/live` | The process is up. It touches nothing else, so it stays 200 when a dependency is down. |
|
|
| `/api/v1/health/ready` | The manifest set is loadable. |
|
|
| `/api/v1/version` | Which build is answering, and what it is compatible with. |
|
|
| Operations workspace | Capability health, capacity, SLO evaluations, active alerts. |
|
|
| Recovery dashboard | Newest verified backup, unprotected assets, recovery readiness. |
|
|
|
|
## The health model
|
|
|
|
`liveness`, `readiness`, `OBSERVABILITY_DEGRADED` and `CAPACITY_CONSTRAINED` each mean something
|
|
specific and different — see
|
|
[FAILURE_AND_HEALTH_MODEL.md](operations/FAILURE_AND_HEALTH_MODEL.md).
|
|
|
|
The one that surprises people: **a capacity rejection is a correct outcome.** The scheduler refusing
|
|
work it cannot place safely is the system working, not an incident.
|
|
|
|
## Runbooks
|
|
|
|
| Situation | Runbook |
|
|
| --- | --- |
|
|
| A node stops reporting | [RUNBOOK_NODE_OFFLINE.md](operations/RUNBOOK_NODE_OFFLINE.md) |
|
|
| A node is gone for good | [RUNBOOK_NODE_LOSS.md](operations/RUNBOOK_NODE_LOSS.md) |
|
|
| GPU pressure | [RUNBOOK_GPU_PRESSURE.md](operations/RUNBOOK_GPU_PRESSURE.md) |
|
|
| Gateway errors | [RUNBOOK_GATEWAY_ERRORS.md](operations/RUNBOOK_GATEWAY_ERRORS.md) |
|
|
| A runtime fails | [RUNBOOK_RUNTIME_FAILURE.md](operations/RUNBOOK_RUNTIME_FAILURE.md) |
|
|
| A backup fails | [RUNBOOK_BACKUP_FAILURE.md](operations/RUNBOOK_BACKUP_FAILURE.md) |
|
|
| Restoring the database | [RUNBOOK_DATABASE_RESTORE.md](operations/RUNBOOK_DATABASE_RESTORE.md) |
|
|
| Losing the control plane | [RUNBOOK_CONTROL_PLANE_LOSS.md](operations/RUNBOOK_CONTROL_PLANE_LOSS.md) |
|
|
| Full disaster recovery | [RUNBOOK_FULL_DR.md](operations/RUNBOOK_FULL_DR.md) |
|
|
| An artifact is lost | [RUNBOOK_ARTIFACT_LOSS.md](operations/RUNBOOK_ARTIFACT_LOSS.md) |
|
|
| A migration cutover is stuck | [RUNBOOK_MIGRATION_FAILURE.md](operations/RUNBOOK_MIGRATION_FAILURE.md) |
|
|
| Storage is low | [RUNBOOK_STORAGE_LOW.md](operations/RUNBOOK_STORAGE_LOW.md) |
|
|
|
|
## Backups and recovery
|
|
|
|
Backup classification, the immutable manifest, encryption and measured RPO/RTO are in
|
|
[BACKUP_AND_RECOVERY.md](architecture/BACKUP_AND_RECOVERY.md).
|
|
|
|
Three properties worth knowing before you need them:
|
|
|
|
- a backup interrupted mid-write can **never** become restore eligible — it is reconciled to
|
|
`FAILED` / `MANIFEST_INCOMPLETE` on the next start;
|
|
- an unwritable destination fails closed with `DESTINATION_UNAVAILABLE` rather than producing a
|
|
partial recovery point;
|
|
- **point-in-time recovery is `NOT_SUPPORTED`** in v1. Verified snapshot restore is what exists.
|
|
|
|
Losing `MODELFORGE_BACKUP_ENCRYPTION_KEY` makes every existing backup unrecoverable. Store it
|
|
outside the deployment it protects.
|
|
|
|
## Lifecycle and migration
|
|
|
|
- [MODEL_LIFECYCLE.md](product/MODEL_LIFECYCLE.md) — discover, verify, evaluate, approve, promote,
|
|
canary, roll back, deprecate.
|
|
- [MIGRATION_RECOVERY.md](architecture/MIGRATION_RECOVERY.md) — why an interrupted cutover is
|
|
reported and never auto-resolved.
|
|
|
|
Lifecycle **resolves** after an interruption; migration **reports**; backups **fail closed**. Each
|
|
subsystem declares which of the three it does — see
|
|
[ADR-0045](architecture/adr/0045-chaos-safe-reconciliation-semantics.md).
|
|
|
|
## Observability
|
|
|
|
[OBSERVABILITY.md](architecture/OBSERVABILITY.md). Monitoring is never a safety dependency: when
|
|
observability persistence is unavailable, serving and recovery continue on their own authoritative
|
|
state.
|
|
|
|
## Verifying the platform is sound
|
|
|
|
```bash
|
|
python scripts/m16_invariants.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge
|
|
```
|
|
|
|
Fifteen read-only checks over authoritative state. They never write, never call an external system,
|
|
and deliberately never read the observability tables — so they can answer "is state still sound?"
|
|
precisely when monitoring is the thing that failed.
|
|
|
|
## Chaos and soak
|
|
|
|
[CHAOS_TESTING.md](operations/CHAOS_TESTING.md), [SOAK_TESTING.md](operations/SOAK_TESTING.md),
|
|
[FAULT_INJECTION.md](operations/FAULT_INJECTION.md),
|
|
[RESTART_RECOVERY.md](operations/RESTART_RECOVERY.md).
|
|
|
|
## Deployment
|
|
|
|
- [INSTALLATION.md](INSTALLATION.md) — clean install
|
|
- [UPGRADE.md](UPGRADE.md) — upgrading, and rolling back
|
|
- [UNRAID_DEPLOYMENT.md](UNRAID_DEPLOYMENT.md) — the Unraid/GPU Node deployment specifically
|
|
- [NODE_AGENT.md](NODE_AGENT.md) — enrolling and running a compute node
|
|
- [CONFIGURATION.md](CONFIGURATION.md) — every setting, generated from the code
|
|
- [RELEASE_NOTES_v1.0.0.md](RELEASE_NOTES_v1.0.0.md) — release behavior and operator-facing changes from v1.0.0
|
|
production cutover: what was measured, and what was left as a documented limitation
|