Initial public ModelForge release
This commit is contained in:
Jens
@@ -0,0 +1,99 @@
|
||||
# Operations
|
||||
|
||||
The index an operator works from. Everything here already exists in the repository; this page is the
|
||||
map, not a copy.
|
||||
|
||||
## Daily signals
|
||||
|
||||
| Where | What it tells you |
|
||||
| --- | --- |
|
||||
| `/api/v1/health/live` | The process is up. It touches nothing else, so it stays 200 when a dependency is down. |
|
||||
| `/api/v1/health/ready` | The manifest set is loadable. |
|
||||
| `/api/v1/version` | Which build is answering, and what it is compatible with. |
|
||||
| Operations workspace | Capability health, capacity, SLO evaluations, active alerts. |
|
||||
| Recovery dashboard | Newest verified backup, unprotected assets, recovery readiness. |
|
||||
|
||||
## The health model
|
||||
|
||||
`liveness`, `readiness`, `OBSERVABILITY_DEGRADED` and `CAPACITY_CONSTRAINED` each mean something
|
||||
specific and different — see
|
||||
[FAILURE_AND_HEALTH_MODEL.md](operations/FAILURE_AND_HEALTH_MODEL.md).
|
||||
|
||||
The one that surprises people: **a capacity rejection is a correct outcome.** The scheduler refusing
|
||||
work it cannot place safely is the system working, not an incident.
|
||||
|
||||
## Runbooks
|
||||
|
||||
| Situation | Runbook |
|
||||
| --- | --- |
|
||||
| A node stops reporting | [RUNBOOK_NODE_OFFLINE.md](operations/RUNBOOK_NODE_OFFLINE.md) |
|
||||
| A node is gone for good | [RUNBOOK_NODE_LOSS.md](operations/RUNBOOK_NODE_LOSS.md) |
|
||||
| GPU pressure | [RUNBOOK_GPU_PRESSURE.md](operations/RUNBOOK_GPU_PRESSURE.md) |
|
||||
| Gateway errors | [RUNBOOK_GATEWAY_ERRORS.md](operations/RUNBOOK_GATEWAY_ERRORS.md) |
|
||||
| A runtime fails | [RUNBOOK_RUNTIME_FAILURE.md](operations/RUNBOOK_RUNTIME_FAILURE.md) |
|
||||
| A backup fails | [RUNBOOK_BACKUP_FAILURE.md](operations/RUNBOOK_BACKUP_FAILURE.md) |
|
||||
| Restoring the database | [RUNBOOK_DATABASE_RESTORE.md](operations/RUNBOOK_DATABASE_RESTORE.md) |
|
||||
| Losing the control plane | [RUNBOOK_CONTROL_PLANE_LOSS.md](operations/RUNBOOK_CONTROL_PLANE_LOSS.md) |
|
||||
| Full disaster recovery | [RUNBOOK_FULL_DR.md](operations/RUNBOOK_FULL_DR.md) |
|
||||
| An artifact is lost | [RUNBOOK_ARTIFACT_LOSS.md](operations/RUNBOOK_ARTIFACT_LOSS.md) |
|
||||
| A migration cutover is stuck | [RUNBOOK_MIGRATION_FAILURE.md](operations/RUNBOOK_MIGRATION_FAILURE.md) |
|
||||
| Storage is low | [RUNBOOK_STORAGE_LOW.md](operations/RUNBOOK_STORAGE_LOW.md) |
|
||||
|
||||
## Backups and recovery
|
||||
|
||||
Backup classification, the immutable manifest, encryption and measured RPO/RTO are in
|
||||
[BACKUP_AND_RECOVERY.md](architecture/BACKUP_AND_RECOVERY.md).
|
||||
|
||||
Three properties worth knowing before you need them:
|
||||
|
||||
- a backup interrupted mid-write can **never** become restore eligible — it is reconciled to
|
||||
`FAILED` / `MANIFEST_INCOMPLETE` on the next start;
|
||||
- an unwritable destination fails closed with `DESTINATION_UNAVAILABLE` rather than producing a
|
||||
partial recovery point;
|
||||
- **point-in-time recovery is `NOT_SUPPORTED`** in v1. Verified snapshot restore is what exists.
|
||||
|
||||
Losing `MODELFORGE_BACKUP_ENCRYPTION_KEY` makes every existing backup unrecoverable. Store it
|
||||
outside the deployment it protects.
|
||||
|
||||
## Lifecycle and migration
|
||||
|
||||
- [MODEL_LIFECYCLE.md](product/MODEL_LIFECYCLE.md) — discover, verify, evaluate, approve, promote,
|
||||
canary, roll back, deprecate.
|
||||
- [MIGRATION_RECOVERY.md](architecture/MIGRATION_RECOVERY.md) — why an interrupted cutover is
|
||||
reported and never auto-resolved.
|
||||
|
||||
Lifecycle **resolves** after an interruption; migration **reports**; backups **fail closed**. Each
|
||||
subsystem declares which of the three it does — see
|
||||
[ADR-0045](architecture/adr/0045-chaos-safe-reconciliation-semantics.md).
|
||||
|
||||
## Observability
|
||||
|
||||
[OBSERVABILITY.md](architecture/OBSERVABILITY.md). Monitoring is never a safety dependency: when
|
||||
observability persistence is unavailable, serving and recovery continue on their own authoritative
|
||||
state.
|
||||
|
||||
## Verifying the platform is sound
|
||||
|
||||
```bash
|
||||
python scripts/m16_invariants.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge
|
||||
```
|
||||
|
||||
Fifteen read-only checks over authoritative state. They never write, never call an external system,
|
||||
and deliberately never read the observability tables — so they can answer "is state still sound?"
|
||||
precisely when monitoring is the thing that failed.
|
||||
|
||||
## Chaos and soak
|
||||
|
||||
[CHAOS_TESTING.md](operations/CHAOS_TESTING.md), [SOAK_TESTING.md](operations/SOAK_TESTING.md),
|
||||
[FAULT_INJECTION.md](operations/FAULT_INJECTION.md),
|
||||
[RESTART_RECOVERY.md](operations/RESTART_RECOVERY.md).
|
||||
|
||||
## Deployment
|
||||
|
||||
- [INSTALLATION.md](INSTALLATION.md) — clean install
|
||||
- [UPGRADE.md](UPGRADE.md) — upgrading, and rolling back
|
||||
- [UNRAID_DEPLOYMENT.md](UNRAID_DEPLOYMENT.md) — the Unraid/GPU Node deployment specifically
|
||||
- [NODE_AGENT.md](NODE_AGENT.md) — enrolling and running a compute node
|
||||
- [CONFIGURATION.md](CONFIGURATION.md) — every setting, generated from the code
|
||||
- [RELEASE_NOTES_v1.0.0.md](RELEASE_NOTES_v1.0.0.md) — release behavior and operator-facing changes from v1.0.0
|
||||
production cutover: what was measured, and what was left as a documented limitation
|
||||
Reference in New Issue
Block a user