# Operations

The index an operator works from. Everything here already exists in the repository; this page is the
map, not a copy.

## Daily signals

| Where | What it tells you |
| --- | --- |
| `/api/v1/health/live` | The process is up. It touches nothing else, so it stays 200 when a dependency is down. |
| `/api/v1/health/ready` | The manifest set is loadable. |
| `/api/v1/version` | Which build is answering, and what it is compatible with. |
| Operations workspace | Capability health, capacity, SLO evaluations, active alerts. |
| Recovery dashboard | Newest verified backup, unprotected assets, recovery readiness. |

## The health model

`liveness`, `readiness`, `OBSERVABILITY_DEGRADED` and `CAPACITY_CONSTRAINED` each mean something
specific and different — see
[FAILURE_AND_HEALTH_MODEL.md](operations/FAILURE_AND_HEALTH_MODEL.md).

The one that surprises people: **a capacity rejection is a correct outcome.** The scheduler refusing
work it cannot place safely is the system working, not an incident.

## Runbooks

| Situation | Runbook |
| --- | --- |
| A node stops reporting | [RUNBOOK_NODE_OFFLINE.md](operations/RUNBOOK_NODE_OFFLINE.md) |
| A node is gone for good | [RUNBOOK_NODE_LOSS.md](operations/RUNBOOK_NODE_LOSS.md) |
| GPU pressure | [RUNBOOK_GPU_PRESSURE.md](operations/RUNBOOK_GPU_PRESSURE.md) |
| Gateway errors | [RUNBOOK_GATEWAY_ERRORS.md](operations/RUNBOOK_GATEWAY_ERRORS.md) |
| A runtime fails | [RUNBOOK_RUNTIME_FAILURE.md](operations/RUNBOOK_RUNTIME_FAILURE.md) |
| A backup fails | [RUNBOOK_BACKUP_FAILURE.md](operations/RUNBOOK_BACKUP_FAILURE.md) |
| Restoring the database | [RUNBOOK_DATABASE_RESTORE.md](operations/RUNBOOK_DATABASE_RESTORE.md) |
| Losing the control plane | [RUNBOOK_CONTROL_PLANE_LOSS.md](operations/RUNBOOK_CONTROL_PLANE_LOSS.md) |
| Full disaster recovery | [RUNBOOK_FULL_DR.md](operations/RUNBOOK_FULL_DR.md) |
| An artifact is lost | [RUNBOOK_ARTIFACT_LOSS.md](operations/RUNBOOK_ARTIFACT_LOSS.md) |
| A migration cutover is stuck | [RUNBOOK_MIGRATION_FAILURE.md](operations/RUNBOOK_MIGRATION_FAILURE.md) |
| Storage is low | [RUNBOOK_STORAGE_LOW.md](operations/RUNBOOK_STORAGE_LOW.md) |

## Backups and recovery

Backup classification, the immutable manifest, encryption and measured RPO/RTO are in
[BACKUP_AND_RECOVERY.md](architecture/BACKUP_AND_RECOVERY.md).

Three properties worth knowing before you need them:

- a backup interrupted mid-write can **never** become restore eligible — it is reconciled to
  `FAILED` / `MANIFEST_INCOMPLETE` on the next start;
- an unwritable destination fails closed with `DESTINATION_UNAVAILABLE` rather than producing a
  partial recovery point;
- **point-in-time recovery is `NOT_SUPPORTED`** in v1. Verified snapshot restore is what exists.

Losing `MODELFORGE_BACKUP_ENCRYPTION_KEY` makes every existing backup unrecoverable. Store it
outside the deployment it protects.

## Lifecycle and migration

- [MODEL_LIFECYCLE.md](product/MODEL_LIFECYCLE.md) — discover, verify, evaluate, approve, promote,
  canary, roll back, deprecate.
- [MIGRATION_RECOVERY.md](architecture/MIGRATION_RECOVERY.md) — why an interrupted cutover is
  reported and never auto-resolved.

Lifecycle **resolves** after an interruption; migration **reports**; backups **fail closed**. Each
subsystem declares which of the three it does — see
[ADR-0045](architecture/adr/0045-chaos-safe-reconciliation-semantics.md).

## Observability

[OBSERVABILITY.md](architecture/OBSERVABILITY.md). Monitoring is never a safety dependency: when
observability persistence is unavailable, serving and recovery continue on their own authoritative
state.

## Verifying the platform is sound

```bash
python scripts/m16_invariants.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge
```

Fifteen read-only checks over authoritative state. They never write, never call an external system,
and deliberately never read the observability tables — so they can answer "is state still sound?"
precisely when monitoring is the thing that failed.

## Chaos and soak

[CHAOS_TESTING.md](operations/CHAOS_TESTING.md), [SOAK_TESTING.md](operations/SOAK_TESTING.md),
[FAULT_INJECTION.md](operations/FAULT_INJECTION.md),
[RESTART_RECOVERY.md](operations/RESTART_RECOVERY.md).

## Deployment

- [INSTALLATION.md](INSTALLATION.md) — clean install
- [UPGRADE.md](UPGRADE.md) — upgrading, and rolling back
- [UNRAID_DEPLOYMENT.md](UNRAID_DEPLOYMENT.md) — the Unraid/GPU Node deployment specifically
- [NODE_AGENT.md](NODE_AGENT.md) — enrolling and running a compute node
- [CONFIGURATION.md](CONFIGURATION.md) — every setting, generated from the code
- [RELEASE_NOTES_v1.0.0.md](RELEASE_NOTES_v1.0.0.md) — release behavior and operator-facing changes from v1.0.0
  production cutover: what was measured, and what was left as a documented limitation