# Chaos testing

## Fault matrix

Every subsystem, the failure modes it can present, how far the damage reaches, and the invariant
that must survive.

| Subsystem | Failure mode | Class | Blast radius | Recovery | Alert | Invariant |
| --- | --- | --- | --- | --- | --- | --- |
| Control Plane | process crash / restart | TRANSIENT | in-flight API requests | startup reconciliation | none by design | no authoritative drift |
| Control Plane | restart during backup | TRANSIENT | that backup only | interrupted backup fails closed | `BACKUP_FAILED` | `restore_requires_verified_backup` |
| PostgreSQL | unavailable | DEPENDENCY | all persistence | pool reconnects, no restart needed | `PLATFORM_API` SLO | `audit_chain_intact` |
| PostgreSQL | deadlock / serialisation failure | CONCURRENCY | one transaction | bounded retry | none | no lost update |
| PostgreSQL | latency | CAPACITY | request queueing | timeouts, typed errors | SLO burn | no connection explosion |
| Redis | unavailable | DEPENDENCY | queues, transient payloads | queues start empty, leases reconciled | dependency health | no duplicate serving job |
| Node Agent | network loss | DEPENDENCY | that node's capacity | heartbeat stale → offline → reconnect | `NODE_OFFLINE` | `single_node_identity` |
| Node Agent | credential revoked | AUTH | that node | re-enrol, identity preserved | `NODE_OFFLINE` | `revoked_credentials_stay_revoked` |
| Node Agent | restart storm | TRANSIENT | that node | same identity, no duplicate inventory | none | `single_active_node_credential` |
| Runtime Worker | crash during load/invoke/unload | TRANSIENT | one request | typed failure, generation fencing | `RUNTIME_CRASH_LOOP` | `no_orphan_serving_work` |
| Gateway | invalid credential burst | AUTH | none | identical refusal | none | no enumeration signal |
| Gateway | malformed or hostile input | SECURITY | none | typed 4xx | none | no execution, no 5xx |
| Scheduler | GPU pressure | CAPACITY | admission | reject or queue | `GPU_PRESSURE` | `no_stale_gpu_lease` |
| Scheduler | external VRAM spike | CAPACITY | admission | schedulable headroom shrinks | `CAPABILITY_CAPACITY` | external workloads untouched |
| GPU | telemetry stale | DEPENDENCY | placement decisions | admission blocked, never guessed | `GPU_PRESSURE` | no stale snapshot as truth |
| Artifact storage | location missing | CORRUPTION | one artifact set | exact-revision rehydration | `ARTIFACT_INTEGRITY` | `no_unsafe_artifact_promoted` |
| Artifact storage | disk low / full | CAPACITY | acquisition | capacity guard refuses first | `STORAGE_LOW` | no partial promotion |
| Backup storage | unwritable / absent | CAPACITY | that backup | typed `DESTINATION_UNAVAILABLE` | `BACKUP_FAILED` | serving unaffected |
| Backup storage | payload corrupted | CORRUPTION | that backup | verification refuses | `BACKUP_VERIFICATION_FAILED` | `restore_requires_verified_backup` |
| Lifecycle | operation interrupted | TRANSIENT | one LAB subject | rollback from immutable snapshot, exactly once | `ROLLBACK_FAILURE` | `lifecycle_commit_has_evidence` |
| Migration | cutover interrupted | CORRUPTION | one plan | never auto-resolved; operator reconciles | `MIGRATION_FAILURE` | `cutover_commit_has_validation` |
| Migration | backfill worker failure | TRANSIENT | one plan | checkpoint preserves completed work | `MIGRATION_FAILURE` | no mixed embedding space |
| Observability | persistence unavailable | DEPENDENCY | monitoring only | degraded gauge, serving continues | `modelforge_observability_degraded` | safety never depends on the alert store |
| Recovery | restore interrupted | TRANSIENT | that restore | manual intervention, resumable | `RESTORE_FAILED` | `restore_requires_verified_backup` |
| Credentials | rotation under load | AUTH | one client | old refused, new accepted | none | one usable secret |
| External | Hugging Face unavailable | DEPENDENCY | acquisition only | `ARTIFACT_REHYDRATION_BLOCKED` | none | no fallback source |
| External | ExampleRAG / ExampleVision down | EXTERNAL | none | not owned | none | never written to |

## Failure classes

```text
TRANSIENT     recoverable by retry or restart; state converges
PERSISTENT    requires operator action; state is safe but stuck
CORRUPTION    integrity is in question; the platform must refuse rather than proceed
CAPACITY      the safe answer is to reject work, not to risk exhaustion
DEPENDENCY    something ModelForge does not own is unavailable
AUTH          a credential is missing, wrong, revoked or out of scope
SECURITY      hostile input or an attempt to cross a boundary
CONCURRENCY   two operations race; exactly one must win
OPERATOR      a human action that must be gated, audited and reversible
```

A capacity rejection is not a failure. The scheduler refusing unsafe work is the system behaving
correctly, and the soak counts those separately from internal errors for exactly that reason.

## Invariants

Fifteen properties, each a query the platform runs against its own authoritative state. They are
read-only, never call an external system, and never depend on the observability database — so they
stay usable exactly when monitoring is the thing that failed.

```sh
python scripts/m16_invariants.py --label "before change"
```

See `docs/security/CHAOS_SECURITY_GATE.md` for the full list and what each one protects.

## Running the suite

```sh
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --all --report chaos.json
python scripts/m16_soak.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --minutes 45 --secret <secret>
python scripts/m16_invariants.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL"
```

Chaos scenarios and the soak may be run together deliberately — a backup taken while the soak is
generating real traffic is the M16 "backup under load" evidence — but a soak whose numbers are
meant to characterise steady state should not overlap an injected fault. When they do overlap, say
so in the report rather than presenting the mixed result as a clean baseline.