# Fault injection

## Where faults come from

ModelForge has no chaos endpoint and no way to run a command. A control plane that can be told to
break itself is a control plane an attacker can tell to break itself, so every fault is injected
from outside the product:

```text
container runtime   stop, start, restart, network disconnect, filesystem permissions
rehearsal seams     the M12 lifecycle pause, the M15 restore interruption
real credentials    revoked, rotated, malformed, wrongly scoped
real load           the soak harness, running the actual capability path
```

A static test asserts the product exposes no `/chaos`, `/shell`, `/exec`, `/command` or `/debug`
route segment, and that subprocess use stays allowlisted to the recovery plane.

## The shape every scenario has

```text
preflight → baseline → inject → observe → assert → recover → verify invariants → cleanup
```

`preflight` refuses to start if an invariant is already violated: injecting a fault into a broken
platform measures nothing. `verify invariants` re-runs the full set afterwards, so a scenario that
survives its fault but corrupts state is reported as `INVARIANT_VIOLATED` rather than passing.
A scenario that cannot restore the platform reports `RECOVERY_FAILED` rather than leaving the
environment dirty and calling itself done.

## Running it

```sh
python scripts/m16_chaos.py --list
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --scenario redis-outage --seed 20260827
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --all --report chaos.json
```

The database URL is required for executable scenarios and must name the non-owner runtime role;
the harness has no embedded or owner-credential fallback. `--list` does not connect to PostgreSQL.

Every run records its scenario id, seed, target, expected fault, observed result, recovery, cleanup
outcome and the invariant results before and after. The seed is logged even where behaviour is
deterministic, so a surprising result can be reproduced exactly.

## Scenarios

| Key | Class | Subsystem | Expected |
| --- | --- | --- | --- |
| `redis-outage` | DEPENDENCY | Redis | control plane stays up, replays nothing |
| `postgres-restart` | DEPENDENCY | PostgreSQL | fails safely, recovers its pool, no duplicate work |
| `node-agent-disconnect` | DEPENDENCY | Node Agent | heartbeat stops, liveness degrades, one identity survives |
| `control-plane-restart-storm` | TRANSIENT | Control Plane | idempotent reconciliation, no authoritative drift |
| `invalid-auth-burst` | AUTH | Control Plane | identical refusal, no amplification, no enumeration signal |
| `backup-destination-unavailable` | CAPACITY | Backup storage | typed failure, serving unaffected |

## Safety rules

Disposable targets only. The node-disconnect scenario enrols a throwaway agent rather than
isolating GPU Node: GPU Node reaches the control plane over the LAN rather than the Compose network, so
disconnecting a container would not interrupt it anyway — and isolating the production node to
prove a point is exactly what a release gate must not do.

Nothing in the harness touches ExampleRAG, ExampleVision, Ollama, Plex or Tdarr. Their VRAM counts as
external pressure and is observed; their processes are never stopped, paused, throttled or
reconfigured.

The harness deletes its own fixtures. `backup-destination-unavailable` removes the `chaos-*` backup
sets it created; `node-agent-disconnect` tears down its agent, revokes its credential and disables
the node it enrolled.

## When a scenario finds something

Both defects the harness found in M16 were real product defects rather than harness noise: an
unwritable backup root escaping as an unhandled error, and a Node Agent logging a failure with no
diagnostic. Each was fixed with a regression test before the scenario was re-run.

One scenario was itself wrong first: the node-disconnect measured GPU Node while disconnecting an
unrelated container, and reported that the heartbeat kept advancing during the "outage". That
contradiction is why the harness records what it observed rather than only whether it passed.

See `docs/operations/CHAOS_TESTING.md` for the fault matrix and
`docs/security/CHAOS_SECURITY_GATE.md` for the invariants.