4.2 KiB
Fault injection
Where faults come from
ModelForge has no chaos endpoint and no way to run a command. A control plane that can be told to break itself is a control plane an attacker can tell to break itself, so every fault is injected from outside the product:
container runtime stop, start, restart, network disconnect, filesystem permissions
rehearsal seams the M12 lifecycle pause, the M15 restore interruption
real credentials revoked, rotated, malformed, wrongly scoped
real load the soak harness, running the actual capability path
A static test asserts the product exposes no /chaos, /shell, /exec, /command or /debug
route segment, and that subprocess use stays allowlisted to the recovery plane.
The shape every scenario has
preflight → baseline → inject → observe → assert → recover → verify invariants → cleanup
preflight refuses to start if an invariant is already violated: injecting a fault into a broken
platform measures nothing. verify invariants re-runs the full set afterwards, so a scenario that
survives its fault but corrupts state is reported as INVARIANT_VIOLATED rather than passing.
A scenario that cannot restore the platform reports RECOVERY_FAILED rather than leaving the
environment dirty and calling itself done.
Running it
python scripts/m16_chaos.py --list
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --scenario redis-outage --seed 20260827
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --all --report chaos.json
The database URL is required for executable scenarios and must name the non-owner runtime role;
the harness has no embedded or owner-credential fallback. --list does not connect to PostgreSQL.
Every run records its scenario id, seed, target, expected fault, observed result, recovery, cleanup outcome and the invariant results before and after. The seed is logged even where behaviour is deterministic, so a surprising result can be reproduced exactly.
Scenarios
| Key | Class | Subsystem | Expected |
|---|---|---|---|
redis-outage |
DEPENDENCY | Redis | control plane stays up, replays nothing |
postgres-restart |
DEPENDENCY | PostgreSQL | fails safely, recovers its pool, no duplicate work |
node-agent-disconnect |
DEPENDENCY | Node Agent | heartbeat stops, liveness degrades, one identity survives |
control-plane-restart-storm |
TRANSIENT | Control Plane | idempotent reconciliation, no authoritative drift |
invalid-auth-burst |
AUTH | Control Plane | identical refusal, no amplification, no enumeration signal |
backup-destination-unavailable |
CAPACITY | Backup storage | typed failure, serving unaffected |
Safety rules
Disposable targets only. The node-disconnect scenario enrols a throwaway agent rather than isolating GPU Node: GPU Node reaches the control plane over the LAN rather than the Compose network, so disconnecting a container would not interrupt it anyway — and isolating the production node to prove a point is exactly what a release gate must not do.
Nothing in the harness touches ExampleRAG, ExampleVision, Ollama, Plex or Tdarr. Their VRAM counts as external pressure and is observed; their processes are never stopped, paused, throttled or reconfigured.
The harness deletes its own fixtures. backup-destination-unavailable removes the chaos-* backup
sets it created; node-agent-disconnect tears down its agent, revokes its credential and disables
the node it enrolled.
When a scenario finds something
Both defects the harness found in M16 were real product defects rather than harness noise: an unwritable backup root escaping as an unhandled error, and a Node Agent logging a failure with no diagnostic. Each was fixed with a regression test before the scenario was re-run.
One scenario was itself wrong first: the node-disconnect measured GPU Node while disconnecting an unrelated container, and reported that the heartbeat kept advancing during the "outage". That contradiction is why the harness records what it observed rather than only whether it passed.
See docs/operations/CHAOS_TESTING.md for the fault matrix and
docs/security/CHAOS_SECURITY_GATE.md for the invariants.