84 lines
4.2 KiB
Markdown
84 lines
4.2 KiB
Markdown
# Fault injection
|
|
|
|
## Where faults come from
|
|
|
|
ModelForge has no chaos endpoint and no way to run a command. A control plane that can be told to
|
|
break itself is a control plane an attacker can tell to break itself, so every fault is injected
|
|
from outside the product:
|
|
|
|
```text
|
|
container runtime stop, start, restart, network disconnect, filesystem permissions
|
|
rehearsal seams the M12 lifecycle pause, the M15 restore interruption
|
|
real credentials revoked, rotated, malformed, wrongly scoped
|
|
real load the soak harness, running the actual capability path
|
|
```
|
|
|
|
A static test asserts the product exposes no `/chaos`, `/shell`, `/exec`, `/command` or `/debug`
|
|
route segment, and that subprocess use stays allowlisted to the recovery plane.
|
|
|
|
## The shape every scenario has
|
|
|
|
```text
|
|
preflight → baseline → inject → observe → assert → recover → verify invariants → cleanup
|
|
```
|
|
|
|
`preflight` refuses to start if an invariant is already violated: injecting a fault into a broken
|
|
platform measures nothing. `verify invariants` re-runs the full set afterwards, so a scenario that
|
|
survives its fault but corrupts state is reported as `INVARIANT_VIOLATED` rather than passing.
|
|
A scenario that cannot restore the platform reports `RECOVERY_FAILED` rather than leaving the
|
|
environment dirty and calling itself done.
|
|
|
|
## Running it
|
|
|
|
```sh
|
|
python scripts/m16_chaos.py --list
|
|
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --scenario redis-outage --seed 20260827
|
|
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --all --report chaos.json
|
|
```
|
|
|
|
The database URL is required for executable scenarios and must name the non-owner runtime role;
|
|
the harness has no embedded or owner-credential fallback. `--list` does not connect to PostgreSQL.
|
|
|
|
Every run records its scenario id, seed, target, expected fault, observed result, recovery, cleanup
|
|
outcome and the invariant results before and after. The seed is logged even where behaviour is
|
|
deterministic, so a surprising result can be reproduced exactly.
|
|
|
|
## Scenarios
|
|
|
|
| Key | Class | Subsystem | Expected |
|
|
| --- | --- | --- | --- |
|
|
| `redis-outage` | DEPENDENCY | Redis | control plane stays up, replays nothing |
|
|
| `postgres-restart` | DEPENDENCY | PostgreSQL | fails safely, recovers its pool, no duplicate work |
|
|
| `node-agent-disconnect` | DEPENDENCY | Node Agent | heartbeat stops, liveness degrades, one identity survives |
|
|
| `control-plane-restart-storm` | TRANSIENT | Control Plane | idempotent reconciliation, no authoritative drift |
|
|
| `invalid-auth-burst` | AUTH | Control Plane | identical refusal, no amplification, no enumeration signal |
|
|
| `backup-destination-unavailable` | CAPACITY | Backup storage | typed failure, serving unaffected |
|
|
|
|
## Safety rules
|
|
|
|
Disposable targets only. The node-disconnect scenario enrols a throwaway agent rather than
|
|
isolating GPU Node: GPU Node reaches the control plane over the LAN rather than the Compose network, so
|
|
disconnecting a container would not interrupt it anyway — and isolating the production node to
|
|
prove a point is exactly what a release gate must not do.
|
|
|
|
Nothing in the harness touches ExampleRAG, ExampleVision, Ollama, Plex or Tdarr. Their VRAM counts as
|
|
external pressure and is observed; their processes are never stopped, paused, throttled or
|
|
reconfigured.
|
|
|
|
The harness deletes its own fixtures. `backup-destination-unavailable` removes the `chaos-*` backup
|
|
sets it created; `node-agent-disconnect` tears down its agent, revokes its credential and disables
|
|
the node it enrolled.
|
|
|
|
## When a scenario finds something
|
|
|
|
Both defects the harness found in M16 were real product defects rather than harness noise: an
|
|
unwritable backup root escaping as an unhandled error, and a Node Agent logging a failure with no
|
|
diagnostic. Each was fixed with a regression test before the scenario was re-run.
|
|
|
|
One scenario was itself wrong first: the node-disconnect measured GPU Node while disconnecting an
|
|
unrelated container, and reported that the heartbeat kept advancing during the "outage". That
|
|
contradiction is why the harness records what it observed rather than only whether it passed.
|
|
|
|
See `docs/operations/CHAOS_TESTING.md` for the fault matrix and
|
|
`docs/security/CHAOS_SECURITY_GATE.md` for the invariants.
|