Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,83 @@
+# Fault injection
+
+## Where faults come from
+
+ModelForge has no chaos endpoint and no way to run a command. A control plane that can be told to
+break itself is a control plane an attacker can tell to break itself, so every fault is injected
+from outside the product:
+
+```text
+container runtime   stop, start, restart, network disconnect, filesystem permissions
+rehearsal seams     the M12 lifecycle pause, the M15 restore interruption
+real credentials    revoked, rotated, malformed, wrongly scoped
+real load           the soak harness, running the actual capability path
+```
+
+A static test asserts the product exposes no `/chaos`, `/shell`, `/exec`, `/command` or `/debug`
+route segment, and that subprocess use stays allowlisted to the recovery plane.
+
+## The shape every scenario has
+
+```text
+preflight → baseline → inject → observe → assert → recover → verify invariants → cleanup
+```
+
+`preflight` refuses to start if an invariant is already violated: injecting a fault into a broken
+platform measures nothing. `verify invariants` re-runs the full set afterwards, so a scenario that
+survives its fault but corrupts state is reported as `INVARIANT_VIOLATED` rather than passing.
+A scenario that cannot restore the platform reports `RECOVERY_FAILED` rather than leaving the
+environment dirty and calling itself done.
+
+## Running it
+
+```sh
+python scripts/m16_chaos.py --list
+python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --scenario redis-outage --seed 20260827
+python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --all --report chaos.json
+```
+
+The database URL is required for executable scenarios and must name the non-owner runtime role;
+the harness has no embedded or owner-credential fallback. `--list` does not connect to PostgreSQL.
+
+Every run records its scenario id, seed, target, expected fault, observed result, recovery, cleanup
+outcome and the invariant results before and after. The seed is logged even where behaviour is
+deterministic, so a surprising result can be reproduced exactly.
+
+## Scenarios
+
+| Key | Class | Subsystem | Expected |
+| --- | --- | --- | --- |
+| `redis-outage` | DEPENDENCY | Redis | control plane stays up, replays nothing |
+| `postgres-restart` | DEPENDENCY | PostgreSQL | fails safely, recovers its pool, no duplicate work |
+| `node-agent-disconnect` | DEPENDENCY | Node Agent | heartbeat stops, liveness degrades, one identity survives |
+| `control-plane-restart-storm` | TRANSIENT | Control Plane | idempotent reconciliation, no authoritative drift |
+| `invalid-auth-burst` | AUTH | Control Plane | identical refusal, no amplification, no enumeration signal |
+| `backup-destination-unavailable` | CAPACITY | Backup storage | typed failure, serving unaffected |
+
+## Safety rules
+
+Disposable targets only. The node-disconnect scenario enrols a throwaway agent rather than
+isolating GPU Node: GPU Node reaches the control plane over the LAN rather than the Compose network, so
+disconnecting a container would not interrupt it anyway — and isolating the production node to
+prove a point is exactly what a release gate must not do.
+
+Nothing in the harness touches ExampleRAG, ExampleVision, Ollama, Plex or Tdarr. Their VRAM counts as
+external pressure and is observed; their processes are never stopped, paused, throttled or
+reconfigured.
+
+The harness deletes its own fixtures. `backup-destination-unavailable` removes the `chaos-*` backup
+sets it created; `node-agent-disconnect` tears down its agent, revokes its credential and disables
+the node it enrolled.
+
+## When a scenario finds something
+
+Both defects the harness found in M16 were real product defects rather than harness noise: an
+unwritable backup root escaping as an unhandled error, and a Node Agent logging a failure with no
+diagnostic. Each was fixed with a regression test before the scenario was re-run.
+
+One scenario was itself wrong first: the node-disconnect measured GPU Node while disconnecting an
+unrelated container, and reported that the heartbeat kept advancing during the "outage". That
+contradiction is why the harness records what it observed rather than only whether it passed.
+
+See `docs/operations/CHAOS_TESTING.md` for the fault matrix and
+`docs/security/CHAOS_SECURITY_GATE.md` for the invariants.