2.0 KiB
ADR-0044: Bounded fault injection and executable invariants
Status
Accepted in M16 on 2026-08-27.
Decision
Inject every fault from outside the product — the container runtime, existing rehearsal seams, real credentials, real load — and give ModelForge no chaos endpoint, no command route and no way to run a shell. Subprocess use stays allowlisted to the recovery plane, with argv built as a list and never a string, and static tests enforce all of it.
Express the platform's safety properties as fifteen read-only invariant queries over authoritative
state rather than as prose. Run them before a fault, after it, and after recovery. A scenario that
survives its fault but breaks an invariant reports INVARIANT_VIOLATED rather than passing, and a
scenario that cannot restore the platform reports RECOVERY_FAILED rather than leaving the
environment dirty.
Where a property is already enforced by a partial unique index, keep the index as the primary defence and treat the invariant as defence in depth for a bypassed guard.
Consequences
Fault injection cannot become an attack surface: there is nothing in the product to call. The cost is that some faults can only be reached with runtime access, which is the correct trade for a control plane.
Invariants are usable during an incident because they never write, never call an external system and never read the observability database — the case where monitoring itself has failed is exactly when the question "is state still sound?" matters most.
Each invariant carries a negative test that constructs its violation, because a check that never fires reports safety it did not verify. Two of the M16 scenarios found real product defects and one scenario was itself wrong first, reporting that a heartbeat kept advancing during an outage it was not actually causing; recording observations rather than only pass/fail is what surfaced that.
Every new milestone must classify the tables it adds, or the fingerprint and invariant coverage tests fail the build.