40 lines
2.0 KiB
Markdown
40 lines
2.0 KiB
Markdown
# ADR-0044: Bounded fault injection and executable invariants
|
|
|
|
## Status
|
|
|
|
Accepted in M16 on 2026-08-27.
|
|
|
|
## Decision
|
|
|
|
Inject every fault from outside the product — the container runtime, existing rehearsal seams, real
|
|
credentials, real load — and give ModelForge no chaos endpoint, no command route and no way to run
|
|
a shell. Subprocess use stays allowlisted to the recovery plane, with argv built as a list and
|
|
never a string, and static tests enforce all of it.
|
|
|
|
Express the platform's safety properties as fifteen read-only invariant queries over authoritative
|
|
state rather than as prose. Run them before a fault, after it, and after recovery. A scenario that
|
|
survives its fault but breaks an invariant reports `INVARIANT_VIOLATED` rather than passing, and a
|
|
scenario that cannot restore the platform reports `RECOVERY_FAILED` rather than leaving the
|
|
environment dirty.
|
|
|
|
Where a property is already enforced by a partial unique index, keep the index as the primary
|
|
defence and treat the invariant as defence in depth for a bypassed guard.
|
|
|
|
## Consequences
|
|
|
|
Fault injection cannot become an attack surface: there is nothing in the product to call. The cost
|
|
is that some faults can only be reached with runtime access, which is the correct trade for a
|
|
control plane.
|
|
|
|
Invariants are usable during an incident because they never write, never call an external system
|
|
and never read the observability database — the case where monitoring itself has failed is exactly
|
|
when the question "is state still sound?" matters most.
|
|
|
|
Each invariant carries a negative test that constructs its violation, because a check that never
|
|
fires reports safety it did not verify. Two of the M16 scenarios found real product defects and one
|
|
scenario was itself wrong first, reporting that a heartbeat kept advancing during an outage it was
|
|
not actually causing; recording observations rather than only pass/fail is what surfaced that.
|
|
|
|
Every new milestone must classify the tables it adds, or the fingerprint and invariant coverage
|
|
tests fail the build.
|