ModelForge/docs/operations/CHAOS_TESTING.md

Chaos testing

Fault matrix

Every subsystem, the failure modes it can present, how far the damage reaches, and the invariant that must survive.

Subsystem	Failure mode	Class	Blast radius	Recovery	Alert	Invariant
Control Plane	process crash / restart	TRANSIENT	in-flight API requests	startup reconciliation	none by design	no authoritative drift
Control Plane	restart during backup	TRANSIENT	that backup only	interrupted backup fails closed	BACKUP_FAILED	restore_requires_verified_backup
PostgreSQL	unavailable	DEPENDENCY	all persistence	pool reconnects, no restart needed	PLATFORM_API SLO	audit_chain_intact
PostgreSQL	deadlock / serialisation failure	CONCURRENCY	one transaction	bounded retry	none	no lost update
PostgreSQL	latency	CAPACITY	request queueing	timeouts, typed errors	SLO burn	no connection explosion
Redis	unavailable	DEPENDENCY	queues, transient payloads	queues start empty, leases reconciled	dependency health	no duplicate serving job
Node Agent	network loss	DEPENDENCY	that node's capacity	heartbeat stale → offline → reconnect	NODE_OFFLINE	single_node_identity
Node Agent	credential revoked	AUTH	that node	re-enrol, identity preserved	NODE_OFFLINE	revoked_credentials_stay_revoked
Node Agent	restart storm	TRANSIENT	that node	same identity, no duplicate inventory	none	single_active_node_credential
Runtime Worker	crash during load/invoke/unload	TRANSIENT	one request	typed failure, generation fencing	RUNTIME_CRASH_LOOP	no_orphan_serving_work
Gateway	invalid credential burst	AUTH	none	identical refusal	none	no enumeration signal
Gateway	malformed or hostile input	SECURITY	none	typed 4xx	none	no execution, no 5xx
Scheduler	GPU pressure	CAPACITY	admission	reject or queue	GPU_PRESSURE	no_stale_gpu_lease
Scheduler	external VRAM spike	CAPACITY	admission	schedulable headroom shrinks	CAPABILITY_CAPACITY	external workloads untouched
GPU	telemetry stale	DEPENDENCY	placement decisions	admission blocked, never guessed	GPU_PRESSURE	no stale snapshot as truth
Artifact storage	location missing	CORRUPTION	one artifact set	exact-revision rehydration	ARTIFACT_INTEGRITY	no_unsafe_artifact_promoted
Artifact storage	disk low / full	CAPACITY	acquisition	capacity guard refuses first	STORAGE_LOW	no partial promotion
Backup storage	unwritable / absent	CAPACITY	that backup	typed DESTINATION_UNAVAILABLE	BACKUP_FAILED	serving unaffected
Backup storage	payload corrupted	CORRUPTION	that backup	verification refuses	BACKUP_VERIFICATION_FAILED	restore_requires_verified_backup
Lifecycle	operation interrupted	TRANSIENT	one LAB subject	rollback from immutable snapshot, exactly once	ROLLBACK_FAILURE	lifecycle_commit_has_evidence
Migration	cutover interrupted	CORRUPTION	one plan	never auto-resolved; operator reconciles	MIGRATION_FAILURE	cutover_commit_has_validation
Migration	backfill worker failure	TRANSIENT	one plan	checkpoint preserves completed work	MIGRATION_FAILURE	no mixed embedding space
Observability	persistence unavailable	DEPENDENCY	monitoring only	degraded gauge, serving continues	modelforge_observability_degraded	safety never depends on the alert store
Recovery	restore interrupted	TRANSIENT	that restore	manual intervention, resumable	RESTORE_FAILED	restore_requires_verified_backup
Credentials	rotation under load	AUTH	one client	old refused, new accepted	none	one usable secret
External	Hugging Face unavailable	DEPENDENCY	acquisition only	ARTIFACT_REHYDRATION_BLOCKED	none	no fallback source
External	ExampleRAG / ExampleVision down	EXTERNAL	none	not owned	none	never written to

Failure classes

TRANSIENT     recoverable by retry or restart; state converges
PERSISTENT    requires operator action; state is safe but stuck
CORRUPTION    integrity is in question; the platform must refuse rather than proceed
CAPACITY      the safe answer is to reject work, not to risk exhaustion
DEPENDENCY    something ModelForge does not own is unavailable
AUTH          a credential is missing, wrong, revoked or out of scope
SECURITY      hostile input or an attempt to cross a boundary
CONCURRENCY   two operations race; exactly one must win
OPERATOR      a human action that must be gated, audited and reversible

A capacity rejection is not a failure. The scheduler refusing unsafe work is the system behaving correctly, and the soak counts those separately from internal errors for exactly that reason.

Invariants

Fifteen properties, each a query the platform runs against its own authoritative state. They are read-only, never call an external system, and never depend on the observability database — so they stay usable exactly when monitoring is the thing that failed.

python scripts/m16_invariants.py --label "before change"

See docs/security/CHAOS_SECURITY_GATE.md for the full list and what each one protects.

Running the suite

python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --all --report chaos.json
python scripts/m16_soak.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --minutes 45 --secret <secret>
python scripts/m16_invariants.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL"

Chaos scenarios and the soak may be run together deliberately — a backup taken while the soak is generating real traffic is the M16 "backup under load" evidence — but a soak whose numbers are meant to characterise steady state should not overlap an injected fault. When they do overlap, say so in the report rather than presenting the mixed result as a clean baseline.