79 lines
5.9 KiB
Markdown
79 lines
5.9 KiB
Markdown
# Chaos testing
|
|
|
|
## Fault matrix
|
|
|
|
Every subsystem, the failure modes it can present, how far the damage reaches, and the invariant
|
|
that must survive.
|
|
|
|
| Subsystem | Failure mode | Class | Blast radius | Recovery | Alert | Invariant |
|
|
| --- | --- | --- | --- | --- | --- | --- |
|
|
| Control Plane | process crash / restart | TRANSIENT | in-flight API requests | startup reconciliation | none by design | no authoritative drift |
|
|
| Control Plane | restart during backup | TRANSIENT | that backup only | interrupted backup fails closed | `BACKUP_FAILED` | `restore_requires_verified_backup` |
|
|
| PostgreSQL | unavailable | DEPENDENCY | all persistence | pool reconnects, no restart needed | `PLATFORM_API` SLO | `audit_chain_intact` |
|
|
| PostgreSQL | deadlock / serialisation failure | CONCURRENCY | one transaction | bounded retry | none | no lost update |
|
|
| PostgreSQL | latency | CAPACITY | request queueing | timeouts, typed errors | SLO burn | no connection explosion |
|
|
| Redis | unavailable | DEPENDENCY | queues, transient payloads | queues start empty, leases reconciled | dependency health | no duplicate serving job |
|
|
| Node Agent | network loss | DEPENDENCY | that node's capacity | heartbeat stale → offline → reconnect | `NODE_OFFLINE` | `single_node_identity` |
|
|
| Node Agent | credential revoked | AUTH | that node | re-enrol, identity preserved | `NODE_OFFLINE` | `revoked_credentials_stay_revoked` |
|
|
| Node Agent | restart storm | TRANSIENT | that node | same identity, no duplicate inventory | none | `single_active_node_credential` |
|
|
| Runtime Worker | crash during load/invoke/unload | TRANSIENT | one request | typed failure, generation fencing | `RUNTIME_CRASH_LOOP` | `no_orphan_serving_work` |
|
|
| Gateway | invalid credential burst | AUTH | none | identical refusal | none | no enumeration signal |
|
|
| Gateway | malformed or hostile input | SECURITY | none | typed 4xx | none | no execution, no 5xx |
|
|
| Scheduler | GPU pressure | CAPACITY | admission | reject or queue | `GPU_PRESSURE` | `no_stale_gpu_lease` |
|
|
| Scheduler | external VRAM spike | CAPACITY | admission | schedulable headroom shrinks | `CAPABILITY_CAPACITY` | external workloads untouched |
|
|
| GPU | telemetry stale | DEPENDENCY | placement decisions | admission blocked, never guessed | `GPU_PRESSURE` | no stale snapshot as truth |
|
|
| Artifact storage | location missing | CORRUPTION | one artifact set | exact-revision rehydration | `ARTIFACT_INTEGRITY` | `no_unsafe_artifact_promoted` |
|
|
| Artifact storage | disk low / full | CAPACITY | acquisition | capacity guard refuses first | `STORAGE_LOW` | no partial promotion |
|
|
| Backup storage | unwritable / absent | CAPACITY | that backup | typed `DESTINATION_UNAVAILABLE` | `BACKUP_FAILED` | serving unaffected |
|
|
| Backup storage | payload corrupted | CORRUPTION | that backup | verification refuses | `BACKUP_VERIFICATION_FAILED` | `restore_requires_verified_backup` |
|
|
| Lifecycle | operation interrupted | TRANSIENT | one LAB subject | rollback from immutable snapshot, exactly once | `ROLLBACK_FAILURE` | `lifecycle_commit_has_evidence` |
|
|
| Migration | cutover interrupted | CORRUPTION | one plan | never auto-resolved; operator reconciles | `MIGRATION_FAILURE` | `cutover_commit_has_validation` |
|
|
| Migration | backfill worker failure | TRANSIENT | one plan | checkpoint preserves completed work | `MIGRATION_FAILURE` | no mixed embedding space |
|
|
| Observability | persistence unavailable | DEPENDENCY | monitoring only | degraded gauge, serving continues | `modelforge_observability_degraded` | safety never depends on the alert store |
|
|
| Recovery | restore interrupted | TRANSIENT | that restore | manual intervention, resumable | `RESTORE_FAILED` | `restore_requires_verified_backup` |
|
|
| Credentials | rotation under load | AUTH | one client | old refused, new accepted | none | one usable secret |
|
|
| External | Hugging Face unavailable | DEPENDENCY | acquisition only | `ARTIFACT_REHYDRATION_BLOCKED` | none | no fallback source |
|
|
| External | ExampleRAG / ExampleVision down | EXTERNAL | none | not owned | none | never written to |
|
|
|
|
## Failure classes
|
|
|
|
```text
|
|
TRANSIENT recoverable by retry or restart; state converges
|
|
PERSISTENT requires operator action; state is safe but stuck
|
|
CORRUPTION integrity is in question; the platform must refuse rather than proceed
|
|
CAPACITY the safe answer is to reject work, not to risk exhaustion
|
|
DEPENDENCY something ModelForge does not own is unavailable
|
|
AUTH a credential is missing, wrong, revoked or out of scope
|
|
SECURITY hostile input or an attempt to cross a boundary
|
|
CONCURRENCY two operations race; exactly one must win
|
|
OPERATOR a human action that must be gated, audited and reversible
|
|
```
|
|
|
|
A capacity rejection is not a failure. The scheduler refusing unsafe work is the system behaving
|
|
correctly, and the soak counts those separately from internal errors for exactly that reason.
|
|
|
|
## Invariants
|
|
|
|
Fifteen properties, each a query the platform runs against its own authoritative state. They are
|
|
read-only, never call an external system, and never depend on the observability database — so they
|
|
stay usable exactly when monitoring is the thing that failed.
|
|
|
|
```sh
|
|
python scripts/m16_invariants.py --label "before change"
|
|
```
|
|
|
|
See `docs/security/CHAOS_SECURITY_GATE.md` for the full list and what each one protects.
|
|
|
|
## Running the suite
|
|
|
|
```sh
|
|
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --all --report chaos.json
|
|
python scripts/m16_soak.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --minutes 45 --secret <secret>
|
|
python scripts/m16_invariants.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL"
|
|
```
|
|
|
|
Chaos scenarios and the soak may be run together deliberately — a backup taken while the soak is
|
|
generating real traffic is the M16 "backup under load" evidence — but a soak whose numbers are
|
|
meant to characterise steady state should not overlap an injected fault. When they do overlap, say
|
|
so in the report rather than presenting the mixed result as a clean baseline.
|