49 lines
2.4 KiB
Markdown
49 lines
2.4 KiB
Markdown
# Deduplicated alerting
|
|
|
|
`AlertRuleRevision` is an immutable, fingerprinted rule containing signal/SLO, condition, pending
|
|
duration, severity, bounded labels, cooldown and recovery condition. Changes create a revision.
|
|
|
|
## State machine
|
|
|
|
```text
|
|
clear → PENDING → FIRING → ACKNOWLEDGED
|
|
│ │ │
|
|
└──────────┴────────────┴→ RESOLVED
|
|
bad + maintenance window → SUPPRESSED
|
|
SUPPRESSED + window ended + still bad → PENDING
|
|
RESOLVED + cooldown elapsed + recurrence → PENDING/FIRING
|
|
```
|
|
|
|
Acknowledgement records actor/reason but cannot alter the signal or SLO. Resolution is an explicit
|
|
history event; alerts do not disappear. The deterministic fingerprint uses rule key, subject type
|
|
and subject reference, so every polling cycle updates one logical alert rather than creating spam.
|
|
|
|
Severities are `INFO`, `WARNING` and `CRITICAL`. Initial rules cover production node liveness, GPU
|
|
pressure, storage reserve, gateway errors, stable capability availability, runtime crash loops,
|
|
queue saturation, migration/manual-intervention failure, rollback failure, artifact integrity and
|
|
production capability envelope/headroom.
|
|
|
|
Maintenance windows have bounded matchers (`rule_key`, `alert_type`, `subject_ref`), start/end,
|
|
reason and creator. Creation is audited and suppression remains visible. M14 delivers persistent
|
|
records, API and UI; it has no mandatory Slack/email dependency and performs no automatic restart,
|
|
kill, promotion, rollback or deletion.
|
|
|
|
## M15 recovery rules
|
|
|
|
Five deduplicated rules extend the same lifecycle to recovery readiness:
|
|
|
|
```text
|
|
BACKUP_STALE WARNING no verified backup inside the policy window
|
|
BACKUP_FAILED CRITICAL a backup failed within the window
|
|
BACKUP_VERIFICATION_FAILED CRITICAL a backup failed specifically on hash or manifest verification
|
|
RESTORE_FAILED CRITICAL a restore failed or needs manual intervention
|
|
RECOVERY_READINESS_DEGRADED WARNING an authoritative asset is unprotected
|
|
```
|
|
|
|
They read the same authoritative records the recovery dashboard reads, so an alert and the dashboard
|
|
can never disagree. A `CREATED` backup that was never verified does not clear `BACKUP_STALE`.
|
|
|
|
Live evidence: `BACKUP_STALE` and `RECOVERY_READINESS_DEGRADED` fired while no verified backup
|
|
existed and resolved once one did, each as a single alert with an occurrence count rather than one
|
|
alert per evaluation.
|