ModelForge/docs/architecture/ALERTING.md

Deduplicated alerting

AlertRuleRevision is an immutable, fingerprinted rule containing signal/SLO, condition, pending duration, severity, bounded labels, cooldown and recovery condition. Changes create a revision.

State machine

clear → PENDING → FIRING → ACKNOWLEDGED
          │          │            │
          └──────────┴────────────┴→ RESOLVED
bad + maintenance window → SUPPRESSED
SUPPRESSED + window ended + still bad → PENDING
RESOLVED + cooldown elapsed + recurrence → PENDING/FIRING

Acknowledgement records actor/reason but cannot alter the signal or SLO. Resolution is an explicit history event; alerts do not disappear. The deterministic fingerprint uses rule key, subject type and subject reference, so every polling cycle updates one logical alert rather than creating spam.

Severities are INFO, WARNING and CRITICAL. Initial rules cover production node liveness, GPU pressure, storage reserve, gateway errors, stable capability availability, runtime crash loops, queue saturation, migration/manual-intervention failure, rollback failure, artifact integrity and production capability envelope/headroom.

Maintenance windows have bounded matchers (rule_key, alert_type, subject_ref), start/end, reason and creator. Creation is audited and suppression remains visible. M14 delivers persistent records, API and UI; it has no mandatory Slack/email dependency and performs no automatic restart, kill, promotion, rollback or deletion.

M15 recovery rules

Five deduplicated rules extend the same lifecycle to recovery readiness:

BACKUP_STALE                 WARNING   no verified backup inside the policy window
BACKUP_FAILED                CRITICAL  a backup failed within the window
BACKUP_VERIFICATION_FAILED   CRITICAL  a backup failed specifically on hash or manifest verification
RESTORE_FAILED               CRITICAL  a restore failed or needs manual intervention
RECOVERY_READINESS_DEGRADED  WARNING   an authoritative asset is unprotected

They read the same authoritative records the recovery dashboard reads, so an alert and the dashboard can never disagree. A CREATED backup that was never verified does not clear BACKUP_STALE.

Live evidence: BACKUP_STALE and RECOVERY_READINESS_DEGRADED fired while no verified backup existed and resolved once one did, each as a single alert with an occurrence count rather than one alert per evaluation.