Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,48 @@
+# Deduplicated alerting
+
+`AlertRuleRevision` is an immutable, fingerprinted rule containing signal/SLO, condition, pending
+duration, severity, bounded labels, cooldown and recovery condition. Changes create a revision.
+
+## State machine
+
+```text
+clear → PENDING → FIRING → ACKNOWLEDGED
+          │          │            │
+          └──────────┴────────────┴→ RESOLVED
+bad + maintenance window → SUPPRESSED
+SUPPRESSED + window ended + still bad → PENDING
+RESOLVED + cooldown elapsed + recurrence → PENDING/FIRING
+```
+
+Acknowledgement records actor/reason but cannot alter the signal or SLO. Resolution is an explicit
+history event; alerts do not disappear. The deterministic fingerprint uses rule key, subject type
+and subject reference, so every polling cycle updates one logical alert rather than creating spam.
+
+Severities are `INFO`, `WARNING` and `CRITICAL`. Initial rules cover production node liveness, GPU
+pressure, storage reserve, gateway errors, stable capability availability, runtime crash loops,
+queue saturation, migration/manual-intervention failure, rollback failure, artifact integrity and
+production capability envelope/headroom.
+
+Maintenance windows have bounded matchers (`rule_key`, `alert_type`, `subject_ref`), start/end,
+reason and creator. Creation is audited and suppression remains visible. M14 delivers persistent
+records, API and UI; it has no mandatory Slack/email dependency and performs no automatic restart,
+kill, promotion, rollback or deletion.
+
+## M15 recovery rules
+
+Five deduplicated rules extend the same lifecycle to recovery readiness:
+
+```text
+BACKUP_STALE                 WARNING   no verified backup inside the policy window
+BACKUP_FAILED                CRITICAL  a backup failed within the window
+BACKUP_VERIFICATION_FAILED   CRITICAL  a backup failed specifically on hash or manifest verification
+RESTORE_FAILED               CRITICAL  a restore failed or needs manual intervention
+RECOVERY_READINESS_DEGRADED  WARNING   an authoritative asset is unprotected
+```
+
+They read the same authoritative records the recovery dashboard reads, so an alert and the dashboard
+can never disagree. A `CREATED` backup that was never verified does not clear `BACKUP_STALE`.
+
+Live evidence: `BACKUP_STALE` and `RECOVERY_READINESS_DEGRADED` fired while no verified backup
+existed and resolved once one did, each as a single alert with an occurrence count rather than one
+alert per evaluation.