ITWorx-Pulse-Public/docs/architecture/ALERTING_AND_INCIDENTS.md

Alerting and incident architecture

1. Concepts

• Rule: versioned definition of a condition.
• Alert instance: stable rule + entity/label fingerprint.
• Occurrence: immutable state transition/evaluation history.
• Incident: grouped operational problem containing related alerts/entities/events.
• Silence: user-defined temporary notification/state suppression.
• Maintenance window: scheduled policy for selected entities.
• Inhibition/suppression: dependency-aware prevention of downstream noise.

2. State machine

Primary states:

inactive -> pending -> firing -> acknowledged -> resolved

Orthogonal/contextual states:

• unknown;
• silenced;
• suppressed;
• maintenance.

Acknowledgement does not mean resolved. A firing alert may be acknowledged.

3. Evaluation

Every rule defines:

• schedule/evaluation interval;
• semantic query/event/status input;
• scope selector;
• comparator/condition;
• pending duration;
• recovery duration;
• hysteresis;
• severity;
• message template;
• grouping/suppression metadata;
• unknown behavior.

Evaluation is idempotent and transactional.

4. Unknown behavior

When required data is stale/unavailable:

• do not auto-resolve a firing alert;
• transition/display unknown according to rule policy;
• retain last known value/time;
• emit source-health context;
• avoid repeated notifications for the same unknown episode.

5. Hysteresis example

Disk temperature:

• enter pending above 50°C;
• fire after 5 minutes;
• remain firing until below 46°C for 5 minutes;
• cooldown duplicate notifications.

Both thresholds and durations are visible in rule UI.

6. Deduplication/grouping

Fingerprint is based on:

• rule ID/version behavior;
• stable entity ID;
• bounded grouping labels.

Notifications group by:

• incident;
• host/application;
• severity/time window;
• configured labels.

7. Dependency suppression

Examples:

• host unreachable suppresses container/service unreachable alerts;
• DNS failure suppresses dependent service resolution alerts;
• reverse proxy outage groups external endpoint failures while internal probes remain separate;
• Prometheus unavailable suppresses metric-rule noise and creates a monitoring-source incident.

Suppressed alerts remain inspectable.

Pulse uses bounded, deterministic cause keys for suppression (host.unreachable, dns.failure, source.unavailable). A downstream alert is suppressed only when the configured cause is active and confirmed or high-confidence; the downstream signal, fingerprint and suppression reason remain inspectable. Group keys use rule identity, severity and configured stable labels with bounded cardinality. Duplicate evaluation signals are collapsed by instance and evaluation key before grouping.

8. Maintenance and silences

Maintenance:

• scheduled;
• selector-based;
• audited;
• optionally changes visual status to maintenance;
• prevents configured notifications without deleting evidence.

Silence:

• explicit reason;
• owner;
• expiry required;
• matchers bounded/previewable;
• audited.

9. Incident correlation

Deterministic rules first:

• shared parent dependency;
• same application/host;
• close start time;
• known causal hierarchy;
• common event.

Heuristic correlation may be added later, but must expose confidence and rationale.

Incident severity is derived from:

• highest unsuppressed alert;
• criticality of affected entities;
• duration/scope;
• explicit operator override.

10. Notifications

Channels may include email/webhook/other selected integrations.

Requirements:

• encrypted secret references;
• idempotency;
• retry with bounded exponential backoff;
• delivery audit;
• templates with safe escaping;
• grouping and cooldown;
• recovery notification;
• test action;
• channel failure health.

11. Default rules

Provide conservative baseline rules for:

• monitoring source stale/down;
• host resource/temperature;
• container stop/health/restart loop;
• array/pool/disk/SMART/capacity;
• service availability/latency/TLS;
• UPS battery/on-battery when available.

Defaults are versioned, visible and editable. Avoid hard thresholds where hardware-specific policy is needed; use sensible presets with onboarding confirmation.

M8-06 implementation note: silences and maintenance windows use explicit bounded matchers (rule IDs, entity IDs, entity types, severities, and exact labels). Preview returns deterministic matching instance IDs. Expiry is mandatory, state transitions are retained, and revocation or expiry never deletes alert occurrences. Maintenance state is exposed as scheduled, active, expired, or revoked.

M8-07 operation semantics: acknowledgement is an operator state annotation, not resolution. Acknowledgement and unacknowledgement are persisted as immutable occurrence events, use optimistic revisions, and are idempotent by evaluation key. Evaluator transitions remain authoritative for resolved state.

M8-08 delivery semantics

The notification framework uses the alert lifecycle event type (firing, recovery, or unknown) as the outbox contract. Enqueue is idempotent by caller-supplied key. Workers claim due rows with a bounded lease and PostgreSQL row locking, write a delivering audit record, and complete the same attempt transactionally. Failures are redacted, retried with bounded exponential delay, and terminally marked failed after ten attempts. A recovery event is delivered through the same path.

The production transport is a bounded HTTPS webhook configured with PULSE_NOTIFICATION_WEBHOOK_URL, PULSE_NOTIFICATION_WEBHOOK_TOKEN, and an optional 1–30 second PULSE_NOTIFICATION_WEBHOOK_TIMEOUT. The worker reconciles one system-owned webhook channel at startup. PostgreSQL stores only the URL, timeout, and an opaque runtime secret reference; the bearer credential never enters channel JSON, delivery audit, API responses, or logs. Requests contain the lifecycle payload and repeat the stable outbox idempotency key in both the body and Idempotency-Key header. Receivers must use that key to collapse a replay when delivery succeeded but acknowledgement or local completion was interrupted. Redirects, URL credentials, query strings, fragments, unbounded timeouts, and non-HTTPS production endpoints are rejected.

Channel test actions are in-memory only, do not enqueue an outbox record, and are rate limited per actor/channel pair.

M8-09 correlation behavior

Correlation starts with deterministic keys: a primary dependency outage groups the dependency alert and downstream alerts sharing its parent; otherwise stable application, host, or entity keys are used. Groups are sorted by alert ID and correlation output is stable across input ordering. Every candidate stores the method and confidence used; the model does not present heuristic correlation as proven causation. Manual alert association is persisted separately, audited at the API boundary, and protected from automatic overwrite.

M8-10 incident timeline and operator context

Incident detail presents correlated alerts and operator notes in a deterministic UTC timeline. Ownership and notes are auditable metadata only. The UI explicitly states that correlation is an evidence-backed indication rather than proven causality and that confidence describes the correlation rule, not certainty of root cause. External workflow linking is a disabled placeholder; Pulse remains observational and does not remediate infrastructure.

M8-11 default rule seeding

The implementation-owned embedded default bundle is schema-versioned and validated against the semantic metric catalog before startup reconciliation. Seeding is additive and idempotent: a missing default is inserted, while an existing rule ID—including a customized version—is preserved and counted as existing. Event-based restart-loop rules explicitly support count aggregation. Duplicate evaluation signals are collapsed before deterministic grouping; dependency causes remain inspectable while downstream notifications are suppressed.

Default storage-pool rule — evaluability fix (2026-08-17)

The seeded default "Opslagpool bijna vol" originally bound storage.pool.utilization, whose query template requires a {{pool}} value; the alert evaluator issues one instant query per rule without a pool scope and does not fan out over groupBy, so the rule could never evaluate (PROMQL_BINDING_VALUE_REQUIRED (pool)). It now uses storage.pool.utilization.maximum (placeholder-free: the highest utilisation across all pools) so the default fires when any pool crosses the threshold; per-pool rules can still be created with a poolId scope. The seed refreshes implementation-owned defaults that are still at revision 1 (never edited by an operator) through the normal versioned update path (alertdefaults.Seed → RuleUpdater), and never touches edited rules. The pulse_storage_pool_* series must be provided by the environment — for Unraid via the recording rules in deploy/prometheus.pulse-storage-pool.rules.yaml.