Publish ITWorx Pulse source

2026-09-03 02:09:19 +02:00
commit bd774932d5
614 changed files with 77116 additions and 0 deletions
@@ -0,0 +1,87 @@
+# Observability of Pulse
+
+Pulse must expose its own health without creating a circular single point of truth.
+
+## Internal metrics
+
+At minimum:
+- API request count/duration/error by bounded route code;
+- active WebSocket clients/subscriptions;
+- dropped/coalesced live samples;
+- Prometheus query duration/errors/series/points/cache;
+- discovery runs and reconciliation counts;
+- worker job duration/failure/lag;
+- alert evaluation duration/state transitions;
+- notification attempts/failures;
+- probe queue/concurrency/results;
+- database pool/queries/migration;
+- agent heartbeat/capabilities/errors;
+- backup success/age;
+- process CPU/memory/goroutines.
+
+Avoid high-cardinality path/user/entity labels.
+
+## Health endpoints
+
+- liveness: process loop alive, no dependency checks that cause restart storms;
+- readiness: mandatory dependencies usable;
+- detailed system status: authenticated, shows component state/freshness.
+
+## Logs
+
+Structured production logs:
+- timestamp;
+- level;
+- service/version;
+- correlation/request/job ID;
+- operation;
+- safe entity/resource IDs;
+- error code and wrapped message;
+- duration/result.
+
+Redact:
+- tokens/cookies/authorization;
+- passwords/connection secrets;
+- sensitive headers;
+- full upstream bodies;
+- private keys.
+
+## External dead-man
+
+An independent existing monitor or simple external check should detect:
+- Pulse HTTPS unavailable;
+- Pulse heartbeat absent;
+- backup too old if supported.
+
+Do not rely solely on Pulse to alert that Pulse is completely down.
+
+## System status page
+
+Shows:
+- build/version/commit;
+- database;
+- Prometheus;
+- Unraid/agent;
+- OIDC;
+- worker heartbeats;
+- query health;
+- probes;
+- notifications;
+- backup age;
+- active clients;
+- storage use;
+- last successful critical jobs.
+
+No secret/config values.
+
+## M9-08 operating contract
+
+The public liveness contract is GET /healthz and returns exactly ok with HTTP 200 when the Pulse process is serving requests. It deliberately does not query PostgreSQL or upstream datasources, so an independent monitor can distinguish a dead Pulse process from a dependency outage. GET /readyz is the dependency-aware readiness contract and returns HTTP 503 while the database is unavailable.
+
+The authenticated status surface is GET /api/v1/system/status and is available to viewers. It reports bounded component state, explicit Unknown/Disabled reasons, backup age (Unknown until a verified backup exists), and source lag (Unknown until a source reports an observed timestamp). It never reports unsampled sources as Healthy. Operators with the operate permission can read /api/v1/system/diagnostics and /api/v1/system/metrics; these routes are not public monitoring endpoints.
+
+Configure an independent monitor outside Pulse to run every 60 seconds with two consecutive failures required for alerting:
+
+    python tools/deadman_check.py https://pulse.example.invalid/healthz --timeout 5
+
+The script uses only the Python standard library, refuses embedded credentials, bounds the timeout to 1-30 seconds, accepts only HTTP 200 with the exact ok body, and exits non-zero for transport, timeout, redirect-result, or HTTP failures. Keep the monitor and its alerting path independent of the Pulse API and database.