ITWorx Pulse release export
bd774932d5
Public source validation / validate (push) Failing after 3m8s
87 lines
3.2 KiB
Markdown
87 lines
3.2 KiB
Markdown
# Observability of Pulse
|
|
|
|
Pulse must expose its own health without creating a circular single point of truth.
|
|
|
|
## Internal metrics
|
|
|
|
At minimum:
|
|
- API request count/duration/error by bounded route code;
|
|
- active WebSocket clients/subscriptions;
|
|
- dropped/coalesced live samples;
|
|
- Prometheus query duration/errors/series/points/cache;
|
|
- discovery runs and reconciliation counts;
|
|
- worker job duration/failure/lag;
|
|
- alert evaluation duration/state transitions;
|
|
- notification attempts/failures;
|
|
- probe queue/concurrency/results;
|
|
- database pool/queries/migration;
|
|
- agent heartbeat/capabilities/errors;
|
|
- backup success/age;
|
|
- process CPU/memory/goroutines.
|
|
|
|
Avoid high-cardinality path/user/entity labels.
|
|
|
|
## Health endpoints
|
|
|
|
- liveness: process loop alive, no dependency checks that cause restart storms;
|
|
- readiness: mandatory dependencies usable;
|
|
- detailed system status: authenticated, shows component state/freshness.
|
|
|
|
## Logs
|
|
|
|
Structured production logs:
|
|
- timestamp;
|
|
- level;
|
|
- service/version;
|
|
- correlation/request/job ID;
|
|
- operation;
|
|
- safe entity/resource IDs;
|
|
- error code and wrapped message;
|
|
- duration/result.
|
|
|
|
Redact:
|
|
- tokens/cookies/authorization;
|
|
- passwords/connection secrets;
|
|
- sensitive headers;
|
|
- full upstream bodies;
|
|
- private keys.
|
|
|
|
## External dead-man
|
|
|
|
An independent existing monitor or simple external check should detect:
|
|
- Pulse HTTPS unavailable;
|
|
- Pulse heartbeat absent;
|
|
- backup too old if supported.
|
|
|
|
Do not rely solely on Pulse to alert that Pulse is completely down.
|
|
|
|
## System status page
|
|
|
|
Shows:
|
|
- build/version/commit;
|
|
- database;
|
|
- Prometheus;
|
|
- Unraid/agent;
|
|
- OIDC;
|
|
- worker heartbeats;
|
|
- query health;
|
|
- probes;
|
|
- notifications;
|
|
- backup age;
|
|
- active clients;
|
|
- storage use;
|
|
- last successful critical jobs.
|
|
|
|
No secret/config values.
|
|
|
|
## M9-08 operating contract
|
|
|
|
The public liveness contract is GET /healthz and returns exactly ok with HTTP 200 when the Pulse process is serving requests. It deliberately does not query PostgreSQL or upstream datasources, so an independent monitor can distinguish a dead Pulse process from a dependency outage. GET /readyz is the dependency-aware readiness contract and returns HTTP 503 while the database is unavailable.
|
|
|
|
The authenticated status surface is GET /api/v1/system/status and is available to viewers. It reports bounded component state, explicit Unknown/Disabled reasons, backup age (Unknown until a verified backup exists), and source lag (Unknown until a source reports an observed timestamp). It never reports unsampled sources as Healthy. Operators with the operate permission can read /api/v1/system/diagnostics and /api/v1/system/metrics; these routes are not public monitoring endpoints.
|
|
|
|
Configure an independent monitor outside Pulse to run every 60 seconds with two consecutive failures required for alerting:
|
|
|
|
python tools/deadman_check.py https://pulse.example.invalid/healthz --timeout 5
|
|
|
|
The script uses only the Python standard library, refuses embedded credentials, bounds the timeout to 1-30 seconds, accepts only HTTP 200 with the exact ok body, and exits non-zero for transport, timeout, redirect-result, or HTTP failures. Keep the monitor and its alerting path independent of the Pulse API and database. |