Observability of Pulse

Pulse must expose its own health without creating a circular single point of truth.

Internal metrics

At minimum:

API request count/duration/error by bounded route code;
active WebSocket clients/subscriptions;
dropped/coalesced live samples;
Prometheus query duration/errors/series/points/cache;
discovery runs and reconciliation counts;
worker job duration/failure/lag;
alert evaluation duration/state transitions;
notification attempts/failures;
probe queue/concurrency/results;
database pool/queries/migration;
agent heartbeat/capabilities/errors;
backup success/age;
process CPU/memory/goroutines.

Avoid high-cardinality path/user/entity labels.

Health endpoints

liveness: process loop alive, no dependency checks that cause restart storms;
readiness: mandatory dependencies usable;
detailed system status: authenticated, shows component state/freshness.

Logs

Structured production logs:

timestamp;
level;
service/version;
correlation/request/job ID;
operation;
safe entity/resource IDs;
error code and wrapped message;
duration/result.

Redact:

tokens/cookies/authorization;
passwords/connection secrets;
sensitive headers;
full upstream bodies;
private keys.

External dead-man

An independent existing monitor or simple external check should detect:

Pulse HTTPS unavailable;
Pulse heartbeat absent;
backup too old if supported.

Do not rely solely on Pulse to alert that Pulse is completely down.

System status page

Shows:

build/version/commit;
database;
Prometheus;
Unraid/agent;
OIDC;
worker heartbeats;
query health;
probes;
notifications;
backup age;
active clients;
storage use;
last successful critical jobs.

No secret/config values.

M9-08 operating contract

The public liveness contract is GET /healthz and returns exactly ok with HTTP 200 when the Pulse process is serving requests. It deliberately does not query PostgreSQL or upstream datasources, so an independent monitor can distinguish a dead Pulse process from a dependency outage. GET /readyz is the dependency-aware readiness contract and returns HTTP 503 while the database is unavailable.

The authenticated status surface is GET /api/v1/system/status and is available to viewers. It reports bounded component state, explicit Unknown/Disabled reasons, backup age (Unknown until a verified backup exists), and source lag (Unknown until a source reports an observed timestamp). It never reports unsampled sources as Healthy. Operators with the operate permission can read /api/v1/system/diagnostics and /api/v1/system/metrics; these routes are not public monitoring endpoints.

Configure an independent monitor outside Pulse to run every 60 seconds with two consecutive failures required for alerting:

python tools/deadman_check.py https://pulse.example.invalid/healthz --timeout 5

The script uses only the Python standard library, refuses embedded credentials, bounds the timeout to 1-30 seconds, accepts only HTTP 200 with the exact ok body, and exits non-zero for transport, timeout, redirect-result, or HTTP failures. Keep the monitor and its alerting path independent of the Pulse API and database.

3.2 KiB Raw Permalink Blame History