# Observability of Pulse

Pulse must expose its own health without creating a circular single point of truth.

## Internal metrics

At minimum:
- API request count/duration/error by bounded route code;
- active WebSocket clients/subscriptions;
- dropped/coalesced live samples;
- Prometheus query duration/errors/series/points/cache;
- discovery runs and reconciliation counts;
- worker job duration/failure/lag;
- alert evaluation duration/state transitions;
- notification attempts/failures;
- probe queue/concurrency/results;
- database pool/queries/migration;
- agent heartbeat/capabilities/errors;
- backup success/age;
- process CPU/memory/goroutines.

Avoid high-cardinality path/user/entity labels.

## Health endpoints

- liveness: process loop alive, no dependency checks that cause restart storms;
- readiness: mandatory dependencies usable;
- detailed system status: authenticated, shows component state/freshness.

## Logs

Structured production logs:
- timestamp;
- level;
- service/version;
- correlation/request/job ID;
- operation;
- safe entity/resource IDs;
- error code and wrapped message;
- duration/result.

Redact:
- tokens/cookies/authorization;
- passwords/connection secrets;
- sensitive headers;
- full upstream bodies;
- private keys.

## External dead-man

An independent existing monitor or simple external check should detect:
- Pulse HTTPS unavailable;
- Pulse heartbeat absent;
- backup too old if supported.

Do not rely solely on Pulse to alert that Pulse is completely down.

## System status page

Shows:
- build/version/commit;
- database;
- Prometheus;
- Unraid/agent;
- OIDC;
- worker heartbeats;
- query health;
- probes;
- notifications;
- backup age;
- active clients;
- storage use;
- last successful critical jobs.

No secret/config values.

## M9-08 operating contract

The public liveness contract is GET /healthz and returns exactly ok with HTTP 200 when the Pulse process is serving requests. It deliberately does not query PostgreSQL or upstream datasources, so an independent monitor can distinguish a dead Pulse process from a dependency outage. GET /readyz is the dependency-aware readiness contract and returns HTTP 503 while the database is unavailable.

The authenticated status surface is GET /api/v1/system/status and is available to viewers. It reports bounded component state, explicit Unknown/Disabled reasons, backup age (Unknown until a verified backup exists), and source lag (Unknown until a source reports an observed timestamp). It never reports unsampled sources as Healthy. Operators with the operate permission can read /api/v1/system/diagnostics and /api/v1/system/metrics; these routes are not public monitoring endpoints.

Configure an independent monitor outside Pulse to run every 60 seconds with two consecutive failures required for alerting:

    python tools/deadman_check.py https://pulse.example.invalid/healthz --timeout 5

The script uses only the Python standard library, refuses embedded credentials, bounds the timeout to 1-30 seconds, accepts only HTTP 200 with the exact ok body, and exits non-zero for transport, timeout, redirect-result, or HTTP failures. Keep the monitor and its alerting path independent of the Pulse API and database.