Publish ITWorx Pulse source

docs/operations/BACKUP_RESTORE.md

@@ -0,0 +1,128 @@
+# Backup and restore
+
+## Backup contents
+
+- PostgreSQL logical backup or selected physical method with documented compatibility;
+- Pulse configuration;
+- dashboard versions;
+- alert rules and maintenance windows;
+- inventory/events/incidents/audit according to policy;
+- non-secret datasource/channel configuration;
+- schema/application version;
+- image digests;
+- checksums and manifest.
+
+Secrets:
+- are backed up only through an explicitly secured secret-store procedure;
+- never appear plaintext in portable exports;
+- are documented as required reattachment steps if excluded.
+
+## Backup behavior
+
+- on-demand administrator action;
+- scheduled optional job;
+- unique immutable backup ID;
+- temporary file + atomic finalize;
+- checksum;
+- size and duration;
+- success/failure audit;
+- retention;
+- destination health;
+- no backup only inside the source database volume.
+
+## Verification
+
+A backup is not trusted until:
+- checksum passes;
+- archive/format opens;
+- manifest/schema version valid;
+- secret scan/redaction policy passes;
+- restore test has succeeded for the release.
+
+## Restore test
+
+Use an empty isolated environment:
+
+1. deploy compatible database/application version;
+2. restore backup;
+3. run migrations if expected;
+4. start services;
+5. authenticate;
+6. verify dashboards/rules/inventory/events/incidents;
+7. query metrics datasource through restored config;
+8. run smoke tests;
+9. compare key counts/checksums;
+10. document credential reattachment.
+
+Do not overwrite production during routine validation.
+
+## Upgrade rollback
+
+Before schema upgrade:
+- create verified backup;
+- record current migration version and image digests;
+- test upgrade from prior release fixture;
+- test supported downgrade or forward-recovery strategy;
+- document compatibility window.
+
+If down migration is unsafe, use restore/forward-fix and state this explicitly.
+
+### Supported M9-10 recovery path
+
+Pulse migrations are forward-only. Do not manually remove rows from
+`schema_migrations`, run ad-hoc down SQL, or point an older image at a schema
+whose compatibility has not been proven. The supported recovery path after a
+failed upgrade is:
+
+1. Freeze only the Pulse compose project and preserve the migration/API logs.
+2. Record the image digests, `schema_migrations` rows and the verified backup
+   ID before making another attempt.
+3. Correct the release/configuration fault, use the same or a newer compatible
+   Pulse image, and re-run `cmd/migrate`. Each migration is committed with its
+   schema-migration record in one transaction and repeat runs are safe.
+4. Restart the Pulse API/worker/agent, then verify health, schema migration
+   count, dashboard/rule current-version links and the operational smoke
+   checks.
+5. If forward recovery cannot be completed, build a new empty isolated target,
+   restore the last verified portable backup, reattach excluded secrets, run
+   compatible migrations and validate it before any production cutover.
+
+The old image/configuration is a rollback point only when its documented
+schema compatibility is satisfied. M9-10 verified the baseline `61cb2a4` to
+`11e3496` transition, a repeat migration and a PostgreSQL transaction fault
+rollback. It does not claim that arbitrary future down-migrations are safe.
+
+## Recovery objectives
+
+Codex must measure and document practical RPO/RTO for the environment. Do not invent guarantees.
+
+Baseline intent:
+- configuration/incident loss limited by backup schedule;
+- restore procedure executable by the operator;
+- no dependency on the failed Pulse API to access backup instructions.
+
+## M9-09 portable backup
+
+Configure a protected, operator-owned destination with `PULSE_BACKUP_DIR` and an optional retention count with `PULSE_BACKUP_RETENTION` (default 5, maximum 100). The API exposes the following administrator-only endpoint:
+
+- `GET /api/v1/system/backups` lists verified backup archives.
+- `POST /api/v1/system/backups` creates a temporary archive, writes it atomically, emits a sidecar SHA-256 checksum and applies retention.
+
+Format version 2 of the archive is a ZIP containing a manifest and deterministic JSONL entries for the approved non-secret persistence tables. Durable discovery identity in `container_aliases` is included so a restored worker does not assign new entity identities to unchanged containers. Runtime configuration, `system_settings`, notification channel configuration, secret references, notification delivery state and bounded `agent_snapshots` and `capacity_samples` runtime telemetry are excluded and must be reattached or republished through the approved procedure. The exporter rejects sensitive JSON keys rather than writing them. The manifest records the migration count, table row counts and per-entry SHA-256 values. Restore first verifies the archive and sidecar, then only restores into a database where every backed-up table is empty, inside one transaction. Current dashboard/rule version foreign keys are restored after their version rows exist. Version 1 archives predate durable container-alias coverage and are deliberately rejected instead of being accepted as complete recovery points.
+
+The PostgreSQL clean-room test classifies every migrated application table as
+either portable or deliberately excluded and validates every configured export
+and restore column against the migrated schema. CI sets
+`PULSE_REQUIRE_BACKUP_INTEGRATION=true` and provisions separate source and
+restore databases, so a missing DSN or a newly unclassified table fails rather
+than silently skipping this release gate.
+
+A reproducible DSN-gated test is:
+
+```text
+$env:PULSE_TEST_DATABASE_URL = 'postgresql://...'
+$env:PULSE_TEST_RESTORE_DATABASE_URL = 'postgresql://...'
+go test ./internal/backup -run TestPostgreSQLBackupRestoreCleanRoom -count=1 -v
+```
+
+Do not put actual DSNs, passwords, secret-store values or backup bytes in the repository, evidence or shell history. The restore test uses an isolated empty environment and never overwrites production.

docs/operations/DEVELOPMENT_SETUP.md

@@ -0,0 +1,61 @@
+# Development setup
+
+## Prerequisites
+
+- Git;
+- Go 1.26.6;
+- Node.js 24 or newer and pnpm 10.33 or newer;
+- Python 3 with the packages in `requirements-dev.txt`;
+- PowerShell 7;
+- Docker Engine and Docker Compose;
+- Chromium for Playwright end-to-end tests.
+
+## Bootstrap
+
+From a clean checkout:
+
+```powershell
+Copy-Item .env.example .env
+pwsh -NoProfile -File scripts/bootstrap.ps1
+```
+
+The example environment uses mock authentication and local-only credentials. Never reuse production credentials, backups, dashboards, telemetry, or host inventories.
+
+## Run the development stack
+
+```powershell
+docker compose -f deploy/compose.yaml -f deploy/compose.dev.yaml up --build
+```
+
+The development override publishes the web, API, and PostgreSQL ports for local testing. Treat those ports as development-only and use a host firewall when working on an untrusted network.
+
+Stop only this stack with:
+
+```powershell
+docker compose -f deploy/compose.yaml -f deploy/compose.dev.yaml down --volumes
+```
+
+## Validation
+
+For a public source checkout:
+
+```powershell
+pwsh -NoProfile -File scripts/public-verify.ps1
+```
+
+The optional real source-to-browser smoke uses an isolated Compose project and tears it down after the run:
+
+```powershell
+pwsh -NoProfile -File scripts/integration-smoke.ps1
+```
+
+PostgreSQL integration tests use explicit process-local test DSNs. Never point them at a database that contains data you need to keep.
+
+## Production separation
+
+- production uses OIDC and rejects mock authentication;
+- the base/production profiles do not publish PostgreSQL or the API;
+- runtime secrets remain outside Git;
+- fixtures are synthetic;
+- the agent remains read-only and receives no Docker socket;
+- production changes follow [`../PUBLIC_DEPLOYMENT.md`](../PUBLIC_DEPLOYMENT.md), not the development procedure.

docs/operations/OBSERVABILITY_OF_PULSE.md

@@ -0,0 +1,87 @@
+# Observability of Pulse
+
+Pulse must expose its own health without creating a circular single point of truth.
+
+## Internal metrics
+
+At minimum:
+- API request count/duration/error by bounded route code;
+- active WebSocket clients/subscriptions;
+- dropped/coalesced live samples;
+- Prometheus query duration/errors/series/points/cache;
+- discovery runs and reconciliation counts;
+- worker job duration/failure/lag;
+- alert evaluation duration/state transitions;
+- notification attempts/failures;
+- probe queue/concurrency/results;
+- database pool/queries/migration;
+- agent heartbeat/capabilities/errors;
+- backup success/age;
+- process CPU/memory/goroutines.
+
+Avoid high-cardinality path/user/entity labels.
+
+## Health endpoints
+
+- liveness: process loop alive, no dependency checks that cause restart storms;
+- readiness: mandatory dependencies usable;
+- detailed system status: authenticated, shows component state/freshness.
+
+## Logs
+
+Structured production logs:
+- timestamp;
+- level;
+- service/version;
+- correlation/request/job ID;
+- operation;
+- safe entity/resource IDs;
+- error code and wrapped message;
+- duration/result.
+
+Redact:
+- tokens/cookies/authorization;
+- passwords/connection secrets;
+- sensitive headers;
+- full upstream bodies;
+- private keys.
+
+## External dead-man
+
+An independent existing monitor or simple external check should detect:
+- Pulse HTTPS unavailable;
+- Pulse heartbeat absent;
+- backup too old if supported.
+
+Do not rely solely on Pulse to alert that Pulse is completely down.
+
+## System status page
+
+Shows:
+- build/version/commit;
+- database;
+- Prometheus;
+- Unraid/agent;
+- OIDC;
+- worker heartbeats;
+- query health;
+- probes;
+- notifications;
+- backup age;
+- active clients;
+- storage use;
+- last successful critical jobs.
+
+No secret/config values.
+
+## M9-08 operating contract
+
+The public liveness contract is GET /healthz and returns exactly ok with HTTP 200 when the Pulse process is serving requests. It deliberately does not query PostgreSQL or upstream datasources, so an independent monitor can distinguish a dead Pulse process from a dependency outage. GET /readyz is the dependency-aware readiness contract and returns HTTP 503 while the database is unavailable.
+
+The authenticated status surface is GET /api/v1/system/status and is available to viewers. It reports bounded component state, explicit Unknown/Disabled reasons, backup age (Unknown until a verified backup exists), and source lag (Unknown until a source reports an observed timestamp). It never reports unsampled sources as Healthy. Operators with the operate permission can read /api/v1/system/diagnostics and /api/v1/system/metrics; these routes are not public monitoring endpoints.
+
+Configure an independent monitor outside Pulse to run every 60 seconds with two consecutive failures required for alerting:
+
+    python tools/deadman_check.py https://pulse.example.invalid/healthz --timeout 5
+
+The script uses only the Python standard library, refuses embedded credentials, bounds the timeout to 1-30 seconds, accepts only HTTP 200 with the exact ok body, and exits non-zero for transport, timeout, redirect-result, or HTTP failures. Keep the monitor and its alerting path independent of the Pulse API and database.

docs/operations/WORKER_AGENT_HEALTHCHECK_CONTRACT.md

@@ -0,0 +1,118 @@
+# Worker/agent healthcheck contract
+
+Applies to `pulse-worker` (`cmd/worker`) and `pulse-agent` (`cmd/agent`). Both
+are currently foundation stubs (`cmd/worker/main.go`, `cmd/agent/main.go`)
+that will gain real scheduling loops. This document is the contract the Go
+implementation must satisfy so the compose-side healthcheck already wired in
+`deploy/compose.yaml` reports true liveness instead of "process exists".
+
+## Why not an HTTP `/healthz`
+
+`pulse-api` and `pulse-web` already expose a network port, so an HTTP
+`/healthz` costs nothing extra there. `pulse-worker` and `pulse-agent` sit on
+`pulse-internal` only and have no other reason to bind a port; opening one
+solely for a healthcheck would be an unnecessary internal attack surface
+(see `apps/agent/AGENTS.md`: "No generic shell/exec/file-write endpoint" and
+the general least-privilege posture of ADR-0005/ADR-0010). A heartbeat file
+on the already-mounted `tmpfs` `/tmp` needs no new port, no new capability,
+and no new listener.
+
+## What the compose/image side already does
+
+- `deploy/pulse-entrypoint.sh` is the container `ENTRYPOINT`. It writes the
+  current unix time to `/tmp/.pulse-started-at` and then `exec`s the real
+  binary (PID 1 becomes the Go process). This exists only so the healthcheck
+  script can grant a startup grace period independent of Docker's
+  `start_period`, which suppresses unhealthy *status* but does not stop the
+  `HEALTHCHECK CMD` (or its side effects) from running during that window.
+- `deploy/healthcheck-heartbeat.sh` is the `HEALTHCHECK CMD` (also wired
+  explicitly in `deploy/compose.yaml`, `interval=15s timeout=5s
+  start_period=20s retries=3`). It checks the mtime of the heartbeat file
+  described below. If the file is missing for longer than
+  `PULSE_HEARTBEAT_MAX_AGE_SECONDS` (default **45s**) after container start,
+  or exists but its mtime is older than that same threshold, the script logs
+  why and sends `SIGKILL` to PID 1. `docker compose up` (non-swarm) does not
+  restart a container merely because it is reported "unhealthy" — killing
+  PID 1 turns that into a real container exit, which `restart:
+  unless-stopped` then recovers automatically. Same-UID `SIGKILL` needs no
+  Linux capability, so this works under `cap_drop: [ALL]`.
+
+You do not need to change either script or the Dockerfiles to implement this
+contract — only the points below, inside `cmd/worker` / `cmd/agent` and
+whatever internal packages they call.
+
+## What the Go side must implement
+
+1. **Path.** Write the heartbeat to the file named by the
+   `PULSE_HEARTBEAT_FILE` environment variable if set, otherwise
+   `/tmp/healthy`. That path is on the container's `tmpfs` `/tmp` mount
+   (already present in `deploy/compose.yaml` for both services), so no new
+   volume or mount is needed.
+2. **Cadence tied to real progress, not a free-running ticker.** Update the
+   heartbeat only after the main scheduling loop completes an iteration (a
+   scheduler tick, a due-job scan, a lease renewal, a completed
+   discovery/capability-collection pass — whatever the loop's unit of work
+   is for that binary). Do **not** heartbeat from an independent goroutine
+   that ticks on a timer regardless of whether the main loop is stuck — that
+   would silently defeat the whole point of this contract (the exact failure
+   mode `kill -0 1` had).
+3. **Interval.** The loop must complete an iteration, and therefore heartbeat,
+   at least every **10 seconds** even when there is no work to do (an empty
+   poll is still a completed iteration). This gives a comfortable margin
+   under the 45s staleness threshold enforced by
+   `deploy/healthcheck-heartbeat.sh`, tolerating a couple of missed/slow
+   cycles before the container is killed.
+4. **First heartbeat.** Write one heartbeat immediately after startup
+   (config loaded, DB reachable) and before blocking on the first real unit
+   of work, so a slow-but-healthy cold start is not mistaken for a hang.
+5. **Bounded blocking calls.** Every blocking call inside the loop iteration
+   (DB queries, Unraid/agent-protocol calls, probe dials, notification
+   sends) must use a `context` with a timeout well under 10s, and
+   long-running job execution must happen in a separate goroutine so the
+   scheduler loop itself stays responsive. This is also required by
+   `apps/worker/AGENTS.md` ("Every job is idempotent, cancellable, bounded
+   and observable") independent of this contract — the two requirements
+   reinforce each other: if a call isn't bounded, the loop stalls, the
+   heartbeat goes stale, and the container is killed and restarted,
+   correctly surfacing the hang instead of hiding it.
+6. **Write semantics.** A simple truncate-and-write (or an `os.Chtimes` touch
+   if content does not need to change) is sufficient; the healthcheck reads
+   only the file's mtime, not its contents, so no locking or atomic
+   rename is required for correctness. For operator debuggability, write the
+   current UTC time in RFC 3339 (e.g. `2026-08-04T12:00:03Z\n`) as the file
+   content so `docker exec <container> cat /tmp/healthy` is meaningful.
+7. **Failure handling.** If writing the heartbeat file itself fails (e.g.
+   tmpfs write error), log it at error level and continue the loop — do not
+   crash the process for a heartbeat I/O failure alone. A sustained write
+   failure will naturally show up as staleness and get caught by the
+   healthcheck.
+8. **Shutdown.** No special handling is required on `SIGTERM`/graceful
+   shutdown; leaving the heartbeat file in place is fine since the container
+   is stopping anyway (`service.WaitForStop` already handles the
+   signal-driven shutdown path in both `cmd/worker/main.go` and
+   `cmd/agent/main.go`).
+
+## Tunables
+
+| Env var | Default | Set where | Meaning |
+|---|---|---|---|
+| `PULSE_HEARTBEAT_FILE` | `/tmp/healthy` | worker/agent process env (compose) | Path the Go process writes and the healthcheck script reads. |
+| `PULSE_HEARTBEAT_MAX_AGE_SECONDS` | `45` | healthcheck script env (compose, if overridden) | Staleness threshold before the healthcheck force-restarts the container. |
+| `PULSE_STARTED_AT_FILE` | `/tmp/.pulse-started-at` | entrypoint/healthcheck script env (compose, if overridden) | Where the entrypoint records container start time, used only for startup grace. |
+
+None of these are currently set as explicit environment variables in
+`deploy/compose.yaml` (the defaults baked into the scripts are used); add
+them there if the recommended values above ever need to change per
+deployment.
+
+## Reviewer double-check
+
+Self-terminating a container from inside its own healthcheck is an unusual
+pattern. It was chosen because `docker compose up` (non-swarm, confirmed by
+`docs/operations/DEVELOPMENT_SETUP.md`) does not auto-restart merely
+"unhealthy" containers, and an external auto-heal watcher would need Docker
+socket access, which ADR-0005 forbids host-wide. If that trade-off is
+unacceptable, the alternative is to drop the `kill -9 1` and rely on the
+Unraid dashboard surfacing "unhealthy" status for manual operator action —
+but that leaves the original "never restarts" finding only partially fixed
+(visible, not self-healing).