Publish ITWorx Pulse source

docs/operations/WORKER_AGENT_HEALTHCHECK_CONTRACT.md

@@ -0,0 +1,118 @@
+# Worker/agent healthcheck contract
+
+Applies to `pulse-worker` (`cmd/worker`) and `pulse-agent` (`cmd/agent`). Both
+are currently foundation stubs (`cmd/worker/main.go`, `cmd/agent/main.go`)
+that will gain real scheduling loops. This document is the contract the Go
+implementation must satisfy so the compose-side healthcheck already wired in
+`deploy/compose.yaml` reports true liveness instead of "process exists".
+
+## Why not an HTTP `/healthz`
+
+`pulse-api` and `pulse-web` already expose a network port, so an HTTP
+`/healthz` costs nothing extra there. `pulse-worker` and `pulse-agent` sit on
+`pulse-internal` only and have no other reason to bind a port; opening one
+solely for a healthcheck would be an unnecessary internal attack surface
+(see `apps/agent/AGENTS.md`: "No generic shell/exec/file-write endpoint" and
+the general least-privilege posture of ADR-0005/ADR-0010). A heartbeat file
+on the already-mounted `tmpfs` `/tmp` needs no new port, no new capability,
+and no new listener.
+
+## What the compose/image side already does
+
+- `deploy/pulse-entrypoint.sh` is the container `ENTRYPOINT`. It writes the
+  current unix time to `/tmp/.pulse-started-at` and then `exec`s the real
+  binary (PID 1 becomes the Go process). This exists only so the healthcheck
+  script can grant a startup grace period independent of Docker's
+  `start_period`, which suppresses unhealthy *status* but does not stop the
+  `HEALTHCHECK CMD` (or its side effects) from running during that window.
+- `deploy/healthcheck-heartbeat.sh` is the `HEALTHCHECK CMD` (also wired
+  explicitly in `deploy/compose.yaml`, `interval=15s timeout=5s
+  start_period=20s retries=3`). It checks the mtime of the heartbeat file
+  described below. If the file is missing for longer than
+  `PULSE_HEARTBEAT_MAX_AGE_SECONDS` (default **45s**) after container start,
+  or exists but its mtime is older than that same threshold, the script logs
+  why and sends `SIGKILL` to PID 1. `docker compose up` (non-swarm) does not
+  restart a container merely because it is reported "unhealthy" — killing
+  PID 1 turns that into a real container exit, which `restart:
+  unless-stopped` then recovers automatically. Same-UID `SIGKILL` needs no
+  Linux capability, so this works under `cap_drop: [ALL]`.
+
+You do not need to change either script or the Dockerfiles to implement this
+contract — only the points below, inside `cmd/worker` / `cmd/agent` and
+whatever internal packages they call.
+
+## What the Go side must implement
+
+1. **Path.** Write the heartbeat to the file named by the
+   `PULSE_HEARTBEAT_FILE` environment variable if set, otherwise
+   `/tmp/healthy`. That path is on the container's `tmpfs` `/tmp` mount
+   (already present in `deploy/compose.yaml` for both services), so no new
+   volume or mount is needed.
+2. **Cadence tied to real progress, not a free-running ticker.** Update the
+   heartbeat only after the main scheduling loop completes an iteration (a
+   scheduler tick, a due-job scan, a lease renewal, a completed
+   discovery/capability-collection pass — whatever the loop's unit of work
+   is for that binary). Do **not** heartbeat from an independent goroutine
+   that ticks on a timer regardless of whether the main loop is stuck — that
+   would silently defeat the whole point of this contract (the exact failure
+   mode `kill -0 1` had).
+3. **Interval.** The loop must complete an iteration, and therefore heartbeat,
+   at least every **10 seconds** even when there is no work to do (an empty
+   poll is still a completed iteration). This gives a comfortable margin
+   under the 45s staleness threshold enforced by
+   `deploy/healthcheck-heartbeat.sh`, tolerating a couple of missed/slow
+   cycles before the container is killed.
+4. **First heartbeat.** Write one heartbeat immediately after startup
+   (config loaded, DB reachable) and before blocking on the first real unit
+   of work, so a slow-but-healthy cold start is not mistaken for a hang.
+5. **Bounded blocking calls.** Every blocking call inside the loop iteration
+   (DB queries, Unraid/agent-protocol calls, probe dials, notification
+   sends) must use a `context` with a timeout well under 10s, and
+   long-running job execution must happen in a separate goroutine so the
+   scheduler loop itself stays responsive. This is also required by
+   `apps/worker/AGENTS.md` ("Every job is idempotent, cancellable, bounded
+   and observable") independent of this contract — the two requirements
+   reinforce each other: if a call isn't bounded, the loop stalls, the
+   heartbeat goes stale, and the container is killed and restarted,
+   correctly surfacing the hang instead of hiding it.
+6. **Write semantics.** A simple truncate-and-write (or an `os.Chtimes` touch
+   if content does not need to change) is sufficient; the healthcheck reads
+   only the file's mtime, not its contents, so no locking or atomic
+   rename is required for correctness. For operator debuggability, write the
+   current UTC time in RFC 3339 (e.g. `2026-08-04T12:00:03Z\n`) as the file
+   content so `docker exec <container> cat /tmp/healthy` is meaningful.
+7. **Failure handling.** If writing the heartbeat file itself fails (e.g.
+   tmpfs write error), log it at error level and continue the loop — do not
+   crash the process for a heartbeat I/O failure alone. A sustained write
+   failure will naturally show up as staleness and get caught by the
+   healthcheck.
+8. **Shutdown.** No special handling is required on `SIGTERM`/graceful
+   shutdown; leaving the heartbeat file in place is fine since the container
+   is stopping anyway (`service.WaitForStop` already handles the
+   signal-driven shutdown path in both `cmd/worker/main.go` and
+   `cmd/agent/main.go`).
+
+## Tunables
+
+| Env var | Default | Set where | Meaning |
+|---|---|---|---|
+| `PULSE_HEARTBEAT_FILE` | `/tmp/healthy` | worker/agent process env (compose) | Path the Go process writes and the healthcheck script reads. |
+| `PULSE_HEARTBEAT_MAX_AGE_SECONDS` | `45` | healthcheck script env (compose, if overridden) | Staleness threshold before the healthcheck force-restarts the container. |
+| `PULSE_STARTED_AT_FILE` | `/tmp/.pulse-started-at` | entrypoint/healthcheck script env (compose, if overridden) | Where the entrypoint records container start time, used only for startup grace. |
+
+None of these are currently set as explicit environment variables in
+`deploy/compose.yaml` (the defaults baked into the scripts are used); add
+them there if the recommended values above ever need to change per
+deployment.
+
+## Reviewer double-check
+
+Self-terminating a container from inside its own healthcheck is an unusual
+pattern. It was chosen because `docker compose up` (non-swarm, confirmed by
+`docs/operations/DEVELOPMENT_SETUP.md`) does not auto-restart merely
+"unhealthy" containers, and an external auto-heal watcher would need Docker
+socket access, which ADR-0005 forbids host-wide. If that trade-off is
+unacceptable, the alternative is to drop the `kill -9 1` and rely on the
+Unraid dashboard surfacing "unhealthy" status for manual operator action —
+but that leaves the original "never restarts" finding only partially fixed
+(visible, not self-healing).