ITWorx-Pulse-Public/docs/operations/WORKER_AGENT_HEALTHCHECK_CONTRACT.md

# Worker/agent healthcheck contract

Applies to `pulse-worker` (`cmd/worker`) and `pulse-agent` (`cmd/agent`). Both
are currently foundation stubs (`cmd/worker/main.go`, `cmd/agent/main.go`)
that will gain real scheduling loops. This document is the contract the Go
implementation must satisfy so the compose-side healthcheck already wired in
`deploy/compose.yaml` reports true liveness instead of "process exists".

## Why not an HTTP `/healthz`

`pulse-api` and `pulse-web` already expose a network port, so an HTTP
`/healthz` costs nothing extra there. `pulse-worker` and `pulse-agent` sit on
`pulse-internal` only and have no other reason to bind a port; opening one
solely for a healthcheck would be an unnecessary internal attack surface
(see `apps/agent/AGENTS.md`: "No generic shell/exec/file-write endpoint" and
the general least-privilege posture of ADR-0005/ADR-0010). A heartbeat file
on the already-mounted `tmpfs` `/tmp` needs no new port, no new capability,
and no new listener.

## What the compose/image side already does

- `deploy/pulse-entrypoint.sh` is the container `ENTRYPOINT`. It writes the
  current unix time to `/tmp/.pulse-started-at` and then `exec`s the real
  binary (PID 1 becomes the Go process). This exists only so the healthcheck
  script can grant a startup grace period independent of Docker's
  `start_period`, which suppresses unhealthy *status* but does not stop the
  `HEALTHCHECK CMD` (or its side effects) from running during that window.
- `deploy/healthcheck-heartbeat.sh` is the `HEALTHCHECK CMD` (also wired
  explicitly in `deploy/compose.yaml`, `interval=15s timeout=5s
  start_period=20s retries=3`). It checks the mtime of the heartbeat file
  described below. If the file is missing for longer than
  `PULSE_HEARTBEAT_MAX_AGE_SECONDS` (default **45s**) after container start,
  or exists but its mtime is older than that same threshold, the script logs
  why and sends `SIGKILL` to PID 1. `docker compose up` (non-swarm) does not
  restart a container merely because it is reported "unhealthy" — killing
  PID 1 turns that into a real container exit, which `restart:
  unless-stopped` then recovers automatically. Same-UID `SIGKILL` needs no
  Linux capability, so this works under `cap_drop: [ALL]`.

You do not need to change either script or the Dockerfiles to implement this
contract — only the points below, inside `cmd/worker` / `cmd/agent` and
whatever internal packages they call.

## What the Go side must implement

1. **Path.** Write the heartbeat to the file named by the
   `PULSE_HEARTBEAT_FILE` environment variable if set, otherwise
   `/tmp/healthy`. That path is on the container's `tmpfs` `/tmp` mount
   (already present in `deploy/compose.yaml` for both services), so no new
   volume or mount is needed.
2. **Cadence tied to real progress, not a free-running ticker.** Update the
   heartbeat only after the main scheduling loop completes an iteration (a
   scheduler tick, a due-job scan, a lease renewal, a completed
   discovery/capability-collection pass — whatever the loop's unit of work
   is for that binary). Do **not** heartbeat from an independent goroutine
   that ticks on a timer regardless of whether the main loop is stuck — that
   would silently defeat the whole point of this contract (the exact failure
   mode `kill -0 1` had).
3. **Interval.** The loop must complete an iteration, and therefore heartbeat,
   at least every **10 seconds** even when there is no work to do (an empty
   poll is still a completed iteration). This gives a comfortable margin
   under the 45s staleness threshold enforced by
   `deploy/healthcheck-heartbeat.sh`, tolerating a couple of missed/slow
   cycles before the container is killed.
4. **First heartbeat.** Write one heartbeat immediately after startup
   (config loaded, DB reachable) and before blocking on the first real unit
   of work, so a slow-but-healthy cold start is not mistaken for a hang.
5. **Bounded blocking calls.** Every blocking call inside the loop iteration
   (DB queries, Unraid/agent-protocol calls, probe dials, notification
   sends) must use a `context` with a timeout well under 10s, and
   long-running job execution must happen in a separate goroutine so the
   scheduler loop itself stays responsive. This is also required by
   `apps/worker/AGENTS.md` ("Every job is idempotent, cancellable, bounded
   and observable") independent of this contract — the two requirements
   reinforce each other: if a call isn't bounded, the loop stalls, the
   heartbeat goes stale, and the container is killed and restarted,
   correctly surfacing the hang instead of hiding it.
6. **Write semantics.** A simple truncate-and-write (or an `os.Chtimes` touch
   if content does not need to change) is sufficient; the healthcheck reads
   only the file's mtime, not its contents, so no locking or atomic
   rename is required for correctness. For operator debuggability, write the
   current UTC time in RFC 3339 (e.g. `2026-08-04T12:00:03Z\n`) as the file
   content so `docker exec <container> cat /tmp/healthy` is meaningful.
7. **Failure handling.** If writing the heartbeat file itself fails (e.g.
   tmpfs write error), log it at error level and continue the loop — do not
   crash the process for a heartbeat I/O failure alone. A sustained write
   failure will naturally show up as staleness and get caught by the
   healthcheck.
8. **Shutdown.** No special handling is required on `SIGTERM`/graceful
   shutdown; leaving the heartbeat file in place is fine since the container
   is stopping anyway (`service.WaitForStop` already handles the
   signal-driven shutdown path in both `cmd/worker/main.go` and
   `cmd/agent/main.go`).

## Tunables

| Env var | Default | Set where | Meaning |
|---|---|---|---|
| `PULSE_HEARTBEAT_FILE` | `/tmp/healthy` | worker/agent process env (compose) | Path the Go process writes and the healthcheck script reads. |
| `PULSE_HEARTBEAT_MAX_AGE_SECONDS` | `45` | healthcheck script env (compose, if overridden) | Staleness threshold before the healthcheck force-restarts the container. |
| `PULSE_STARTED_AT_FILE` | `/tmp/.pulse-started-at` | entrypoint/healthcheck script env (compose, if overridden) | Where the entrypoint records container start time, used only for startup grace. |

None of these are currently set as explicit environment variables in
`deploy/compose.yaml` (the defaults baked into the scripts are used); add
them there if the recommended values above ever need to change per
deployment.

## Reviewer double-check

Self-terminating a container from inside its own healthcheck is an unusual
pattern. It was chosen because `docker compose up` (non-swarm, confirmed by
`docs/operations/DEVELOPMENT_SETUP.md`) does not auto-restart merely
"unhealthy" containers, and an external auto-heal watcher would need Docker
socket access, which ADR-0005 forbids host-wide. If that trade-off is
unacceptable, the alternative is to drop the `kill -9 1` and rely on the
Unraid dashboard surfacing "unhealthy" status for manual operator action —
but that leaves the original "never restarts" finding only partially fixed
(visible, not self-healing).