M22: implement operational observability

docs/17-runbook.md

@@ -30,6 +30,30 @@ make test                                  # isolated test project/database, all
 docker compose run --rm api ruff check .   # clean
 ```
 
+## Observability
+
+The API emits one JSON log record per request with UTC timestamp, method, route, status,
+duration, client IP and a UUID correlation ID. A valid incoming `X-Correlation-Id` is
+propagated into the response and API error body; invalid values are replaced. Docker log
+rotation is capped at five 10 MB files.
+
+`GET /metrics` exposes Prometheus request counters, latency histograms, in-flight work,
+database readiness and operational-versus-synthetic outbox state. The endpoint is only
+reachable inside the production Compose network. If it is exposed elsewhere, configure
+`METRICS_BEARER_TOKEN` and send it as a Bearer token.
+
+Start the optional pinned monitoring stack with:
+
+```bash
+docker compose -f compose.yaml -f compose.unraid.yaml -f compose.observability.yaml \
+  --profile observability up -d prometheus grafana
+```
+
+Prometheus listens on host loopback port 19090 and Grafana on loopback port 13000. Set a
+unique `GRAFANA_ADMIN_PASSWORD` before first start. Provisioning includes the MobilityOps
+overview dashboard and alerts for API/database outage, 5xx rate, p95 latency, real outbox
+backlog and real outbox failures. Synthetic retry scenarios never trigger outbox alerts.
+
 Never execute `pytest` inside the deployed API container: the acceptance fixtures reset
 their database deliberately. `make test` uses `compose.test.yaml`, a fixed
 `mobilityops-test` Compose project and its own disposable PostgreSQL volume, and removes