# PoC runbook

For the demo-specific 5-minute/10-minute walkthroughs, reset behaviour, Unraid
redeploy/rollback steps and troubleshooting, see `docs/demo-release/demo-runbook.md`.
This document covers general environment bootstrap and n8n setup.
Privacy retention, data-subject exports, anonymisation and incident governance are
defined in `docs/18-privacy-governance.md`.

## Bootstrap (clean checkout)

```bash
cp .env.example .env
make demo
```

`make demo` runs `docker compose up --build -d` (migrations run automatically on API
container startup, see `backend/entrypoint.sh`) and then seeds the deterministic dataset.
Equivalently, without `make`:

```bash
cp .env.example .env
docker compose up --build -d
docker compose exec api python -m app.cli seed --reset
```

Verify:

```bash
curl http://localhost:8128/health/ready    # database-backed readiness
curl -o /dev/null -w "%{http_code}\n" http://localhost:1228/   # 200
make test                                  # isolated test project/database, all tests pass
docker compose run --rm api ruff check .   # clean
```

## Observability

The API emits one JSON log record per request with UTC timestamp, method, route, status,
duration, client IP and a UUID correlation ID. A valid incoming `X-Correlation-Id` is
propagated into the response and API error body; invalid values are replaced. Docker log
rotation is capped at five 10 MB files.

`GET /metrics` exposes Prometheus request counters, latency histograms, in-flight work,
database readiness and operational-versus-synthetic outbox state. The endpoint is only
reachable inside the production Compose network. If it is exposed elsewhere, configure
`METRICS_BEARER_TOKEN` and send it as a Bearer token.

Start the optional pinned monitoring stack with:

```bash
docker compose -f compose.yaml -f compose.unraid.yaml -f compose.observability.yaml \
  --profile observability up -d prometheus grafana
```

Prometheus listens on host loopback port 19090 and Grafana on loopback port 13000. Set a
unique `GRAFANA_ADMIN_PASSWORD` before first start. Provisioning includes the MobilityOps
overview dashboard and alerts for API/database outage, 5xx rate, p95 latency, real outbox
backlog and real outbox failures. Synthetic retry scenarios never trigger outbox alerts.

The hourly `live-canary.yml` workflow independently checks HTTPS readiness, the TLS
certificate horizon and the non-destructive Chromium/Firefox production journey. Configure
the optional `LIVE_CANARY_HEARTBEAT_URL` repository secret to make missed scheduled jobs
visible in an external dead-man monitor. A successful app release does not restart the
database, backup or monitoring services; use `deploy/unraid/refresh-infrastructure.sh`
deliberately when those definitions change.

Never execute `pytest` inside the deployed API container: the acceptance fixtures reset
their database deliberately. `make test` uses `compose.test.yaml`, a fixed
`mobilityops-test` Compose project and its own disposable PostgreSQL volume, and removes
that project on success or failure. The Gitea workflow uses the same isolation boundary.
CI also runs `scripts/run-readonly-load-smoke.py` against persisted list/dashboard routes;
the gate requires zero HTTP errors and a p95 below 1.5 seconds at its bounded concurrency.

## Operational mode (non-demo login)

Keep the current demonstration environment on `MOBILITYOPS_DEMO_MODE=true`. For an
operational deployment, set `MOBILITYOPS_DEMO_MODE=false`, `DEMO_ALLOW_RESET=false` and
provide `INITIAL_ADMIN_EMAIL`, `INITIAL_ADMIN_PASSWORD` (at least 8 characters) and an
optional `INITIAL_ADMIN_DISPLAY_NAME` in the deployment's untracked secret environment.
On startup the API creates the first active Operations Manager only when no user with that
email exists. The sign-in page then accepts email/password instead of exposing demo roles;
demo reset, the guided tour and the synthetic-data badge are hidden.

Use a long unique `APP_SECRET`; production requires an HTTPS public endpoint and
`SESSION_COOKIE_SECURE=true`. Keep `INITIAL_ADMIN_PASSWORD` out of Git and logs. Existing sessions are
revalidated against the current user record on every request, so deactivating an account
invalidates its next request.

### Optional organisation login (OIDC)

OpenID Connect can coexist with the public demo. Set `OIDC_ENABLED=true`, issuer URL,
client ID and client secret; register
`<MOBILITYOPS_PUBLIC_URL>/api/v1/auth/oidc/callback` at the identity provider. The login
screen then adds an organisation-login action without removing either public demo role.
State and nonce validation use a short-lived signed HttpOnly cookie. Identity binding is
unique on issuer plus `sub`; only a verified email is accepted. Optionally restrict
domains with `OIDC_ALLOWED_EMAIL_DOMAINS`. New identities receive the least-privileged
`rental_employee` role by default and can subsequently be promoted by an Operations
Manager. Set `OIDC_AUTO_PROVISION=false` when every account must be pre-created.

OIDC must use HTTPS outside a trusted local network. Set `SESSION_COOKIE_SECURE=true` and
keep `OIDC_CLIENT_SECRET` in the deployment secret store. Disabling OIDC immediately
removes the organisation-login action but does not affect public demo access.

## n8n automation (one-time per environment)

The optional bundled fallback image (`n8nio/n8n:2.33.7`) requires an owner account before any
workflow — including webhook registration — works reliably; `N8N_BASIC_AUTH_ACTIVE` no
longer gates this. This is a one-time step per fresh `docker compose down -v`:

1. Open `http://localhost:5678/setup` and create an owner account (any email/password
   meeting the 8+ characters / 1 number / 1 capital rule — no email verification is
   required). Skip the optional survey/license-key dialogs that follow.
2. Import and activate the return-processing workflow:

   ```bash
   make n8n-setup
   ```

   which runs:

   ```bash
   docker compose exec n8n n8n import:workflow --input=//imports/workflows/fleet-ops-vehicle-return.json
   docker compose exec n8n n8n publish:workflow --id=mobilityops-return-processing
   docker compose restart n8n
   ```

   (`n8n import:workflow` always leaves the workflow deactivated regardless of its
   `"active"` field; `publish:workflow` + a restart is what actually activates it.)

   Before it will actually process a return, create two Header Auth credentials in the n8n
   UI — `Fleet Ops Webhook Trigger Token` (value: `MOBILITYOPS_WEBHOOK_TRIGGER_TOKEN` from
   `.env`) and `Fleet Ops Service Token` (value: `MOBILITYOPS_CALLBACK_TOKEN` from `.env`) —
   the workflow's webhook trigger and outbound HTTP call reference these credentials by
   name; no secret value is embedded in the workflow file itself.

Verify the full round trip:

```bash
# after logging in and registering any return via the UI or API
curl -b cookies.txt http://localhost:8128/api/v1/workflows | grep succeeded
```

A failed/offline n8n does not roll back the return — the outbox event simply stays
`pending`/`failed` and is safely retryable from the Automation page.

### Second workflow: scheduled quality scan

Import and publish the same way:

```bash
make n8n-setup-scan
```

which runs:

```bash
docker compose exec n8n n8n import:workflow --input=//imports/workflows/fleet-ops-data-quality-scan.json
docker compose exec n8n n8n publish:workflow --id=mobilityops-scheduled-quality-scan
docker compose restart n8n
```

This workflow also needs the `Fleet Ops Service Token` Header Auth credential created in
the n8n UI before a run will succeed.

Verify:

```bash
curl -X POST http://localhost:8128/api/v1/integrations/n8n/scheduled-scan \
  -H "X-Service-Token: <MOBILITYOPS_CALLBACK_TOKEN from .env>"
# {"created": {...}}
```

Trigger a live run from n8n's own UI ("Manual test trigger" node → Execute Workflow) to
confirm the round trip without waiting for the hourly schedule. It does not depend on
RAGcore or MCP Hub and ships `"active": false`, so it never fires anywhere until
deliberately published with a real service token.

### Existing shared n8n on the Unraid review server

The Unraid deployment uses the existing n8n at `https://n8n.itworx.tech`; it does not
start MobilityOps's bundled n8n service. `compose.unraid.yaml` places that fallback behind
the opt-in `bundled-n8n` profile. Configure the API target and publish the workflow with:

```bash
sed -i \
  's|^N8N_WEBHOOK_URL=.*|N8N_WEBHOOK_URL=https://n8n.itworx.tech/webhook/mobilityops-return|' \
  .env
./deploy/unraid/setup-existing-n8n.sh \
  n8n \
  https://fleetops.itworx.tech/api/v1/integrations/n8n/return-callback
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up -d db api web
```

For an update of already configured workflows, preserve the live credential IDs. n8n
2.x's CLI can bind every Header Auth node to the same credential when two credentials of
that type exist, even when the committed names differ. Never import the name-only JSON
directly over a live workflow. Export first, merge only the non-secret references, then
import the generated files:

```bash
docker exec n8n n8n export:workflow --all \
  --output=/data/backups/mobilityops-pre-update.json
python n8n/workflows/merge_credential_refs.py \
  /mnt/cache/appdata/n8n/backups/mobilityops-pre-update.json \
  n8n/workflows /mnt/cache/appdata/n8n/imports/mobilityops-safe
# import each generated fleet-ops-*.json, publish the four known IDs, then restart n8n
```

The merge utility copies credential IDs and names only; credential values remain inside
n8n's encrypted credential store. After restart, export the return workflow and verify
that `Return webhook` uses `Fleet Ops Webhook Trigger Token`, while `Record follow-up`
and `Report workflow heartbeat` use `Fleet Ops Service Token`. Finally run a real return
and verify both a succeeded outbox delivery and a heartbeat execution ID in Automation.

The setup script reads the callback token from the mode-0600 deployment `.env`, builds and
removes a temporary server-side import without writing the token to Git, and restarts the
existing n8n so the production webhook is registered. The imported configuration remains
inside n8n's protected application data. The callback travels through the MobilityOps web
proxy, so the shared n8n container does not need direct database access or membership of
the MobilityOps Docker network.

Publish the scheduled quality-scan workflow the same way:

```bash
./deploy/unraid/setup-scheduled-scan.sh \
  n8n \
  https://fleetops.itworx.tech/api/v1/integrations/n8n/scheduled-scan
```

## Required operational checks

- API liveness (`GET /health/live`), database-backed readiness (`GET /health/ready`) and
  web health (web root `200`);
- database migration level (`docker compose exec api alembic current`);
- pending/failed outbox count (Automation page, or `GET /api/v1/workflows?status=failed`);
- RAGcore provider state (`GET /api/v1/knowledge/status`; demo provider is always
  `available`, RAGcore adapter reports `unavailable` when unreachable);
- n8n connectivity (`docker compose logs n8n`, or submit a return and watch `/automation`);
- MCP provider endpoint authorization (`curl` the four `/api/v1/integrations/mcp/*`
  routes with and without a valid `X-Service-Token` — see
  `artifacts/evidence/final-summary.md` for sample calls);
- deterministic demo reset (`POST /api/v1/demo/reset` as Operations Manager, or `make seed`).

## Recovery expectations

- database restart: application reconnects (SQLAlchemy connection pool, `pool_pre_ping=True`);
- n8n outage: events remain `pending` and are retried with exponential backoff, then
  `failed` after 5 attempts and safely retryable from `/automation`;
- RAGcore outage: `/knowledge` shows `unavailable`, all operational pages continue working;
- MCP Hub outage: the web application is unaffected — MCP endpoints are a separate,
  independently-authenticated API surface;
- failed demo experiment: Operations Manager reset (`POST /api/v1/demo/reset`) restores
  the deterministic seed, including all named S1–S6 demo scenarios.
- loss of the Unraid host: recover the latest OneDrive dump only after its downloaded
  checksum and `pg_restore --list` pass; use the guarded restore command above. OneDrive
  synchronization health requires both a daily verified round trip and a weekly real
  restore drill. OAuth setup is documented in `deploy/unraid/README.md`.