258 lines
12 KiB
Markdown
258 lines
12 KiB
Markdown
# PoC runbook
|
||
|
||
For the demo-specific 5-minute/10-minute walkthroughs, reset behaviour, Unraid
|
||
redeploy/rollback steps and troubleshooting, see `docs/demo-release/demo-runbook.md`.
|
||
This document covers general environment bootstrap and n8n setup.
|
||
Privacy retention, data-subject exports, anonymisation and incident governance are
|
||
defined in `docs/18-privacy-governance.md`.
|
||
|
||
## Bootstrap (clean checkout)
|
||
|
||
```bash
|
||
cp .env.example .env
|
||
make demo
|
||
```
|
||
|
||
`make demo` runs `docker compose up --build -d` (migrations run automatically on API
|
||
container startup, see `backend/entrypoint.sh`) and then seeds the deterministic dataset.
|
||
Equivalently, without `make`:
|
||
|
||
```bash
|
||
cp .env.example .env
|
||
docker compose up --build -d
|
||
docker compose exec api python -m app.cli seed --reset
|
||
```
|
||
|
||
Verify:
|
||
|
||
```bash
|
||
curl http://localhost:8128/health/ready # database-backed readiness
|
||
curl -o /dev/null -w "%{http_code}\n" http://localhost:1228/ # 200
|
||
make test # isolated test project/database, all tests pass
|
||
docker compose run --rm api ruff check . # clean
|
||
```
|
||
|
||
## Observability
|
||
|
||
The API emits one JSON log record per request with UTC timestamp, method, route, status,
|
||
duration, client IP and a UUID correlation ID. A valid incoming `X-Correlation-Id` is
|
||
propagated into the response and API error body; invalid values are replaced. Docker log
|
||
rotation is capped at five 10 MB files.
|
||
|
||
`GET /metrics` exposes Prometheus request counters, latency histograms, in-flight work,
|
||
database readiness and operational-versus-synthetic outbox state. The endpoint is only
|
||
reachable inside the production Compose network. If it is exposed elsewhere, configure
|
||
`METRICS_BEARER_TOKEN` and send it as a Bearer token.
|
||
|
||
Start the optional pinned monitoring stack with:
|
||
|
||
```bash
|
||
docker compose -f compose.yaml -f compose.unraid.yaml -f compose.observability.yaml \
|
||
--profile observability up -d prometheus grafana
|
||
```
|
||
|
||
Prometheus listens on host loopback port 19090 and Grafana on loopback port 13000. Set a
|
||
unique `GRAFANA_ADMIN_PASSWORD` before first start. Provisioning includes the MobilityOps
|
||
overview dashboard and alerts for API/database outage, 5xx rate, p95 latency, real outbox
|
||
backlog and real outbox failures. Synthetic retry scenarios never trigger outbox alerts.
|
||
|
||
The hourly `live-canary.yml` workflow independently checks HTTPS readiness, the TLS
|
||
certificate horizon and the non-destructive Chromium/Firefox production journey. Configure
|
||
the optional `LIVE_CANARY_HEARTBEAT_URL` repository secret to make missed scheduled jobs
|
||
visible in an external dead-man monitor. A successful app release does not restart the
|
||
database, backup or monitoring services; use `deploy/unraid/refresh-infrastructure.sh`
|
||
deliberately when those definitions change.
|
||
|
||
Never execute `pytest` inside the deployed API container: the acceptance fixtures reset
|
||
their database deliberately. `make test` uses `compose.test.yaml`, a fixed
|
||
`mobilityops-test` Compose project and its own disposable PostgreSQL volume, and removes
|
||
that project on success or failure. The Gitea workflow uses the same isolation boundary.
|
||
CI also runs `scripts/run-readonly-load-smoke.py` against persisted list/dashboard routes;
|
||
the gate requires zero HTTP errors and a p95 below 1.5 seconds at its bounded concurrency.
|
||
|
||
## Operational mode (non-demo login)
|
||
|
||
Keep the current demonstration environment on `MOBILITYOPS_DEMO_MODE=true`. For an
|
||
operational deployment, set `MOBILITYOPS_DEMO_MODE=false`, `DEMO_ALLOW_RESET=false` and
|
||
provide `INITIAL_ADMIN_EMAIL`, `INITIAL_ADMIN_PASSWORD` (at least 8 characters) and an
|
||
optional `INITIAL_ADMIN_DISPLAY_NAME` in the deployment's untracked secret environment.
|
||
On startup the API creates the first active Operations Manager only when no user with that
|
||
email exists. The sign-in page then accepts email/password instead of exposing demo roles;
|
||
demo reset, the guided tour and the synthetic-data badge are hidden.
|
||
|
||
Use a long unique `APP_SECRET`; production requires an HTTPS public endpoint and
|
||
`SESSION_COOKIE_SECURE=true`. Keep `INITIAL_ADMIN_PASSWORD` out of Git and logs. Existing sessions are
|
||
revalidated against the current user record on every request, so deactivating an account
|
||
invalidates its next request.
|
||
|
||
### Optional organisation login (OIDC)
|
||
|
||
OpenID Connect can coexist with the public demo. Set `OIDC_ENABLED=true`, issuer URL,
|
||
client ID and client secret; register
|
||
`<MOBILITYOPS_PUBLIC_URL>/api/v1/auth/oidc/callback` at the identity provider. The login
|
||
screen then adds an organisation-login action without removing either public demo role.
|
||
State and nonce validation use a short-lived signed HttpOnly cookie. Identity binding is
|
||
unique on issuer plus `sub`; only a verified email is accepted. Optionally restrict
|
||
domains with `OIDC_ALLOWED_EMAIL_DOMAINS`. New identities receive the least-privileged
|
||
`rental_employee` role by default and can subsequently be promoted by an Operations
|
||
Manager. Set `OIDC_AUTO_PROVISION=false` when every account must be pre-created.
|
||
|
||
OIDC must use HTTPS outside a trusted local network. Set `SESSION_COOKIE_SECURE=true` and
|
||
keep `OIDC_CLIENT_SECRET` in the deployment secret store. Disabling OIDC immediately
|
||
removes the organisation-login action but does not affect public demo access.
|
||
|
||
## n8n automation (one-time per environment)
|
||
|
||
The optional bundled fallback image (`n8nio/n8n:2.33.7`) requires an owner account before any
|
||
workflow — including webhook registration — works reliably; `N8N_BASIC_AUTH_ACTIVE` no
|
||
longer gates this. This is a one-time step per fresh `docker compose down -v`:
|
||
|
||
1. Open `http://localhost:5678/setup` and create an owner account (any email/password
|
||
meeting the 8+ characters / 1 number / 1 capital rule — no email verification is
|
||
required). Skip the optional survey/license-key dialogs that follow.
|
||
2. Import and activate the return-processing workflow:
|
||
|
||
```bash
|
||
make n8n-setup
|
||
```
|
||
|
||
which runs:
|
||
|
||
```bash
|
||
docker compose exec n8n n8n import:workflow --input=//imports/workflows/fleet-ops-vehicle-return.json
|
||
docker compose exec n8n n8n publish:workflow --id=mobilityops-return-processing
|
||
docker compose restart n8n
|
||
```
|
||
|
||
(`n8n import:workflow` always leaves the workflow deactivated regardless of its
|
||
`"active"` field; `publish:workflow` + a restart is what actually activates it.)
|
||
|
||
Before it will actually process a return, create two Header Auth credentials in the n8n
|
||
UI — `Fleet Ops Webhook Trigger Token` (value: `MOBILITYOPS_WEBHOOK_TRIGGER_TOKEN` from
|
||
`.env`) and `Fleet Ops Service Token` (value: `MOBILITYOPS_CALLBACK_TOKEN` from `.env`) —
|
||
the workflow's webhook trigger and outbound HTTP call reference these credentials by
|
||
name; no secret value is embedded in the workflow file itself.
|
||
|
||
Verify the full round trip:
|
||
|
||
```bash
|
||
# after logging in and registering any return via the UI or API
|
||
curl -b cookies.txt http://localhost:8128/api/v1/workflows | grep succeeded
|
||
```
|
||
|
||
A failed/offline n8n does not roll back the return — the outbox event simply stays
|
||
`pending`/`failed` and is safely retryable from the Automation page.
|
||
|
||
### Second workflow: scheduled quality scan
|
||
|
||
Import and publish the same way:
|
||
|
||
```bash
|
||
make n8n-setup-scan
|
||
```
|
||
|
||
which runs:
|
||
|
||
```bash
|
||
docker compose exec n8n n8n import:workflow --input=//imports/workflows/fleet-ops-data-quality-scan.json
|
||
docker compose exec n8n n8n publish:workflow --id=mobilityops-scheduled-quality-scan
|
||
docker compose restart n8n
|
||
```
|
||
|
||
This workflow also needs the `Fleet Ops Service Token` Header Auth credential created in
|
||
the n8n UI before a run will succeed.
|
||
|
||
Verify:
|
||
|
||
```bash
|
||
curl -X POST http://localhost:8128/api/v1/integrations/n8n/scheduled-scan \
|
||
-H "X-Service-Token: <MOBILITYOPS_CALLBACK_TOKEN from .env>"
|
||
# {"created": {...}}
|
||
```
|
||
|
||
Trigger a live run from n8n's own UI ("Manual test trigger" node → Execute Workflow) to
|
||
confirm the round trip without waiting for the hourly schedule. It does not depend on
|
||
RAGcore or MCP Hub and ships `"active": false`, so it never fires anywhere until
|
||
deliberately published with a real service token.
|
||
|
||
### Existing shared n8n on the Unraid review server
|
||
|
||
The Unraid deployment uses the existing n8n at `https://n8n.itworx.tech`; it does not
|
||
start MobilityOps's bundled n8n service. `compose.unraid.yaml` places that fallback behind
|
||
the opt-in `bundled-n8n` profile. Configure the API target and publish the workflow with:
|
||
|
||
```bash
|
||
sed -i \
|
||
's|^N8N_WEBHOOK_URL=.*|N8N_WEBHOOK_URL=https://n8n.itworx.tech/webhook/mobilityops-return|' \
|
||
.env
|
||
./deploy/unraid/setup-existing-n8n.sh \
|
||
n8n \
|
||
https://fleetops.itworx.tech/api/v1/integrations/n8n/return-callback
|
||
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up -d db api web
|
||
```
|
||
|
||
For an update of already configured workflows, preserve the live credential IDs. n8n
|
||
2.x's CLI can bind every Header Auth node to the same credential when two credentials of
|
||
that type exist, even when the committed names differ. Never import the name-only JSON
|
||
directly over a live workflow. Export first, merge only the non-secret references, then
|
||
import the generated files:
|
||
|
||
```bash
|
||
docker exec n8n n8n export:workflow --all \
|
||
--output=/data/backups/mobilityops-pre-update.json
|
||
python n8n/workflows/merge_credential_refs.py \
|
||
/mnt/cache/appdata/n8n/backups/mobilityops-pre-update.json \
|
||
n8n/workflows /mnt/cache/appdata/n8n/imports/mobilityops-safe
|
||
# import each generated fleet-ops-*.json, publish the four known IDs, then restart n8n
|
||
```
|
||
|
||
The merge utility copies credential IDs and names only; credential values remain inside
|
||
n8n's encrypted credential store. After restart, export the return workflow and verify
|
||
that `Return webhook` uses `Fleet Ops Webhook Trigger Token`, while `Record follow-up`
|
||
and `Report workflow heartbeat` use `Fleet Ops Service Token`. Finally run a real return
|
||
and verify both a succeeded outbox delivery and a heartbeat execution ID in Automation.
|
||
|
||
The setup script reads the callback token from the mode-0600 deployment `.env`, builds and
|
||
removes a temporary server-side import without writing the token to Git, and restarts the
|
||
existing n8n so the production webhook is registered. The imported configuration remains
|
||
inside n8n's protected application data. The callback travels through the MobilityOps web
|
||
proxy, so the shared n8n container does not need direct database access or membership of
|
||
the MobilityOps Docker network.
|
||
|
||
Publish the scheduled quality-scan workflow the same way:
|
||
|
||
```bash
|
||
./deploy/unraid/setup-scheduled-scan.sh \
|
||
n8n \
|
||
https://fleetops.itworx.tech/api/v1/integrations/n8n/scheduled-scan
|
||
```
|
||
|
||
## Required operational checks
|
||
|
||
- API liveness (`GET /health/live`), database-backed readiness (`GET /health/ready`) and
|
||
web health (web root `200`);
|
||
- database migration level (`docker compose exec api alembic current`);
|
||
- pending/failed outbox count (Automation page, or `GET /api/v1/workflows?status=failed`);
|
||
- RAGcore provider state (`GET /api/v1/knowledge/status`; demo provider is always
|
||
`available`, RAGcore adapter reports `unavailable` when unreachable);
|
||
- n8n connectivity (`docker compose logs n8n`, or submit a return and watch `/automation`);
|
||
- MCP provider endpoint authorization (`curl` the four `/api/v1/integrations/mcp/*`
|
||
routes with and without a valid `X-Service-Token` — see
|
||
`artifacts/evidence/final-summary.md` for sample calls);
|
||
- deterministic demo reset (`POST /api/v1/demo/reset` as Operations Manager, or `make seed`).
|
||
|
||
## Recovery expectations
|
||
|
||
- database restart: application reconnects (SQLAlchemy connection pool, `pool_pre_ping=True`);
|
||
- n8n outage: events remain `pending` and are retried with exponential backoff, then
|
||
`failed` after 5 attempts and safely retryable from `/automation`;
|
||
- RAGcore outage: `/knowledge` shows `unavailable`, all operational pages continue working;
|
||
- MCP Hub outage: the web application is unaffected — MCP endpoints are a separate,
|
||
independently-authenticated API surface;
|
||
- failed demo experiment: Operations Manager reset (`POST /api/v1/demo/reset`) restores
|
||
the deterministic seed, including all named S1–S6 demo scenarios.
|
||
- loss of the Unraid host: recover the latest OneDrive dump only after its downloaded
|
||
checksum and `pg_restore --list` pass; use the guarded restore command above. OneDrive
|
||
synchronization health requires both a daily verified round trip and a weekly real
|
||
restore drill. OAuth setup is documented in `deploy/unraid/README.md`.
|