MobilityOps/docs/17-runbook.md

PoC runbook

For the demo-specific 5-minute/10-minute walkthroughs, reset behaviour, Unraid redeploy/rollback steps and troubleshooting, see docs/demo-release/demo-runbook.md. This document covers general environment bootstrap and n8n setup.

Bootstrap (clean checkout)

cp .env.example .env
make demo

make demo runs docker compose up --build -d (migrations run automatically on API container startup, see backend/entrypoint.sh) and then seeds the deterministic dataset. Equivalently, without make:

cp .env.example .env
docker compose up --build -d
docker compose exec api python -m app.cli seed --reset

Verify:

curl http://localhost:8128/health/ready    # database-backed readiness
curl -o /dev/null -w "%{http_code}\n" http://localhost:1228/   # 200
make test                                  # isolated test project/database, all tests pass
docker compose run --rm api ruff check .   # clean

Never execute pytest inside the deployed API container: the acceptance fixtures reset their database deliberately. make test uses compose.test.yaml, a fixed mobilityops-test Compose project and its own disposable PostgreSQL volume, and removes that project on success or failure. The Gitea workflow uses the same isolation boundary.

Operational mode (non-demo login)

Keep the current demonstration environment on MOBILITYOPS_DEMO_MODE=true. For an operational deployment, set MOBILITYOPS_DEMO_MODE=false, DEMO_ALLOW_RESET=false and provide INITIAL_ADMIN_EMAIL, INITIAL_ADMIN_PASSWORD (at least 8 characters) and an optional INITIAL_ADMIN_DISPLAY_NAME in the deployment's untracked secret environment. On startup the API creates the first active Operations Manager only when no user with that email exists. The sign-in page then accepts email/password instead of exposing demo roles; demo reset, the guided tour and the synthetic-data badge are hidden.

Use a long unique APP_SECRET, set SESSION_COOKIE_SECURE=true once the public endpoint uses HTTPS, and keep INITIAL_ADMIN_PASSWORD out of Git and logs. Existing sessions are revalidated against the current user record on every request, so deactivating an account invalidates its next request.

n8n automation (one-time per environment)

The optional bundled fallback image (n8nio/n8n:2.33.7) requires an owner account before any workflow — including webhook registration — works reliably; N8N_BASIC_AUTH_ACTIVE no longer gates this. This is a one-time step per fresh docker compose down -v:

1. Open http://localhost:5678/setup and create an owner account (any email/password meeting the 8+ characters / 1 number / 1 capital rule — no email verification is required). Skip the optional survey/license-key dialogs that follow.

2. Import and activate the return-processing workflow:

   make n8n-setup
   

   which runs:

   docker compose exec n8n n8n import:workflow --input=//imports/workflows/fleet-ops-vehicle-return.json
   docker compose exec n8n n8n publish:workflow --id=mobilityops-return-processing
   docker compose restart n8n
   

   (n8n import:workflow always leaves the workflow deactivated regardless of its "active" field; publish:workflow + a restart is what actually activates it.)

   Before it will actually process a return, create two Header Auth credentials in the n8n UI — Fleet Ops Webhook Trigger Token (value: MOBILITYOPS_WEBHOOK_TRIGGER_TOKEN from .env) and Fleet Ops Service Token (value: MOBILITYOPS_CALLBACK_TOKEN from .env) — the workflow's webhook trigger and outbound HTTP call reference these credentials by name; no secret value is embedded in the workflow file itself.

Verify the full round trip:

# after logging in and registering any return via the UI or API
curl -b cookies.txt http://localhost:8128/api/v1/workflows | grep succeeded

A failed/offline n8n does not roll back the return — the outbox event simply stays pending/failed and is safely retryable from the Automation page.

Second workflow: scheduled quality scan

Import and publish the same way:

make n8n-setup-scan

which runs:

docker compose exec n8n n8n import:workflow --input=//imports/workflows/fleet-ops-data-quality-scan.json
docker compose exec n8n n8n publish:workflow --id=mobilityops-scheduled-quality-scan
docker compose restart n8n

This workflow also needs the Fleet Ops Service Token Header Auth credential created in the n8n UI before a run will succeed.

Verify:

curl -X POST http://localhost:8128/api/v1/integrations/n8n/scheduled-scan \
  -H "X-Service-Token: <MOBILITYOPS_CALLBACK_TOKEN from .env>"
# {"created": {...}}

Trigger a live run from n8n's own UI ("Manual test trigger" node → Execute Workflow) to confirm the round trip without waiting for the hourly schedule. It does not depend on RAGcore or MCP Hub and ships "active": false, so it never fires anywhere until deliberately published with a real service token.

Existing shared n8n on the Unraid review server

The Unraid deployment uses the existing n8n at http://192.168.10.150:5678; it does not start MobilityOps's bundled n8n service. compose.unraid.yaml places that fallback behind the opt-in bundled-n8n profile. Configure the API target and publish the workflow with:

sed -i \
  's|^N8N_WEBHOOK_URL=.*|N8N_WEBHOOK_URL=http://192.168.10.150:5678/webhook/mobilityops-return|' \
  .env
./deploy/unraid/setup-existing-n8n.sh \
  n8n \
  http://192.168.10.150:1236/api/v1/integrations/n8n/return-callback
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up -d db api web

For an update of already configured workflows, preserve the live credential IDs. n8n 2.x's CLI can bind every Header Auth node to the same credential when two credentials of that type exist, even when the committed names differ. Never import the name-only JSON directly over a live workflow. Export first, merge only the non-secret references, then import the generated files:

docker exec n8n n8n export:workflow --all \
  --output=/data/backups/mobilityops-pre-update.json
python n8n/workflows/merge_credential_refs.py \
  /mnt/cache/appdata/n8n/backups/mobilityops-pre-update.json \
  n8n/workflows /mnt/cache/appdata/n8n/imports/mobilityops-safe
# import each generated fleet-ops-*.json, publish the four known IDs, then restart n8n

The merge utility copies credential IDs and names only; credential values remain inside n8n's encrypted credential store. After restart, export the return workflow and verify that Return webhook uses Fleet Ops Webhook Trigger Token, while Record follow-up and Report workflow heartbeat use Fleet Ops Service Token. Finally run a real return and verify both a succeeded outbox delivery and a heartbeat execution ID in Automation.

The setup script reads the callback token from the mode-0600 deployment .env, builds and removes a temporary server-side import without writing the token to Git, and restarts the existing n8n so the production webhook is registered. The imported configuration remains inside n8n's protected application data. The callback travels through the MobilityOps web proxy, so the shared n8n container does not need direct database access or membership of the MobilityOps Docker network.

Publish the scheduled quality-scan workflow the same way:

./deploy/unraid/setup-scheduled-scan.sh \
  n8n \
  http://192.168.10.150:1236/api/v1/integrations/n8n/scheduled-scan

Required operational checks

• API liveness (GET /health/live), database-backed readiness (GET /health/ready) and web health (web root 200);
• database migration level (docker compose exec api alembic current);
• pending/failed outbox count (Automation page, or GET /api/v1/workflows?status=failed);
• RAGcore provider state (GET /api/v1/knowledge/status; demo provider is always available, RAGcore adapter reports unavailable when unreachable);
• n8n connectivity (docker compose logs n8n, or submit a return and watch /automation);
• MCP provider endpoint authorization (curl the four /api/v1/integrations/mcp/* routes with and without a valid X-Service-Token — see artifacts/evidence/final-summary.md for sample calls);
• deterministic demo reset (POST /api/v1/demo/reset as Operations Manager, or make seed).

Recovery expectations

• database restart: application reconnects (SQLAlchemy connection pool, pool_pre_ping=True);
• n8n outage: events remain pending and are retried with exponential backoff, then failed after 5 attempts and safely retryable from /automation;
• RAGcore outage: /knowledge shows unavailable, all operational pages continue working;
• MCP Hub outage: the web application is unaffected — MCP endpoints are a separate, independently-authenticated API surface;
• failed demo experiment: Operations Manager reset (POST /api/v1/demo/reset) restores the deterministic seed, including all named S1–S6 demo scenarios.