MobilityOps/docs/17-runbook.md

PoC runbook

Bootstrap (clean checkout)

cp .env.example .env
make demo

make demo runs docker compose up --build -d (migrations run automatically on API container startup, see backend/entrypoint.sh) and then seeds the deterministic dataset. Equivalently, without make:

cp .env.example .env
docker compose up --build -d
docker compose exec api python -m app.cli seed --reset

Verify:

curl http://localhost:8128/health          # {"status":"ok",...}
curl -o /dev/null -w "%{http_code}\n" http://localhost:1228/   # 200
docker compose run --rm api pytest -q      # all tests pass
docker compose run --rm api ruff check .   # clean

n8n automation (one-time per environment)

The n8n image used here (n8nio/n8n:latest, 2.x) requires an owner account before any workflow — including webhook registration — works reliably; N8N_BASIC_AUTH_ACTIVE no longer gates this. This is a one-time step per fresh docker compose down -v:

1. Open http://localhost:5678/setup and create an owner account (any email/password meeting the 8+ characters / 1 number / 1 capital rule — no email verification is required). Skip the optional survey/license-key dialogs that follow.

2. Import and activate the return-processing workflow:

   make n8n-setup
   

   which runs:

   docker compose exec n8n n8n import:workflow --input=//imports/mobilityops-return-processing.json
   docker compose exec n8n n8n publish:workflow --id=mobilityops-return-processing
   docker compose restart n8n
   

   (n8n import:workflow always leaves the workflow deactivated regardless of its "active" field; publish:workflow + a restart is what actually activates it.)

Verify the full round trip:

# after logging in and registering any return via the UI or API
curl -b cookies.txt http://localhost:8128/api/v1/workflows | grep succeeded

A failed/offline n8n does not roll back the return — the outbox event simply stays pending/failed and is safely retryable from the Automation page.

Required operational checks

• API and web health (GET /health, web root 200);
• database migration level (docker compose exec api alembic current);
• pending/failed outbox count (Automation page, or GET /api/v1/workflows?status=failed);
• RAGcore provider state (GET /api/v1/knowledge/status; demo provider is always available, RAGcore adapter reports unavailable when unreachable);
• n8n connectivity (docker compose logs n8n, or submit a return and watch /automation);
• MCP provider endpoint authorization (curl the four /api/v1/integrations/mcp/* routes with and without a valid X-Service-Token — see artifacts/evidence/final-summary.md for sample calls);
• deterministic demo reset (POST /api/v1/demo/reset as Operations Manager, or make seed).

Recovery expectations

• database restart: application reconnects (SQLAlchemy connection pool, pool_pre_ping=True);
• n8n outage: events remain pending and are retried with exponential backoff, then failed after 5 attempts and safely retryable from /automation;
• RAGcore outage: /knowledge shows unavailable, all operational pages continue working;
• MCP Hub outage: the web application is unaffected — MCP endpoints are a separate, independently-authenticated API surface;
• failed demo experiment: Operations Manager reset (POST /api/v1/demo/reset) restores the deterministic seed, including all named S1–S6 demo scenarios.