MobilityOps/docs/17-runbook.md

# PoC runbook

For the demo-specific 5-minute/10-minute walkthroughs, reset behaviour, Unraid
redeploy/rollback steps and troubleshooting, see `docs/demo-release/demo-runbook.md`.
This document covers general environment bootstrap and n8n setup.

## Bootstrap (clean checkout)

```bash
cp .env.example .env
make demo
```

`make demo` runs `docker compose up --build -d` (migrations run automatically on API
container startup, see `backend/entrypoint.sh`) and then seeds the deterministic dataset.
Equivalently, without `make`:

```bash
cp .env.example .env
docker compose up --build -d
docker compose exec api python -m app.cli seed --reset
```

Verify:

```bash
curl http://localhost:8128/health          # {"status":"ok",...}
curl -o /dev/null -w "%{http_code}\n" http://localhost:1228/   # 200
docker compose run --rm api pytest -q      # all tests pass
docker compose run --rm api ruff check .   # clean
```

## n8n automation (one-time per environment)

The n8n image used here (n8nio/n8n:latest, 2.x) requires an owner account before any
workflow — including webhook registration — works reliably; `N8N_BASIC_AUTH_ACTIVE` no
longer gates this. This is a one-time step per fresh `docker compose down -v`:

1. Open `http://localhost:5678/setup` and create an owner account (any email/password
   meeting the 8+ characters / 1 number / 1 capital rule — no email verification is
   required). Skip the optional survey/license-key dialogs that follow.
2. Import and activate the return-processing workflow:

   ```bash
   make n8n-setup
   ```

   which runs:

   ```bash
   docker compose exec n8n n8n import:workflow --input=//imports/mobilityops-return-processing.json
   docker compose exec n8n n8n publish:workflow --id=mobilityops-return-processing
   docker compose restart n8n
   ```

   (`n8n import:workflow` always leaves the workflow deactivated regardless of its
   `"active"` field; `publish:workflow` + a restart is what actually activates it.)

Verify the full round trip:

```bash
# after logging in and registering any return via the UI or API
curl -b cookies.txt http://localhost:8128/api/v1/workflows | grep succeeded
```

A failed/offline n8n does not roll back the return — the outbox event simply stays
`pending`/`failed` and is safely retryable from the Automation page.

### Second workflow: scheduled quality scan

Import and publish the same way:

```bash
make n8n-setup-scan
```

which runs:

```bash
docker compose exec n8n n8n import:workflow --input=//imports/mobilityops-scheduled-quality-scan.json
docker compose exec n8n n8n publish:workflow --id=mobilityops-scheduled-quality-scan
docker compose restart n8n
```

Verify:

```bash
curl -X POST http://localhost:8128/api/v1/integrations/n8n/scheduled-scan \
  -H "X-Service-Token: <MOBILITYOPS_CALLBACK_TOKEN from .env>"
# {"created": {...}}
```

Trigger a live run from n8n's own UI ("Manual test trigger" node → Execute Workflow) to
confirm the round trip without waiting for the hourly schedule. It does not depend on
RAGcore or MCP Hub and ships `"active": false`, so it never fires anywhere until
deliberately published with a real service token.

### Existing shared n8n on the Unraid review server

The Unraid deployment uses the existing n8n at `http://192.168.10.150:5678`; it does not
start MobilityOps's bundled n8n service. `compose.unraid.yaml` places that fallback behind
the opt-in `bundled-n8n` profile. Configure the API target and publish the workflow with:

```bash
sed -i \
  's|^N8N_WEBHOOK_URL=.*|N8N_WEBHOOK_URL=http://192.168.10.150:5678/webhook/mobilityops-return|' \
  .env
./deploy/unraid/setup-existing-n8n.sh \
  n8n \
  http://192.168.10.150:1236/api/v1/integrations/n8n/return-callback
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up -d db api web
```

The setup script reads the callback token from the mode-0600 deployment `.env`, builds and
removes a temporary server-side import without writing the token to Git, and restarts the
existing n8n so the production webhook is registered. The imported configuration remains
inside n8n's protected application data. The callback travels through the MobilityOps web
proxy, so the shared n8n container does not need direct database access or membership of
the MobilityOps Docker network.

Publish the scheduled quality-scan workflow the same way:

```bash
./deploy/unraid/setup-scheduled-scan.sh \
  n8n \
  http://192.168.10.150:1236/api/v1/integrations/n8n/scheduled-scan
```

## Required operational checks

- API and web health (`GET /health`, web root `200`);
- database migration level (`docker compose exec api alembic current`);
- pending/failed outbox count (Automation page, or `GET /api/v1/workflows?status=failed`);
- RAGcore provider state (`GET /api/v1/knowledge/status`; demo provider is always
  `available`, RAGcore adapter reports `unavailable` when unreachable);
- n8n connectivity (`docker compose logs n8n`, or submit a return and watch `/automation`);
- MCP provider endpoint authorization (`curl` the four `/api/v1/integrations/mcp/*`
  routes with and without a valid `X-Service-Token` — see
  `artifacts/evidence/final-summary.md` for sample calls);
- deterministic demo reset (`POST /api/v1/demo/reset` as Operations Manager, or `make seed`).

## Recovery expectations

- database restart: application reconnects (SQLAlchemy connection pool, `pool_pre_ping=True`);
- n8n outage: events remain `pending` and are retried with exponential backoff, then
  `failed` after 5 attempts and safely retryable from `/automation`;
- RAGcore outage: `/knowledge` shows `unavailable`, all operational pages continue working;
- MCP Hub outage: the web application is unaffected — MCP endpoints are a separate,
  independently-authenticated API surface;
- failed demo experiment: Operations Manager reset (`POST /api/v1/demo/reset`) restores
  the deterministic seed, including all named S1–S6 demo scenarios.