Files
MobilityOps/docs/final-integrations/ai-operations-brief-runbook.md
T
NuklearRabbitandClaude Sonnet 5 57992bf153 M10: AI Operations Brief runbook, full live e2e regression, final evidence
Ran a real AI Operations Brief via the live ITWorx MCP Hub connector's own
MobilityOpsClient against production Fleet Ops: real operations summary, real
most-pressing vehicle, real grounded knowledge answer with citations, real
correlation IDs verified end-to-end in Fleet Ops's own audit log. No write
actions performed. Runbook and full output in
docs/final-integrations/ai-operations-brief-runbook.md.

Ran the full Playwright e2e suite against the live deployed instance and fixed
two pre-existing fragile locators unrelated to this session's feature work
(both broke because Automation now legitimately has two tables sharing the
same generic selectors, exposed by running the full suite rather than
individual files) plus one pre-existing untranslated-loanword false positive.
All specs pass.

artifacts/final-integrations/final-summary.md has the complete evidence
write-up: repository/deployment state, what was fixed vs. handed off, test
results, and known limitations stated plainly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-05 13:49:23 +02:00

88 lines
5.0 KiB
Markdown

# AI Operations Brief — runbook and live evidence
Answers, using a real MCP client through the real ITWorx MCP Hub connector against live
production Fleet Ops: *"Which vehicles need the most attention today, why, and which
internal procedure should be followed for the most important issue?"*
## What this is not
Not a chatbot. No write actions exist under `/api/v1/integrations/mcp/*` (verified by
`test_no_write_endpoints_exist_under_mcp_namespace`). Every call below is a plain
read-only tool invocation, exactly as the deployed Hub connector performs them.
## Reproducible command
Run from inside the live `itworx-mcp-hub-connector-mobilityops-1` container (has the
real `MOBILITYOPS_ENDPOINT`, `MOBILITYOPS_SERVICE_TOKEN_FILE` and the real
`packages.connector_kit.mobilityops.MobilityOpsClient` already available — the same
client class the deployed connector uses):
```sh
docker exec -e PYTHONPATH=/opt/hub -w /opt/hub \
itworx-mcp-hub-connector-mobilityops-1 python3 - <<'PY'
from packages.connector_kit.mobilityops import FileTokenProvider, MobilityOpsClient, MobilityOpsSettings
import os, json
settings = MobilityOpsSettings(
base_url=os.environ["MOBILITYOPS_ENDPOINT"],
token_reference=os.environ["MOBILITYOPS_SERVICE_TOKEN_FILE"],
)
client = MobilityOpsClient(settings, FileTokenProvider())
client_id = "ai-ops-brief"
summary = client.operations_summary(client_id)
attention = client.attention_vehicles(client_id, minimum_severity="high", date=None, limit=20)
top_ref = attention[0]["vehicle_ref"] if attention else None
details = client.vehicle_details(client_id, top_ref) if top_ref else None
answer = client.search_knowledge(client_id, "What must I do when a vehicle returns with damage?", max_sources=2)
print(json.dumps({"summary": summary, "top_attention": attention[0] if attention else None, "vehicle": details, "answer": answer}, indent=2))
client.close()
PY
```
This calls, in order: `fleet_ops_get_operations_summary``fleet_ops_list_attention_vehicles`
`fleet_ops_get_vehicle_details``fleet_ops_search_knowledge`. No result is invented —
every field printed is exactly what Fleet Ops's live API returned.
## Live run, 2026-08-05
- **Operations summary**: 21 available, 11 rented, 6 cleaning, 5 maintenance,
**7 blocked**; 23 open quality issues; 1 pending/failed workflow.
- **Most pressing vehicle**: `MO-031``missing_required_field`, "near-future booking;
required operational inspection missing", detected `2026-08-05T11:08:33Z`. (16 vehicles
currently carry a `high`-severity open issue; `MO-031` was the earliest-detected.)
- **Vehicle detail (`MO-031`)**: Adria Matrix, 2022, Geel, `operational_status: blocked`,
41,149 km (service due at 50,000 km), no current booking.
- **Grounded procedure (English)**: *Damage handling procedure* v1.3, section "1.
Immediate actions" — "When a vehicle returns with visible or reported damage, mark
damage in the return inspection, add a concise factual description and keep the
vehicle blocked. Do not promise the customer a repair cost or liability decision."
Second source: *Vehicle return procedure* v2.0, section "3. Determine next state".
`evidence_state: grounded`, 2 real citations, `provider: demo`.
- **Dutch and French variants of the damage question** (`Wat moet ik doen wanneer een
voertuig beschadigd terugkomt?`, `Que dois-je faire lorsqu'un véhicule revient
endommagé ?`) both returned `evidence_state: insufficient` — an honest, non-fabricated
"no match" rather than a wrong or invented answer. Root cause: the currently-deployed
Hub connector does not yet send the new `locale` field this session added to
`search-knowledge` (Fleet Ops defaults to `en-GB`), so non-English question text
doesn't match the demo provider's English-tokenized index. A real fix needs a Hub-side
connector update to pass `locale`, tracked as a follow-up, not silently worked around.
- **Correlation ID, end to end, verified**: each call's response `correlation_id` (e.g.
`abd643a7-2aeb-4af3-806a-da04acb3e444` for the English grounded answer) appears
verbatim in Fleet Ops's own audit log (`GET /api/v1/audit?action=mcp_tool_request`),
alongside `actor_label: ai-ops-brief-2026-08-05` and the real tool name
(`fleet_ops_search_knowledge`). No write actions were performed; only `mcp_tool_request`
audit rows were created, matching every other live MCP call this integration makes.
## Known limitation, stated plainly
The demo knowledge provider (not RAGcore — `KNOWLEDGE_PROVIDER=demo`, see
`docs/final-integrations/current-state-audit.md` for why) is what grounds the English
answer here. It is deterministic, extractive, and never fabricates — but it is not the
live RAGcore integration the brief brief for this task set out to exercise; that
remains blocked on RAGcore's own reranker gap. This run is honest about that: it proves
the full MCP-client → Hub → Fleet Ops → knowledge-provider → audit chain works for real,
live, in production, with real data and real citations — using the knowledge provider
that is actually configured live today.