diff --git a/artifacts/final-integrations/final-summary.md b/artifacts/final-integrations/final-summary.md new file mode 100644 index 0000000..9ebdc6d --- /dev/null +++ b/artifacts/final-integrations/final-summary.md @@ -0,0 +1,161 @@ +# Fleet Ops final integrations — evidence summary + +Session date: 2026-08-05. Branch `feat/fleet-ops-final-integrations`. + +## Repository state + +| Repo | Start | End | Branch | Notes | +|---|---|---|---|---| +| Fleet Ops (MobilityOps) | `3ebca9e` (from `feat/live-n8n-ragcore-integration`) | `727c19a` (+ e2e test fixes, uncommitted at write time) | `feat/fleet-ops-final-integrations`, pushed to `origin` | 3 commits: `34df66d`, `2ae2044`, `727c19a` | +| RAGcore | `64a908a` | `64a908a` (+1 isolated commit `ce0ad56`) | `main` | Only a backlog handoff entry committed; no code changes (36-file concurrent-session collision — see below) | +| ITWorx MCP Hub | not modified this session | — | `feature/wp240-final-acceptance` | Connector already live in production before this session started; not touched | + +## Deployed revisions + +- Fleet Ops: `http://192.168.10.150:1236`, redeployed twice this session (after Batches + 1-3 and after Batch 4), `docker compose -p mobilityops -f compose.yaml -f + compose.unraid.yaml up --build -d db api web`, `.deploy/source-revision` = `727c19a...`. +- RAGcore: `http://192.168.10.150:1237`, `ragcore-app-1`. No image redeploy — the two live + fixes (filesystem permissions, reranker model pull) were applied directly to the + running container/Ollama instance, not via a code deploy. +- ITWorx MCP Hub: `http://192.168.10.150:1100` (Tower), unchanged, already live before + this session at commit `c4a0f6d` per the Hub's own state. + +## GUI polish (Batch 1) + +- Dashboard Attention Queue: curated severity mix (grouped "Handle now / Follow up + today / Review later"), replacing pure severity-sort that let `high` crowd out + everything else. +- Today's Movements: seed data curated (`seed/bookings.csv`) so a fresh reset shows ≥2 + departures and ≥2 returns; new `test_seed_today_movements_are_a_credible_mix` test. + Live-verified after a real demo reset: 2 returns + 2 departures shown. +- About Demo: restructured into a compact grid with `
` progressive disclosure + for architecture/security/testing sections. +- Duplicate Customer Merge: match/conflict counts shown, matching fields hidden by + default (toggle to reveal), compact preview of the merged record before confirmation. +- Repo hygiene: removed a stray empty `backend;C` dir and an untracked 31MB zip export; + `.gitignore` now excludes future archive exports. +- All four live-verified via browser against the deployed instance (see screenshots + taken during the session — not separately saved to disk). + +## n8n (Batch 2) + +- 4 canonical workflows confirmed live: Vehicle Return Orchestration, Scheduled Data + Quality Scan, RAGcore Procedure Sync, Workflow Error Handler. +- Fixed genuinely invalid JSON in the committed `fleet-ops-vehicle-return.json` (a + missing `},` between two node objects — the file could not be parsed). +- Workflow 3 (RAGcore Procedure Sync): confirmed 6 real nodes built and saved. Found and + fixed two real defects via the safe `n8n import:workflow` CLI path (not the REST API, + which caused a documented wipe incident in an earlier session): three body-parameter + expressions had a stray trailing `}}`, and `settings.errorWorkflow` was unset. Exported + the corrected definition to `n8n/workflows/fleet-ops-ragcore-procedure-sync.json`, + added to `MANIFEST.md` and `check_drift.py`. +- **Not published** — the Schedule Trigger runs daily at midnight; activating it starts + real unattended production runs, deliberately left as a separate go-live decision. +- No no-op/sync/error-handler live-execution smoke test was run this session beyond the + structural CLI-export verification above (workflow remains unpublished). + +## RAGcore (Batch 3) + +- **Root cause found and fixed, live, user-approved**: the "zero retrieval candidates" + bug was a filesystem permission bug (`/workspace/.state/models/embedding_profiles.json` + was `root:root` mode `600` on the host bind mount, unreadable by the app's actual + runtime uid 10001) — not authorization, not Qdrant, not embeddings, all independently + verified healthy first. Fixed via `chown`/`chmod`; re-verified in-process (5 real hits, + up from 0). +- **Second, deeper gap found, not fixed**: the reranker adapter calls + `{ollama}/api/rerank`, a route this Ollama version (`0.32.5`) does not serve (404). + Pulled a working model (`xitao/bge-reranker-v2-m3:latest`, 1.2GB, approved) — did not + fix it, since the problem is the HTTP route, not the model. `/v1/answers` still returns + `not_answerable`/0 citations for real questions against real matching content. +- User decision: leave `KNOWLEDGE_PROVIDER=demo`; hand the reranker fix off to RAGcore's + own backlog (`docs/ai/BACKLOG.yaml`, task `M8-01`, committed in that repo as `ce0ad56` + — the only commit made in RAGcore this session) rather than editing RAGcore code amid + its own 36-file concurrent-session collision. +- Side effect: minting the live-verification credential rotated the existing "Fleet Ops + Knowledge Assistant (production)" service account's credential (2-active-credential cap + reached). A fresh credential must be issued before actually flipping the provider live. + +## MCP Hub (Batch 4) + +- Confirmed the Fleet Ops connector is already live in production on the Hub side + (Tower, commit `c4a0f6d`), with a real contract fix already applied there + (`vehicle.get`'s wire parameter normalized to camelCase `vehicleRef`). +- Fixed two concrete gaps in Fleet Ops's own `search-knowledge` endpoint: no `locale` + field existed at all (now `nl-BE`/`en-GB`/`fr-BE`, wired to the knowledge provider's + existing `language` param), and the correlation ID was always freshly minted, ignoring + any inbound `X-Correlation-Id` header. Added `get_correlation_id`, applied to all four + MCP endpoints. +- `MCP_HUB_BASE_URL` was dead config (declared, never read); wired it for a real, + bounded Hub-reachability health check instead of an unneeded self-registration push + (the Hub's own registration is catalog-driven). +- Renamed Fleet Ops's own internal audit tool labels `mobilityops_*` → `fleet_ops_*` + (mirrored in `contracts/mcp-tools.json`, `mobilityops_*` kept as deprecated aliases). + The live Hub connector's own dotted tool namespace (`mobilityops.operations.summary` + etc.) is a separate, Hub-owned naming layer, deliberately not touched. +- Automation page's MCP card now shows real evidence (last tool/client/count/timestamp) + instead of only the registration-enabled boolean. + +## AI Operations Brief (Batch 5) + +Real MCP-client-shaped run via the live ITWorx MCP Hub connector's own +`MobilityOpsClient` class against production Fleet Ops. Full runbook and live output in +`docs/final-integrations/ai-operations-brief-runbook.md`. Summary: + +- Real operations summary (21 available / 11 rented / 6 cleaning / 5 maintenance / + 7 blocked; 23 open quality issues). +- Real most-pressing vehicle identified (`MO-031`, missing operational inspection). +- Real vehicle detail lookup. +- Real grounded knowledge answer (English damage-handling question): 2 real citations, + `evidence_state: grounded`. +- Dutch/French variants of the same question honestly returned `insufficient` (no + fabrication) — root cause: the live Hub connector doesn't yet send the new `locale` + field, a Hub-side follow-up, not silently worked around. +- Correlation IDs verified end-to-end in Fleet Ops's own audit log + (`GET /api/v1/audit?action=mcp_tool_request`), matching the response payloads exactly. +- No write actions performed at any point. + +## Testing per batch + +- Backend: **176 passed**, `ruff check .` clean, `mypy app` clean (50 source files) — + verified against a freshly rebuilt image after discovering mid-session that + `docker compose run --rm api` (no bind mount on the `api` service) silently tests a + stale image otherwise. One genuinely stale test assertion found and fixed as a result. +- Frontend: `tsc -b && vite build` clean. +- E2e (Playwright, against the live deployed instance, + `MOBILITYOPS_PUBLIC_URL=http://192.168.10.150:1236`): every spec file run this + session passed — `demo.spec.ts`, `interactive-elements.spec.ts` (26), + `responsive-i18n.spec.ts` + `demo-accessibility.spec.ts` + `guided-demo-full.spec.ts` + (28), `i18n-coverage.spec.ts` + `error-messages.spec.ts` + `clickable-rows.spec.ts` + + `demo-guide.spec.ts` + `demo-entry.spec.ts` + `demo-legibility.spec.ts` + + `fleet-ops-correction.spec.ts` + `ui-redesign.spec.ts` + `greeting.spec.ts` + + `greeting-live.spec.ts` (28, after fixing 2 pre-existing fragile locators unrelated to + this session's feature work — a `.data-table` ambiguity now that Automation has two + tables, and a `Technische details` toggle ambiguity for the same reason; plus one + pre-existing untranslated-loanword false positive in `i18n-coverage.spec.ts`). + +## Known limitations, stated plainly + +- `KNOWLEDGE_PROVIDER` is still `demo`, not `ragcore` — blocked on RAGcore's own + reranker gap (handed off, not fixed this session). +- n8n workflow 3 is built and correct but not published (deliberate, separate decision). +- The live MCP Hub connector doesn't yet send the new `locale` field, so + locale-aware knowledge search only works when called directly against Fleet Ops (as + proven by the backend tests), not yet through the live Hub connector as deployed. +- No public-demo-readiness checklist, About Demo Guide "completed" end-state polish + (section 4E), or dashboard MCP "activity showcase after Demo Complete" gating were + built this session — the MCP evidence display exists on the Automation page + unconditionally rather than gated behind guided-demo completion. +- No security-review pass was run separately this session (existing gates: ruff, mypy, + the repo's own auth/audit test coverage). + +## Rollback + +- Fleet Ops: prior working revision `0571a40` remains in `.deploy/` as + `source-0571a40.tar.gz` on the Unraid host; redeploy by re-extracting and re-running + the same `docker compose up --build -d` sequence with that archive. +- RAGcore: `chown`/`chmod` change is trivially reversible (`chown 0:0` + + `chmod 600` on the same path) if needed, though there is no reason to revert a + permission fix. Ollama model pull (`xitao/bge-reranker-v2-m3:latest`) can be removed + with `ollama rm` if unwanted; it is inert until RAGcore's own code is changed to use it. +- MCP Hub: not modified this session. diff --git a/docs/final-integrations/ai-operations-brief-runbook.md b/docs/final-integrations/ai-operations-brief-runbook.md new file mode 100644 index 0000000..00832f2 --- /dev/null +++ b/docs/final-integrations/ai-operations-brief-runbook.md @@ -0,0 +1,87 @@ +# AI Operations Brief — runbook and live evidence + +Answers, using a real MCP client through the real ITWorx MCP Hub connector against live +production Fleet Ops: *"Which vehicles need the most attention today, why, and which +internal procedure should be followed for the most important issue?"* + +## What this is not + +Not a chatbot. No write actions exist under `/api/v1/integrations/mcp/*` (verified by +`test_no_write_endpoints_exist_under_mcp_namespace`). Every call below is a plain +read-only tool invocation, exactly as the deployed Hub connector performs them. + +## Reproducible command + +Run from inside the live `itworx-mcp-hub-connector-mobilityops-1` container (has the +real `MOBILITYOPS_ENDPOINT`, `MOBILITYOPS_SERVICE_TOKEN_FILE` and the real +`packages.connector_kit.mobilityops.MobilityOpsClient` already available — the same +client class the deployed connector uses): + +```sh +docker exec -e PYTHONPATH=/opt/hub -w /opt/hub \ + itworx-mcp-hub-connector-mobilityops-1 python3 - <<'PY' +from packages.connector_kit.mobilityops import FileTokenProvider, MobilityOpsClient, MobilityOpsSettings +import os, json + +settings = MobilityOpsSettings( + base_url=os.environ["MOBILITYOPS_ENDPOINT"], + token_reference=os.environ["MOBILITYOPS_SERVICE_TOKEN_FILE"], +) +client = MobilityOpsClient(settings, FileTokenProvider()) +client_id = "ai-ops-brief" + +summary = client.operations_summary(client_id) +attention = client.attention_vehicles(client_id, minimum_severity="high", date=None, limit=20) +top_ref = attention[0]["vehicle_ref"] if attention else None +details = client.vehicle_details(client_id, top_ref) if top_ref else None +answer = client.search_knowledge(client_id, "What must I do when a vehicle returns with damage?", max_sources=2) + +print(json.dumps({"summary": summary, "top_attention": attention[0] if attention else None, "vehicle": details, "answer": answer}, indent=2)) +client.close() +PY +``` + +This calls, in order: `fleet_ops_get_operations_summary` → `fleet_ops_list_attention_vehicles` +→ `fleet_ops_get_vehicle_details` → `fleet_ops_search_knowledge`. No result is invented — +every field printed is exactly what Fleet Ops's live API returned. + +## Live run, 2026-08-05 + +- **Operations summary**: 21 available, 11 rented, 6 cleaning, 5 maintenance, + **7 blocked**; 23 open quality issues; 1 pending/failed workflow. +- **Most pressing vehicle**: `MO-031` — `missing_required_field`, "near-future booking; + required operational inspection missing", detected `2026-08-05T11:08:33Z`. (16 vehicles + currently carry a `high`-severity open issue; `MO-031` was the earliest-detected.) +- **Vehicle detail (`MO-031`)**: Adria Matrix, 2022, Geel, `operational_status: blocked`, + 41,149 km (service due at 50,000 km), no current booking. +- **Grounded procedure (English)**: *Damage handling procedure* v1.3, section "1. + Immediate actions" — "When a vehicle returns with visible or reported damage, mark + damage in the return inspection, add a concise factual description and keep the + vehicle blocked. Do not promise the customer a repair cost or liability decision." + Second source: *Vehicle return procedure* v2.0, section "3. Determine next state". + `evidence_state: grounded`, 2 real citations, `provider: demo`. +- **Dutch and French variants of the damage question** (`Wat moet ik doen wanneer een + voertuig beschadigd terugkomt?`, `Que dois-je faire lorsqu'un véhicule revient + endommagé ?`) both returned `evidence_state: insufficient` — an honest, non-fabricated + "no match" rather than a wrong or invented answer. Root cause: the currently-deployed + Hub connector does not yet send the new `locale` field this session added to + `search-knowledge` (Fleet Ops defaults to `en-GB`), so non-English question text + doesn't match the demo provider's English-tokenized index. A real fix needs a Hub-side + connector update to pass `locale`, tracked as a follow-up, not silently worked around. +- **Correlation ID, end to end, verified**: each call's response `correlation_id` (e.g. + `abd643a7-2aeb-4af3-806a-da04acb3e444` for the English grounded answer) appears + verbatim in Fleet Ops's own audit log (`GET /api/v1/audit?action=mcp_tool_request`), + alongside `actor_label: ai-ops-brief-2026-08-05` and the real tool name + (`fleet_ops_search_knowledge`). No write actions were performed; only `mcp_tool_request` + audit rows were created, matching every other live MCP call this integration makes. + +## Known limitation, stated plainly + +The demo knowledge provider (not RAGcore — `KNOWLEDGE_PROVIDER=demo`, see +`docs/final-integrations/current-state-audit.md` for why) is what grounds the English +answer here. It is deterministic, extractive, and never fabricates — but it is not the +live RAGcore integration the brief brief for this task set out to exercise; that +remains blocked on RAGcore's own reranker gap. This run is honest about that: it proves +the full MCP-client → Hub → Fleet Ops → knowledge-provider → audit chain works for real, +live, in production, with real data and real citations — using the knowledge provider +that is actually configured live today. diff --git a/frontend/e2e/fleet-ops-correction.spec.ts b/frontend/e2e/fleet-ops-correction.spec.ts index cbdb37f..a43a5fe 100644 --- a/frontend/e2e/fleet-ops-correction.spec.ts +++ b/frontend/e2e/fleet-ops-correction.spec.ts @@ -302,7 +302,11 @@ test("automation shows a localized error explanation with the raw error only und await expect(page.getByText(/tijdelijk niet bereikbaar/).first()).toBeVisible(); await expect(page.getByText("Synthetic connection timeout to n8n")).not.toBeVisible(); - await page.getByText("Technische details").first().click(); + // Scope to the failed job's own row -- the workflow-evidence table above it also has + // "Technische details" toggles (one per workflow), so an unscoped .first() can open + // the wrong one. + const failedJobRow = page.locator("tr", { has: page.getByText(/tijdelijk niet bereikbaar/) }); + await failedJobRow.getByText("Technische details").click(); await expect(page.getByText("Synthetic connection timeout to n8n")).toBeVisible(); }); diff --git a/frontend/e2e/i18n-coverage.spec.ts b/frontend/e2e/i18n-coverage.spec.ts index bd7c3f9..2cb0c64 100644 --- a/frontend/e2e/i18n-coverage.spec.ts +++ b/frontend/e2e/i18n-coverage.spec.ts @@ -144,6 +144,7 @@ const IDENTICAL_VALUE_ALLOWLIST = new Set([ "knowledge.retrievalFlow.question", "knowledge.questionLabelExchange", "returns.result.inspection", // "Inspection" -- identical spelling in French + "integrations.workflows.columns.name", // "Workflow" -- used as-is in Dutch and French ]); function isTranslatableProse(value: unknown): value is string { diff --git a/frontend/e2e/interactive-elements.spec.ts b/frontend/e2e/interactive-elements.spec.ts index c771b80..cbafb7d 100644 --- a/frontend/e2e/interactive-elements.spec.ts +++ b/frontend/e2e/interactive-elements.spec.ts @@ -243,11 +243,12 @@ test("data quality: manual scan runs and shows a result summary", async ({ page, test("automation page: status filter and retry button work", async ({ page, request }) => { await resetDemoData(request); await page.goto("/automation"); - await expect(page.locator(".data-table")).toBeVisible(); + const jobsTable = page.getByRole("table", { name: "Automation jobs" }); + await expect(jobsTable).toBeVisible(); await page.getByLabel("Status").selectOption("failed"); - await expect(page.locator(".data-table tbody tr").first()).toBeVisible(); - const failedRowCountBefore = await page.locator(".data-table tbody tr").count(); + await expect(jobsTable.locator("tbody tr").first()).toBeVisible(); + const failedRowCountBefore = await jobsTable.locator("tbody tr").count(); const retryButton = page.getByRole("button", { name: "Retry" }).first(); await expect(retryButton).toBeVisible(); await retryButton.click(); @@ -256,7 +257,7 @@ test("automation page: status filter and retry button work", async ({ page, requ // filtered to "failed" — the row correctly disappears from this view rather than // showing "pending" in place. Confirm the filtered list shrank by one. await expect(async () => { - const count = await page.locator(".data-table tbody tr").count(); + const count = await jobsTable.locator("tbody tr").count(); expect(count).toBe(failedRowCountBefore - 1); }).toPass({ timeout: 5000 }); });