M5: implement RAGcore knowledge integration

KnowledgeProvider protocol with a deterministic TF-IDF-weighted extractive demo provider (never generative, always cites real excerpts) and a RAGcore HTTP adapter that degrades cleanly to unavailable. Knowledge nav + chat-style Q&A UI with source cards and honest grounded/insufficient/unavailable states. 57 backend tests passing, ruff clean. Fixed a real relevance bug (generic terms like "vehicle" crowding out distinctive matches) via IDF weighting, found by testing the actual S6 scenario. Verified end-to-end in the browser: grounded damage question cites both expected procedures; unrelated question honestly returns insufficient evidence with no fabrication.

PROJECT_STATE.md

@@ -2,7 +2,7 @@
 
 ## Current milestone
 
-M4 — complete. Starting M5 next.
+M5 — complete. Starting M6 next.
 
 ## Locked decisions
 
@@ -94,10 +94,24 @@ M4 — complete. Starting M5 next.
   - Full live round trip (not mocked): registered a real return on `BK-DEMO-RETURN` → outbox event queued → background dispatcher delivered it to the now-activated n8n workflow within its 3s poll interval → n8n called back into `/api/v1/integrations/n8n/return-callback` (200 OK, confirmed in `docker compose logs api`) → dispatcher's original POST received n8n's success response → event flipped to `succeeded` on attempt 1, visible on `/automation`.
   - S5 scenario end-to-end in the browser: seeded `BK-H-0020` (`failed`, 3 attempts, "Synthetic connection timeout to n8n") → clicked Retry → `pending` → within ~3s, live dispatcher delivered it through the real n8n instance → `succeeded`, 4 attempts. This is the full documented S5 scenario working for real, not simulated.
 
+### M5 — RAGcore knowledge integration
+- `app/services/knowledge/__init__.py`: `KnowledgeProvider` Protocol (sync, not async — the rest of the backend is sync SQLAlchemy/FastAPI, so an async provider interface would have meant bridging paradigms for no benefit) with `health()`/`ask()`, plus `GroundedAnswer`/`SourceCard`/`KnowledgeHealth` Pydantic models matching `contracts/openapi.yaml`'s `GroundedAnswer` schema exactly. `get_knowledge_provider()` factory switches on `settings.knowledge_provider` ("demo" default, "ragcore" opt-in).
+- `app/services/knowledge/demo.py` — `DemoKnowledgeProvider`: parses the 10 `knowledge/procedures/*.md` files' YAML frontmatter (hand-rolled flat parser, not PyYAML — avoided adding a dependency for a 6-key flat block) and `## `-delimited sections at startup, then does **TF-IDF-weighted keyword retrieval** (not naive keyword counting) with light suffix-stripping stemming (`returns`→`return`, `damaged`→`damage`). This is extractive, not generative: it returns real excerpts and a templated answer sentence, never invented text.
+  - **Real bug found and fixed by testing the actual S6 question, not by inspection**: naive flat keyword-overlap scoring (first cut) let the word "vehicle" — present in nearly every document's title — crowd out the actually-relevant `damage-procedure` document from the top-3 results for "What must I do when a vehicle returns with damage?", because generic words scored the same as distinctive ones. Fixed by computing corpus-wide IDF per token (`log((N+1)/(df+1)) + 1`) and weighting matches by it, so common terms contribute little and rare/distinctive terms (like "damage") dominate the ranking. Verified: the S6 question now returns `damage-procedure` and `vehicle-return-procedure` in the top 3, matching the documented expectation exactly.
+- `app/services/knowledge/ragcore.py` — `RAGcoreKnowledgeProvider`: real `httpx` adapter guessing a plausible REST contract (`GET /health`, `POST /api/v1/ask`) per `contracts/ragcore-contract-assumptions.md` (RAGcore is built separately; no live instance was reachable this session to verify against). Any connection error, timeout, or malformed response degrades to `evidence_state: "unavailable"` rather than raising — this is the adapter that actually exercises the architecture's "RAGcore failure disables knowledge answers only" reliability boundary. Not wired as the active provider by default; `KNOWLEDGE_PROVIDER=ragcore` would need a real, verified base URL to turn on.
+- `POST /api/v1/knowledge/questions` + `GET /api/v1/knowledge/status` (`app/api/routers/knowledge.py`). Audit event `knowledge_question_asked` logs `evidence_state`, `provider`, `source_ids`, and `question_length` only — **not** the question text itself, per `docs/12-security-and-audit.md` ("log question metadata and source IDs, not unnecessary full prompts").
+- Knowledge nav + page (`pages/Knowledge.tsx`): chat-style question box, source cards (title/version/section/excerpt) prioritized over the answer text per `docs/06-ui-ux.md`, explicit `grounded`/`insufficient`/`unavailable` states with distinct visual treatment — never a fabricated-looking answer for the latter two.
+- Dockerfile now also `COPY knowledge ./knowledge`; added `KNOWLEDGE_DIR` setting (`/app/knowledge/procedures` in-container, same pattern as `SEED_DIR`) rather than deriving the path from `__file__` — simpler and doesn't break if the module moves.
+- Commands run and verified from this checkout:
+  - `docker compose run --rm api pytest -q` — **57 passed** (new `tests/test_knowledge.py`: S6 grounded-with-expected-sources, unrelated question is honestly insufficient with no fabrication, demo provider health/document count, endpoint auth required, audit doesn't leak question text, RAGcore adapter degrades to unavailable on a simulated connection error).
+  - `docker compose run --rm api ruff check .` — All checks passed.
+  - `npm run build` — clean.
+  - Full browser run of S6 end-to-end: asked "What must I do when a vehicle returns with damage?" on `/knowledge` → grounded answer citing "Vehicle return procedure" (2 sections) and "Damage handling procedure" with real excerpts. Also asked an unrelated question ("What is the weather forecast for tomorrow?") → correctly returned "Insufficient evidence" / "No matching procedure was found" with zero sources, confirming no fabrication.
+
 ## Known blockers
 
-None. External service credentials may be absent; use the documented demo/degraded providers. The n8n workflow-activation steps above are a one-time manual setup requirement in this environment, not a blocker — but not yet scripted; M7 should either automate it (e.g. a bootstrap script CI/compose can run) or document it clearly enough for `docs/14-testing-and-acceptance.md`'s clean-checkout criteria.
+None. External service credentials may be absent; use the documented demo/degraded providers. The n8n workflow-activation steps from M4 are a one-time manual setup requirement in this environment, not a blocker — but not yet scripted; M7 should either automate it (e.g. a bootstrap script CI/compose can run) or document it clearly enough for `docs/14-testing-and-acceptance.md`'s clean-checkout criteria. RAGcore itself was never reachable this session — `RAGcoreKnowledgeProvider` is implemented and unit-tested for its unavailable-degradation path but its actual request/response contract against a real RAGcore instance is unverified; the demo provider is what M5's acceptance criteria are actually satisfied by.
 
 ## Exact next action
 
-Start M5 (RAGcore knowledge integration): read `docs/09-ragcore-integration.md`, `knowledge/manifest.json`. Implement the `KnowledgeProvider` protocol (`health`, `ask`, `sync_manifest`), a `DemoKnowledgeProvider` doing deterministic keyword/BM25-style retrieval directly over `knowledge/procedures/*.md` (extractive, not generative — must return real excerpts + `insufficient`/`unavailable` states honestly, never fabricate), and a `RAGcoreKnowledgeProvider` adapter stub for the real service (`RAGCORE_BASE_URL` etc. are already in config/compose from M0, but no actual RAGcore instance is confirmed reachable this session — build the demo provider as the one that actually has to work for the acceptance criteria, and make the RAGcore adapter degrade to `unavailable` cleanly if unreachable, per `docs/03-architecture.md`'s reliability boundary "RAGcore failure disables knowledge answers only"). Add `POST /api/v1/knowledge/questions` and `GET /api/v1/knowledge/status`, a Knowledge nav item + page (chat-style question box, source cards with title/version/section/excerpt, explicit unavailable/insufficient states — no fabricated answers ever). S6 demo scenario: "What must I do when a vehicle returns with damage?" must cite `knowledge/procedures/03-damage-handling.md` and `02-vehicle-return.md`.
+Start M6 (ITWorx MCP Hub publication): read `docs/10-mcp-hub-integration.md`, `contracts/mcp-tools.json`. Implement the four read-only, service-token-protected provider endpoints under `/api/v1/integrations/mcp/` (`operations-summary`, `attention-vehicles`, `vehicles/{public_ref}`, and a knowledge-search façade wrapping the M5 `KnowledgeProvider` — per `contracts/mcp-tools.json`'s `mobilityops_search_knowledge` tool and `docs/10-mcp-hub-integration.md`'s note to route through a narrow façade rather than duplicate retrieval logic). Auth: reuse the same shared-secret-header pattern already built for the n8n callback in M4 (`X-Service-Token`, a new `MCP_HUB_SERVICE_TOKEN` setting — `.env.example` already has the env var name reserved) rather than inventing a second auth mechanism. Must not expose generic SQL, arbitrary fetch, write/mutation actions, or secrets — these are read-only summaries only. Add MobilityOps-side service-request audit events (the Hub owns its own central tool-call audit; MobilityOps only needs to record that its provider APIs were reached, with tool name/correlation ID/client identity/result status). No actual ITWorx MCP Hub instance is confirmed reachable in this environment (same situation as RAGcore in M5) — validate the four provider endpoints directly via authenticated `curl`/tests rather than a live Hub round trip, and note that gap explicitly rather than claiming an unverified integration works.