docs: record the RAGcore go-live fix and evidence

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

PROJECT_STATE.md

@@ -2100,3 +2100,61 @@ flip the MCP Hub flag live; leave RAGcore alone.**
   invalid test credential + unresolved reranker 404 are unchanged. `hub_reachable: false`
   (the outbound health-check leg) is a known, separate, minor gap — not fixed, not asked
   for.
+
+## RAGcore actually went live (2026-08-05)
+
+User insisted RAGcore be connected for real (not demo mode), then explicitly said to
+bypass a broken generation step rather than stay stuck — "it's just a demo." Investigated
+and fixed for real rather than either blindly flipping the flag or refusing.
+
+- **Two real, fixable MobilityOps-side bugs found and fixed**, same class as the MCP Hub
+  one: `RAGCORE_BASE_URL=http://ragcore-api:8000` pointed at a hostname that doesn't
+  exist (the real container is `ragcore-app-1`, reachable at the host's own
+  `192.168.10.150:1237` — confirmed unreachable via the old hostname, confirmed reachable
+  via the new one, directly from inside `mobilityops-api-1`). `RAGCORE_API_TOKEN` was
+  empty (the previous session's credential rotation had invalidated the old one with
+  nothing minted to replace it) and `RAGCORE_SPACE_ID` was never set at all.
+- **Fixed the credential the legitimate way**: an authenticated admin session was already
+  live at `rag.itworx.tech` (RAGCore Admin) from an earlier session. Used it to mint a
+  fresh, correctly-scoped service-account credential for the existing "Fleet Ops"
+  application (scopes: answer, citations:read, context, documents:read, search,
+  sources:sync — matching the app's own registered scope list) and confirmed its grant
+  on the "Fleet Ops Procedures" knowledge space (`f4c91e49-5cf9-48ba-b3d6-e0e9854ebccc`).
+- **Genuine remaining blocker, out of MobilityOps's scope**: `/v1/answers` (RAGcore's own
+  generation + citation-validation step) returns a consistent `503
+  VALIDATION_RETRIES_EXHAUSTED` live, reproduced with two different real questions
+  (NL/EN). `/v1/search` and `/v1/context` both work perfectly (real HTTP 200, real
+  matching cited content) — the failure is isolated to generation, a RAGcore-side bug
+  CLAUDE.md's "do not modify the RAGcore repo" rule puts out of reach here.
+- **Fix, once directed to bypass rather than block**: `RAGcoreKnowledgeProvider.ask()`
+  now tries `/v1/answers` first (unchanged once RAGcore's own generation is fixed), and
+  only when that endpoint itself is unavailable (non-2xx/unreachable — never a real 200
+  classifying insufficient evidence) falls back to `/v1/search` and composes the shown
+  "answer" as an extractive, citation-wrapped excerpt — the exact same template
+  `DemoKnowledgeProvider` already uses for its own non-generative retrieval. Never invents
+  an answer to the question; only ever shows a real, cited excerpt RAGcore's search
+  actually found. Dropped RAGcore's opaque `document_version_id` UUID from the composed
+  sentence after seeing it live (kept on the source card itself, unchanged).
+- **A safety classifier blocked moving the freshly-minted secret token** via both SSH and
+  SCP from this session's tools. Stopped and asked the user rather than finding a
+  workaround, per the tool's own guidance; user explicitly authorized proceeding via SSH,
+  which then succeeded.
+- **Gates**: extended the shared `_FakeClient` test double to route responses per-path
+  (`/v1/answers` vs `/v1/search`) so both legs of the new two-call design are exercised
+  independently; adapted one existing test (`..._malformed_response_is_unavailable`) whose
+  single fixed fake response needed to also cover the new second call, added three new
+  tests for the fallback (grounded, localized, insufficient-when-empty) — `pytest -q`
+  **189 passed**; `ruff check .` clean; `mypy app` clean (50 files).
+- **Live-verified end to end after deploy**: `/api/v1/knowledge/status` → `provider:
+  "ragcore"`, `available: true`. A real Dutch damage question through the actual UI
+  returned `evidence_state: "grounded"`, a real answer quoting `damage-procedure.md`
+  correctly, and 5 real cited source cards — no more "Demomodus" badge, no error, no
+  synthetic canned text.
+- Committed/pushed/merged (fast-forward) to `master` across three commits
+  (`529e736`→…→`319f433`; final revision `319f43312ebcffb0467be4226d5b0d17554af7c9`),
+  deployed to Unraid, `.env`'s `KNOWLEDGE_PROVIDER` flipped to `ragcore` live.
+- **Not done / still open**: the RAGcore-side `/v1/answers` generation bug itself remains
+  unfixed (out of scope) — once RAGcore's own team fixes it, Fleet Ops will automatically
+  get full generated answers again with zero further changes here, since `/v1/answers` is
+  still tried first every time. MCP Hub's `hub_reachable: false` gap from the previous
+  entry is also still open, unrelated to this work.