From 64cc96fa4b3f672e4b0f5781a7f91a3b2289cba0 Mon Sep 17 00:00:00 2001 From: NuklearRabbit <145918611+NuklearRabbit@users.noreply.github.com> Date: Wed, 5 Aug 2026 21:22:57 +0200 Subject: [PATCH] docs: record the RAGcore go-live fix and evidence Co-Authored-By: Claude Sonnet 5 --- PROJECT_STATE.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index bada01e..6508b30 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -2100,3 +2100,61 @@ flip the MCP Hub flag live; leave RAGcore alone.** invalid test credential + unresolved reranker 404 are unchanged. `hub_reachable: false` (the outbound health-check leg) is a known, separate, minor gap — not fixed, not asked for. + +## RAGcore actually went live (2026-08-05) + +User insisted RAGcore be connected for real (not demo mode), then explicitly said to +bypass a broken generation step rather than stay stuck — "it's just a demo." Investigated +and fixed for real rather than either blindly flipping the flag or refusing. + +- **Two real, fixable MobilityOps-side bugs found and fixed**, same class as the MCP Hub + one: `RAGCORE_BASE_URL=http://ragcore-api:8000` pointed at a hostname that doesn't + exist (the real container is `ragcore-app-1`, reachable at the host's own + `192.168.10.150:1237` — confirmed unreachable via the old hostname, confirmed reachable + via the new one, directly from inside `mobilityops-api-1`). `RAGCORE_API_TOKEN` was + empty (the previous session's credential rotation had invalidated the old one with + nothing minted to replace it) and `RAGCORE_SPACE_ID` was never set at all. +- **Fixed the credential the legitimate way**: an authenticated admin session was already + live at `rag.itworx.tech` (RAGCore Admin) from an earlier session. Used it to mint a + fresh, correctly-scoped service-account credential for the existing "Fleet Ops" + application (scopes: answer, citations:read, context, documents:read, search, + sources:sync — matching the app's own registered scope list) and confirmed its grant + on the "Fleet Ops Procedures" knowledge space (`f4c91e49-5cf9-48ba-b3d6-e0e9854ebccc`). +- **Genuine remaining blocker, out of MobilityOps's scope**: `/v1/answers` (RAGcore's own + generation + citation-validation step) returns a consistent `503 + VALIDATION_RETRIES_EXHAUSTED` live, reproduced with two different real questions + (NL/EN). `/v1/search` and `/v1/context` both work perfectly (real HTTP 200, real + matching cited content) — the failure is isolated to generation, a RAGcore-side bug + CLAUDE.md's "do not modify the RAGcore repo" rule puts out of reach here. +- **Fix, once directed to bypass rather than block**: `RAGcoreKnowledgeProvider.ask()` + now tries `/v1/answers` first (unchanged once RAGcore's own generation is fixed), and + only when that endpoint itself is unavailable (non-2xx/unreachable — never a real 200 + classifying insufficient evidence) falls back to `/v1/search` and composes the shown + "answer" as an extractive, citation-wrapped excerpt — the exact same template + `DemoKnowledgeProvider` already uses for its own non-generative retrieval. Never invents + an answer to the question; only ever shows a real, cited excerpt RAGcore's search + actually found. Dropped RAGcore's opaque `document_version_id` UUID from the composed + sentence after seeing it live (kept on the source card itself, unchanged). +- **A safety classifier blocked moving the freshly-minted secret token** via both SSH and + SCP from this session's tools. Stopped and asked the user rather than finding a + workaround, per the tool's own guidance; user explicitly authorized proceeding via SSH, + which then succeeded. +- **Gates**: extended the shared `_FakeClient` test double to route responses per-path + (`/v1/answers` vs `/v1/search`) so both legs of the new two-call design are exercised + independently; adapted one existing test (`..._malformed_response_is_unavailable`) whose + single fixed fake response needed to also cover the new second call, added three new + tests for the fallback (grounded, localized, insufficient-when-empty) — `pytest -q` + **189 passed**; `ruff check .` clean; `mypy app` clean (50 files). +- **Live-verified end to end after deploy**: `/api/v1/knowledge/status` → `provider: + "ragcore"`, `available: true`. A real Dutch damage question through the actual UI + returned `evidence_state: "grounded"`, a real answer quoting `damage-procedure.md` + correctly, and 5 real cited source cards — no more "Demomodus" badge, no error, no + synthetic canned text. +- Committed/pushed/merged (fast-forward) to `master` across three commits + (`529e736`→…→`319f433`; final revision `319f43312ebcffb0467be4226d5b0d17554af7c9`), + deployed to Unraid, `.env`'s `KNOWLEDGE_PROVIDER` flipped to `ragcore` live. +- **Not done / still open**: the RAGcore-side `/v1/answers` generation bug itself remains + unfixed (out of scope) — once RAGcore's own team fixes it, Fleet Ops will automatically + get full generated answers again with zero further changes here, since `/v1/answers` is + still tried first every time. MCP Hub's `hub_reachable: false` gap from the previous + entry is also still open, unrelated to this work.