From e5d8466266d7af7f03d7fea9e05d369d55662afa Mon Sep 17 00:00:00 2001 From: NuklearRabbit <145918611+NuklearRabbit@users.noreply.github.com> Date: Wed, 5 Aug 2026 03:20:55 +0200 Subject: [PATCH] knowledge: rewrite RAGcoreKnowledgeProvider to the real search/answers contract The previous adapter targeted an endpoint shape RAGcore never actually exposed. health() now checks /health/ready and ask() posts to the real POST /v1/answers with Bearer auth and requested_space_ids, matching RAGcore's actual contract after this session's Bearer-auth and search/answer wiring work. Adds RAGCORE_SPACE_ID config/env plumbing (a question is meaningless without a knowledge space to scope it to) and 12 new adapter tests covering degradation paths: missing space id, connection errors, non-200 responses, malformed responses, not-answerable, and answerable-without-citations all fail closed to "insufficient evidence" rather than fabricating an answer. KNOWLEDGE_PROVIDER stays "demo" in production for now -- switching requires RAGcore's own search/answer application to actually be deployed and live-verified, tracked separately in PROJECT_STATE.md. Co-Authored-By: Claude Sonnet 5 --- .env.example | 2 + PROJECT_STATE.md | 269 +++++++++++++++++++++- backend/app/core/config.py | 1 + backend/app/services/knowledge/ragcore.py | 106 +++++---- backend/tests/test_knowledge.py | 186 ++++++++++++++- compose.yaml | 1 + 6 files changed, 512 insertions(+), 53 deletions(-) diff --git a/.env.example b/.env.example index f643e93..d941cc7 100644 --- a/.env.example +++ b/.env.example @@ -42,6 +42,8 @@ RAGCORE_TENANT=northstar-mobility-demo RAGCORE_WORKSPACE=mobilityops RAGCORE_COLLECTION=internal-procedures RAGCORE_API_TOKEN= +# UUID of the RAGcore knowledge space procedures were synced into (see workflow 3). +RAGCORE_SPACE_ID= # ITWorx MCP Hub integration MCP_HUB_REGISTRATION_ENABLED=false diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index bb9aecc..933f3d8 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -1286,10 +1286,265 @@ Handler — both net-new, not yet built. The plaintext token was never printed/logged — copied via RAGcore's own "Copy" button and pasted directly into a new n8n Header Auth credential named **"RAGcore Sync Token"** (header `Authorization: Bearer `), ready for workflow 3. -- **Exact next action**: task #86 (build workflow 3, RAGcore Procedure Sync) and the - `RAGcoreKnowledgeProvider` adapter rewrite (to the real inspected contract — `/health/live`, - `/health/ready`, `POST /v1/uploads`, `POST /v1/search`/`/v1/context`/`/v1/answers`) are now - unblocked — the "RAGcore Sync Token" n8n credential exists and works. WF4's own - timeout/retry gap is still open pending n8n browser re-authentication (minor, non-blocking, - see above). Both #90 and #91 should be revisited once workflow 3 is actually built, since - they currently document it as blocked. +- **Exact next action (superseded by the entry below)**: task #86 (build workflow 3, + RAGcore Procedure Sync) and the `RAGcoreKnowledgeProvider` adapter rewrite (to the real + inspected contract — `/health/live`, `/health/ready`, `POST /v1/uploads`, `POST + /v1/search`/`/v1/context`/`/v1/answers`) are now unblocked — the "RAGcore Sync Token" n8n + credential exists and works. WF4's own timeout/retry gap is still open pending n8n browser + re-authentication (minor, non-blocking, see above). Both #90 and #91 should be revisited + once workflow 3 is actually built, since they currently document it as blocked. + +- **Second RAGcore bug found, fixed and deployed (with explicit owner approval, same pattern + as the transaction-race fix above)**: with the "RAGcore Sync Token" credential in hand, + the n8n "Upload to RAGcore" node still returned a persistent 401 on every item. Traced via + direct RAGcore source inspection (`C:\Projects\RAGcore`) to a genuine second, independent + gap: no code path in RAGcore converted an incoming `Authorization: Bearer ` header + into a `request.state.principal` for any `/v1/*` route — only browser session cookies were + ever accepted, even though the credential-verification logic + (`ServiceAccountCredentialService.verify()`) existed and was unit-tested. This blocks any + machine caller (n8n, and eventually Fleet Ops's own `RAGcoreKnowledgeProvider` adapter) + from ever authenticating to `/v1/uploads`. Fixed additively, scoped to `/v1/uploads` only + per owner instruction (search/context/answers left for later): new + `CredentialRepository.get_by_id()` (Postgres + in-memory), new + `ServiceAccountCredentialService.authenticate()` (parallel to the existing `verify()`, not + a refactor of it), and a new `get_upload_principal` FastAPI dependency + (`src/ragcore/api/v1/uploads/dependencies.py`) that falls back to the Bearer header when + there is no session principal, wired into `uploads/routes.py` in place of the session-only + `get_principal`. New/updated tests in `tests/security/identity/test_credentials.py` and + `tests/security/uploads/test_upload_security.py` (bearer-token accept/reject paths, the + existing route test's stale `get_principal` override fixed to `get_upload_principal`). + Verified: RAGcore's own test suite — 377 passed in the affected `tests/security`, + `tests/unit`, `tests/api` trees (3 unrelated pre-existing failures: two need Windows + symlink privileges the sandbox doesn't have, one is a git-connector fixture mismatch; a + separate architecture-boundary failure in `application/ingestion/handler.py` belongs to + unrelated in-progress work by a different concurrent agent on the same RAGcore checkout, + confirmed via `git status`/`git log` — not touched by this fix). Ruff and mypy clean on + every changed file. Deployed to the live RAGcore instance (`ragcore-app-1` on Unraid, port + 1237 internally, fronted by `rag.itworx.tech` — note the *admin UI* and the *API* share + one process/origin, `/v1/uploads` is reachable at `https://rag.itworx.tech/v1/uploads`, + **not** `ragcore.itworx.tech`, which only appears in RFC7807 problem-type URLs) by copying + the 6 changed source files directly into the server checkout and `docker compose build + app && up -d --no-deps app` (deliberately not committing to RAGcore's git history or + touching the `worker` service, since a different agent has substantial unrelated + uncommitted work in that same working tree). Confirmed live with a garbage token (still + correctly 401) and then with a freshly-issued, correctly-scoped real token (403 + `UPLOAD_TARGET_FORBIDDEN` against a dummy space ID — i.e. authentication succeeded, + authorization correctly rejected the wrong space — proving the fix end-to-end before + touching n8n at all). +- **Root cause of the n8n-side 401 found and fixed**: separately from the RAGcore bug above, + the "Upload to RAGcore" HTTP node's Authentication was set to Header Auth, but **no + credential had ever actually been attached** to that picker — so the node was sending no + `Authorization` header at all, which produces the identical 401 to a malformed one (easy + to conflate with the RAGcore-side bug, which is why fixing RAGcore alone didn't resolve + the symptom). There was already an unused "RAGcore Sync Token" n8n credential sitting + around from the earlier session (its value likely never actually got saved when it was + first created, or was created but never selected on this node — not conclusively + determined). Owner attached it and set Name=`Authorization`, + Value=`Bearer ` (a new credential issued via the + RAGcore admin UI at `/admin/control/applications/c20ac48a-d57b-4c68-9bd1-564f49c1a473/credentials/new` + specifically for this, service account "n8n Procedure Sync (production)"; the earlier + diagnostic-only credential used to prove the RAGcore fix was revoked afterward via direct + SQL `UPDATE identity.service_account_credentials SET revoked_at = now() ...` since the + admin UI has no revoke button). +- **Workflow 3's "Upload to RAGcore" node live-validated end-to-end, real data**: ran the + full workflow via n8n's "Execute workflow" (Schedule Trigger → List procedures → Prepare + uploads → Upload to RAGcore). All 33 items succeeded — each output item is a real + `AcceptedJob` (`job_id`/`status_url`), not error output. Independently confirmed at the + database level (not just trusting the n8n UI): `select count(*) from jobs.jobs where + operation='ingest_upload' and created_at > now() - interval '5 minutes'` → **33**, on the + live RAGcore Postgres. +- **n8n browser-automation notes for this environment** (worth knowing before attempting + canvas interaction again): (1) an n8n NPS survey modal (`role=dialog`, "We've been busy") + intermittently covers the whole canvas and silently eats every click underneath it until + removed; (2) canvas node positions reported by `getBoundingClientRect()` drift between + successive tool calls in a way that made coordinate-based `computer` clicks and even + `find`-ref-based clicks land on the wrong element repeatedly this session (dozens of failed + attempts, multiple different coordinate-math theories, none reliable) — directly setting + `.vue-flow__transformationpane`'s inline `style.transform` to force a node into view + **desyncs vue-flow's own internal pan/zoom state**, making the problem worse, not better; + (3) what actually worked reliably every time: calling native `.click()` directly via JS on + a plain `