M5: implement RAGcore knowledge integration
KnowledgeProvider protocol with a deterministic TF-IDF-weighted extractive demo provider (never generative, always cites real excerpts) and a RAGcore HTTP adapter that degrades cleanly to unavailable. Knowledge nav + chat-style Q&A UI with source cards and honest grounded/insufficient/unavailable states. 57 backend tests passing, ruff clean. Fixed a real relevance bug (generic terms like "vehicle" crowding out distinctive matches) via IDF weighting, found by testing the actual S6 scenario. Verified end-to-end in the browser: grounded damage question cites both expected procedures; unrelated question honestly returns insufficient evidence with no fabrication.
This commit is contained in:
NuklearRabbit
+17
-3
@@ -2,7 +2,7 @@
|
||||
|
||||
## Current milestone
|
||||
|
||||
M4 — complete. Starting M5 next.
|
||||
M5 — complete. Starting M6 next.
|
||||
|
||||
## Locked decisions
|
||||
|
||||
@@ -94,10 +94,24 @@ M4 — complete. Starting M5 next.
|
||||
- Full live round trip (not mocked): registered a real return on `BK-DEMO-RETURN` → outbox event queued → background dispatcher delivered it to the now-activated n8n workflow within its 3s poll interval → n8n called back into `/api/v1/integrations/n8n/return-callback` (200 OK, confirmed in `docker compose logs api`) → dispatcher's original POST received n8n's success response → event flipped to `succeeded` on attempt 1, visible on `/automation`.
|
||||
- S5 scenario end-to-end in the browser: seeded `BK-H-0020` (`failed`, 3 attempts, "Synthetic connection timeout to n8n") → clicked Retry → `pending` → within ~3s, live dispatcher delivered it through the real n8n instance → `succeeded`, 4 attempts. This is the full documented S5 scenario working for real, not simulated.
|
||||
|
||||
### M5 — RAGcore knowledge integration
|
||||
- `app/services/knowledge/__init__.py`: `KnowledgeProvider` Protocol (sync, not async — the rest of the backend is sync SQLAlchemy/FastAPI, so an async provider interface would have meant bridging paradigms for no benefit) with `health()`/`ask()`, plus `GroundedAnswer`/`SourceCard`/`KnowledgeHealth` Pydantic models matching `contracts/openapi.yaml`'s `GroundedAnswer` schema exactly. `get_knowledge_provider()` factory switches on `settings.knowledge_provider` ("demo" default, "ragcore" opt-in).
|
||||
- `app/services/knowledge/demo.py` — `DemoKnowledgeProvider`: parses the 10 `knowledge/procedures/*.md` files' YAML frontmatter (hand-rolled flat parser, not PyYAML — avoided adding a dependency for a 6-key flat block) and `## `-delimited sections at startup, then does **TF-IDF-weighted keyword retrieval** (not naive keyword counting) with light suffix-stripping stemming (`returns`→`return`, `damaged`→`damage`). This is extractive, not generative: it returns real excerpts and a templated answer sentence, never invented text.
|
||||
- **Real bug found and fixed by testing the actual S6 question, not by inspection**: naive flat keyword-overlap scoring (first cut) let the word "vehicle" — present in nearly every document's title — crowd out the actually-relevant `damage-procedure` document from the top-3 results for "What must I do when a vehicle returns with damage?", because generic words scored the same as distinctive ones. Fixed by computing corpus-wide IDF per token (`log((N+1)/(df+1)) + 1`) and weighting matches by it, so common terms contribute little and rare/distinctive terms (like "damage") dominate the ranking. Verified: the S6 question now returns `damage-procedure` and `vehicle-return-procedure` in the top 3, matching the documented expectation exactly.
|
||||
- `app/services/knowledge/ragcore.py` — `RAGcoreKnowledgeProvider`: real `httpx` adapter guessing a plausible REST contract (`GET /health`, `POST /api/v1/ask`) per `contracts/ragcore-contract-assumptions.md` (RAGcore is built separately; no live instance was reachable this session to verify against). Any connection error, timeout, or malformed response degrades to `evidence_state: "unavailable"` rather than raising — this is the adapter that actually exercises the architecture's "RAGcore failure disables knowledge answers only" reliability boundary. Not wired as the active provider by default; `KNOWLEDGE_PROVIDER=ragcore` would need a real, verified base URL to turn on.
|
||||
- `POST /api/v1/knowledge/questions` + `GET /api/v1/knowledge/status` (`app/api/routers/knowledge.py`). Audit event `knowledge_question_asked` logs `evidence_state`, `provider`, `source_ids`, and `question_length` only — **not** the question text itself, per `docs/12-security-and-audit.md` ("log question metadata and source IDs, not unnecessary full prompts").
|
||||
- Knowledge nav + page (`pages/Knowledge.tsx`): chat-style question box, source cards (title/version/section/excerpt) prioritized over the answer text per `docs/06-ui-ux.md`, explicit `grounded`/`insufficient`/`unavailable` states with distinct visual treatment — never a fabricated-looking answer for the latter two.
|
||||
- Dockerfile now also `COPY knowledge ./knowledge`; added `KNOWLEDGE_DIR` setting (`/app/knowledge/procedures` in-container, same pattern as `SEED_DIR`) rather than deriving the path from `__file__` — simpler and doesn't break if the module moves.
|
||||
- Commands run and verified from this checkout:
|
||||
- `docker compose run --rm api pytest -q` — **57 passed** (new `tests/test_knowledge.py`: S6 grounded-with-expected-sources, unrelated question is honestly insufficient with no fabrication, demo provider health/document count, endpoint auth required, audit doesn't leak question text, RAGcore adapter degrades to unavailable on a simulated connection error).
|
||||
- `docker compose run --rm api ruff check .` — All checks passed.
|
||||
- `npm run build` — clean.
|
||||
- Full browser run of S6 end-to-end: asked "What must I do when a vehicle returns with damage?" on `/knowledge` → grounded answer citing "Vehicle return procedure" (2 sections) and "Damage handling procedure" with real excerpts. Also asked an unrelated question ("What is the weather forecast for tomorrow?") → correctly returned "Insufficient evidence" / "No matching procedure was found" with zero sources, confirming no fabrication.
|
||||
|
||||
## Known blockers
|
||||
|
||||
None. External service credentials may be absent; use the documented demo/degraded providers. The n8n workflow-activation steps above are a one-time manual setup requirement in this environment, not a blocker — but not yet scripted; M7 should either automate it (e.g. a bootstrap script CI/compose can run) or document it clearly enough for `docs/14-testing-and-acceptance.md`'s clean-checkout criteria.
|
||||
None. External service credentials may be absent; use the documented demo/degraded providers. The n8n workflow-activation steps from M4 are a one-time manual setup requirement in this environment, not a blocker — but not yet scripted; M7 should either automate it (e.g. a bootstrap script CI/compose can run) or document it clearly enough for `docs/14-testing-and-acceptance.md`'s clean-checkout criteria. RAGcore itself was never reachable this session — `RAGcoreKnowledgeProvider` is implemented and unit-tested for its unavailable-degradation path but its actual request/response contract against a real RAGcore instance is unverified; the demo provider is what M5's acceptance criteria are actually satisfied by.
|
||||
|
||||
## Exact next action
|
||||
|
||||
Start M5 (RAGcore knowledge integration): read `docs/09-ragcore-integration.md`, `knowledge/manifest.json`. Implement the `KnowledgeProvider` protocol (`health`, `ask`, `sync_manifest`), a `DemoKnowledgeProvider` doing deterministic keyword/BM25-style retrieval directly over `knowledge/procedures/*.md` (extractive, not generative — must return real excerpts + `insufficient`/`unavailable` states honestly, never fabricate), and a `RAGcoreKnowledgeProvider` adapter stub for the real service (`RAGCORE_BASE_URL` etc. are already in config/compose from M0, but no actual RAGcore instance is confirmed reachable this session — build the demo provider as the one that actually has to work for the acceptance criteria, and make the RAGcore adapter degrade to `unavailable` cleanly if unreachable, per `docs/03-architecture.md`'s reliability boundary "RAGcore failure disables knowledge answers only"). Add `POST /api/v1/knowledge/questions` and `GET /api/v1/knowledge/status`, a Knowledge nav item + page (chat-style question box, source cards with title/version/section/excerpt, explicit unavailable/insufficient states — no fabricated answers ever). S6 demo scenario: "What must I do when a vehicle returns with damage?" must cite `knowledge/procedures/03-damage-handling.md` and `02-vehicle-return.md`.
|
||||
Start M6 (ITWorx MCP Hub publication): read `docs/10-mcp-hub-integration.md`, `contracts/mcp-tools.json`. Implement the four read-only, service-token-protected provider endpoints under `/api/v1/integrations/mcp/` (`operations-summary`, `attention-vehicles`, `vehicles/{public_ref}`, and a knowledge-search façade wrapping the M5 `KnowledgeProvider` — per `contracts/mcp-tools.json`'s `mobilityops_search_knowledge` tool and `docs/10-mcp-hub-integration.md`'s note to route through a narrow façade rather than duplicate retrieval logic). Auth: reuse the same shared-secret-header pattern already built for the n8n callback in M4 (`X-Service-Token`, a new `MCP_HUB_SERVICE_TOKEN` setting — `.env.example` already has the env var name reserved) rather than inventing a second auth mechanism. Must not expose generic SQL, arbitrary fetch, write/mutation actions, or secrets — these are read-only summaries only. Add MobilityOps-side service-request audit events (the Hub owns its own central tool-call audit; MobilityOps only needs to record that its provider APIs were reached, with tool name/correlation ID/client identity/result status). No actual ITWorx MCP Hub instance is confirmed reachable in this environment (same situation as RAGcore in M5) — validate the four provider endpoints directly via authenticated `curl`/tests rather than a live Hub round trip, and note that gap explicitly rather than claiming an unverified integration works.
|
||||
|
||||
@@ -9,6 +9,7 @@ COPY backend/alembic ./alembic
|
||||
COPY backend/alembic.ini ./
|
||||
COPY backend/tests ./tests
|
||||
COPY seed ./seed
|
||||
COPY knowledge ./knowledge
|
||||
RUN pip install --no-cache-dir --no-deps -e .
|
||||
EXPOSE 8000
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.deps import get_current_user, get_db
|
||||
from app.schemas import CurrentUser
|
||||
from app.services.audit import record_audit_event
|
||||
from app.services.knowledge import GroundedAnswer, KnowledgeHealth, get_knowledge_provider
|
||||
|
||||
router = APIRouter(prefix="/api/v1/knowledge", tags=["knowledge"])
|
||||
|
||||
|
||||
class AskQuestionRequest(BaseModel):
|
||||
question: str = Field(min_length=3, max_length=1000)
|
||||
|
||||
|
||||
@router.post("/questions", response_model=GroundedAnswer)
|
||||
def ask_question(
|
||||
body: AskQuestionRequest,
|
||||
db: Session = Depends(get_db),
|
||||
user: CurrentUser = Depends(get_current_user),
|
||||
) -> GroundedAnswer:
|
||||
correlation_id = str(uuid.uuid4())
|
||||
provider = get_knowledge_provider()
|
||||
answer = provider.ask(body.question, correlation_id)
|
||||
|
||||
record_audit_event(
|
||||
db,
|
||||
actor_type="user",
|
||||
actor_label=user.display_name,
|
||||
action="knowledge_question_asked",
|
||||
entity_type="knowledge",
|
||||
correlation_id=uuid.UUID(correlation_id),
|
||||
metadata={
|
||||
"evidence_state": answer.evidence_state,
|
||||
"provider": answer.provider,
|
||||
"source_ids": [s.document_id for s in answer.sources],
|
||||
"question_length": len(body.question),
|
||||
},
|
||||
)
|
||||
db.commit()
|
||||
return answer
|
||||
|
||||
|
||||
@router.get("/status", response_model=KnowledgeHealth)
|
||||
def knowledge_status(_user: CurrentUser = Depends(get_current_user)) -> KnowledgeHealth:
|
||||
return get_knowledge_provider().health()
|
||||
@@ -14,6 +14,8 @@ class Settings(BaseSettings):
|
||||
ragcore_tenant: str = "northstar-mobility-demo"
|
||||
ragcore_workspace: str = "mobilityops"
|
||||
ragcore_collection: str = "internal-procedures"
|
||||
ragcore_api_token: str = ""
|
||||
ragcore_http_timeout_seconds: float = 5.0
|
||||
n8n_webhook_url: str = "http://n8n:5678/webhook/mobilityops-return"
|
||||
n8n_callback_token: str = "replace-me-n8n-callback-token"
|
||||
n8n_dispatch_enabled: bool = True
|
||||
@@ -24,6 +26,7 @@ class Settings(BaseSettings):
|
||||
session_cookie_name: str = "mobilityops_session"
|
||||
session_ttl_seconds: int = 60 * 60 * 8
|
||||
seed_dir: str = "/app/seed"
|
||||
knowledge_dir: str = "/app/knowledge/procedures"
|
||||
cors_allow_origins: str = "http://localhost:1228"
|
||||
demo_today: str = "2026-08-01"
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ from app.api.routers import (
|
||||
data_quality,
|
||||
demo,
|
||||
integrations,
|
||||
knowledge,
|
||||
vehicles,
|
||||
workflows,
|
||||
)
|
||||
@@ -84,3 +85,4 @@ app.include_router(audit.router)
|
||||
app.include_router(data_quality.router)
|
||||
app.include_router(workflows.router)
|
||||
app.include_router(integrations.router)
|
||||
app.include_router(knowledge.router)
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from functools import lru_cache
|
||||
from typing import Literal, Protocol
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.core.config import get_settings
|
||||
|
||||
EvidenceState = Literal["grounded", "insufficient", "unavailable"]
|
||||
|
||||
|
||||
class SourceCard(BaseModel):
|
||||
document_id: str
|
||||
title: str
|
||||
version: str
|
||||
section: str
|
||||
excerpt: str
|
||||
|
||||
|
||||
class GroundedAnswer(BaseModel):
|
||||
answer: str
|
||||
evidence_state: EvidenceState
|
||||
sources: list[SourceCard]
|
||||
provider: str
|
||||
correlation_id: str
|
||||
|
||||
|
||||
class KnowledgeHealth(BaseModel):
|
||||
provider: str
|
||||
available: bool
|
||||
detail: str
|
||||
tenant: str
|
||||
workspace: str
|
||||
collection: str
|
||||
document_count: int
|
||||
|
||||
|
||||
class KnowledgeProvider(Protocol):
|
||||
name: str
|
||||
|
||||
def health(self) -> KnowledgeHealth: ...
|
||||
|
||||
def ask(self, question: str, correlation_id: str) -> GroundedAnswer: ...
|
||||
|
||||
|
||||
@lru_cache
|
||||
def get_knowledge_provider() -> KnowledgeProvider:
|
||||
settings = get_settings()
|
||||
if settings.knowledge_provider == "ragcore":
|
||||
from app.services.knowledge.ragcore import RAGcoreKnowledgeProvider
|
||||
|
||||
return RAGcoreKnowledgeProvider()
|
||||
from app.services.knowledge.demo import DemoKnowledgeProvider
|
||||
|
||||
return DemoKnowledgeProvider()
|
||||
@@ -0,0 +1,221 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
from app.core.config import get_settings
|
||||
from app.services.knowledge import GroundedAnswer, KnowledgeHealth, SourceCard
|
||||
|
||||
STOPWORDS = {
|
||||
"a", "an", "the", "is", "are", "was", "were", "be", "been", "being",
|
||||
"to", "of", "in", "on", "at", "for", "and", "or", "but", "if", "then",
|
||||
"do", "does", "did", "must", "may", "can", "could", "should", "would",
|
||||
"i", "you", "it", "we", "they", "my", "your", "what", "when", "how",
|
||||
"with", "without", "this", "that", "these", "those", "not", "no",
|
||||
}
|
||||
|
||||
_WORD_RE = re.compile(r"[a-z0-9]+")
|
||||
|
||||
|
||||
def _stem(word: str) -> str:
|
||||
# Deterministic, intentionally crude suffix stripping — good enough to match "returns"
|
||||
# with "return" or "damaged" with "damage" without pulling in a stemming dependency.
|
||||
for suffix in ("ing", "edly", "ed", "es", "s"):
|
||||
if len(word) > len(suffix) + 2 and word.endswith(suffix):
|
||||
return word[: -len(suffix)]
|
||||
return word
|
||||
|
||||
|
||||
def _tokenize(text: str) -> set[str]:
|
||||
words = _WORD_RE.findall(text.lower())
|
||||
return {_stem(w) for w in words if w not in STOPWORDS and len(w) > 2}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Document:
|
||||
document_id: str
|
||||
title: str
|
||||
version: str
|
||||
title_tokens: set[str] = field(default_factory=set)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScoredSection:
|
||||
document: Document
|
||||
heading: str
|
||||
text: str
|
||||
heading_tokens: set[str]
|
||||
body_tokens: set[str]
|
||||
|
||||
|
||||
def _parse_frontmatter(raw: str) -> tuple[dict[str, str], str]:
|
||||
if not raw.startswith("---"):
|
||||
return {}, raw
|
||||
end = raw.find("\n---", 3)
|
||||
if end == -1:
|
||||
return {}, raw
|
||||
block = raw[3:end].strip()
|
||||
body = raw[end + 4 :].lstrip("\n")
|
||||
meta: dict[str, str] = {}
|
||||
for line in block.splitlines():
|
||||
if ":" not in line:
|
||||
continue
|
||||
key, _, value = line.partition(":")
|
||||
meta[key.strip()] = value.strip().strip('"')
|
||||
return meta, body
|
||||
|
||||
|
||||
def _split_sections(body: str) -> list[tuple[str, str]]:
|
||||
sections: list[tuple[str, str]] = []
|
||||
current_heading = "Overview"
|
||||
current_lines: list[str] = []
|
||||
for line in body.splitlines():
|
||||
if line.startswith("## "):
|
||||
if current_lines:
|
||||
sections.append((current_heading, "\n".join(current_lines).strip()))
|
||||
current_heading = line[3:].strip()
|
||||
current_lines = []
|
||||
elif line.startswith("# "):
|
||||
continue
|
||||
else:
|
||||
current_lines.append(line)
|
||||
if current_lines:
|
||||
sections.append((current_heading, "\n".join(current_lines).strip()))
|
||||
return sections
|
||||
|
||||
|
||||
def _load_sections(procedures_dir: Path) -> list[ScoredSection]:
|
||||
sections: list[ScoredSection] = []
|
||||
for path in sorted(procedures_dir.glob("*.md")):
|
||||
raw = path.read_text(encoding="utf-8")
|
||||
meta, body = _parse_frontmatter(raw)
|
||||
title = meta.get("title", path.stem)
|
||||
doc = Document(
|
||||
document_id=meta.get("document_id", path.stem),
|
||||
title=title,
|
||||
version=meta.get("version", "1.0"),
|
||||
title_tokens=_tokenize(title),
|
||||
)
|
||||
for heading, text in _split_sections(body):
|
||||
sections.append(
|
||||
ScoredSection(
|
||||
document=doc,
|
||||
heading=heading,
|
||||
text=text,
|
||||
heading_tokens=_tokenize(heading),
|
||||
body_tokens=_tokenize(text),
|
||||
)
|
||||
)
|
||||
return sections
|
||||
|
||||
|
||||
class DemoKnowledgeProvider:
|
||||
"""Deterministic extractive retrieval over the local procedure Markdown files.
|
||||
|
||||
Not a generative model: it scores sections with TF-IDF-weighted keyword overlap
|
||||
(downweighting terms common across the whole corpus, like "vehicle", in favor of
|
||||
distinctive ones, like "damage") and returns real excerpts, never invented text.
|
||||
"""
|
||||
|
||||
name = "demo"
|
||||
|
||||
def __init__(self) -> None:
|
||||
settings = get_settings()
|
||||
self._settings = settings
|
||||
self._procedures_dir = Path(settings.knowledge_dir)
|
||||
self._sections = _load_sections(self._procedures_dir)
|
||||
self._document_count = len({s.document.document_id for s in self._sections})
|
||||
self._idf = self._build_idf(self._sections)
|
||||
|
||||
@staticmethod
|
||||
def _build_idf(sections: list[ScoredSection]) -> dict[str, float]:
|
||||
n = len(sections) or 1
|
||||
doc_freq: dict[str, int] = {}
|
||||
for section in sections:
|
||||
doc = section.document
|
||||
all_tokens = doc.title_tokens | section.heading_tokens | section.body_tokens
|
||||
for token in all_tokens:
|
||||
doc_freq[token] = doc_freq.get(token, 0) + 1
|
||||
return {token: math.log((n + 1) / (df + 1)) + 1 for token, df in doc_freq.items()}
|
||||
|
||||
def health(self) -> KnowledgeHealth:
|
||||
return KnowledgeHealth(
|
||||
provider=self.name,
|
||||
available=True,
|
||||
detail="Deterministic keyword-matching demo provider; no external service.",
|
||||
tenant=self._settings.ragcore_tenant,
|
||||
workspace=self._settings.ragcore_workspace,
|
||||
collection=self._settings.ragcore_collection,
|
||||
document_count=self._document_count,
|
||||
)
|
||||
|
||||
def _score(self, query_tokens: set[str], section: ScoredSection) -> float:
|
||||
score = 0.0
|
||||
for token in query_tokens:
|
||||
idf = self._idf.get(token, 0.0)
|
||||
if idf == 0.0:
|
||||
continue
|
||||
if token in section.heading_tokens:
|
||||
score += 3 * idf
|
||||
elif token in section.document.title_tokens:
|
||||
score += 2 * idf
|
||||
elif token in section.body_tokens:
|
||||
score += idf
|
||||
return score
|
||||
|
||||
def ask(self, question: str, correlation_id: str) -> GroundedAnswer:
|
||||
query_tokens = _tokenize(question)
|
||||
scored = [
|
||||
(self._score(query_tokens, section), section)
|
||||
for section in self._sections
|
||||
]
|
||||
scored = [(score, section) for score, section in scored if score > 0]
|
||||
scored.sort(key=lambda item: item[0], reverse=True)
|
||||
top = scored[:3]
|
||||
|
||||
if not top:
|
||||
return GroundedAnswer(
|
||||
answer="No matching procedure was found for this question.",
|
||||
evidence_state="insufficient",
|
||||
sources=[],
|
||||
provider=self.name,
|
||||
correlation_id=correlation_id,
|
||||
)
|
||||
|
||||
sources = [
|
||||
SourceCard(
|
||||
document_id=section.document.document_id,
|
||||
title=section.document.title,
|
||||
version=section.document.version,
|
||||
section=section.heading,
|
||||
excerpt=(section.text[:400] + "…") if len(section.text) > 400 else section.text,
|
||||
)
|
||||
for _, section in top
|
||||
]
|
||||
|
||||
if top[0][0] < 3:
|
||||
return GroundedAnswer(
|
||||
answer=(
|
||||
"The available procedures do not clearly answer this question. "
|
||||
"The closest matches are included below for review."
|
||||
),
|
||||
evidence_state="insufficient",
|
||||
sources=sources,
|
||||
provider=self.name,
|
||||
correlation_id=correlation_id,
|
||||
)
|
||||
|
||||
lead_section = top[0][1]
|
||||
answer = (
|
||||
f'Per "{lead_section.document.title}" (v{lead_section.document.version}), '
|
||||
f'section "{lead_section.heading}": {lead_section.text.splitlines()[0][:300]}'
|
||||
)
|
||||
return GroundedAnswer(
|
||||
answer=answer,
|
||||
evidence_state="grounded",
|
||||
sources=sources,
|
||||
provider=self.name,
|
||||
correlation_id=correlation_id,
|
||||
)
|
||||
@@ -0,0 +1,98 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core.config import get_settings
|
||||
from app.services.knowledge import GroundedAnswer, KnowledgeHealth, SourceCard
|
||||
|
||||
|
||||
class RAGcoreKnowledgeProvider:
|
||||
"""Adapter for the central RAGcore service.
|
||||
|
||||
RAGcore is built and owned separately (see contracts/ragcore-contract-assumptions.md).
|
||||
No live RAGcore instance was reachable during this build, so the exact request/response
|
||||
shape below is a best-effort guess at a REST contract; any failure (connection, timeout,
|
||||
malformed response) degrades to `unavailable` rather than raising, per the architecture's
|
||||
reliability boundary: RAGcore failure disables knowledge answers only, never the rest of
|
||||
the app, and never fabricates an answer.
|
||||
"""
|
||||
|
||||
name = "ragcore"
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._settings = get_settings()
|
||||
|
||||
def _client(self) -> httpx.Client:
|
||||
headers = {}
|
||||
if self._settings.ragcore_api_token:
|
||||
headers["Authorization"] = f"Bearer {self._settings.ragcore_api_token}"
|
||||
return httpx.Client(
|
||||
base_url=self._settings.ragcore_base_url,
|
||||
headers=headers,
|
||||
timeout=self._settings.ragcore_http_timeout_seconds,
|
||||
)
|
||||
|
||||
def health(self) -> KnowledgeHealth:
|
||||
try:
|
||||
with self._client() as client:
|
||||
response = client.get("/health")
|
||||
response.raise_for_status()
|
||||
available = True
|
||||
detail = "RAGcore reachable."
|
||||
except httpx.HTTPError as exc:
|
||||
available = False
|
||||
detail = f"RAGcore unavailable: {type(exc).__name__}: {exc}"
|
||||
return KnowledgeHealth(
|
||||
provider=self.name,
|
||||
available=available,
|
||||
detail=detail,
|
||||
tenant=self._settings.ragcore_tenant,
|
||||
workspace=self._settings.ragcore_workspace,
|
||||
collection=self._settings.ragcore_collection,
|
||||
document_count=0,
|
||||
)
|
||||
|
||||
def ask(self, question: str, correlation_id: str) -> GroundedAnswer:
|
||||
try:
|
||||
with self._client() as client:
|
||||
response = client.post(
|
||||
"/api/v1/ask",
|
||||
json={
|
||||
"tenant": self._settings.ragcore_tenant,
|
||||
"workspace": self._settings.ragcore_workspace,
|
||||
"collection": self._settings.ragcore_collection,
|
||||
"question": question,
|
||||
"correlation_id": correlation_id,
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
body = response.json()
|
||||
except (httpx.HTTPError, ValueError):
|
||||
return GroundedAnswer(
|
||||
answer="",
|
||||
evidence_state="unavailable",
|
||||
sources=[],
|
||||
provider=self.name,
|
||||
correlation_id=correlation_id,
|
||||
)
|
||||
|
||||
try:
|
||||
sources = [SourceCard(**s) for s in body.get("sources", [])]
|
||||
evidence_state = body.get("evidence_state", "insufficient")
|
||||
if evidence_state not in ("grounded", "insufficient", "unavailable"):
|
||||
evidence_state = "insufficient"
|
||||
return GroundedAnswer(
|
||||
answer=body.get("answer", ""),
|
||||
evidence_state=evidence_state,
|
||||
sources=sources,
|
||||
provider=self.name,
|
||||
correlation_id=correlation_id,
|
||||
)
|
||||
except (TypeError, ValueError):
|
||||
return GroundedAnswer(
|
||||
answer="",
|
||||
evidence_state="unavailable",
|
||||
sources=[],
|
||||
provider=self.name,
|
||||
correlation_id=correlation_id,
|
||||
)
|
||||
@@ -0,0 +1,84 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
|
||||
from app.services.knowledge.demo import DemoKnowledgeProvider
|
||||
from app.services.knowledge.ragcore import RAGcoreKnowledgeProvider
|
||||
|
||||
|
||||
def test_s6_damage_question_is_grounded_with_expected_sources():
|
||||
provider = DemoKnowledgeProvider()
|
||||
answer = provider.ask(
|
||||
"What must I do when a vehicle returns with damage?", "test-correlation-1"
|
||||
)
|
||||
assert answer.evidence_state == "grounded"
|
||||
document_ids = {s.document_id for s in answer.sources}
|
||||
assert "damage-procedure" in document_ids
|
||||
assert "vehicle-return-procedure" in document_ids
|
||||
assert answer.answer # never empty for a grounded answer
|
||||
for source in answer.sources:
|
||||
assert source.excerpt
|
||||
|
||||
|
||||
def test_unrelated_question_is_insufficient():
|
||||
provider = DemoKnowledgeProvider()
|
||||
answer = provider.ask("What is the capital of France?", "test-correlation-2")
|
||||
assert answer.evidence_state == "insufficient"
|
||||
assert "France" not in answer.answer # never fabricates an answer beyond the procedures
|
||||
|
||||
|
||||
def test_demo_provider_health_reports_document_count():
|
||||
provider = DemoKnowledgeProvider()
|
||||
health = provider.health()
|
||||
assert health.provider == "demo"
|
||||
assert health.available is True
|
||||
assert health.document_count == 10
|
||||
|
||||
|
||||
def test_ask_question_endpoint_grounded(ops_client):
|
||||
response = ops_client.post(
|
||||
"/api/v1/knowledge/questions",
|
||||
json={"question": "What must I do when a vehicle returns with damage?"},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["evidence_state"] == "grounded"
|
||||
assert body["provider"] == "demo"
|
||||
assert len(body["sources"]) > 0
|
||||
|
||||
|
||||
def test_ask_question_requires_authentication(client):
|
||||
response = client.post("/api/v1/knowledge/questions", json={"question": "Anything?"})
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_ask_question_is_audited_without_leaking_full_text(ops_client):
|
||||
ops_client.post(
|
||||
"/api/v1/knowledge/questions",
|
||||
json={"question": "What must I do when a vehicle returns with damage?"},
|
||||
)
|
||||
events = ops_client.get(
|
||||
"/api/v1/audit", params={"action": "knowledge_question_asked"}
|
||||
).json()
|
||||
assert len(events) >= 1
|
||||
metadata = events[0]["metadata"]
|
||||
assert "evidence_state" in metadata
|
||||
assert "source_ids" in metadata
|
||||
assert "question" not in metadata
|
||||
|
||||
|
||||
def test_knowledge_status_endpoint(ops_client):
|
||||
response = ops_client.get("/api/v1/knowledge/status")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["provider"] == "demo"
|
||||
|
||||
|
||||
def test_ragcore_provider_degrades_to_unavailable(monkeypatch):
|
||||
def fake_client(*args, **kwargs):
|
||||
raise httpx.ConnectError("no ragcore in this environment")
|
||||
|
||||
provider = RAGcoreKnowledgeProvider()
|
||||
monkeypatch.setattr(provider, "_client", fake_client)
|
||||
answer = provider.ask("Anything?", "test-correlation-3")
|
||||
assert answer.evidence_state == "unavailable"
|
||||
assert answer.sources == []
|
||||
@@ -11,6 +11,7 @@ import { BookingDetail } from "./pages/BookingDetail";
|
||||
import { DataQuality } from "./pages/DataQuality";
|
||||
import { DataQualityIssueDetail } from "./pages/DataQualityIssueDetail";
|
||||
import { Automation } from "./pages/Automation";
|
||||
import { Knowledge } from "./pages/Knowledge";
|
||||
import { Audit } from "./pages/Audit";
|
||||
|
||||
export function App() {
|
||||
@@ -33,6 +34,7 @@ export function App() {
|
||||
<Route path="/data-quality" element={<DataQuality />} />
|
||||
<Route path="/data-quality/:publicRef" element={<DataQualityIssueDetail />} />
|
||||
<Route path="/automation" element={<Automation />} />
|
||||
<Route path="/knowledge" element={<Knowledge />} />
|
||||
<Route path="/audit" element={<Audit />} />
|
||||
</Route>
|
||||
<Route path="/" element={<Navigate to="/dashboard" replace />} />
|
||||
|
||||
@@ -167,6 +167,32 @@ export interface MergeCustomersResult {
|
||||
rewired_bookings: number;
|
||||
}
|
||||
|
||||
export interface SourceCard {
|
||||
document_id: string;
|
||||
title: string;
|
||||
version: string;
|
||||
section: string;
|
||||
excerpt: string;
|
||||
}
|
||||
|
||||
export interface GroundedAnswer {
|
||||
answer: string;
|
||||
evidence_state: "grounded" | "insufficient" | "unavailable";
|
||||
sources: SourceCard[];
|
||||
provider: string;
|
||||
correlation_id: string;
|
||||
}
|
||||
|
||||
export interface KnowledgeHealth {
|
||||
provider: string;
|
||||
available: boolean;
|
||||
detail: string;
|
||||
tenant: string;
|
||||
workspace: string;
|
||||
collection: string;
|
||||
document_count: number;
|
||||
}
|
||||
|
||||
export interface AuditEvent {
|
||||
id: string;
|
||||
actor_type: string;
|
||||
|
||||
@@ -6,6 +6,7 @@ const NAV_ITEMS = [
|
||||
{ to: "/vehicles", label: "Vehicles" },
|
||||
{ to: "/bookings", label: "Bookings" },
|
||||
{ to: "/data-quality", label: "Data Quality" },
|
||||
{ to: "/knowledge", label: "Knowledge" },
|
||||
{ to: "/automation", label: "Automation" },
|
||||
{ to: "/audit", label: "Audit" },
|
||||
];
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
import { useEffect, useState, type FormEvent } from "react";
|
||||
import { api, ApiError } from "../api/client";
|
||||
import type { GroundedAnswer, KnowledgeHealth } from "../api/types";
|
||||
|
||||
interface Exchange {
|
||||
question: string;
|
||||
answer: GroundedAnswer;
|
||||
}
|
||||
|
||||
const EVIDENCE_LABEL: Record<GroundedAnswer["evidence_state"], string> = {
|
||||
grounded: "Grounded in cited procedures",
|
||||
insufficient: "Insufficient evidence",
|
||||
unavailable: "Knowledge service unavailable",
|
||||
};
|
||||
|
||||
export function Knowledge() {
|
||||
const [status, setStatus] = useState<KnowledgeHealth | null>(null);
|
||||
const [question, setQuestion] = useState("");
|
||||
const [submitting, setSubmitting] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [exchanges, setExchanges] = useState<Exchange[]>([]);
|
||||
|
||||
useEffect(() => {
|
||||
api
|
||||
.get<KnowledgeHealth>("/api/v1/knowledge/status")
|
||||
.then(setStatus)
|
||||
.catch(() => setStatus(null));
|
||||
}, []);
|
||||
|
||||
async function handleSubmit(e: FormEvent) {
|
||||
e.preventDefault();
|
||||
if (!question.trim()) return;
|
||||
setError(null);
|
||||
setSubmitting(true);
|
||||
try {
|
||||
const answer = await api.post<GroundedAnswer>("/api/v1/knowledge/questions", { question });
|
||||
setExchanges((prev) => [{ question, answer }, ...prev]);
|
||||
setQuestion("");
|
||||
} catch (err) {
|
||||
setError(err instanceof ApiError ? err.message : "Could not reach the knowledge service.");
|
||||
} finally {
|
||||
setSubmitting(false);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="page">
|
||||
<h1>Knowledge</h1>
|
||||
{status && (
|
||||
<p className="knowledge-status">
|
||||
Provider: <strong>{status.provider}</strong> ·{" "}
|
||||
{status.available ? "available" : "unavailable"} · {status.document_count} procedures
|
||||
indexed
|
||||
</p>
|
||||
)}
|
||||
|
||||
<form className="panel knowledge-form" onSubmit={handleSubmit} aria-labelledby="ask-heading">
|
||||
<h2 id="ask-heading">Ask a procedure question</h2>
|
||||
<label htmlFor="knowledge-question" className="visually-hidden">
|
||||
Question
|
||||
</label>
|
||||
<div className="knowledge-input-row">
|
||||
<input
|
||||
id="knowledge-question"
|
||||
type="text"
|
||||
value={question}
|
||||
onChange={(e) => setQuestion(e.target.value)}
|
||||
placeholder="e.g. What must I do when a vehicle returns with damage?"
|
||||
minLength={3}
|
||||
maxLength={1000}
|
||||
required
|
||||
/>
|
||||
<button type="submit" disabled={submitting}>
|
||||
{submitting ? "Asking…" : "Ask"}
|
||||
</button>
|
||||
</div>
|
||||
{error && <p className="error" role="alert">{error}</p>}
|
||||
</form>
|
||||
|
||||
{exchanges.length === 0 && !error && (
|
||||
<p>Ask a question about one of the ten operational procedures to see cited sources.</p>
|
||||
)}
|
||||
|
||||
<ul className="exchange-list">
|
||||
{exchanges.map((exchange, index) => (
|
||||
<li key={index} className="panel exchange">
|
||||
<p className="exchange-question">
|
||||
<strong>Q:</strong> {exchange.question}
|
||||
</p>
|
||||
<p className={`evidence-state evidence-${exchange.answer.evidence_state}`}>
|
||||
{EVIDENCE_LABEL[exchange.answer.evidence_state]}
|
||||
</p>
|
||||
|
||||
{exchange.answer.evidence_state === "unavailable" ? (
|
||||
<p>
|
||||
The knowledge service is currently unreachable. Operational features are
|
||||
unaffected — try again later.
|
||||
</p>
|
||||
) : (
|
||||
<p>{exchange.answer.answer}</p>
|
||||
)}
|
||||
|
||||
{exchange.answer.sources.length > 0 && (
|
||||
<ul className="source-cards">
|
||||
{exchange.answer.sources.map((source) => (
|
||||
<li key={`${source.document_id}-${source.section}`} className="source-card">
|
||||
<p className="source-title">
|
||||
{source.title} <span className="source-version">v{source.version}</span>
|
||||
</p>
|
||||
<p className="source-section">{source.section}</p>
|
||||
<p className="source-excerpt">{source.excerpt}</p>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -226,6 +226,33 @@ a { color: #1f5c8f; }
|
||||
}
|
||||
.data-table td button:disabled { opacity: 0.6; cursor: not-allowed; }
|
||||
|
||||
.knowledge-status { color: #607084; font-size: 0.9rem; margin-bottom: 16px; }
|
||||
.knowledge-form { margin-bottom: 20px; }
|
||||
.knowledge-input-row { display: flex; gap: 10px; flex-wrap: wrap; }
|
||||
.knowledge-input-row input {
|
||||
flex: 1; min-width: 240px; padding: 10px 12px; border: 1px solid #cfd8e2;
|
||||
border-radius: 8px; font-size: 0.95rem;
|
||||
}
|
||||
.knowledge-input-row button {
|
||||
padding: 10px 20px; border-radius: 8px; border: none;
|
||||
background: #14324f; color: white; font-weight: 700; cursor: pointer;
|
||||
}
|
||||
.knowledge-input-row button:disabled { opacity: 0.6; cursor: not-allowed; }
|
||||
|
||||
.exchange-list { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column; gap: 16px; }
|
||||
.exchange-question { font-size: 1rem; margin: 0 0 6px; }
|
||||
.evidence-state { display: inline-block; margin: 0 0 10px; padding: 3px 10px; border-radius: 999px; font-weight: 700; font-size: 0.8rem; }
|
||||
.evidence-grounded { background: #e6f5ec; color: #1f6d3d; }
|
||||
.evidence-insufficient { background: #fdf1de; color: #8a5a10; }
|
||||
.evidence-unavailable { background: #fbe6e6; color: #8f2323; }
|
||||
|
||||
.source-cards { list-style: none; margin: 12px 0 0; padding: 0; display: grid; gap: 10px; grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); }
|
||||
.source-card { background: #f6f8fb; border: 1px solid #dce3eb; border-radius: 10px; padding: 12px; }
|
||||
.source-title { margin: 0; font-weight: 700; }
|
||||
.source-version { color: #607084; font-weight: 400; font-size: 0.85rem; }
|
||||
.source-section { margin: 2px 0 6px; color: #375065; font-size: 0.85rem; font-weight: 600; }
|
||||
.source-excerpt { margin: 0; font-size: 0.88rem; color: #47566b; }
|
||||
|
||||
@media (max-width: 700px) {
|
||||
.app-header { flex-direction: column; align-items: flex-start; }
|
||||
.user-badge { margin-left: 0; }
|
||||
|
||||
Reference in New Issue
Block a user