# Current-state audit — Fleet Ops final integrations

Date: 2026-08-05. Compiled from direct repository inspection (git log/status/diff across
all three repos), `PROJECT_STATE.md` history, and read-only investigation of the sibling
repos' own state docs. No live server SSH/curl evidence is included in this pass yet —
see `integration-release-state.md` for the live-verification checklist as it is executed.

## Repository revisions at audit time

| Repo | Path | Branch | HEAD | Notes |
|---|---|---|---|---|
| Fleet Ops (MobilityOps) | `C:\Projects\MobilityOps` | `feat/fleet-ops-final-integrations` (new, branched from `feat/live-n8n-ragcore-integration`) | `3ebca9e` | `feat/live-n8n-ragcore-integration` was pushed to `origin` at `0571a40` and deployed live; `3ebca9e` (logo rebrand) is one commit ahead, not yet deployed. `master` is 19 commits behind and stale (localization-round only). |
| RAGcore | `C:\Projects\RAGcore` | `main` | `64a908a` | Up to date with `origin/main`. Uncommitted local work in progress (see below) — not Fleet-Ops-related, left untouched. |
| ITWorx MCP Hub | `C:\Projects\ITWorx_MCP_Hub` | `feature/wp240-final-acceptance` | `26e6bd8` (+ later `75bb16a`) | Contains `f107544` (MobilityOps connector) as a direct ancestor, plus a real contract fix (`96de385`, vehicleRef camelCase). Already deployed live to Tower at `c4a0f6d`. |

## Branch-name correction (recorded assumption)

The task brief names the working branch `feat/fleet-ops-final-integrations` as already
selected and "branched from the most recently validated, localized, deployed master
branch." That literal branch did not exist. `master` is in fact stale (19 commits behind,
last touched for a localization round only) — the actually-validated, deployed line of
work is `feat/live-n8n-ragcore-integration` (pushed to `origin`, deployed to
`http://192.168.10.150:1236` at `0571a40`, one commit behind current HEAD). Created
`feat/fleet-ops-final-integrations` from that branch's HEAD (`3ebca9e`) instead of from
`master`, since that satisfies the actual intent (continue from the validated/deployed
line) even though the literal branch name in the brief was inaccurate.

## What is actually already done (contradicts "not yet live" framing in places)

- **n8n**: 3 of 4 canonical workflows are live and active in the shared instance
  (`n8n.itworx.tech`): Vehicle Return Orchestration, Scheduled Data Quality Scan,
  Workflow Error Handler. The 4th, RAGcore Procedure Sync, has all 6 nodes built and
  saved but is **not published** (deliberately left for an explicit activation decision,
  since publishing starts real unattended daily runs against production). The root cause
  of an earlier "auth"-looking failure (`N8N_PROXY_HOPS=0` behind the TLS-terminating
  reverse proxy, breaking the browserId CSRF check on every mutating REST call) was found
  and fixed at the infrastructure level (Unraid template), not worked around.
- **RAGcore**: deployed to `http://192.168.10.150:1237`, application wiring for
  search/context/answer is real (commit `a2905cc`, confirmed present in RAGcore's own
  history at `13 commits behind HEAD`). `KNOWLEDGE_PROVIDER` is still `demo` in Fleet Ops
  because real queries against the "Fleet Ops Procedures" space return **zero dense and
  zero sparse candidates** at the raw retrieval stage — confirmed not a Fleet-Ops-side
  wiring bug (RAGcore's own trusted Query Lab tool reproduces the identical zero-candidate
  result against the same space). Root cause not yet found as of this audit; ruled out so
  far: point count/scoping (83 published, correctly scoped), embedding digest mismatch
  (matches), collection alias resolution (resolves correctly). One separate, confirmed,
  pre-existing bug: `_DEFAULT_LANGUAGE = "en"` is hardcoded in RAGcore's ingestion handler
  — every chunk is stamped `language: "en"` regardless of actual content; RAGcore has never
  done real language detection. Not the cause of zero candidates, but must be fixed for
  trilingual retrieval (task 6A) once the space is answerable at all.
- **MCP Hub**: the Fleet Ops read-only connector (4 tools, `mobilityops.*`) is **already
  live in production** on Tower (commit `c4a0f6d`), reachable via `fleetops.itworx.tech`,
  end-to-end verified once already per the Hub's own `CLAUDE.md`/`BUILD_STATE.json`. A
  real contract bug was found and fixed there (`vehicle.get`'s input schema disagreed with
  the actual wire parameter name — `vehicle_ref` vs `vehicleRef`). What is **not** yet done:
  the Hub's own formal production-acceptance checklist row for MobilityOps (`CON-P04`) has
  not been executed, and Fleet Ops's own `MCP_HUB_REGISTRATION_ENABLED`/base-URL
  configuration has not been confirmed as actually wired and flipped on from the Fleet Ops
  side (open item for this audit's Batch 4).

## Confirmed contradictions to resolve (task section 3)

- "Twee versus vier n8n-workflows": resolved above — 3 active + 1 built-but-unpublished.
  Canonical set is 4; only 3 are live.
- "Demo-provider versus live RAGcore": Fleet Ops is still on the demo knowledge provider
  by deliberate, documented decision (not an oversight) pending the retrieval root cause.
- "MCP Hub-status": prior Fleet Ops docs (`.env.example`, `MCP_HUB_REGISTRATION_ENABLED`)
  predate the Hub-side deployment and need reconciling against the fact that the connector
  is already live on the Hub side.
- Repo hygiene: removed an untracked, empty `backend;C` directory and an untracked 31 MB
  `MobilityOps.zip` stray export; added `*.zip`/`*.tar.gz` to `.gitignore`. No accidentally
  committed `__pycache__`/`.pytest_cache`/`test-results` were found in git history.

## RAGcore retrieval root cause — found and partially fixed (2026-08-05, this session)

Investigated live against production (`192.168.10.150`, containers `ragcore-app-1`,
`ragcore-qdrant-1`, `ragcore-postgres-1`, `ollama`), read-only first, then two approved
live changes.

**Root cause #1 (FIXED): filesystem permission bug, not authorization/data.** Verified,
in order, that every earlier suspect was actually healthy: the `control.grants` row
(active, `editor` role, correct application/space), the real
`ControlPlaneAuthorizationInputsProvider` + `RetrievalAuthorizationService.resolve()` code
path run in-process against the live DB (resolves a non-empty `effective_space_ids`), the
exact production Qdrant filter run directly against the live collection (returns real
matching points), and a real ANN vector query under that filter (real hits, sensible
scores). The actual break: `/workspace/.state/models/embedding_profiles.json` — the file
`RetrievalPipeline.run()` reads on every single query to resolve the active embedding
profile — was owned by container-side `root:root` mode `600` on the bind-mounted
`/mnt/cache/appdata/ragcore/state/models` host path, while the real running app process
is uid 10001 (`ragcore`). Every retrieval call hit a `PermissionError` reading its own
state file before ever reaching Qdrant — a plain filesystem-ownership bug, invisible to
every DB/Qdrant-level check. **Fixed live**: `chown 10001:10001` +
`chmod 644`/`755` on that file/directory (approved by the user beforehand). Re-verified
in-process: `RetrievalPipeline.run()` now returns 5 real, relevant hits for an English
damage-procedure question (previously 0).

**Root cause #2 (found, NOT fixed — needs a design decision): reranking is
architecturally unavailable.** `DEFAULT_RERANKER_PROFILE.model_identifier` is
`bge-reranker-v2-m3:v1`, which was never actually present in Ollama's model list (0 of 14
installed models matched). With the user's approval, pulled a working GGUF
(`xitao/bge-reranker-v2-m3:latest`, 1.2 GB) into the shared Ollama instance. **This did
not fix reranking**: `OllamaRerankAdapter` posts to `{ollama_base_url}/api/rerank`, and
this Ollama server (version `0.32.5`) returns a plain `404` for that route — it has no
rerank endpoint at all. This is not a missing-model problem, it is that RAGcore's
reranker adapter was built against an Ollama HTTP API that does not exist in the deployed
version (matches the code's own comment that no reranker-profile registry or live
validation existed yet). The retrieval pipeline degrades gracefully on rerank failure
(RRF-fusion-only hits still returned, confirmed above), but the `/v1/answers` endpoint's
answerability classifier still returns `not_answerable`/0 citations for real NL/EN/FR
questions against real matching content, live-verified after fix #1 with a freshly
minted, correctly-scoped credential.

Options for #2, not decided yet: (a) find/confirm whether a newer Ollama version adds a
real `/api/rerank` route and upgrade the shared instance (affects every other project on
this Ollama — needs its own explicit approval and blast-radius review); (b) change
RAGcore's reranker adapter to call a route Ollama actually supports (e.g. score via
`/api/embed` + a manual similarity/cross-encoder computation, or drop the separate
rerank step and let the answerability classifier trust RRF-fused scores) — a RAGcore
code/design change, out of Fleet Ops's own mandate to decide unilaterally; (c) leave
`KNOWLEDGE_PROVIDER=demo` until RAGcore's own team/session resolves this.

**Side effect to flag**: minting the live-verification credential used `rotate=True` on
the existing "Fleet Ops Knowledge Assistant (production)" service account (a second
credential would have exceeded RAGcore's own 2-active-credential cap), which invalidates
whatever token was previously issued for that account. Since Fleet Ops is still on
`KNOWLEDGE_PROVIDER=demo`, this has no live user-facing impact today, but a fresh
credential must be issued and wired into Fleet Ops's `RAGCORE_API_TOKEN` at actual
cutover time — do not assume the old one still works.

**Concurrency note**: `C:\Projects\RAGcore` had substantial uncommitted local changes
from what appears to be a different, actively-running session (36 modified/untracked
files by the end of this investigation, including files this investigation also read).
No commits or file edits were made in that checkout this session precisely because of
that collision risk — the two live fixes above were applied directly to the running
containers/Ollama instance (approved), not to the RAGcore git repository. **Follow-up
required**: once the concurrent session's work lands, the reranker-profile fix (whichever
option above is chosen) still needs an actual code change + commit + redeploy in
`C:\Projects\RAGcore`, which was not safe to do mid-collision this session.

## Minimal remaining implementation order

1. Root-cause the RAGcore zero-candidate retrieval bug (blocks flipping `KNOWLEDGE_PROVIDER`
   and blocks the trilingual live-acceptance and AI Operations Brief tasks).
2. Fix RAGcore's hardcoded `language: "en"` chunk metadata for trilingual retrieval.
3. Decide on and execute n8n workflow 3 publication, with live no-op-on-rerun verification.
4. Confirm/complete Fleet Ops-side MCP Hub registration wiring and run the Hub's own
   CON-P04 acceptance row.
5. Build the AI Operations Brief runbook once RAGcore and MCP Hub are both live-green.
6. GUI polish batch (dashboard Today/Attention presentation, duplicate-merge presentation,
   About Demo scannability, Demo Guide completion state).
7. Final regression gates and evidence write-up.