MobilityOps/artifacts/live-ai-integration/final-summary.md

# Live n8n + RAGcore integration — final evidence

No credential values, tokens, or secrets appear anywhere in this document. Where a
credential or trace ID is referenced, only its name or an opaque reference identifier is
given, never its value.

## Commit

Built on branch `feat/live-n8n-ragcore-integration`, HEAD at commit
`aaa16305354d34f9c1f4d57253d33d9062c38faa` ("docs: record WF2 retry fix and WF4's
n8n-session-expiry blocker"). Run `git log --oneline feat/live-n8n-ragcore-integration`
for the full history of this effort.

## Scope

The brief required treating n8n (`https://n8n.itworx.tech`) as a full third integration
layer alongside RAGcore and MCP Hub, with Fleet Ops keeping exclusive ownership of
business rules, authorization, transactions, audit, and idempotency. Four canonical n8n
workflows were required. The repository (`n8n/workflows/*.json` + `MANIFEST.md` +
`n8n/workflows/check_drift.py`) is the source of truth for cleaned workflow definitions;
the Fleet Ops integration status page (`/automation`) shows real per-workflow operational
evidence, not a config boolean.

## Result summary

| # | Workflow | Status | Live evidence this round |
|---|---|---|---|
| 1 | Fleet Ops — Vehicle Return Orchestration | **Live, hardened** | Timeout+bounded-retry gap found and fixed |
| 2 | Fleet Ops — Scheduled Data Quality Scan | **Live, hardened** | Same gap found and fixed |
| 3 | Fleet Ops — RAGcore Procedure Sync | **Blocked** | Not built — RAGcore rejects credential issuance (see below) |
| 4 | Fleet Ops — Workflow Error Handler | **Live, validated** | Mock + genuine induced-failure test; own hardening incomplete (see below) |

Full per-workflow detail (purpose, trigger, event contract, required credentials, live
workflow ID, checksum) is in `n8n/workflows/MANIFEST.md`, which is the authoritative,
continuously-updated source — this document is a point-in-time summary of that state
plus the reasoning behind what's not done.

## Workflow 1 — Vehicle Return Orchestration

Live workflow ID `mobilityops-return-processing`. Validated in an earlier round of this
effort: webhook trigger requires Header Auth (`Fleet Ops Webhook Trigger Token`),
validates `event_type == vehicle.returned.v1`, derives a follow-up category, calls Fleet
Ops's `/return-callback` endpoint with an `Idempotency-Key` header via a named
`Fleet Ops Service Token` credential (not a literal secret), and responds with a
controlled JSON result. Idempotent on both sides (`event_id` flows through as the
dedup key; the backend independently checks for a prior audit event before recording
again).

**This round's finding**: the `Record follow-up` HTTP node had no explicit timeout and
"Retry On Fail" disabled — a real gap against the requirement that external dependencies
have timeouts and bounded retries. Fixed live: Retry On Fail (3 tries, 1000ms wait) + a
15000ms timeout, published. Safe to retry because the callback is idempotent. Repo
definition and manifest checksum synced (commit `0562893`).

Attached to workflow 4 as its Error Workflow.

## Workflow 2 — Scheduled Data Quality Scan

Live workflow ID `mobilityops-scheduled-quality-scan`. Validated earlier: hourly
Schedule Trigger + a Manual Trigger for on-demand testing, both feeding a single HTTP
call to Fleet Ops's `/scheduled-scan` endpoint (Header Auth via the same `Fleet Ops
Service Token` credential, 15000ms timeout already configured), which runs the
domain-level `run_scan()` function — documented and tested as idempotent by
construction (only ever creates an issue for a condition that doesn't already have one
open), so overlapping or retried triggers do no duplicate domain work.

**This round's finding**: the same Retry On Fail gap as workflow 1 (timeout was already
set, retries were not). Fixed live the same way (3 tries, 1000ms wait), published. Repo
definition and manifest checksum synced (commit `167bf49`).

Attached to workflow 4 as its Error Workflow.

## Workflow 3 — RAGcore Procedure Sync — blocked

**Not built.** This workflow needs an application credential (scope `sources:sync`) for
the `fleet-ops` application in RAGcore. Two independent issuance attempts, in two
separate rounds of this effort, both failed with an opaque server-side rejection:

1. **Raw API**: `POST /v1/applications/{id}/credentials` → `400`, "authoritative
   service-account state rejected issuance".
2. **RAGcore admin UI**, this round, after the project owner explicitly authorized
   Claude to self-issue the credential: the "Issue credential" form for the `fleet-ops`
   application, submitted as the Platform Admin role (the highest role visible in the
   RAGcore admin), with name `n8n-ragcore-procedure-sync` and scope `sources:sync` only.
   Result: "Something went wrong. The credential could not be issued with those
   values.", trace reference `1955c6a8968c4941a22a1faef39e17a7`.

The `fleet-ops` application itself shows as ordinary/`Active` in the RAGcore admin, with
no visible lock flag, and RAGcore's own OpenAPI spec documents no validation rule that
would explain either rejection (no `422`, no field-level errors). Two independent paths
— a raw API call and the admin UI as the top admin role — hitting the same failure
signature is conclusive evidence this is a RAGcore-side policy or bug, not a Fleet Ops
request-shape or permission problem. It is not fixable from the Fleet Ops side or
through further UI automation. Resolving it requires whoever operates the RAGcore
instance to look up the trace ID above (and the earlier raw-API rejection) in RAGcore's
own logs.

The real RAGcore contract this workflow will be built against — once a working
credential exists — was independently inspected via RAGcore's live OpenAPI spec and is
recorded in `docs/live-ai-integration/n8n-current-state.md` and
`contracts/ragcore-contract-assumptions.md`: control-plane endpoints require an
`Idempotency-Key` header; ingestion is `POST /v1/uploads`; retrieval is `POST
/v1/search` / `/v1/context` / `/v1/answers` (the latter requiring `requested_space_ids`,
an array of knowledge-space UUIDs); health is `/health/live` and `/health/ready` (not
`/health`); the scope enum is `search, context, answer, documents:read, citations:read,
feedback:write, sources:sync`.

**`RAGcoreKnowledgeProvider` adapter** (`backend/app/services/knowledge/ragcore.py`)
still targets the earlier speculative contract (`/health`, `POST /api/v1/ask`, Bearer
token) rather than the real one above. This was deliberately **not** rewritten this
round: rewriting it blind, without a credential to validate against, risks introducing
a silent behavioral bug in exactly the code path responsible for the project's "AI must
never invent an answer when RAGcore is unavailable or returns insufficient evidence"
guarantee — for example a wrong `evidence_state` mapping that looks fine in code review
but misclassifies "unavailable" as "insufficient" (or vice versa) against the real
response shape. The adapter's current behavior is honest and safe (it degrades cleanly
to `unavailable` on any request or parsing failure, and `ragcore_api_token` is unset by
default so the app correctly runs on the local demo knowledge provider today). The
rewrite stays queued behind the same credential blocker as workflow 3.

## Workflow 4 — Workflow Error Handler

Live workflow ID `Xppn2rAEqUuyiCJF`. Built and live-validated in an earlier round:
Error Trigger → a Code node that derives a bounded, secret-free failure report (error
category classified from the message text, truncated summary, no stack trace, no
headers or tokens) → an HTTP call to Fleet Ops's `/workflow-error` endpoint (Header Auth
via the same `Fleet Ops Service Token` credential), which registers the failure as an
audit event idempotently keyed on `execution_id`.

Set as the Error Workflow on both workflow 1 and workflow 2. Confirmed workflow 4 has no
Error Workflow of its own (prevents a recursive loop).

**Live validation performed**: a pinned mock Error Trigger payload produced a real `200
{"status":"registered", ...}` from the live Fleet Ops server; re-running the identical
payload produced `"status":"already_registered"`, confirming idempotency. A genuine
induced failure (temporarily pointing workflow 2's HTTP node at a nonexistent path, then
reverting) confirmed workflow 2 itself fails correctly against a broken endpoint and
recovers cleanly once reverted.

**Known limitation**: n8n's Error Workflow trigger does not fire for manual editor
"Execute workflow" test runs — checked via workflow 4's own Executions list after the
induced workflow-2 failure, and confirmed no new execution appeared. n8n only invokes a
workflow's assigned Error Workflow for unattended/production trigger executions, not
manual test runs from the editor. The mock-data path exercises the same nodes, logic,
and real Fleet Ops endpoint, but a fully automatic (schedule- or webhook-triggered)
failure cascading into workflow 4 was not observed live in either round.

**Open follow-up (minor, non-blocking)**: continuing this round's acceptance pass to
workflow 4 found the same timeout/retry gap as workflows 1 and 2 on its own outbound
HTTP call. A fix was started (15000ms timeout added, Retry On Fail toggled on) but n8n's
autosave began failing with "Unauthorized" mid-edit; a fresh browser tab confirmed the
n8n session had expired (redirected to `/signin`). Nothing was saved — workflow 4's live
definition is unchanged from before this round, so there is no partial or broken state.
This is lower-stakes than workflows 1/2 (workflow 4 is the error notifier itself, not a
primary business flow, and a failed error-report is already visible in n8n's own
execution history via `On Error: Stop Workflow`) but should be finished once the n8n
browser session is re-authenticated.

## Repository source of truth

`n8n/workflows/` holds cleaned, credential-value-free JSON definitions for all built
workflows, `n8n/workflows/MANIFEST.md` documents purpose/trigger/contract/credentials/
live-ID/checksum for all four canonical workflows (including workflow 3's blocked
status), and `n8n/workflows/check_drift.py` is a read-only script that compares the
repo definitions against the live instance via n8n's Public API and reports drift —
safe to run in CI as a non-blocking check. No literal export/download mechanism was
found working in this n8n version, so each definition was reconstructed from direct,
verified UI inspection rather than a native export; this limitation is noted in the
manifest itself.

## Integration status page

`/automation` (Operations Manager only) surfaces real per-workflow evidence derived
purely from Fleet Ops's own audit/outbox tables — no new dependency on n8n's API was
added to the backend. Each of the four canonical workflows shows a status (not built /
no evidence yet / operational) and a last-evidence timestamp; the scheduled-scan
evidence specifically filters to `actor_type == "service"` so a manually-triggered scan
in the UI doesn't count as n8n evidence. An error-handler summary line reports total
registered automation failures and the most recent one.

Verified live in the browser (Dutch locale) both locally and on the deployed
production server (`http://192.168.10.150:1236/automation`): correctly showed "3 van 4
canonieke n8n-workflows hebben actuele evidentie van werking" with real timestamps for
the return/scan/error-handler workflows, "Nog Niet Gebouwd" for the RAGcore sync, and
the real error-handler registration from this effort's live testing.

## Deployments performed (all explicitly user-approved)

1. Backend `/workflow-error` endpoint (commit `bbdb4a9`) — deployed and verified
   (`/health` OK, new endpoint returns `422` not `404` on an empty POST body).
2. Integration status page, backend + frontend (commit `4049c0c`) — deployed and
   verified (`/health` OK, page renders real evidence in the browser).

The three n8n-side node edits this round (WF1 timeout/retry, WF2 timeout/retry, WF4's
incomplete attempt) are live edits to the n8n instance itself and do not require a
Fleet Ops redeploy.

## What's left

1. **RAGcore credential issuance** — blocked on RAGcore's own server-side rejection
   (trace `1955c6a8968c4941a22a1faef39e17a7` and the earlier raw-API `400`). Needs
   RAGcore's operator to investigate. Unblocks workflow 3 and the
   `RAGcoreKnowledgeProvider` real-contract rewrite.
2. **Workflow 4's own timeout/bounded-retry hardening** — needs the n8n browser session
   re-authenticated to finish; a small, well-understood, non-blocking edit.
3. **Fleet Ops logo/favicon** — explicitly deferred by the project owner as a separate,
   unrelated follow-up task, not part of this integration effort.