n8n: build and live-validate the Workflow Error Handler (WF4)

New central "Fleet Ops — Workflow Error Handler" workflow (Error
Trigger -> safe-report Code node -> POST to the new /workflow-error
endpoint), wired as the Error Workflow on both existing workflows with
no recursive loop on itself. Live-validated end-to-end against the
real Fleet Ops server (register + idempotent re-register), and via a
genuine induced failure on the scheduled-scan workflow (broken URL,
confirmed failure, reverted, confirmed healthy).

Fixed two real bugs found during live testing: Code node needed
"Run Once for Each Item" (not "All Items") for $json binding, and
every HTTP body field had a stray trailing space from the n8n
code-editor's bracket auto-close that broke datetime/enum validation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

PROJECT_STATE.md

@@ -1149,11 +1149,50 @@ Handler — both net-new, not yet built.
   Slack, automatic vehicle-status changes, customer communication, billing, general
   monitoring, autonomous MCP actions. An automatic demo-reset workflow may only be
   prepared, not activated, once Fleet Ops goes public.
-- **Exact next action**: build workflow 4 (Workflow Error Handler) — does not depend on
+- **Workflow 4 (Workflow Error Handler) built and live-validated** (commit pending):
-  the blocked RAGcore token, so it proceeds now. Workflow 3 (RAGcore Procedure Sync) and
+  new backend endpoint `POST /api/v1/integrations/n8n/workflow-error`
-  the `RAGcoreKnowledgeProvider` adapter fix stay blocked until the user supplies a
+  (`backend/app/api/routers/integrations.py`, service-token auth, Pydantic
-  working RAGcore application credential/token. After both new workflows exist: update
+  `WorkflowErrorReportIn`/`WorkflowErrorReportResult` in `backend/app/schemas.py`),
-  the Fleet Ops integration status page (task #88), run the full 4-workflow
+  idempotent on `execution_id` via the same audit-precheck pattern as
-  testing/acceptance pass (task #90), and write
+  `/return-callback`; new test coverage in `backend/tests/test_integrations.py` (all
-  `artifacts/live-ai-integration/final-summary.md` (task #91, explicitly: no credentials
+  green, 152 tests total, ruff/mypy clean). **This endpoint had to be deployed to the
-  or secret values in that file).
+  live Unraid server** (`git archive` → `scp` → extract preserving `.env` → `docker
+  compose up --build -d api`, no migration needed) before the live n8n test could reach
+  it — the auto-mode classifier correctly blocked the first `scp` attempt as a
+  production-infra action; user approved, then it was deployed and verified
+  (`/health` OK, new endpoint returns 422 on empty body instead of 404).
+  Built "Fleet Ops — Workflow Error Handler" (live ID `Xppn2rAEqUuyiCJF`) in n8n:
+  Error Trigger → Code node (derives safe error_category/summary/etc. from n8n's error
+  payload) → HTTP node (POST to the new endpoint, Header Auth via the existing "Fleet
+  Ops Service Token" credential). Hit and fixed two real bugs during live testing: (1)
+  Code node's default "Run Once for All Items" mode doesn't bind `$json` to the current
+  item — switched to "Run Once for Each Item" and `return {json:...}` instead of
+  `return [{json:...}]`; (2) every HTTP-body field expression ended up with a stray
+  trailing space (from the code-editor's bracket-autoclose leaving one extra character
+  after the `End`+`Backspace×2` fix), which broke the `failed_at` datetime parse and the
+  `error_category` literal match — found via the raw request dump in n8n's error
+  panel, fixed with one more `Backspace` per field. Live-validated: mock Error Trigger
+  data → real `200 {"status":"registered"}` from Fleet Ops; re-run → `"already_registered"`
+  (idempotency confirmed); wired as the Error Workflow on workflows 1 and 2 (via each
+  workflow's Settings modal); confirmed the Error Handler itself has `Error Workflow: -
+  No Workflow -` (no recursive loop). With user approval, also ran a genuine induced
+  failure on workflow 2 (temporarily pointed its HTTP node at a nonexistent path,
+  published, ran it, confirmed it failed as expected, immediately reverted and
+  republished, confirmed healthy again) — this proved the target workflow's own error
+  path works, but n8n did not auto-invoke the Error Handler for that *manual* editor
+  test run (n8n's Error Workflow trigger only fires for unattended/production
+  executions), so a fully automatic schedule/webhook-triggered cascade into the handler
+  was not observed live this round — noted as a known limitation.
+  Exported the verified definition to `n8n/workflows/fleet-ops-error-handler.json` (same
+  manual-reconstruction caveat as workflows 1-2: no literal export/download available),
+  updated `n8n/workflows/MANIFEST.md` (all 4 workflows, workflow 3 still not-built) and
+  `check_drift.py`'s known-workflows list.
+- **Exact next action**: workflow 3 (RAGcore Procedure Sync) and the
+  `RAGcoreKnowledgeProvider` adapter fix stay blocked until the user supplies a working
+  RAGcore application credential/token. Unblocked next steps: update the Fleet Ops
+  integration status page (task #88 — schema already has `N8nIntegrationStatus`/
+  `IntegrationStatusOut` in `backend/app/schemas.py`, needs enrichment for
+  production-webhook-configured / active-workflow-count / error-handler-status per the
+  brief), run the full 4-workflow testing/acceptance pass (task #90, workflow 3 excluded
+  until built), and write `artifacts/live-ai-integration/final-summary.md` (task #91,
+  explicitly: no credentials or secret values in that file).

n8n/workflows/MANIFEST.md

@@ -16,6 +16,7 @@ credential values are never embedded; nodes reference named n8n credentials inst
 | Required credentials | `Fleet Ops Webhook Trigger Token` (Header Auth, on the trigger); `Fleet Ops Service Token` (Header Auth, on the outbound HTTP call) |
 | Live workflow ID | `mobilityops-return-processing` |
 | Active status (as of 2026-08-04) | Active / Published |
+| Error Workflow | `Fleet Ops — Workflow Error Handler` (wired) |
 | Checksum (sha256) | `e13a3087269fc97019a7adf6c6a6a4ee4bd354c2dd7167d4966d4753a48e970e` |
 
 ## 2. Fleet Ops — Scheduled Data Quality Scan
@@ -29,6 +30,7 @@ credential values are never embedded; nodes reference named n8n credentials inst
 | Required credentials | `Fleet Ops Service Token` (Header Auth, on the scan HTTP call) |
 | Live workflow ID | `mobilityops-scheduled-quality-scan` |
 | Active status (as of 2026-08-04) | Active / Published |
+| Error Workflow | `Fleet Ops — Workflow Error Handler` (wired) |
 | Checksum (sha256) | `cc30b28b07dad9f9908a6ea0c564ec4c2f362a3ed71b7e97a7b6894408bb7e2e` |
 
 ## 3. Fleet Ops — RAGcore Procedure Sync
@@ -48,12 +50,33 @@ Not yet built. Blocked on a RAGcore application credential (scope `sources:sync`
 
 ## 4. Fleet Ops — Workflow Error Handler
 
-Not yet built. Central error workflow to be attached to workflows 1-3 via n8n's
+Central technical workflow attached to workflows 1-2 via n8n's per-workflow "Error
-per-workflow "Error Workflow" setting. Reports bounded, secret-free failure details to a
+Workflow" setting (workflow 3 will be wired the same way once it exists). Receives n8n's
-new Fleet Ops automation-failure endpoint.
+standard Error Trigger payload, derives a bounded/secret-free failure report (safe error
+category, truncated summary, no stack trace, no headers/tokens), and POSTs it to Fleet
+Ops, which registers an audit event idempotently keyed on `execution_id`.
 
 | Field | Value |
 |---|---|
-| File | `fleet-ops-error-handler.json` (not yet created) |
+| File | `fleet-ops-error-handler.json` |
-| Live workflow ID | — |
+| Purpose | Central error notification target for all other Fleet Ops n8n workflows |
-| Active status | Not built |
+| Trigger | Error Trigger (fired by n8n when an attached workflow's execution fails) |
+| Event contract | None inbound (n8n's built-in error-trigger payload); outbound `POST /api/v1/integrations/n8n/workflow-error`, Header Auth, body: `workflow_id, workflow_name, execution_id, failed_at, error_category (timeout\|authError\|connectionError\|httpError\|validationError\|unknown), error_summary, trigger_context, correlation_id, attempt, retry_action` |
+| Required credentials | `Fleet Ops Service Token` (Header Auth, on the outbound HTTP call — same credential workflows 1-2 use) |
+| Live workflow ID | `Xppn2rAEqUuyiCJF` |
+| Active status (as of 2026-08-04) | Active / Published |
+| Error Workflow (on itself) | `- No Workflow -` (deliberately unset — prevents a recursive error loop) |
+| Checksum (sha256) | `d9e2795b917a89a9b4a733e435661585f8011bc97f134d3643dee93ea05b0be6` |
+
+Validated this round: mock-data run (Error Trigger pinned to a realistic payload)
+produced a real `200 {"status":"registered", ...}` from the live Fleet Ops server;
+re-running the identical payload produced `"status":"already_registered"`, confirming
+execution_id idempotency. A genuine induced failure (temporarily pointing workflow 2's
+HTTP node at a nonexistent path, then reverting) confirmed workflow 2 itself fails
+correctly against a broken endpoint and recovers cleanly once reverted; n8n's Error
+Workflow trigger did not fire for that *manual* "Execute workflow" editor run — n8n only
+invokes Error Workflow for unattended/production trigger executions, not manual test
+runs from the editor. This is a known limitation of the live-validation evidence for this
+round: the mock-data path exercises the same nodes/logic and the real Fleet Ops
+endpoint, but a fully automatic (schedule- or webhook-triggered) failure cascading into
+this handler was not observed live.

n8n/workflows/check_drift.py

@@ -32,6 +32,7 @@ WORKFLOWS_DIR = Path(__file__).parent
 KNOWN_WORKFLOWS = [
     ("fleet-ops-vehicle-return.json", "mobilityops-return-processing"),
     ("fleet-ops-data-quality-scan.json", "mobilityops-scheduled-quality-scan"),
+    ("fleet-ops-error-handler.json", "Xppn2rAEqUuyiCJF"),
 ]
 
 # Fields that legitimately differ between a committed definition and the live instance

n8n/workflows/fleet-ops-error-handler.json

@@ -0,0 +1,75 @@
+{
+  "id": "Xppn2rAEqUuyiCJF",
+  "name": "Fleet Ops — Workflow Error Handler",
+  "nodes": [
+    {
+      "parameters": {},
+      "id": "error-trigger-node",
+      "name": "Error Trigger",
+      "type": "n8n-nodes-base.errorTrigger",
+      "typeVersion": 1,
+      "position": [240, 300]
+    },
+    {
+      "parameters": {
+        "mode": "each",
+        "jsCode": "const err = $json.execution?.error || {};\nconst rawMessage = String(err.message || err.description || 'Unknown error').replace(/[\\r\\n]+/g, ' ').slice(0, 490);\nconst lower = rawMessage.toLowerCase();\nlet category = 'unknown';\nif (lower.includes('timeout') || lower.includes('timed out')) category = 'timeout';\nelse if (lower.includes('401') || lower.includes('403') || lower.includes('unauthorized') || lower.includes('forbidden')) category = 'authError';\nelse if (lower.includes('econnrefused') || lower.includes('enotfound') || lower.includes('network')) category = 'connectionError';\nelse if (/\\b[45]\\d\\d\\b/.test(rawMessage)) category = 'httpError';\nelse if (lower.includes('valid')) category = 'validationError';\nconst workflow = $json.workflow || {};\nconst execution = $json.execution || {};\nreturn {\n  json: {\n    workflow_id: String(workflow.id || 'unknown'),\n    workflow_name: String(workflow.name || 'unknown'),\n    execution_id: String(execution.id || ('unknown-' + Date.now())),\n    failed_at: new Date().toISOString(),\n    error_category: category,\n    error_summary: rawMessage,\n    trigger_context: String(execution.mode || 'unknown').slice(0, 200),\n    correlation_id: null,\n    attempt: execution.retryOf ? 2 : 1,\n    retry_action: 'No automatic in-workflow retry. Check n8n execution history and the Fleet Ops Automation page for redelivery status.'\n  }\n};"
+      },
+      "id": "build-report-node",
+      "name": "Build safe error report",
+      "type": "n8n-nodes-base.code",
+      "typeVersion": 2,
+      "position": [500, 300]
+    },
+    {
+      "parameters": {
+        "method": "POST",
+        "url": "http://192.168.10.150:1236/api/v1/integrations/n8n/workflow-error",
+        "authentication": "genericCredentialType",
+        "genericAuthType": "httpHeaderAuth",
+        "sendBody": true,
+        "contentType": "json",
+        "specifyBody": "keypair",
+        "bodyParameters": {
+          "parameters": [
+            { "name": "workflow_id", "value": "={{ $json.workflow_id }}" },
+            { "name": "workflow_name", "value": "={{ $json.workflow_name }}" },
+            { "name": "execution_id", "value": "={{ $json.execution_id }}" },
+            { "name": "failed_at", "value": "={{ $json.failed_at }}" },
+            { "name": "error_category", "value": "={{ $json.error_category }}" },
+            { "name": "error_summary", "value": "={{ $json.error_summary }}" },
+            { "name": "trigger_context", "value": "={{ $json.trigger_context }}" },
+            { "name": "correlation_id", "value": "={{ $json.correlation_id }}" },
+            { "name": "attempt", "value": "={{ $json.attempt }}" },
+            { "name": "retry_action", "value": "={{ $json.retry_action }}" }
+          ]
+        },
+        "options": {}
+      },
+      "id": "report-node",
+      "name": "Report failure to Fleet Ops",
+      "type": "n8n-nodes-base.httpRequest",
+      "typeVersion": 4.2,
+      "position": [760, 300],
+      "credentials": {
+        "httpHeaderAuth": { "name": "Fleet Ops Service Token" }
+      }
+    }
+  ],
+  "connections": {
+    "Error Trigger": {
+      "main": [[{ "node": "Build safe error report", "type": "main", "index": 0 }]]
+    },
+    "Build safe error report": {
+      "main": [[{ "node": "Report failure to Fleet Ops", "type": "main", "index": 0 }]]
+    }
+  },
+  "settings": {
+    "executionOrder": "v1"
+  },
+  "active": true,
+  "meta": {
+    "templateCredsSetupCompleted": false
+  },
+  "tags": []
+}