diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index 5c53fb8..8053e89 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -1149,11 +1149,50 @@ Handler — both net-new, not yet built. Slack, automatic vehicle-status changes, customer communication, billing, general monitoring, autonomous MCP actions. An automatic demo-reset workflow may only be prepared, not activated, once Fleet Ops goes public. -- **Exact next action**: build workflow 4 (Workflow Error Handler) — does not depend on - the blocked RAGcore token, so it proceeds now. Workflow 3 (RAGcore Procedure Sync) and - the `RAGcoreKnowledgeProvider` adapter fix stay blocked until the user supplies a - working RAGcore application credential/token. After both new workflows exist: update - the Fleet Ops integration status page (task #88), run the full 4-workflow - testing/acceptance pass (task #90), and write - `artifacts/live-ai-integration/final-summary.md` (task #91, explicitly: no credentials - or secret values in that file). +- **Workflow 4 (Workflow Error Handler) built and live-validated** (commit pending): + new backend endpoint `POST /api/v1/integrations/n8n/workflow-error` + (`backend/app/api/routers/integrations.py`, service-token auth, Pydantic + `WorkflowErrorReportIn`/`WorkflowErrorReportResult` in `backend/app/schemas.py`), + idempotent on `execution_id` via the same audit-precheck pattern as + `/return-callback`; new test coverage in `backend/tests/test_integrations.py` (all + green, 152 tests total, ruff/mypy clean). **This endpoint had to be deployed to the + live Unraid server** (`git archive` → `scp` → extract preserving `.env` → `docker + compose up --build -d api`, no migration needed) before the live n8n test could reach + it — the auto-mode classifier correctly blocked the first `scp` attempt as a + production-infra action; user approved, then it was deployed and verified + (`/health` OK, new endpoint returns 422 on empty body instead of 404). + Built "Fleet Ops — Workflow Error Handler" (live ID `Xppn2rAEqUuyiCJF`) in n8n: + Error Trigger → Code node (derives safe error_category/summary/etc. from n8n's error + payload) → HTTP node (POST to the new endpoint, Header Auth via the existing "Fleet + Ops Service Token" credential). Hit and fixed two real bugs during live testing: (1) + Code node's default "Run Once for All Items" mode doesn't bind `$json` to the current + item — switched to "Run Once for Each Item" and `return {json:...}` instead of + `return [{json:...}]`; (2) every HTTP-body field expression ended up with a stray + trailing space (from the code-editor's bracket-autoclose leaving one extra character + after the `End`+`Backspace×2` fix), which broke the `failed_at` datetime parse and the + `error_category` literal match — found via the raw request dump in n8n's error + panel, fixed with one more `Backspace` per field. Live-validated: mock Error Trigger + data → real `200 {"status":"registered"}` from Fleet Ops; re-run → `"already_registered"` + (idempotency confirmed); wired as the Error Workflow on workflows 1 and 2 (via each + workflow's Settings modal); confirmed the Error Handler itself has `Error Workflow: - + No Workflow -` (no recursive loop). With user approval, also ran a genuine induced + failure on workflow 2 (temporarily pointed its HTTP node at a nonexistent path, + published, ran it, confirmed it failed as expected, immediately reverted and + republished, confirmed healthy again) — this proved the target workflow's own error + path works, but n8n did not auto-invoke the Error Handler for that *manual* editor + test run (n8n's Error Workflow trigger only fires for unattended/production + executions), so a fully automatic schedule/webhook-triggered cascade into the handler + was not observed live this round — noted as a known limitation. + Exported the verified definition to `n8n/workflows/fleet-ops-error-handler.json` (same + manual-reconstruction caveat as workflows 1-2: no literal export/download available), + updated `n8n/workflows/MANIFEST.md` (all 4 workflows, workflow 3 still not-built) and + `check_drift.py`'s known-workflows list. +- **Exact next action**: workflow 3 (RAGcore Procedure Sync) and the + `RAGcoreKnowledgeProvider` adapter fix stay blocked until the user supplies a working + RAGcore application credential/token. Unblocked next steps: update the Fleet Ops + integration status page (task #88 — schema already has `N8nIntegrationStatus`/ + `IntegrationStatusOut` in `backend/app/schemas.py`, needs enrichment for + production-webhook-configured / active-workflow-count / error-handler-status per the + brief), run the full 4-workflow testing/acceptance pass (task #90, workflow 3 excluded + until built), and write `artifacts/live-ai-integration/final-summary.md` (task #91, + explicitly: no credentials or secret values in that file). diff --git a/n8n/workflows/MANIFEST.md b/n8n/workflows/MANIFEST.md index a23fe24..a7e3061 100644 --- a/n8n/workflows/MANIFEST.md +++ b/n8n/workflows/MANIFEST.md @@ -16,6 +16,7 @@ credential values are never embedded; nodes reference named n8n credentials inst | Required credentials | `Fleet Ops Webhook Trigger Token` (Header Auth, on the trigger); `Fleet Ops Service Token` (Header Auth, on the outbound HTTP call) | | Live workflow ID | `mobilityops-return-processing` | | Active status (as of 2026-08-04) | Active / Published | +| Error Workflow | `Fleet Ops — Workflow Error Handler` (wired) | | Checksum (sha256) | `e13a3087269fc97019a7adf6c6a6a4ee4bd354c2dd7167d4966d4753a48e970e` | ## 2. Fleet Ops — Scheduled Data Quality Scan @@ -29,6 +30,7 @@ credential values are never embedded; nodes reference named n8n credentials inst | Required credentials | `Fleet Ops Service Token` (Header Auth, on the scan HTTP call) | | Live workflow ID | `mobilityops-scheduled-quality-scan` | | Active status (as of 2026-08-04) | Active / Published | +| Error Workflow | `Fleet Ops — Workflow Error Handler` (wired) | | Checksum (sha256) | `cc30b28b07dad9f9908a6ea0c564ec4c2f362a3ed71b7e97a7b6894408bb7e2e` | ## 3. Fleet Ops — RAGcore Procedure Sync @@ -48,12 +50,33 @@ Not yet built. Blocked on a RAGcore application credential (scope `sources:sync` ## 4. Fleet Ops — Workflow Error Handler -Not yet built. Central error workflow to be attached to workflows 1-3 via n8n's -per-workflow "Error Workflow" setting. Reports bounded, secret-free failure details to a -new Fleet Ops automation-failure endpoint. +Central technical workflow attached to workflows 1-2 via n8n's per-workflow "Error +Workflow" setting (workflow 3 will be wired the same way once it exists). Receives n8n's +standard Error Trigger payload, derives a bounded/secret-free failure report (safe error +category, truncated summary, no stack trace, no headers/tokens), and POSTs it to Fleet +Ops, which registers an audit event idempotently keyed on `execution_id`. | Field | Value | |---|---| -| File | `fleet-ops-error-handler.json` (not yet created) | -| Live workflow ID | — | -| Active status | Not built | +| File | `fleet-ops-error-handler.json` | +| Purpose | Central error notification target for all other Fleet Ops n8n workflows | +| Trigger | Error Trigger (fired by n8n when an attached workflow's execution fails) | +| Event contract | None inbound (n8n's built-in error-trigger payload); outbound `POST /api/v1/integrations/n8n/workflow-error`, Header Auth, body: `workflow_id, workflow_name, execution_id, failed_at, error_category (timeout\|authError\|connectionError\|httpError\|validationError\|unknown), error_summary, trigger_context, correlation_id, attempt, retry_action` | +| Required credentials | `Fleet Ops Service Token` (Header Auth, on the outbound HTTP call — same credential workflows 1-2 use) | +| Live workflow ID | `Xppn2rAEqUuyiCJF` | +| Active status (as of 2026-08-04) | Active / Published | +| Error Workflow (on itself) | `- No Workflow -` (deliberately unset — prevents a recursive error loop) | +| Checksum (sha256) | `d9e2795b917a89a9b4a733e435661585f8011bc97f134d3643dee93ea05b0be6` | + +Validated this round: mock-data run (Error Trigger pinned to a realistic payload) +produced a real `200 {"status":"registered", ...}` from the live Fleet Ops server; +re-running the identical payload produced `"status":"already_registered"`, confirming +execution_id idempotency. A genuine induced failure (temporarily pointing workflow 2's +HTTP node at a nonexistent path, then reverting) confirmed workflow 2 itself fails +correctly against a broken endpoint and recovers cleanly once reverted; n8n's Error +Workflow trigger did not fire for that *manual* "Execute workflow" editor run — n8n only +invokes Error Workflow for unattended/production trigger executions, not manual test +runs from the editor. This is a known limitation of the live-validation evidence for this +round: the mock-data path exercises the same nodes/logic and the real Fleet Ops +endpoint, but a fully automatic (schedule- or webhook-triggered) failure cascading into +this handler was not observed live. diff --git a/n8n/workflows/check_drift.py b/n8n/workflows/check_drift.py index e389620..9204172 100644 --- a/n8n/workflows/check_drift.py +++ b/n8n/workflows/check_drift.py @@ -32,6 +32,7 @@ WORKFLOWS_DIR = Path(__file__).parent KNOWN_WORKFLOWS = [ ("fleet-ops-vehicle-return.json", "mobilityops-return-processing"), ("fleet-ops-data-quality-scan.json", "mobilityops-scheduled-quality-scan"), + ("fleet-ops-error-handler.json", "Xppn2rAEqUuyiCJF"), ] # Fields that legitimately differ between a committed definition and the live instance diff --git a/n8n/workflows/fleet-ops-error-handler.json b/n8n/workflows/fleet-ops-error-handler.json new file mode 100644 index 0000000..d3bd41b --- /dev/null +++ b/n8n/workflows/fleet-ops-error-handler.json @@ -0,0 +1,75 @@ +{ + "id": "Xppn2rAEqUuyiCJF", + "name": "Fleet Ops — Workflow Error Handler", + "nodes": [ + { + "parameters": {}, + "id": "error-trigger-node", + "name": "Error Trigger", + "type": "n8n-nodes-base.errorTrigger", + "typeVersion": 1, + "position": [240, 300] + }, + { + "parameters": { + "mode": "each", + "jsCode": "const err = $json.execution?.error || {};\nconst rawMessage = String(err.message || err.description || 'Unknown error').replace(/[\\r\\n]+/g, ' ').slice(0, 490);\nconst lower = rawMessage.toLowerCase();\nlet category = 'unknown';\nif (lower.includes('timeout') || lower.includes('timed out')) category = 'timeout';\nelse if (lower.includes('401') || lower.includes('403') || lower.includes('unauthorized') || lower.includes('forbidden')) category = 'authError';\nelse if (lower.includes('econnrefused') || lower.includes('enotfound') || lower.includes('network')) category = 'connectionError';\nelse if (/\\b[45]\\d\\d\\b/.test(rawMessage)) category = 'httpError';\nelse if (lower.includes('valid')) category = 'validationError';\nconst workflow = $json.workflow || {};\nconst execution = $json.execution || {};\nreturn {\n json: {\n workflow_id: String(workflow.id || 'unknown'),\n workflow_name: String(workflow.name || 'unknown'),\n execution_id: String(execution.id || ('unknown-' + Date.now())),\n failed_at: new Date().toISOString(),\n error_category: category,\n error_summary: rawMessage,\n trigger_context: String(execution.mode || 'unknown').slice(0, 200),\n correlation_id: null,\n attempt: execution.retryOf ? 2 : 1,\n retry_action: 'No automatic in-workflow retry. Check n8n execution history and the Fleet Ops Automation page for redelivery status.'\n }\n};" + }, + "id": "build-report-node", + "name": "Build safe error report", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [500, 300] + }, + { + "parameters": { + "method": "POST", + "url": "http://192.168.10.150:1236/api/v1/integrations/n8n/workflow-error", + "authentication": "genericCredentialType", + "genericAuthType": "httpHeaderAuth", + "sendBody": true, + "contentType": "json", + "specifyBody": "keypair", + "bodyParameters": { + "parameters": [ + { "name": "workflow_id", "value": "={{ $json.workflow_id }}" }, + { "name": "workflow_name", "value": "={{ $json.workflow_name }}" }, + { "name": "execution_id", "value": "={{ $json.execution_id }}" }, + { "name": "failed_at", "value": "={{ $json.failed_at }}" }, + { "name": "error_category", "value": "={{ $json.error_category }}" }, + { "name": "error_summary", "value": "={{ $json.error_summary }}" }, + { "name": "trigger_context", "value": "={{ $json.trigger_context }}" }, + { "name": "correlation_id", "value": "={{ $json.correlation_id }}" }, + { "name": "attempt", "value": "={{ $json.attempt }}" }, + { "name": "retry_action", "value": "={{ $json.retry_action }}" } + ] + }, + "options": {} + }, + "id": "report-node", + "name": "Report failure to Fleet Ops", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.2, + "position": [760, 300], + "credentials": { + "httpHeaderAuth": { "name": "Fleet Ops Service Token" } + } + } + ], + "connections": { + "Error Trigger": { + "main": [[{ "node": "Build safe error report", "type": "main", "index": 0 }]] + }, + "Build safe error report": { + "main": [[{ "node": "Report failure to Fleet Ops", "type": "main", "index": 0 }]] + } + }, + "settings": { + "executionOrder": "v1" + }, + "active": true, + "meta": { + "templateCredsSetupCompleted": false + }, + "tags": [] +}