Files

T

NuklearRabbit 6f77a30dce fix: stop the prepared demo failure from degrading n8n integration health

The demo seed plants exactly one failed delivery (BK-H-0020) to demonstrate
retry and audit. Because derive_n8n_status() counted any failure, every fresh
reset pinned the n8n integration to "degraded" -- the demo showed a warning
about a prop, which tells a viewer something untrue about the automation.

The seeded failure now carries its own error code, demoScenarioTimeout, rather
than the generic connectionError a real timeout produces. No column and no
migration: last_error_code already existed, is already surfaced to the UI and is
already localizable.

- integration status splits failed into unexpected_failed and
  demo_scenario_failed; only unexpected failures may move the state. A staged
  failure alone leaves n8n operational.
- latest_failure_at is a health signal and now ignores the staged failure;
  latest_demo_scenario_at reports it separately.
- /api/v1/workflows exposes is_demo_scenario. The Automation page labels the run
  as a prepared demo scenario, explains that it is a simulated temporary failure
  that does not affect automation health, and offers a distinct "retry demo
  scenario" action. Translated in nl-BE, en-GB and fr-BE.
- the carve-out stays narrow: a real failure still degrades n8n, and a genuine
  later failure of the same event overwrites the demo code with the real one.
- the retry itself is unchanged and real: the event goes back on the outbox and
  the dispatcher delivers it to n8n like any other, so 19+1 becomes 20+0 only on
  an actual round trip. The audit records which kind of failure was retried.

Tests that assert on the seeded scenario now reseed first, since earlier test
files legitimately mutate the outbox and the suite shares one database.

Verified locally against a real PostgreSQL 16: 181 passed, ruff clean, mypy
clean (50 files), tsc clean, frontend build clean. Not deployed and not
browser-verified.

2026-08-05 14:07:05 +00:00

5.0 KiB

Raw Blame History

n8n integration

Role

n8n orchestrates secondary cross-system work after MobilityOps commits canonical state. It is not the domain engine.

Required live workflow: return processing

Input: vehicle.returned.v1 webhook event.

Steps:

validate event type and schema;
derive a follow-up category from the already-calculated state;
call the narrow MobilityOps callback endpoint with event ID and follow-up summary;
return a stable workflow result;
on errors, fail visibly so the outbox dispatcher can retry.

The canonical, live-validated definition is n8n/workflows/fleet-ops-vehicle-return.json (see n8n/workflows/MANIFEST.md); it authenticates via named Header Auth credentials rather than a literal token, per the live-hardening pass documented in docs/live-ai-integration/n8n-current-state.md.

Second live workflow: scheduled quality scan

RAGcore is not connected in this environment, so the originally sketched "knowledge sync" workflow below remains deferred (see "Deferred: knowledge sync"). The second implemented workflow does not depend on RAGcore or MCP Hub, so it is not blocked by them.

Input: hourly schedule trigger, or a manual trigger for on-demand testing.

Steps:

call the narrow, service-token-protected POST /api/v1/integrations/n8n/scheduled-scan endpoint;
the endpoint runs the same deterministic run_scan() domain function the manual "Run quality scan" UI action uses, and records a data_quality_scan_run audit event with actor_type=service;
return counts of newly created issues per rule type.

run_scan() only ever creates an issue for a condition that does not already have one open, so a duplicate or overlapping trigger (a manual test run firing close to the scheduled one, or a retried HTTP call) does no duplicate domain work.

The canonical, live-validated definition is n8n/workflows/fleet-ops-data-quality-scan.json (see n8n/workflows/MANIFEST.md), imported and published the same way as the return workflow (see deploy/unraid/setup-scheduled-scan.sh and docs/17-runbook.md). It is active on the live instance; a fresh import ships inactive until credentials are wired up and it is deliberately published.

RAGcore procedure sync (in progress)

RAGcore is now reachable in this environment; a live inspection of its real contract is recorded in docs/live-ai-integration/n8n-current-state.md. Workflow 3, "Fleet Ops — RAGcore Procedure Sync", is being built against that real contract (not the sketch originally in this section) — see n8n/workflows/MANIFEST.md for current status.

Workflow error handler (in progress)

Workflow 4, "Fleet Ops — Workflow Error Handler", is a central technical workflow attached to workflows 1-3 via n8n's per-workflow "Error Workflow" setting, reporting bounded, secret-free failure details to Fleet Ops. See n8n/workflows/MANIFEST.md for status.

Outbox dispatcher

polls pending records;
claims with FOR UPDATE SKIP LOCKED or equivalent;
sends event with timeout;
exponential backoff with a small maximum attempt count;
supports explicit manual retry;
preserves last error and response metadata;
does not hold a database transaction open during network I/O.

Prepared demo failure versus real failure

The demo seed deliberately plants exactly one failed delivery (BK-H-0020, see seed/workflow_runs.csv). It exists to demonstrate retry and audit, so it must never be read as evidence that the automation is unhealthy.

It is distinguished by its last_error_code, demoScenarioTimeout (app.models.outbox.DEMO_SCENARIO_ERROR_CODE) — not by a new column, so no migration is involved. A real timeout produces connectionError; the two are never confused.

Consequences, all enforced by tests:

/api/v1/integrations/status reports failed (everything), unexpected_failed (real failures only) and demo_scenario_failed separately.
Only unexpected_failed can move n8n off operational. A prepared failure alone leaves the integration operational — a staged prop may not raise a red flag.
latest_failure_at is a health signal and therefore ignores the prepared failure; latest_demo_scenario_at reports it separately.
/api/v1/workflows marks the run with is_demo_scenario: true. The Automation page labels it "Prepared demo scenario", explains that it is a simulated temporary failure, and offers a distinct "Retry demo scenario" action.
A genuine later failure of that same event overwrites the code with the real one, and from that moment it counts as a real failure — the carve-out is narrow by construction.

The retry itself is real in both cases: the event goes back on the outbox and the dispatcher delivers it to the configured n8n webhook like any other, so 19 succeeded + 1 failed becomes 20 succeeded + 0 failed only when n8n genuinely accepts the delivery. Nothing is marked succeeded without a real round trip. The audit entry records demo_scenario: true/false so a staged retry is never mistaken for a production fix. A demo reset recreates the original 19 + 1 scenario.

5.0 KiB Raw Blame History