fix: stop the prepared demo failure from degrading n8n integration health
The demo seed plants exactly one failed delivery (BK-H-0020) to demonstrate retry and audit. Because derive_n8n_status() counted any failure, every fresh reset pinned the n8n integration to "degraded" -- the demo showed a warning about a prop, which tells a viewer something untrue about the automation. The seeded failure now carries its own error code, demoScenarioTimeout, rather than the generic connectionError a real timeout produces. No column and no migration: last_error_code already existed, is already surfaced to the UI and is already localizable. - integration status splits failed into unexpected_failed and demo_scenario_failed; only unexpected failures may move the state. A staged failure alone leaves n8n operational. - latest_failure_at is a health signal and now ignores the staged failure; latest_demo_scenario_at reports it separately. - /api/v1/workflows exposes is_demo_scenario. The Automation page labels the run as a prepared demo scenario, explains that it is a simulated temporary failure that does not affect automation health, and offers a distinct "retry demo scenario" action. Translated in nl-BE, en-GB and fr-BE. - the carve-out stays narrow: a real failure still degrades n8n, and a genuine later failure of the same event overwrites the demo code with the real one. - the retry itself is unchanged and real: the event goes back on the outbox and the dispatcher delivers it to n8n like any other, so 19+1 becomes 20+0 only on an actual round trip. The audit records which kind of failure was retried. Tests that assert on the seeded scenario now reseed first, since earlier test files legitimately mutate the outbox and the suite shares one database. Verified locally against a real PostgreSQL 16: 181 passed, ruff clean, mypy clean (50 files), tsc clean, frontend build clean. Not deployed and not browser-verified.
This commit is contained in:
@@ -1862,3 +1862,35 @@ correlation propagation, real upstream readiness) was implemented in the sibling
|
||||
- Local gates were **not** re-run this session: the environment this ran in has no
|
||||
network and no Docker, so `docker compose build api` + `pytest`/`ruff`/`mypy` could not
|
||||
be executed. Run them before deploying.
|
||||
|
||||
## Prepared demo failure separated from real integration health (2026-08-05)
|
||||
|
||||
The demo seed's single staged delivery failure (`BK-H-0020`) pinned the n8n integration
|
||||
to **degraded** on every fresh reset. A viewer therefore saw a red-ish automation badge
|
||||
for a failure that exists on purpose — the demo told an untrue story about itself.
|
||||
|
||||
- The seeded failure now carries its own error code, `demoScenarioTimeout`
|
||||
(`app.models.outbox.DEMO_SCENARIO_ERROR_CODE`), instead of the generic
|
||||
`connectionError` a real timeout produces. No schema change and no migration: the code
|
||||
column already existed, is already surfaced and is already localizable.
|
||||
- `derive_n8n_status()` counts `unexpected_failed` and `demo_scenario_failed` separately
|
||||
and only lets real failures move the state. `latest_failure_at` (a health signal) now
|
||||
ignores the staged failure; `latest_demo_scenario_at` reports it separately.
|
||||
- `/api/v1/workflows` exposes `is_demo_scenario`; the Automation page shows a "Prepared
|
||||
demo scenario" badge, an explanation that it is a simulated temporary failure that does
|
||||
not affect automation health, and a distinct "Retry demo scenario" action. Translated
|
||||
in nl-BE, en-GB and fr-BE; i18n key parity verified against en-GB.
|
||||
- The carve-out is deliberately narrow: a real failure still degrades n8n, proven by
|
||||
`test_a_real_failure_still_degrades_the_integration`. The retry stays a real
|
||||
redelivery through the dispatcher — nothing is marked succeeded without an actual n8n
|
||||
round trip — and the audit records `demo_scenario: true/false`.
|
||||
- Also in this pass (earlier commit `e5307a7`): the demo manifest's MCP Hub summary no
|
||||
longer derives "operational" from `MCP_HUB_REGISTRATION_ENABLED` alone.
|
||||
- **Local gates, actually executed this session** against a real PostgreSQL 16 and a
|
||||
fresh install of the pinned dependencies: `pytest` — **181 passed**; `ruff check .` —
|
||||
clean; `mypy app` — clean (50 files); `tsc --noEmit` — clean; `npm run build` — clean.
|
||||
The suite was made order-independent where it asserts on the seeded scenario
|
||||
(`_reseed()` helpers), since earlier test files legitimately mutate the outbox.
|
||||
- **Not done, and not claimed**: no deployment and no browser verification — the
|
||||
environment this ran in has no network to `192.168.10.150` and no Docker, so the live
|
||||
Unraid instance still runs the previous revision. Playwright e2e was not re-run.
|
||||
|
||||
@@ -8,7 +8,7 @@ from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.deps import get_db, require_operations_manager
|
||||
from app.core.errors import AppError
|
||||
from app.models.outbox import OutboxEvent
|
||||
from app.models.outbox import OutboxEvent, is_demo_scenario_failure
|
||||
from app.schemas import AutomationRunOut, CurrentUser
|
||||
from app.services.audit import record_audit_event
|
||||
|
||||
@@ -24,6 +24,7 @@ def _to_out(event: OutboxEvent) -> AutomationRunOut:
|
||||
attempts=event.attempts,
|
||||
last_error=event.last_error,
|
||||
last_error_code=event.last_error_code,
|
||||
is_demo_scenario=is_demo_scenario_failure(event),
|
||||
occurred_at=event.occurred_at,
|
||||
)
|
||||
|
||||
@@ -63,15 +64,27 @@ def retry_workflow(
|
||||
status_code=409,
|
||||
)
|
||||
|
||||
# Captured before the status flips, so the audit records what was actually retried.
|
||||
was_demo_scenario = is_demo_scenario_failure(event)
|
||||
|
||||
event.delivery_status = "pending"
|
||||
event.next_attempt_at = None
|
||||
# The retry itself is real either way: the event goes back on the outbox and the
|
||||
# dispatcher delivers it to the configured n8n webhook like any other. The only
|
||||
# difference recorded here is *what* was retried -- a staged demo failure or a real
|
||||
# one -- so the audit trail never implies a production incident was resolved when a
|
||||
# prop was.
|
||||
record_audit_event(
|
||||
db,
|
||||
actor_type="user",
|
||||
actor_label=user.display_name,
|
||||
action="workflow_retry",
|
||||
entity_type="outbox_event",
|
||||
metadata={"event_id": event_id, "previous_attempts": event.attempts},
|
||||
metadata={
|
||||
"event_id": event_id,
|
||||
"previous_attempts": event.attempts,
|
||||
"demo_scenario": was_demo_scenario,
|
||||
},
|
||||
)
|
||||
db.commit()
|
||||
return _to_out(event)
|
||||
|
||||
@@ -10,6 +10,25 @@ from app.models.mixins import TimestampMixin
|
||||
|
||||
DELIVERY_STATUSES = ("pending", "delivering", "succeeded", "failed")
|
||||
|
||||
# The one delivery failure the demo seed deliberately plants (BK-H-0020, see
|
||||
# seed/workflow_runs.csv). It exists to show retry and audit working, so it must never
|
||||
# be read as an integration-health problem: it is a scripted prop, not evidence that
|
||||
# n8n is unhealthy. A dedicated error code -- rather than the generic
|
||||
# "connectionError" a real timeout produces -- is what lets every reader tell the two
|
||||
# apart without guessing from the message text.
|
||||
#
|
||||
# It is deliberately a `last_error_code` value and not a new column: the code is
|
||||
# already persisted, already surfaced to the UI, and already localizable, so no schema
|
||||
# change or migration is needed. A genuine later failure of this same event overwrites
|
||||
# the code with the real one, which is exactly right -- from that moment it *is* a real
|
||||
# failure.
|
||||
DEMO_SCENARIO_ERROR_CODE = "demoScenarioTimeout"
|
||||
|
||||
|
||||
def is_demo_scenario_failure(event: "OutboxEvent") -> bool:
|
||||
"""True for the prepared demo failure, false for every real one."""
|
||||
return event.delivery_status == "failed" and event.last_error_code == DEMO_SCENARIO_ERROR_CODE
|
||||
|
||||
|
||||
class OutboxEvent(TimestampMixin, Base):
|
||||
__tablename__ = "outbox_events"
|
||||
|
||||
+15
-1
@@ -266,11 +266,21 @@ class N8nIntegrationStatus(BaseModel):
|
||||
dispatch_enabled: bool
|
||||
state: Literal["disabled", "unavailable", "degraded", "operational", "no_evidence"]
|
||||
pending: int
|
||||
delivering: int
|
||||
#: Every failed delivery, staged and real together -- the number a viewer sees in
|
||||
#: the run list.
|
||||
failed: int
|
||||
#: Failures that were not planted by the demo seed. This is the only failure count
|
||||
#: that may influence `state`.
|
||||
unexpected_failed: int = 0
|
||||
#: Prepared demo failures (see `app.models.outbox.DEMO_SCENARIO_ERROR_CODE`).
|
||||
#: Present so the UI can label them instead of implying the automation is broken.
|
||||
demo_scenario_failed: int = 0
|
||||
delivering: int
|
||||
succeeded: int
|
||||
latest_success_at: datetime | None
|
||||
#: Most recent *real* failure; a staged one never sets this.
|
||||
latest_failure_at: datetime | None
|
||||
latest_demo_scenario_at: datetime | None = None
|
||||
expected_workflow_count: int
|
||||
known_workflow_count: int
|
||||
workflows: list[N8nWorkflowEvidence]
|
||||
@@ -365,6 +375,10 @@ class AutomationRunOut(BaseModel):
|
||||
attempts: int
|
||||
last_error: str | None
|
||||
last_error_code: str | None
|
||||
#: True for the deliberately seeded demo failure. The UI uses this to label the run
|
||||
#: as a prepared scenario and to offer the demo retry, instead of presenting it as
|
||||
#: an unexplained production error.
|
||||
is_demo_scenario: bool = False
|
||||
occurred_at: datetime
|
||||
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ from app.models.data_quality import DataQualityIssue
|
||||
from app.models.idempotency import IdempotencyRecord
|
||||
from app.models.inspection import Inspection
|
||||
from app.models.maintenance import MaintenanceRecord
|
||||
from app.models.outbox import OutboxEvent
|
||||
from app.models.outbox import DEMO_SCENARIO_ERROR_CODE, OutboxEvent
|
||||
from app.models.user import User
|
||||
from app.models.vehicle import Vehicle
|
||||
from app.services.audit import record_audit_event
|
||||
@@ -323,7 +323,10 @@ def load_seed(db: Session) -> SeedResult:
|
||||
"last_error": row["last_error"] or None,
|
||||
# The seed dataset's one synthetic failure (BK-H-0020) models a
|
||||
# connection-timeout-style delivery failure -- see workflow_runs.csv.
|
||||
"last_error_code": "connectionError" if row["last_error"] else None,
|
||||
# It is coded as a *prepared demo scenario*, not as a real
|
||||
# connectionError, so integration health never degrades because of a
|
||||
# prop and a viewer is told plainly that this failure is staged.
|
||||
"last_error_code": DEMO_SCENARIO_ERROR_CODE if row["last_error"] else None,
|
||||
"external_run_id": None,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -8,7 +8,7 @@ from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import get_settings
|
||||
from app.models.audit import AuditEvent
|
||||
from app.models.outbox import OutboxEvent
|
||||
from app.models.outbox import DEMO_SCENARIO_ERROR_CODE, OutboxEvent
|
||||
from app.schemas import (
|
||||
McpHubIntegrationStatus,
|
||||
N8nErrorHandlerStatus,
|
||||
@@ -40,19 +40,49 @@ def derive_n8n_status(db: Session) -> N8nIntegrationStatus:
|
||||
failed = counts.get("failed", 0)
|
||||
succeeded = counts.get("succeeded", 0)
|
||||
|
||||
# Prepared demo failures are props, not health signals. They stay visible and
|
||||
# counted -- hiding them would be its own kind of lie -- but they are counted
|
||||
# *separately*, and only genuinely unexpected failures are allowed to move n8n off
|
||||
# "operational". Without this split the demo seed's single staged failure pins the
|
||||
# integration to "degraded" forever, which tells a viewer something untrue about
|
||||
# the automation.
|
||||
demo_scenario_failed = (
|
||||
db.scalar(
|
||||
select(func.count())
|
||||
.select_from(OutboxEvent)
|
||||
.where(
|
||||
OutboxEvent.delivery_status == "failed",
|
||||
OutboxEvent.last_error_code == DEMO_SCENARIO_ERROR_CODE,
|
||||
)
|
||||
)
|
||||
or 0
|
||||
)
|
||||
unexpected_failed = max(failed - demo_scenario_failed, 0)
|
||||
|
||||
latest_success_at = db.scalar(
|
||||
select(func.max(OutboxEvent.updated_at)).where(OutboxEvent.delivery_status == "succeeded")
|
||||
)
|
||||
# Health talks about real failures only, so the "latest failure" a health reader
|
||||
# sees must exclude the staged one too.
|
||||
latest_failure_at = db.scalar(
|
||||
select(func.max(OutboxEvent.updated_at)).where(OutboxEvent.delivery_status == "failed")
|
||||
select(func.max(OutboxEvent.updated_at)).where(
|
||||
OutboxEvent.delivery_status == "failed",
|
||||
OutboxEvent.last_error_code != DEMO_SCENARIO_ERROR_CODE,
|
||||
)
|
||||
)
|
||||
latest_demo_scenario_at = db.scalar(
|
||||
select(func.max(OutboxEvent.updated_at)).where(
|
||||
OutboxEvent.delivery_status == "failed",
|
||||
OutboxEvent.last_error_code == DEMO_SCENARIO_ERROR_CODE,
|
||||
)
|
||||
)
|
||||
|
||||
state: Literal["disabled", "unavailable", "degraded", "operational", "no_evidence"]
|
||||
if not settings.n8n_dispatch_enabled:
|
||||
state = "disabled"
|
||||
elif failed > 0 and succeeded == 0:
|
||||
elif unexpected_failed > 0 and succeeded == 0:
|
||||
state = "unavailable"
|
||||
elif failed > 0:
|
||||
elif unexpected_failed > 0:
|
||||
state = "degraded"
|
||||
elif succeeded > 0 or pending > 0 or delivering > 0:
|
||||
state = "operational"
|
||||
@@ -112,9 +142,12 @@ def derive_n8n_status(db: Session) -> N8nIntegrationStatus:
|
||||
pending=pending,
|
||||
delivering=delivering,
|
||||
failed=failed,
|
||||
unexpected_failed=unexpected_failed,
|
||||
demo_scenario_failed=demo_scenario_failed,
|
||||
succeeded=succeeded,
|
||||
latest_success_at=latest_success_at,
|
||||
latest_failure_at=latest_failure_at,
|
||||
latest_demo_scenario_at=latest_demo_scenario_at,
|
||||
expected_workflow_count=len(_CANONICAL_WORKFLOWS),
|
||||
known_workflow_count=sum(1 for w in workflows if w.last_seen_at is not None),
|
||||
workflows=workflows,
|
||||
|
||||
@@ -1,3 +1,24 @@
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.db import SessionLocal
|
||||
from app.models.outbox import OutboxEvent
|
||||
from app.seed_loader import reset_and_seed
|
||||
|
||||
|
||||
def _reseed() -> None:
|
||||
"""Restore the canonical demo dataset (19 succeeded + 1 prepared failure).
|
||||
|
||||
The suite shares one session-scoped database and earlier files legitimately mutate
|
||||
the outbox, so any test that asserts on the *seeded* scenario has to re-establish it
|
||||
rather than depend on file ordering.
|
||||
"""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
reset_and_seed(db)
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def test_integration_status_requires_operations_manager(employee_client):
|
||||
response = employee_client.get("/api/v1/integrations/status")
|
||||
assert response.status_code == 403
|
||||
@@ -9,6 +30,7 @@ def test_integration_status_requires_authentication(client):
|
||||
|
||||
|
||||
def test_integration_status_reflects_seeded_mixed_outcomes(ops_client):
|
||||
_reseed()
|
||||
response = ops_client.get("/api/v1/integrations/status")
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
@@ -16,20 +38,65 @@ def test_integration_status_reflects_seeded_mixed_outcomes(ops_client):
|
||||
n8n = body["n8n"]
|
||||
assert n8n["dispatch_enabled"] is True
|
||||
assert n8n["succeeded"] >= 1
|
||||
# The seeded failure stays visible and counted...
|
||||
assert n8n["failed"] >= 1
|
||||
# The seed deliberately carries both failed and succeeded events, so a single most-
|
||||
# recent-event read would misreport health -- the aggregate must call this "degraded",
|
||||
# not "operational" or "unavailable".
|
||||
assert n8n["state"] == "degraded"
|
||||
assert n8n["demo_scenario_failed"] == 1
|
||||
# ...but it is a prepared prop, so it is not an unexpected failure and must not
|
||||
# move the integration off "operational". A staged failure that degrades the
|
||||
# health badge tells a viewer something untrue about the automation.
|
||||
assert n8n["unexpected_failed"] == 0
|
||||
assert n8n["state"] == "operational"
|
||||
assert n8n["latest_success_at"] is not None
|
||||
assert n8n["latest_failure_at"] is not None
|
||||
# "latest failure" is a health signal, so the staged one never sets it; it is
|
||||
# reported separately instead.
|
||||
assert n8n["latest_failure_at"] is None
|
||||
assert n8n["latest_demo_scenario_at"] is not None
|
||||
|
||||
mcp_hub = body["mcp_hub"]
|
||||
assert mcp_hub["registration_enabled"] is False
|
||||
assert mcp_hub["state"] == "not_configured"
|
||||
|
||||
|
||||
def test_a_real_failure_still_degrades_the_integration(ops_client):
|
||||
"""The demo carve-out must be narrow: a failure that is not the prepared scenario
|
||||
still degrades n8n, otherwise this change would hide real breakage."""
|
||||
_reseed()
|
||||
db = SessionLocal()
|
||||
try:
|
||||
real_failure = db.scalar(
|
||||
select(OutboxEvent).where(OutboxEvent.delivery_status == "succeeded").limit(1)
|
||||
)
|
||||
assert real_failure is not None
|
||||
restore = (real_failure.delivery_status, real_failure.last_error_code)
|
||||
real_failure.delivery_status = "failed"
|
||||
real_failure.last_error_code = "connectionError"
|
||||
db.commit()
|
||||
|
||||
n8n = ops_client.get("/api/v1/integrations/status").json()["n8n"]
|
||||
assert n8n["unexpected_failed"] == 1
|
||||
assert n8n["state"] == "degraded"
|
||||
assert n8n["latest_failure_at"] is not None
|
||||
finally:
|
||||
real_failure.delivery_status, real_failure.last_error_code = restore
|
||||
db.commit()
|
||||
db.close()
|
||||
|
||||
|
||||
def test_prepared_demo_failure_is_reset_back_by_a_demo_reset(ops_client):
|
||||
"""A demo reset must recreate the intended 19 succeeded + 1 prepared failure, so the
|
||||
scenario can be shown again after it has been retried away."""
|
||||
_reseed()
|
||||
|
||||
n8n = ops_client.get("/api/v1/integrations/status").json()["n8n"]
|
||||
assert n8n["succeeded"] == 19
|
||||
assert n8n["failed"] == 1
|
||||
assert n8n["demo_scenario_failed"] == 1
|
||||
assert n8n["unexpected_failed"] == 0
|
||||
assert n8n["state"] == "operational"
|
||||
|
||||
|
||||
def test_integration_status_is_operational_once_all_failed_events_resolved(ops_client):
|
||||
_reseed()
|
||||
failed = ops_client.get("/api/v1/workflows", params={"status": "failed"}).json()
|
||||
for run in failed:
|
||||
retried = ops_client.post(f"/api/v1/workflows/{run['event_id']}/retry")
|
||||
|
||||
@@ -7,7 +7,7 @@ from app.models.audit import AuditEvent
|
||||
from app.models.booking import Booking
|
||||
from app.models.customer import Customer
|
||||
from app.models.data_quality import DataQualityIssue
|
||||
from app.models.outbox import OutboxEvent
|
||||
from app.models.outbox import DEMO_SCENARIO_ERROR_CODE, OutboxEvent
|
||||
from app.models.user import User
|
||||
from app.models.vehicle import Vehicle
|
||||
from app.seed_loader import SEED_AUTHORED_ANCHOR, reset_and_seed
|
||||
@@ -142,7 +142,10 @@ def test_seed_scenario_s5_failed_workflow_run():
|
||||
assert failed.delivery_status == "failed"
|
||||
assert failed.attempts >= 1
|
||||
assert failed.last_error
|
||||
assert failed.last_error_code == "connectionError"
|
||||
# Coded as a prepared demo scenario, not as a real connectionError: the whole
|
||||
# point of this row is to demonstrate retry and audit, so nothing downstream
|
||||
# may read it as evidence that the n8n integration is unhealthy.
|
||||
assert failed.last_error_code == DEMO_SCENARIO_ERROR_CODE
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@@ -1,9 +1,28 @@
|
||||
from app.core.db import SessionLocal
|
||||
from app.seed_loader import reset_and_seed
|
||||
|
||||
|
||||
def _reseed() -> None:
|
||||
"""Restore the canonical demo dataset (19 succeeded + 1 prepared failure).
|
||||
|
||||
The suite shares one session-scoped database and earlier files legitimately mutate
|
||||
the outbox, so any test that asserts on the *seeded* scenario has to re-establish it
|
||||
rather than depend on file ordering.
|
||||
"""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
reset_and_seed(db)
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def test_list_workflows_requires_operations_manager(employee_client):
|
||||
response = employee_client.get("/api/v1/workflows")
|
||||
assert response.status_code == 403
|
||||
|
||||
|
||||
def test_list_workflows_includes_seeded_failed_run(ops_client):
|
||||
_reseed()
|
||||
response = ops_client.get("/api/v1/workflows", params={"status": "failed"})
|
||||
assert response.status_code == 200
|
||||
runs = response.json()
|
||||
@@ -20,6 +39,7 @@ def test_retry_requires_failed_status(ops_client):
|
||||
|
||||
|
||||
def test_retry_failed_run_moves_to_pending_and_audits(ops_client):
|
||||
_reseed()
|
||||
failed = ops_client.get("/api/v1/workflows", params={"status": "failed"}).json()
|
||||
target = failed[0]["event_id"]
|
||||
|
||||
@@ -36,3 +56,42 @@ def test_retry_requires_operations_manager(employee_client):
|
||||
"/api/v1/workflows/00000000-0000-4000-8000-000000000020/retry"
|
||||
)
|
||||
assert response.status_code == 403
|
||||
|
||||
|
||||
def test_seeded_failure_is_labelled_as_a_prepared_demo_scenario(ops_client):
|
||||
"""The one seeded failure must announce itself as staged. An unexplained red row in
|
||||
a demo reads as a broken product; a labelled one reads as the retry story it is."""
|
||||
_reseed()
|
||||
runs = ops_client.get("/api/v1/workflows", params={"status": "failed"}).json()
|
||||
demo_runs = [run for run in runs if run["is_demo_scenario"]]
|
||||
assert len(demo_runs) == 1
|
||||
assert demo_runs[0]["last_error_code"] == "demoScenarioTimeout"
|
||||
assert demo_runs[0]["aggregate_ref"] == "BK-H-0020"
|
||||
|
||||
|
||||
def test_succeeded_runs_are_never_marked_as_a_demo_scenario(ops_client):
|
||||
runs = ops_client.get("/api/v1/workflows", params={"status": "succeeded"}).json()
|
||||
assert runs
|
||||
assert all(run["is_demo_scenario"] is False for run in runs)
|
||||
|
||||
|
||||
def test_retry_of_the_demo_scenario_is_audited_as_a_demo_scenario(ops_client):
|
||||
"""The retry is a real redelivery either way; the audit records which kind of
|
||||
failure it resolved so a staged retry is never mistaken for a production fix."""
|
||||
_reseed()
|
||||
failed = ops_client.get("/api/v1/workflows", params={"status": "failed"}).json()
|
||||
demo_run = next(run for run in failed if run["is_demo_scenario"])
|
||||
|
||||
response = ops_client.post(f"/api/v1/workflows/{demo_run['event_id']}/retry")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "pending"
|
||||
|
||||
audit = ops_client.get("/api/v1/audit", params={"action": "workflow_retry"}).json()
|
||||
entry = next(
|
||||
event
|
||||
for event in audit
|
||||
if (event.get("metadata") or event.get("metadata_json") or {}).get("event_id")
|
||||
== demo_run["event_id"]
|
||||
)
|
||||
metadata = entry.get("metadata") or entry.get("metadata_json") or {}
|
||||
assert metadata["demo_scenario"] is True
|
||||
|
||||
@@ -67,3 +67,34 @@ secret-free failure details to Fleet Ops. See `n8n/workflows/MANIFEST.md` for st
|
||||
- supports explicit manual retry;
|
||||
- preserves last error and response metadata;
|
||||
- does not hold a database transaction open during network I/O.
|
||||
|
||||
## Prepared demo failure versus real failure
|
||||
|
||||
The demo seed deliberately plants exactly one failed delivery (`BK-H-0020`, see
|
||||
`seed/workflow_runs.csv`). It exists to demonstrate retry and audit, so it must never be
|
||||
read as evidence that the automation is unhealthy.
|
||||
|
||||
It is distinguished by its `last_error_code`, `demoScenarioTimeout`
|
||||
(`app.models.outbox.DEMO_SCENARIO_ERROR_CODE`) — not by a new column, so no migration is
|
||||
involved. A real timeout produces `connectionError`; the two are never confused.
|
||||
|
||||
Consequences, all enforced by tests:
|
||||
|
||||
- `/api/v1/integrations/status` reports `failed` (everything), `unexpected_failed` (real
|
||||
failures only) and `demo_scenario_failed` separately.
|
||||
- Only `unexpected_failed` can move n8n off `operational`. A prepared failure alone
|
||||
leaves the integration **operational** — a staged prop may not raise a red flag.
|
||||
- `latest_failure_at` is a health signal and therefore ignores the prepared failure;
|
||||
`latest_demo_scenario_at` reports it separately.
|
||||
- `/api/v1/workflows` marks the run with `is_demo_scenario: true`. The Automation page
|
||||
labels it "Prepared demo scenario", explains that it is a simulated temporary failure,
|
||||
and offers a distinct "Retry demo scenario" action.
|
||||
- A genuine later failure of that same event overwrites the code with the real one, and
|
||||
from that moment it counts as a real failure — the carve-out is narrow by construction.
|
||||
|
||||
The retry itself is real in both cases: the event goes back on the outbox and the
|
||||
dispatcher delivers it to the configured n8n webhook like any other, so 19 succeeded +
|
||||
1 failed becomes 20 succeeded + 0 failed only when n8n genuinely accepts the delivery.
|
||||
Nothing is marked succeeded without a real round trip. The audit entry records
|
||||
`demo_scenario: true/false` so a staged retry is never mistaken for a production fix.
|
||||
A demo reset recreates the original 19 + 1 scenario.
|
||||
|
||||
@@ -36,7 +36,12 @@ Vehicle `MO-016` has two imported overlapping reservations. Expected: visible qu
|
||||
|
||||
### S5 — Failed workflow
|
||||
|
||||
One seeded outbox/workflow record is failed with a safe simulated connection error. Expected: dashboard and Automation page show it; Operations Manager can retry.
|
||||
One seeded outbox/workflow record is failed with a safe simulated connection error, coded
|
||||
`demoScenarioTimeout` so it is recognisable as a prepared scenario rather than a real
|
||||
incident. Expected: dashboard and Automation page show it, labelled as a prepared demo
|
||||
scenario; n8n stays "Operational"; the Operations Manager can retry it, after which the
|
||||
overview reads 20 succeeded and 0 failed. See `docs/11-n8n-integration.md`, "Prepared demo
|
||||
failure versus real failure".
|
||||
|
||||
### S6 — Grounded damage question
|
||||
|
||||
|
||||
@@ -110,6 +110,10 @@ export interface AutomationRun {
|
||||
attempts: number;
|
||||
last_error: string | null;
|
||||
last_error_code: string | null;
|
||||
/** True for the failure the demo seed plants on purpose (see the backend's
|
||||
* DEMO_SCENARIO_ERROR_CODE). Drives the "prepared scenario" labelling instead of
|
||||
* showing it as an unexplained production error. */
|
||||
is_demo_scenario: boolean;
|
||||
occurred_at: string;
|
||||
}
|
||||
|
||||
@@ -272,10 +276,17 @@ export interface N8nIntegrationStatus {
|
||||
state: "disabled" | "unavailable" | "degraded" | "operational" | "no_evidence";
|
||||
pending: number;
|
||||
delivering: number;
|
||||
/** Every failed delivery, prepared and real together. */
|
||||
failed: number;
|
||||
/** Failures that were not planted by the demo seed -- the only ones that affect state. */
|
||||
unexpected_failed: number;
|
||||
/** Prepared demo failures. Visible and counted, but never a health signal. */
|
||||
demo_scenario_failed: number;
|
||||
succeeded: number;
|
||||
latest_success_at: string | null;
|
||||
/** Most recent real failure; a prepared one never sets this. */
|
||||
latest_failure_at: string | null;
|
||||
latest_demo_scenario_at: string | null;
|
||||
expected_workflow_count: number;
|
||||
known_workflow_count: number;
|
||||
workflows: N8nWorkflowEvidence[];
|
||||
|
||||
@@ -19,7 +19,8 @@
|
||||
"mcpNoEvidence": "Registered, but no tool call has been recorded yet.",
|
||||
"mcpEvidence": "Last call: {{tool}} by {{client}} · {{count}} total calls",
|
||||
"mcpHubReachable": "Hub reachable",
|
||||
"mcpHubUnreachable": "Hub unreachable"
|
||||
"mcpHubUnreachable": "Hub unreachable",
|
||||
"n8nDemoScenario": "Plus {{count}} prepared demo scenario — a simulated temporary failure, not an integration problem."
|
||||
},
|
||||
"statusLabels": {
|
||||
"notConnected": "Not connected",
|
||||
@@ -97,7 +98,11 @@
|
||||
"malformedPayload": "The job contained incomplete data and could not be delivered. The underlying data remains safely stored.",
|
||||
"remoteReportedFailure": "The workflow service declined the delivery. The job can be retried.",
|
||||
"staleLeaseRecovered": "This job was recovered after an earlier delivery attempt stalled without a result.",
|
||||
"unknownError": "An unexpected error occurred while delivering this job."
|
||||
}
|
||||
"unknownError": "An unexpected error occurred while delivering this job.",
|
||||
"demoScenarioTimeout": "Simulated timeout towards the workflow service. This failure is part of the prepared demo scenario, not a real incident."
|
||||
},
|
||||
"demoScenarioBadge": "Prepared demo scenario",
|
||||
"demoScenarioExplanation": "Simulated temporary failure, planted by the demo data on purpose to show retry and audit. It does not affect the health of the automation.",
|
||||
"retryDemoScenario": "Retry demo scenario"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,7 +19,8 @@
|
||||
"mcpNoEvidence": "Enregistré, mais aucun appel d'outil n'a encore été consigné.",
|
||||
"mcpEvidence": "Dernier appel : {{tool}} par {{client}} · {{count}} appels au total",
|
||||
"mcpHubReachable": "Hub accessible",
|
||||
"mcpHubUnreachable": "Hub inaccessible"
|
||||
"mcpHubUnreachable": "Hub inaccessible",
|
||||
"n8nDemoScenario": "Plus {{count}} scénario de démonstration préparé — une panne temporaire simulée, pas un problème d'intégration."
|
||||
},
|
||||
"statusLabels": {
|
||||
"notConnected": "Non connecté",
|
||||
@@ -97,7 +98,11 @@
|
||||
"malformedPayload": "La tâche contenait des données incomplètes et n'a pas pu être livrée. Les données sous-jacentes restent enregistrées en toute sécurité.",
|
||||
"remoteReportedFailure": "Le service d'automatisation a refusé la livraison. La tâche peut être soumise à nouveau.",
|
||||
"staleLeaseRecovered": "Cette tâche a été récupérée après qu'une tentative de livraison précédente s'est arrêtée sans résultat.",
|
||||
"unknownError": "Une erreur inattendue s'est produite lors de la livraison de cette tâche."
|
||||
}
|
||||
"unknownError": "Une erreur inattendue s'est produite lors de la livraison de cette tâche.",
|
||||
"demoScenarioTimeout": "Délai d'attente simulé vers le service de workflow. Cette panne fait partie du scénario de démonstration préparé, ce n'est pas un incident réel."
|
||||
},
|
||||
"demoScenarioBadge": "Scénario de démonstration préparé",
|
||||
"demoScenarioExplanation": "Panne temporaire simulée, placée volontairement dans les données de démonstration pour montrer la relance et l'audit. Elle n'affecte pas la santé de l'automatisation.",
|
||||
"retryDemoScenario": "Relancer le scénario de démonstration"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,7 +19,8 @@
|
||||
"mcpNoEvidence": "Geregistreerd, maar er is nog geen tool-aanroep geregistreerd.",
|
||||
"mcpEvidence": "Laatste aanroep: {{tool}} door {{client}} · {{count}} aanroepen in totaal",
|
||||
"mcpHubReachable": "Hub bereikbaar",
|
||||
"mcpHubUnreachable": "Hub onbereikbaar"
|
||||
"mcpHubUnreachable": "Hub onbereikbaar",
|
||||
"n8nDemoScenario": "Plus {{count}} voorbereid demoscenario — een gesimuleerde tijdelijke fout, geen integratieprobleem."
|
||||
},
|
||||
"statusLabels": {
|
||||
"notConnected": "Niet gekoppeld",
|
||||
@@ -97,7 +98,11 @@
|
||||
"malformedPayload": "De opdracht bevatte onvolledige gegevens en kon niet worden afgeleverd. De onderliggende gegevens blijven veilig bewaard.",
|
||||
"remoteReportedFailure": "De workflowdienst heeft de aflevering geweigerd. De opdracht kan opnieuw worden aangeboden.",
|
||||
"staleLeaseRecovered": "Deze opdracht werd hersteld nadat een eerdere afleverpoging vastliep zonder resultaat.",
|
||||
"unknownError": "Er is een onverwachte fout opgetreden bij het afleveren van deze opdracht."
|
||||
}
|
||||
"unknownError": "Er is een onverwachte fout opgetreden bij het afleveren van deze opdracht.",
|
||||
"demoScenarioTimeout": "Gesimuleerde time-out richting de workflowdienst. Deze fout hoort bij het voorbereide demoscenario en is geen echt incident."
|
||||
},
|
||||
"demoScenarioBadge": "Voorbereid demoscenario",
|
||||
"demoScenarioExplanation": "Gesimuleerde tijdelijke fout, bewust in de demodata gezet om retry en audit te tonen. Ze heeft geen invloed op de gezondheid van de automatisering.",
|
||||
"retryDemoScenario": "Demoscenario opnieuw proberen"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -117,12 +117,16 @@ export function Automation() {
|
||||
<td data-label={t("ledger.columns.booking")}>{r.aggregate_ref}</td>
|
||||
<td data-label={t("ledger.columns.status")}>
|
||||
<StatusBadge status={r.status} label={t(`ledger.status${r.status.charAt(0).toUpperCase()}${r.status.slice(1)}`)} />
|
||||
{r.is_demo_scenario && (
|
||||
<StatusBadge status="demo_scenario" label={t("ledger.demoScenarioBadge")} />
|
||||
)}
|
||||
</td>
|
||||
<td data-label={t("ledger.columns.attempts")}>{r.attempts}</td>
|
||||
<td data-label={t("ledger.columns.lastError")}>
|
||||
{r.last_error_code ? (
|
||||
<>
|
||||
<span>{t(`ledger.errorCodes.${r.last_error_code}`, { defaultValue: r.last_error ?? r.last_error_code })}</span>
|
||||
{r.is_demo_scenario && <p className="table-subtext">{t("ledger.demoScenarioExplanation")}</p>}
|
||||
{r.last_error && (
|
||||
<details className="evidence-disclosure">
|
||||
<summary>{t("common:actions.technicalDetails")}</summary>
|
||||
@@ -140,7 +144,9 @@ export function Automation() {
|
||||
<td data-label={t("ledger.columns.action")}>
|
||||
{r.status === "failed" ? (
|
||||
<button type="button" onClick={() => handleRetry(r.event_id)} disabled={retrying === r.event_id}>
|
||||
{retrying === r.event_id ? t("ledger.retrying") : t("ledger.retry")}
|
||||
{retrying === r.event_id
|
||||
? t("ledger.retrying")
|
||||
: t(r.is_demo_scenario ? "ledger.retryDemoScenario" : "ledger.retry")}
|
||||
</button>
|
||||
) : (
|
||||
t("ledger.noAction")
|
||||
@@ -164,12 +170,19 @@ export function Automation() {
|
||||
{integrationStatus
|
||||
? t("cards.n8nSummary", {
|
||||
succeeded: integrationStatus.n8n.succeeded,
|
||||
failed: integrationStatus.n8n.failed,
|
||||
failed: integrationStatus.n8n.unexpected_failed,
|
||||
pending: integrationStatus.n8n.pending,
|
||||
delivering: integrationStatus.n8n.delivering,
|
||||
})
|
||||
: t("cards.n8nFallback")}
|
||||
</p>
|
||||
{/* Prepared demo failures are stated separately and never folded into the
|
||||
health count above, so the badge and the numbers tell the same story. */}
|
||||
{integrationStatus && integrationStatus.n8n.demo_scenario_failed > 0 && (
|
||||
<p className="table-subtext">
|
||||
{t("cards.n8nDemoScenario", { count: integrationStatus.n8n.demo_scenario_failed })}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
{(() => {
|
||||
const meta = integrationStatus ? N8N_STATE_META[integrationStatus.n8n.state] : null;
|
||||
|
||||
@@ -210,7 +210,7 @@ a:hover { color: var(--teal); }
|
||||
.badge::before, .badge-dot { content: ""; width: 5px; height: 5px; border-radius: 50%; background: currentColor; }
|
||||
.severity-high, .status-failed, .status-blocked, .status-rejected, .status-unavailable { color: #9f2929; background: var(--critical-pale); border-color: #f3c5c5; }
|
||||
.severity-medium, .status-pending, .status-delivering, .status-needs_attention { color: #93520c; background: var(--warning-pale); border-color: #eed4aa; }
|
||||
.severity-low { color: #315f79; background: var(--info-pale); border-color: #c8dfeb; }
|
||||
.severity-low, .status-demo_scenario { color: #315f79; background: var(--info-pale); border-color: #c8dfeb; }
|
||||
.status-succeeded, .status-resolved, .status-available, .status-returned, .status-active { color: #17633f; background: var(--success-pale); border-color: #bfe3d0; }
|
||||
.status-cancelled, .status-deferred, .status-cleaning, .status-maintenance, .status-rented, .status-reserved, .status-not_configured, .status-no_events, .status-no_delivery_yet { color: #536172; background: #f0f3f6; border-color: #d8dfe6; }
|
||||
.badge-dot { display: inline-block; }
|
||||
|
||||
Reference in New Issue
Block a user