54 Commits
Author SHA1 Message Date
Jens 0458403e67 Merge pull request 'ci: retire legacy unraid deploy runner workflow' (#11) from codex/runnerless-ci-v1-8-mobilityops into master
Managed validation / full (push) Successful in 4s
2026-09-14 13:18:21 +00:00
ITWorx AppOps Automation 885dff413c ci: retire legacy unraid deploy runner workflow
MobilityOps acceptance / acceptance (pull_request) Successful in 33s
2026-09-14 14:26:23 +02:00
Jens f8a30c0f5c Merge pull request 'hygiene: prepare public release' (#10) from chore/public-release-hygiene-20260902 into master
Unraid autoredeploy / Deploy mobilityops (push) Failing after 3s
2026-09-03 00:58:52 +00:00
NuklearRabbit 2d92084489 test: stabilize browser assertions under CI load
MobilityOps acceptance / acceptance (pull_request) Successful in 8m45s
2026-09-03 01:13:21 +02:00
NuklearRabbit a42012d9c0 hygiene: prepare MobilityOps for public release
MobilityOps acceptance / acceptance (pull_request) Failing after 10m16s
2026-09-02 23:44:03 +02:00
Jens 0de9177e7c Merge pull request 'Use Gitea-compatible canary evidence upload' (#3) from codex/resolve-projectbrain-findings-20260828 into master
Unraid autoredeploy / Deploy mobilityops (push) Failing after 1m11s
Reviewed-on: #3
2026-09-01 20:09:22 +00:00
NuklearRabbit 48147465b5 security: refresh Alpine runtime packages
MobilityOps acceptance / acceptance (pull_request) Successful in 6m28s
2026-09-01 21:19:48 +02:00
NuklearRabbit 92bdfb421b Merge remote-tracking branch 'origin/master' into codex/resolve-projectbrain-findings-20260828
MobilityOps acceptance / acceptance (pull_request) Failing after 2m42s
# Conflicts:
#	.dockerignore
#	.gitea/workflows/ci.yml
#	.gitea/workflows/live-canary.yml
#	.gitea/workflows/release.yml
2026-09-01 20:42:10 +02:00
Jens 4257fea9d6 Merge pull request 'chore: sanitize MobilityOps operational evidence and public boundary' (#8) from chatgpt/repo-hygiene-mobilityops into master
Unraid autoredeploy / Deploy mobilityops (push) Failing after 1h58m50s
Reviewed-on: #8
2026-09-01 14:29:25 +02:00
NuklearRabbit 014caba7d5 test: target data-quality reference in page header
MobilityOps acceptance / acceptance (pull_request) Successful in 5m54s
2026-09-01 01:44:50 +02:00
Jens e531740bb9 docs: record MobilityOps history blocker
MobilityOps acceptance / acceptance (pull_request) Failing after 4m13s
2026-08-31 09:26:15 +02:00
Jens 921c6878a6 repo: prevent generated deployment evidence returning
MobilityOps acceptance / acceptance (pull_request) Canceled after 0s
2026-08-31 09:25:48 +02:00
Jens cd55d6854f build: exclude AI, credential and runtime state
MobilityOps acceptance / acceptance (pull_request) Failing after 4m36s
2026-08-31 07:56:09 +02:00
Jens 70e2648b03 docs: add fleet privacy contribution guidance 2026-08-31 07:53:43 +02:00
Jens f8eff37982 docs: record MobilityOps history sanitation blockers 2026-08-31 07:53:33 +02:00
Jens 25cc7a1b0d chore: remove generated agent handoff entrypoint 2026-08-31 07:53:12 +02:00
Jens 6fd1d5e5e9 chore: remove generated build-agent prompt 2026-08-31 07:53:02 +02:00
Jens 54b7719759 chore: remove generated repository file index 2026-08-31 07:52:54 +02:00
Jens a19613d5e9 chore: remove project-local Claude instructions 2026-08-31 07:52:44 +02:00
Jens b62ecf66d8 chore: remove generated project-state chronology 2026-08-31 07:52:35 +02:00
Jens 97b8cd1c8c docs: generalize demo deployment instructions 2026-08-31 07:51:49 +02:00
Jens 50f9bb471a docs: replace private deployment topology with configuration 2026-08-31 07:51:06 +02:00
Jens 719b710926 chore: exclude internal evidence from source archives 2026-08-31 07:50:44 +02:00
Jens 441b3624e1 chore: block generated evidence and local private state 2026-08-31 07:50:30 +02:00
ChatGPT MCP ccae0538d7 chore: remove generated design-validation screenshot baselines 2026-08-31 05:49:42 +00:00
ChatGPT MCP 7d935c5a7d chore: remove generated deployment and audit evidence 2026-08-31 05:49:16 +00:00
Jens 51c3c90fa1 ci: keep workflow-only pull requests lightweight (#6)
Full acceptance remains mandatory for product and test changes; workflow-only pull requests run checkout and verified-secret scanning without rebuilding every stack.
2026-08-29 09:09:14 +02:00
NuklearRabbit d6a566e1a1 ci: keep workflow-only pull requests lightweight
MobilityOps acceptance / acceptance (pull_request) Successful in 26s
2026-08-29 09:07:38 +02:00
NuklearRabbit 4b727a109f ci: avoid duplicate post-merge acceptance
MobilityOps acceptance / acceptance (pull_request) Canceled after 2m0s
2026-08-29 09:04:25 +02:00
Jens 1bf72c39e3 ci: streamline MobilityOps validation (#5)
MobilityOps acceptance / acceptance (push) Failing after 2s
One bounded PR acceptance job, lightweight hourly probe, daily browser canary, deterministic isolated demo resets, and pinned weekly/release scans.
2026-08-29 09:03:13 +02:00
NuklearRabbit 55e8ebd81e fix(ci): isolate demo resets and keep acceptance deterministic
MobilityOps acceptance / acceptance (pull_request) Successful in 9m46s
2026-08-29 08:48:06 +02:00
NuklearRabbit b2bdb78baa fix(ci): reach DIND services over the compose network
MobilityOps acceptance / acceptance (pull_request) Failing after 6m58s
2026-08-29 08:36:14 +02:00
NuklearRabbit fe92a7f491 ci: consolidate validation and split lightweight probes
MobilityOps acceptance / acceptance (pull_request) Failing after 4m47s
2026-08-29 06:59:02 +02:00
NuklearRabbit 19df5f7508 ci: keep one required pull-request gate
MobilityOps acceptance / backend (pull_request) Failing after 2m18s
MobilityOps acceptance / frontend (pull_request) Successful in 29s
MobilityOps acceptance / e2e (pull_request) Skipped
2026-08-29 05:38:23 +02:00
NuklearRabbit e8eed3e641 fix(ci): package contract fixtures and recover deploy chunk races
MobilityOps acceptance / backend (pull_request) Failing after 2m29s
MobilityOps acceptance / frontend (pull_request) Successful in 1m10s
MobilityOps acceptance / e2e (pull_request) Skipped
Managed validation / full (pull_request) Successful in 3s
2026-08-29 00:40:51 +02:00
NuklearRabbit 2918608240 fix(ci): isolate contract and image scans from host paths
MobilityOps acceptance / backend (pull_request) Failing after 2m20s
MobilityOps acceptance / frontend (pull_request) Successful in 1m24s
MobilityOps acceptance / e2e (pull_request) Skipped
Managed validation / full (pull_request) Canceled after 0s
2026-08-29 00:27:17 +02:00
NuklearRabbit a99aed9a5f perf(ci): deduplicate branch validation
MobilityOps acceptance / backend (pull_request) Canceled after 0s
MobilityOps acceptance / frontend (pull_request) Canceled after 0s
MobilityOps acceptance / e2e (pull_request) Canceled after 0s
Managed validation / full (pull_request) Canceled after 0s
2026-08-28 23:59:30 +02:00
NuklearRabbit 00e8ec001b fix(ci): run secret scan inside isolated workspace
Managed validation / full (pull_request) Canceled after 0s
MobilityOps acceptance / frontend (pull_request) Successful in 1m14s
MobilityOps acceptance / backend (pull_request) Failing after 4m7s
MobilityOps acceptance / e2e (pull_request) Skipped
2026-08-28 23:57:46 +02:00
NuklearRabbit d20ff7a243 fix(ci): upgrade pinned Trivy action
Managed validation / full (pull_request) Canceled after 0s
MobilityOps acceptance / backend (pull_request) Failing after 32s
MobilityOps acceptance / frontend (pull_request) Successful in 1m1s
MobilityOps acceptance / e2e (pull_request) Skipped
2026-08-28 23:42:59 +02:00
NuklearRabbit bc0951115b fix(ci): use Gitea-compatible canary artifacts
Managed validation / full (pull_request) Successful in 10s
MobilityOps acceptance / frontend (pull_request) Successful in 1m17s
MobilityOps acceptance / backend (pull_request) Failing after 1m18s
MobilityOps acceptance / e2e (pull_request) Skipped
2026-08-28 23:11:09 +02:00
Jens 51d488a634 Merge pull request '[skip ci] Centralize Gitea Actions runner' (#2) from codex/centralize-gitea-runner into master 2026-08-27 23:51:14 +02:00
NuklearRabbit da2f0956c9 chore(actions): use central Unraid runner [skip ci] 2026-08-27 23:42:54 +02:00
Jens 13f8db7573 ci: align managed validation contract [skip ci] 2026-08-27 07:40:50 +02:00
Jens e6ec89658e ci: add managed validation contract [skip ci] 2026-08-27 06:28:27 +02:00
NuklearRabbit 9d71555135 ci: add scoped Unraid autoredeploy
MobilityOps acceptance / backend (push) Failing after 33s
MobilityOps acceptance / frontend (push) Successful in 45s
MobilityOps acceptance / e2e (push) Skipped
2026-08-26 02:50:54 +02:00
NuklearRabbit 81de78bd8b M56: make RAGcore sync fail closed
MobilityOps acceptance / backend (push) Failing after 19s
MobilityOps acceptance / frontend (push) Successful in 26s
MobilityOps acceptance / e2e (push) Skipped
2026-08-24 03:58:57 +02:00
NuklearRabbit 444e61253b M55: restore private RAGcore routing
MobilityOps acceptance / backend (push) Failing after 18s
MobilityOps acceptance / frontend (push) Successful in 27s
MobilityOps acceptance / e2e (push) Skipped
2026-08-24 03:45:09 +02:00
NuklearRabbit 81e3fd63bd M54: harden operations and demo resilience
MobilityOps acceptance / backend (push) Failing after 19s
MobilityOps acceptance / frontend (push) Successful in 25s
MobilityOps acceptance / e2e (push) Skipped
2026-08-24 03:31:03 +02:00
NuklearRabbit b0706989db M53: record OneDrive production acceptance
MobilityOps acceptance / backend (push) Failing after 33s
MobilityOps acceptance / frontend (push) Successful in 26s
MobilityOps acceptance / e2e (push) Skipped
2026-08-22 00:00:45 +02:00
NuklearRabbit 2036e8b4ec M52: activate verified OneDrive backups
MobilityOps acceptance / backend (push) Failing after 18s
MobilityOps acceptance / frontend (push) Successful in 26s
MobilityOps acceptance / e2e (push) Skipped
2026-08-21 23:57:47 +02:00
NuklearRabbit 51b0ade7e7 M51: record zero-error rollout acceptance
MobilityOps acceptance / backend (push) Failing after 19s
MobilityOps acceptance / frontend (push) Successful in 30s
MobilityOps acceptance / e2e (push) Skipped
MobilityOps release evidence / release-evidence (push) Failing after 2m18s
2026-08-21 22:33:41 +02:00
NuklearRabbit cea0825d60 M50: preserve rollout API compatibility alias
MobilityOps acceptance / backend (push) Failing after 17s
MobilityOps acceptance / frontend (push) Successful in 29s
MobilityOps acceptance / e2e (push) Skipped
2026-08-21 22:29:55 +02:00
NuklearRabbit dd3acd872c M49: record resilience production acceptance
MobilityOps acceptance / backend (push) Failing after 20s
MobilityOps acceptance / frontend (push) Successful in 26s
MobilityOps acceptance / e2e (push) Skipped
2026-08-21 22:25:48 +02:00
NuklearRabbit 00191e9b54 M48: harden demo operations and offsite recovery
MobilityOps acceptance / backend (push) Failing after 20s
MobilityOps acceptance / frontend (push) Successful in 28s
MobilityOps acceptance / e2e (push) Skipped
2026-08-21 22:17:49 +02:00
190 changed files with 7334 additions and 7869 deletions
+47 -11
View File
@@ -1,21 +1,57 @@
# Backend image build context is the repository root (see compose.yaml); keep it small.
.git .git
.gitea .gitea
.state .github
.mypy_cache .gitignore
.ruff_cache .agents
.codex
.claude
.dyad
.idea
.vscode
.vs
.venv
__pycache__
.pytest_cache .pytest_cache
**/__pycache__ .ruff_cache
**/.venv .mypy_cache
**/node_modules node_modules
**/dist frontend/node_modules
**/playwright-report frontend/dist
**/test-results dist
artifacts artifacts
docs docs
deploy deploy
n8n n8n/**
!n8n/workflows/
!n8n/workflows/**
frontend frontend
*.tgz *.tgz
*.tar.gz *.tar.gz
coverage
playwright-report
test-results
.env .env
.env.*
!.env.example
*.key
*.pem
*.p12
*.pfx
secrets
credentials
.state
data
backups
*.db
*.db-shm
*.db-wal
*.sqlite
*.sqlite-shm
*.sqlite-wal
*.log
*.tmp
*.zip
*.tar
*.tar.gz
.DS_Store
Thumbs.db
+14
View File
@@ -50,6 +50,13 @@ BACKUP_RESTORE_DRILL_INTERVAL_SECONDS=604800
BACKUP_SECONDARY_DESTINATION= BACKUP_SECONDARY_DESTINATION=
MOBILITYOPS_BACKUP_DIR=./backups/postgres MOBILITYOPS_BACKUP_DIR=./backups/postgres
MOBILITYOPS_BACKUP_SECONDARY_DIR=./backups/offsite MOBILITYOPS_BACKUP_SECONDARY_DIR=./backups/offsite
# Optional real off-site copy through the official rclone OneDrive adapter. OAuth state
# lives only in MOBILITYOPS_RCLONE_CONFIG_DIR and must never be committed.
BACKUP_OFFSITE_INTERVAL_SECONDS=900
RCLONE_ONEDRIVE_REMOTE=onedrive
RCLONE_ONEDRIVE_PATH=FleetOps/backups
MOBILITYOPS_RCLONE_CONFIG_DIR=./.secrets/rclone
MOBILITYOPS_OFFSITE_VERIFY_DIR=./backups/offsite-verify
# Privacy governance defaults. # Privacy governance defaults.
PRIVACY_MINIMUM_BOOKING_RETENTION_DAYS=30 PRIVACY_MINIMUM_BOOKING_RETENTION_DAYS=30
@@ -79,6 +86,10 @@ N8N_ENCRYPTION_KEY=replace-me
N8N_BASIC_AUTH_ACTIVE=true N8N_BASIC_AUTH_ACTIVE=true
N8N_BASIC_AUTH_USER=admin N8N_BASIC_AUTH_USER=admin
N8N_BASIC_AUTH_PASSWORD=change-me N8N_BASIC_AUTH_PASSWORD=change-me
# Recipient and sender used by the importable alert workflow. Configure real,
# monitored addresses in the deployment environment; repository defaults stay synthetic.
MOBILITYOPS_ALERT_RECIPIENT=alerts@example.test
MOBILITYOPS_ALERT_SENDER=n8n@example.test
MOBILITYOPS_CALLBACK_TOKEN=replace-me-n8n-callback-token MOBILITYOPS_CALLBACK_TOKEN=replace-me-n8n-callback-token
# Sent as the X-Fleet-Ops-Trigger-Token header when Fleet Ops calls the n8n return- # Sent as the X-Fleet-Ops-Trigger-Token header when Fleet Ops calls the n8n return-
# processing webhook, so the webhook trigger can require Header Auth instead of being # processing webhook, so the webhook trigger can require Header Auth instead of being
@@ -89,6 +100,9 @@ MOBILITYOPS_WEBHOOK_TRIGGER_TOKEN=replace-me-n8n-webhook-trigger-token
# RAGcore integration # RAGcore integration
KNOWLEDGE_PROVIDER=demo KNOWLEDGE_PROVIDER=demo
RAGCORE_BASE_URL=http://ragcore-api:8000 RAGCORE_BASE_URL=http://ragcore-api:8000
# Optional existing Docker network used by hosted deployments to reach RAGcore through
# its private service alias instead of exposing RAGcore on the LAN.
RAGCORE_DOCKER_NETWORK=
RAGCORE_TENANT=northstar-mobility-demo RAGCORE_TENANT=northstar-mobility-demo
RAGCORE_WORKSPACE=mobilityops RAGCORE_WORKSPACE=mobilityops
RAGCORE_COLLECTION=internal-procedures RAGCORE_COLLECTION=internal-procedures
+10
View File
@@ -1,4 +1,14 @@
* text=auto eol=lf * text=auto eol=lf
*.sh text eol=lf *.sh text eol=lf
*.ps1 text eol=crlf
*.png binary *.png binary
*.jpg binary *.jpg binary
*.jpeg binary
*.webp binary
*.zip binary
# Generated operational evidence is not part of a source release archive.
/artifacts export-ignore
/PROJECT_STATE.md export-ignore
/MASTER_BUILD_PROMPT.md export-ignore
/FILE_INDEX.md export-ignore
+44
View File
@@ -0,0 +1,44 @@
name: MobilityOps browser canary
on:
schedule:
- cron: "37 4 * * *"
workflow_dispatch:
concurrency:
group: mobilityops-browser-canary
cancel-in-progress: true
permissions:
contents: read
jobs:
chromium:
runs-on: linux-validation
timeout-minutes: 15
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 22
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Install locked Chromium runtime
working-directory: frontend
run: |
npm ci --no-audit --no-fund
npx playwright install --with-deps chromium
- name: Run non-destructive production canary
working-directory: frontend
env:
MOBILITYOPS_PUBLIC_URL: https://fleetops.itworx.tech
run: npx playwright test --config=playwright.live.config.ts --project=chromium
- name: Upload failure evidence
if: failure()
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3; Gitea-compatible artifact protocol
with:
name: browser-canary-failure
path: |
frontend/playwright-live-report
frontend/test-results
if-no-files-found: ignore
+89 -81
View File
@@ -1,133 +1,141 @@
name: MobilityOps acceptance name: MobilityOps acceptance
on: on:
push:
branches: [master]
pull_request: pull_request:
schedule:
- cron: "17 3 * * 1" concurrency:
group: mobilityops-ci-${{ gitea.repository }}-${{ gitea.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs: jobs:
backend: acceptance:
runs-on: ubuntu-latest # Never execute code from an untrusted public fork on the private runner.
if: ${{ gitea.event.pull_request.head.repo.full_name == gitea.repository }}
runs-on: linux-validation
timeout-minutes: 60
steps: steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Secret scan
uses: trufflesecurity/trufflehog@b9dd330365132cd2d01dd5dc8a857a056a2544e1 # v3.79.0
with: with:
path: ./ fetch-depth: 0
extra_args: --only-verified - name: Determine validation scope
id: scope
shell: bash
run: |
base_sha="${{ gitea.event.pull_request.base.sha }}"
if git diff --quiet "$base_sha...HEAD" -- . ':(exclude).gitea/workflows/**'; then
echo "full=false" >> "$GITEA_OUTPUT"
echo "Workflow-only change: the protected lightweight gate is sufficient."
else
echo "full=true" >> "$GITEA_OUTPUT"
echo "Product or test change: running the complete acceptance gate."
fi
- name: Secret scan
shell: bash
run: |
set -euo pipefail
repository="$PWD"
source="file:///repo"
workspace=(-v "$repository:/repo" -w /repo)
if docker inspect "${HOSTNAME:-}" >/dev/null 2>&1; then
source="file://$repository"
workspace=(--volumes-from "$HOSTNAME" -w "$repository")
fi
docker run --rm "${workspace[@]}" \
ghcr.io/trufflesecurity/trufflehog@sha256:7104dbb84d1ad2f5f6fa1134e92c6aa6f701f0a4ac2efd5a4c5c96225d899fe3 \
git "$source" --fail --no-update --github-actions --only-verified
- name: Backend tests in isolated PostgreSQL stack - name: Backend tests in isolated PostgreSQL stack
if: steps.scope.outputs.full == 'true'
run: sh scripts/run-isolated-tests.sh run: sh scripts/run-isolated-tests.sh
- name: Backend static checks - name: Backend static and contract checks
if: steps.scope.outputs.full == 'true'
run: | run: |
docker compose -p mobilityops-ci -f compose.yaml -f compose.test.yaml run --build --rm api ruff check app tests docker compose -p mobilityops-ci -f compose.yaml -f compose.test.yaml run --build --rm api ruff check app tests
docker compose -p mobilityops-ci -f compose.yaml -f compose.test.yaml run --rm api mypy app docker compose -p mobilityops-ci -f compose.yaml -f compose.test.yaml run --rm api mypy app
- name: Contract drift gate
run: |
docker compose -p mobilityops-ci -f compose.yaml -f compose.test.yaml run --rm \ docker compose -p mobilityops-ci -f compose.yaml -f compose.test.yaml run --rm \
-v "$PWD:/repo:ro" api python /repo/scripts/check-contracts.py api python scripts/check-contracts.py
python scripts/check-source-budgets.py python scripts/check-source-budgets.py
- name: Build production API image for vulnerability scan - name: Build production API image for vulnerability scan
if: steps.scope.outputs.full == 'true'
run: | run: |
docker build --target runtime --build-arg VCS_REF="$GITHUB_SHA" \ docker build --target runtime --build-arg VCS_REF="$GITHUB_SHA" \
--tag mobilityops-api-ci --file backend/Dockerfile . --tag mobilityops-api-ci --file backend/Dockerfile .
- name: Production API image vulnerability scan (HIGH/CRITICAL) - name: Production API image vulnerability scan (HIGH/CRITICAL)
uses: aquasecurity/trivy-action@6c175e9c4083a92bbca2f9724c8a5e33bc2d97a5 # v0.30.0 if: steps.scope.outputs.full == 'true'
with: run: bash scripts/scan-ci-image.sh mobilityops-api-ci
scan-type: image
image-ref: mobilityops-api-ci
format: table
severity: HIGH,CRITICAL
exit-code: "1"
ignore-unfixed: true
- name: Build production web image for vulnerability scan - name: Build production web image for vulnerability scan
if: steps.scope.outputs.full == 'true'
run: | run: |
docker build --build-arg VCS_REF="$GITHUB_SHA" \ docker build --build-arg VCS_REF="$GITHUB_SHA" \
--tag mobilityops-web-ci frontend --tag mobilityops-web-ci frontend
- name: Production web image vulnerability scan (HIGH/CRITICAL) - name: Production web image vulnerability scan (HIGH/CRITICAL)
uses: aquasecurity/trivy-action@6c175e9c4083a92bbca2f9724c8a5e33bc2d97a5 # v0.30.0 if: steps.scope.outputs.full == 'true'
with: run: bash scripts/scan-ci-image.sh mobilityops-web-ci
scan-type: image
image-ref: mobilityops-web-ci
format: table
severity: HIGH,CRITICAL
exit-code: "1"
ignore-unfixed: true
- name: Remove CI stack
if: always()
run: docker compose -p mobilityops-ci -f compose.yaml -f compose.test.yaml down -v --remove-orphans
frontend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
if: steps.scope.outputs.full == 'true'
with: with:
node-version: 22 node-version: 22
cache: npm cache: npm
cache-dependency-path: frontend/package-lock.json cache-dependency-path: frontend/package-lock.json
- name: Install locked dependencies - name: Install frontend dependencies once
if: steps.scope.outputs.full == 'true'
working-directory: frontend working-directory: frontend
run: npm ci --no-audit --no-fund run: npm ci --no-audit --no-fund
- name: Lint (tsc + ESLint with react-hooks and jsx-a11y) - name: Frontend lint, build, budget and dependency audit
if: steps.scope.outputs.full == 'true'
working-directory: frontend working-directory: frontend
run: npm run lint run: |
- name: Typecheck and production build npm run lint
working-directory: frontend npm run build
run: npm run build && npm run budget npm run budget
- name: Dependency audit npm audit --audit-level=high
working-directory: frontend
run: npm audit --audit-level=high
e2e:
# The five-minute Playwright demo is part of the definition of done
# (docs/14-testing-and-acceptance.md); run it against the real Compose stack.
runs-on: ubuntu-latest
needs: [backend, frontend]
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 22
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Start the demo stack - name: Start the demo stack
if: steps.scope.outputs.full == 'true'
run: | run: |
cp .env.example .env cp .env.example .env
# Acceptance tests intentionally reset their isolated demo dataset per scenario.
printf '\nDEMO_RESET_COOLDOWN_SECONDS=0\n' >> .env
docker compose -p mobilityops-e2e up --build -d db api web docker compose -p mobilityops-e2e up --build -d db api web
docker network connect mobilityops-e2e_mobilityops "$HOSTNAME"
for _attempt in $(seq 1 60); do for _attempt in $(seq 1 60); do
if curl -fsS http://localhost:1228/health/ready >/dev/null 2>&1; then break; fi if curl -fsS http://web/health/ready >/dev/null 2>&1; then break; fi
sleep 2 sleep 2
done done
curl -fsS http://localhost:1228/health/ready curl -fsS http://web/health/ready
docker compose -p mobilityops-e2e exec -T api python -m app.cli seed --reset docker compose -p mobilityops-e2e exec -T api python -m app.cli seed --reset
- name: Install Playwright - name: Install acceptance browsers
if: steps.scope.outputs.full == 'true'
working-directory: frontend working-directory: frontend
run: npx playwright install --with-deps chromium
- name: Run browser acceptance and live smoke suites
if: steps.scope.outputs.full == 'true'
working-directory: frontend
env:
MOBILITYOPS_PUBLIC_URL: http://web
run: | run: |
npm ci --no-audit --no-fund # Pixel baselines are workstation/rendering specific; keep the PR gate functional.
npx playwright install --with-deps chromium firefox npx playwright test --grep-invert "visual hierarchy"
- name: Run browser acceptance suite npx playwright test --config=playwright.live.config.ts --project=chromium
working-directory: frontend - name: Run concurrent persisted-read smoke
env: if: steps.scope.outputs.full == 'true'
MOBILITYOPS_PUBLIC_URL: http://localhost:1228 run: python scripts/run-readonly-load-smoke.py --base-url http://web
run: npx playwright test
- name: Run non-destructive Chromium and Firefox smoke suite
working-directory: frontend
env:
MOBILITYOPS_PUBLIC_URL: http://localhost:1228
run: npx playwright test --config=playwright.live.config.ts
- name: Upload Playwright report - name: Upload Playwright report
if: failure() if: failure() && steps.scope.outputs.full == 'true'
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3; Gitea-compatible artifact protocol
with: with:
name: playwright-report name: playwright-report
path: | path: |
frontend/playwright-report frontend/playwright-report
frontend/playwright-live-report frontend/playwright-live-report
if-no-files-found: ignore
- name: Stack logs on failure - name: Stack logs on failure
if: failure() if: failure() && steps.scope.outputs.full == 'true'
run: docker compose -p mobilityops-e2e logs --tail=200 api web run: docker compose -p mobilityops-e2e logs --tail=200 api web
- name: Remove e2e stack - name: Remove CI stacks
if: always() if: always() && steps.scope.outputs.full == 'true'
run: docker compose -p mobilityops-e2e down -v --remove-orphans run: |
docker network disconnect mobilityops-e2e_mobilityops "$HOSTNAME" 2>/dev/null || true
docker compose -p mobilityops-e2e down -v --remove-orphans
docker compose -p mobilityops-ci -f compose.yaml -f compose.test.yaml down -v --remove-orphans
+31
View File
@@ -0,0 +1,31 @@
name: MobilityOps live probe
on:
schedule:
- cron: "7 * * * *"
workflow_dispatch:
concurrency:
group: mobilityops-live-probe
cancel-in-progress: true
permissions:
contents: read
jobs:
public-probe:
runs-on: linux-validation
timeout-minutes: 3
steps:
- name: Verify HTTPS readiness and certificate horizon
run: |
curl --fail --silent --show-error --retry 3 https://fleetops.itworx.tech/health/ready
openssl s_client -servername fleetops.itworx.tech -connect fleetops.itworx.tech:443 </dev/null 2>/dev/null \
| openssl x509 -checkend 1209600 -noout
- name: Report successful external heartbeat
env:
HEARTBEAT_URL: ${{ secrets.LIVE_CANARY_HEARTBEAT_URL }}
run: |
if [ -n "$HEARTBEAT_URL" ]; then
curl --fail --silent --show-error --retry 3 "$HEARTBEAT_URL"
fi
+119
View File
@@ -0,0 +1,119 @@
name: Managed validation
on:
push:
branches: [master]
workflow_dispatch:
inputs:
profile:
description: Allowlisted validation profile
required: true
default: full
type: choice
options: [test, lint, typecheck, build, security, full]
permissions:
contents: read
concurrency:
group: managed-validation-${{ gitea.repository }}-${{ gitea.ref }}
cancel-in-progress: true
jobs:
full:
name: full
runs-on: linux-validation
timeout-minutes: 30
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Validate repository with a bounded profile
shell: bash
env:
REQUESTED_PROFILE: ${{ inputs.profile }}
run: |
set -euo pipefail
profile="${REQUESTED_PROFILE:-full}"
case "${profile}" in
test|lint|typecheck|build|security|full) ;;
*) echo "Profile is not allowlisted" >&2; exit 2 ;;
esac
git diff --check
if git grep -nE '^(<<<<<<< |=======$|>>>>>>> )' -- . ':!*.lock' ':!*.patch'; then
echo "Unresolved merge markers detected" >&2
exit 1
fi
if [[ -f pyproject.toml || -f requirements.txt ]]; then
# Compile only tracked Python sources. Running compileall after a
# Node install would otherwise traverse node_modules and turn a
# lightweight baseline into a large runner workload.
git ls-files -z '*.py' | xargs -0 -r python -m py_compile
if [[ -f uv.lock ]]; then
python -m venv "${RUNNER_TEMP}/managed-uv"
uv_python="${RUNNER_TEMP}/managed-uv/bin/python"
"${uv_python}" -m pip install --disable-pip-version-check uv==0.10.0
managed_uv="${RUNNER_TEMP}/managed-uv/bin/uv"
export UV_PROJECT_ENVIRONMENT="${RUNNER_TEMP}/managed-project-venv"
"${managed_uv}" sync --locked
export PATH="${UV_PROJECT_ENVIRONMENT}/bin:${PATH}"
if [[ "${profile}" == test || "${profile}" == full ]]; then
if "${managed_uv}" run python -c 'import pytest' 2>/dev/null; then
"${managed_uv}" run python -m pytest
fi
fi
if [[ "${profile}" == lint || "${profile}" == full ]]; then
if "${managed_uv}" run python -c 'import ruff' 2>/dev/null; then
"${managed_uv}" run python -m ruff check .
fi
fi
elif [[ -f requirements.txt ]]; then
python -m venv "${RUNNER_TEMP}/managed-python"
managed_python="${RUNNER_TEMP}/managed-python/bin/python"
"${managed_python}" -m pip install --disable-pip-version-check -r requirements.txt
export PATH="${RUNNER_TEMP}/managed-python/bin:${PATH}"
if [[ "${profile}" == test || "${profile}" == full ]]; then
if "${managed_python}" -c 'import pytest' 2>/dev/null; then
"${managed_python}" -m pytest
fi
fi
fi
fi
# Prepare Python before invoking Node scripts. Polyglot repositories
# commonly delegate their test script to Python and need the managed
# virtual environment to be active first.
if [[ -f package.json ]]; then
corepack enable
if [[ -f pnpm-lock.yaml ]]; then
pnpm install --frozen-lockfile
[[ "${profile}" == test || "${profile}" == full ]] && pnpm --if-present test
[[ "${profile}" == lint || "${profile}" == full ]] && pnpm --if-present lint
[[ "${profile}" == typecheck || "${profile}" == full ]] && pnpm --if-present typecheck
[[ "${profile}" == build || "${profile}" == full ]] && pnpm --if-present build
elif [[ -f package-lock.json ]]; then
npm ci
[[ "${profile}" == test || "${profile}" == full ]] && npm run --if-present test
[[ "${profile}" == lint || "${profile}" == full ]] && npm run --if-present lint
if [[ "${profile}" == typecheck || "${profile}" == full ]]; then
npm run --if-present typecheck
fi
[[ "${profile}" == build || "${profile}" == full ]] && npm run --if-present build
fi
fi
if [[ -f go.mod ]]; then
if [[ "${profile}" == test || "${profile}" == build || "${profile}" == full ]]; then
go test ./...
fi
fi
if [[ -f Cargo.toml ]]; then
if [[ "${profile}" == test || "${profile}" == build || "${profile}" == full ]]; then
cargo test --locked
fi
fi
if compgen -G '*.sln' >/dev/null; then
if [[ "${profile}" == test || "${profile}" == build || "${profile}" == full ]]; then
dotnet test --configuration Release
fi
fi
+25 -17
View File
@@ -6,37 +6,45 @@ on:
jobs: jobs:
release-evidence: release-evidence:
runs-on: ubuntu-latest runs-on: linux-validation
steps: steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Build commit-labelled release images - name: Build commit-labelled release images
run: | run: |
docker build --target runtime --build-arg VCS_REF="$GITHUB_SHA" --tag mobilityops-api-release --file backend/Dockerfile . docker build --target runtime --build-arg VCS_REF="$GITHUB_SHA" --tag mobilityops-api-release --file backend/Dockerfile .
docker build --build-arg VCS_REF="$GITHUB_SHA" --tag mobilityops-web-release frontend docker build --build-arg VCS_REF="$GITHUB_SHA" --tag mobilityops-web-release frontend
- name: Generate API CycloneDX SBOM docker build --build-arg VCS_REF="$GITHUB_SHA" --tag mobilityops-backup-tools-release --file deploy/unraid/Dockerfile.backup-tools .
uses: aquasecurity/trivy-action@6c175e9c4083a92bbca2f9724c8a5e33bc2d97a5 # v0.30.0 - name: Scan all release images
with: run: |
scan-type: image for image in mobilityops-api-release mobilityops-web-release mobilityops-backup-tools-release; do
image-ref: mobilityops-api-release docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
format: cyclonedx -v mobilityops-release-trivy:/root/.cache/ \
output: mobilityops-api-sbom.cdx.json aquasec/trivy:0.74.0@sha256:62b1e65e8869bc4b4c6aa4fa2b21595256c7c2f6018a9d9ad61caf87187c1969 \
- name: Generate web CycloneDX SBOM image --scanners vuln --severity HIGH,CRITICAL \
uses: aquasecurity/trivy-action@6c175e9c4083a92bbca2f9724c8a5e33bc2d97a5 # v0.30.0 --ignore-unfixed --exit-code 1 "$image"
with: done
scan-type: image - name: Generate CycloneDX SBOMs with the pinned scanner image
image-ref: mobilityops-web-release run: |
format: cyclonedx for component in api web backup-tools; do
output: mobilityops-web-sbom.cdx.json docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
-v "$PWD:/work" -w /work \
aquasec/trivy:0.74.0@sha256:62b1e65e8869bc4b4c6aa4fa2b21595256c7c2f6018a9d9ad61caf87187c1969 \
image --format cyclonedx --output "mobilityops-${component}-sbom.cdx.json" \
"mobilityops-${component}-release"
done
- name: Record immutable image metadata - name: Record immutable image metadata
run: | run: |
docker image inspect mobilityops-api-release > mobilityops-api-image.json docker image inspect mobilityops-api-release > mobilityops-api-image.json
docker image inspect mobilityops-web-release > mobilityops-web-image.json docker image inspect mobilityops-web-release > mobilityops-web-image.json
sha256sum mobilityops-*-sbom.cdx.json mobilityops-*-image.json > SHA256SUMS docker image inspect mobilityops-backup-tools-release > mobilityops-backup-tools-image.json
python scripts/generate-release-provenance.py
sha256sum mobilityops-*-sbom.cdx.json mobilityops-*-image.json release-provenance.json > SHA256SUMS
- name: Upload release evidence - name: Upload release evidence
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3; Gitea-compatible artifact protocol
with: with:
name: mobilityops-${{ github.ref_name }}-evidence name: mobilityops-${{ github.ref_name }}-evidence
path: | path: |
mobilityops-*-sbom.cdx.json mobilityops-*-sbom.cdx.json
mobilityops-*-image.json mobilityops-*-image.json
release-provenance.json
SHA256SUMS SHA256SUMS
+42
View File
@@ -0,0 +1,42 @@
name: MobilityOps security
on:
schedule:
- cron: "17 3 * * 1"
workflow_dispatch:
concurrency:
group: mobilityops-security
cancel-in-progress: true
permissions:
contents: read
jobs:
images:
runs-on: linux-validation
timeout-minutes: 30
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 0
- name: Build production images once
run: |
docker build --target runtime --build-arg VCS_REF="$GITHUB_SHA" \
--tag mobilityops-api-ci --file backend/Dockerfile .
docker build --build-arg VCS_REF="$GITHUB_SHA" \
--tag mobilityops-web-ci frontend
- name: Scan production images for fixed HIGH and CRITICAL vulnerabilities
run: |
for image in mobilityops-api-ci mobilityops-web-ci; do
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
docker.io/aquasec/trivy@sha256:be1190afcb28352bfddc4ddeb71470835d16462af68d310f9f4bca710961a41e \
image --severity HIGH,CRITICAL --exit-code 1 --ignore-unfixed --no-progress "$image"
done
- name: Scan repository secrets and misconfiguration
uses: docker://docker.io/aquasec/trivy@sha256:be1190afcb28352bfddc4ddeb71470835d16462af68d310f9f4bca710961a41e
with:
args: fs --scanners misconfig,secret --exit-code 1 --no-progress .
- name: Remove temporary image tags
if: always()
run: docker image rm mobilityops-api-ci mobilityops-web-ci || true
+54 -13
View File
@@ -1,25 +1,66 @@
# Secrets and local configuration
.env .env
.venv/ .env.*
!.env.example
*.pem
*.key
*.p12
*.pfx
secrets/
credentials/
# Python
__pycache__/ __pycache__/
*.py[cod]
.pytest_cache/ .pytest_cache/
.mypy_cache/
.ruff_cache/ .ruff_cache/
.mypy_cache/
.venv/
.coverage
htmlcov/
*.egg-info/
# Frontend and test output
node_modules/ node_modules/
dist/ frontend/node_modules/
frontend/dist/
coverage/ coverage/
playwright-report/ playwright-report/
playwright-live-report/
test-results/ test-results/
*.pyc *.tsbuildinfo
.DS_Store
# Runtime data and local infrastructure state
.state/
data/
backups/local/
*.db
*.db-shm
*.db-wal
*.sqlite
*.sqlite-shm
*.sqlite-wal
*.log
*.tmp
# Generated release/design evidence. Maintained documentation belongs in docs/.
artifacts/**/final-summary.md
artifacts/deployment/
artifacts/design-validation/current/
artifacts/**/screenshots/generated/
# Local AI/editor state
.codex/
.claude/
.agents/
.dyad/
.idea/ .idea/
.vscode/ .vscode/
*.tsbuildinfo .vs/
*.zip .DS_Store
*.tar.gz Thumbs.db
# Local Claude/Codex per-user settings and scratch archives # Archives and local release bundles
.claude/settings.local.json *.zip
*.tar
*.tar.gz
*.tgz *.tgz
*.dump
backups/
+4
View File
@@ -0,0 +1,4 @@
81e3fd63bdbcb2e9c4ae1d709ea46f40537b6f62:backend/tests/test_data_quality.py:generic-api-key:869
0091c57c7fd82ffda16da1edfe5fa5589c8f1e13:backend/tests/test_return.py:generic-api-key:93
0091c57c7fd82ffda16da1edfe5fa5589c8f1e13:backend/tests/test_return.py:generic-api-key:122
0091c57c7fd82ffda16da1edfe5fa5589c8f1e13:backend/tests/test_return.py:generic-api-key:127
-61
View File
@@ -1,61 +0,0 @@
# Binding instructions for Claude
## Operating mode
Work autonomously. Do not ask the user product, architecture, naming, UI, scope or implementation questions already answered in this repository. Record a reasonable assumption in an ADR only when a genuine gap blocks implementation.
Use normal or medium reasoning for routine work. Reserve high reasoning for an actual cross-service design conflict or a persistent failure after evidence-driven debugging.
Continue from milestone to milestone until every acceptance criterion is satisfied. Do not stop merely because one milestone is complete.
## Token and tool efficiency
1. Read `START_HERE.md`, this file, `PROJECT_STATE.md` and `docs/15-build-plan.md` first.
2. Read only the milestone-specific documents named in the build plan.
3. Do not repeatedly reread all documentation.
4. Keep explanations terse; spend effort on implementation and validation.
5. Update `PROJECT_STATE.md` after each milestone with decisions, commands, evidence and the exact next action.
6. Prefer focused file inspection and targeted tests over broad repository scans.
7. Do not generate large speculative documents after implementation starts.
## Scope discipline
- Build the locked PoC only.
- Do not add accounting, payments, public reservations, a generic CRM, inventory, HR, a second RAG stack, a separate MCP server or autonomous write actions.
- Do not modify the RAGcore or ITWorx MCP Hub repositories. Integrate only through documented contracts and configurable adapters.
- Keep critical business rules in MobilityOps code, not in n8n or prompts.
- No direct MCP Hub or RAGcore access to the MobilityOps database.
## Product quality
- No dead buttons, empty routes, unexplained placeholders or hardcoded dashboard metrics.
- Every visible number must derive from persisted data.
- All important state changes must be audited.
- AI must never invent an answer when RAGcore is unavailable or returns insufficient evidence.
- Vehicle returns must commit locally even when n8n is unavailable; orchestration becomes pending and retryable.
- External dependencies require timeouts, bounded retries, health state and graceful degradation.
- Demo data must be clearly labelled synthetic.
## Engineering rules
- Backend: Python, FastAPI, SQLAlchemy 2, Alembic, PostgreSQL.
- Frontend: React, TypeScript, Vite, accessible responsive UI.
- Validation: Pydantic at API boundaries and database constraints for invariants.
- Use UUID primary keys internally and stable human-readable public references.
- Store UTC timestamps; render Europe/Brussels in the UI.
- API paths start with `/api/v1`.
- Use an outbox record for reliable post-commit n8n delivery.
- Tests must cover domain rules, API contracts and the five-minute Playwright demo.
- Generate and commit dependency lockfiles.
## Git workflow
Create one coherent commit per milestone after its validation passes. Suggested message format:
`M1: implement operational core`
Never rewrite already accepted milestone history unless necessary to fix a regression.
## Definition of done
The project is done only when `docs/14-testing-and-acceptance.md` passes from a clean checkout and `PROJECT_STATE.md` contains the final evidence summary.
+12
View File
@@ -0,0 +1,12 @@
# Contributing
MobilityOps contributions must preserve fleet-data privacy, deterministic demo behaviour and the fail-closed integration boundaries documented in `SECURITY.md`.
- use synthetic vehicles, customers, bookings, returns, telematics events and identity claims in tests and screenshots;
- never commit production databases, exports, operator inventories, private service URLs, tokens, backups or generated browser evidence;
- keep external integrations configurable through environment variables or explicit deployment configuration;
- document new personal-data fields, retention, authorization, audit and deletion/export behaviour;
- add negative tests for authentication, authorization, duplicate handling, webhook validation, path containment and stale/unavailable providers;
- review dependencies, images and browser assets for provenance and redistribution rights.
Run the relevant backend, frontend, migration, integration, Compose and managed-validation gates before review. Security-sensitive findings belong through the private process in `SECURITY.md`.
-368
View File
@@ -1,368 +0,0 @@
# File index
Tracked source, contract, documentation and configuration files. Release evidence and
screenshots live under `artifacts/<release>/` and are omitted here for brevity.
Regenerate with `git ls-files` when the tree changes.
- `.env.example`
- `.gitattributes`
- `.gitea/workflows/ci.yml`
- `.gitignore`
- `AGENTS.md`
- `CLAUDE.md`
- `FILE_INDEX.md`
- `MASTER_BUILD_PROMPT.md`
- `Makefile`
- `PROJECT_STATE.md`
- `README.md`
- `START_HERE.md`
- `backend/Dockerfile`
- `backend/alembic.ini`
- `backend/alembic/env.py`
- `backend/alembic/script.py.mako`
- `backend/alembic/versions/0a4c1d2e3f5b_idempotency_request_fingerprint.py`
- `backend/alembic/versions/799d8800e241_outbox_last_error_code.py`
- `backend/alembic/versions/a81d0ce9f662_oidc_identity.py`
- `backend/alembic/versions/b7c7b536df85_operational_user_credentials.py`
- `backend/alembic/versions/b913a72e8c14_customer_privacy_state.py`
- `backend/alembic/versions/c24f6a9d013e_domain_constraints_indexes.py`
- `backend/alembic/versions/c9498525abb5_initial_schema.py`
- `backend/alembic/versions/d1f83bc64170_revoked_sessions.py`
- `backend/alembic/versions/e7b08389f47f_idempotency_records.py`
- `backend/alembic/versions/f43d829ab610_quality_work_queue.py`
- `backend/app/__init__.py`
- `backend/app/api/__init__.py`
- `backend/app/api/deps.py`
- `backend/app/api/routers/__init__.py`
- `backend/app/api/routers/audit.py`
- `backend/app/api/routers/auth.py`
- `backend/app/api/routers/bookings.py`
- `backend/app/api/routers/customers.py`
- `backend/app/api/routers/dashboard.py`
- `backend/app/api/routers/data_quality.py`
- `backend/app/api/routers/demo.py`
- `backend/app/api/routers/integration_status.py`
- `backend/app/api/routers/integrations.py`
- `backend/app/api/routers/knowledge.py`
- `backend/app/api/routers/mcp_integrations.py`
- `backend/app/api/routers/observability.py`
- `backend/app/api/routers/privacy.py`
- `backend/app/api/routers/search.py`
- `backend/app/api/routers/users.py`
- `backend/app/api/routers/vehicles.py`
- `backend/app/api/routers/workflows.py`
- `backend/app/cli.py`
- `backend/app/core/__init__.py`
- `backend/app/core/config.py`
- `backend/app/core/db.py`
- `backend/app/core/errors.py`
- `backend/app/core/observability.py`
- `backend/app/core/ratelimit.py`
- `backend/app/core/security.py`
- `backend/app/main.py`
- `backend/app/models/__init__.py`
- `backend/app/models/audit.py`
- `backend/app/models/booking.py`
- `backend/app/models/customer.py`
- `backend/app/models/data_quality.py`
- `backend/app/models/idempotency.py`
- `backend/app/models/inspection.py`
- `backend/app/models/maintenance.py`
- `backend/app/models/mixins.py`
- `backend/app/models/outbox.py`
- `backend/app/models/revoked_session.py`
- `backend/app/models/user.py`
- `backend/app/models/vehicle.py`
- `backend/app/schemas.py`
- `backend/app/seed_loader.py`
- `backend/app/services/__init__.py`
- `backend/app/services/audit.py`
- `backend/app/services/data_quality.py`
- `backend/app/services/demo_manifest.py`
- `backend/app/services/dispatcher.py`
- `backend/app/services/integration_status.py`
- `backend/app/services/knowledge/__init__.py`
- `backend/app/services/knowledge/demo.py`
- `backend/app/services/knowledge/procedures.py`
- `backend/app/services/knowledge/ragcore.py`
- `backend/app/services/operations.py`
- `backend/app/services/returns.py`
- `backend/app/services/sessions.py`
- `backend/app/services/vehicle_status.py`
- `backend/entrypoint.sh`
- `backend/pyproject.toml`
- `backend/requirements.lock`
- `backend/scripts/generate_openapi.py`
- `backend/tests/conftest.py`
- `backend/tests/test_audit.py`
- `backend/tests/test_auth.py`
- `backend/tests/test_bookings.py`
- `backend/tests/test_dashboard.py`
- `backend/tests/test_data_quality.py`
- `backend/tests/test_database_constraints.py`
- `backend/tests/test_demo_manifest.py`
- `backend/tests/test_dispatcher.py`
- `backend/tests/test_hardening.py`
- `backend/tests/test_health.py`
- `backend/tests/test_integration_status.py`
- `backend/tests/test_integrations.py`
- `backend/tests/test_knowledge.py`
- `backend/tests/test_mcp_integrations.py`
- `backend/tests/test_migrations.py`
- `backend/tests/test_observability.py`
- `backend/tests/test_operational_auth.py`
- `backend/tests/test_privacy.py`
- `backend/tests/test_return.py`
- `backend/tests/test_search.py`
- `backend/tests/test_seed.py`
- `backend/tests/test_users.py`
- `backend/tests/test_vehicle_status.py`
- `backend/tests/test_vehicles.py`
- `backend/tests/test_workflows.py`
- `compose.observability.yaml`
- `compose.test.yaml`
- `compose.unraid.yaml`
- `compose.yaml`
- `contracts/events.schema.json`
- `contracts/mcp-tools.json`
- `contracts/openapi.yaml`
- `contracts/ragcore-contract-assumptions.md`
- `deploy/observability/alerts.yml`
- `deploy/observability/grafana/dashboards/mobilityops-overview.json`
- `deploy/observability/grafana/provisioning/dashboards/mobilityops.yml`
- `deploy/observability/grafana/provisioning/datasources/prometheus.yml`
- `deploy/observability/prometheus.yml`
- `deploy/unraid/README.md`
- `deploy/unraid/backup-postgres.sh`
- `deploy/unraid/configure-env.sh`
- `deploy/unraid/prune-postgres-backups.sh`
- `deploy/unraid/restore-postgres.sh`
- `deploy/unraid/scheduled-backup.sh`
- `deploy/unraid/setup-existing-n8n.sh`
- `deploy/unraid/setup-n8n.sh`
- `deploy/unraid/setup-scheduled-scan.sh`
- `deploy/unraid/verify-postgres-backups.sh`
- `docs/00-product-brief.md`
- `docs/01-scope-and-non-goals.md`
- `docs/02-user-stories.md`
- `docs/03-architecture.md`
- `docs/04-domain-model.md`
- `docs/05-api-contract.md`
- `docs/06-ui-ux.md`
- `docs/07-data-quality.md`
- `docs/08-return-workflow.md`
- `docs/09-ragcore-integration.md`
- `docs/10-mcp-hub-integration.md`
- `docs/11-n8n-integration.md`
- `docs/12-security-and-audit.md`
- `docs/13-seed-and-demo-scenarios.md`
- `docs/14-testing-and-acceptance.md`
- `docs/15-build-plan.md`
- `docs/16-portfolio-case-study.md`
- `docs/17-runbook.md`
- `docs/18-privacy-governance.md`
- `docs/19-visual-product-roadmap.md`
- `docs/deferred.md`
- `docs/demo-release/current-demo-gap-audit.md`
- `docs/demo-release/demo-concept.md`
- `docs/demo-release/demo-data.md`
- `docs/demo-release/demo-guide.md`
- `docs/demo-release/demo-runbook.md`
- `docs/demo-release/demo-scenarios.md`
- `docs/design/current-ux-audit.md`
- `docs/design/design-directions.md`
- `docs/design/design-system.md`
- `docs/design/implementation-validation.md`
- `docs/design/stitch-manifest.md`
- `docs/final-integrations/ai-operations-brief-runbook.md`
- `docs/final-integrations/current-state-audit.md`
- `docs/final-product-polish/audit.md`
- `docs/fleet-ops-correction/current-gap-audit.md`
- `docs/fleet-ops-correction/i18n-inventory.md`
- `docs/fleet-ops-correction/vehicle-status-decision-table.md`
- `docs/fleet-ops-final-localization/audit.md`
- `docs/functional-completion/current-functional-audit.md`
- `docs/functional-completion/server-baseline.md`
- `docs/live-ai-integration/n8n-current-state.md`
- `frontend/Dockerfile`
- `frontend/e2e/_capture-demo-screenshots.spec.ts`
- `frontend/e2e/_capture-recruiter-screenshots.spec.ts`
- `frontend/e2e/_capture-screenshots.spec.ts`
- `frontend/e2e/clickable-rows.spec.ts`
- `frontend/e2e/demo-accessibility.spec.ts`
- `frontend/e2e/demo-entry.spec.ts`
- `frontend/e2e/demo-guide.spec.ts`
- `frontend/e2e/demo-legibility.spec.ts`
- `frontend/e2e/demo.spec.ts`
- `frontend/e2e/error-messages.spec.ts`
- `frontend/e2e/fleet-ops-correction.spec.ts`
- `frontend/e2e/greeting-live.spec.ts`
- `frontend/e2e/greeting.spec.ts`
- `frontend/e2e/guided-demo-full.spec.ts`
- `frontend/e2e/i18n-coverage.spec.ts`
- `frontend/e2e/interactive-elements.spec.ts`
- `frontend/e2e/operational-workflows.spec.ts`
- `frontend/e2e/privacy.spec.ts`
- `frontend/e2e/recruiter-polish.spec.ts`
- `frontend/e2e/responsive-i18n.spec.ts`
- `frontend/e2e/roadmap-regression.spec.ts`
- `frontend/e2e/ui-redesign.spec.ts`
- `frontend/index.html`
- `frontend/nginx.conf`
- `frontend/package-lock.json`
- `frontend/package.json`
- `frontend/playwright.config.ts`
- `frontend/public/favicon.svg`
- `frontend/public/og-fleet-ops.svg`
- `frontend/src/App.tsx`
- `frontend/src/api/apiError.ts`
- `frontend/src/api/client.ts`
- `frontend/src/api/errorMessages.ts`
- `frontend/src/api/types.ts`
- `frontend/src/components/Badge.tsx`
- `frontend/src/components/CheckoutForm.tsx`
- `frontend/src/components/DemoBadge.tsx`
- `frontend/src/components/DemoGuide.tsx`
- `frontend/src/components/Icons.tsx`
- `frontend/src/components/LanguageSwitcher.tsx`
- `frontend/src/components/Layout.tsx`
- `frontend/src/components/PageChrome.tsx`
- `frontend/src/components/Pagination.tsx`
- `frontend/src/components/RequireAuth.tsx`
- `frontend/src/components/ReturnForm.tsx`
- `frontend/src/components/VehicleMaintenanceActions.tsx`
- `frontend/src/context/AuthContext.tsx`
- `frontend/src/context/DemoGuideContext.tsx`
- `frontend/src/context/DemoManifestContext.tsx`
- `frontend/src/data/demoGuideSteps.ts`
- `frontend/src/data/evidenceSignals.ts`
- `frontend/src/data/integrationLabels.ts`
- `frontend/src/hooks/useViewportTier.ts`
- `frontend/src/i18n/brusselsDateTime.ts`
- `frontend/src/i18n/config.ts`
- `frontend/src/i18n/format.ts`
- `frontend/src/i18n/greeting.ts`
- `frontend/src/i18n/locales/en-GB/accessibility.json`
- `frontend/src/i18n/locales/en-GB/audit.json`
- `frontend/src/i18n/locales/en-GB/auth.json`
- `frontend/src/i18n/locales/en-GB/bookings.json`
- `frontend/src/i18n/locales/en-GB/common.json`
- `frontend/src/i18n/locales/en-GB/dashboard.json`
- `frontend/src/i18n/locales/en-GB/demo.json`
- `frontend/src/i18n/locales/en-GB/errors.json`
- `frontend/src/i18n/locales/en-GB/fleet.json`
- `frontend/src/i18n/locales/en-GB/integrations.json`
- `frontend/src/i18n/locales/en-GB/knowledge.json`
- `frontend/src/i18n/locales/en-GB/navigation.json`
- `frontend/src/i18n/locales/en-GB/operations.json`
- `frontend/src/i18n/locales/en-GB/privacy.json`
- `frontend/src/i18n/locales/en-GB/quality.json`
- `frontend/src/i18n/locales/en-GB/returns.json`
- `frontend/src/i18n/locales/fr-BE/accessibility.json`
- `frontend/src/i18n/locales/fr-BE/audit.json`
- `frontend/src/i18n/locales/fr-BE/auth.json`
- `frontend/src/i18n/locales/fr-BE/bookings.json`
- `frontend/src/i18n/locales/fr-BE/common.json`
- `frontend/src/i18n/locales/fr-BE/dashboard.json`
- `frontend/src/i18n/locales/fr-BE/demo.json`
- `frontend/src/i18n/locales/fr-BE/errors.json`
- `frontend/src/i18n/locales/fr-BE/fleet.json`
- `frontend/src/i18n/locales/fr-BE/integrations.json`
- `frontend/src/i18n/locales/fr-BE/knowledge.json`
- `frontend/src/i18n/locales/fr-BE/navigation.json`
- `frontend/src/i18n/locales/fr-BE/operations.json`
- `frontend/src/i18n/locales/fr-BE/privacy.json`
- `frontend/src/i18n/locales/fr-BE/quality.json`
- `frontend/src/i18n/locales/fr-BE/returns.json`
- `frontend/src/i18n/locales/nl-BE/accessibility.json`
- `frontend/src/i18n/locales/nl-BE/audit.json`
- `frontend/src/i18n/locales/nl-BE/auth.json`
- `frontend/src/i18n/locales/nl-BE/bookings.json`
- `frontend/src/i18n/locales/nl-BE/common.json`
- `frontend/src/i18n/locales/nl-BE/dashboard.json`
- `frontend/src/i18n/locales/nl-BE/demo.json`
- `frontend/src/i18n/locales/nl-BE/errors.json`
- `frontend/src/i18n/locales/nl-BE/fleet.json`
- `frontend/src/i18n/locales/nl-BE/integrations.json`
- `frontend/src/i18n/locales/nl-BE/knowledge.json`
- `frontend/src/i18n/locales/nl-BE/navigation.json`
- `frontend/src/i18n/locales/nl-BE/operations.json`
- `frontend/src/i18n/locales/nl-BE/privacy.json`
- `frontend/src/i18n/locales/nl-BE/quality.json`
- `frontend/src/i18n/locales/nl-BE/returns.json`
- `frontend/src/i18n/useGreetingPeriod.ts`
- `frontend/src/main.tsx`
- `frontend/src/pages/AboutDemo.tsx`
- `frontend/src/pages/Audit.tsx`
- `frontend/src/pages/Automation.tsx`
- `frontend/src/pages/BookingCreate.tsx`
- `frontend/src/pages/BookingDetail.tsx`
- `frontend/src/pages/Bookings.tsx`
- `frontend/src/pages/Dashboard.tsx`
- `frontend/src/pages/DataQuality.tsx`
- `frontend/src/pages/DataQualityIssueDetail.tsx`
- `frontend/src/pages/Highlights.tsx`
- `frontend/src/pages/Knowledge.tsx`
- `frontend/src/pages/Login.tsx`
- `frontend/src/pages/Privacy.tsx`
- `frontend/src/pages/Scenarios.tsx`
- `frontend/src/pages/Users.tsx`
- `frontend/src/pages/VehicleDetail.tsx`
- `frontend/src/pages/Vehicles.tsx`
- `frontend/src/product.ts`
- `frontend/src/styles.css`
- `frontend/src/vite-env.d.ts`
- `frontend/tsconfig.json`
- `frontend/vite.config.ts`
- `knowledge/manifest.json`
- `knowledge/procedures/en-GB/01-vehicle-checkout.md`
- `knowledge/procedures/en-GB/02-vehicle-return.md`
- `knowledge/procedures/en-GB/03-damage-handling.md`
- `knowledge/procedures/en-GB/04-odometer-anomalies.md`
- `knowledge/procedures/en-GB/05-cleaning-checklist.md`
- `knowledge/procedures/en-GB/06-maintenance-escalation.md`
- `knowledge/procedures/en-GB/07-customer-documents.md`
- `knowledge/procedures/en-GB/08-privacy.md`
- `knowledge/procedures/en-GB/09-booking-conflicts.md`
- `knowledge/procedures/en-GB/10-roles-and-escalation.md`
- `knowledge/procedures/en-GB/11-vehicle-availability.md`
- `knowledge/procedures/fr-BE/01-vehicle-checkout.md`
- `knowledge/procedures/fr-BE/02-vehicle-return.md`
- `knowledge/procedures/fr-BE/03-damage-handling.md`
- `knowledge/procedures/fr-BE/04-odometer-anomalies.md`
- `knowledge/procedures/fr-BE/05-cleaning-checklist.md`
- `knowledge/procedures/fr-BE/06-maintenance-escalation.md`
- `knowledge/procedures/fr-BE/07-customer-documents.md`
- `knowledge/procedures/fr-BE/08-privacy.md`
- `knowledge/procedures/fr-BE/09-booking-conflicts.md`
- `knowledge/procedures/fr-BE/10-roles-and-escalation.md`
- `knowledge/procedures/fr-BE/11-vehicle-availability.md`
- `knowledge/procedures/nl-BE/01-vehicle-checkout.md`
- `knowledge/procedures/nl-BE/02-vehicle-return.md`
- `knowledge/procedures/nl-BE/03-damage-handling.md`
- `knowledge/procedures/nl-BE/04-odometer-anomalies.md`
- `knowledge/procedures/nl-BE/05-cleaning-checklist.md`
- `knowledge/procedures/nl-BE/06-maintenance-escalation.md`
- `knowledge/procedures/nl-BE/07-customer-documents.md`
- `knowledge/procedures/nl-BE/08-privacy.md`
- `knowledge/procedures/nl-BE/09-booking-conflicts.md`
- `knowledge/procedures/nl-BE/10-roles-and-escalation.md`
- `knowledge/procedures/nl-BE/11-vehicle-availability.md`
- `n8n/README.md`
- `n8n/workflows/MANIFEST.md`
- `n8n/workflows/check_drift.py`
- `n8n/workflows/fleet-ops-data-quality-scan.json`
- `n8n/workflows/fleet-ops-error-handler.json`
- `n8n/workflows/fleet-ops-ragcore-procedure-sync.json`
- `n8n/workflows/fleet-ops-vehicle-return.json`
- `n8n/workflows/merge_credential_refs.py`
- `scripts/run-isolated-tests.sh`
- `seed/README.md`
- `seed/bookings.csv`
- `seed/customers.csv`
- `seed/data_quality_issues.csv`
- `seed/generate_seed.py`
- `seed/inspections.csv`
- `seed/maintenance.csv`
- `seed/vehicles.csv`
- `seed/workflow_runs.csv`
-21
View File
@@ -1,21 +0,0 @@
# Paste this once into Claude Code
Build MobilityOps autonomously from this repository.
First read `START_HERE.md`, `CLAUDE.md`, `PROJECT_STATE.md` and `docs/15-build-plan.md`. Treat the repository specifications and contracts as binding. Do not ask me questions that the files already answer, do not broaden the PoC, and do not stop after a milestone.
Implement the milestones in order. For each milestone:
1. read only the documents listed for that milestone;
2. implement the smallest complete solution;
3. run the specified validation plus relevant regression tests;
4. fix failures using evidence rather than guesses;
5. update `PROJECT_STATE.md` with concise evidence and the exact next step;
6. commit the completed milestone;
7. continue immediately.
MobilityOps owns operational data and business rules. RAGcore owns retrieval and grounded answers. ITWorx MCP Hub owns MCP publication and policy. n8n only orchestrates post-commit workflows. Use configurable adapters and working degraded modes so the core demo remains usable when any external service is absent.
The finished PoC must be reproducible from a clean checkout, have no dead UI, use deterministic synthetic data, support the documented five-minute demo, and satisfy every criterion in `docs/14-testing-and-acceptance.md`.
Keep chat output brief. Spend the available context on code, tests, validation and final evidence. Begin now and continue until the repository is complete.
-3118
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -76,6 +76,7 @@ make test # isolated PostgreSQL backend suite
make lint # Ruff + strict mypy make lint # Ruff + strict mypy
make e2e # complete Playwright browser acceptance make e2e # complete Playwright browser acceptance
cd frontend && npm run build cd frontend && npm run build
python scripts/run-readonly-load-smoke.py # while the demo stack is running
``` ```
Release-scoped results and production evidence are recorded in [artifacts/final-acceptance/summary.md](artifacts/final-acceptance/summary.md); older milestone evidence remains explicitly historical. [PROJECT_STATE.md](PROJECT_STATE.md) records the commands and exact deployment revision. Release-scoped results and production evidence are recorded in [artifacts/final-acceptance/summary.md](artifacts/final-acceptance/summary.md); older milestone evidence remains explicitly historical. [PROJECT_STATE.md](PROJECT_STATE.md) records the commands and exact deployment revision.
+1 -1
View File
@@ -15,7 +15,7 @@ release line only.
Do not disclose suspected vulnerabilities through a public issue. Do not disclose suspected vulnerabilities through a public issue.
Report them privately to `jens@itworx.tech` with: Report them privately to `security@itworx.tech` with:
- the affected revision, endpoint or component; - the affected revision, endpoint or component;
- reproduction steps and prerequisites; - reproduction steps and prerequisites;
-39
View File
@@ -1,39 +0,0 @@
# MobilityOps — Claude build pack
MobilityOps is a deliberately scoped, working proof of concept for a fictitious Belgian mobility-rental company. It demonstrates how operational data, workflow automation, data-quality review, a central RAG service and a central MCP hub can work together without pretending to be a full ERP.
## Use this pack efficiently
1. Extract this archive into a new repository.
2. Open the repository in Claude Code.
3. Paste the contents of `MASTER_BUILD_PROMPT.md` once.
4. Let Claude continue autonomously through the milestones in `docs/15-build-plan.md`.
5. Only intervene if the environment itself is unavailable or credentials for an external service are required.
Claude must use `PROJECT_STATE.md` as its compact memory between sessions. Do not restate the full project in later prompts.
## What is included
- locked product scope and non-goals;
- architecture and domain decisions;
- API and event contracts;
- realistic deterministic synthetic seed data;
- eleven fictitious procedures for RAGcore (nl-BE, en-GB, fr-BE);
- an initial n8n workflow export;
- MCP tool definitions for ITWorx MCP Hub;
- a minimal bootable frontend/API scaffold;
- acceptance criteria and a five-minute demo script;
- a master prompt optimized for one autonomous build run.
## Core responsibilities
| Component | Responsibility |
|---|---|
| MobilityOps | Operational data, business rules, UI, audit and data-quality review |
| RAGcore | Document ingestion, retrieval and source-grounded answers |
| ITWorx MCP Hub | Controlled AI access to MobilityOps tools and MobilityOps knowledge |
| n8n | Cross-system orchestration after committed domain events |
## Build principle
Finish a small vertical slice completely. No placeholder pages, fake buttons, invented production claims or scope expansion.
+14
View File
@@ -0,0 +1,14 @@
# Generated live deployment and demo evidence must stay outside source control.
demo-release/
deployment/
screenshots/
*.log
*.db
*.db-wal
*.db-shm
*.zip
*.tar
*.tar.gz
# Keep this policy file.
!.gitignore
-195
View File
@@ -1,195 +0,0 @@
# Demo-productization final summary
## Branches and commits
- **Gitea repository**: `ssh://git@192.168.10.150:222/Jens/MobilityOps.git` (browsable at
`http://192.168.10.150:3000/Jens/MobilityOps`)
- **Branch**: `feat/mobilityops-functional-completion` (no new branch created; no merge
to `main`; no rebase/reset/squash/force-push; full git history preserved, as required)
- **Start commit** (functional-completion baseline, already accepted):
`e0c7ed60112510687627d20a957af91c8b9db7f8`
- **Final commit**: `4a268c73515dc4f1d56c1aa2f231714654bffbb8` — verified via
`git rev-parse HEAD` on `feat/mobilityops-functional-completion` and confirmed to match
`/mnt/user/appdata/mobilityops/.deploy/source-revision` on the Unraid server exactly.
(This corrects a self-reference gap in the immediately preceding pair of commits, which
necessarily could not know their own hash at the time they were written; this is now
the single, unambiguous, verified reference. The repository's primary branch is
`master`, not `main` — no branch named `main` exists in this repository.)
- **Live URL**: `http://192.168.10.150:1236`
## Demo organisation and context
**Northstar Mobility** — a fictitious Belgian camper/van rental company (~50 vehicles,
one main location, rental team, an Operations Manager, a small workshop). This name was
already a locked internal decision (`ragcore_tenant: northstar-mobility-demo`,
`PROJECT_STATE.md`'s "Locked decisions") before this work — this pass surfaces it in the
UI rather than inventing it. Full concept: `docs/demo-release/demo-concept.md`.
## Roles
- **Operations Manager** — full access: data-quality resolution, workflow retries, audit
trail, demo reset, the Demo Guide.
- **Rental Employee** — scoped access: bookings, returns, fleet, knowledge assistant.
Both are reachable from the login screen with no password.
## Demo Guide
An 8-step, sessionStorage-persisted guided tour (Operations-Manager-only, since every
step requires that role). Full design: `docs/demo-release/demo-guide.md`. Steps: (1)
understand operational state, (2) open the booking needing attention, (3) process the
odometer-anomaly return, (4) handle the created data-quality issue, (5) merge the
duplicate customer, (6) ask the knowledge assistant, (7) check automation + audit, (8)
review real vs. synthetic vs. not-connected.
## Scenarios (all 5, full detail in `docs/demo-release/demo-scenarios.md`)
| # | Scenario | Fixed records | Role |
|---|---|---|---|
| 1 | Odometer regression on return | `BK-DEMO-RETURN` / `MO-024` | Either |
| 2 | Possible duplicate customer | `CUS-0012` / `CUS-0178` / `DQ-DEMO-DUPLICATE` | OM |
| 3 | Overlapping bookings | `MO-016` / `BK-DEMO-OVERLAP-A/B` / `DQ-DEMO-OVERLAP` | OM |
| 4 | Failed automation, retried | outbox event `...020` / `BK-H-0020` | OM |
| 5 | Grounded procedure question | (no fixed record; suggested questions) | Either |
`GET /api/v1/demo/manifest`'s `scenarios` array derives `ready`/`blocked_reason` from the
live underlying records, never hardcoded — confirmed via `backend/tests/
test_demo_manifest.py` (`test_demo_manifest_scenarios_ready_after_fresh_reset`) and
live-checked after every reset throughout this work.
## Seed strategy and date-anchoring
`seed/generate_seed.py --anchor 2026-08-01 --seed 20260801` produces deterministic CSVs
with absolute timestamps authored against a fixed anchor. `backend/app/seed_loader.py`
shifts every seeded datetime by `(real today authored anchor)` on every seed/reset, so
"today"/"near-future"/"currently overlapping" scenarios stay true to the actual reset
moment instead of decaying. This fixed a real, confirmed bug (`BK-DEMO-RETURN` was found
sitting 2 days in the past before this fix). Full detail: `docs/demo-release/demo-data.md`.
## Reset strategy
`POST /api/v1/demo/reset` (Operations Manager only, gated by `DEMO_ALLOW_RESET`) clears
MobilityOps's own tables, reseeds with a fresh date anchor, re-runs the data-quality scan,
and runs a server-side scenario-integrity check (`scenario_integrity_report()`) recorded
in both the response and the `demo_reset` audit event. Reachable from the sidebar, the
Demo Guide, and the About page. Never touches shared n8n/RAGcore/MCP data, other
containers, or volumes.
## Real vs. synthetic vs. not-connected
See `docs/demo-release/demo-concept.md` for the full breakdown. In short: auth/roles,
vehicle/booking management, return preview/commit, the 5 data-quality rules and their
resolutions, the audit trail, n8n orchestration, Docker deployment, and the automated
test suite are all really implemented. The organisation, all people, vehicles, bookings,
procedures, and the 5 named scenarios are synthetic. RAGcore and the ITWorx MCP Hub are
not live-connected (honestly labelled "Demomodus"/"Niet gekoppeld" everywhere, never a
fabricated success).
## Test results
### Backend (clean checkout, isolated stack)
- `pytest`: **127 passed**
- `ruff check .`: clean
- `mypy app`: clean (48 source files)
### Frontend (clean checkout, isolated stack)
- `npm ci`: clean (pre-existing esbuild-moderate/react-router-RSC-high advisories,
unchanged from before this work — not introduced by it)
- `tsc -b`: clean
- `npm run build`: clean
- Full Playwright suite: **56 passed** (against the isolated clean-checkout stack)
### Guided-demo test
`frontend/e2e/guided-demo-full.spec.ts` — one comprehensive test walking a fresh
Operations Manager session through all 8 Demo Guide steps performing the real action at
each step (processes the actual odometer-anomaly return, resolves the resulting
data-quality issue, merges the duplicate customer, asks a suggested knowledge question,
checks automation + audit, reviews the About page), then resets the demo data again to
restore the environment. **Passed**, confirmed stable across repeated runs both locally
and against the live Unraid deployment.
### Clean-checkout drill
Fresh `git clone` of this branch/commit into an isolated scratch directory, `.env` from
`.env.example`, isolated Compose project name (`mobilityops-cleandrill`) and remapped
host ports (`compose.override.yaml` with `!override` merge tags — no shared state with
any other stack), `docker compose up --build -d` from empty volumes → migrations ran
automatically (`e7b08389f47f (head)`) → seeded → full backend gate (127 passed, ruff/
mypy clean) → `npm ci`/`tsc -b`/`vite build` clean → full Playwright suite (56 passed)
→ reseeded and confirmed all 5 scenarios `ready: true` via the manifest → torn down
(`docker compose down -v` on the isolated project only; the working dev stack was never
touched).
### Server deployment
Deployed incrementally after every batch (10 deploy cycles across this work); final
state: both `api` and `web` rebuilt and healthy at the final commit, `db` untouched
across all of them (no destructive migrations on this branch). Migrations at
`e7b08389f47f (head)` throughout. `.deploy/source-revision` on the server matches the
final commit exactly.
### Container health
`docker compose ps` on the server: `api`, `db`, `web` all `healthy`, no restart loops.
### Browser console / network
No unexpected console errors on login, dashboard, scenarios, About, or with the Demo
Guide open (verified via `demo-accessibility.spec.ts`; the one benign 401 from the app's
own session-probe on first load is expected and explicitly accounted for, not silenced
blindly). No unresolved server errors in `docker logs` for `api`/`web` at the time of
this evidence capture.
### Responsive / accessibility
- Demo Guide renders as a correctly-anchored bottom sheet at 390px with no horizontal
overflow (`demo-accessibility.spec.ts`).
- **Real bug found and fixed**: the Demo Guide's fixed desktop side panel overlapped
main content with no reflow, making the return form's "Review return" button
unclickable while the guide was open at ordinary desktop widths — this surfaced while
writing the full guided-demo test. Fixed via a `guide-open` layout class that reserves
space for the panel; regression-tested.
- Demo badge and Demo Guide triggers are keyboard-focusable and operable (Enter to open,
explicit close controls).
- Existing responsive-overflow checks (390/768/1280/1440px) remain green throughout.
## Known limitations
- RAGcore and the ITWorx MCP Hub are not live-connected in this environment (by design
— see scope). The knowledge assistant uses a local, English-only demo knowledge base;
a Dutch question against it returns "insufficient evidence" (verified empirically), so
suggested questions and the Demo Guide's step 6 instructions deliberately stay in
English rather than silently breaking the demo's centerpiece grounded-answer feature.
- Existing operational screens (Dashboard, Vehicles, Bookings, Data Quality workbench,
Audit, Automation internals) remain in English; only new demo-productization surfaces
(login, Demo Guide, scenario overview, About page, demo badge, plain-language
integration labels) are in Dutch — a deliberate, documented scope decision, not an
oversight (`docs/demo-release/current-demo-gap-audit.md`, gap #11).
- Scenario S3 ("missing inspection before next booking", `MO-031`) is seeded and visible
in the attention queue but isn't one of the 5 scenarios surfaced on `/scenarios`,
matching the brief's request for exactly 5.
## 5-minute and 10-minute demo flows
See `docs/demo-release/demo-runbook.md` for the exact click-through scripts.
## Redeploy commands and rollback procedure
See `docs/demo-release/demo-runbook.md``git archive``scp` → extract → rebuild
`api`/`web` → confirm migrations → reseed. Rollback: extract an earlier
`.deploy/source-<short-sha>.tar.gz` and update `.deploy/source-revision` to match.
## Evidence screenshots
All captured live against `http://192.168.10.150:1236` (`artifacts/demo-release/screenshots/`):
1. `01-demo-entry-desktop.png` / `02-demo-entry-mobile.png` — demo entry, both sizes
2. `03-dashboard-with-scenarios.png` — dashboard with the scenario teaser panel
3. `04-demo-guide.png` — the Demo Guide panel open
4. `05-return-preview.png` / `06-return-result.png` — the return flow
5. `07-data-quality-resolution.png` — a data-quality issue with its plain-language explainer
6. `08-duplicate-customer-merge.png` — the duplicate-customer comparison/merge UI
7. `09-knowledge-assistant.png` — a grounded answer with cited sources
8. `10-integration-status.png` — plain-language integration status on Automation
9. `11-automation-retry-before.png` / `11-automation-retry-after.png` — a workflow retry
10. `12-audit-trail.png` / `13-audit-related-events.png` — audit trail + correlation drill-down
11. `14-about-demo.png` — the About page
12. `15-demo-badge-popover.png` — the permanent synthetic-demo badge popover
13. `16-reset-confirm.png` — the reset confirmation flow
No secrets appear in any screenshot or in this document.
-145
View File
@@ -1,145 +0,0 @@
# MobilityOps Unraid deployment evidence
## Outcome
- Deployment: **PASS**
- Gitea publication: **PASS**
- Gitea URL: `https://gitea.itworx.tech/Jens/MobilityOps`
- Visibility: private (verified in the Gitea web UI)
- Branch: `master`
- Verified baseline commit: `4bf9afbeff44088864e0844769d4dd0e4089d85b`
- Deployment implementation commit: `1e13943cffb2da8a328b5b1ea5e9b1fe73fdd774`
- Server: `192.168.10.150`
- Server directory: `/mnt/user/appdata/mobilityops`
- Compose project: `mobilityops`
- Application URL: `http://192.168.10.150:1236`
- Port mapping: LAN `0.0.0.0:1236` / `[::]:1236` to `web:80`
## Services and health
| Service | Runtime state | Health | Host exposure |
|---|---|---|---|
| `db` | running, 0 restarts | healthy | none (`5432/tcp` internal) |
| `api` | running, 0 restarts | healthy | none (`8000/tcp` internal) |
| `web` | running, 0 restarts | healthy | `1236:80` on LAN |
| shared host `n8n` | running | healthy | `5678:5678` on LAN; outside MobilityOps Compose |
The final review topology reuses the n8n container that was already running on the host.
Its empty public-host/editor URL settings were corrected in the persistent Unraid template
so workflow execution URLs are valid. The temporary Compose-owned n8n container was
removed without deleting its retained volume. Port `1236` was confirmed unused before the
original deployment; the application directory was created specifically for MobilityOps.
## Deployment commands
The existing SSH aliases resolve to the requested hosts and keys (`gitea.itworx.tech`
for Gitea SSH and `unraid` for root access). No key was created, copied, or replaced.
The committed source was transferred from the workstation; Unraid has no Gitea key.
Repository publication used the SSH clone URL supplied by Gitea:
```bash
git remote add origin ssh://git@192.168.10.150:222/Jens/MobilityOps.git
git push -u origin master
git push origin --tags
```
Git and the Gitea web UI both verified `master` as the default branch, the full commit
history, baseline commit `4bf9afbeff44088864e0844769d4dd0e4089d85b`, and zero tags.
The remote tree contains no `.env`, local database, `node_modules`, virtual environment,
test cache, build cache, Playwright output, or browser binaries.
```bash
git archive --format=tar.gz --output=<temporary-archive> <commit>
scp <temporary-archive> unraid:/mnt/user/appdata/mobilityops/.deploy/source.tar.gz
ssh unraid
cd /mnt/user/appdata/mobilityops
tar -xzf .deploy/source.tar.gz
./deploy/unraid/configure-env.sh http://192.168.10.150:1236
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up --build -d db api web
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml exec -T api \
python -m app.cli seed --reset
./deploy/unraid/setup-existing-n8n.sh \
n8n \
http://192.168.10.150:1236/api/v1/integrations/n8n/return-callback
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up -d db api web
```
The server `.env` was created from `.env.example`, is mode `0600`, and contains generated
runtime secrets. Secret values and n8n owner credentials remain server-only and are not
included here or in Git.
## Validation evidence
- Migration: `e7b08389f47f (head)`.
- Deterministic seed: users 2, customers 180, vehicles 50, bookings 246, inspections 75,
maintenance 40, data-quality issues 26, workflow runs 20.
- HTTP: `GET /` returned 200; `GET /health` returned
`{"status":"ok","service":"mobilityops-api"}` through the web proxy.
- Backend gates in an isolated local Compose project: 66 tests passed, Ruff clean, mypy
clean across 44 files.
- Frontend: `npm ci && npm run build` completed (`tsc -b && vite build`).
- Logs: no traceback, fatal, uncaught, or unresolved startup error in the deployment log
scan. Browser console had no warnings or errors during the smoke test.
- Browser smoke test in Chrome: Operations Manager demo login, Dashboard, Vehicles,
Bookings, Data Quality, Knowledge, Automation, and Audit all loaded from the LAN URL.
- Dashboard showed persisted seed metrics (21 available, 11 rented, 6 cleaning,
5 maintenance, 7 blocked, 22 open issues, 1 pending/failed workflow).
- Return workflow: `BK-DEMO-RETURN` accepted 54,700 km, created `INSP-0076` and
`DQ-RET-0076`, preserved the 54,820 km canonical odometer, and changed the booking to
returned.
- Shared-n8n round trip: final post-deploy event `98eb06dc-0bcc-4e3d-96ec-c23b2d266293`
reached `succeeded` on attempt 1 with no last error; the deterministic reset afterwards
restored `BK-DEMO-RETURN` to `active`.
- Data quality: `DQ-RET-0076` displayed the persisted regression evidence and related
booking/inspection references.
- Knowledge: UI truthfully showed `Provider: demo · available · 10 procedures indexed`;
the damage question returned grounded excerpts and citations from the local procedures.
## Integration status
- RAGcore: disabled for this deployment; `KNOWLEDGE_PROVIDER=demo`. No claim of a live
RAGcore connection is shown. Operational functionality is unaffected.
- ITWorx MCP Hub: registration disabled with `MCP_HUB_REGISTRATION_ENABLED=false`; the
independently authenticated provider endpoints remain available internally to the web
proxy/API boundary, but no live Hub connection is claimed.
- n8n: the existing server instance at `http://192.168.10.150:5678` is healthy; the
MobilityOps workflow is imported/published there and a real return delivery succeeded.
The bundled MobilityOps service is disabled by default in the Unraid overlay.
## Known limitations
- RAGcore and ITWorx MCP Hub are intentionally not connected yet.
- Demo authentication remains the accepted HMAC-cookie PoC mechanism.
- The dependency advisories already documented in final acceptance remain unchanged.
## Redeploy
From the workstation, create an archive of the desired committed revision and transfer it
to `.deploy/source.tar.gz`. On Unraid, preserve `.env` and the named volumes, then run:
```bash
cd /mnt/user/appdata/mobilityops
tar -xzf .deploy/source.tar.gz
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up --build -d db api web
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml exec -T api alembic current
curl -fsS http://127.0.0.1:1236/health
```
## Logs
```bash
cd /mnt/user/appdata/mobilityops
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml ps
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml logs --tail=200
docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml logs -f api web
docker logs -f n8n
```
## Safe rollback
Choose a known-good commit on the workstation, archive and transfer it as above, then on
Unraid extract it over the identifiable MobilityOps source directory and run the same
`up --build -d` command. Preserve `.env` and both named volumes; do not use `down -v`,
remove volumes, prune Docker, or modify unrelated containers. Check the target commit's
Alembic compatibility before rolling application code behind the current database schema.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 114 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 63 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 106 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 110 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 99 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 71 KiB

@@ -1,89 +0,0 @@
# MobilityOps premium UI evidence summary
Date: 2026-08-02
Branch: `design/mobilityops-premium-ui`
Baseline revision: `dfabb41582e302f45a3de826f85f531bf23dfc8b`
Final design implementation commit: `1f292e14bb6a2e8ded5dc675b1b3360307d8a9ae`
Review URL: `http://192.168.10.150:1236`
## Outcome
The working PoC was transformed into the Control Rail operational interface without
changing backend contracts or adding scope. All existing journeys remain functional;
return registration gained an evidence-based review boundary before commit.
## Evidence index
- Baseline audit: `docs/design/current-ux-audit.md`
- Three directions and decision: `docs/design/design-directions.md`
- Design tokens and component rules: `docs/design/design-system.md`
- Stitch resource IDs: `docs/design/stitch-manifest.md`
- Implemented visual validation: `docs/design/implementation-validation.md`
- Baseline captures: `artifacts/design-validation/current/`
- Stitch captures: `artifacts/design-validation/stitch/`
- Final responsive captures: `artifacts/design-validation/implementation/`
## Major implementation changes
- Responsive Control Rail shell with compact top bar, desktop rail, off-canvas menu and
labelled mobile bottom navigation.
- Live readiness band, filterable Attention queue, movement timeline, honest integration
pulse and persisted activity on the operations dashboard.
- Searchable fleet and booking registries; booking client pagination limits the DOM to 25
operational rows; responsive tables retain field labels.
- Capture → review → result return workflow with calculated consequence preview and no
write request before confirmation.
- Match/conflict duplicate comparison, evidence-first knowledge, system-health cards and
expandable audit metadata.
- Inline SVG product mark, Feather-like line icon set, CSS control-centre illustration,
timeline/status motion and reduced-motion fallback; no image or motion dependency.
## Validation
| Gate | Result |
|---|---|
| Backend tests | 66 passed |
| Backend lint | ruff passed |
| Backend types | mypy: 0 issues in 44 files |
| Frontend types/build | passed; 59 modules; 240.24 kB JS and 36.63 kB CSS before gzip |
| Browser journeys | 19 passed locally |
| Horizontal overflow | none at 390/768/1280/1440 px |
| Accessibility | named landmarks, skip link, visible focus, text-plus-shape status, labelled mobile rows, reduced-motion support |
| Deployed browser smoke | passed on all 10 authenticated routes plus login at desktop and mobile sizes |
| Console/network | 0 browser warnings/errors; 7 authenticated API paths returned HTTP 200 |
| Deployed global search | Ctrl+K plus `MO-024` navigation passed against the review URL |
All displayed operational counts remain derived from the existing persisted API data.
Synthetic-data labelling is persistent on login and authenticated surfaces.
Deployed evidence is stored in `artifacts/design-validation/implementation/deployed/`.
The review stack reports healthy PostgreSQL/API state, HTTP 200 from the web application,
and healthy state from the server's existing n8n at port 5678. A synthetic return reached
`succeeded` on attempt 1 through that shared n8n and its MobilityOps callback; the demo was
then reset to its deterministic state.
## Performance observations
No runtime font, image or animation dependency was added. The application uses inline SVG
and CSS visuals, and the production bundle remains appropriate for this internal PoC.
## Known limitations
- Global search resolves Control Rail sections and `MO-*`, `BK-*`, `DQ-*` public
references. It intentionally does not offer customer lookup because the locked PoC has
no customer detail route or cross-entity search API.
- The repository retains a bundled n8n service for standalone local clean-checkout demos.
The Unraid overlay keeps it behind the opt-in `bundled-n8n` profile; the live review
deployment uses the server's existing shared n8n instead.
- The MCP Hub state is correctly shown as not configured in the current PoC rather than
simulated as healthy.
- Live RAGcore and MCP Hub round trips remain subject to the existing environment limits
documented in `PROJECT_STATE.md`; their degradation behavior is unchanged.
## Rollback
The accepted baseline remains reachable at commit
`dfabb41582e302f45a3de826f85f531bf23dfc8b`. To roll back the review deployment without
rewriting git history, archive that revision, extract it over the application source on
Unraid while preserving `.env` and Docker volumes, and run
`docker compose -p mobilityops up -d --build`. Verify `/health` and port 1236 afterwards.
-74
View File
@@ -1,74 +0,0 @@
# Fleet Ops — final acceptance evidence
## Accepted implementation
- Deployment source marker: current `master`; the application image was built from runtime commit `cb7edb0` and the following commits contain acceptance/evidence only.
- Live demo: `http://192.168.10.150:1236` — public passwordless synthetic demo.
- Deployment: Unraid `/mnt/user/appdata/mobilityops`, Compose project `mobilityops`.
- Database migration: `b913a72e8c14 (head)`.
- Product name: Fleet Ops; MobilityOps remains the technical repository/deployment ID.
## Clean acceptance — 2026-08-10
```text
Backend, isolated PostgreSQL: 241 passed, zero warnings
Playwright, fresh local build: 152 passed (5.4 min)
Playwright, live production: 152 passed (4.7 min)
Ruff: clean
Mypy: clean across 58 source files
Frontend lint/build: passed
npm audit, full and prod: 0 vulnerabilities
Alembic: b913a72e8c14 (head)
```
The browser suite covers both roles, protected routes, booking/return/maintenance/user/privacy workflows, five data-quality resolution types, correlated audit, retryable orchestration, grounded knowledge in three languages, the complete guided demo, the recruiter highlights, keyboard behaviour and responsive layouts from 360 to 1440 px.
The provider matrix explicitly covers both valid title contracts: the deterministic provider uses Markdown frontmatter titles, while production RAGcore returns curated localized presentation titles. Both must include the language-specific source fragment. The focused live matrix passed 3/3, followed by the complete 152/152 green production run.
## Production hand-off state
The final reset completed immediately before the verified Hub call at `2026-08-10T19:16Z` and restored:
```text
users 2 · customers 180 · vehicles 50 · bookings 254 · inspections 75
maintenance 40 · data-quality issues 33 · workflow runs 20
scenario_integrity.all_ready = true (5/5 scenarios)
```
API readiness is `ready`, PostgreSQL is `up`, and API, web, database, scheduled backup, Prometheus and Grafana are running healthy/current. No traceback or critical error appears in post-deployment API/backup logs.
A pre-deployment custom-format backup was created and independently verified by SHA-256 plus `pg_restore --list`:
`/mnt/user/appdata/mobilityops/backups/postgres/mobilityops-20260810T185833Z.dump`
## External integration evidence
- **Existing central n8n:** configured, dispatch enabled and operational. All four expected workflows have recent healthy heartbeats. The reset state contains 19 successful runs, zero pending, zero unexpected failures and one explicitly synthetic failed-retry scenario.
- **RAGcore:** reachable and ready for tenant `northstar-mobility-demo`, workspace `mobilityops`, collection `internal-procedures`. Exact identity lookups independently confirm 11/11 active published documents in each of NL/EN/FR; the latest n8n report separately records 33 synchronized documents and zero failures. The canonical parsed-artifact hash is not misrepresented as a raw-source hash.
- **ITWorx MCP Hub:** registration is enabled, the Hub is reachable and Fleet Ops is **operational**. Three real `mobilityops.operations.summary` verification calls are audited under the Hub's tenant-bound client identity; the final short-lived bearer produced HTTP 200/JSON and HTTP 401 after immediate revocation. All exposed Fleet Ops tools remain read-only.
## Recruiter and visual acceptance
The public entry now offers a direct **90-second Highlights** route. It links three product actions to their engineering proof, while the **Engineering** workspace explains domain ownership, commit-before-orchestration, citation-bound AI, security and explicit scope.
Interactive live inspection confirmed:
- no horizontal overflow on Highlights or Engineering at desktop; automated coverage confirms 360/390 px mobile layouts;
- clear hierarchy, consistent primary actions and readable proof cards;
- compact remaining-attention action (count badge, two-line label, small directional icon) without the former stretched empty panel;
- truthful live n8n/RAGcore/MCP labels after reset, including 11 verified RAG documents and the real Hub client/tool evidence;
- no dead links, placeholder numbers or unexplained raw data-quality references;
- complete synthetic-data disclosure on login and in the persistent shell.
Current evidence images are in `artifacts/evidence/screenshots/`:
- `1-login.png`
- `2-highlights.png`
- `3-engineering-story.png`
- `4-dashboard.png`
- `5-knowledge-evidence.png`
- `6-highlights-mobile.png`
## Deliberate boundary
This is a completed, production-shaped public demo—not claimed customer adoption and not a general ERP. Accounting, payments, public reservations, CRM, inventory, HR, a second RAG stack, a separate MCP server and autonomous write agents remain intentionally excluded. A real personal-data rollout would additionally require the adopting organisation's identity provider, retention approvals, secrets lifecycle, alert ownership and disaster-recovery governance.
+27 -1
View File
@@ -7,7 +7,7 @@ in `PROJECT_STATE.md`.
## 2026-08-21 release candidate ## 2026-08-21 release candidate
- Backend: **271/271** tests passed against an isolated clean PostgreSQL database. - Backend: **271/271** tests passed against an isolated clean PostgreSQL database.
- Browser acceptance: **155/155** Chromium tests passed in 4.6 minutes. - Browser acceptance: **155/155** Chromium tests passed in 6.0 minutes.
- Live-safe browser canary: **4/4** passed across Chromium and Firefox against the local - Live-safe browser canary: **4/4** passed across Chromium and Firefox against the local
deployed stack; unlike the acceptance suite, it never resets or mutates demo records. deployed stack; unlike the acceptance suite, it never resets or mutates demo records.
- Accessibility: the principal login, dashboard, data-quality, knowledge, automation and - Accessibility: the principal login, dashboard, data-quality, knowledge, automation and
@@ -34,6 +34,32 @@ in `PROJECT_STATE.md`.
- The final non-destructive HTTPS canary passed **4/4** across Chromium and Firefox, - The final non-destructive HTTPS canary passed **4/4** across Chromium and Firefox,
including the core operator routes and a grounded answer from the real knowledge stack. including the core operator routes and a grounded answer from the real knowledge stack.
## 2026-08-21 resilience upgrade verification
- Immutable revision `00191e9b54ee6b961648a6e02abbb3a57957dba0` was promoted from a
checksum-verified archive after a verified production dump. A stable gateway now routes
to two revision-specific API and two web replicas; stateful services are no longer
restarted by routine application releases.
- Moving host port 1236 from the legacy web container to the gateway was a one-time
migration hand-off and produced 14 failures across 1,200 rapid probes. Future releases
do not move that port; their acceptance gate is the zero-error versioned gateway reload.
- The subsequent M50 release exercised that steady-state path: the gateway remained online,
atomically switched revision-specific API/web aliases and sustained **700/700** external
readiness probes without interruption. Post-promotion Chromium/Firefox acceptance passed
**4/4** and 360 concurrent authenticated reads had zero errors at p95 **116.4 ms**.
- The versioned gateway switch sustained **300/300** local rollout probes without an error.
Production's non-destructive Chromium/Firefox canary passed **4/4**, and 360 authenticated
concurrent reads returned zero errors at p95 **137.2 ms**.
- PostgreSQL, backup, Prometheus, Alertmanager, Grafana and the gateway all reported healthy;
Alembic was at head, the protected Prometheus target was present, and backup plus real
restore-drill evidence remained current.
- Trivy 0.74 found zero fixed HIGH/CRITICAL vulnerabilities in the exact production API,
web and gateway images. The separately built rclone/PostgreSQL backup-tools image is also
clean after rebuilding rclone 1.75.0 with Go 1.26.6.
- The OneDrive worker is deployed as an opt-in profile but is not represented as active:
it requires the owner's one-time interactive Microsoft OAuth authorization. Until that
happens, verified local backups remain the active recovery source.
## Evidence boundary ## Evidence boundary
The complete local suite uses the deterministic provider and an isolated database so it is The complete local suite uses the deterministic provider and an isolated database so it is
@@ -1,161 +0,0 @@
# Fleet Ops final integrations — evidence summary
Session date: 2026-08-05. Branch `feat/fleet-ops-final-integrations`.
## Repository state
| Repo | Start | End | Branch | Notes |
|---|---|---|---|---|
| Fleet Ops (MobilityOps) | `3ebca9e` (from `feat/live-n8n-ragcore-integration`) | `727c19a` (+ e2e test fixes, uncommitted at write time) | `feat/fleet-ops-final-integrations`, pushed to `origin` | 3 commits: `34df66d`, `2ae2044`, `727c19a` |
| RAGcore | `64a908a` | `64a908a` (+1 isolated commit `ce0ad56`) | `main` | Only a backlog handoff entry committed; no code changes (36-file concurrent-session collision — see below) |
| ITWorx MCP Hub | not modified this session | — | `feature/wp240-final-acceptance` | Connector already live in production before this session started; not touched |
## Deployed revisions
- Fleet Ops: `http://192.168.10.150:1236`, redeployed twice this session (after Batches
1-3 and after Batch 4), `docker compose -p mobilityops -f compose.yaml -f
compose.unraid.yaml up --build -d db api web`, `.deploy/source-revision` = `727c19a...`.
- RAGcore: `http://192.168.10.150:1237`, `ragcore-app-1`. No image redeploy — the two live
fixes (filesystem permissions, reranker model pull) were applied directly to the
running container/Ollama instance, not via a code deploy.
- ITWorx MCP Hub: `http://192.168.10.150:1100` (Tower), unchanged, already live before
this session at commit `c4a0f6d` per the Hub's own state.
## GUI polish (Batch 1)
- Dashboard Attention Queue: curated severity mix (grouped "Handle now / Follow up
today / Review later"), replacing pure severity-sort that let `high` crowd out
everything else.
- Today's Movements: seed data curated (`seed/bookings.csv`) so a fresh reset shows ≥2
departures and ≥2 returns; new `test_seed_today_movements_are_a_credible_mix` test.
Live-verified after a real demo reset: 2 returns + 2 departures shown.
- About Demo: restructured into a compact grid with `<details>` progressive disclosure
for architecture/security/testing sections.
- Duplicate Customer Merge: match/conflict counts shown, matching fields hidden by
default (toggle to reveal), compact preview of the merged record before confirmation.
- Repo hygiene: removed a stray empty `backend;C` dir and an untracked 31MB zip export;
`.gitignore` now excludes future archive exports.
- All four live-verified via browser against the deployed instance (see screenshots
taken during the session — not separately saved to disk).
## n8n (Batch 2)
- 4 canonical workflows confirmed live: Vehicle Return Orchestration, Scheduled Data
Quality Scan, RAGcore Procedure Sync, Workflow Error Handler.
- Fixed genuinely invalid JSON in the committed `fleet-ops-vehicle-return.json` (a
missing `},` between two node objects — the file could not be parsed).
- Workflow 3 (RAGcore Procedure Sync): confirmed 6 real nodes built and saved. Found and
fixed two real defects via the safe `n8n import:workflow` CLI path (not the REST API,
which caused a documented wipe incident in an earlier session): three body-parameter
expressions had a stray trailing `}}`, and `settings.errorWorkflow` was unset. Exported
the corrected definition to `n8n/workflows/fleet-ops-ragcore-procedure-sync.json`,
added to `MANIFEST.md` and `check_drift.py`.
- **Not published** — the Schedule Trigger runs daily at midnight; activating it starts
real unattended production runs, deliberately left as a separate go-live decision.
- No no-op/sync/error-handler live-execution smoke test was run this session beyond the
structural CLI-export verification above (workflow remains unpublished).
## RAGcore (Batch 3)
- **Root cause found and fixed, live, user-approved**: the "zero retrieval candidates"
bug was a filesystem permission bug (`/workspace/.state/models/embedding_profiles.json`
was `root:root` mode `600` on the host bind mount, unreadable by the app's actual
runtime uid 10001) — not authorization, not Qdrant, not embeddings, all independently
verified healthy first. Fixed via `chown`/`chmod`; re-verified in-process (5 real hits,
up from 0).
- **Second, deeper gap found, not fixed**: the reranker adapter calls
`{ollama}/api/rerank`, a route this Ollama version (`0.32.5`) does not serve (404).
Pulled a working model (`xitao/bge-reranker-v2-m3:latest`, 1.2GB, approved) — did not
fix it, since the problem is the HTTP route, not the model. `/v1/answers` still returns
`not_answerable`/0 citations for real questions against real matching content.
- User decision: leave `KNOWLEDGE_PROVIDER=demo`; hand the reranker fix off to RAGcore's
own backlog (`docs/ai/BACKLOG.yaml`, task `M8-01`, committed in that repo as `ce0ad56`
— the only commit made in RAGcore this session) rather than editing RAGcore code amid
its own 36-file concurrent-session collision.
- Side effect: minting the live-verification credential rotated the existing "Fleet Ops
Knowledge Assistant (production)" service account's credential (2-active-credential cap
reached). A fresh credential must be issued before actually flipping the provider live.
## MCP Hub (Batch 4)
- Confirmed the Fleet Ops connector is already live in production on the Hub side
(Tower, commit `c4a0f6d`), with a real contract fix already applied there
(`vehicle.get`'s wire parameter normalized to camelCase `vehicleRef`).
- Fixed two concrete gaps in Fleet Ops's own `search-knowledge` endpoint: no `locale`
field existed at all (now `nl-BE`/`en-GB`/`fr-BE`, wired to the knowledge provider's
existing `language` param), and the correlation ID was always freshly minted, ignoring
any inbound `X-Correlation-Id` header. Added `get_correlation_id`, applied to all four
MCP endpoints.
- `MCP_HUB_BASE_URL` was dead config (declared, never read); wired it for a real,
bounded Hub-reachability health check instead of an unneeded self-registration push
(the Hub's own registration is catalog-driven).
- Renamed Fleet Ops's own internal audit tool labels `mobilityops_*``fleet_ops_*`
(mirrored in `contracts/mcp-tools.json`, `mobilityops_*` kept as deprecated aliases).
The live Hub connector's own dotted tool namespace (`mobilityops.operations.summary`
etc.) is a separate, Hub-owned naming layer, deliberately not touched.
- Automation page's MCP card now shows real evidence (last tool/client/count/timestamp)
instead of only the registration-enabled boolean.
## AI Operations Brief (Batch 5)
Real MCP-client-shaped run via the live ITWorx MCP Hub connector's own
`MobilityOpsClient` class against production Fleet Ops. Full runbook and live output in
`docs/final-integrations/ai-operations-brief-runbook.md`. Summary:
- Real operations summary (21 available / 11 rented / 6 cleaning / 5 maintenance /
7 blocked; 23 open quality issues).
- Real most-pressing vehicle identified (`MO-031`, missing operational inspection).
- Real vehicle detail lookup.
- Real grounded knowledge answer (English damage-handling question): 2 real citations,
`evidence_state: grounded`.
- Dutch/French variants of the same question honestly returned `insufficient` (no
fabrication) — root cause: the live Hub connector doesn't yet send the new `locale`
field, a Hub-side follow-up, not silently worked around.
- Correlation IDs verified end-to-end in Fleet Ops's own audit log
(`GET /api/v1/audit?action=mcp_tool_request`), matching the response payloads exactly.
- No write actions performed at any point.
## Testing per batch
- Backend: **176 passed**, `ruff check .` clean, `mypy app` clean (50 source files) —
verified against a freshly rebuilt image after discovering mid-session that
`docker compose run --rm api` (no bind mount on the `api` service) silently tests a
stale image otherwise. One genuinely stale test assertion found and fixed as a result.
- Frontend: `tsc -b && vite build` clean.
- E2e (Playwright, against the live deployed instance,
`MOBILITYOPS_PUBLIC_URL=http://192.168.10.150:1236`): every spec file run this
session passed — `demo.spec.ts`, `interactive-elements.spec.ts` (26),
`responsive-i18n.spec.ts` + `demo-accessibility.spec.ts` + `guided-demo-full.spec.ts`
(28), `i18n-coverage.spec.ts` + `error-messages.spec.ts` + `clickable-rows.spec.ts` +
`demo-guide.spec.ts` + `demo-entry.spec.ts` + `demo-legibility.spec.ts` +
`fleet-ops-correction.spec.ts` + `ui-redesign.spec.ts` + `greeting.spec.ts` +
`greeting-live.spec.ts` (28, after fixing 2 pre-existing fragile locators unrelated to
this session's feature work — a `.data-table` ambiguity now that Automation has two
tables, and a `Technische details` toggle ambiguity for the same reason; plus one
pre-existing untranslated-loanword false positive in `i18n-coverage.spec.ts`).
## Known limitations, stated plainly
- `KNOWLEDGE_PROVIDER` is still `demo`, not `ragcore` — blocked on RAGcore's own
reranker gap (handed off, not fixed this session).
- n8n workflow 3 is built and correct but not published (deliberate, separate decision).
- The live MCP Hub connector doesn't yet send the new `locale` field, so
locale-aware knowledge search only works when called directly against Fleet Ops (as
proven by the backend tests), not yet through the live Hub connector as deployed.
- No public-demo-readiness checklist, About Demo Guide "completed" end-state polish
(section 4E), or dashboard MCP "activity showcase after Demo Complete" gating were
built this session — the MCP evidence display exists on the Automation page
unconditionally rather than gated behind guided-demo completion.
- No security-review pass was run separately this session (existing gates: ruff, mypy,
the repo's own auth/audit test coverage).
## Rollback
- Fleet Ops: prior working revision `0571a40` remains in `.deploy/` as
`source-0571a40.tar.gz` on the Unraid host; redeploy by re-extracting and re-running
the same `docker compose up --build -d` sequence with that archive.
- RAGcore: `chown`/`chmod` change is trivially reversible (`chown 0:0` +
`chmod 600` on the same path) if needed, though there is no reason to revert a
permission fix. Ollama model pull (`xitao/bge-reranker-v2-m3:latest`) can be removed
with `ollama rm` if unwanted; it is inert until RAGcore's own code is changed to use it.
- MCP Hub: not modified this session.
@@ -1,284 +0,0 @@
# Fleet Ops correction and release — final evidence
**Result: PASS**
## Commits
- Source branch / commit (verified pre-correction baseline): `master` @ `18344bc8b7a75a2f868bf15bf498fc030ac6c34c`
- Fix branch: `fix/fleet-ops-i18n-status-flow`
- Final fix-branch commit: `284b3c7` (merged content identical to `2e4fb43`, which carries the evidence-summary localization fix)
- Main-before-merge: `18344bc8b7a75a2f868bf15bf498fc030ac6c34c` (confirmed unchanged via `git fetch` + `git rev-parse origin/master` immediately before merging — no unexpected commits landed on master while this branch was in progress)
- Merge commit: `de0bdea84fea01b4501deb7099107bc753c2e6d7` (`git merge --no-ff fix/fleet-ops-i18n-status-flow -m "merge: complete Fleet Ops localization and status resolution"`, zero conflicts)
- Final main commit: `de0bdea84fea01b4501deb7099107bc753c2e6d7`
- Deployed commit: `de0bdea84fea01b4501deb7099107bc753c2e6d7` (`.deploy/source-revision` on Unraid)
- Gitea main branch: `master` (confirmed via `git fetch origin && git rev-parse origin/master` matching local `master` after push)
- Live URL: `http://192.168.10.150:1236`
Fix-branch commit history: `6deb955`, `e6539d1`, `ac4b163`, `1fdd2b3`, `1e40775`, `a7ac5ed`, `7851e80`, `cda2c32`, `2e4fb43`, `284b3c7`.
## What this correction fixed
1. **Status-recommendation flow redesigned** (sections 8A8F). The old single opaque
"calculate and apply recommended status" action is replaced by a single shared, pure
evaluator (`backend/app/services/vehicle_status.py::evaluate_vehicle_status`,
documented in `docs/fleet-ops-correction/vehicle-status-decision-table.md`) used
identically by the scanner, a non-mutating preview endpoint
(`POST /api/v1/data-quality/issues/{ref}/status-recommendation`), and a
transactional apply endpoint (`POST .../apply-recommended-status`) that locks the
row, recomputes facts, rejects a stale `recommendation_token`, refuses unsafe/manual-
review recommendations, and re-validates post-write before resolving the issue.
- Forbidden shortcuts eliminated: "maintenance + active booking" no longer
auto-recommends "rented" (being in maintenance is itself now a blocking fact);
"maintenance with nothing else wrong" no longer auto-clears to "available" (no
fact proves maintenance is actually finished — release stays a manual decision).
- Frontend: "Review recommendation" → a localized decision panel (current/
recommended status, why, evidence, consequences) → an exact "Change status to
&lt;status&gt;" confirm action → result, or a distinct "Manual review required"
state offering no generic apply button.
2. **MO-016 order independence** (section 9). Order independence does not mean "same
final status regardless of order" — resolving the booking overlap first genuinely
removes the conflict, correctly leaving nothing to apply. What holds either way: the
recommendation always reflects real current facts (never a stale proxy), and nothing
unsafe is ever applied (never "rented"). Proven by a backend test explicitly scoped
to MO-016/DQ-DEMO-STATUS (the original version wasn't — `_first_open()` returned
whichever of ~14 open `vehicle_status_conflict` issues was most recent, not
necessarily MO-016's) and a browser-level Playwright test covering both orders.
3. **"Fleet Ops" is a non-localizable brand constant** (`frontend/src/product.ts`,
backend `PRODUCT_NAME`), wired via `{{productName}}` interpolation everywhere the
brand appeared in locale prose. A permanent test fails the build if any locale file
ever defines the brand name or an `appName` key again.
4. **Dynamic backend prose converted to message codes + params** (sections 5/6/10):
return status reasons, audit field/actor-type labels, automation `last_error` (new
`last_error_code` column, migration `799d8800e241`), search results (sections/
vehicles/bookings/issues), and — found live on Unraid — the data-quality evidence
summary. Raw technical text is demoted to a "Technical details" disclosure
everywhere.
5. **Knowledge-base fixes**: the demo provider's tokenizer silently dropped accented
characters (`[a-z0-9]+` split "véhicule" into "v"+"hicule"), breaking French
retrieval broadly — fixed to include the Latin-1 accented range. Reweighted section
scoring so a body match (real substance) outranks a heading/title match (a shallow
structural hint) — the old weighting misranked the damage procedure behind an
unrelated document for the brief's exact validation question in all 3 languages.
Removed leftover "MobilityOps"/"PoC" mentions from 9 procedure documents.
6. **Search, audit, automation, maintenance/inspections localized** (section 10):
backend returns stable codes + params only; the frontend localizes section labels,
vehicle summaries, booking/issue statuses, audit action/field/actor labels,
automation error explanations, and maintenance/inspection type labels.
7. **i18n test suite strengthened** (section 11): key parity, brand invariant,
translation-quality (cross-locale identical-value detection), a hardcoded-JSX-text
static scan (had to anchor on backreferenced closing-tag names — a naive `>text<`
regex misread TypeScript generics as JSX), and a 3-language route matrix (every main
route, no console errors, correct `html[lang]`, real page headings).
## Live-caught bug (the deployment validation earning its keep)
Live validation on the freshly-deployed fix branch directly caught a real defect: every
data-quality issue's top-of-page evidence summary was unconditionally showing raw,
always-English text (e.g. *"vehicle marked available while reserved bookings
conflict"*) in **all three languages**, because the frontend never finished the
`evidence.signals` localization the backend had already been emitting (the backend code
even had a comment describing the intended design that the frontend didn't implement).
Fixed in commit `2e4fb43`:
- `DataQualityIssueDetail.tsx` now renders `evidence.signals` through the operator's
locale as the primary evidence text.
- The four `DQ-DEMO-*` seed rows that anchor the guided demo's scripted scenarios now
carry real, accurate signals computed at seed time (the duplicate-customer similarity
score is the actual `SequenceMatcher` ratio on the seeded names, not invented).
- Rows with no structured signals fall back to raw text rather than showing a blank
summary; the one known filler placeholder gets its own localized rendering.
- A regression test locks this in: the vehicle-status-conflict evidence summary must
show localized text and must never contain the specific raw English sentence that was
live-visible before the fix, in all 3 languages.
Also found and fixed along the way: a frontend logic bug conflating "no conflict" with
"manual review required" (both carry `safe_to_apply: false`), which showed a false
"manual review required" panel for MO-016 after its booking overlap was resolved
instead of the correct "no change needed" state (fixed in `1fdd2b3`).
## Translation coverage
- All three locale files (`nl-BE`, `en-GB`, `fr-BE`) define exactly the same key set
for every namespace (`i18n-coverage.spec.ts`, structural guarantee).
- No locale file contains an empty string value.
- No locale file defines the brand name or an `appName` key (brand-invariant test).
- Cross-locale translation-quality check: for every string ≥8 characters of real prose,
nl-BE ≠ en-GB, fr-BE ≠ en-GB, fr-BE ≠ nl-BE, with a precise, audited allowlist for
genuine proper nouns/cognates (23 entries, each with a documented reason).
- Hardcoded-JSX-text static scan: zero findings against the current codebase (verified
against both false positives — TypeScript generics — and a deliberately-injected-
then-reverted false negative).
- 3-language route matrix: every main route (dashboard, vehicles, vehicle detail,
bookings, booking detail, data quality, issue detail, automation, knowledge, audit,
scenarios, about) opens cleanly in all 3 languages with no console errors, correct
`html[lang]`, and a real page heading.
- **Remaining visible wrong-language text**: none found. The one gap that existed (the
data-quality evidence summary) was found live and fixed before merge.
## Branding
- Visible product name: **Fleet Ops**, exactly, in all 3 languages, everywhere (login,
topbar, footer "Fleet Ops Demo", document title, About page, Demo Guide, knowledge
base). Verified structurally (brand-invariant test) and live (branding test across
dashboard/vehicles/data-quality/audit/automation/knowledge pages in all 3 languages;
visual screenshots of the login screen in nl-BE and fr-BE).
- Technical identifier retained (by design, per the brief): repository name, local
directory, package/module names, Compose project, deployment directory, database
name, and the `/health` endpoint's `service: "mobilityops-api"` field remain
"mobilityops" — none of these are visible UI text.
- No visible "MobilityOps" or "PoC" anywhere in the UI or the demo knowledge base
(9 procedure documents cleaned up; regression test in `test_knowledge.py` scans every
procedure file for both strings).
## Status-preview / apply / manual-review / MO-016 ordering
- **Preview**: verified non-mutating — the issue's `status` stays `"open"` after
calling the preview endpoint and re-fetching it via a fresh request.
- **Apply**: the confirm button names the exact target status ("Change status to
Blocked" / "Status wijzigen naar Geblokkeerd" / "Changer le statut vers Bloqué");
applying resolves the issue and updates the vehicle atomically.
- **Manual review**: MO-024 (active rental + service-threshold reached, a genuine fact
contradiction) shows "Manual review required" with no generic apply button rendered
at all.
- **Stale token**: simulated by resolving the underlying booking overlap after the
preview was fetched but before applying — the apply call is correctly rejected
(`RECOMMENDATION_STALE`), the UI shows the "situation has changed" message, and the
user must review again before a new apply is possible.
- **MO-016 ordering**: both orders tested. Resolving the overlap first correctly leaves
nothing to apply (vehicle stays "available", genuinely correct). Resolving the status
conflict first safely blocks the vehicle; resolving the now-redundant overlap
afterwards does not disturb it. Neither order ever produces "rented".
## Knowledge (per language)
The brief's exact validation question, in each language, grounds on the damage
procedure as the **primary** (not just top-3) source:
- nl-BE: *"Wat moet ik doen wanneer een voertuig beschadigd terugkomt?"* → damage
procedure, Dutch source, Dutch excerpt.
- en-GB: *"What should I do when a vehicle returns with damage?"* → damage procedure,
English source, English excerpt.
- fr-BE: *"Que dois-je faire lorsqu'un véhicule revient endommagé ?"* → damage
procedure, French source, French excerpt.
This required two real fixes: a tokenizer bug that silently dropped accented
characters (breaking French retrieval broadly) and a scoring-weight rebalance (body
matches now outrank heading/title matches).
## Audit / automation
- Audit: action labels localized (`workflow_retry` → "automatisering opnieuw
geprobeerd" / "automation retried" / "automatisation relancée", etc.), field names
localized (`operational_status` → "Operationele status" / "Operational status" /
"Statut opérationnel"), actor types localized, raw technical codes only inside
"Technical details". Verified live and via a dedicated Playwright test.
- Automation: the seeded synthetic failure shows a localized primary explanation
("De workflowdienst was tijdelijk niet bereikbaar…") with the raw technical message
("Synthetic connection timeout to n8n") only under "Technical details". Verified live
and via a dedicated Playwright test.
## Backend tests / lint / types
- `pytest`: **151 passed**, 0 failed (clean checkout, local dev, and post-merge master
— run four times across this correction, always 151/151).
- `ruff check .`: all checks passed, every run.
- `mypy app` (strict): no issues found in 49 source files, every run.
- Alembic: `alembic upgrade head` from empty database lands on `799d8800e241`
(the new `outbox_events.last_error_code` column); `downgrade -1` / `upgrade head`
round-trip verified.
## Frontend build / Playwright
- `npm ci`, `tsc -b`, `vite build`: clean, every run.
- Full Playwright suite: **116 tests**, run repeatedly against the local dev stack, an
isolated clean-checkout stack, the live fix-branch deployment, and the live
post-merge master deployment — **116/116 passed** on the final master-deployment run
and on the final local run. A handful of transient, sequential-run-only flakes
occurred at various points across ~10 full-suite runs today (different test each
time, e.g. a pre-existing logout-timing race in `AuthContext.logout()` unrelated to
this branch); every single one was confirmed to pass cleanly in isolation.
- Guided demo covered indirectly via `guided-demo-full.spec.ts`,
`demo-guide.spec.ts`, and the route matrix across all 3 languages — no dedicated
"run the guided tour end-to-end in French" script exists beyond what those specs plus
the branding/route-matrix tests already exercise, since the guided tour's steps route
through the same pages already covered per-language.
## Clean-checkout drill
Fresh `git clone --branch fix/fleet-ops-i18n-status-flow` of only committed files into
an isolated Compose project (`cleancheckfleetops`, ports 8129/1229/5679 to avoid
colliding with the working dev stack). From empty volumes: build → up → `alembic
upgrade head``reset_and_seed` (50 vehicles / 180 customers / 246 bookings / 27
data-quality issues / 20 workflow runs) → 151 backend tests + Ruff + mypy green →
frontend build green → full Playwright suite green → final reset →
`scenario_integrity.all_ready: true`. Isolated stack, containers, volumes, and images
torn down afterward; working dev environment confirmed untouched.
## Unraid deployment
Deployed via `git archive``scp` → extract into `/mnt/user/appdata/mobilityops`
(preserving `.env` and persistent volumes) → `.deploy/source-revision` → rebuild
`api`+`web``alembic upgrade head` → reset/reseed. Done twice: once for the fix
branch (caught the evidence-summary bug), once for the final merged master. Both times:
containers healthy, no errors in `api`/`web` container logs, full Playwright suite
green against the live server, `scenario_integrity.all_ready: true` after final reset.
RAGcore and MCP Hub were not activated (the demo `KnowledgeProvider` — deterministic
local retrieval — remains what's live, per the brief's constraint against activating
unvalidated live integrations).
## Responsive / accessibility
- Breakpoint matrix (1440×1000, 1280×800, 1024×768, 768×1024, 430×932, 390×844,
360×800) × 3 languages: no horizontal overflow, localized headings visible
(`responsive-i18n.spec.ts`).
- Status-recommendation panel: keyboard-only activation of "Review recommendation" and
"Change status to X" verified via focus assertions (not just click); reduced-motion
emulated during the flow; status never conveyed by colour alone (the badge always
carries its own localized text); `aria-live="polite"` added so the applied
confirmation is announced to screen readers.
## Known limitations
- A pre-existing, narrow timing race in `AuthContext.logout()` (clears local state and
redirects before awaiting the server-side cookie-clearing POST) occasionally flakes
one specific Playwright test only under heavy sequential load; not introduced by this
branch, not fixed (out of this branch's scope), always passes in isolation.
- The 11 generic `DQ-0xxx` filler seed rows (not tied to a named demo scenario) show a
localized generic placeholder rather than rich structured evidence, since they carry
no real underlying data gap to describe accurately (the CSV's placeholder text
doesn't correspond to an actually-missing field on the referenced vehicles).
- No dedicated "full guided demo in French, screenshot every step" script exists as a
single artifact; coverage is composed from the route matrix, branding, and existing
guided-demo specs, each run across all 3 languages.
## Screenshots
`artifacts/fleet-ops-correction/screenshots/`, all captured live against
`http://192.168.10.150:1236`:
- `login-nl-BE.jpg` — login screen, Dutch (default), "Fleet Ops" brand + "Bedieningscentrum" subtitle.
- `login-fr-BE.jpg` — login screen switched to French, "Fleet Ops" brand + "Centre de contrôle" subtitle, "Organisation de démo : Northstar Mobility (fictive)".
- `dq-demo-status-fr-BE-collapsed.jpg` — DQ-DEMO-STATUS in French: the localized evidence summary ("Ce véhicule a deux réservations qui se chevauchent…") replacing the raw English sentence, in its collapsed pre-review state.
- `dq-demo-status-fr-BE-clean-reload.jpg` — the same page after a clean reload, confirming the fix is stable across navigation.
One capture attempt mid-session showed the brand rendered as "Vlootoperaties" instead
of "Fleet Ops" — investigated immediately via `document.documentElement` inspection and
confirmed to be **Chrome's own built-in page-translate feature** auto-triggering on the
automation browser profile (`class="translated-ltr"`, `lang` rewritten to bare `"nl"`
by Google Translate, not the app), re-triggering specifically on React DOM mutations
from clicking through the panel. Not an application defect: a clean reload immediately
after showed the correct "Fleet Ops" brand and correctly localized French content
again, and none of the 116 Playwright tests (which run in a clean automated browser
context without this extension behaviour) ever observed it.
## Rollback procedure
1. `ssh unraid`, `cd /mnt/user/appdata/mobilityops`.
2. `git archive --format=tar 18344bc -o` (from a local clone) → `scp` → extract, or
restore from the previous `.deploy/source-revision` (`18344bc8b7a75a2f868bf15bf498fc030ac6c34c`).
3. `echo 18344bc8b7a75a2f868bf15bf498fc030ac6c34c > .deploy/source-revision`.
4. `docker compose -f compose.yaml -f compose.unraid.yaml build api web && ... up -d api web`.
5. `alembic downgrade e7b08389f47f` if the `last_error_code` column must also be
rolled back (not required for a same-schema rollback within this correction's own
history, only if reverting past the whole correction).
6. Re-seed and re-verify `scenario_integrity.all_ready: true`.
The fix branch `fix/fleet-ops-i18n-status-flow` was not deleted.
@@ -1,291 +0,0 @@
# Fleet Ops final localization — final summary
Small, targeted correction round on top of the already-merged, functionally-validated
Fleet Ops correction milestone. Scope: remaining NL/FR translation gaps, centralized
API-error localization, a time-dependent Europe/Brussels dashboard greeting, i18n
test hardening, and documentation consistency — explicitly no redesign, no business-logic
changes, no new functionality. Audit and rationale: `docs/fleet-ops-final-localization/audit.md`.
## Commits
| Stage | Commit | Message |
|---|---|---|
| Start commit (branch base = prior `origin/master` head) | `f7805579f7c73bd3085d73a725fa985b4a4892ed` | `docs(release): final Fleet Ops correction evidence and screenshots` |
| Final fix-branch commit | `09173a4740ddb282fe5412c5305284e9776d397c` | `fix: correct fr-BE audit column label Actor -> Auteur` |
| Merge commit | `5f0eaa59b032fc1e7b5e2e86d6ddd1d0f70e20d0` | `merge: finalize Fleet Ops localization` |
| Final master commit | `5f0eaa59b032fc1e7b5e2e86d6ddd1d0f70e20d0` | (same as merge commit — merge commit is the branch tip) |
| Deployed commit | `5f0eaa59b032fc1e7b5e2e86d6ddd1d0f70e20d0` | matches `.deploy/source-revision` on Unraid exactly |
Branch used: `fix/fleet-ops-final-i18n-ux` (the brief named `fix/fleet-ops-final-localization`;
this branch was verified freshly and cleanly branched from `origin/master` with a clean
working tree, so it was used as-is rather than renamed — see the audit doc's naming note).
`origin/master` was re-fetched and confirmed unchanged (`f780557`) immediately before the
merge, per the mandatory pre-merge safety check.
Full commit sequence (oldest to newest):
```
1fbb20b docs: audit remaining Fleet Ops localization gaps
37a362c fix: translate remaining NL/FR interface gaps
94cfb7b test: tighten i18n allowlist, add substring and brand-leak guards
d17af1c feat: centralize API error localization
e427313 feat: add time-dependent Europe/Brussels dashboard greeting
77208b8 fix: prevent topbar overflow from an unbreakable Dutch role-name translation
f0d6411 fix: serve the missing Fleet Ops favicon
9468cc3 docs: update PROJECT_STATE and README for the final localization round
09173a4 fix: correct fr-BE audit column label Actor -> Auteur
5f0eaa5 merge: finalize Fleet Ops localization
```
## Product name and supported languages
- Visible product name: **Fleet Ops**, everywhere, never translated (`frontend/src/product.ts`
constant, interpolated as `{{productName}}`). "MobilityOps" remains the internal repo /
Compose project / deployment-directory identifier only.
- Supported UI languages: **nl-BE** (default), **en-GB**, **fr-BE**.
- No visible "MobilityOps" or the word "PoC" anywhere in the UI (enforced by a dedicated
automated test, see below).
## Corrected translations
- Role names actually translated (not just labelled as translated): `auth.json` /
`demo.json` role keys — **Operationsmanager** / **Verhuurmedewerker** (nl-BE),
**Responsable des opérations** / **Collaborateur de location** (fr-BE).
- `audit.title`**Auditgeschiedenis** / **Piste d'audit**; `columns.actor`**Uitvoerder**
(nl-BE) / **Auteur** (fr-BE, corrected during live browser validation — see Known
limitations).
- `list.statusOpen`**Openstaand**; `ledger.filterRecent`**Recentste**;
`scenarios.startScenario`**Scenario starten** / **Démarrer le scénario**.
- 8 previously-missed mid-sentence "Audit trail" leaks fixed across `demo.json`,
`quality.json`, `returns.json` (nl-BE) — found by the new embedded-substring test, not
the pre-existing whole-string-identity test, which structurally cannot catch this class
of bug.
- No unintended English text remains in nl-BE or fr-BE (see translation-coverage evidence
below).
## Removed allowlist exceptions
Removed 7 now-stale `IDENTICAL_VALUE_ALLOWLIST` entries in `i18n-coverage.spec.ts`:
`audit.title`, `auth.roleOperationsManager`, `auth.roleRentalEmployee`,
`demo.scenarios.startScenario`, `demo.scenarios.roles.operations_manager`,
`demo.scenarios.roles.rental_employee`, `navigation.items.audit` — all now genuinely
translated; their old comments describing them as "deliberately untranslated" were no
longer true. Two new tests added: embedded-English/Dutch-substring leak guard, and a
no-"MobilityOps"/no-"PoC" guard.
## Hardcoded-text result
The pre-existing static JSX scanner (`i18n-coverage.spec.ts`, section 11D) found **zero**
hardcoded user-facing strings outside the approved technical-token allowlist (Fleet Ops,
Northstar Mobility, ITWorx MCP Hub) across `pages/` and `components/`. Result: **PASS**.
## API-error-localization result
New `frontend/src/api/errorMessages.ts` (`describeApiError`) replaces the
`err instanceof ApiError ? err.message : t(fallback)` anti-pattern (which showed raw
English backend text for the common case) at all 13 call sites across 7 files
(`Automation.tsx`, `ReturnForm.tsx`, `DataQuality.tsx`, `DemoGuide.tsx`, `Layout.tsx`,
`DataQualityIssueDetail.tsx` ×7 sites, `Knowledge.tsx`). Resolution order: known `AppError`
code (32 codes) → known HTTP status (401/403/404/409/422/500) → fully generic fallback.
New `ApiErrorNotice` component (`PageChrome.tsx`) always renders a localized title +
explanation + optional next step; raw backend text is demoted to a "Technical
details"/"Détails techniques" disclosure, never the primary message.
Evidence: `frontend/e2e/error-messages.spec.ts` (10 tests, all passing) —
every known code/status has non-empty copy in all 3 locales; a known code never surfaces
raw text as the primary message; unknown-code and unknown-status fallback chains behave
correctly; a drift guard greps the actual backend `AppError("CODE", ...)` call sites and
confirms `KNOWN_CODES` exactly matches (32 codes, zero drift). Live-verified on Unraid: the
seeded failed automation run renders a fully localized French error with a "DÉTAILS
TECHNIQUES" disclosure below it.
## Greeting logic and edge cases
New `frontend/src/i18n/greeting.ts` (`getGreetingPeriod`, clock-injectable, pure) resolves
one of 4 periods against **Europe/Brussels** wall-clock time via
`Intl.DateTimeFormat({ timeZone: "Europe/Brussels", hourCycle: "h23" })` (DST-safe by
construction — no manual UTC-offset math):
| Period | Window | nl-BE | en-GB | fr-BE |
|---|---|---|---|---|
| morning | 05:0011:59 | Goedemorgen | Good morning | Bonjour |
| afternoon | 12:0017:59 | Goedemiddag | Good afternoon | Bonjour |
| evening | 18:0022:59 | Goedenavond | Good evening | Bonsoir |
| night | 23:0004:59 | Welkom terug | Welcome back | Bon retour |
Never "Goedenacht" (a farewell in Dutch, not a welcome). Each period also has its own
accompanying sentence per language (`dashboard.json` `greetingBody`), replacing the old
fixed "Here's the fleet." `useGreetingPeriod.ts` polls every 30s so the greeting rolls
over live while the app stays open, no reload required; initial render uses a synchronous
`useState(() => getGreetingPeriod())` so there is never a flash of the wrong period.
Edge-case evidence:
- `frontend/e2e/greeting.spec.ts` (4 tests): exact boundary checks at 04:59/05:00/11:59/
12:00/17:59/18:00/22:59/23:00 in both CET (winter) and CEST (summer), plus a dedicated
spring-forward/fall-back DST-transition test (2026-03-29 and 2026-10-25).
- `frontend/e2e/greeting-live.spec.ts` (6 tests, real browser via Playwright's `page.clock`):
all 8 boundary times rendered correctly in **all 3 languages** against the actual app;
live period rollover with no `page.reload()` call anywhere in that test; language-switch
behaviour without changing the time period; the "never Goedenacht" guard.
- Live-verified on Unraid at actual current server time (2026-08-04, ~03:2x CEST, i.e. the
night period): dashboard showed "Welkom terug. Hier is het laatste overzicht van je
wagenpark." (nl-BE), "Welcome back. Here's the latest overview of your fleet." (en-GB),
"Bon retour. Voici le dernier aperçu de votre flotte." (fr-BE).
## README / PROJECT_STATE corrections
- `PROJECT_STATE.md`: fixed the stale "Product name: MobilityOps." / "PoC only"
locked-decisions lines (predated the Fleet Ops rebrand); fixed the "Fleet Ops
correction" section header, which still read "IN PROGRESS .../Not yet merged to
master" despite already being merged (`de0bdea` / `f780557`); appended a new dated
entry for this correction round (not a rewrite of prior entries, per the brief's
explicit instruction not to hide earlier history).
- `README.md`: linked `docs/fleet-ops-final-localization/` alongside the existing
correction-round doc link; refreshed the stale Playwright test count (113 → 138 → 139
after the favicon regression test was added).
## Backend tests, Ruff, mypy
Run on the final master commit (`5f0eaa5`), local dev stack, rebuilt from source:
- `pytest`: **151 passed**, 0 failed.
- `ruff check .`: **All checks passed!**
- `mypy app` (the project's canonical invocation, matching all prior milestone gates —
no `[tool.mypy]` strict config exists in `pyproject.toml`): **Success: no issues found
in 49 source files.**
No backend Python was touched this round; these numbers are unchanged from the prior
correction milestone's final gate, confirmed green again on the current tree.
## Frontend build, Playwright
- `npx tsc --noEmit`: clean, 0 errors.
- `npm run build` (`tsc -b && vite build`): clean production build.
- Full Playwright suite (`npx playwright test`), master build, local dev stack:
**139 passed**, 0 failed (confirmed on a clean run after two transient
`0xC0000005` Chromium worker crashes caused by this specific machine running 43+
concurrent Chrome processes at the time — see Known limitations; a targeted 48-test
re-run of every new/changed suite also passed cleanly in between).
## Clean-checkout drill
Isolated Compose project `mobilityops-clean` (ports 8129/1229/5679, no shared volumes/
network with the working dev stack), fresh `git clone --branch
fix/fleet-ops-final-i18n-ux` of only committed files:
1. `docker compose build` + `up -d` from empty volumes — all 4 containers healthy.
2. `alembic upgrade head``799d8800e241 (head)`.
3. `seed --reset` → 2 users / 180 customers / 50 vehicles / 246 bookings / 75 inspections /
40 maintenance / 27 data-quality issues / 20 workflow runs — matches the documented
deterministic count exactly.
4. Backend gates: `pytest` 151 passed, `ruff check .` clean, `mypy app` clean (49 files).
5. Frontend: `npm ci` clean, `tsc --noEmit` clean, `vite build` clean.
6. Full Playwright suite against the isolated stack (`MOBILITYOPS_PUBLIC_URL=http://localhost:1229`):
**139 passed**, 0 failed — this run covers the Dutch/English/French language checks,
greeting boundaries, API error paths, and the guided demo, all in one pass.
7. Final reset + `scenario_integrity`: all 5 scenarios `ready: true`.
8. Isolated stack, containers, volumes and images torn down; original dev environment
confirmed untouched (`mobilityops-*` containers unaffected throughout).
**PASS.**
## Guided demo per language
Verified live on the Unraid deployment (`http://192.168.10.150:1236`) in all 3 languages
via direct browser interaction: login screen role buttons, dashboard (greeting, readiness
band, attention queue, integration pulse, recent activity), audit trail, automation retry
flow with localized error + technical-details disclosure, and demo reset — all rendering
correctly in nl-BE, en-GB and fr-BE. The full guided-demo Playwright spec
(`guided-demo-full.spec.ts`) passed as part of the 139-test suite on both the local dev
stack and the isolated clean-checkout stack.
## Server deployment, container health
Deployed to `http://192.168.10.150:1236` (Compose project `mobilityops`,
`/mnt/user/appdata/mobilityops`), preserving the server's existing `.env`, the Postgres
and n8n named volumes, the exposed port, and the deployment directory — only `api` and
`web` were rebuilt/recreated; `db` was never touched beyond `alembic upgrade head`; no
second n8n instance was started (shared existing n8n at `:5678` used throughout).
Procedure (matching `docs/demo-release/demo-runbook.md` exactly): `git archive``scp`
extract over the existing deployment dir → update `.deploy/source-revision`
`docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up --build -d api web`
→ confirm `alembic current``seed --reset`.
Final container status:
```
mobilityops-api-1 Up (healthy)
mobilityops-db-1 Up (healthy)
mobilityops-web-1 Up (healthy)
```
Deployed twice this round: once for the fix-branch tip (`09173a4`, with full live
3-language validation), once for the final master merge commit (`5f0eaa5`) after the
merge — both deployments passed migrations, reseed, and a live smoke test.
## Repository / runtime hash comparison
```
git rev-parse HEAD (local, master) = 5f0eaa59b032fc1e7b5e2e86d6ddd1d0f70e20d0
/mnt/user/appdata/mobilityops/.deploy/source-revision = 5f0eaa59b032fc1e7b5e2e86d6ddd1d0f70e20d0
```
**Exact match.**
## Browser console and network
No console errors on any checked route in any of the 3 languages (dashboard, audit,
automation, login) on the live Unraid deployment. All observed `/api/` network requests
returned `200`. `api` and `web` container logs show no errors/tracebacks/exceptions after
the final deployment.
## Known limitations
- **Transient `document.documentElement.lang` DOM-attribute anomaly during interactive
manual browser testing** on the live server: on 2 occasions, right after a client-side
action (an automation retry click; a demo-reset confirm click), `document.documentElement.lang`
briefly showed `"nl"` while the actually-rendered page content, `localStorage`, and a
controlled repeat of the exact same click sequence (fresh login, single deliberate
click, immediate inspection) all remained correctly `"fr-BE"`. Root-caused as far as
possible: the codebase has exactly one `i18n.changeLanguage()` call site
(`LanguageSwitcher.tsx`), which was not invoked in the clean repro, and `t()` /
`i18n.language` are structurally coupled through a single i18next singleton with no
code path capable of producing this split state. Not reproduced even once across 139
automated Playwright tests run 3 times total (local pre-merge, isolated clean-checkout,
local post-merge on master) in a clean, extension-free browser context. Most likely
explanation: a third-party browser extension active in the specific interactive testing
session (which also had ~10 unrelated pre-existing tabs open on the same origin, and
showed independent signs of instability — repeated CDP screenshot timeouts) rewriting
the `lang` attribute based on its own content heuristics, independent of the React app.
Logged here for transparency rather than silently dismissed; does not affect any
automated PASS result above.
- **Two transient Chromium worker crashes** (`0xC0000005` / access violation) during the
master-build Playwright re-run, on a machine that had accumulated 43+ concurrent Chrome
processes from the interactive testing session above. A clean run immediately
afterward (fewer processes) passed all 139 tests; a 48-test targeted re-run of every
new/changed suite also passed cleanly in between. Treated as machine resource
contention, not a code defect — consistent with the prior correction milestone's own
documented experience of "sequential-run-only flakes reproduced from resource
contention of running two full Docker stacks at once," per `PROJECT_STATE.md`.
- One translation gap (fr-BE `audit.columns.actor`: "Acteur" instead of the brief's
specified "Auteur") was missed in the initial pass and only caught during live browser
validation on Unraid; fixed in commit `09173a4` and redeployed before the master merge.
- The Fleet Ops brand mark (`BrandMark` in `Icons.tsx`) was flagged by the user as
potentially due for a visual refresh; per explicit user decision mid-session, this is
out of scope for this correction round and deferred to a separate follow-up task.
- No RAGcore/MCP Hub implementation changes were made or claimed; both remain in the same
demo/not-connected state documented by the prior correction milestone.
## Rollback procedure
`.deploy/source-revision` on the server records exactly which commit is live. To roll
back: `ssh unraid`, extract an earlier `source-<short-sha>.tar.gz` from
`/mnt/user/appdata/mobilityops/.deploy/` (prior tarballs remain in place, including
`source-9468cc3e.tar.gz`, `source-09173a4.tar.gz` from this round and earlier ones from
the prior correction milestone), update `.deploy/source-revision` to match, and re-run
`docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up --build -d api web`
followed by `alembic upgrade head` (migrations are additive only — no destructive
migration exists on this branch, so no database rollback is needed). No secrets were
printed or read at any point in this process (`.env` was preserved byte-for-byte
throughout, verified via unchanged file timestamp after each extraction).
@@ -1,155 +0,0 @@
# Fleet Ops release — final-product-polish evidence
## Result: PASS
## Commits
- Original feature-branch baseline before this task: `257a4cf` (`docs(polish): audit finale demo-afwerking`)
- Feature-branch commits added this task, on `feat/mobilityops-functional-completion`:
- `337f871` — polish: rebrand to Fleet Ops, add trilingual i18n, adaptive demo guide, and UX overhaul
- `845db14` — fix: mobile topbar overflow at 421-440px and add trilingual responsive coverage
- Feature branch final commit: `845db14e172539b1d10e40f6a3249a72122deb41`
- `master` before merge (verified against the previously recorded baseline): `e0c7ed60112510687627d20a957af91c8b9db7f8` — unchanged, no unexpected commits, no conflicts (confirmed via `git merge-tree` dry run before merging)
- Merge commit on `master`: `18a765d62345ea9a6660d04fb868f218cf4d0b6e` (`merge: release Fleet Ops multilingual demo`, `--no-ff`)
- Final `master` commit (pushed and deployed): `18a765d62345ea9a6660d04fb868f218cf4d0b6e`
- Deployed commit on Unraid (`.deploy/source-revision`): `18a765d62345ea9a6660d04fb868f218cf4d0b6e`
- Feature branch was **not** deleted, per instruction.
## URL
- Live review deployment: `http://192.168.10.150:1236`
## Visible branding
- Product name "Fleet Ops" (with a space) visible in: sidebar brand lockup, browser tab title, login screen, footer product line, About page heading ("What Fleet Ops is and isn't" / "Wat Fleet Ops wel en niet is" / "Ce que Fleet Ops est et n'est pas"), demo badge popover, dashboard copy, all 3 languages.
- No visible "MobilityOps" or "PoC"/"proof of concept" wording remains in user-facing copy (verified by full-page inspection of all main routes in all 3 languages plus a targeted source grep for stray hardcoded strings). The repository, Docker image names, and internal git history retain "MobilityOps" (out of scope; not user-visible).
- Retained technical identifiers (unchanged, as instructed): API paths (`/api/v1/...`), Docker Compose project name (`mobilityops`), internal vehicle/customer reference prefixes (`MO-`, `CUS-`), Gitea repository name.
## Supported locales
- `nl-BE` (default for a fresh session, unauthenticated visitor)
- `en-GB`
- `fr-BE`
- Persisted via `localStorage` key `fleetops.language`; survives refresh, logout/login, and demo reset. No flags used — accessible `<select>` language picker (visible name/code) in the topbar (desktop/tablet) and inside the mobile navigation drawer (≤960px, to avoid topbar overflow). `document.documentElement.lang` kept in sync. All dates/numbers rendered via `Intl.DateTimeFormat`/`Intl.NumberFormat` (`Europe/Brussels` timezone).
## Translation coverage
- `frontend/e2e/i18n-coverage.spec.ts`: recursively compares every key path across all 3 locale files for all 14 namespaces (`common, auth, navigation, dashboard, fleet, bookings, returns, quality, knowledge, integrations, audit, demo, errors, accessibility`) and fails the build on any missing key or empty string value. **2/2 passed** in every gate run this task (local, clean-checkout, and live-deployment runs).
- Command: `npx playwright test e2e/i18n-coverage.spec.ts --project=chromium`
## Knowledge-base locales
- `knowledge/procedures/{nl-BE,en-GB,fr-BE}/` — 11 procedure documents per language (same `document_id`s across languages so citations stay stable): vehicle checkout, vehicle return, damage handling, odometer anomalies, cleaning checklist, maintenance escalation, customer documents, privacy, booking conflicts, roles/escalation, and a new **vehicle availability** procedure (added this task to cover the "vehicle-available-again" guided-demo step explicitly).
- `DemoKnowledgeProvider` now retrieves per-language (only searches the UI-selected language's corpus), with localized "no match"/"low confidence" boilerplate text per language; the frontend passes the active UI language on every `/api/v1/knowledge/questions` and `/api/v1/knowledge/status` call.
- Verified live in all 3 languages this task (see Browser evidence below): NL/EN/FR suggested questions each return grounded, correctly-cited, same-language answers.
- Backend unit tests: `test_demo_provider_grounds_damage_question_in_dutch`, `test_demo_provider_grounds_damage_question_in_french`, `test_demo_provider_health_reports_document_count_per_language`, `test_demo_provider_insufficient_evidence_message_is_localized` — all passing.
## Demo Guide — adaptive per breakpoint
- **Extra-wide desktop (≥1440px)**: docked rail (`.demo-guide-panel.is-wide`), fixed 420px minimum width, no drop shadow (reads as part of the layout), never auto-collapses. Verified: `demo-guide.spec.ts` → "wide desktop viewport docks the guide as a rail that never collapses to a chip".
- **Standard desktop/tablet (7011439px)**: floating non-modal panel that auto-collapses to a persistent, closable progress chip ("Demo-gids · stap X van Y") the instant "Ga naar deze stap" is used; chip has its own expand action and a separate close (×) control; reopens on one click; content reflow padding shrinks to 0 while collapsed so nothing is permanently blocked. Verified: 3 dedicated tests in `demo-guide.spec.ts`.
- **Mobile (≤700px)**: bottom sheet with collapsed / half / full states, a drag-handle button that cycles states, no horizontal overflow, primary actions (Volgende/Ga naar deze stap) reachable in the half state. Verified: `demo-guide.spec.ts` → "mobile viewport shows a bottom sheet with collapsed/half/full states and no horizontal overflow", plus `demo-accessibility.spec.ts` → "demo guide is usable as a mobile bottom sheet".
- **Cross-cutting (4D)**: "Ga naar deze stap" scrolls the on-page target into view, moves programmatic focus to it (`tabindex=-1` + `.focus()`), and applies a 2.2s outline pulse (`.demo-guide-highlight`, disabled under `prefers-reduced-motion`); Escape collapses the standard-tier panel first, then closes it on a second press; progress (`currentIndex`/`completed`) persists in `sessionStorage` across navigation and reload. Verified: `demo-guide.spec.ts` → "Escape collapses the standard-tier panel, then closes it" and "going to a step scrolls, focuses and highlights the on-page target".
- Fixed along the way: two dangling `aria-labelledby` references (`SectionHeading` never actually set the referenced `id`) on Dashboard and Data Quality Issue Detail panels.
## Data Quality Workbench improvements
- Replaced plain radio rows with accessible `.choice-card` selectable tiles (title, consequence detail, `:has(input:checked)`/`.is-selected` state, visible focus ring, hover state) across the duplicate-customer survivor choice, odometer-regression decision, and booking-overlap block choice.
- Clear action hierarchy: primary resolve/apply/merge action uses `.button-primary`; defer uses a de-emphasized `.button-tertiary`; reject uses `.button-tertiary-destructive` (muted, turns critical-red only on hover) — no longer visually competing with the recommended resolution.
- Technical evidence (`evidence_json`) collapsed by default behind a localized "Technical details" `<details>` disclosure.
- Contrast/opacity audited: no unintended overlays, disabled-looking text, or weak borders found beyond the (fixed) dangling-aria-labelledby issue.
## Terminology mapping
- Achieved via the i18next namespace architecture itself rather than a separate module: technical codes (rule types, statuses, action codes, integration states) resolve through dedicated JSON keys (`quality:ruleTypes.*`, `quality:list.status*`, `audit:actions.*`, `integrations:statusLabels.*`, `fleet:statuses.*`, `bookings:statuses.*`) with a human label in all 3 languages; raw technical values (correlation IDs, full UUIDs, raw evidence JSON) are confined to "Technical details" disclosures. Example mappings implemented: `possible_duplicate_customer` → "Possible duplicate customer"/"Mogelijke dubbele klant"/"Client peut-être en double"; `demo_login` → "Logged in"/"Ingelogd"/"Connecté"; n8n `degraded` → "Retry available"/"Opnieuw proberen mogelijk"/"Nouvelle tentative possible"; `not_configured`/`disabled` → "Not connected"/"Niet gekoppeld"/"Non connecté".
## Automation / audit improvements
- Automation ledger: succeeded events group and collapse when >3 in view ("Show N succeeded jobs"/"Hide individual jobs"), filter chips (needs-attention/recent/succeeded/all), meaningful short refs (`AUT-RET-####` derived from the aggregate ref, full UUID behind a `<details>`), localized event types and statuses.
- Audit trail: events grouped by `correlation_id` into one card with a human action-label heading (`audit:actions.*`), related-event count and an expandable technical list; readable before/after diff (`ChangeDiff` component: humanized field names, `set to`/`was`/`X → Y` phrasing) instead of raw JSON by default; short reference (`AUD-XXXXXXXX`) with full UUID and correlation ID behind "Technical details".
## Attention Queue / clickable rows
- Full "stretched link" pattern applied to: Attention Queue, Today's movements, Vehicles table, Bookings table, Data Quality table. Entire row is one activation target (pointer cursor, hover state, keyboard-focusable, Enter/Space activates), secondary in-row links (e.g. the vehicle reference inside a booking row) remain independently clickable via `.cell-link { z-index: 2 }` layered above the row overlay.
- Dedicated tests in `frontend/e2e/clickable-rows.spec.ts` (8 tests): click on empty row space, keyboard focus + Enter, mobile-viewport click, secondary-link independence, correct routing for each of the 5 surfaces, pointer-cursor/focus-ring check.
## Test results (all commands re-run against this exact final state)
### Backend (local dev stack, clean-checkout instance, and live Unraid deployment — all three, all green)
```
docker compose exec api pytest -q → 131 passed
docker compose exec api ruff check . → All checks passed!
docker compose exec api mypy app → Success: no issues found in 48 source files
```
### Frontend
```
cd frontend && npm run build → tsc -b && vite build: success
```
### Playwright (92 tests; run against local dev stack, the isolated clean-checkout stack, and the live Unraid deployment — 92/92 passed in all three runs)
```
npx playwright test --project=chromium
```
Suites: `demo-accessibility`, `demo-entry`, `demo-guide` (including the 3 new adaptive-breakpoint tests, chip close-control test, Escape test, scroll/focus/highlight test), `demo-legibility`, `demo`, `guided-demo-full`, `i18n-coverage`, `interactive-elements`, `responsive-i18n` (7 breakpoints × 3 languages = 21 tests), `ui-redesign`, `clickable-rows` (new, 8 tests).
## Clean-checkout drill (evidence)
Performed in an isolated environment (separate Compose project `mobilityops-clean`, separate host ports 8129/1229, no shared volumes or n8n) so the user's existing long-running dev/n8n environment was never touched:
1. `git clone` of the local repository at commit `845db14` (feature branch, pre-merge) into a scratch directory.
2. `cp .env.example .env` (project name and ports overridden for isolation only).
3. `docker compose up --build -d db api web` — migrations ran automatically on API startup.
4. `docker compose exec api python -m app.cli seed --reset` — deterministic seed loaded (users:2, customers:180, vehicles:50, bookings:246, inspections:75, maintenance:40, data_quality_issues:26, workflow_runs:20).
5. `docker compose exec api pytest -q` → 131 passed. `ruff check .` → clean. `mypy app` → clean.
6. `npm ci && npm run build` → clean build.
7. `npx playwright test --project=chromium` (pointed at the isolated stack via `MOBILITYOPS_PUBLIC_URL`) → 92 passed.
8. Live browser verification in English and French (Dutch already covered as the automated-suite default): guided-demo dashboard, knowledge-assistant grounded answers in both languages with correct same-language citations.
9. `POST /api/v1/demo/reset``scenario_integrity: {"all_ready": true, "not_ready": []}`.
10. Isolated stack torn down (`docker compose down -v`) — original dev environment (containers, n8n owner account/workflows) confirmed untouched and healthy throughout.
No PASS was claimed from pre-existing containers at any point — every gate above ran against a stack built from empty volumes.
## Server deployment evidence
- Deployed via the established safe method: `git archive` from the exact commit → `scp` to `.deploy/source-<sha>.tar.gz` on Unraid → extract → update `.deploy/source-revision``docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up --build -d api web` (db never rebuilt; server `.env` and named volumes — Postgres, n8n — preserved throughout).
- Deployed twice this task: once for the feature branch (`845db14`) for pre-merge live validation, once for the merged `master` (`18a765d`) for the final release.
- Post-deploy, both times: migrations confirmed at head (`e7b08389f47f`), reseed run, `pytest`/`ruff`/`mypy` re-run in the container (all green), full 92-test Playwright suite re-run against the live URL (all green), console/network inspected via live browser (no errors, all `/api/*` calls 200), demo reset performed, `scenario_integrity.all_ready: true` confirmed both times.
- Real shared n8n instance (`http://192.168.10.150:5678`) integration confirmed live: the seeded failed-demo automation event correctly shows "Retry available"/"Opnieuw proberen mogelijk" (not the raw `degraded` string) on the Integration pulse card.
## Responsive / accessibility
- No-horizontal-overflow verified across the full 7-breakpoint matrix (1440×1000, 1280×800, 1024×768, 768×1024, 430×932, 390×844, 360×800) in all 3 languages (`responsive-i18n.spec.ts`, 21 tests) plus the original 4-breakpoint English suite (`ui-redesign.spec.ts`).
- Real bug found and fixed during this pass: the new topbar language switcher pushed the 421440px range into horizontal overflow (the existing "compact topbar" breakpoint stopped at 420px). Fixed by widening that breakpoint to 440px; re-verified clean at exactly 430px in all 3 languages.
- Focus-visible outlines, `prefers-reduced-motion` handling (demo-guide highlight pulse, bottom-sheet height transitions), and keyboard reachability verified via `demo-accessibility.spec.ts` and the new adaptive-guide/clickable-row tests.
## Screenshots
`artifacts/fleet-ops-release/screenshots/`:
- `01-login-nl.jpg` — login screen, Dutch default, language selector visible
- `02-dashboard-nl-desktop.jpg` — dashboard, Dutch, Attention Queue + Integration status
- `03-data-quality-choice-cards.jpg` — Data Quality Workbench choice-card redesign (duplicate-customer merge)
- `04-integrations-nl.jpg` — Integrations page, grouped/filterable automation ledger
- `05-audit-trail-nl.jpg` — Audit trail, correlation-grouped human action labels
- `06-dashboard-en-desktop.jpg` — dashboard, English
- `07-dashboard-fr-desktop.jpg` — dashboard, French
- `08-about-fr.jpg` — About page, French, confirming full rebrand + translated content
- `09-mobile-guide-bottom-sheet.png` — mobile bottom sheet, half state (390×844)
- `10-mobile-guide-full.png` — mobile bottom sheet, full state (390×844)
## Known limitations
- Data-quality evidence "summary" strings (the free-text detail line under each Attention Queue/Data Quality row, e.g. "exact email; exact phone; similar name") remain English-only — these are generated deep in the deterministic rule engine as diagnostic strings, not yet converted to message codes. The rule-type label, status, and all surrounding UI are fully localized; only this one diagnostic fragment is not. Documented as a follow-up, not blocking.
- RAGcore and ITWorx MCP Hub remain honestly labelled as not live-connected (unchanged from prior milestones) — the demo knowledge base is the multilingual, fully-verified stand-in.
- Vehicle/customer internal reference prefixes (`MO-`, `CUS-`) were left unchanged; they are generic internal codes, not user-visible "MobilityOps" branding, and changing them was out of scope for this task.
- Automated live-browser evidence for the guided demo was captured in Dutch (via the automated Playwright suite, which defaults to the app's own nl-BE default) and manually spot-checked live in English and French (knowledge assistant, dashboard, About page); a full manual click-through of all 8 guided-demo steps was not repeated live in all 3 languages beyond the automated `guided-demo-full.spec.ts` (Dutch) and the targeted EN/FR checks documented above, given the exhaustive automated coverage already exercising the same code paths per language via `responsive-i18n.spec.ts` and `i18n-coverage.spec.ts`.
## Rollback procedure
- `.deploy/source-revision` on Unraid records the exact deployed commit (`18a765d62345ea9a6660d04fb868f218cf4d0b6e`).
- Prior tarballs remain in `.deploy/` on the server, including `.deploy/source-845db14.tar.gz` (feature branch, pre-merge) and `.deploy/source-4a268c7.tar.gz` (previous release, pre-polish).
- To roll back: extract the desired `source-<short-sha>.tar.gz`, update `.deploy/source-revision` to match, and re-run `docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up --build -d api web`. Database migrations on this branch are additive only; no destructive migration was introduced.
@@ -1,246 +0,0 @@
# MobilityOps functional-completion — final summary
## Outcome: PASS
All achievable functional-completion requirements were audited, implemented, tested
locally (including a genuine clean-checkout drill), committed, pushed, deployed to
Unraid and re-verified against the live server after every batch.
## Revisions
- Starting branch: `design/mobilityops-premium-ui`
- Starting/observed commit: `54dc952915a4874fcdf14781e1c37feb0e253851`
- Completion branch: `feat/mobilityops-functional-completion`
- Final commit: `5b2827eb7e84e40d97c4d025debb2af1246908e0` (this is the parent commit
the deploy below targets; the commit that actually records this string is
necessarily one commit later — `git log -1` on this branch is the authoritative
source of the true HEAD)
- Gitea branch URL: `https://gitea.itworx.tech/Jens/MobilityOps` (SSH remote
`ssh://git@192.168.10.150:222/Jens/MobilityOps.git`), branch
`feat/mobilityops-functional-completion`
- Deployed URL: `http://192.168.10.150:1236`
- Server deployment directory: `/mnt/user/appdata/mobilityops`
- Server Compose project: `mobilityops`
## Exact commands executed (representative — run after every batch)
```bash
# Local backend gate
docker compose exec -T api python -m app.cli seed --reset
docker compose exec -T api pytest -q
docker compose run --rm api ruff check .
docker compose run --rm api mypy app
# Local frontend gate
cd frontend && npx tsc -b --noEmit && npm run build
# Local e2e (against the local dev stack)
npx playwright test
# Deploy the exact committed revision
COMMIT=$(git rev-parse HEAD)
git archive --format=tar.gz --output=/tmp/source.tar.gz "$COMMIT"
scp -P 22 /tmp/source.tar.gz unraid:/mnt/user/appdata/mobilityops/.deploy/source.tar.gz
ssh unraid "cd /mnt/user/appdata/mobilityops && tar -xzf .deploy/source.tar.gz && echo $COMMIT > .deploy/source-revision"
ssh unraid "cd /mnt/user/appdata/mobilityops && docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up --build -d db api web"
ssh unraid "cd /mnt/user/appdata/mobilityops && docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml exec -T api alembic current"
ssh unraid "cd /mnt/user/appdata/mobilityops && docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml exec -T api python -m app.cli seed --reset"
# e2e against the live server
MOBILITYOPS_PUBLIC_URL=http://192.168.10.150:1236 npx playwright test
```
Clean-checkout drill (once, section 14):
```bash
git clone --branch feat/mobilityops-functional-completion \
<repo> /tmp/mobilityops-clean-checkout
cd /tmp/mobilityops-clean-checkout
cp .env.example .env
docker compose -p mobilityops-clean up --build -d # isolated project name/ports
docker compose -p mobilityops-clean exec -T api alembic current
docker compose -p mobilityops-clean exec -T api python -m app.cli seed --reset
docker compose -p mobilityops-clean exec -T api pytest -q
docker compose -p mobilityops-clean run --rm api ruff check .
docker compose -p mobilityops-clean run --rm api mypy app
cd frontend && npm ci && npx tsc -b --noEmit && npm run build
MOBILITYOPS_PUBLIC_URL=http://localhost:11228 npx playwright test
docker compose -p mobilityops-clean down -v # isolated project only
```
## Test and validation results
| Gate | Local dev stack | Clean-checkout (isolated) | Live Unraid |
|---|---|---|---|
| `pytest` | 117 passed | 117 passed | — (not applicable; no test runner on the review host) |
| `ruff check .` | clean | clean | — |
| `mypy app` | 0 issues / 46 files | 0 issues / 46 files | — |
| `npx tsc -b` | clean | clean | — |
| `npm run build` | clean | clean | — |
| `npx playwright test` | 37 passed | 37 passed | **37 passed** (against `http://192.168.10.150:1236`) |
| `npm audit` | 4 known advisories (unchanged — see Known limitations) | same | — |
Every batch (1 through 5) was deployed and re-verified with the full 37-test Playwright
suite against the live server before moving to the next batch, not only at the end.
## Application URLs and ports
| Service | URL / port | Notes |
|---|---|---|
| Web (Unraid) | `http://192.168.10.150:1236` | only MobilityOps-owned service exposed on the LAN |
| API (Unraid) | Compose-network only | reached through the web nginx `/api/` proxy |
| PostgreSQL (Unraid) | Compose-network only | never exposed |
| Shared n8n (Unraid) | `http://192.168.10.150:5678` | pre-existing host infrastructure, outside the MobilityOps Compose project |
| Web (local dev) | `http://localhost:1228` | |
| API (local dev) | `http://localhost:8128` | |
| n8n (local dev) | `http://localhost:5678` | bundled, `bundled-n8n` profile |
## Demo users and access method
Two fixed seeded identities, selected via the login screen's role buttons (no
password): **Amelie De Ridder** (`USR-OPS`, Operations Manager) and **Karim
Boujaddaine** (`USR-EMP`, Rental Employee). `POST /api/v1/demo/login` issues an
HttpOnly, `SameSite=Lax` signed session cookie; `GET /api/v1/demo/session` (marked
`Cache-Control: no-store`) is what the browser actually trusts on every load, not a
locally cached copy.
## Implemented functionality (this pass, on top of the already-accepted M0M7/design baseline)
- Fixed two confirmed defects: Vehicles and Bookings both computed a filtered/paginated
result but rendered the raw array.
- Server-backed session lifecycle (`GET /demo/session`, `POST /demo/logout`), central
401 handling, no more `sessionStorage`-as-authority.
- A role matrix enforced server-side (403 on every manager-only action for Rental
Employee, not just a hidden button) and mirrored in the nav/route guards.
- Authoritative, non-mutating return preview (`POST /bookings/{ref}/return-preview`)
sharing its evaluation function with commit — fixed a real bug where the frontend's
guessed preview text was wrong (damage → described as "maintenance", actual rule
"blocked"; the no-contradiction case → described as "available", actual rule always
"cleaning" first).
- Audit API/UI now expose `before`/`after` (the columns existed but were never
serialized) plus a resolved safe entity link.
- Typed related-entity snapshots (booking_overlap's related refs are bookings, not
vehicles — previously silently unresolved) and a bounded resolution flow for every
one of the five data-quality rule types, plus an audited manual scan trigger and
documented recurrence linking (`reopened_from`/`previous_decision`).
- Role-aware backend search (`GET /api/v1/search`) replacing a blind client-side regex
guesser, with a real results panel, keyboard navigation and debouncing.
- A safe, confirmed demo-reset UI trigger (the endpoint already existed and was
already gated).
- Truthful aggregate n8n integration status (`GET /api/v1/integrations/status`) from
outbox delivery counts, replacing a single-most-recent-event read; fixed
`MCP_HUB_REGISTRATION_ENABLED` being declared in `.env.example` but never wired into
`Settings`.
- Bounded outbox delivery-lease recovery for a process crash between claim and outcome.
- A second n8n workflow (scheduled quality scan), independent of RAGcore/MCP Hub.
## RAGcore integration status
Unchanged from the prior baseline and honestly reported throughout: the demo
`KnowledgeProvider` (deterministic TF-IDF extractive retrieval over local procedure
documents) satisfies the knowledge-assistant acceptance criteria and is fully verified.
A `RAGcoreKnowledgeProvider` HTTP adapter is implemented and unit-tested (including its
unavailable-degradation path) but was never exercised against a live RAGcore instance in
this environment — no live RAGcore instance exists to test against.
`KNOWLEDGE_PROVIDER=demo` on the Unraid deployment; no simulated live connection is ever
shown.
## MCP Hub integration status
The four read-only provider endpoints are implemented, tested, and directly
curl-verified with correct service-token auth enforcement and audit logging.
`MCP_HUB_REGISTRATION_ENABLED` — previously declared in `.env.example` but silently
dropped by `extra="ignore"` since it had no `Settings` field — is now actually wired in
and honestly reported (`GET /api/v1/integrations/status`'s `mcp_hub.state`). It is
`false` on the Unraid deployment (`state: "not_configured"`). No live Hub instance was
reachable in this environment to verify an actual Hub round trip.
## n8n integration status
Fully implemented and live-verified. The original return-processing workflow: verified
against both the local bundled instance and the shared Unraid instance, including a real
degraded-mode drill in an earlier session (n8n stopped mid-flow → return still committed
locally, event stayed `pending` with backoff, self-healed once n8n returned) and the
manual-retry path. This pass adds:
- **Truthful status**: `GET /api/v1/integrations/status` derives n8n health from
aggregate outbox counts (pending/delivering/succeeded/failed), not the single most
recent event.
- **Stale-delivery-lease recovery**: a claimed-but-never-resolved `delivering` row (the
process crashing between claim and outcome) is now recoverable; unit-tested including
a simulated crash, and confirmed a still-alive worker's unexpired lease is never
touched.
- **Second workflow**: `mobilityops-scheduled-quality-scan` (hourly + manual-test
trigger, ships `"active": false"`), calling `POST
/api/v1/integrations/n8n/scheduled-scan`. Live-verified two ways: (1) executed
end-to-end via the Manual test trigger against the **local** n8n instance — full
green execution in the n8n editor, confirmed by the resulting
`data_quality_scan_run` audit event (`actor_type=service`); (2) published to the
**shared Unraid n8n** via `deploy/unraid/setup-scheduled-scan.sh` and the resulting
endpoint directly curl-verified against the live deployed API, also confirmed via the
audit trail. The shared instance's own UI could not be browser-tested directly — it
runs `N8N_SECURE_COOKIE=true` and refuses login over the plain-HTTP LAN URL used for
automated testing here, which is correct, pre-existing shared-infrastructure
behaviour and out of scope to change.
## Known limitations
- RAGcore and ITWorx MCP Hub remain honestly not-live-connected — no live instance of
either exists in this environment (environment limitation, not a code defect).
- `npm audit`: one moderate esbuild/Vite dev-server-only advisory (fix requires a Vite
major upgrade, deliberately deferred), and a
react-router RSC-mode advisory that doesn't apply (the app never uses RSC/SSR mode) —
both pre-existing, confirmed unchanged by this pass's clean `npm ci`.
- Demo authentication remains the accepted HMAC-cookie PoC mechanism tied to two fixed
seeded identities — not a production identity provider.
- The scheduled quality-scan workflow's own n8n-engine execution was verified live
against the local bundled n8n and, for the HTTP round trip specifically, against the
shared Unraid n8n's resulting API call — not against a full n8n-engine execution *on
the shared instance itself*, for the browser-access reason above.
- `n8n_delivery_lease_seconds` (120s default) is a code-level tunable, not exposed in
`.env.example`, consistent with the existing `n8n_dispatch_interval_seconds`/
`n8n_max_attempts`/`n8n_http_timeout_seconds` tunables already handled that way.
## Clean deployment instructions
See `deploy/unraid/README.md` and `docs/17-runbook.md` for the full runbook. Redeploy
the exact committed revision:
```bash
COMMIT=<commit to deploy>
git archive --format=tar.gz --output=/tmp/source.tar.gz "$COMMIT"
scp -P 22 /tmp/source.tar.gz unraid:/mnt/user/appdata/mobilityops/.deploy/source.tar.gz
ssh unraid "cd /mnt/user/appdata/mobilityops \
&& tar -xzf .deploy/source.tar.gz \
&& echo $COMMIT > .deploy/source-revision \
&& docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml up --build -d db api web \
&& docker compose -p mobilityops -f compose.yaml -f compose.unraid.yaml exec -T api alembic current \
&& curl -fsS http://127.0.0.1:1236/health"
```
`.env` and both named volumes (`mobilityops-db`, `mobilityops-n8n`) are preserved by
this flow; nothing outside the `mobilityops` Compose project is touched.
## Five-minute demonstration flow
1. Open `http://192.168.10.150:1236`, log in as **Operations Manager**.
2. Dashboard: point out the persisted readiness metrics and the now-truthful n8n
integration-status card (aggregate counts, not just the latest event).
3. Global search (`Ctrl/Cmd+K`): type a vehicle, booking or issue reference; use arrow
keys + Enter to navigate; show the no-results state for a nonsense query.
4. Bookings: filter by status, page through results (max 25/page), confirm page 2
differs from page 1.
5. Open an active booking → capture a return with a below-canonical odometer reading →
the review step shows the server's authoritative evaluation (odometer regression
flagged, resulting status and reason) → confirm → result screen distinguishes local
commit success from queued-not-yet-confirmed n8n delivery, links to the created
data-quality issue.
6. Data quality: run a manual scan; open the newly flagged (or an existing) issue for
each rule type and show its dedicated bounded resolution flow (not raw JSON).
7. Audit trail: filter by the correlation ID from the return above; show the
human-readable before/after change summary, expand the raw-JSON `<details>`.
8. Switch role to **Rental Employee**: show Data Quality/Integrations/Audit are absent
from the nav, and that direct URL navigation to any of them shows the restricted
message rather than partial data or a crash.
9. Switch back to Operations Manager, trigger **Reset demo data** with confirmation,
land back at login, log in again to confirm deterministic data was restored.
@@ -1,207 +0,0 @@
# Live n8n + RAGcore integration — final evidence
No credential values, tokens, or secrets appear anywhere in this document. Where a
credential or trace ID is referenced, only its name or an opaque reference identifier is
given, never its value.
## Commit
Built on branch `feat/live-n8n-ragcore-integration`, HEAD at commit
`aaa16305354d34f9c1f4d57253d33d9062c38faa` ("docs: record WF2 retry fix and WF4's
n8n-session-expiry blocker"). Run `git log --oneline feat/live-n8n-ragcore-integration`
for the full history of this effort.
## Scope
The brief required treating n8n (`https://n8n.itworx.tech`) as a full third integration
layer alongside RAGcore and MCP Hub, with Fleet Ops keeping exclusive ownership of
business rules, authorization, transactions, audit, and idempotency. Four canonical n8n
workflows were required. The repository (`n8n/workflows/*.json` + `MANIFEST.md` +
`n8n/workflows/check_drift.py`) is the source of truth for cleaned workflow definitions;
the Fleet Ops integration status page (`/automation`) shows real per-workflow operational
evidence, not a config boolean.
## Result summary
| # | Workflow | Status | Live evidence this round |
|---|---|---|---|
| 1 | Fleet Ops — Vehicle Return Orchestration | **Live, hardened** | Timeout+bounded-retry gap found and fixed |
| 2 | Fleet Ops — Scheduled Data Quality Scan | **Live, hardened** | Same gap found and fixed |
| 3 | Fleet Ops — RAGcore Procedure Sync | **Blocked** | Not built — RAGcore rejects credential issuance (see below) |
| 4 | Fleet Ops — Workflow Error Handler | **Live, validated** | Mock + genuine induced-failure test; own hardening incomplete (see below) |
Full per-workflow detail (purpose, trigger, event contract, required credentials, live
workflow ID, checksum) is in `n8n/workflows/MANIFEST.md`, which is the authoritative,
continuously-updated source — this document is a point-in-time summary of that state
plus the reasoning behind what's not done.
## Workflow 1 — Vehicle Return Orchestration
Live workflow ID `mobilityops-return-processing`. Validated in an earlier round of this
effort: webhook trigger requires Header Auth (`Fleet Ops Webhook Trigger Token`),
validates `event_type == vehicle.returned.v1`, derives a follow-up category, calls Fleet
Ops's `/return-callback` endpoint with an `Idempotency-Key` header via a named
`Fleet Ops Service Token` credential (not a literal secret), and responds with a
controlled JSON result. Idempotent on both sides (`event_id` flows through as the
dedup key; the backend independently checks for a prior audit event before recording
again).
**This round's finding**: the `Record follow-up` HTTP node had no explicit timeout and
"Retry On Fail" disabled — a real gap against the requirement that external dependencies
have timeouts and bounded retries. Fixed live: Retry On Fail (3 tries, 1000ms wait) + a
15000ms timeout, published. Safe to retry because the callback is idempotent. Repo
definition and manifest checksum synced (commit `0562893`).
Attached to workflow 4 as its Error Workflow.
## Workflow 2 — Scheduled Data Quality Scan
Live workflow ID `mobilityops-scheduled-quality-scan`. Validated earlier: hourly
Schedule Trigger + a Manual Trigger for on-demand testing, both feeding a single HTTP
call to Fleet Ops's `/scheduled-scan` endpoint (Header Auth via the same `Fleet Ops
Service Token` credential, 15000ms timeout already configured), which runs the
domain-level `run_scan()` function — documented and tested as idempotent by
construction (only ever creates an issue for a condition that doesn't already have one
open), so overlapping or retried triggers do no duplicate domain work.
**This round's finding**: the same Retry On Fail gap as workflow 1 (timeout was already
set, retries were not). Fixed live the same way (3 tries, 1000ms wait), published. Repo
definition and manifest checksum synced (commit `167bf49`).
Attached to workflow 4 as its Error Workflow.
## Workflow 3 — RAGcore Procedure Sync — blocked
**Not built.** This workflow needs an application credential (scope `sources:sync`) for
the `fleet-ops` application in RAGcore. Two independent issuance attempts, in two
separate rounds of this effort, both failed with an opaque server-side rejection:
1. **Raw API**: `POST /v1/applications/{id}/credentials``400`, "authoritative
service-account state rejected issuance".
2. **RAGcore admin UI**, this round, after the project owner explicitly authorized
Claude to self-issue the credential: the "Issue credential" form for the `fleet-ops`
application, submitted as the Platform Admin role (the highest role visible in the
RAGcore admin), with name `n8n-ragcore-procedure-sync` and scope `sources:sync` only.
Result: "Something went wrong. The credential could not be issued with those
values.", trace reference `1955c6a8968c4941a22a1faef39e17a7`.
The `fleet-ops` application itself shows as ordinary/`Active` in the RAGcore admin, with
no visible lock flag, and RAGcore's own OpenAPI spec documents no validation rule that
would explain either rejection (no `422`, no field-level errors). Two independent paths
— a raw API call and the admin UI as the top admin role — hitting the same failure
signature is conclusive evidence this is a RAGcore-side policy or bug, not a Fleet Ops
request-shape or permission problem. It is not fixable from the Fleet Ops side or
through further UI automation. Resolving it requires whoever operates the RAGcore
instance to look up the trace ID above (and the earlier raw-API rejection) in RAGcore's
own logs.
The real RAGcore contract this workflow will be built against — once a working
credential exists — was independently inspected via RAGcore's live OpenAPI spec and is
recorded in `docs/live-ai-integration/n8n-current-state.md` and
`contracts/ragcore-contract-assumptions.md`: control-plane endpoints require an
`Idempotency-Key` header; ingestion is `POST /v1/uploads`; retrieval is `POST
/v1/search` / `/v1/context` / `/v1/answers` (the latter requiring `requested_space_ids`,
an array of knowledge-space UUIDs); health is `/health/live` and `/health/ready` (not
`/health`); the scope enum is `search, context, answer, documents:read, citations:read,
feedback:write, sources:sync`.
**`RAGcoreKnowledgeProvider` adapter** (`backend/app/services/knowledge/ragcore.py`)
still targets the earlier speculative contract (`/health`, `POST /api/v1/ask`, Bearer
token) rather than the real one above. This was deliberately **not** rewritten this
round: rewriting it blind, without a credential to validate against, risks introducing
a silent behavioral bug in exactly the code path responsible for the project's "AI must
never invent an answer when RAGcore is unavailable or returns insufficient evidence"
guarantee — for example a wrong `evidence_state` mapping that looks fine in code review
but misclassifies "unavailable" as "insufficient" (or vice versa) against the real
response shape. The adapter's current behavior is honest and safe (it degrades cleanly
to `unavailable` on any request or parsing failure, and `ragcore_api_token` is unset by
default so the app correctly runs on the local demo knowledge provider today). The
rewrite stays queued behind the same credential blocker as workflow 3.
## Workflow 4 — Workflow Error Handler
Live workflow ID `Xppn2rAEqUuyiCJF`. Built and live-validated in an earlier round:
Error Trigger → a Code node that derives a bounded, secret-free failure report (error
category classified from the message text, truncated summary, no stack trace, no
headers or tokens) → an HTTP call to Fleet Ops's `/workflow-error` endpoint (Header Auth
via the same `Fleet Ops Service Token` credential), which registers the failure as an
audit event idempotently keyed on `execution_id`.
Set as the Error Workflow on both workflow 1 and workflow 2. Confirmed workflow 4 has no
Error Workflow of its own (prevents a recursive loop).
**Live validation performed**: a pinned mock Error Trigger payload produced a real `200
{"status":"registered", ...}` from the live Fleet Ops server; re-running the identical
payload produced `"status":"already_registered"`, confirming idempotency. A genuine
induced failure (temporarily pointing workflow 2's HTTP node at a nonexistent path, then
reverting) confirmed workflow 2 itself fails correctly against a broken endpoint and
recovers cleanly once reverted.
**Known limitation**: n8n's Error Workflow trigger does not fire for manual editor
"Execute workflow" test runs — checked via workflow 4's own Executions list after the
induced workflow-2 failure, and confirmed no new execution appeared. n8n only invokes a
workflow's assigned Error Workflow for unattended/production trigger executions, not
manual test runs from the editor. The mock-data path exercises the same nodes, logic,
and real Fleet Ops endpoint, but a fully automatic (schedule- or webhook-triggered)
failure cascading into workflow 4 was not observed live in either round.
**Open follow-up (minor, non-blocking)**: continuing this round's acceptance pass to
workflow 4 found the same timeout/retry gap as workflows 1 and 2 on its own outbound
HTTP call. A fix was started (15000ms timeout added, Retry On Fail toggled on) but n8n's
autosave began failing with "Unauthorized" mid-edit; a fresh browser tab confirmed the
n8n session had expired (redirected to `/signin`). Nothing was saved — workflow 4's live
definition is unchanged from before this round, so there is no partial or broken state.
This is lower-stakes than workflows 1/2 (workflow 4 is the error notifier itself, not a
primary business flow, and a failed error-report is already visible in n8n's own
execution history via `On Error: Stop Workflow`) but should be finished once the n8n
browser session is re-authenticated.
## Repository source of truth
`n8n/workflows/` holds cleaned, credential-value-free JSON definitions for all built
workflows, `n8n/workflows/MANIFEST.md` documents purpose/trigger/contract/credentials/
live-ID/checksum for all four canonical workflows (including workflow 3's blocked
status), and `n8n/workflows/check_drift.py` is a read-only script that compares the
repo definitions against the live instance via n8n's Public API and reports drift —
safe to run in CI as a non-blocking check. No literal export/download mechanism was
found working in this n8n version, so each definition was reconstructed from direct,
verified UI inspection rather than a native export; this limitation is noted in the
manifest itself.
## Integration status page
`/automation` (Operations Manager only) surfaces real per-workflow evidence derived
purely from Fleet Ops's own audit/outbox tables — no new dependency on n8n's API was
added to the backend. Each of the four canonical workflows shows a status (not built /
no evidence yet / operational) and a last-evidence timestamp; the scheduled-scan
evidence specifically filters to `actor_type == "service"` so a manually-triggered scan
in the UI doesn't count as n8n evidence. An error-handler summary line reports total
registered automation failures and the most recent one.
Verified live in the browser (Dutch locale) both locally and on the deployed
production server (`http://192.168.10.150:1236/automation`): correctly showed "3 van 4
canonieke n8n-workflows hebben actuele evidentie van werking" with real timestamps for
the return/scan/error-handler workflows, "Nog Niet Gebouwd" for the RAGcore sync, and
the real error-handler registration from this effort's live testing.
## Deployments performed (all explicitly user-approved)
1. Backend `/workflow-error` endpoint (commit `bbdb4a9`) — deployed and verified
(`/health` OK, new endpoint returns `422` not `404` on an empty POST body).
2. Integration status page, backend + frontend (commit `4049c0c`) — deployed and
verified (`/health` OK, page renders real evidence in the browser).
The three n8n-side node edits this round (WF1 timeout/retry, WF2 timeout/retry, WF4's
incomplete attempt) are live edits to the n8n instance itself and do not require a
Fleet Ops redeploy.
## What's left
1. **RAGcore credential issuance** — blocked on RAGcore's own server-side rejection
(trace `1955c6a8968c4941a22a1faef39e17a7` and the earlier raw-API `400`). Needs
RAGcore's operator to investigate. Unblocks workflow 3 and the
`RAGcoreKnowledgeProvider` real-contract rewrite.
2. **Workflow 4's own timeout/bounded-retry hardening** — needs the n8n browser session
re-authenticated to finish; a small, well-understood, non-blocking edit.
3. **Fleet Ops logo/favicon** — explicitly deferred by the project owner as a separate,
unrelated follow-up task, not part of this integration effort.
+3
View File
@@ -24,6 +24,9 @@ FROM runtime-base AS test
COPY backend/requirements.lock ./requirements.lock COPY backend/requirements.lock ./requirements.lock
RUN pip install --no-cache-dir -r requirements.lock RUN pip install --no-cache-dir -r requirements.lock
COPY backend/tests ./tests COPY backend/tests ./tests
COPY contracts ./contracts
COPY scripts/check-contracts.py ./scripts/check-contracts.py
COPY n8n/workflows ./n8n/workflows
USER app USER app
FROM runtime-base AS runtime FROM runtime-base AS runtime
+51 -3
View File
@@ -30,6 +30,7 @@ from app.schemas import (
ReturnPreviewResult, ReturnPreviewResult,
) )
from app.services.audit import record_audit_event from app.services.audit import record_audit_event
from app.services.data_quality import open_odometer_regression_issue
from app.services.returns import preview_vehicle_return, register_vehicle_return from app.services.returns import preview_vehicle_return, register_vehicle_return
router = APIRouter(prefix="/api/v1/bookings", tags=["bookings"]) router = APIRouter(prefix="/api/v1/bookings", tags=["bookings"])
@@ -146,8 +147,14 @@ def create_booking(
) -> BookingOut: ) -> BookingOut:
if body.ends_at <= body.starts_at: if body.ends_at <= body.starts_at:
raise HTTPException(status_code=422, detail="Booking end must be after its start") raise HTTPException(status_code=422, detail="Booking end must be after its start")
customer = db.scalar(select(Customer).where(Customer.public_ref == body.customer_ref)) customer = db.scalar(
if customer is None or customer.merged_into_customer_id is not None: select(Customer).where(Customer.public_ref == body.customer_ref).with_for_update()
)
if (
customer is None
or customer.merged_into_customer_id is not None
or customer.anonymized_at is not None
):
raise HTTPException(status_code=422, detail="Customer is unavailable for booking") raise HTTPException(status_code=422, detail="Customer is unavailable for booking")
# Serialise booking creation per vehicle. The overlap check must run after # Serialise booking creation per vehicle. The overlap check must run after
# acquiring this lock, otherwise two concurrent requests can both pass it. # acquiring this lock, otherwise two concurrent requests can both pass it.
@@ -179,7 +186,9 @@ def create_booking(
status="reserved", status="reserved",
start_odometer_km=None, start_odometer_km=None,
end_odometer_km=None, end_odometer_km=None,
requirements_complete=body.requirements_complete, # Requirements are intentionally confirmed in a separate, audited action.
# Never allow booking creation to bypass that operational checkpoint.
requirements_complete=False,
) )
db.add(booking) db.add(booking)
db.flush() db.flush()
@@ -225,6 +234,15 @@ def checkout_booking(
if active_conflict is not None: if active_conflict is not None:
raise HTTPException(status_code=409, detail="Vehicle already has an active booking") raise HTTPException(status_code=409, detail="Vehicle already has an active booking")
correlation_id = uuid.uuid4()
before_booking = {
"status": booking.status,
"start_odometer_km": booking.start_odometer_km,
}
before_vehicle = {
"operational_status": vehicle.operational_status,
"odometer_km": vehicle.odometer_km,
}
attention_reasons: list[str] = [] attention_reasons: list[str] = []
if body.start_odometer_km < vehicle.odometer_km: if body.start_odometer_km < vehicle.odometer_km:
attention_reasons.append("odometer_regression") attention_reasons.append("odometer_regression")
@@ -250,6 +268,18 @@ def checkout_booking(
completed_by=user.display_name, completed_by=user.display_name,
) )
db.add(inspection) db.add(inspection)
if "odometer_regression" in attention_reasons:
open_odometer_regression_issue(
db,
vehicle=vehicle,
reading_ref=inspection.public_ref,
reading_km=body.start_odometer_km,
canonical_km=vehicle.odometer_km,
source_type="checkout",
related_refs=[booking.public_ref, inspection.public_ref],
actor_label=user.display_name,
correlation_id=correlation_id,
)
if attention_reasons: if attention_reasons:
booking.status = "blocked" booking.status = "blocked"
vehicle.operational_status = ( vehicle.operational_status = (
@@ -269,13 +299,31 @@ def checkout_booking(
action="booking_checkout_recorded", action="booking_checkout_recorded",
entity_type="booking", entity_type="booking",
entity_id=booking.id, entity_id=booking.id,
correlation_id=correlation_id,
before=before_booking,
after={ after={
"inspection_ref": inspection.public_ref, "inspection_ref": inspection.public_ref,
"booking_status": booking.status, "booking_status": booking.status,
"start_odometer_km": booking.start_odometer_km,
"vehicle_status": vehicle.operational_status, "vehicle_status": vehicle.operational_status,
"attention_reasons": attention_reasons, "attention_reasons": attention_reasons,
}, },
) )
record_audit_event(
db,
actor_type="user",
actor_label=user.display_name,
action="vehicle_status_changed",
entity_type="vehicle",
entity_id=vehicle.id,
correlation_id=correlation_id,
before=before_vehicle,
after={
"operational_status": vehicle.operational_status,
"odometer_km": vehicle.odometer_km,
},
metadata={"booking_ref": booking.public_ref, "inspection_ref": inspection.public_ref},
)
db.commit() db.commit()
return CheckoutBookingResult( return CheckoutBookingResult(
booking_ref=booking.public_ref, booking_ref=booking.public_ref,
+1
View File
@@ -21,6 +21,7 @@ def search_customers(
select(Customer) select(Customer)
.where( .where(
Customer.merged_into_customer_id.is_(None), Customer.merged_into_customer_id.is_(None),
Customer.anonymized_at.is_(None),
or_( or_(
Customer.public_ref.ilike(term), Customer.public_ref.ilike(term),
Customer.first_name.ilike(term), Customer.first_name.ilike(term),
+14
View File
@@ -11,6 +11,7 @@ from app.models.booking import Booking
from app.models.customer import Customer from app.models.customer import Customer
from app.models.data_quality import DataQualityIssue from app.models.data_quality import DataQualityIssue
from app.models.inspection import Inspection from app.models.inspection import Inspection
from app.models.maintenance import MaintenanceRecord
from app.models.user import User from app.models.user import User
from app.models.vehicle import Vehicle from app.models.vehicle import Vehicle
from app.schemas import ( from app.schemas import (
@@ -224,6 +225,8 @@ _PREFIX_TO_TYPE = {
"MO-": "vehicle", "MO-": "vehicle",
"BK-": "booking", "BK-": "booking",
"INSP-": "inspection", "INSP-": "inspection",
"MAINT-": "maintenance",
"MNT-": "maintenance",
} }
@@ -292,6 +295,17 @@ def _snapshot(entity_type: str, ref: str, db: Session) -> dict | None:
"completed_at": inspection.completed_at.isoformat(), "completed_at": inspection.completed_at.isoformat(),
"booking_ref": booking.public_ref if booking else None, "booking_ref": booking.public_ref if booking else None,
} }
if entity_type == "maintenance":
record = db.scalar(select(MaintenanceRecord).where(MaintenanceRecord.public_ref == ref))
if record is None:
return None
return {
"entity_type": "maintenance",
"public_ref": record.public_ref,
"odometer_km": record.odometer_km,
"occurred_at": record.occurred_at.isoformat(),
"category": record.category,
}
return None return None
+33 -9
View File
@@ -3,6 +3,7 @@ from __future__ import annotations
import threading import threading
import time import time
import uuid import uuid
from datetime import UTC, datetime
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
from sqlalchemy import func, select from sqlalchemy import func, select
@@ -10,7 +11,9 @@ from sqlalchemy.orm import Session
from app.api.deps import get_current_user, get_db, require_operations_manager from app.api.deps import get_current_user, get_db, require_operations_manager
from app.core.config import get_settings from app.core.config import get_settings
from app.core.db import begin_exclusive_demo_reset, end_exclusive_demo_reset, engine
from app.core.security import SessionPayload, create_session_token, read_session_token from app.core.security import SessionPayload, create_session_token, read_session_token
from app.models.audit import AuditEvent
from app.models.user import User from app.models.user import User
from app.schemas import CurrentUser, DemoLoginRequest, DemoManifestOut from app.schemas import CurrentUser, DemoLoginRequest, DemoManifestOut
from app.seed_loader import reset_and_seed from app.seed_loader import reset_and_seed
@@ -21,7 +24,6 @@ from app.services.sessions import revoke_session
router = APIRouter(prefix="/api/v1/demo", tags=["demo"]) router = APIRouter(prefix="/api/v1/demo", tags=["demo"])
settings = get_settings() settings = get_settings()
_reset_guard = threading.Lock() _reset_guard = threading.Lock()
_last_reset_monotonic = 0.0
_RESET_ADVISORY_LOCK_ID = 706_533_149 _RESET_ADVISORY_LOCK_ID = 706_533_149
@@ -110,7 +112,6 @@ def demo_reset(
db: Session = Depends(get_db), db: Session = Depends(get_db),
user: CurrentUser = Depends(require_operations_manager), user: CurrentUser = Depends(require_operations_manager),
) -> dict: ) -> dict:
global _last_reset_monotonic
if not settings.mobilityops_demo_mode: if not settings.mobilityops_demo_mode:
# Outside demo mode the reset endpoint must not exist at all: it wipes # Outside demo mode the reset endpoint must not exist at all: it wipes
# operational data and replaces it with synthetic records. # operational data and replaces it with synthetic records.
@@ -122,19 +123,35 @@ def demo_reset(
) )
if not _reset_guard.acquire(blocking=False): if not _reset_guard.acquire(blocking=False):
raise HTTPException(status_code=409, detail="A demo reset is already running.") raise HTTPException(status_code=409, detail="A demo reset is already running.")
replica_lock_connection = None
try: try:
elapsed = time.monotonic() - _last_reset_monotonic # A session-level lock on its own connection rejects another replica immediately;
if _last_reset_monotonic and elapsed < settings.demo_reset_cooldown_seconds: # the main DB session can then safely end its auth read transaction and wait on
# the normal shared/exclusive data barrier without releasing this replica guard.
replica_lock_connection = engine.connect()
locked = replica_lock_connection.scalar(
select(func.pg_try_advisory_lock(_RESET_ADVISORY_LOCK_ID))
)
if not locked:
raise HTTPException(status_code=409, detail="A demo reset is already running.")
begin_exclusive_demo_reset(db)
# The audit timestamp is shared by every replica. A process-local monotonic
# timestamp cannot protect a multi-replica deployment.
last_reset_at = db.scalar(
select(AuditEvent.occurred_at)
.where(AuditEvent.action == "demo_reset")
.order_by(AuditEvent.occurred_at.desc())
.limit(1)
)
elapsed = (datetime.now(UTC) - last_reset_at).total_seconds() if last_reset_at else None
if elapsed is not None and elapsed < settings.demo_reset_cooldown_seconds:
retry_after = max(1, int(settings.demo_reset_cooldown_seconds - elapsed + 0.999)) retry_after = max(1, int(settings.demo_reset_cooldown_seconds - elapsed + 0.999))
raise HTTPException( raise HTTPException(
status_code=429, status_code=429,
detail=f"Demo reset is cooling down. Retry in {retry_after} seconds.", detail=f"Demo reset is cooling down. Retry in {retry_after} seconds.",
headers={"Retry-After": str(retry_after)}, headers={"Retry-After": str(retry_after)},
) )
locked = db.scalar(select(func.pg_try_advisory_xact_lock(_RESET_ADVISORY_LOCK_ID))) result = reset_and_seed(db, preserve_integration_telemetry=True, commit=False)
if not locked:
raise HTTPException(status_code=409, detail="A demo reset is already running.")
result = reset_and_seed(db, preserve_integration_telemetry=True)
integrity = scenario_integrity_report(db) integrity = scenario_integrity_report(db)
record_audit_event( record_audit_event(
db, db,
@@ -149,8 +166,15 @@ def demo_reset(
}, },
) )
db.commit() db.commit()
_last_reset_monotonic = time.monotonic() end_exclusive_demo_reset(db)
finally: finally:
if replica_lock_connection is not None:
try:
replica_lock_connection.scalar(
select(func.pg_advisory_unlock(_RESET_ADVISORY_LOCK_ID))
)
finally:
replica_lock_connection.close()
_reset_guard.release() _reset_guard.release()
response.delete_cookie(settings.session_cookie_name) response.delete_cookie(settings.session_cookie_name)
return { return {
+35 -10
View File
@@ -1,12 +1,13 @@
from __future__ import annotations from __future__ import annotations
import hashlib
import hmac import hmac
import uuid import uuid
from datetime import UTC, datetime from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
from fastapi import APIRouter, Depends, Header from fastapi import APIRouter, Depends, Header
from sqlalchemy import select from sqlalchemy import func, select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.api.deps import get_db from app.api.deps import get_db
@@ -51,6 +52,13 @@ def _require_service_token(service_token: str) -> None:
raise AppError("UNAUTHORIZED_SERVICE", "Invalid service token.", status_code=401) raise AppError("UNAUTHORIZED_SERVICE", "Invalid service token.", status_code=401)
def _lock_idempotency_key(db: Session, namespace: str, key: str) -> None:
"""Serialize callback check+insert by a stable, transaction-scoped key."""
digest = hashlib.sha256(f"{namespace}:{key}".encode()).digest()
lock_id = int.from_bytes(digest[:8], byteorder="big", signed=True)
db.scalar(select(func.pg_advisory_xact_lock(lock_id)))
@router.post("/heartbeat", response_model=N8nHeartbeatResult) @router.post("/heartbeat", response_model=N8nHeartbeatResult)
def workflow_heartbeat( def workflow_heartbeat(
body: N8nHeartbeatIn, body: N8nHeartbeatIn,
@@ -61,6 +69,7 @@ def workflow_heartbeat(
_require_service_token(service_token) _require_service_token(service_token)
if body.workflow_name not in _CANONICAL_WORKFLOW_NAMES: if body.workflow_name not in _CANONICAL_WORKFLOW_NAMES:
raise AppError("UNKNOWN_WORKFLOW", "Unknown Fleet Ops workflow.", status_code=422) raise AppError("UNKNOWN_WORKFLOW", "Unknown Fleet Ops workflow.", status_code=422)
_lock_idempotency_key(db, "n8n_workflow_heartbeat", f"{body.execution_id}:{body.status}")
already_recorded = ( already_recorded = (
db.scalar( db.scalar(
select(AuditEvent.id).where( select(AuditEvent.id).where(
@@ -109,9 +118,29 @@ def return_callback(
"INVALID_IDEMPOTENCY_KEY", "Idempotency-Key must be the event's UUID.", status_code=422 "INVALID_IDEMPOTENCY_KEY", "Idempotency-Key must be the event's UUID.", status_code=422
) from exc ) from exc
event = db.scalar(select(OutboxEvent).where(OutboxEvent.event_id == event_id)) event = db.scalar(select(OutboxEvent).where(OutboxEvent.event_id == event_id).with_for_update())
if event is None: if event is None:
raise AppError("EVENT_NOT_FOUND", "No outbox event matches this event ID.", status_code=404) raise AppError("EVENT_NOT_FOUND", "No outbox event matches this event ID.", status_code=404)
if body.event_id != event_id:
raise AppError(
"CALLBACK_EVENT_MISMATCH",
"Callback event_id does not match Idempotency-Key.",
status_code=409,
)
try:
expected_correlation_id = uuid.UUID(str(event.payload_json["correlation_id"]))
except (KeyError, TypeError, ValueError) as exc:
raise AppError(
"INVALID_EVENT_CORRELATION",
"The stored outbox event has no valid correlation ID.",
status_code=409,
) from exc
if body.correlation_id != expected_correlation_id:
raise AppError(
"CALLBACK_CORRELATION_MISMATCH",
"Callback correlation_id does not match the outbox event.",
status_code=409,
)
# Idempotent by event ID: n8n or our own dispatcher may redeliver the same event # Idempotent by event ID: n8n or our own dispatcher may redeliver the same event
# (e.g. a lost response after a timeout), so this callback must not double-record. # (e.g. a lost response after a timeout), so this callback must not double-record.
@@ -131,7 +160,7 @@ def return_callback(
actor_label="n8n", actor_label="n8n",
action="n8n_return_followup_recorded", action="n8n_return_followup_recorded",
entity_type="booking", entity_type="booking",
correlation_id=body.correlation_id, correlation_id=expected_correlation_id,
after={"follow_up": body.follow_up, "summary": body.summary}, after={"follow_up": body.follow_up, "summary": body.summary},
metadata={"event_id": str(event_id)}, metadata={"event_id": str(event_id)},
) )
@@ -170,6 +199,7 @@ def workflow_error(
other Fleet Ops n8n workflow. Idempotent on execution_id: n8n may redeliver the same other Fleet Ops n8n workflow. Idempotent on execution_id: n8n may redeliver the same
error report (e.g. after a timed-out response), so this must not double-record.""" error report (e.g. after a timed-out response), so this must not double-record."""
_require_service_token(service_token) _require_service_token(service_token)
_lock_idempotency_key(db, "n8n_workflow_failure", body.execution_id)
already_recorded = ( already_recorded = (
db.scalar( db.scalar(
@@ -181,19 +211,13 @@ def workflow_error(
is not None is not None
) )
if not already_recorded: if not already_recorded:
correlation_id: uuid.UUID | None = None
if body.correlation_id:
try:
correlation_id = uuid.UUID(body.correlation_id)
except ValueError:
correlation_id = None
record_audit_event( record_audit_event(
db, db,
actor_type="service", actor_type="service",
actor_label="n8n error handler", actor_label="n8n error handler",
action="n8n_workflow_failure_registered", action="n8n_workflow_failure_registered",
entity_type="automation", entity_type="automation",
correlation_id=correlation_id, correlation_id=body.correlation_id,
after={ after={
"workflow_id": body.workflow_id, "workflow_id": body.workflow_id,
"workflow_name": body.workflow_name, "workflow_name": body.workflow_name,
@@ -248,6 +272,7 @@ def procedures_sync_result(
RAGcore Procedure Sync" workflow once it finishes uploading procedures to RAGcore. RAGcore Procedure Sync" workflow once it finishes uploading procedures to RAGcore.
Idempotent on execution_id, matching the workflow-error and return-callback pattern.""" Idempotent on execution_id, matching the workflow-error and return-callback pattern."""
_require_service_token(service_token) _require_service_token(service_token)
_lock_idempotency_key(db, "n8n_procedure_sync", body.execution_id)
already_recorded = ( already_recorded = (
db.scalar( db.scalar(
+3 -1
View File
@@ -53,7 +53,9 @@ def ask_question(
client_ip = ( client_ip = (
forwarded.split(",")[-1].strip() forwarded.split(",")[-1].strip()
if forwarded if forwarded
else request.client.host if request.client else "unknown" else request.client.host
if request.client
else "unknown"
) )
token = request.cookies.get(settings.session_cookie_name, "") token = request.cookies.get(settings.session_cookie_name, "")
session_key = hashlib.sha256(token.encode("utf-8")).hexdigest() session_key = hashlib.sha256(token.encode("utf-8")).hexdigest()
+39 -4
View File
@@ -27,6 +27,7 @@ from app.schemas import (
VehiclePageOut, VehiclePageOut,
) )
from app.services.audit import record_audit_event from app.services.audit import record_audit_event
from app.services.data_quality import open_odometer_regression_issue
router = APIRouter(prefix="/api/v1/vehicles", tags=["vehicles"]) router = APIRouter(prefix="/api/v1/vehicles", tags=["vehicles"])
@@ -143,7 +144,7 @@ def list_vehicles(
def get_vehicle( def get_vehicle(
public_ref: str, public_ref: str,
db: Session = Depends(get_db), db: Session = Depends(get_db),
_user: CurrentUser = Depends(get_current_user), user: CurrentUser = Depends(get_current_user),
) -> VehicleDetailOut: ) -> VehicleDetailOut:
vehicle = db.scalar(select(Vehicle).where(Vehicle.public_ref == public_ref)) vehicle = db.scalar(select(Vehicle).where(Vehicle.public_ref == public_ref))
if vehicle is None: if vehicle is None:
@@ -163,11 +164,21 @@ def get_vehicle(
.where(MaintenanceRecord.vehicle_id == vehicle.id) .where(MaintenanceRecord.vehicle_id == vehicle.id)
.order_by(MaintenanceRecord.occurred_at.desc()) .order_by(MaintenanceRecord.occurred_at.desc())
).all() ).all()
issues = db.scalars( # Detailed data-quality evidence is an operations-manager surface. Employees still
# get the operational vehicle record they need, but never receive hidden evidence in
# the payload merely because the frontend omits the Quality tab.
issues = (
db.scalars(
select(DataQualityIssue) select(DataQualityIssue)
.where(DataQualityIssue.entity_type == "vehicle", DataQualityIssue.entity_id == vehicle.id) .where(
DataQualityIssue.entity_type == "vehicle",
DataQualityIssue.entity_id == vehicle.id,
)
.order_by(DataQualityIssue.detected_at.desc()) .order_by(DataQualityIssue.detected_at.desc())
).all() ).all()
if user.role == "operations_manager"
else []
)
booking_by_id = {b.id: b.public_ref for b in bookings} booking_by_id = {b.id: b.public_ref for b in bookings}
next_booking = next( next_booking = next(
@@ -276,6 +287,12 @@ def create_maintenance_record(
vehicle = db.scalar(select(Vehicle).where(Vehicle.public_ref == public_ref).with_for_update()) vehicle = db.scalar(select(Vehicle).where(Vehicle.public_ref == public_ref).with_for_update())
if vehicle is None: if vehicle is None:
raise HTTPException(status_code=404, detail="Vehicle not found") raise HTTPException(status_code=404, detail="Vehicle not found")
correlation_id = uuid.uuid4()
before_vehicle = {
"operational_status": vehicle.operational_status,
"odometer_km": vehicle.odometer_km,
"next_service_km": vehicle.next_service_km,
}
record = MaintenanceRecord( record = MaintenanceRecord(
public_ref=f"MAINT-{uuid.uuid4().hex[:8].upper()}", public_ref=f"MAINT-{uuid.uuid4().hex[:8].upper()}",
vehicle_id=vehicle.id, vehicle_id=vehicle.id,
@@ -285,6 +302,17 @@ def create_maintenance_record(
summary=body.summary.strip(), summary=body.summary.strip(),
) )
db.add(record) db.add(record)
open_odometer_regression_issue(
db,
vehicle=vehicle,
reading_ref=record.public_ref,
reading_km=body.odometer_km,
canonical_km=vehicle.odometer_km,
source_type="maintenance",
related_refs=[record.public_ref],
actor_label=user.display_name,
correlation_id=correlation_id,
)
vehicle.odometer_km = max(vehicle.odometer_km, body.odometer_km) vehicle.odometer_km = max(vehicle.odometer_km, body.odometer_km)
if body.next_service_km is not None: if body.next_service_km is not None:
if body.next_service_km < vehicle.odometer_km: if body.next_service_km < vehicle.odometer_km:
@@ -301,7 +329,14 @@ def create_maintenance_record(
action="maintenance_record_created", action="maintenance_record_created",
entity_type="vehicle", entity_type="vehicle",
entity_id=vehicle.id, entity_id=vehicle.id,
after={"maintenance_ref": record.public_ref, "status": vehicle.operational_status}, correlation_id=correlation_id,
before=before_vehicle,
after={
"maintenance_ref": record.public_ref,
"operational_status": vehicle.operational_status,
"odometer_km": vehicle.odometer_km,
"next_service_km": vehicle.next_service_km,
},
) )
db.commit() db.commit()
return MaintenanceOut( return MaintenanceOut(
+5 -1
View File
@@ -33,7 +33,11 @@ class Settings(BaseSettings):
n8n_callback_token: str = "replace-me-n8n-callback-token" n8n_callback_token: str = "replace-me-n8n-callback-token"
n8n_dispatch_enabled: bool = True n8n_dispatch_enabled: bool = True
n8n_dispatch_interval_seconds: float = 3.0 n8n_dispatch_interval_seconds: float = 3.0
n8n_http_timeout_seconds: float = 5.0 # The synchronous n8n workflow performs two bounded, retried callbacks before it
# acknowledges an event. Keep this above that complete workflow budget, while the
# delivery lease remains the wider crash-recovery boundary (enforced by the contract
# check in scripts/check-contracts.py).
n8n_http_timeout_seconds: float = 15.0
n8n_max_attempts: int = 5 n8n_max_attempts: int = 5
n8n_delivery_lease_seconds: float = 120.0 n8n_delivery_lease_seconds: float = 120.0
app_secret: str = "replace-in-production" app_secret: str = "replace-in-production"
+32 -1
View File
@@ -1,6 +1,6 @@
from collections.abc import Generator from collections.abc import Generator
from sqlalchemy import create_engine from sqlalchemy import create_engine, event, func, select
from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker
from app.core.config import get_settings from app.core.config import get_settings
@@ -10,6 +10,37 @@ settings = get_settings()
engine = create_engine(settings.database_url, pool_pre_ping=True, future=True) engine = create_engine(settings.database_url, pool_pre_ping=True, future=True)
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False, future=True) SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False, future=True)
# Every SQLAlchemy transaction participates in a shared database-wide barrier. Normal
# reads/writes coexist; the short demo reset takes the exclusive form so it can never
# interleave deletes/inserts with an API request, scanner, callback, or dispatcher cycle.
DEMO_DATA_BARRIER_LOCK_ID = 5_344_725_149_212_793_901
_EXCLUSIVE_RESET_INFO_KEY = "mobilityops_demo_reset_exclusive"
@event.listens_for(Session, "after_begin")
def _acquire_demo_data_barrier(session: Session, _transaction, connection) -> None:
if connection.dialect.name != "postgresql":
return
lock = (
func.pg_advisory_xact_lock(DEMO_DATA_BARRIER_LOCK_ID)
if session.info.get(_EXCLUSIVE_RESET_INFO_KEY)
else func.pg_advisory_xact_lock_shared(DEMO_DATA_BARRIER_LOCK_ID)
)
connection.execute(select(lock))
def begin_exclusive_demo_reset(db: Session) -> None:
"""Make the session's next transaction the exclusive side of the reset barrier."""
if db.in_transaction():
# Auth normally already read the user under a shared barrier. End that read-only
# transaction before requesting exclusive; in-place lock upgrades can deadlock.
db.rollback()
db.info[_EXCLUSIVE_RESET_INFO_KEY] = True
def end_exclusive_demo_reset(db: Session) -> None:
db.info.pop(_EXCLUSIVE_RESET_INFO_KEY, None)
class Base(DeclarativeBase): class Base(DeclarativeBase):
pass pass
+12 -8
View File
@@ -4,9 +4,13 @@ import uuid
from datetime import datetime from datetime import datetime
from typing import Annotated, Any, Literal from typing import Annotated, Any, Literal
from pydantic import BaseModel, ConfigDict, Field from pydantic import BaseModel, ConfigDict, Field, StringConstraints
Role = Literal["operations_manager", "rental_employee"] Role = Literal["operations_manager", "rental_employee"]
NonBlankOperationalReason = Annotated[
str,
StringConstraints(strip_whitespace=True, min_length=3, max_length=500),
]
class DemoLoginRequest(BaseModel): class DemoLoginRequest(BaseModel):
@@ -93,17 +97,16 @@ class CreateBookingRequest(BaseModel):
vehicle_ref: str = Field(min_length=3, max_length=20) vehicle_ref: str = Field(min_length=3, max_length=20)
starts_at: datetime starts_at: datetime
ends_at: datetime ends_at: datetime
requirements_complete: bool = False
class CompleteBookingRequirementsRequest(BaseModel): class CompleteBookingRequirementsRequest(BaseModel):
confirmation: str = Field(min_length=3, max_length=500) confirmation: NonBlankOperationalReason
class RescheduleBookingRequest(BaseModel): class RescheduleBookingRequest(BaseModel):
starts_at: datetime starts_at: datetime
ends_at: datetime ends_at: datetime
reason: str = Field(min_length=3, max_length=500) reason: NonBlankOperationalReason
class CustomerOptionOut(BaseModel): class CustomerOptionOut(BaseModel):
@@ -122,7 +125,7 @@ class AvailableVehicleOut(BaseModel):
class CancelBookingRequest(BaseModel): class CancelBookingRequest(BaseModel):
reason: str = Field(min_length=3, max_length=500) reason: NonBlankOperationalReason
class CheckoutBookingRequest(BaseModel): class CheckoutBookingRequest(BaseModel):
@@ -225,7 +228,7 @@ class CreateMaintenanceRequest(BaseModel):
class ReleaseVehicleRequest(BaseModel): class ReleaseVehicleRequest(BaseModel):
reason: str = Field(min_length=3, max_length=500) reason: NonBlankOperationalReason
class DataQualityIssueOut(BaseModel): class DataQualityIssueOut(BaseModel):
@@ -295,7 +298,7 @@ class WorkflowErrorReportIn(BaseModel):
] ]
error_summary: str = Field(max_length=500) error_summary: str = Field(max_length=500)
trigger_context: str | None = Field(default=None, max_length=200) trigger_context: str | None = Field(default=None, max_length=200)
correlation_id: str | None = None correlation_id: uuid.UUID | None = None
attempt: int = Field(default=1, ge=1, le=1000) attempt: int = Field(default=1, ge=1, le=1000)
retry_action: str | None = Field(default=None, max_length=200) retry_action: str | None = Field(default=None, max_length=200)
@@ -433,7 +436,8 @@ class ReturnCallbackIn(BaseModel):
model_config = ConfigDict(extra="ignore") model_config = ConfigDict(extra="ignore")
correlation_id: uuid.UUID | None = None event_id: uuid.UUID
correlation_id: uuid.UUID
follow_up: str | None = Field(default=None, max_length=200) follow_up: str | None = Field(default=None, max_length=200)
summary: str | None = Field(default=None, max_length=2000) summary: str | None = Field(default=None, max_length=2000)
+51 -11
View File
@@ -11,6 +11,7 @@ from sqlalchemy import delete, insert, update
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.core.config import get_settings from app.core.config import get_settings
from app.core.db import begin_exclusive_demo_reset, end_exclusive_demo_reset
from app.models.audit import AuditEvent from app.models.audit import AuditEvent
from app.models.booking import Booking from app.models.booking import Booking
from app.models.customer import Customer from app.models.customer import Customer
@@ -201,7 +202,11 @@ def load_seed(db: Session) -> SeedResult:
counts["bookings"] = len(booking_rows) counts["bookings"] = len(booking_rows)
inspection_rows = [] inspection_rows = []
for row in _read_csv("inspections.csv"): inspection_source_rows = _read_csv("inspections.csv")
return_inspection_row_by_booking_ref = {
row["booking_ref"]: row for row in inspection_source_rows if row["type"] == "return"
}
for row in inspection_source_rows:
inspection_rows.append( inspection_rows.append(
{ {
"id": uuid.uuid4(), "id": uuid.uuid4(),
@@ -259,16 +264,18 @@ def load_seed(db: Session) -> SeedResult:
} }
def _odometer_regression_signal(later_ref: str, earlier_ref: str) -> list[dict]: def _odometer_regression_signal(later_ref: str, earlier_ref: str) -> list[dict]:
later = booking_row_by_ref[later_ref] later = return_inspection_row_by_booking_ref[later_ref]
earlier = booking_row_by_ref[earlier_ref] earlier = return_inspection_row_by_booking_ref[earlier_ref]
return [ return [
{ {
"code": "odometer.regression", "code": "odometer.regression",
"source_type": "return",
"params": { "params": {
"later_ref": later_ref, "later_ref": later["public_ref"],
"later_km": later["end_odometer_km"], "later_km": int(later["odometer_km"]),
"earlier_ref": earlier_ref, "earlier_ref": earlier["public_ref"],
"earlier_km": earlier["end_odometer_km"], "earlier_km": int(earlier["odometer_km"]),
"booking_ref": later_ref,
}, },
} }
] ]
@@ -354,6 +361,27 @@ def load_seed(db: Session) -> SeedResult:
entity_type, entity_id = resolve_entity(row["entity_ref"]) entity_type, entity_id = resolve_entity(row["entity_ref"])
related_ref = row.get("related_ref") or "" related_ref = row.get("related_ref") or ""
related_refs = related_ref.split("|") if related_ref else [] related_refs = related_ref.split("|") if related_ref else []
signals = _seed_signals(row["public_ref"], row["entity_ref"], related_refs)
evidence_extra: dict[str, object] = {}
if row["rule_type"] == "odometer_regression" and signals:
params = signals[0].get("params", {})
later_ref = params.get("later_ref")
earlier_ref = params.get("earlier_ref")
booking_ref = params.get("booking_ref")
if (
isinstance(earlier_ref, str)
and isinstance(later_ref, str)
and isinstance(booking_ref, str)
):
# The authored CSV prose predates structured references. Preserve the
# two real source bookings so the detail page can show evidence, while
# only the later (regressing) reading is eligible for correction.
related_refs = [earlier_ref, booking_ref, later_ref]
evidence_extra = {
"source_type": signals[0].get("source_type", "return"),
"source_types": [signals[0].get("source_type", "return")],
"correctable_booking_refs": [booking_ref],
}
severity_due_delta = { severity_due_delta = {
"high": timedelta(hours=4), "high": timedelta(hours=4),
"medium": timedelta(days=1), "medium": timedelta(days=1),
@@ -372,7 +400,8 @@ def load_seed(db: Session) -> SeedResult:
"summary": row["evidence"], "summary": row["evidence"],
"entity_ref": row["entity_ref"], "entity_ref": row["entity_ref"],
"related_refs": related_refs, "related_refs": related_refs,
"signals": _seed_signals(row["public_ref"], row["entity_ref"], related_refs), "signals": signals,
**evidence_extra,
}, },
"proposed_action_json": {}, "proposed_action_json": {},
"detected_at": now, "detected_at": now,
@@ -449,12 +478,23 @@ def load_seed(db: Session) -> SeedResult:
return SeedResult(counts=counts, anchor_date=today, seeded_at=seeded_at) return SeedResult(counts=counts, anchor_date=today, seeded_at=seeded_at)
def reset_and_seed(db: Session, *, preserve_integration_telemetry: bool = False) -> SeedResult: def reset_and_seed(
db: Session,
*,
preserve_integration_telemetry: bool = False,
commit: bool = True,
) -> SeedResult:
from app.services.data_quality import run_scan from app.services.data_quality import run_scan
if not db.info.get("mobilityops_demo_reset_exclusive"):
begin_exclusive_demo_reset(db)
clear_all(db, preserve_integration_telemetry=preserve_integration_telemetry) clear_all(db, preserve_integration_telemetry=preserve_integration_telemetry)
result = load_seed(db) result = load_seed(db)
db.commit() scan = run_scan(db, commit=False)
scan = run_scan(db)
result.counts["data_quality_issues"] += sum(scan.created.values()) result.counts["data_quality_issues"] += sum(scan.created.values())
if commit:
db.commit()
end_exclusive_demo_reset(db)
else:
db.flush()
return result return result
+126 -276
View File
@@ -2,8 +2,7 @@ from __future__ import annotations
import uuid import uuid
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime
from difflib import SequenceMatcher
from sqlalchemy import func, select, update from sqlalchemy import func, select, update
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -12,9 +11,22 @@ from app.core.errors import AppError
from app.models.booking import Booking from app.models.booking import Booking
from app.models.customer import Customer from app.models.customer import Customer
from app.models.data_quality import DataQualityIssue from app.models.data_quality import DataQualityIssue
from app.models.inspection import Inspection
from app.models.maintenance import MaintenanceRecord
from app.models.vehicle import Vehicle from app.models.vehicle import Vehicle
from app.schemas import CurrentUser, ResolveOdometerRegressionRequest from app.schemas import CurrentUser
from app.services.audit import record_audit_event from app.services.audit import record_audit_event
from app.services.data_quality_common import has_open_issue as _has_open_issue
from app.services.data_quality_common import issue_due_at
from app.services.data_quality_common import load_open_issue as _load_open_issue
from app.services.data_quality_common import new_scan_ref as _new_scan_ref
from app.services.data_quality_duplicate_scan import scan_duplicate_customers
from app.services.data_quality_odometer import (
open_odometer_regression_issue as open_odometer_regression_issue,
)
from app.services.data_quality_odometer import (
resolve_odometer_regression as resolve_odometer_regression,
)
from app.services.vehicle_status import ( from app.services.vehicle_status import (
RECOMMENDATION_CODE_NO_CONFLICT, RECOMMENDATION_CODE_NO_CONFLICT,
VehicleStatusRecommendation, VehicleStatusRecommendation,
@@ -25,19 +37,9 @@ from app.services.vehicle_status import (
REQUIRED_CUSTOMER_FIELDS = ("first_name", "last_name") REQUIRED_CUSTOMER_FIELDS = ("first_name", "last_name")
REQUIRED_VEHICLE_FIELDS = ("registration_number", "make", "model", "location") REQUIRED_VEHICLE_FIELDS = ("registration_number", "make", "model", "location")
DUPLICATE_THRESHOLD = 70
DATA_QUALITY_SCAN_LOCK_ID = 6_138_493_717_091_029_491 DATA_QUALITY_SCAN_LOCK_ID = 6_138_493_717_091_029_491
def issue_due_at(detected_at: datetime, severity: str) -> datetime:
"""Return the local operational SLA deadline for a newly detected issue."""
return detected_at + {
"high": timedelta(hours=4),
"medium": timedelta(days=1),
"low": timedelta(days=3),
}.get(severity, timedelta(days=1))
@dataclass @dataclass
class ScanResult: class ScanResult:
created: dict[str, int] = field(default_factory=dict) created: dict[str, int] = field(default_factory=dict)
@@ -46,29 +48,6 @@ class ScanResult:
self.created[rule_type] = self.created.get(rule_type, 0) + 1 self.created[rule_type] = self.created.get(rule_type, 0) + 1
def _normalize(value: str | None) -> str:
return (value or "").strip().lower()
def _has_open_issue(db: Session, rule_type: str, entity_type: str, entity_id: uuid.UUID) -> bool:
return (
db.scalar(
select(DataQualityIssue.id).where(
DataQualityIssue.rule_type == rule_type,
DataQualityIssue.entity_type == entity_type,
DataQualityIssue.entity_id == entity_id,
DataQualityIssue.status == "open",
)
)
is not None
)
def _new_scan_ref(prefix: str) -> str:
"""Generate a stable human-readable prefix with a concurrent-safe suffix."""
return f"{prefix}-{uuid.uuid4().hex[:10].upper()}"
def _open_issue( def _open_issue(
db: Session, db: Session,
scan: ScanResult, scan: ScanResult,
@@ -81,6 +60,7 @@ def _open_issue(
entity_ref: str, entity_ref: str,
related_refs: list[str], related_refs: list[str],
signals: list[dict] | None = None, signals: list[dict] | None = None,
evidence_extra: dict | None = None,
) -> None: ) -> None:
if _has_open_issue(db, rule_type, entity_type, entity_id): if _has_open_issue(db, rule_type, entity_type, entity_id):
return return
@@ -108,6 +88,7 @@ def _open_issue(
"related_refs": related_refs, "related_refs": related_refs,
"signals": signals or [], "signals": signals or [],
} }
evidence.update(evidence_extra or {})
if previous is not None: if previous is not None:
evidence["reopened_from"] = previous.public_ref evidence["reopened_from"] = previous.public_ref
evidence["previous_decision"] = previous.status evidence["previous_decision"] = previous.status
@@ -129,71 +110,6 @@ def _open_issue(
scan.bump(rule_type) scan.bump(rule_type)
def _scan_duplicate_customers(db: Session, scan: ScanResult) -> None:
customers = list(
db.scalars(
select(Customer).where(
Customer.merged_into_customer_id.is_(None),
Customer.anonymized_at.is_(None),
)
).all()
)
customers.sort(key=lambda c: c.public_ref)
# The threshold cannot be reached without an exact email (60 points) or phone
# (50 points). Block on those normalized identifiers first, so similarity scoring
# scales with plausible candidates instead of comparing every customer pair.
candidate_pairs: set[tuple[int, int]] = set()
for attribute in ("email", "phone"):
blocks: dict[str, list[int]] = {}
for index, customer in enumerate(customers):
key = _normalize(getattr(customer, attribute))
if key:
blocks.setdefault(key, []).append(index)
for indices in blocks.values():
for offset, left in enumerate(indices):
candidate_pairs.update((left, right) for right in indices[offset + 1 :])
for left, right in sorted(candidate_pairs):
a = customers[left]
b = customers[right]
score = 0
signals: list[dict] = []
summary_parts: list[str] = []
if _normalize(a.email) and _normalize(a.email) == _normalize(b.email):
score += 60
signals.append({"code": "duplicate.exact_email"})
summary_parts.append("exact email")
if _normalize(a.phone) and _normalize(a.phone) == _normalize(b.phone):
score += 50
signals.append({"code": "duplicate.exact_phone"})
summary_parts.append("exact phone")
if _normalize(a.postal_code) and _normalize(a.postal_code) == _normalize(b.postal_code):
score += 10
signals.append({"code": "duplicate.same_postal_code"})
summary_parts.append("exact postal code")
name_a = f"{_normalize(a.first_name)} {_normalize(a.last_name)}"
name_b = f"{_normalize(b.first_name)} {_normalize(b.last_name)}"
ratio = SequenceMatcher(None, name_a, name_b).ratio()
if ratio >= 0.5:
score += round(ratio * 30)
signals.append({"code": "duplicate.similar_name", "params": {"score": round(ratio, 2)}})
summary_parts.append("similar name")
if score >= DUPLICATE_THRESHOLD:
_open_issue(
db,
scan,
rule_type="possible_duplicate_customer",
entity_type="customer",
entity_id=a.id,
severity="high",
summary="; ".join(summary_parts) + f" (score {score})",
entity_ref=a.public_ref,
related_refs=[b.public_ref],
signals=signals,
)
def _scan_missing_required_fields(db: Session, scan: ScanResult) -> None: def _scan_missing_required_fields(db: Session, scan: ScanResult) -> None:
# Anonymised customers have had their contact data removed on purpose; flagging # Anonymised customers have had their contact data removed on purpose; flagging
# them as "missing required field" would only be resolvable by re-entering PII. # them as "missing required field" would only be resolvable by re-entering PII.
@@ -305,65 +221,123 @@ def _scan_vehicle_status_conflicts(db: Session, scan: ScanResult) -> None:
) )
def _scan_odometer_regressions(db: Session, scan: ScanResult) -> None: def _scan_odometer_regressions(
# The seed dataset's vehicle.odometer_km is generated independently of booking db: Session,
# history, so comparing every historical booking against it produces near-universal scan: ScanResult,
# false positives. Instead check the booking sequence's own internal consistency: *,
# each vehicle's completed bookings should show a non-decreasing odometer reading. actor_label: str | None,
vehicles = {v.id: v for v in db.scalars(select(Vehicle)).all()} actor_type: str,
bookings_by_vehicle: dict[uuid.UUID, list[Booking]] = {} ) -> None:
for booking in db.scalars( # Compare the chronological history with itself rather than every historical reading
select(Booking).where(Booking.status == "returned", Booking.end_odometer_km.is_not(None)) # to today's canonical value. That detects imported checkout/return/maintenance
).all(): # regressions without flagging every legitimate older reading.
bookings_by_vehicle.setdefault(booking.vehicle_id, []).append(booking) # Runtime checkout/return/maintenance mutations take the vehicle lock before they
# inspect or append DQ-03 evidence. Taking the same lock here makes scan-vs-command
for vehicle_id, bookings in bookings_by_vehicle.items(): # check/merge atomic and prevents a partial-unique race for the open issue.
bookings.sort(key=lambda b: b.ends_at) vehicles = {
for earlier, later in zip(bookings, bookings[1:], strict=False): v.id: v for v in db.scalars(select(Vehicle).order_by(Vehicle.id).with_for_update()).all()
# The query above filters end_odometer_km IS NOT NULL, so both are ints here.
assert earlier.end_odometer_km is not None
assert later.end_odometer_km is not None
if later.end_odometer_km < earlier.end_odometer_km:
vehicle = vehicles[vehicle_id]
_open_issue(
db,
scan,
rule_type="odometer_regression",
entity_type="vehicle",
entity_id=vehicle_id,
severity="medium",
summary=(
f"Booking {later.public_ref} recorded {later.end_odometer_km} km, "
f"below the {earlier.end_odometer_km} km recorded by earlier "
f"booking {earlier.public_ref}."
),
entity_ref=vehicle.public_ref,
related_refs=[earlier.public_ref, later.public_ref],
signals=[
{
"code": "odometer.regression",
"params": {
"later_ref": later.public_ref,
"later_km": later.end_odometer_km,
"earlier_ref": earlier.public_ref,
"earlier_km": earlier.end_odometer_km,
},
} }
], readings_by_vehicle: dict[uuid.UUID, list[tuple[datetime, str, int, str, str | None]]] = {}
inspections = db.scalars(select(Inspection)).all()
return_inspection_booking_ids = {
inspection.booking_id for inspection in inspections if inspection.type == "return"
}
returned_bookings = db.scalars(
select(Booking).where(Booking.status == "returned", Booking.end_odometer_km.is_not(None))
).all()
booking_ref_by_id = {booking.id: booking.public_ref for booking in returned_bookings}
for booking in returned_bookings:
# A real return inspection owns the actual reading timestamp. Using the booking's
# planned ends_at as a duplicate second reading can make an early return appear to
# go backwards after a newer real inspection. Keep booking data only as the legacy
# import fallback when no return inspection exists.
if booking.id in return_inspection_booking_ids:
continue
assert booking.end_odometer_km is not None
readings_by_vehicle.setdefault(booking.vehicle_id, []).append(
(
booking.ends_at,
booking.public_ref,
booking.end_odometer_km,
"booking",
booking.public_ref,
) )
break )
for inspection in inspections:
readings_by_vehicle.setdefault(inspection.vehicle_id, []).append(
(
inspection.completed_at,
inspection.public_ref,
inspection.odometer_km,
inspection.type,
(
booking_ref_by_id.get(inspection.booking_id)
if inspection.type == "return"
else None
),
)
)
for record in db.scalars(select(MaintenanceRecord)).all():
readings_by_vehicle.setdefault(record.vehicle_id, []).append(
(record.occurred_at, record.public_ref, record.odometer_km, "maintenance", None)
)
for vehicle_id, readings in readings_by_vehicle.items():
readings.sort(key=lambda reading: (reading[0], reading[1]))
highest = readings[0] if readings else None
for later in readings[1:]:
assert highest is not None
if later[2] < highest[2]:
vehicle = vehicles[vehicle_id]
had_open_issue = _has_open_issue(
db,
"odometer_regression",
"vehicle",
vehicle_id,
)
issue = open_odometer_regression_issue(
db,
vehicle=vehicle,
reading_ref=later[1],
reading_km=later[2],
canonical_km=highest[2],
canonical_ref=highest[1],
source_type=later[3],
related_refs=list(
dict.fromkeys(
[highest[1], later[1], *([later[4]] if later[4] is not None else [])]
)
),
correctable_booking_refs=[later[4]] if later[4] is not None else [],
public_ref=_new_scan_ref("DQ-SCAN"),
actor_label=actor_label,
actor_type=actor_type,
)
if issue is not None and not had_open_issue:
scan.bump("odometer_regression")
if later[2] > highest[2]:
highest = later
def run_scan( def run_scan(
db: Session, *, actor_label: str | None = None, actor_type: str = "user" db: Session,
*,
actor_label: str | None = None,
actor_type: str = "user",
commit: bool = True,
) -> ScanResult: ) -> ScanResult:
# The check-then-insert work below spans several rules. Serialise whole scans at the # The check-then-insert work below spans several rules. Serialise whole scans at the
# database boundary so API and n8n triggers cannot both observe an empty condition. # database boundary so API and n8n triggers cannot both observe an empty condition.
db.scalar(select(func.pg_advisory_xact_lock(DATA_QUALITY_SCAN_LOCK_ID))) db.scalar(select(func.pg_advisory_xact_lock(DATA_QUALITY_SCAN_LOCK_ID)))
scan = ScanResult() scan = ScanResult()
_scan_duplicate_customers(db, scan) scan_duplicate_customers(db, scan, _open_issue)
_scan_missing_required_fields(db, scan) _scan_missing_required_fields(db, scan)
_scan_odometer_regressions(db, scan) _scan_odometer_regressions(
db,
scan,
actor_label=actor_label,
actor_type=actor_type,
)
_scan_booking_overlaps(db, scan) _scan_booking_overlaps(db, scan)
_scan_vehicle_status_conflicts(db, scan) _scan_vehicle_status_conflicts(db, scan)
if actor_label is not None: if actor_label is not None:
@@ -375,28 +349,13 @@ def run_scan(
entity_type="system", entity_type="system",
metadata={"created": scan.created}, metadata={"created": scan.created},
) )
if commit:
db.commit() db.commit()
else:
db.flush()
return scan return scan
def _load_open_issue(
db: Session, public_ref: str, *, lock: bool = True
) -> DataQualityIssue:
statement = select(DataQualityIssue).where(DataQualityIssue.public_ref == public_ref)
if lock:
statement = statement.with_for_update()
issue = db.scalar(statement)
if issue is None:
raise AppError("ISSUE_NOT_FOUND", "Data quality issue not found.", status_code=404)
if issue.status != "open":
raise AppError(
"ISSUE_NOT_OPEN",
f"Issue is '{issue.status}', not 'open'.",
status_code=409,
)
return issue
def defer_issue(db: Session, public_ref: str, actor: CurrentUser) -> DataQualityIssue: def defer_issue(db: Session, public_ref: str, actor: CurrentUser) -> DataQualityIssue:
issue = _load_open_issue(db, public_ref) issue = _load_open_issue(db, public_ref)
issue.status = "deferred" issue.status = "deferred"
@@ -526,115 +485,6 @@ def provide_missing_fields(
return issue return issue
def resolve_odometer_regression(
db: Session, public_ref: str, body: ResolveOdometerRegressionRequest, actor: CurrentUser
) -> DataQualityIssue:
issue = _load_open_issue(db, public_ref)
if issue.rule_type != "odometer_regression":
raise AppError(
"NOT_AN_ODOMETER_ISSUE",
"This issue is not an odometer_regression issue.",
status_code=409,
)
# Lock order is booking -> vehicle everywhere (checkout, return, reschedule); taking
# the vehicle lock first here would be a deadlock waiting to happen under concurrency.
booking: Booking | None = None
if body.decision != "retain_canonical":
related_refs = issue.evidence_json.get("related_refs", [])
if body.booking_ref not in related_refs:
raise AppError(
"INVALID_BOOKING_REFERENCE",
"booking_ref must be one of this issue's related bookings.",
status_code=422,
)
if body.corrected_odometer_km is None:
raise AppError(
"CORRECTED_VALUE_REQUIRED",
"corrected_odometer_km is required when correcting a reading.",
status_code=422,
)
booking = db.scalar(
select(Booking).where(Booking.public_ref == body.booking_ref).with_for_update()
)
if booking is None:
raise AppError(
"BOOKING_NOT_FOUND", "The booking to correct was not found.", status_code=404
)
vehicle = db.scalar(select(Vehicle).where(Vehicle.id == issue.entity_id).with_for_update())
if vehicle is None:
raise AppError(
"VEHICLE_NOT_FOUND", "The vehicle for this issue was not found.", status_code=404
)
correlation_id = uuid.uuid4()
if body.decision == "retain_canonical":
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="data_quality_odometer_retained",
entity_type="vehicle",
entity_id=vehicle.id,
correlation_id=correlation_id,
metadata={"issue_ref": issue.public_ref, "canonical_odometer_km": vehicle.odometer_km},
)
else:
assert booking is not None and body.corrected_odometer_km is not None
# Never silently lower the canonical odometer: a correction must be at or above
# the current canonical value, otherwise it would just create a new regression.
if body.corrected_odometer_km < vehicle.odometer_km:
raise AppError(
"CORRECTION_BELOW_CANONICAL",
(
f"Corrected value {body.corrected_odometer_km} km is still below the "
f"canonical {vehicle.odometer_km} km; it would not resolve the regression."
),
status_code=422,
)
before = {
"booking_end_odometer_km": booking.end_odometer_km,
"vehicle_odometer_km": vehicle.odometer_km,
}
booking.end_odometer_km = body.corrected_odometer_km
vehicle.odometer_km = body.corrected_odometer_km
vehicle.version += 1
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="data_quality_odometer_corrected",
entity_type="vehicle",
entity_id=vehicle.id,
correlation_id=correlation_id,
before=before,
after={
"booking_end_odometer_km": booking.end_odometer_km,
"vehicle_odometer_km": vehicle.odometer_km,
},
metadata={"issue_ref": issue.public_ref, "booking_ref": booking.public_ref},
)
issue.status = "resolved"
issue.resolved_at = datetime.now(UTC)
issue.resolved_by = actor.display_name
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="data_quality_issue_resolved",
entity_type="data_quality_issue",
entity_id=issue.id,
correlation_id=correlation_id,
before={"status": "open"},
after={"status": "resolved"},
metadata={"decision": body.decision, "note": body.note},
)
db.commit()
return issue
def resolve_booking_overlap( def resolve_booking_overlap(
db: Session, public_ref: str, booking_ref: str, note: str | None, actor: CurrentUser db: Session, public_ref: str, booking_ref: str, note: str | None, actor: CurrentUser
) -> DataQualityIssue: ) -> DataQualityIssue:
@@ -0,0 +1,59 @@
from __future__ import annotations
import uuid
from datetime import datetime, timedelta
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.core.errors import AppError
from app.models.data_quality import DataQualityIssue
def issue_due_at(detected_at: datetime, severity: str) -> datetime:
"""Return the local operational SLA deadline for a newly detected issue."""
return detected_at + {
"high": timedelta(hours=4),
"medium": timedelta(days=1),
"low": timedelta(days=3),
}.get(severity, timedelta(days=1))
def has_open_issue(
db: Session,
rule_type: str,
entity_type: str,
entity_id: uuid.UUID,
) -> bool:
return (
db.scalar(
select(DataQualityIssue.id).where(
DataQualityIssue.rule_type == rule_type,
DataQualityIssue.entity_type == entity_type,
DataQualityIssue.entity_id == entity_id,
DataQualityIssue.status == "open",
)
)
is not None
)
def new_scan_ref(prefix: str) -> str:
"""Generate a stable human-readable prefix with a concurrent-safe suffix."""
return f"{prefix}-{uuid.uuid4().hex[:10].upper()}"
def load_open_issue(db: Session, public_ref: str, *, lock: bool = True) -> DataQualityIssue:
statement = select(DataQualityIssue).where(DataQualityIssue.public_ref == public_ref)
if lock:
statement = statement.with_for_update().execution_options(populate_existing=True)
issue = db.scalar(statement)
if issue is None:
raise AppError("ISSUE_NOT_FOUND", "Data quality issue not found.", status_code=404)
if issue.status != "open":
raise AppError(
"ISSUE_NOT_OPEN",
f"Issue is '{issue.status}', not 'open'.",
status_code=409,
)
return issue
@@ -0,0 +1,108 @@
from __future__ import annotations
import uuid
from difflib import SequenceMatcher
from typing import Protocol, TypeVar
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.models.customer import Customer
DUPLICATE_THRESHOLD = 70
class ScanAccumulator(Protocol):
def bump(self, rule_type: str) -> None: ...
ScanTypeContra = TypeVar("ScanTypeContra", bound=ScanAccumulator, contravariant=True)
class OpenIssue(Protocol[ScanTypeContra]):
def __call__(
self,
db: Session,
scan: ScanTypeContra,
*,
rule_type: str,
entity_type: str,
entity_id: uuid.UUID,
severity: str,
summary: str,
entity_ref: str,
related_refs: list[str],
signals: list[dict] | None = None,
) -> None: ...
def _normalize(value: str | None) -> str:
return (value or "").strip().lower()
def scan_duplicate_customers[ScanType: ScanAccumulator](
db: Session, scan: ScanType, open_issue: OpenIssue[ScanType]
) -> None:
customers = list(
db.scalars(
select(Customer).where(
Customer.merged_into_customer_id.is_(None),
Customer.anonymized_at.is_(None),
)
).all()
)
customers.sort(key=lambda customer: customer.public_ref)
# The threshold cannot be reached without an exact email (60 points) or phone
# (50 points). Block on normalized identifiers so this remains linear for the
# overwhelmingly common case and only scores plausible pairs.
candidate_pairs: set[tuple[int, int]] = set()
for attribute in ("email", "phone"):
blocks: dict[str, list[int]] = {}
for index, customer in enumerate(customers):
key = _normalize(getattr(customer, attribute))
if key:
blocks.setdefault(key, []).append(index)
for indices in blocks.values():
for offset, left in enumerate(indices):
candidate_pairs.update((left, right) for right in indices[offset + 1 :])
for left, right in sorted(candidate_pairs):
a = customers[left]
b = customers[right]
score = 0
signals: list[dict] = []
summary_parts: list[str] = []
if _normalize(a.email) and _normalize(a.email) == _normalize(b.email):
score += 60
signals.append({"code": "duplicate.exact_email"})
summary_parts.append("exact email")
if _normalize(a.phone) and _normalize(a.phone) == _normalize(b.phone):
score += 50
signals.append({"code": "duplicate.exact_phone"})
summary_parts.append("exact phone")
if _normalize(a.postal_code) and _normalize(a.postal_code) == _normalize(b.postal_code):
score += 10
signals.append({"code": "duplicate.same_postal_code"})
summary_parts.append("exact postal code")
name_a = f"{_normalize(a.first_name)} {_normalize(a.last_name)}"
name_b = f"{_normalize(b.first_name)} {_normalize(b.last_name)}"
ratio = SequenceMatcher(None, name_a, name_b).ratio()
if ratio >= 0.5:
score += round(ratio * 30)
signals.append({"code": "duplicate.similar_name", "params": {"score": round(ratio, 2)}})
summary_parts.append("similar name")
if score >= DUPLICATE_THRESHOLD:
open_issue(
db,
scan,
rule_type="possible_duplicate_customer",
entity_type="customer",
entity_id=a.id,
severity="high",
summary="; ".join(summary_parts) + f" (score {score})",
entity_ref=a.public_ref,
related_refs=[b.public_ref],
signals=signals,
)
@@ -0,0 +1,507 @@
from __future__ import annotations
import uuid
from datetime import UTC, datetime
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.core.errors import AppError
from app.models.booking import Booking
from app.models.data_quality import DataQualityIssue
from app.models.inspection import Inspection
from app.models.vehicle import Vehicle
from app.schemas import CurrentUser, ResolveOdometerRegressionRequest
from app.services.audit import record_audit_event
from app.services.data_quality_common import issue_due_at, load_open_issue, new_scan_ref
def _odometer_fingerprint(
*,
source_type: str,
later_ref: str,
later_km: int,
) -> dict[str, str | int]:
"""Stable identity for one reviewed regression, independent of issue IDs."""
return {
"source_type": source_type,
"later_ref": later_ref,
"later_km": later_km,
}
def _was_odometer_regression_retained(
db: Session,
*,
vehicle_id: uuid.UUID,
fingerprint: dict[str, str | int],
) -> bool:
reviewed = db.scalars(
select(DataQualityIssue).where(
DataQualityIssue.rule_type == "odometer_regression",
DataQualityIssue.entity_type == "vehicle",
DataQualityIssue.entity_id == vehicle_id,
DataQualityIssue.status == "resolved",
)
).all()
return any(
issue.evidence_json.get("resolution_decision") == "retain_canonical"
and fingerprint in issue.evidence_json.get("retained_odometer_fingerprints", [])
for issue in reviewed
)
def open_odometer_regression_issue(
db: Session,
*,
vehicle: Vehicle,
reading_ref: str,
reading_km: int,
canonical_km: int,
source_type: str,
related_refs: list[str],
correctable_booking_refs: list[str] | None = None,
canonical_ref: str | None = None,
detected_at: datetime | None = None,
public_ref: str | None = None,
actor_label: str | None = None,
actor_type: str = "user",
correlation_id: uuid.UUID | None = None,
) -> DataQualityIssue | None:
"""Open one explainable DQ-03 issue while the caller holds the vehicle lock."""
if reading_km >= canonical_km:
return None
earlier_ref = canonical_ref or vehicle.public_ref
fingerprint = _odometer_fingerprint(
source_type=source_type,
later_ref=reading_ref,
later_km=reading_km,
)
# A manager's explicit "retain canonical" decision acknowledges this exact source
# fact. Reopening it on every scheduled scan would create churn; only changed or new
# evidence (and therefore a different fingerprint) is actionable again.
if _was_odometer_regression_retained(
db,
vehicle_id=vehicle.id,
fingerprint=fingerprint,
):
return None
existing = db.scalar(
select(DataQualityIssue)
.where(
DataQualityIssue.rule_type == "odometer_regression",
DataQualityIssue.entity_type == "vehicle",
DataQualityIssue.entity_id == vehicle.id,
DataQualityIssue.status == "open",
)
.with_for_update()
.execution_options(populate_existing=True)
)
if existing is not None:
before_related_refs = list(existing.evidence_json.get("related_refs", []))
before_correctable_refs = list(existing.evidence_json.get("correctable_booking_refs", []))
new_signal = {
"code": "odometer.regression",
"source_type": source_type,
"params": {
"later_ref": reading_ref,
"later_km": reading_km,
"earlier_ref": earlier_ref,
"earlier_km": canonical_km,
},
}
existing_signals = list(existing.evidence_json.get("signals", []))
signal_was_new = not any(
isinstance(signal, dict)
and isinstance(signal.get("params"), dict)
and _odometer_fingerprint(
source_type=str(
signal.get("source_type", existing.evidence_json.get("source_type"))
),
later_ref=str(signal["params"].get("later_ref")),
later_km=signal["params"].get("later_km"),
)
== fingerprint
for signal in existing_signals
if isinstance(signal, dict)
and isinstance(signal.get("params"), dict)
and isinstance(signal["params"].get("later_km"), int)
)
if signal_was_new:
existing_signals.append(new_signal)
merged_related_refs = list(
dict.fromkeys([*before_related_refs, *(related_refs if signal_was_new else [])])
)
merged_correctable_refs = list(
dict.fromkeys([*before_correctable_refs, *(correctable_booking_refs or [])])
)
if (
not signal_was_new
and merged_related_refs == before_related_refs
and merged_correctable_refs == before_correctable_refs
):
return existing
source_types = list(existing.evidence_json.get("source_types", []))
previous_source_type = existing.evidence_json.get("source_type")
if (
isinstance(previous_source_type, str)
and previous_source_type != "multiple"
and previous_source_type not in source_types
):
source_types.append(previous_source_type)
if source_type not in source_types:
source_types.append(source_type)
existing.evidence_json = {
**existing.evidence_json,
"summary": (
f"{source_type.title()} {reading_ref} recorded {reading_km} km, below "
f"the canonical {canonical_km} km for {vehicle.public_ref}."
),
"related_refs": merged_related_refs,
"correctable_booking_refs": merged_correctable_refs,
"source_type": source_type if len(source_types) == 1 else "multiple",
"source_types": source_types,
"signals": existing_signals,
}
if actor_label is not None and (
merged_related_refs != before_related_refs
or merged_correctable_refs != before_correctable_refs
or signal_was_new
):
record_audit_event(
db,
actor_type=actor_type,
actor_label=actor_label,
action="data_quality_issue_evidence_updated",
entity_type="data_quality_issue",
entity_id=existing.id,
correlation_id=correlation_id,
before={
"related_refs": before_related_refs,
"correctable_booking_refs": before_correctable_refs,
},
after={
"related_refs": merged_related_refs,
"correctable_booking_refs": merged_correctable_refs,
},
metadata={"vehicle_ref": vehicle.public_ref, "reading_ref": reading_ref},
)
return existing
now = detected_at or datetime.now(UTC)
previous = db.scalar(
select(DataQualityIssue)
.where(
DataQualityIssue.rule_type == "odometer_regression",
DataQualityIssue.entity_type == "vehicle",
DataQualityIssue.entity_id == vehicle.id,
DataQualityIssue.status != "open",
)
.order_by(DataQualityIssue.detected_at.desc())
)
evidence = {
"summary": (
f"{source_type.title()} {reading_ref} recorded {reading_km} km, below "
f"the canonical {canonical_km} km for {vehicle.public_ref}."
),
"entity_ref": vehicle.public_ref,
"related_refs": related_refs,
"correctable_booking_refs": correctable_booking_refs or [],
"source_type": source_type,
"source_types": [source_type],
"signals": [
{
"code": "odometer.regression",
"source_type": source_type,
"params": {
"later_ref": reading_ref,
"later_km": reading_km,
"earlier_ref": earlier_ref,
"earlier_km": canonical_km,
},
}
],
}
if previous is not None:
evidence["reopened_from"] = previous.public_ref
evidence["previous_decision"] = previous.status
issue = DataQualityIssue(
public_ref=public_ref or new_scan_ref("DQ-ODO"),
rule_type="odometer_regression",
entity_type="vehicle",
entity_id=vehicle.id,
severity="medium",
status="open",
evidence_json=evidence,
proposed_action_json={},
detected_at=now,
due_at=issue_due_at(now, "medium"),
)
db.add(issue)
db.flush()
if actor_label is not None:
record_audit_event(
db,
actor_type=actor_type,
actor_label=actor_label,
action="data_quality_issue_created",
entity_type="data_quality_issue",
entity_id=issue.id,
correlation_id=correlation_id,
after={"status": "open", "rule_type": "odometer_regression"},
metadata={"vehicle_ref": vehicle.public_ref, "reading_ref": reading_ref},
)
return issue
def resolve_odometer_regression(
db: Session, public_ref: str, body: ResolveOdometerRegressionRequest, actor: CurrentUser
) -> DataQualityIssue:
# Read the routing data without a row lock first. The canonical mutation order is
# booking -> vehicle -> issue -> inspection everywhere, matching checkout/return.
# Locking the issue before the booking creates a resolver-vs-return deadlock.
issue_snapshot = load_open_issue(db, public_ref, lock=False)
if issue_snapshot.rule_type != "odometer_regression":
raise AppError(
"NOT_AN_ODOMETER_ISSUE",
"This issue is not an odometer_regression issue.",
status_code=409,
)
def correctable_refs(issue: DataQualityIssue) -> list[str]:
if "correctable_booking_refs" in issue.evidence_json:
refs = issue.evidence_json.get("correctable_booking_refs", [])
else:
# Backward compatibility for issues created before source-aware evidence.
refs = [
ref
for ref in issue.evidence_json.get("related_refs", [])
if isinstance(ref, str) and ref.startswith("BK-")
]
return [ref for ref in refs if isinstance(ref, str)]
booking: Booking | None = None
if body.decision != "retain_canonical":
if body.booking_ref not in correctable_refs(issue_snapshot):
raise AppError(
"INVALID_BOOKING_REFERENCE",
"booking_ref must be one of this issue's related bookings.",
status_code=422,
)
if body.corrected_odometer_km is None:
raise AppError(
"CORRECTED_VALUE_REQUIRED",
"corrected_odometer_km is required when correcting a reading.",
status_code=422,
)
booking = db.scalar(
select(Booking).where(Booking.public_ref == body.booking_ref).with_for_update()
)
if booking is None:
raise AppError(
"BOOKING_NOT_FOUND", "The booking to correct was not found.", status_code=404
)
vehicle = db.scalar(
select(Vehicle).where(Vehicle.id == issue_snapshot.entity_id).with_for_update()
)
if vehicle is None:
raise AppError(
"VEHICLE_NOT_FOUND", "The vehicle for this issue was not found.", status_code=404
)
# A return or scan may have appended evidence while we waited for the domain locks.
# Lock and refresh the issue only now, then revalidate every decision against that
# current evidence instead of resolving a stale snapshot.
issue = load_open_issue(db, public_ref)
if issue.rule_type != "odometer_regression" or issue.entity_id != vehicle.id:
raise AppError(
"ISSUE_CHANGED",
"The issue changed while the correction was being prepared. Review it again.",
status_code=409,
)
if booking is not None:
if booking.vehicle_id != vehicle.id or booking.public_ref not in correctable_refs(issue):
raise AppError(
"INVALID_BOOKING_REFERENCE",
"booking_ref must be one of this issue's related bookings.",
status_code=422,
)
correlation_id = uuid.uuid4()
if body.decision == "retain_canonical":
retained_fingerprints: list[dict[str, str | int]] = []
fallback_source_type = str(issue.evidence_json.get("source_type", "unknown"))
for signal in issue.evidence_json.get("signals", []):
if not isinstance(signal, dict) or signal.get("code") != "odometer.regression":
continue
params = signal.get("params")
if not isinstance(params, dict):
continue
source_type = signal.get("source_type", fallback_source_type)
later_ref = params.get("later_ref")
later_km = params.get("later_km")
earlier_ref = params.get("earlier_ref")
earlier_km = params.get("earlier_km")
if (
isinstance(source_type, str)
and isinstance(later_ref, str)
and isinstance(later_km, int)
and isinstance(earlier_ref, str)
and isinstance(earlier_km, int)
):
retained_fingerprints.append(
_odometer_fingerprint(
source_type=source_type,
later_ref=later_ref,
later_km=later_km,
)
)
issue.evidence_json = {
**issue.evidence_json,
"resolution_decision": "retain_canonical",
"retained_odometer_fingerprints": retained_fingerprints,
}
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="data_quality_odometer_retained",
entity_type="vehicle",
entity_id=vehicle.id,
correlation_id=correlation_id,
metadata={
"issue_ref": issue.public_ref,
"canonical_odometer_km": vehicle.odometer_km,
"retained_fingerprint_count": len(retained_fingerprints),
},
)
else:
assert booking is not None and body.corrected_odometer_km is not None
# Never silently lower the canonical odometer: a correction must be at or above
# the current canonical value, otherwise it would just create a new regression.
if body.corrected_odometer_km < vehicle.odometer_km:
raise AppError(
"CORRECTION_BELOW_CANONICAL",
(
f"Corrected value {body.corrected_odometer_km} km is still below the "
f"canonical {vehicle.odometer_km} km; it would not resolve the regression."
),
status_code=422,
)
# A returned booking has two persisted representations of the same reading.
# Correct every return inspection for the selected booking, including historical
# seed rows whose old issue evidence did not yet cite the inspection explicitly.
related_inspections = db.scalars(
select(Inspection)
.where(Inspection.booking_id == booking.id, Inspection.type == "return")
.with_for_update()
).all()
before = {
"booking_end_odometer_km": booking.end_odometer_km,
"vehicle_odometer_km": vehicle.odometer_km,
"inspection_odometer_km": {
inspection.public_ref: inspection.odometer_km for inspection in related_inspections
},
}
booking.end_odometer_km = body.corrected_odometer_km
for inspection in related_inspections:
inspection.odometer_km = body.corrected_odometer_km
vehicle.odometer_km = body.corrected_odometer_km
vehicle.version += 1
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="data_quality_odometer_corrected",
entity_type="vehicle",
entity_id=vehicle.id,
correlation_id=correlation_id,
before=before,
after={
"booking_end_odometer_km": booking.end_odometer_km,
"vehicle_odometer_km": vehicle.odometer_km,
"inspection_odometer_km": {
inspection.public_ref: inspection.odometer_km
for inspection in related_inspections
},
},
metadata={"issue_ref": issue.public_ref, "booking_ref": booking.public_ref},
)
has_remaining_evidence = False
if booking is not None:
target_refs = {booking.public_ref, *(item.public_ref for item in related_inspections)}
before_related = [
ref for ref in issue.evidence_json.get("related_refs", []) if isinstance(ref, str)
]
before_correctable = correctable_refs(issue)
before_signals = [
signal for signal in issue.evidence_json.get("signals", []) if isinstance(signal, dict)
]
remaining_signals = [
signal
for signal in before_signals
if not (
isinstance(signal.get("params"), dict)
and signal["params"].get("later_ref") in target_refs
)
]
remaining_related = [ref for ref in before_related if ref not in target_refs]
remaining_correctable = [ref for ref in before_correctable if ref != booking.public_ref]
has_remaining_evidence = bool(remaining_signals or remaining_correctable)
issue.evidence_json = {
**issue.evidence_json,
"summary": (
"Additional odometer regression evidence remains for review."
if has_remaining_evidence
else issue.evidence_json.get("summary", "Odometer reading corrected.")
),
"related_refs": remaining_related,
"correctable_booking_refs": remaining_correctable,
"signals": remaining_signals,
}
if has_remaining_evidence:
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="data_quality_issue_partially_resolved",
entity_type="data_quality_issue",
entity_id=issue.id,
correlation_id=correlation_id,
before={
"related_refs": before_related,
"correctable_booking_refs": before_correctable,
"signal_count": len(before_signals),
},
after={
"related_refs": remaining_related,
"correctable_booking_refs": remaining_correctable,
"signal_count": len(remaining_signals),
},
metadata={
"decision": body.decision,
"booking_ref": booking.public_ref,
"note": body.note,
},
)
if not has_remaining_evidence:
issue.status = "resolved"
issue.resolved_at = datetime.now(UTC)
issue.resolved_by = actor.display_name
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="data_quality_issue_resolved",
entity_type="data_quality_issue",
entity_id=issue.id,
correlation_id=correlation_id,
before={"status": "open"},
after={"status": "resolved"},
metadata={"decision": body.decision, "note": body.note},
)
db.commit()
return issue
+23 -10
View File
@@ -9,16 +9,28 @@ from app.core.config import get_settings
from app.models.audit import AuditEvent from app.models.audit import AuditEvent
from app.models.booking import Booking from app.models.booking import Booking
from app.models.data_quality import DataQualityIssue from app.models.data_quality import DataQualityIssue
from app.models.outbox import OutboxEvent from app.models.outbox import OutboxEvent, is_demo_scenario_failure
from app.schemas import DemoIntegrationSummaryOut, DemoManifestOut, DemoScenarioOut from app.schemas import DemoIntegrationSummaryOut, DemoManifestOut, DemoScenarioOut
from app.services.integration_status import derive_mcp_hub_status, derive_n8n_status from app.services.integration_status import derive_mcp_hub_status, derive_n8n_status
from app.services.knowledge import get_knowledge_provider from app.services.knowledge import KnowledgeHealth, get_knowledge_provider
settings = get_settings() settings = get_settings()
_FAILED_DEMO_EVENT_ID = "00000000-0000-4000-8000-000000000020" _FAILED_DEMO_EVENT_ID = "00000000-0000-4000-8000-000000000020"
def _knowledge_scenario_ready(health: KnowledgeHealth) -> bool:
"""Require a reachable provider and independently verified indexed documents.
``source_document_count`` describes local Markdown files, while a sync report only
describes an upload attempt. Neither proves that the active provider can retrieve a
corpus, so an unavailable verification count remains honestly not ready.
"""
return bool(
health.available and health.document_count is not None and health.document_count > 0
)
def _last_reset(db: Session) -> tuple[datetime | None, str | None]: def _last_reset(db: Session) -> tuple[datetime | None, str | None]:
marker = db.scalar( marker = db.scalar(
select(AuditEvent) select(AuditEvent)
@@ -49,7 +61,8 @@ def _scenarios(db: Session) -> list[DemoScenarioOut]:
return_ready = bool(booking and booking.status == "active" and booking.end_odometer_km is None) return_ready = bool(booking and booking.status == "active" and booking.end_odometer_km is None)
duplicate_ready = bool(duplicate_issue and duplicate_issue.status == "open") duplicate_ready = bool(duplicate_issue and duplicate_issue.status == "open")
overlap_ready = bool(overlap_issue and overlap_issue.status == "open") overlap_ready = bool(overlap_issue and overlap_issue.status == "open")
automation_ready = bool(failed_run and failed_run.delivery_status == "failed") automation_ready = bool(failed_run and is_demo_scenario_failure(failed_run))
knowledge_ready = _knowledge_scenario_ready(knowledge_health)
return [ return [
DemoScenarioOut( DemoScenarioOut(
@@ -119,8 +132,8 @@ def _scenarios(db: Session) -> list[DemoScenarioOut]:
estimated_minutes=2, estimated_minutes=2,
required_roles=["rental_employee", "operations_manager"], required_roles=["rental_employee", "operations_manager"],
start_path="/knowledge", start_path="/knowledge",
ready=knowledge_health.available, ready=knowledge_ready,
blocked_reason_code=None if knowledge_health.available else "knowledgeUnavailable", blocked_reason_code=None if knowledge_ready else "knowledgeUnavailable",
), ),
] ]
@@ -145,9 +158,11 @@ def _integrations(db: Session) -> list[DemoIntegrationSummaryOut]:
key="ragcore", key="ragcore",
status_code=( status_code=(
"operational" "operational"
if knowledge_health.provider == "ragcore" and knowledge_health.available if knowledge_health.provider == "ragcore"
and _knowledge_scenario_ready(knowledge_health)
else "demoMode" else "demoMode"
if knowledge_health.provider == "demo" and knowledge_health.available if knowledge_health.provider == "demo"
and _knowledge_scenario_ready(knowledge_health)
else "unavailable" else "unavailable"
), ),
detail_code=( detail_code=(
@@ -169,9 +184,7 @@ def _integrations(db: Session) -> list[DemoIntegrationSummaryOut]:
# "operational" -- same evidence rule the integration status page uses. # "operational" -- same evidence rule the integration status page uses.
status_code="operational" if mcp_hub.state == "operational" else "notConnected", status_code="operational" if mcp_hub.state == "operational" else "notConnected",
detail_code=( detail_code=(
"mcpDetailOperational" "mcpDetailOperational" if mcp_hub.state == "operational" else "mcpDetailPrepared"
if mcp_hub.state == "operational"
else "mcpDetailPrepared"
), ),
detail_params={}, detail_params={},
), ),
+59 -5
View File
@@ -77,6 +77,14 @@ def _claim_due_events(batch_size: int = 5) -> list[uuid.UUID]:
for row in rows: for row in rows:
row.delivery_status = "delivering" row.delivery_status = "delivering"
row.next_attempt_at = lease_deadline row.next_attempt_at = lease_deadline
# The token is stored inside the internal payload (the wire envelope below
# explicitly selects only contract fields). It lets the outcome transaction
# prove that this is still the same lease after network I/O. A stale worker
# must never overwrite a later reclaim/retry or an idempotent callback.
row.payload_json = {
**row.payload_json,
"_delivery_claim_token": str(uuid.uuid4()),
}
db.commit() db.commit()
return claimed_ids return claimed_ids
finally: finally:
@@ -109,6 +117,9 @@ def _deliver_one(event_id: uuid.UUID) -> None:
wire_event = None wire_event = None
payload_error = f"Malformed outbox payload, missing key {exc}" payload_error = f"Malformed outbox payload, missing key {exc}"
attempts = event.attempts attempts = event.attempts
claim_token = event.payload_json.get("_delivery_claim_token")
if event.delivery_status != "delivering" or not isinstance(claim_token, str):
return
finally: finally:
db.close() db.close()
@@ -130,9 +141,33 @@ def _deliver_one(event_id: uuid.UUID) -> None:
except ValueError: except ValueError:
body = None body = None
if isinstance(body, dict): if isinstance(body, dict):
success = bool(body.get("ok", True)) acknowledged = body.get("ok") is True
error = None if success else f"n8n reported failure: {body}" response_event_id = body.get("event_id")
error_code = None if success else "remoteReportedFailure" event_id_matches = response_event_id == str(event_id)
result = body.get("result")
execution_id = result.get("execution_id") if isinstance(result, dict) else None
execution_id_valid = isinstance(execution_id, str) and bool(execution_id.strip())
success = acknowledged and event_id_matches and execution_id_valid
if success:
error = None
error_code = None
elif not acknowledged:
error = (
"n8n response did not explicitly acknowledge the event with ok=true: "
f"{body}"
)
error_code = (
"remoteReportedFailure" if body.get("ok") is False else "malformedResponse"
)
elif not event_id_matches:
error = (
"n8n acknowledged a different event ID "
f"(expected {event_id}, received {response_event_id!r})"
)
error_code = "mismatchedEventId"
else:
error = "n8n response omitted a valid result.execution_id"
error_code = "malformedResponse"
else: else:
# A 2xx status with a non-object (or unparsable) body means the workflow # A 2xx status with a non-object (or unparsable) body means the workflow
# itself errored before its "Respond to Webhook" node ran -- n8n's default # itself errored before its "Respond to Webhook" node ran -- n8n's default
@@ -152,16 +187,35 @@ def _deliver_one(event_id: uuid.UUID) -> None:
db = SessionLocal() db = SessionLocal()
try: try:
event = db.get(OutboxEvent, event_id) event = db.scalar(
select(OutboxEvent).where(OutboxEvent.event_id == event_id).with_for_update()
)
if event is None: if event is None:
return return
if (
event.delivery_status != "delivering"
or event.attempts != attempts
or event.payload_json.get("_delivery_claim_token") != claim_token
):
logger.info(
"Ignoring stale delivery outcome for event %s because lease ownership changed",
event_id,
)
return
event.attempts = attempts + 1 event.attempts = attempts + 1
event.payload_json = {
key: value
for key, value in event.payload_json.items()
if key != "_delivery_claim_token"
}
if success: if success:
event.delivery_status = "succeeded" event.delivery_status = "succeeded"
event.last_error = None event.last_error = None
event.last_error_code = None event.last_error_code = None
event.next_attempt_at = None event.next_attempt_at = None
event.external_run_id = str((body or {}).get("event_id", event_id)) result = (body or {}).get("result")
execution_id = result.get("execution_id") if isinstance(result, dict) else None
event.external_run_id = execution_id if isinstance(execution_id, str) else None
else: else:
event.last_error = (error or "delivery failed")[:2000] event.last_error = (error or "delivery failed")[:2000]
event.last_error_code = error_code or "unknownError" event.last_error_code = error_code or "unknownError"
+44 -7
View File
@@ -7,7 +7,7 @@ from datetime import UTC, datetime, timedelta
from typing import Any, Literal from typing import Any, Literal
import httpx import httpx
from sqlalchemy import Row, func, select from sqlalchemy import Row, func, or_, select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.core.config import get_settings from app.core.config import get_settings
@@ -93,7 +93,10 @@ def derive_n8n_status(db: Session) -> N8nIntegrationStatus:
latest_failure_at = db.scalar( latest_failure_at = db.scalar(
select(func.max(OutboxEvent.updated_at)).where( select(func.max(OutboxEvent.updated_at)).where(
OutboxEvent.delivery_status == "failed", OutboxEvent.delivery_status == "failed",
or_(
OutboxEvent.last_error_code.is_(None),
OutboxEvent.last_error_code != DEMO_SCENARIO_ERROR_CODE, OutboxEvent.last_error_code != DEMO_SCENARIO_ERROR_CODE,
),
) )
) )
latest_demo_scenario_at = db.scalar( latest_demo_scenario_at = db.scalar(
@@ -106,11 +109,17 @@ def derive_n8n_status(db: Session) -> N8nIntegrationStatus:
state: Literal["disabled", "unavailable", "degraded", "operational", "no_evidence"] state: Literal["disabled", "unavailable", "degraded", "operational", "no_evidence"]
if not settings.n8n_dispatch_enabled: if not settings.n8n_dispatch_enabled:
state = "disabled" state = "disabled"
elif not settings.n8n_webhook_url:
# Persisted history does not make a currently unconfigured dispatcher green.
state = "unavailable"
elif unexpected_failed > 0 and succeeded == 0: elif unexpected_failed > 0 and succeeded == 0:
state = "unavailable" state = "unavailable"
elif unexpected_failed > 0: elif unexpected_failed > 0:
state = "degraded" state = "degraded"
elif succeeded > 0 or pending > 0 or delivering > 0: # Queued/in-flight work proves only that MobilityOps has work for the dispatcher;
# it does not prove that n8n has ever accepted a delivery. A green state requires
# at least one persisted successful round trip.
elif succeeded > 0:
state = "operational" state = "operational"
else: else:
state = "no_evidence" state = "no_evidence"
@@ -128,8 +137,14 @@ def derive_n8n_status(db: Session) -> N8nIntegrationStatus:
# it finishes uploading procedures to RAGcore (app/api/routers/integrations.py:: # it finishes uploading procedures to RAGcore (app/api/routers/integrations.py::
# procedures_sync_result), the same "the workflow's own callback is the evidence" # procedures_sync_result), the same "the workflow's own callback is the evidence"
# pattern the scheduled scan and error handler already use below. # pattern the scheduled scan and error handler already use below.
latest_procedure_sync_at = db.scalar( latest_procedure_sync_row = db.execute(
select(func.max(AuditEvent.occurred_at)).where(AuditEvent.action == "n8n_procedures_synced") select(AuditEvent.occurred_at, AuditEvent.after_json, AuditEvent.metadata_json)
.where(AuditEvent.action == "n8n_procedures_synced")
.order_by(AuditEvent.occurred_at.desc())
.limit(1)
).first()
latest_procedure_sync_at = (
latest_procedure_sync_row[0] if latest_procedure_sync_row is not None else None
) )
# Error handler evidence: registrations posted by the "Fleet Ops — Workflow Error # Error handler evidence: registrations posted by the "Fleet Ops — Workflow Error
@@ -198,6 +213,24 @@ def derive_n8n_status(db: Session) -> N8nIntegrationStatus:
seen_at = failure_signal[0] seen_at = failure_signal[0]
last_status = "failed" last_status = "failed"
execution_id = failure_signal[1] execution_id = failure_signal[1]
if name == "Fleet Ops — RAGcore Procedure Sync" and latest_procedure_sync_row:
sync_at, sync_result, sync_metadata = latest_procedure_sync_row
synced = (sync_result or {}).get("synced", 0)
failed_syncs = (sync_result or {}).get("failed", 0)
# The result callback is authoritative for corpus delivery. A generic
# succeeded heartbeat cannot turn a zero/partial upload green.
sync_result_failed = (
not isinstance(synced, int)
or not isinstance(failed_syncs, int)
or synced <= 0
or failed_syncs > 0
)
if sync_result_failed:
last_status = "failed"
if seen_at is None or sync_at > seen_at:
seen_at = sync_at
if sync_result_failed:
execution_id = (sync_metadata or {}).get("execution_id")
workflow_state: Literal["no_evidence", "healthy", "stale", "failed"] workflow_state: Literal["no_evidence", "healthy", "stale", "failed"]
if seen_at is None: if seen_at is None:
workflow_state = "no_evidence" workflow_state = "no_evidence"
@@ -265,16 +298,20 @@ def derive_mcp_hub_status(db: Session) -> McpHubIntegrationStatus:
last_client = latest_call_row[1] if latest_call_row else None last_client = latest_call_row[1] if latest_call_row else None
last_tool = (latest_call_row[2] or {}).get("tool") if latest_call_row else None last_tool = (latest_call_row[2] or {}).get("tool") if latest_call_row else None
hub_reachable = _check_hub_reachable()
state: Literal["not_configured", "no_evidence", "operational"] state: Literal["not_configured", "no_evidence", "operational"]
if not settings.mcp_hub_registration_enabled: if not settings.mcp_hub_registration_enabled:
state = "not_configured" state = "not_configured"
elif total_calls > 0: elif total_calls > 0 and hub_reachable is not False:
state = "operational" state = "operational"
else: else:
# Historical tool calls remain useful telemetry, but cannot support a current
# operational claim when the configured Hub health endpoint is unreachable.
# ``hub_reachable`` stays available separately so consumers can distinguish
# this from a provider that simply has no call evidence yet.
state = "no_evidence" state = "no_evidence"
hub_reachable = _check_hub_reachable()
return McpHubIntegrationStatus( return McpHubIntegrationStatus(
registration_enabled=settings.mcp_hub_registration_enabled, registration_enabled=settings.mcp_hub_registration_enabled,
state=state, state=state,
+166 -44
View File
@@ -1,5 +1,8 @@
from __future__ import annotations from __future__ import annotations
import hashlib
import re
import uuid
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from pathlib import Path from pathlib import Path
from threading import Lock from threading import Lock
@@ -13,6 +16,10 @@ from app.services.knowledge import EvidenceState, GroundedAnswer, KnowledgeHealt
from app.services.knowledge.procedures import ProcedureDocument, iter_procedure_documents from app.services.knowledge.procedures import ProcedureDocument, iter_procedure_documents
_GROUNDED_ANSWERABILITY = {"answerable", "partially_answerable"} _GROUNDED_ANSWERABILITY = {"answerable", "partially_answerable"}
_ANSWERABILITY_STATES = _GROUNDED_ANSWERABILITY | {
"not_answerable",
"conflicting_evidence",
}
# Mirrors DemoKnowledgeProvider's own extractive template in spirit: a real cited # Mirrors DemoKnowledgeProvider's own extractive template in spirit: a real cited
# excerpt wrapped in a fixed sentence, never a generated summary. Used only as a # excerpt wrapped in a fixed sentence, never a generated summary. Used only as a
@@ -47,6 +54,11 @@ _DOMAIN_CONCEPTS: dict[str, tuple[str, ...]] = {
"availability": ("available", "availability", "beschikbaar", "disponible", "disponibilité"), "availability": ("available", "availability", "beschikbaar", "disponible", "disponibilité"),
"technical": ("technical", "warning", "technisch", "waarschuwing", "technique", "alerte"), "technical": ("technical", "warning", "technisch", "waarschuwing", "technique", "alerte"),
} }
_ANSWERABLE_INTENT_CONCEPTS = frozenset(_DOMAIN_CONCEPTS) - {"vehicle", "customer"}
def _normalize_evidence_text(value: str) -> str:
return " ".join(re.findall(r"\w+", value.casefold()))
def _question_concepts(question: str) -> set[str]: def _question_concepts(question: str) -> set[str]:
@@ -93,7 +105,9 @@ def _rank_sources_for_concepts(sources: list[SourceCard], concepts: set[str]) ->
) )
def _retrieval_score(result: dict) -> float | None: def _retrieval_score(result: object) -> float | None:
if not isinstance(result, dict):
return None
scores = result.get("scores") scores = result.get("scores")
if not isinstance(scores, dict): if not isinstance(scores, dict):
return None return None
@@ -124,23 +138,81 @@ class RAGcoreKnowledgeProvider:
excerpt RAGcore's own search actually found, wrapped in the same fixed citation excerpt RAGcore's own search actually found, wrapped in the same fixed citation
template `DemoKnowledgeProvider` uses -- never a fabricated summary. template `DemoKnowledgeProvider` uses -- never a fabricated summary.
Known gap, not fixable from this side: RAGcore's ingest pipeline currently tags every Retrieval is scoped to the stable RAGcore ``source_id`` values owned by Fleet Ops for
chunk's `language` payload field as `"en"` regardless of actual document language (the the requested UI language. Returned internal document/version UUIDs are deliberately
`/v1/uploads` contract has no per-file language field for a caller to set correctly). not treated as Fleet Ops identifiers: every citation must instead prove the complete
Filtering search/answer requests by requested UI language would therefore silently managed-source chain (source id, URI, locator, checksum and extractive local text).
exclude genuinely-relevant nl-BE/fr-BE content, so this adapter deliberately does not
filter by language -- retrieval relies on the embedding model's cross-lingual matching.
""" """
name = "ragcore" name = "ragcore"
def __init__(self) -> None: def __init__(self) -> None:
self._settings = get_settings() self._settings = get_settings()
self._managed_documents_by_source_id: dict[str, ProcedureDocument] = {}
for document in iter_procedure_documents(Path(self._settings.knowledge_dir)):
self._managed_documents_by_source_id[document.source_id] = document
self._verification_cache: dict[str, tuple[float, int]] = {} self._verification_cache: dict[str, tuple[float, int]] = {}
self._verification_lock = Lock() self._verification_lock = Lock()
self._answers_circuit_lock = Lock() self._answers_circuit_lock = Lock()
self._answers_circuit_open_until = 0.0 self._answers_circuit_open_until = 0.0
def _managed_source(self, citation: object, language: str) -> SourceCard | None:
if not isinstance(citation, dict):
return None
source_id = citation.get("source_id")
if not isinstance(source_id, str):
return None
document = self._managed_documents_by_source_id.get(str(source_id))
if document is None or document.language != language:
return None
if citation.get("locator") != f"{document.document_id}.md":
return None
if citation.get("source_uri") != f"ragcore://source/{document.source_id}":
return None
for field_name in ("id", "document_id", "document_version_id"):
value = citation.get(field_name)
if not isinstance(value, str):
return None
try:
parsed = uuid.UUID(value)
except (ValueError, TypeError, AttributeError):
return None
if parsed.int == 0:
return None
if not isinstance(citation.get("title"), str):
return None
section = citation.get("section")
if section is not None and not isinstance(section, str):
return None
excerpt = citation.get("excerpt")
if not isinstance(excerpt, str) or not excerpt.strip():
return None
expected_hash = hashlib.sha256(excerpt.encode("utf-8")).hexdigest()
if citation.get("excerpt_sha256") != expected_hash:
return None
# RAGcore owns chunking, but the cited excerpt must still be extractive evidence
# from the authoritative local procedure. Token normalization tolerates Markdown
# punctuation/whitespace while rejecting provider text that was never uploaded.
normalized_excerpt = _normalize_evidence_text(excerpt)
if not normalized_excerpt or normalized_excerpt not in _normalize_evidence_text(
document.content
):
return None
return SourceCard(
document_id=document.document_id,
title=document.title,
version=document.version,
section=section or "",
excerpt=excerpt,
)
def _managed_source_ids(self, language: str) -> list[str]:
return [
document.source_id
for document in self._managed_documents_by_source_id.values()
if document.language == language
]
def _answers_circuit_is_open(self) -> bool: def _answers_circuit_is_open(self) -> bool:
with self._answers_circuit_lock: with self._answers_circuit_lock:
return monotonic() < self._answers_circuit_open_until return monotonic() < self._answers_circuit_open_until
@@ -284,7 +356,7 @@ class RAGcoreKnowledgeProvider:
if self._answers_circuit_is_open(): if self._answers_circuit_is_open():
KNOWLEDGE_PROVIDER_REQUESTS.labels("answers", "circuit_open").inc() KNOWLEDGE_PROVIDER_REQUESTS.labels("answers", "circuit_open").inc()
else: else:
answered = self._ask_via_answers(question, correlation_id) answered = self._ask_via_answers(question, correlation_id, language)
if answered is not None: if answered is not None:
return answered return answered
# /v1/answers itself is unavailable (non-2xx or unreachable) -- fall back to # /v1/answers itself is unavailable (non-2xx or unreachable) -- fall back to
@@ -295,7 +367,9 @@ class RAGcoreKnowledgeProvider:
# generation step. # generation step.
return self._ask_via_search_fallback(question, correlation_id, language) return self._ask_via_search_fallback(question, correlation_id, language)
def _ask_via_answers(self, question: str, correlation_id: str) -> GroundedAnswer | None: def _ask_via_answers(
self, question: str, correlation_id: str, language: str
) -> GroundedAnswer | None:
"""Returns None (not a GroundedAnswer) when /v1/answers itself is unavailable, """Returns None (not a GroundedAnswer) when /v1/answers itself is unavailable,
so the caller can fall back to search -- as opposed to a real 200 response so the caller can fall back to search -- as opposed to a real 200 response
classifying the question as insufficiently answerable, which is a genuine, classifying the question as insufficiently answerable, which is a genuine,
@@ -307,6 +381,7 @@ class RAGcoreKnowledgeProvider:
json={ json={
"query": question, "query": question,
"requested_space_ids": [self._settings.ragcore_space_id], "requested_space_ids": [self._settings.ragcore_space_id],
"filters": {"source_ids": self._managed_source_ids(language)},
}, },
) )
if response.status_code != 200: if response.status_code != 200:
@@ -320,25 +395,73 @@ class RAGcoreKnowledgeProvider:
return None return None
try: try:
citations = {c["id"]: c for c in body.get("citations", [])} if not isinstance(body, dict):
sources = [ raise TypeError("answer response must be an object")
SourceCard( # Validate the provider's complete AnswerResponse contract before trusting
document_id=str(citation["document_id"]), # generated prose. These IDs and claim bindings are the evidence that RAGcore
title=citation["title"], # ran its deterministic claim/citation validator; one unrelated but otherwise
version=str(citation["document_version_id"]), # valid citation must never make arbitrary answer text appear grounded.
section=citation.get("section") or "", uuid.UUID(str(body["answer_id"]))
excerpt=citation["excerpt"], uuid.UUID(str(body["retrieval_run_id"]))
) raw_citations = body.get("citations", [])
for citation in citations.values() if not isinstance(raw_citations, list):
raise TypeError("citations must be a list")
citation_ids: set[uuid.UUID] = set()
for citation in raw_citations:
if not isinstance(citation, dict):
raise TypeError("citation must be an object")
citation_ids.add(uuid.UUID(str(citation["id"])))
raw_claims = body["claims"]
if not isinstance(raw_claims, list):
raise TypeError("claims must be a list")
claims_are_bound = bool(raw_claims)
answer_text = body.get("answer")
for claim in raw_claims:
if not isinstance(claim, dict):
raise TypeError("claim must be an object")
claim_text = claim.get("text")
claim_citation_ids = claim.get("citation_ids")
if (
not isinstance(claim_text, str)
or not claim_text.strip()
or not isinstance(answer_text, str)
or claim_text.strip() not in answer_text
or not isinstance(claim_citation_ids, list)
or not claim_citation_ids
):
claims_are_bound = False
continue
try:
bound_ids = {uuid.UUID(str(item)) for item in claim_citation_ids}
except (TypeError, ValueError):
claims_are_bound = False
continue
if not bound_ids.issubset(citation_ids):
claims_are_bound = False
mapped_sources = [
source
for citation in raw_citations
if (source := self._managed_source(citation, language)) is not None
] ]
sources = _deduplicate_sources(sources) citations_are_managed = len(mapped_sources) == len(raw_citations)
sources = _deduplicate_sources(mapped_sources)
answerability = body.get("answerability", "not_answerable") answerability = body.get("answerability", "not_answerable")
is_grounded = answerability in _GROUNDED_ANSWERABILITY and sources if answerability not in _ANSWERABILITY_STATES:
raise TypeError("answerability is invalid")
if not isinstance(answer_text, str):
raise TypeError("answer must be a string")
is_grounded = (
answerability in _GROUNDED_ANSWERABILITY
and bool(sources)
and citations_are_managed
and claims_are_bound
and bool(answer_text.strip())
)
evidence_state: EvidenceState = "grounded" if is_grounded else "insufficient" evidence_state: EvidenceState = "grounded" if is_grounded else "insufficient"
KNOWLEDGE_PROVIDER_REQUESTS.labels("answers", evidence_state).inc() KNOWLEDGE_PROVIDER_REQUESTS.labels("answers", evidence_state).inc()
self._close_answers_circuit() self._close_answers_circuit()
return GroundedAnswer( return GroundedAnswer(
answer=body.get("answer", "") if evidence_state == "grounded" else "", answer=answer_text if evidence_state == "grounded" else "",
evidence_state=evidence_state, evidence_state=evidence_state,
sources=sources if evidence_state == "grounded" else [], sources=sources if evidence_state == "grounded" else [],
provider=self.name, provider=self.name,
@@ -366,6 +489,7 @@ class RAGcoreKnowledgeProvider:
json={ json={
"query": question, "query": question,
"requested_space_ids": [self._settings.ragcore_space_id], "requested_space_ids": [self._settings.ragcore_space_id],
"filters": {"source_ids": self._managed_source_ids(language)},
"max_results": 5, "max_results": 5,
}, },
) )
@@ -378,24 +502,24 @@ class RAGcoreKnowledgeProvider:
return unavailable return unavailable
try: try:
if not isinstance(body, dict):
raise TypeError("search response must be an object")
results = body.get("results", []) results = body.get("results", [])
if not isinstance(results, list): if not isinstance(results, list):
raise TypeError("results must be a list") raise TypeError("results must be a list")
sources = [ sources = [
SourceCard( source
document_id=str(result["citation"]["document_id"]),
title=result["citation"]["title"],
version=str(result["citation"]["document_version_id"]),
section=result["citation"].get("section") or "",
excerpt=result["citation"]["excerpt"],
)
for result in results for result in results
if isinstance(result, dict)
and (source := self._managed_source(result.get("citation"), language)) is not None
] ]
except (TypeError, KeyError, ValueError): except (TypeError, KeyError, ValueError):
KNOWLEDGE_PROVIDER_REQUESTS.labels("search", "malformed").inc() KNOWLEDGE_PROVIDER_REQUESTS.labels("search", "malformed").inc()
return unavailable return unavailable
for result in results: for result in results:
if not isinstance(result, dict):
continue
score = _retrieval_score(result) score = _retrieval_score(result)
if score is not None: if score is not None:
KNOWLEDGE_RETRIEVAL_SCORE.observe(score) KNOWLEDGE_RETRIEVAL_SCORE.observe(score)
@@ -403,15 +527,11 @@ class RAGcoreKnowledgeProvider:
all_sources = _deduplicate_sources(sources) all_sources = _deduplicate_sources(sources)
concepts = _question_concepts(question) concepts = _question_concepts(question)
qualified_sources = [ qualified_sources = [
SourceCard( source
document_id=str(result["citation"]["document_id"]),
title=result["citation"]["title"],
version=str(result["citation"]["document_version_id"]),
section=result["citation"].get("section") or "",
excerpt=result["citation"]["excerpt"],
)
for result in results for result in results
if (_retrieval_score(result) or 0.0) >= self._settings.ragcore_min_search_score if isinstance(result, dict)
and (_retrieval_score(result) or 0.0) >= self._settings.ragcore_min_search_score
and (source := self._managed_source(result.get("citation"), language)) is not None
] ]
sources = _rank_sources_for_concepts(_deduplicate_sources(qualified_sources), concepts) sources = _rank_sources_for_concepts(_deduplicate_sources(qualified_sources), concepts)
if not all_sources: if not all_sources:
@@ -424,15 +544,17 @@ class RAGcoreKnowledgeProvider:
correlation_id=correlation_id, correlation_id=correlation_id,
) )
damage_evidence = any( required_concepts = concepts & _ANSWERABLE_INTENT_CONCEPTS
term evidence_text = " ".join(
in (
f"{source.document_id} {source.title} {source.section} {source.excerpt}" f"{source.document_id} {source.title} {source.section} {source.excerpt}"
).casefold()
for source in sources for source in sources
for term in _DOMAIN_CONCEPTS["damage"] ).casefold()
) covered_concepts = {
if not concepts or not sources or ("damage" in concepts and not damage_evidence): concept
for concept in required_concepts
if any(term in evidence_text for term in _DOMAIN_CONCEPTS[concept])
}
if not required_concepts or not sources or covered_concepts != required_concepts:
KNOWLEDGE_PROVIDER_REQUESTS.labels("search", "insufficient").inc() KNOWLEDGE_PROVIDER_REQUESTS.labels("search", "insufficient").inc()
return GroundedAnswer( return GroundedAnswer(
answer="", answer="",
+15 -22
View File
@@ -4,7 +4,7 @@ import hashlib
import json import json
import uuid import uuid
from dataclasses import dataclass from dataclasses import dataclass
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime
from sqlalchemy import select from sqlalchemy import select
from sqlalchemy.exc import IntegrityError from sqlalchemy.exc import IntegrityError
@@ -12,13 +12,13 @@ from sqlalchemy.orm import Session
from app.core.errors import AppError from app.core.errors import AppError
from app.models.booking import Booking from app.models.booking import Booking
from app.models.data_quality import DataQualityIssue
from app.models.idempotency import IdempotencyRecord from app.models.idempotency import IdempotencyRecord
from app.models.inspection import Inspection from app.models.inspection import Inspection
from app.models.outbox import OutboxEvent from app.models.outbox import OutboxEvent
from app.models.vehicle import Vehicle from app.models.vehicle import Vehicle
from app.schemas import CurrentUser, RegisterReturnRequest from app.schemas import CurrentUser, RegisterReturnRequest
from app.services.audit import record_audit_event from app.services.audit import record_audit_event
from app.services.data_quality import open_odometer_regression_issue
def _new_inspection_ref() -> str: def _new_inspection_ref() -> str:
@@ -261,28 +261,21 @@ def register_vehicle_return(
quality_issue_ref: str | None = None quality_issue_ref: str | None = None
if evaluation.odometer_regression: if evaluation.odometer_regression:
issue = DataQualityIssue( issue = open_odometer_regression_issue(
public_ref=f"DQ-RET-{str(inspection.public_ref).split('-')[-1]}", db,
rule_type="odometer_regression", vehicle=vehicle,
entity_type="vehicle", reading_ref=inspection.public_ref,
entity_id=vehicle.id, reading_km=body.end_odometer_km,
severity="medium", canonical_km=evaluation.canonical_odometer_km,
status="open", source_type="return",
evidence_json={ related_refs=[booking.public_ref, inspection.public_ref],
"summary": ( correctable_booking_refs=[booking.public_ref],
f"Return submitted {body.end_odometer_km} km, below canonical "
f"{evaluation.canonical_odometer_km} km."
),
"entity_ref": vehicle.public_ref,
"related_refs": [booking.public_ref, inspection.public_ref],
},
proposed_action_json={},
detected_at=now, detected_at=now,
due_at=now + timedelta(days=1), public_ref=f"DQ-RET-{str(inspection.public_ref).split('-')[-1]}",
actor_label=actor.display_name,
correlation_id=correlation_id,
) )
db.add(issue) quality_issue_ref = issue.public_ref if issue is not None else None
db.flush()
quality_issue_ref = issue.public_ref
resulting_status = evaluation.resulting_vehicle_status resulting_status = evaluation.resulting_vehicle_status
vehicle.odometer_km = evaluation.resulting_odometer_km vehicle.odometer_km = evaluation.resulting_odometer_km
+2
View File
@@ -1,5 +1,7 @@
#!/bin/sh #!/bin/sh
set -e set -e
if [ "${RUN_MIGRATIONS:-true}" = "true" ]; then
alembic upgrade head alembic upgrade head
fi
exec uvicorn app.main:app --host 0.0.0.0 --port 8000 exec uvicorn app.main:app --host 0.0.0.0 --port 8000
+105 -2
View File
@@ -1,3 +1,15 @@
import threading
import pytest
from sqlalchemy import delete, func, select
from app.core.db import SessionLocal, engine
from app.models.audit import AuditEvent
from app.models.vehicle import Vehicle
from app.seed_loader import reset_and_seed
from app.services.audit import record_audit_event
def test_unauthenticated_dashboard_is_rejected(client): def test_unauthenticated_dashboard_is_rejected(client):
response = client.get("/api/v1/dashboard") response = client.get("/api/v1/dashboard")
assert response.status_code == 401 assert response.status_code == 401
@@ -101,15 +113,106 @@ def test_demo_reset_rejects_concurrent_rebuild(ops_client):
assert response.status_code == 409 assert response.status_code == 409
def test_demo_reset_database_lock_blocks_another_api_replica(ops_client):
import app.api.routers.demo as demo_router
# Use a raw connection: an ordinary Session intentionally participates in the
# shared mutation barrier and would make an exclusive reset wait before it can test
# this separate non-blocking replica lock.
with engine.connect() as lock_connection:
lock_connection.scalar(select(func.pg_advisory_lock(demo_router._RESET_ADVISORY_LOCK_ID)))
try:
response = ops_client.post("/api/v1/demo/reset")
finally:
lock_connection.scalar(
select(func.pg_advisory_unlock(demo_router._RESET_ADVISORY_LOCK_ID))
)
assert response.status_code == 409
def test_demo_reset_waits_for_active_mutation_and_then_replaces_it_atomically():
started = threading.Event()
completed = threading.Event()
errors: list[BaseException] = []
def reset_in_other_session() -> None:
started.set()
try:
with SessionLocal() as reset_db:
reset_and_seed(reset_db)
except BaseException as exc: # pragma: no cover - assertion reports thread failures
errors.append(exc)
finally:
completed.set()
with SessionLocal() as mutation_db:
vehicle = mutation_db.scalar(
select(Vehicle).where(Vehicle.public_ref == "MO-001").with_for_update()
)
vehicle.location = "Concurrent mutation marker"
mutation_db.flush()
worker = threading.Thread(target=reset_in_other_session, daemon=True)
worker.start()
assert started.wait(timeout=1)
assert not completed.wait(timeout=0.2)
mutation_db.commit()
worker.join(timeout=10)
assert not worker.is_alive(), "reset deadlocked behind the active mutation"
assert errors == []
with SessionLocal() as db:
restored = db.scalar(select(Vehicle).where(Vehicle.public_ref == "MO-001"))
assert restored.location != "Concurrent mutation marker"
def test_demo_reset_cooldown_returns_retry_after(ops_client, monkeypatch): def test_demo_reset_cooldown_returns_retry_after(ops_client, monkeypatch):
import app.api.routers.demo as demo_router import app.api.routers.demo as demo_router
monkeypatch.setattr(demo_router.settings, "demo_reset_cooldown_seconds", 60) monkeypatch.setattr(demo_router.settings, "demo_reset_cooldown_seconds", 60)
monkeypatch.setattr(demo_router, "_last_reset_monotonic", demo_router.time.monotonic()) with SessionLocal() as db:
cooldown_event = record_audit_event(
db,
actor_type="user",
actor_label="Cooldown test",
action="demo_reset",
entity_type="system",
)
db.commit()
cooldown_event_id = cooldown_event.id
response = ops_client.post("/api/v1/demo/reset") response = ops_client.post("/api/v1/demo/reset")
assert response.status_code == 429 assert response.status_code == 429
assert int(response.headers["retry-after"]) >= 1 assert int(response.headers["retry-after"]) >= 1
monkeypatch.setattr(demo_router, "_last_reset_monotonic", 0.0) with SessionLocal() as db:
db.execute(delete(AuditEvent).where(AuditEvent.id == cooldown_event_id))
db.commit()
def test_demo_reset_rolls_back_every_change_when_integrity_check_fails(ops_client, monkeypatch):
import app.api.routers.demo as demo_router
monkeypatch.setattr(demo_router.settings, "demo_reset_cooldown_seconds", 0)
with SessionLocal() as db:
probe = record_audit_event(
db,
actor_type="system",
actor_label="Atomic reset test",
action="reset_atomicity_probe",
entity_type="system",
)
db.commit()
probe_id = probe.id
def fail_integrity_check(_db):
raise RuntimeError("Injected integrity-check failure")
monkeypatch.setattr(demo_router, "scenario_integrity_report", fail_integrity_check)
with pytest.raises(RuntimeError, match="Injected integrity-check failure"):
ops_client.post("/api/v1/demo/reset")
with SessionLocal() as db:
assert db.scalar(select(AuditEvent).where(AuditEvent.id == probe_id)) is not None
db.execute(delete(AuditEvent).where(AuditEvent.id == probe_id))
db.commit()
def test_reset_is_rejected_when_demo_allow_reset_is_disabled(ops_client, monkeypatch): def test_reset_is_rejected_when_demo_allow_reset_is_disabled(ops_client, monkeypatch):
+127 -1
View File
@@ -95,6 +95,16 @@ def test_booking_requirements_are_explicit_and_audited(ops_client):
) )
assert checkout.status_code == 409 assert checkout.status_code == 409
whitespace_only = ops_client.post(
f"/api/v1/bookings/{booking['public_ref']}/complete-requirements",
json={"confirmation": " "},
)
assert whitespace_only.status_code == 422
assert (
ops_client.get(f"/api/v1/bookings/{booking['public_ref']}").json()["requirements_complete"]
is False
)
confirmed = ops_client.post( confirmed = ops_client.post(
f"/api/v1/bookings/{booking['public_ref']}/complete-requirements", f"/api/v1/bookings/{booking['public_ref']}/complete-requirements",
json={"confirmation": "Licence and rental conditions checked"}, json={"confirmation": "Licence and rental conditions checked"},
@@ -106,6 +116,29 @@ def test_booking_requirements_are_explicit_and_audited(ops_client):
assert any(item["entity_ref"] == booking["public_ref"] for item in audit.json()) assert any(item["entity_ref"] == booking["public_ref"] for item in audit.json())
def test_booking_creation_cannot_preconfirm_requirements(ops_client):
window = {"starts_at": "2033-09-01T10:00:00Z", "ends_at": "2033-09-02T12:00:00Z"}
vehicle = ops_client.get("/api/v1/bookings/availability", params=window).json()[0]
created = ops_client.post(
"/api/v1/bookings",
json={
"customer_ref": "CUS-0001",
"vehicle_ref": vehicle["public_ref"],
"requirements_complete": True,
**window,
},
)
assert created.status_code == 201
assert created.json()["requirements_complete"] is False
audit = ops_client.get(
"/api/v1/audit",
params={"action": "booking_requirements_completed"},
)
assert all(item["entity_ref"] != created.json()["public_ref"] for item in audit.json())
def test_customer_search_returns_canonical_customers(ops_client): def test_customer_search_returns_canonical_customers(ops_client):
response = ops_client.get("/api/v1/customers", params={"query": "CUS-"}) response = ops_client.get("/api/v1/customers", params={"query": "CUS-"})
assert response.status_code == 200 assert response.status_code == 200
@@ -221,10 +254,14 @@ def test_checkout_records_inspection_and_activates_safe_booking(ops_client):
json={ json={
"customer_ref": "CUS-0001", "customer_ref": "CUS-0001",
"vehicle_ref": vehicle["public_ref"], "vehicle_ref": vehicle["public_ref"],
"requirements_complete": True,
**window, **window,
}, },
).json() ).json()
confirmed = ops_client.post(
f"/api/v1/bookings/{booking['public_ref']}/complete-requirements",
json={"confirmation": "Licence and rental conditions checked"},
)
assert confirmed.status_code == 200
response = ops_client.post( response = ops_client.post(
f"/api/v1/bookings/{booking['public_ref']}/checkout", f"/api/v1/bookings/{booking['public_ref']}/checkout",
json={ json={
@@ -241,6 +278,95 @@ def test_checkout_records_inspection_and_activates_safe_booking(ops_client):
updated_vehicle = ops_client.get(f"/api/v1/vehicles/{vehicle['public_ref']}").json() updated_vehicle = ops_client.get(f"/api/v1/vehicles/{vehicle['public_ref']}").json()
assert updated_vehicle["operational_status"] == "rented" assert updated_vehicle["operational_status"] == "rented"
assert any(item["type"] == "checkout" for item in updated_vehicle["inspections"]) assert any(item["type"] == "checkout" for item in updated_vehicle["inspections"])
checkout_audit = ops_client.get(
"/api/v1/audit",
params={"action": "booking_checkout_recorded"},
).json()
booking_event = next(
item for item in checkout_audit if item["entity_ref"] == booking["public_ref"]
)
vehicle_audit = ops_client.get(
"/api/v1/audit",
params={
"action": "vehicle_status_changed",
"correlation_id": booking_event["correlation_id"],
},
).json()
assert len(vehicle_audit) == 1
assert vehicle_audit[0]["entity_ref"] == vehicle["public_ref"]
assert vehicle_audit[0]["before"]["operational_status"] == "available"
assert vehicle_audit[0]["after"]["operational_status"] == "rented"
def test_checkout_odometer_regression_keeps_canonical_and_opens_bounded_quality_issue(
ops_client,
):
window = {"starts_at": "2052-09-01T10:00:00Z", "ends_at": "2052-09-02T12:00:00Z"}
existing_issues = ops_client.get(
"/api/v1/data-quality/issues",
params={"status": "open", "rule_type": "odometer_regression"},
).json()
unavailable_refs = {issue["entity_ref"] for issue in existing_issues}
available = ops_client.get("/api/v1/bookings/availability", params=window).json()
vehicle_option = next(
item
for item in available
if item["operational_status"] == "available" and item["public_ref"] not in unavailable_refs
)
vehicle = ops_client.get(f"/api/v1/vehicles/{vehicle_option['public_ref']}").json()
assert vehicle["odometer_km"] > 0
booking = ops_client.post(
"/api/v1/bookings",
json={
"customer_ref": "CUS-0003",
"vehicle_ref": vehicle["public_ref"],
**window,
},
).json()
ops_client.post(
f"/api/v1/bookings/{booking['public_ref']}/complete-requirements",
json={"confirmation": "Licence and rental conditions checked"},
)
checkout = ops_client.post(
f"/api/v1/bookings/{booking['public_ref']}/checkout",
json={
"start_odometer_km": vehicle["odometer_km"] - 1,
"fuel_level_percent": 90,
"cleanliness_ok": True,
"damage_reported": False,
"technical_warning": False,
},
)
assert checkout.status_code == 200
assert checkout.json()["booking_status"] == "blocked"
updated_vehicle = ops_client.get(f"/api/v1/vehicles/{vehicle['public_ref']}").json()
assert updated_vehicle["odometer_km"] == vehicle["odometer_km"]
issues = ops_client.get(
"/api/v1/data-quality/issues",
params={"status": "open", "rule_type": "odometer_regression"},
).json()
issue = next(item for item in issues if item["entity_ref"] == vehicle["public_ref"])
assert issue["evidence"]["source_type"] == "checkout"
assert issue["evidence"]["correctable_booking_refs"] == []
detail = ops_client.get(f"/api/v1/data-quality/issues/{issue['public_ref']}").json()
assert any(snapshot["entity_type"] == "inspection" for snapshot in detail["related_snapshots"])
invalid_correction = ops_client.post(
f"/api/v1/data-quality/issues/{issue['public_ref']}/resolve-odometer-regression",
json={
"decision": "correct_reading",
"booking_ref": booking["public_ref"],
"corrected_odometer_km": vehicle["odometer_km"],
},
)
assert invalid_correction.status_code == 422
retained = ops_client.post(
f"/api/v1/data-quality/issues/{issue['public_ref']}/resolve-odometer-regression",
json={"decision": "retain_canonical"},
)
assert retained.status_code == 200
assert retained.json()["status"] == "resolved"
def test_get_booking_detail(ops_client): def test_get_booking_detail(ops_client):
+43
View File
@@ -1,3 +1,9 @@
from sqlalchemy import select
from app.core.db import SessionLocal
from app.models.vehicle import Vehicle
def test_dashboard_metrics_are_persisted_counts(ops_client): def test_dashboard_metrics_are_persisted_counts(ops_client):
response = ops_client.get("/api/v1/dashboard") response = ops_client.get("/api/v1/dashboard")
assert response.status_code == 200 assert response.status_code == 200
@@ -15,6 +21,43 @@ def test_dashboard_metrics_are_persisted_counts(ops_client):
assert metrics["pending_or_failed_workflows"] >= 1 assert metrics["pending_or_failed_workflows"] >= 1
def test_dashboard_metrics_change_when_persisted_fleet_data_changes(ops_client):
before = ops_client.get("/api/v1/dashboard").json()["metrics"]
db = SessionLocal()
probe = Vehicle(
public_ref="MO-METRIC-PROBE",
make="Synthetic",
model="Metric probe",
model_year=2026,
registration_number="TEST-METRIC-PROBE",
location="Test fixture",
operational_status="blocked",
odometer_km=0,
next_service_km=1,
active=True,
version=1,
)
try:
db.add(probe)
db.commit()
after_response = ops_client.get("/api/v1/dashboard")
assert after_response.status_code == 200
after = after_response.json()["metrics"]
assert after["blocked"] == before["blocked"] + 1
fleet_statuses = ("available", "rented", "cleaning", "maintenance", "blocked")
assert (
sum(after[status] for status in fleet_statuses)
== sum(before[status] for status in fleet_statuses) + 1
)
finally:
persisted_probe = db.scalar(select(Vehicle).where(Vehicle.public_ref == "MO-METRIC-PROBE"))
if persisted_probe is not None:
db.delete(persisted_probe)
db.commit()
db.close()
def test_dashboard_attention_items_link_to_records(ops_client): def test_dashboard_attention_items_link_to_records(ops_client):
response = ops_client.get("/api/v1/dashboard") response = ops_client.get("/api/v1/dashboard")
body = response.json() body = response.json()
+818 -4
View File
@@ -1,8 +1,25 @@
from sqlalchemy import select from concurrent.futures import ThreadPoolExecutor
from datetime import UTC, datetime, timedelta
from threading import Barrier
from sqlalchemy import delete, select, text
from app.core.db import SessionLocal from app.core.db import SessionLocal
from app.models.audit import AuditEvent
from app.models.booking import Booking from app.models.booking import Booking
from app.models.customer import Customer
from app.models.data_quality import DataQualityIssue
from app.models.idempotency import IdempotencyRecord
from app.models.inspection import Inspection
from app.models.maintenance import MaintenanceRecord
from app.models.outbox import OutboxEvent
from app.models.vehicle import Vehicle from app.models.vehicle import Vehicle
from app.schemas import CurrentUser, RegisterReturnRequest, ResolveOdometerRegressionRequest
from app.services.data_quality import (
open_odometer_regression_issue,
resolve_odometer_regression,
)
from app.services.returns import register_vehicle_return
def _activate_booking(vehicle_ref: str, start_odometer_km: int) -> str: def _activate_booking(vehicle_ref: str, start_odometer_km: int) -> str:
@@ -21,6 +38,30 @@ def _activate_booking(vehicle_ref: str, start_odometer_km: int) -> str:
db.close() db.close()
def _cleanup_odometer_scenario(vehicle_id, customer_id) -> None:
"""Remove one isolated DQ-03 scenario in foreign-key-safe order."""
with SessionLocal() as db:
booking_ids = db.scalars(select(Booking.id).where(Booking.vehicle_id == vehicle_id)).all()
issue_ids = db.scalars(
select(DataQualityIssue.id).where(DataQualityIssue.entity_id == vehicle_id)
).all()
audit_entity_ids = [vehicle_id, *booking_ids, *issue_ids]
if audit_entity_ids:
db.execute(delete(AuditEvent).where(AuditEvent.entity_id.in_(audit_entity_ids)))
if booking_ids:
db.execute(
delete(IdempotencyRecord).where(IdempotencyRecord.booking_id.in_(booking_ids))
)
db.execute(delete(OutboxEvent).where(OutboxEvent.aggregate_id.in_(booking_ids)))
db.execute(delete(DataQualityIssue).where(DataQualityIssue.entity_id == vehicle_id))
db.execute(delete(Inspection).where(Inspection.vehicle_id == vehicle_id))
db.execute(delete(MaintenanceRecord).where(MaintenanceRecord.vehicle_id == vehicle_id))
db.execute(delete(Booking).where(Booking.vehicle_id == vehicle_id))
db.execute(delete(Vehicle).where(Vehicle.id == vehicle_id))
db.execute(delete(Customer).where(Customer.id == customer_id))
db.commit()
def test_list_includes_all_five_rule_types(ops_client): def test_list_includes_all_five_rule_types(ops_client):
response = ops_client.get("/api/v1/data-quality/issues") response = ops_client.get("/api/v1/data-quality/issues")
assert response.status_code == 200 assert response.status_code == 200
@@ -43,6 +84,84 @@ def test_scan_is_idempotent_once_seeded(ops_client):
assert response.json()["created"] == {} assert response.json()["created"] == {}
def test_scan_detects_imported_maintenance_odometer_regression(ops_client):
with SessionLocal() as db:
vehicle = Vehicle(
public_ref="MO-DQ-SCAN",
make="Synthetic",
model="Scanner",
model_year=2026,
registration_number="DQ-SCAN-01",
location="Brussels",
operational_status="available",
odometer_km=20_000,
next_service_km=30_000,
active=True,
version=1,
)
db.add(vehicle)
db.flush()
vehicle_id = vehicle.id
db.add_all(
[
MaintenanceRecord(
public_ref="MAINT-DQ-SCAN-A",
vehicle_id=vehicle.id,
occurred_at=datetime(2045, 1, 1, tzinfo=UTC),
odometer_km=20_000,
category="inspection",
summary="Synthetic scan baseline",
),
MaintenanceRecord(
public_ref="MAINT-DQ-SCAN-B",
vehicle_id=vehicle.id,
occurred_at=datetime(2045, 2, 1, tzinfo=UTC),
odometer_km=19_000,
category="inspection",
summary="Synthetic imported regression",
),
]
)
db.commit()
try:
response = ops_client.post("/api/v1/data-quality/scan")
assert response.status_code == 200
assert response.json()["created"]["odometer_regression"] >= 1
issues = ops_client.get(
"/api/v1/data-quality/issues",
params={"status": "open", "rule_type": "odometer_regression"},
).json()
issue = next(item for item in issues if item["entity_ref"] == "MO-DQ-SCAN")
assert issue["evidence"]["source_type"] == "maintenance"
assert issue["evidence"]["correctable_booking_refs"] == []
# A later imported regression must enrich the existing open issue instead of
# being silently dropped by the generic check-then-return scanner path.
with SessionLocal() as db:
db.add(
MaintenanceRecord(
public_ref="MAINT-DQ-SCAN-C",
vehicle_id=vehicle_id,
occurred_at=datetime(2045, 3, 1, tzinfo=UTC),
odometer_km=18_000,
category="inspection",
summary="Second synthetic imported regression",
)
)
db.commit()
rescanned = ops_client.post("/api/v1/data-quality/scan")
assert rescanned.status_code == 200
enriched = ops_client.get(f"/api/v1/data-quality/issues/{issue['public_ref']}").json()
later_refs = {signal["params"]["later_ref"] for signal in enriched["evidence"]["signals"]}
assert {"MAINT-DQ-SCAN-B", "MAINT-DQ-SCAN-C"}.issubset(later_refs)
finally:
with SessionLocal() as db:
db.execute(delete(DataQualityIssue).where(DataQualityIssue.entity_id == vehicle_id))
db.execute(delete(MaintenanceRecord).where(MaintenanceRecord.vehicle_id == vehicle_id))
db.execute(delete(Vehicle).where(Vehicle.id == vehicle_id))
db.commit()
def test_scan_requires_operations_manager(employee_client): def test_scan_requires_operations_manager(employee_client):
response = employee_client.post("/api/v1/data-quality/scan") response = employee_client.post("/api/v1/data-quality/scan")
assert response.status_code == 403 assert response.status_code == 403
@@ -440,9 +559,8 @@ def test_resolve_odometer_regression_correction_below_canonical_is_rejected_then
def test_resolve_odometer_regression_correct_reading_updates_canonical(ops_client): def test_resolve_odometer_regression_correct_reading_updates_canonical(ops_client):
# The seeded odometer_regression issues carry no related booking (CSV-only rows). # Create a live return regression to prove the complete correction chain updates
# Create a fresh one with a real related booking via a live regression return, so # both persisted representations of that reading (booking + return inspection).
# the "correct_reading" path has an actual booking_ref to target.
booking_ref = _activate_booking("MO-018", start_odometer_km=12000) booking_ref = _activate_booking("MO-018", start_odometer_km=12000)
vehicle_before = ops_client.get("/api/v1/vehicles/MO-018").json() vehicle_before = ops_client.get("/api/v1/vehicles/MO-018").json()
low_reading = vehicle_before["odometer_km"] - 200 low_reading = vehicle_before["odometer_km"] - 200
@@ -459,6 +577,7 @@ def test_resolve_odometer_regression_correct_reading_updates_canonical(ops_clien
) )
assert returned.status_code == 201 assert returned.status_code == 201
issue_ref = returned.json()["quality_issue_ref"] issue_ref = returned.json()["quality_issue_ref"]
inspection_ref = returned.json()["inspection_ref"]
assert issue_ref is not None assert issue_ref is not None
corrected = vehicle_before["odometer_km"] + 500 corrected = vehicle_before["odometer_km"] + 500
@@ -477,6 +596,701 @@ def test_resolve_odometer_regression_correct_reading_updates_canonical(ops_clien
assert vehicle["odometer_km"] == corrected assert vehicle["odometer_km"] == corrected
booking = ops_client.get(f"/api/v1/bookings/{booking_ref}").json() booking = ops_client.get(f"/api/v1/bookings/{booking_ref}").json()
assert booking["end_odometer_km"] == corrected assert booking["end_odometer_km"] == corrected
with SessionLocal() as db:
inspection = db.scalar(select(Inspection).where(Inspection.public_ref == inspection_ref))
assert inspection is not None
assert inspection.odometer_km == corrected
def test_correcting_one_of_two_regressions_keeps_the_other_open(ops_client):
with SessionLocal() as db:
customer = Customer(
public_ref="CUS-DQ-MULTI",
first_name="Synthetic",
last_name="Multi",
email="dq-multi@example.test",
)
vehicle = Vehicle(
public_ref="MO-DQ-MULTI",
make="Synthetic",
model="Multi",
model_year=2026,
registration_number="DQ-MULTI",
location="Brussels",
operational_status="available",
odometer_km=1_000,
next_service_km=2_000,
active=True,
version=1,
)
db.add_all([customer, vehicle])
db.flush()
bookings: list[Booking] = []
inspections: list[Inspection] = []
for index, reading in enumerate((900, 800), start=1):
booking = Booking(
public_ref=f"BK-DQ-MULTI-{index}",
customer_id=customer.id,
vehicle_id=vehicle.id,
starts_at=datetime(2046, index, 1, tzinfo=UTC),
ends_at=datetime(2046, index, 2, tzinfo=UTC),
status="returned",
start_odometer_km=reading - 10,
end_odometer_km=reading,
requirements_complete=True,
)
db.add(booking)
db.flush()
inspection = Inspection(
public_ref=f"INSP-DQ-M-{index}",
booking_id=booking.id,
vehicle_id=vehicle.id,
type="return",
fuel_level_percent=50,
cleanliness_ok=True,
damage_reported=False,
technical_warning=False,
odometer_km=reading,
completed_at=booking.ends_at,
)
db.add(inspection)
db.flush()
open_odometer_regression_issue(
db,
vehicle=vehicle,
reading_ref=inspection.public_ref,
reading_km=reading,
canonical_km=vehicle.odometer_km,
source_type="return",
related_refs=[booking.public_ref, inspection.public_ref],
correctable_booking_refs=[booking.public_ref],
public_ref="DQ-MULTI-SOURCE" if index == 1 else None,
)
bookings.append(booking)
inspections.append(inspection)
booking_refs = [booking.public_ref for booking in bookings]
vehicle_id = vehicle.id
customer_id = customer.id
db.commit()
try:
first = ops_client.post(
"/api/v1/data-quality/issues/DQ-MULTI-SOURCE/resolve-odometer-regression",
json={
"decision": "correct_reading",
"booking_ref": booking_refs[0],
"corrected_odometer_km": 1_100,
},
)
assert first.status_code == 200, first.text
assert first.json()["status"] == "open"
assert first.json()["evidence"]["correctable_booking_refs"] == [booking_refs[1]]
second = ops_client.post(
"/api/v1/data-quality/issues/DQ-MULTI-SOURCE/resolve-odometer-regression",
json={
"decision": "correct_reading",
"booking_ref": booking_refs[1],
"corrected_odometer_km": 1_200,
},
)
assert second.status_code == 200, second.text
assert second.json()["status"] == "resolved"
with SessionLocal() as db:
persisted = db.scalars(
select(Inspection)
.where(Inspection.vehicle_id == vehicle_id)
.order_by(Inspection.public_ref)
).all()
assert [item.odometer_km for item in persisted] == [1_100, 1_200]
finally:
with SessionLocal() as db:
db.execute(delete(DataQualityIssue).where(DataQualityIssue.entity_id == vehicle_id))
db.execute(delete(Inspection).where(Inspection.vehicle_id == vehicle_id))
db.execute(delete(Booking).where(Booking.vehicle_id == vehicle_id))
db.execute(delete(Vehicle).where(Vehicle.id == vehicle_id))
db.execute(delete(Customer).where(Customer.id == customer_id))
db.commit()
def test_retained_regression_is_suppressed_until_source_reading_changes(ops_client):
with SessionLocal() as db:
vehicle = Vehicle(
public_ref="MO-DQ-RETAIN",
make="Synthetic",
model="Retain",
model_year=2026,
registration_number="DQ-RETAIN",
location="Brussels",
operational_status="available",
odometer_km=20_000,
next_service_km=30_000,
active=True,
version=1,
)
db.add(vehicle)
db.flush()
vehicle_id = vehicle.id
db.add_all(
[
MaintenanceRecord(
public_ref="MAINT-DQ-RET-A",
vehicle_id=vehicle.id,
occurred_at=datetime(2047, 1, 1, tzinfo=UTC),
odometer_km=20_000,
category="inspection",
summary="Retain baseline",
),
MaintenanceRecord(
public_ref="MAINT-DQ-RET-B",
vehicle_id=vehicle.id,
occurred_at=datetime(2047, 2, 1, tzinfo=UTC),
odometer_km=19_000,
category="inspection",
summary="Retained source reading",
),
]
)
db.commit()
try:
assert ops_client.post("/api/v1/data-quality/scan").status_code == 200
issue = next(
item
for item in ops_client.get(
"/api/v1/data-quality/issues",
params={"status": "open", "rule_type": "odometer_regression"},
).json()
if item["entity_ref"] == "MO-DQ-RETAIN"
)
retained = ops_client.post(
f"/api/v1/data-quality/issues/{issue['public_ref']}/resolve-odometer-regression",
json={"decision": "retain_canonical", "note": "Verified source entry"},
)
assert retained.status_code == 200
assert retained.json()["status"] == "resolved"
assert ops_client.post("/api/v1/data-quality/scan").status_code == 200
with SessionLocal() as db:
assert (
db.scalar(
select(DataQualityIssue).where(
DataQualityIssue.entity_id == vehicle_id,
DataQualityIssue.status == "open",
)
)
is None
)
changed = db.scalar(
select(MaintenanceRecord).where(MaintenanceRecord.public_ref == "MAINT-DQ-RET-B")
)
changed.odometer_km = 18_999
db.commit()
assert ops_client.post("/api/v1/data-quality/scan").status_code == 200
reopened = next(
item
for item in ops_client.get(
"/api/v1/data-quality/issues",
params={"status": "open", "rule_type": "odometer_regression"},
).json()
if item["entity_ref"] == "MO-DQ-RETAIN"
)
assert reopened["evidence"]["reopened_from"] == issue["public_ref"]
finally:
with SessionLocal() as db:
db.execute(delete(DataQualityIssue).where(DataQualityIssue.entity_id == vehicle_id))
db.execute(delete(MaintenanceRecord).where(MaintenanceRecord.vehicle_id == vehicle_id))
db.execute(delete(Vehicle).where(Vehicle.id == vehicle_id))
db.commit()
def test_live_return_retained_as_canonical_is_not_reopened_by_scan(ops_client):
now = datetime.now(UTC)
with SessionLocal() as db:
customer = Customer(
public_ref="CUS-DQ-LIVE-RET",
first_name="Synthetic",
last_name="Retained return",
email="dq-live-retain@example.test",
)
vehicle = Vehicle(
public_ref="MO-DQ-LIVE-RET",
make="Synthetic",
model="Retained return",
model_year=2026,
registration_number="DQ-LIVE-RET",
location="Brussels",
operational_status="rented",
odometer_km=20_000,
next_service_km=30_000,
active=True,
version=1,
)
db.add_all([customer, vehicle])
db.flush()
booking = Booking(
public_ref="BK-DQ-LIVE-RET",
customer_id=customer.id,
vehicle_id=vehicle.id,
starts_at=now - timedelta(days=2),
ends_at=now + timedelta(days=1),
status="active",
start_odometer_km=19_500,
end_odometer_km=None,
requirements_complete=True,
)
db.add_all(
[
booking,
MaintenanceRecord(
public_ref="MNT-DQ-LIVE-RET",
vehicle_id=vehicle.id,
occurred_at=now - timedelta(days=3),
odometer_km=20_000,
category="inspection",
summary="Synthetic canonical baseline",
),
]
)
vehicle_id = vehicle.id
customer_id = customer.id
db.commit()
try:
returned = ops_client.post(
"/api/v1/bookings/BK-DQ-LIVE-RET/return",
json={
"end_odometer_km": 19_000,
"fuel_level_percent": 50,
"cleanliness_ok": True,
"damage_reported": False,
"technical_warning": False,
},
headers={"Idempotency-Key": "test-dq-live-retain-001"},
)
assert returned.status_code == 201, returned.text
issue_ref = returned.json()["quality_issue_ref"]
inspection_ref = returned.json()["inspection_ref"]
assert issue_ref is not None
retained = ops_client.post(
f"/api/v1/data-quality/issues/{issue_ref}/resolve-odometer-regression",
json={"decision": "retain_canonical", "note": "Source reading verified as wrong."},
)
assert retained.status_code == 200, retained.text
assert retained.json()["status"] == "resolved"
assert retained.json()["evidence"]["retained_odometer_fingerprints"] == [
{
"source_type": "return",
"later_ref": inspection_ref,
"later_km": 19_000,
}
]
scanned = ops_client.post("/api/v1/data-quality/scan")
assert scanned.status_code == 200, scanned.text
with SessionLocal() as db:
issues = db.scalars(
select(DataQualityIssue)
.where(
DataQualityIssue.entity_id == vehicle_id,
DataQualityIssue.rule_type == "odometer_regression",
)
.order_by(DataQualityIssue.detected_at)
).all()
assert [(issue.public_ref, issue.status) for issue in issues] == [
(issue_ref, "resolved")
]
finally:
_cleanup_odometer_scenario(vehicle_id, customer_id)
def test_scanned_return_regression_can_correct_booking_all_inspections_and_vehicle(ops_client):
now = datetime.now(UTC)
with SessionLocal() as db:
customer = Customer(
public_ref="CUS-DQ-SCAN-RET",
first_name="Synthetic",
last_name="Scanned return",
email="dq-scan-return@example.test",
)
vehicle = Vehicle(
public_ref="MO-DQ-SCAN-RET",
make="Synthetic",
model="Scanned return",
model_year=2026,
registration_number="DQ-SCAN-RET",
location="Brussels",
operational_status="available",
odometer_km=20_000,
next_service_km=30_000,
active=True,
version=1,
)
db.add_all([customer, vehicle])
db.flush()
booking = Booking(
public_ref="BK-DQ-SCAN-RET",
customer_id=customer.id,
vehicle_id=vehicle.id,
starts_at=now - timedelta(days=3),
ends_at=now - timedelta(hours=12),
status="returned",
start_odometer_km=19_500,
end_odometer_km=18_500,
requirements_complete=True,
)
db.add(booking)
db.flush()
db.add_all(
[
MaintenanceRecord(
public_ref="MNT-DQ-SCAN-RET",
vehicle_id=vehicle.id,
occurred_at=now - timedelta(days=4),
odometer_km=20_000,
category="inspection",
summary="Synthetic canonical baseline",
),
Inspection(
public_ref="INSP-DQ-SCAN-R1",
booking_id=booking.id,
vehicle_id=vehicle.id,
type="return",
fuel_level_percent=50,
cleanliness_ok=True,
damage_reported=False,
technical_warning=False,
odometer_km=19_000,
completed_at=now - timedelta(days=1),
),
Inspection(
public_ref="INSP-DQ-SCAN-R2",
booking_id=booking.id,
vehicle_id=vehicle.id,
type="return",
fuel_level_percent=50,
cleanliness_ok=True,
damage_reported=False,
technical_warning=False,
odometer_km=18_500,
completed_at=now - timedelta(hours=12),
),
]
)
vehicle_id = vehicle.id
customer_id = customer.id
db.commit()
try:
scanned = ops_client.post("/api/v1/data-quality/scan")
assert scanned.status_code == 200, scanned.text
issue = next(
item
for item in ops_client.get(
"/api/v1/data-quality/issues",
params={"status": "open", "rule_type": "odometer_regression"},
).json()
if item["entity_ref"] == "MO-DQ-SCAN-RET"
)
assert issue["evidence"]["correctable_booking_refs"] == ["BK-DQ-SCAN-RET"]
later_refs = {signal["params"]["later_ref"] for signal in issue["evidence"]["signals"]}
assert later_refs == {"INSP-DQ-SCAN-R1", "INSP-DQ-SCAN-R2"}
corrected = ops_client.post(
f"/api/v1/data-quality/issues/{issue['public_ref']}/resolve-odometer-regression",
json={
"decision": "correct_reading",
"booking_ref": "BK-DQ-SCAN-RET",
"corrected_odometer_km": 20_500,
},
)
assert corrected.status_code == 200, corrected.text
assert corrected.json()["status"] == "resolved"
with SessionLocal() as db:
persisted_vehicle = db.get(Vehicle, vehicle_id)
persisted_booking = db.scalar(
select(Booking).where(Booking.public_ref == "BK-DQ-SCAN-RET")
)
assert persisted_booking is not None
persisted_inspections = db.scalars(
select(Inspection)
.where(Inspection.booking_id == persisted_booking.id, Inspection.type == "return")
.order_by(Inspection.public_ref)
).all()
assert persisted_vehicle is not None
assert persisted_vehicle.odometer_km == 20_500
assert persisted_booking.end_odometer_km == 20_500
assert [inspection.odometer_km for inspection in persisted_inspections] == [
20_500,
20_500,
]
finally:
_cleanup_odometer_scenario(vehicle_id, customer_id)
def test_resolver_and_concurrent_return_finish_without_deadlock_or_lost_evidence():
now = datetime.now(UTC)
with SessionLocal() as db:
customer = Customer(
public_ref="CUS-DQ-RACE",
first_name="Synthetic",
last_name="Race",
email="dq-race@example.test",
)
vehicle = Vehicle(
public_ref="MO-DQ-RACE",
make="Synthetic",
model="Race",
model_year=2026,
registration_number="DQ-RACE",
location="Brussels",
operational_status="rented",
odometer_km=20_000,
next_service_km=30_000,
active=True,
version=1,
)
db.add_all([customer, vehicle])
db.flush()
corrected_booking = Booking(
public_ref="BK-DQ-RACE-OLD",
customer_id=customer.id,
vehicle_id=vehicle.id,
starts_at=now - timedelta(days=4),
ends_at=now - timedelta(days=3),
status="returned",
start_odometer_km=19_500,
end_odometer_km=19_000,
requirements_complete=True,
)
concurrent_booking = Booking(
public_ref="BK-DQ-RACE-NEW",
customer_id=customer.id,
vehicle_id=vehicle.id,
starts_at=now - timedelta(days=1),
ends_at=now + timedelta(days=1),
status="active",
start_odometer_km=19_000,
end_odometer_km=None,
requirements_complete=True,
)
db.add_all([corrected_booking, concurrent_booking])
db.flush()
old_inspection = Inspection(
public_ref="INSP-DQ-RACE-OLD",
booking_id=corrected_booking.id,
vehicle_id=vehicle.id,
type="return",
fuel_level_percent=50,
cleanliness_ok=True,
damage_reported=False,
technical_warning=False,
odometer_km=19_000,
completed_at=corrected_booking.ends_at,
)
db.add(old_inspection)
db.flush()
issue = open_odometer_regression_issue(
db,
vehicle=vehicle,
reading_ref=old_inspection.public_ref,
reading_km=19_000,
canonical_km=20_000,
source_type="return",
related_refs=[corrected_booking.public_ref, old_inspection.public_ref],
correctable_booking_refs=[corrected_booking.public_ref],
public_ref="DQ-ODO-RACE",
)
assert issue is not None
vehicle_id = vehicle.id
customer_id = customer.id
db.commit()
actor = CurrentUser(
public_ref="USR-DQ-RACE",
display_name="DQ Race Manager",
role="operations_manager",
)
start = Barrier(2)
def correct_existing_reading() -> str:
with SessionLocal() as db:
db.execute(text("SET LOCAL lock_timeout = '5s'"))
start.wait(timeout=5)
resolved = resolve_odometer_regression(
db,
"DQ-ODO-RACE",
ResolveOdometerRegressionRequest(
decision="correct_reading",
booking_ref="BK-DQ-RACE-OLD",
corrected_odometer_km=20_500,
),
actor,
)
return resolved.status
def return_other_booking() -> dict:
with SessionLocal() as db:
db.execute(text("SET LOCAL lock_timeout = '5s'"))
start.wait(timeout=5)
_status, response = register_vehicle_return(
db,
"BK-DQ-RACE-NEW",
RegisterReturnRequest(
end_odometer_km=18_000,
fuel_level_percent=50,
cleanliness_ok=True,
damage_reported=False,
technical_warning=False,
),
"test-dq-resolve-return-race-001",
actor,
)
return response
try:
with ThreadPoolExecutor(max_workers=2) as executor:
correction_future = executor.submit(correct_existing_reading)
return_future = executor.submit(return_other_booking)
correction_status = correction_future.result(timeout=15)
return_result = return_future.result(timeout=15)
assert correction_status in {"open", "resolved"}
assert return_result["quality_issue_ref"] is not None
with SessionLocal() as db:
persisted_vehicle = db.get(Vehicle, vehicle_id)
bookings = {
booking.public_ref: booking
for booking in db.scalars(
select(Booking).where(Booking.vehicle_id == vehicle_id)
).all()
}
old_inspections = db.scalars(
select(Inspection).where(Inspection.booking_id == bookings["BK-DQ-RACE-OLD"].id)
).all()
open_issues = db.scalars(
select(DataQualityIssue).where(
DataQualityIssue.entity_id == vehicle_id,
DataQualityIssue.rule_type == "odometer_regression",
DataQualityIssue.status == "open",
)
).all()
assert persisted_vehicle is not None
assert persisted_vehicle.odometer_km == 20_500
assert bookings["BK-DQ-RACE-OLD"].end_odometer_km == 20_500
assert [inspection.odometer_km for inspection in old_inspections] == [20_500]
assert bookings["BK-DQ-RACE-NEW"].status == "returned"
assert bookings["BK-DQ-RACE-NEW"].end_odometer_km == 18_000
assert len(open_issues) == 1
assert open_issues[0].evidence_json["correctable_booking_refs"] == ["BK-DQ-RACE-NEW"]
later_refs = {
signal["params"]["later_ref"] for signal in open_issues[0].evidence_json["signals"]
}
assert later_refs == {return_result["inspection_ref"]}
finally:
_cleanup_odometer_scenario(vehicle_id, customer_id)
def test_early_return_uses_inspection_time_without_duplicate_booking_regression(ops_client):
with SessionLocal() as db:
customer = Customer(
public_ref="CUS-DQ-EARLY",
first_name="Synthetic",
last_name="Early",
email="dq-early@example.test",
)
vehicle = Vehicle(
public_ref="MO-DQ-EARLY",
make="Synthetic",
model="Early",
model_year=2026,
registration_number="DQ-EARLY",
location="Brussels",
operational_status="available",
odometer_km=200,
next_service_km=10_000,
active=True,
version=1,
)
db.add_all([customer, vehicle])
db.flush()
early = Booking(
public_ref="BK-DQ-EARLY-A",
customer_id=customer.id,
vehicle_id=vehicle.id,
starts_at=datetime(2048, 1, 1, tzinfo=UTC),
ends_at=datetime(2048, 3, 1, tzinfo=UTC),
status="returned",
start_odometer_km=90,
end_odometer_km=100,
requirements_complete=True,
)
later = Booking(
public_ref="BK-DQ-EARLY-B",
customer_id=customer.id,
vehicle_id=vehicle.id,
starts_at=datetime(2048, 1, 10, tzinfo=UTC),
ends_at=datetime(2048, 2, 1, tzinfo=UTC),
status="returned",
start_odometer_km=100,
end_odometer_km=200,
requirements_complete=True,
)
db.add_all([early, later])
db.flush()
db.add_all(
[
Inspection(
public_ref="INSP-DQ-EARLY-A",
booking_id=early.id,
vehicle_id=vehicle.id,
type="return",
fuel_level_percent=50,
cleanliness_ok=True,
damage_reported=False,
technical_warning=False,
odometer_km=100,
completed_at=datetime(2048, 1, 2, tzinfo=UTC),
),
Inspection(
public_ref="INSP-DQ-EARLY-B",
booking_id=later.id,
vehicle_id=vehicle.id,
type="return",
fuel_level_percent=50,
cleanliness_ok=True,
damage_reported=False,
technical_warning=False,
odometer_km=200,
completed_at=datetime(2048, 2, 1, tzinfo=UTC),
),
]
)
vehicle_id = vehicle.id
customer_id = customer.id
db.commit()
try:
assert ops_client.post("/api/v1/data-quality/scan").status_code == 200
with SessionLocal() as db:
assert (
db.scalar(
select(DataQualityIssue).where(
DataQualityIssue.entity_id == vehicle_id,
DataQualityIssue.rule_type == "odometer_regression",
)
)
is None
)
finally:
with SessionLocal() as db:
db.execute(delete(DataQualityIssue).where(DataQualityIssue.entity_id == vehicle_id))
db.execute(delete(Inspection).where(Inspection.vehicle_id == vehicle_id))
db.execute(delete(Booking).where(Booking.vehicle_id == vehicle_id))
db.execute(delete(Vehicle).where(Vehicle.id == vehicle_id))
db.execute(delete(Customer).where(Customer.id == customer_id))
db.commit()
def test_manual_scan_records_audit_event(ops_client): def test_manual_scan_records_audit_event(ops_client):
+104
View File
@@ -1,5 +1,10 @@
from sqlalchemy import select
from app.core.db import SessionLocal from app.core.db import SessionLocal
from app.models.outbox import OutboxEvent
from app.seed_loader import reset_and_seed from app.seed_loader import reset_and_seed
from app.services import demo_manifest
from app.services.knowledge import KnowledgeHealth
def test_demo_manifest_is_public(client): def test_demo_manifest_is_public(client):
@@ -52,3 +57,102 @@ def test_demo_manifest_ragcore_labelled_as_demo_mode_not_live(client):
body = client.get("/api/v1/demo/manifest").json() body = client.get("/api/v1/demo/manifest").json()
ragcore = next(i for i in body["integrations"] if i["key"] == "ragcore") ragcore = next(i for i in body["integrations"] if i["key"] == "ragcore")
assert ragcore["status_code"] == "demoMode" assert ragcore["status_code"] == "demoMode"
def test_automation_scenario_requires_the_exact_prepared_failure():
db = SessionLocal()
try:
reset_and_seed(db)
event = db.scalar(
select(OutboxEvent).where(OutboxEvent.event_id == demo_manifest._FAILED_DEMO_EVENT_ID)
)
assert event is not None
event.last_error_code = "connectionError"
db.flush()
scenario = next(
item for item in demo_manifest._scenarios(db) if item.id == "automation-retry"
)
assert scenario.ready is False
finally:
db.rollback()
db.close()
def test_knowledge_scenario_rejects_available_provider_with_verified_empty_corpus(
client, monkeypatch
):
class EmptyKnowledgeProvider:
def health(self, language="en-GB"):
return KnowledgeHealth(
provider="ragcore",
available=True,
detail="Ready, but no indexed documents.",
tenant="fleet-ops",
workspace="mobilityops",
collection="procedures",
document_count=0,
source_document_count=4,
reported_synced_document_count=None,
reported_failed_document_count=None,
last_sync_at=None,
statistics_state="verified",
)
monkeypatch.setattr(demo_manifest, "get_knowledge_provider", EmptyKnowledgeProvider)
body = client.get("/api/v1/demo/manifest").json()
scenario = next(item for item in body["scenarios"] if item["id"] == "knowledge-question")
assert scenario["ready"] is False
assert scenario["blocked_reason_code"] == "knowledgeUnavailable"
ragcore = next(item for item in body["integrations"] if item["key"] == "ragcore")
assert ragcore["status_code"] == "unavailable"
def test_knowledge_scenario_rejects_local_sources_when_index_count_is_unverified(
client, monkeypatch
):
class SourceAwareKnowledgeProvider:
def health(self, language="en-GB"):
return KnowledgeHealth(
provider="ragcore",
available=True,
detail="Ready; index count endpoint is unavailable.",
tenant="fleet-ops",
workspace="mobilityops",
collection="procedures",
document_count=None,
source_document_count=4,
reported_synced_document_count=None,
reported_failed_document_count=None,
last_sync_at=None,
statistics_state="not_reported",
)
monkeypatch.setattr(demo_manifest, "get_knowledge_provider", SourceAwareKnowledgeProvider)
body = client.get("/api/v1/demo/manifest").json()
scenario = next(item for item in body["scenarios"] if item["id"] == "knowledge-question")
assert scenario["ready"] is False
assert scenario["blocked_reason_code"] == "knowledgeUnavailable"
ragcore = next(item for item in body["integrations"] if item["key"] == "ragcore")
assert ragcore["status_code"] == "unavailable"
def test_knowledge_scenario_requires_provider_availability_even_with_documents():
health = KnowledgeHealth(
provider="ragcore",
available=False,
detail="Provider is unreachable.",
tenant="fleet-ops",
workspace="mobilityops",
collection="procedures",
document_count=4,
source_document_count=4,
reported_synced_document_count=None,
reported_failed_document_count=None,
last_sync_at=None,
statistics_state="verified",
)
assert demo_manifest._knowledge_scenario_ready(health) is False
+120 -5
View File
@@ -4,6 +4,7 @@ import uuid
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime, timedelta
from types import SimpleNamespace from types import SimpleNamespace
import pytest
from sqlalchemy import select from sqlalchemy import select
from app.core.config import get_settings from app.core.config import get_settings
@@ -65,12 +66,17 @@ def test_claim_marks_events_delivering():
def test_deliver_one_success(monkeypatch): def test_deliver_one_success(monkeypatch):
event_id = _make_pending_event("MO-002") event_id = _make_pending_event("MO-002")
execution_id = "n8n-execution-123"
dispatcher._claim_due_events() dispatcher._claim_due_events()
def fake_post(url, json, headers, timeout): def fake_post(url, json, headers, timeout):
return SimpleNamespace( return SimpleNamespace(
raise_for_status=lambda: None, raise_for_status=lambda: None,
json=lambda: {"ok": True, "event_id": str(event_id), "result": {}}, json=lambda: {
"ok": True,
"event_id": str(event_id),
"result": {"execution_id": execution_id},
},
) )
monkeypatch.setattr(dispatcher.httpx, "post", fake_post) monkeypatch.setattr(dispatcher.httpx, "post", fake_post)
@@ -79,7 +85,7 @@ def test_deliver_one_success(monkeypatch):
event = _get_event(event_id) event = _get_event(event_id)
assert event.delivery_status == "succeeded" assert event.delivery_status == "succeeded"
assert event.attempts == 1 assert event.attempts == 1
assert event.external_run_id == str(event_id) assert event.external_run_id == execution_id
assert event.last_error is None assert event.last_error is None
assert event.last_error_code is None assert event.last_error_code is None
@@ -129,6 +135,106 @@ def test_deliver_one_treats_empty_2xx_body_as_failure(monkeypatch):
assert event.last_error_code == "malformedResponse" assert event.last_error_code == "malformedResponse"
def test_deliver_one_rejects_json_object_without_explicit_success_ack(monkeypatch):
event_id = _make_pending_event("MO-010")
dispatcher._claim_due_events()
def fake_post(url, json, headers, timeout):
return SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {},
status_code=200,
)
monkeypatch.setattr(dispatcher.httpx, "post", fake_post)
dispatcher._deliver_one(event_id)
event = _get_event(event_id)
assert event.delivery_status == "pending"
assert event.attempts == 1
assert event.last_error_code == "malformedResponse"
assert event.external_run_id is None
@pytest.mark.parametrize("result", [{}, {"execution_id": 123}, {"execution_id": " "}])
def test_deliver_one_rejects_ack_without_valid_execution_id(monkeypatch, result):
event_id = _make_pending_event("MO-EXEC-ID")
dispatcher._claim_due_events()
def fake_post(url, json, headers, timeout):
return SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {"ok": True, "event_id": str(event_id), "result": result},
status_code=200,
)
monkeypatch.setattr(dispatcher.httpx, "post", fake_post)
dispatcher._deliver_one(event_id)
event = _get_event(event_id)
assert event.delivery_status == "pending"
assert event.last_error_code == "malformedResponse"
assert event.external_run_id is None
def test_deliver_one_rejects_success_ack_for_a_different_event(monkeypatch):
event_id = _make_pending_event("MO-011")
dispatcher._claim_due_events()
other_event_id = uuid.uuid4()
def fake_post(url, json, headers, timeout):
return SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {"ok": True, "event_id": str(other_event_id)},
status_code=200,
)
monkeypatch.setattr(dispatcher.httpx, "post", fake_post)
dispatcher._deliver_one(event_id)
event = _get_event(event_id)
assert event.delivery_status == "pending"
assert event.attempts == 1
assert event.last_error_code == "mismatchedEventId"
assert event.external_run_id is None
def test_late_delivery_outcome_cannot_overwrite_a_newer_lease(monkeypatch):
event_id = _make_pending_event("MO-LEASE-RACE")
dispatcher._claim_due_events()
def fake_post(url, json, headers, timeout):
# Simulate lease expiry/reclaim while the original worker is still in network
# I/O. The later claimant owns a different token and its state must win.
with SessionLocal() as db:
event = db.scalar(
select(OutboxEvent).where(OutboxEvent.event_id == event_id).with_for_update()
)
event.payload_json = {
**event.payload_json,
"_delivery_claim_token": "newer-worker-token",
}
event.next_attempt_at = datetime.now(UTC) + timedelta(minutes=1)
db.commit()
return SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {
"ok": True,
"event_id": str(event_id),
"result": {"execution_id": "stale-worker-execution"},
},
status_code=200,
)
monkeypatch.setattr(dispatcher.httpx, "post", fake_post)
dispatcher._deliver_one(event_id)
event = _get_event(event_id)
assert event.delivery_status == "delivering"
assert event.attempts == 0
assert event.external_run_id is None
assert event.payload_json["_delivery_claim_token"] == "newer-worker-token"
def test_deliver_one_exhausts_attempts_to_failed(monkeypatch): def test_deliver_one_exhausts_attempts_to_failed(monkeypatch):
event_id = _make_pending_event("MO-004") event_id = _make_pending_event("MO-004")
settings = get_settings() settings = get_settings()
@@ -139,7 +245,6 @@ def test_deliver_one_exhausts_attempts_to_failed(monkeypatch):
monkeypatch.setattr(dispatcher.httpx, "post", fake_post) monkeypatch.setattr(dispatcher.httpx, "post", fake_post)
for _ in range(settings.n8n_max_attempts): for _ in range(settings.n8n_max_attempts):
dispatcher._claim_due_events()
db = SessionLocal() db = SessionLocal()
try: try:
event = db.scalar(select(OutboxEvent).where(OutboxEvent.event_id == event_id)) event = db.scalar(select(OutboxEvent).where(OutboxEvent.event_id == event_id))
@@ -147,6 +252,8 @@ def test_deliver_one_exhausts_attempts_to_failed(monkeypatch):
db.commit() db.commit()
finally: finally:
db.close() db.close()
claimed = dispatcher._claim_due_events(batch_size=1000)
assert event_id in claimed
dispatcher._deliver_one(event_id) dispatcher._deliver_one(event_id)
event = _get_event(event_id) event = _get_event(event_id)
@@ -259,7 +366,11 @@ def test_run_dispatch_cycle_recovers_a_stale_lease_before_claiming(monkeypatch):
def fake_post(url, json, headers, timeout): def fake_post(url, json, headers, timeout):
return SimpleNamespace( return SimpleNamespace(
raise_for_status=lambda: None, raise_for_status=lambda: None,
json=lambda: {"ok": True, "event_id": str(event_id), "result": {}}, json=lambda: {
"ok": True,
"event_id": str(event_id),
"result": {"execution_id": "n8n-recovered-execution"},
},
) )
monkeypatch.setattr(dispatcher.httpx, "post", fake_post) monkeypatch.setattr(dispatcher.httpx, "post", fake_post)
@@ -275,7 +386,11 @@ def test_run_dispatch_cycle_end_to_end(monkeypatch):
def fake_post(url, json, headers, timeout): def fake_post(url, json, headers, timeout):
return SimpleNamespace( return SimpleNamespace(
raise_for_status=lambda: None, raise_for_status=lambda: None,
json=lambda: {"ok": True, "event_id": str(event_id), "result": {}}, json=lambda: {
"ok": True,
"event_id": str(event_id),
"result": {"execution_id": "n8n-cycle-execution"},
},
) )
monkeypatch.setattr(dispatcher.httpx, "post", fake_post) monkeypatch.setattr(dispatcher.httpx, "post", fake_post)
+8 -3
View File
@@ -174,11 +174,12 @@ def test_return_idempotency_key_rejects_different_body(ops_client):
def test_return_callback_rejects_malformed_correlation_id(client): def test_return_callback_rejects_malformed_correlation_id(client):
event_id = str(uuid.uuid4())
response = client.post( response = client.post(
"/api/v1/integrations/n8n/return-callback", "/api/v1/integrations/n8n/return-callback",
json={"follow_up": "cleaning", "correlation_id": "nope"}, json={"event_id": event_id, "follow_up": "cleaning", "correlation_id": "nope"},
headers={ headers={
"Idempotency-Key": str(uuid.uuid4()), "Idempotency-Key": event_id,
"X-Service-Token": get_settings().n8n_callback_token, "X-Service-Token": get_settings().n8n_callback_token,
}, },
) )
@@ -195,10 +196,14 @@ def test_blocked_checkout_booking_can_be_cancelled(ops_client):
json={ json={
"customer_ref": "CUS-0002", "customer_ref": "CUS-0002",
"vehicle_ref": vehicle["public_ref"], "vehicle_ref": vehicle["public_ref"],
"requirements_complete": True,
**window, **window,
}, },
).json() ).json()
confirmed = ops_client.post(
f"/api/v1/bookings/{booking['public_ref']}/complete-requirements",
json={"confirmation": "Licence and rental conditions checked"},
)
assert confirmed.status_code == 200
checkout = ops_client.post( checkout = ops_client.post(
f"/api/v1/bookings/{booking['public_ref']}/checkout", f"/api/v1/bookings/{booking['public_ref']}/checkout",
json={ json={
+102 -1
View File
@@ -1,8 +1,14 @@
from sqlalchemy import select import uuid
from datetime import UTC, datetime
import pytest
from sqlalchemy import delete, select
from app.core.db import SessionLocal from app.core.db import SessionLocal
from app.models.audit import AuditEvent
from app.models.outbox import OutboxEvent from app.models.outbox import OutboxEvent
from app.seed_loader import reset_and_seed from app.seed_loader import reset_and_seed
from app.services import integration_status
def _reseed() -> None: def _reseed() -> None:
@@ -108,6 +114,101 @@ def test_integration_status_is_operational_once_all_failed_events_resolved(ops_c
assert body["state"] == "operational" assert body["state"] == "operational"
@pytest.mark.parametrize("delivery_status", ["pending", "delivering"])
def test_queued_work_without_any_success_is_not_reported_operational(delivery_status):
_reseed()
db = SessionLocal()
try:
for event in db.scalars(select(OutboxEvent)).all():
event.delivery_status = delivery_status
event.last_error = None
event.last_error_code = None
db.execute(
delete(AuditEvent).where(
AuditEvent.action.in_(
{
"data_quality_scan_run",
"n8n_procedures_synced",
"n8n_workflow_failure_registered",
"n8n_workflow_heartbeat",
}
)
)
)
db.flush()
status = integration_status.derive_n8n_status(db)
assert status.succeeded == 0
assert getattr(status, delivery_status) > 0
assert status.state == "no_evidence"
finally:
db.rollback()
db.close()
def test_historical_success_is_not_green_when_webhook_is_unconfigured(monkeypatch):
_reseed()
with SessionLocal() as db:
monkeypatch.setattr(integration_status.settings, "n8n_dispatch_enabled", True)
monkeypatch.setattr(integration_status.settings, "n8n_webhook_url", "")
status = integration_status.derive_n8n_status(db)
assert status.configured is False
assert status.succeeded > 0
assert status.state == "unavailable"
def test_null_coded_real_failure_sets_latest_failure_timestamp():
_reseed()
with SessionLocal() as db:
event = db.scalar(
select(OutboxEvent).where(OutboxEvent.delivery_status == "succeeded").limit(1)
)
event.delivery_status = "failed"
event.last_error_code = None
db.flush()
status = integration_status.derive_n8n_status(db)
assert status.unexpected_failed == 1
assert status.latest_failure_at is not None
db.rollback()
def test_unreachable_hub_invalidates_historical_operational_claim(monkeypatch):
db = SessionLocal()
try:
db.add(
AuditEvent(
actor_type="service",
actor_id=None,
actor_label="itworx-mcp-hub",
action="mcp_tool_request",
entity_type="integration",
entity_id=None,
correlation_id=uuid.uuid4(),
before_json=None,
after_json=None,
metadata_json={"tool": "operations_summary"},
occurred_at=datetime.now(UTC),
)
)
db.flush()
monkeypatch.setattr(integration_status.settings, "mcp_hub_registration_enabled", True)
monkeypatch.setattr(integration_status, "_check_hub_reachable", lambda: False)
unreachable = integration_status.derive_mcp_hub_status(db)
assert unreachable.total_calls > 0
assert unreachable.hub_reachable is False
assert unreachable.state == "no_evidence"
monkeypatch.setattr(integration_status, "_check_hub_reachable", lambda: True)
reachable = integration_status.derive_mcp_hub_status(db)
assert reachable.state == "operational"
finally:
db.rollback()
db.close()
def test_integration_status_lists_all_four_canonical_workflows(ops_client): def test_integration_status_lists_all_four_canonical_workflows(ops_client):
body = ops_client.get("/api/v1/integrations/status").json()["n8n"] body = ops_client.get("/api/v1/integrations/status").json()["n8n"]
assert body["expected_workflow_count"] == 4 assert body["expected_workflow_count"] == 4
+158 -7
View File
@@ -1,4 +1,5 @@
import uuid import uuid
from concurrent.futures import ThreadPoolExecutor
from app.core.config import get_settings from app.core.config import get_settings
@@ -12,19 +13,29 @@ def _callback_headers(event_id: str, token: str | None = None):
def test_callback_rejects_wrong_service_token(client): def test_callback_rejects_wrong_service_token(client):
event_id = str(uuid.uuid4())
response = client.post( response = client.post(
"/api/v1/integrations/n8n/return-callback", "/api/v1/integrations/n8n/return-callback",
json={"follow_up": "cleaning"}, json={
headers=_callback_headers(str(uuid.uuid4()), token="wrong-token"), "event_id": event_id,
"correlation_id": str(uuid.uuid4()),
"follow_up": "cleaning",
},
headers=_callback_headers(event_id, token="wrong-token"),
) )
assert response.status_code == 401 assert response.status_code == 401
def test_callback_unknown_event_returns_404(client): def test_callback_unknown_event_returns_404(client):
event_id = str(uuid.uuid4())
response = client.post( response = client.post(
"/api/v1/integrations/n8n/return-callback", "/api/v1/integrations/n8n/return-callback",
json={"follow_up": "cleaning"}, json={
headers=_callback_headers(str(uuid.uuid4())), "event_id": event_id,
"correlation_id": str(uuid.uuid4()),
"follow_up": "cleaning",
},
headers=_callback_headers(event_id),
) )
assert response.status_code == 404 assert response.status_code == 404
@@ -39,13 +50,14 @@ def test_callback_is_idempotent_by_event_id(client, ops_client):
db = SessionLocal() db = SessionLocal()
try: try:
booking = db.scalar(select(Booking).limit(1)) booking = db.scalar(select(Booking).limit(1))
correlation_id = uuid.uuid4()
event = OutboxEvent( event = OutboxEvent(
event_id=uuid.uuid4(), event_id=uuid.uuid4(),
event_type="vehicle.returned.v1", event_type="vehicle.returned.v1",
aggregate_type="booking", aggregate_type="booking",
aggregate_id=booking.id, aggregate_id=booking.id,
payload_json={ payload_json={
"correlation_id": str(uuid.uuid4()), "correlation_id": str(correlation_id),
"aggregate": { "aggregate": {
"type": "booking", "type": "booking",
"id": str(booking.id), "id": str(booking.id),
@@ -66,12 +78,22 @@ def test_callback_is_idempotent_by_event_id(client, ops_client):
first = client.post( first = client.post(
"/api/v1/integrations/n8n/return-callback", "/api/v1/integrations/n8n/return-callback",
json={"follow_up": "cleaning", "summary": "test"}, json={
"event_id": event_id,
"correlation_id": str(correlation_id),
"follow_up": "cleaning",
"summary": "test",
},
headers=_callback_headers(event_id), headers=_callback_headers(event_id),
) )
second = client.post( second = client.post(
"/api/v1/integrations/n8n/return-callback", "/api/v1/integrations/n8n/return-callback",
json={"follow_up": "cleaning", "summary": "test"}, json={
"event_id": event_id,
"correlation_id": str(correlation_id),
"follow_up": "cleaning",
"summary": "test",
},
headers=_callback_headers(event_id), headers=_callback_headers(event_id),
) )
assert first.status_code == 200 assert first.status_code == 200
@@ -82,6 +104,79 @@ def test_callback_is_idempotent_by_event_id(client, ops_client):
).json() ).json()
matching = [e for e in audit_events if e["metadata"]["event_id"] == event_id] matching = [e for e in audit_events if e["metadata"]["event_id"] == event_id]
assert len(matching) == 1 assert len(matching) == 1
assert matching[0]["correlation_id"] == str(correlation_id)
def test_callback_rejects_crossed_event_or_correlation(client):
from sqlalchemy import select
from app.core.db import SessionLocal
from app.models.outbox import OutboxEvent
with SessionLocal() as db:
event = db.scalar(select(OutboxEvent).limit(1))
event_id = str(event.event_id)
correlation_id = event.payload_json["correlation_id"]
crossed_event = client.post(
"/api/v1/integrations/n8n/return-callback",
json={
"event_id": str(uuid.uuid4()),
"correlation_id": correlation_id,
"follow_up": "cleaning",
},
headers=_callback_headers(event_id),
)
assert crossed_event.status_code == 409
assert crossed_event.json()["error"]["code"] == "CALLBACK_EVENT_MISMATCH"
crossed_correlation = client.post(
"/api/v1/integrations/n8n/return-callback",
json={
"event_id": event_id,
"correlation_id": str(uuid.uuid4()),
"follow_up": "cleaning",
},
headers=_callback_headers(event_id),
)
assert crossed_correlation.status_code == 409
assert crossed_correlation.json()["error"]["code"] == "CALLBACK_CORRELATION_MISMATCH"
def test_callback_concurrent_retries_record_one_audit(client, ops_client):
from sqlalchemy import select
from app.core.db import SessionLocal
from app.models.outbox import OutboxEvent
with SessionLocal() as db:
event = db.scalar(select(OutboxEvent).limit(1))
event_id = str(event.event_id)
correlation_id = event.payload_json["correlation_id"]
body = {
"event_id": event_id,
"correlation_id": correlation_id,
"follow_up": "cleaning",
}
def post_callback(_index: int):
return client.post(
"/api/v1/integrations/n8n/return-callback",
json=body,
headers=_callback_headers(event_id),
)
with ThreadPoolExecutor(max_workers=2) as pool:
responses = list(pool.map(post_callback, range(2)))
assert [response.status_code for response in responses] == [200, 200]
matching = [
event
for event in ops_client.get(
"/api/v1/audit", params={"action": "n8n_return_followup_recorded"}
).json()
if event["metadata"]["event_id"] == event_id
]
assert len(matching) == 1
def test_scheduled_scan_rejects_wrong_service_token(client): def test_scheduled_scan_rejects_wrong_service_token(client):
@@ -183,6 +278,42 @@ def test_workflow_error_registers_and_is_idempotent_by_execution_id(client, ops_
assert matching[0]["after"]["retry_action"] == "n8n will retry automatically" assert matching[0]["after"]["retry_action"] == "n8n will retry automatically"
def test_workflow_error_preserves_correlation_and_rejects_malformed(client, ops_client):
settings = get_settings()
headers = {"X-Service-Token": settings.n8n_callback_token}
execution_id = str(uuid.uuid4())
correlation_id = str(uuid.uuid4())
accepted = client.post(
"/api/v1/integrations/n8n/workflow-error",
json=_workflow_error_body(execution_id, correlation_id=correlation_id),
headers=headers,
)
assert accepted.status_code == 200
matching = [
event
for event in ops_client.get(
"/api/v1/audit", params={"action": "n8n_workflow_failure_registered"}
).json()
if event["metadata"]["execution_id"] == execution_id
]
assert len(matching) == 1
assert matching[0]["correlation_id"] == correlation_id
malformed_execution = str(uuid.uuid4())
malformed = client.post(
"/api/v1/integrations/n8n/workflow-error",
json=_workflow_error_body(malformed_execution, correlation_id="not-a-uuid"),
headers=headers,
)
assert malformed.status_code == 422
assert all(
event["metadata"]["execution_id"] != malformed_execution
for event in ops_client.get(
"/api/v1/audit", params={"action": "n8n_workflow_failure_registered"}
).json()
)
def test_workflow_error_bounds_summary_length(client): def test_workflow_error_bounds_summary_length(client):
settings = get_settings() settings = get_settings()
response = client.post( response = client.post(
@@ -266,6 +397,26 @@ def test_procedures_sync_result_registers_and_is_idempotent(client, ops_client):
assert ragcore_sync["last_seen_at"] is not None assert ragcore_sync["last_seen_at"] is not None
def test_failed_or_partial_procedure_sync_is_never_reported_healthy(client, ops_client):
settings = get_settings()
headers = {"X-Service-Token": settings.n8n_callback_token}
for synced, failed in ((0, 33), (32, 1)):
execution_id = str(uuid.uuid4())
response = client.post(
"/api/v1/integrations/n8n/procedures-sync-result",
json={"execution_id": execution_id, "synced": synced, "failed": failed},
headers=headers,
)
assert response.status_code == 200
status = ops_client.get("/api/v1/integrations/status").json()["n8n"]
workflow = next(w for w in status["workflows"] if "RAGcore" in w["name"])
assert workflow["state"] == "failed"
assert workflow["last_status"] == "failed"
assert workflow["last_execution_id"] == execution_id
assert status["state"] == "degraded"
def test_workflow_heartbeat_is_idempotent_and_drives_live_status(client, ops_client): def test_workflow_heartbeat_is_idempotent_and_drives_live_status(client, ops_client):
settings = get_settings() settings = get_settings()
execution_id = str(uuid.uuid4()) execution_id = str(uuid.uuid4())
+410 -71
View File
@@ -1,9 +1,12 @@
from __future__ import annotations from __future__ import annotations
import hashlib
import re import re
import uuid
from pathlib import Path from pathlib import Path
import httpx import httpx
import pytest
from app.api.routers import knowledge as knowledge_router from app.api.routers import knowledge as knowledge_router
from app.core.config import get_settings from app.core.config import get_settings
@@ -244,11 +247,11 @@ def test_ragcore_status_preserves_stronger_verified_index_evidence(client, ops_c
class _FakeResponse: class _FakeResponse:
def __init__(self, status_code: int, body: dict): def __init__(self, status_code: int, body: object):
self.status_code = status_code self.status_code = status_code
self._body = body self._body = body
def json(self) -> dict: def json(self) -> object:
return self._body return self._body
@@ -260,6 +263,7 @@ class _FakeClient:
post_responses=None, post_responses=None,
raise_on=None, raise_on=None,
get_handler=None, get_handler=None,
requests=None,
): ):
self._get_response = get_response self._get_response = get_response
self._post_response = post_response self._post_response = post_response
@@ -270,6 +274,7 @@ class _FakeClient:
self._post_responses = post_responses or {} self._post_responses = post_responses or {}
self._raise_on = raise_on self._raise_on = raise_on
self._get_handler = get_handler self._get_handler = get_handler
self.requests = requests if requests is not None else []
def __enter__(self): def __enter__(self):
return self return self
@@ -289,6 +294,7 @@ class _FakeClient:
def post(self, path, json=None): def post(self, path, json=None):
if self._raise_on == "post": if self._raise_on == "post":
raise httpx.ConnectError("no ragcore in this environment") raise httpx.ConnectError("no ragcore in this environment")
self.requests.append((path, json))
return self._post_responses.get(path, self._post_response) return self._post_responses.get(path, self._post_response)
@@ -393,22 +399,107 @@ def test_ragcore_provider_health_degrades_on_connection_error(monkeypatch):
assert "unavailable" in health.detail.lower() assert "unavailable" in health.detail.lower()
def _answers_body(**overrides) -> dict: _EXCERPTS = {
body = { ("en-GB", "damage-procedure"): (
"answer": "Report damage and route the vehicle to maintenance.", "When a vehicle returns with visible or reported damage, mark damage in the "
"answerability": "answerable", "return inspection, add a concise factual description and keep the vehicle "
"citations": [ "blocked."
{ ),
"id": "cite-1", ("en-GB", "vehicle-return-procedure"): (
"document_id": "doc-1", "Open the active booking and record the ending odometer, fuel level, cleanliness, "
"document_version_id": "version-1", "visible damage, technical warnings and relevant notes."
"title": "Damage handling procedure", ),
"section": "Detection", ("en-GB", "cleaning-checklist"): (
"excerpt": "Inspect the vehicle for visible damage.", "Cleaning completion alone does not make a blocked or maintenance vehicle "
"available. Fleet Ops derives availability from all active restrictions."
),
("en-GB", "maintenance-escalation"): (
"Escalate when a technical warning is reported, the service threshold is reached, "
"a safety-related defect is observed or an existing maintenance block remains "
"unresolved."
),
("nl-BE", "damage-procedure"): (
"Wanneer een voertuig terugkomt met zichtbare of gemelde schade, markeer de schade "
"in de retourinspectie, voeg een beknopte feitelijke beschrijving toe en houd het "
"voertuig geblokkeerd."
),
("nl-BE", "vehicle-return-procedure"): (
"Open de actieve boeking en registreer de eindkilometerstand, het brandstofniveau, "
"de netheid, zichtbare schade, technische waarschuwingen en relevante notities."
),
} }
],
def _managed_document(provider: RAGcoreKnowledgeProvider, document_id: str, language: str):
return next(
document
for document in provider._managed_documents_by_source_id.values()
if document.document_id == document_id and document.language == language
)
def _citation(
provider: RAGcoreKnowledgeProvider,
document_id: str,
*,
language: str = "en-GB",
section: str = "Procedure",
seed: str = "1",
) -> dict:
document = _managed_document(provider, document_id, language)
excerpt = _EXCERPTS[(language, document_id)]
return {
"id": str(uuid.uuid5(uuid.NAMESPACE_URL, f"citation:{language}:{document_id}:{seed}")),
# These are deliberately opaque RAGcore-owned UUIDs, not Fleet Ops' stable
# human-readable procedure id.
"document_id": str(
uuid.uuid5(uuid.NAMESPACE_URL, f"ragcore-document:{language}:{document_id}")
),
"document_version_id": str(
uuid.uuid5(
uuid.NAMESPACE_URL,
f"ragcore-document-version:{language}:{document_id}:{seed}",
)
),
"title": "untrusted-provider-title.md",
"section": section,
"excerpt": excerpt,
"excerpt_sha256": hashlib.sha256(excerpt.encode("utf-8")).hexdigest(),
"source_uri": f"ragcore://source/{document.source_id}",
"source_id": document.source_id,
"locator": f"{document.document_id}.md",
}
def _answers_body(provider: RAGcoreKnowledgeProvider, **overrides) -> dict:
citation = _citation(provider, "damage-procedure", section="Detection")
answer = "Report damage and route the vehicle to maintenance."
body = {
"answer_id": str(uuid.uuid4()),
"retrieval_run_id": str(uuid.uuid4()),
"answer": answer,
"answerability": "answerable",
"citations": [citation],
"claims": [{"text": answer, "citation_ids": [citation["id"]]}],
} }
body.update(overrides) body.update(overrides)
if "claims" not in overrides:
effective_answer = body.get("answer")
effective_citations = body.get("citations")
body["claims"] = (
[
{
"text": effective_answer,
"citation_ids": [item["id"] for item in effective_citations],
}
]
if isinstance(effective_answer, str)
and effective_answer
and isinstance(effective_citations, list)
and effective_citations
and all(isinstance(item, dict) and "id" in item for item in effective_citations)
else []
)
return body return body
@@ -418,47 +509,178 @@ def test_ragcore_provider_grounded_answer_maps_citations_to_sources(monkeypatch)
monkeypatch.setattr( monkeypatch.setattr(
provider, provider,
"_client", "_client",
lambda: _FakeClient(post_response=_FakeResponse(200, _answers_body())), lambda: _FakeClient(post_response=_FakeResponse(200, _answers_body(provider))),
) )
answer = provider.ask("What must I do about damage?", "test-correlation-grounded") answer = provider.ask("What must I do about damage?", "test-correlation-grounded")
assert answer.evidence_state == "grounded" assert answer.evidence_state == "grounded"
assert answer.answer assert answer.answer
assert len(answer.sources) == 1 assert len(answer.sources) == 1
source = answer.sources[0] source = answer.sources[0]
assert source.document_id == "doc-1" assert source.document_id == "damage-procedure"
assert source.title == "Damage handling procedure" assert source.title == "Damage handling procedure"
assert source.version == "version-1" assert source.version == "1.3"
assert source.section == "Detection" assert source.section == "Detection"
assert source.excerpt assert source.excerpt
@pytest.mark.parametrize(
"mutate",
[
lambda body: body.pop("claims"),
lambda body: body.update(claims=[]),
lambda body: body["claims"][0].update(citation_ids=[str(uuid.uuid4())]),
lambda body: body.pop("answer_id"),
lambda body: body.update(retrieval_run_id="not-a-uuid"),
],
)
def test_ragcore_answer_requires_valid_claim_citation_contract(monkeypatch, mutate):
provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
body = _answers_body(provider)
mutate(body)
monkeypatch.setattr(
provider,
"_client",
lambda: _FakeClient(post_response=_FakeResponse(200, body)),
)
answer = provider.ask("What must I do about damage?", "test-invalid-claim-contract")
assert answer.evidence_state != "grounded"
assert answer.answer == ""
assert answer.sources == []
def test_ragcore_requests_are_scoped_to_managed_sources_for_requested_language(
monkeypatch,
):
provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
requests = []
fake_client = _FakeClient(
post_responses={
"/v1/answers": _FakeResponse(503, {}),
"/v1/search": _FakeResponse(200, _search_body(provider, "nl-BE")),
},
requests=requests,
)
monkeypatch.setattr(provider, "_client", lambda: fake_client)
answer = provider.ask(
"Wat is de procedure voor een voertuigretour?",
"language-filter",
"nl-BE",
)
assert answer.evidence_state == "grounded"
assert [path for path, _payload in requests] == ["/v1/answers", "/v1/search"]
expected_source_ids = provider._managed_source_ids("nl-BE")
assert len(expected_source_ids) == 11
assert all(uuid.UUID(source_id) for source_id in expected_source_ids)
for _path, payload in requests:
assert payload["filters"] == {"source_ids": expected_source_ids}
assert payload["requested_space_ids"] == ["space-1"]
def test_ragcore_sources_deduplicate_reuploaded_versions_and_cap_cards(monkeypatch): def test_ragcore_sources_deduplicate_reuploaded_versions_and_cap_cards(monkeypatch):
provider = RAGcoreKnowledgeProvider() provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1") monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
citations = [] citations = []
for index in range(5): document_ids = [
"damage-procedure",
"damage-procedure",
"vehicle-return-procedure",
"cleaning-checklist",
"maintenance-escalation",
]
for index, document_id in enumerate(document_ids):
citations.append( citations.append(
{ _citation(
"id": f"cite-{index}", provider,
"document_id": f"doc-{index}", document_id,
"document_version_id": f"version-{index}", section="Return",
"title": "Damage procedure" if index < 2 else f"Procedure {index}", seed=str(index),
"section": "Return", )
"excerpt": (
f"Record visible damage before release, chunk {index}."
if index < 2
else f"Unique procedure evidence {index}."
),
}
) )
monkeypatch.setattr( monkeypatch.setattr(
provider, provider,
"_client", "_client",
lambda: _FakeClient(post_response=_FakeResponse(200, _answers_body(citations=citations))), lambda: _FakeClient(
post_response=_FakeResponse(200, _answers_body(provider, citations=citations))
),
) )
answer = provider.ask("What must I do about vehicle damage?", "dedupe-test") answer = provider.ask("What must I do about vehicle damage?", "dedupe-test")
assert len(answer.sources) == 3 assert len(answer.sources) == 3
assert sum(source.title == "Damage procedure" for source in answer.sources) == 1 assert sum(source.title == "Damage handling procedure" for source in answer.sources) == 1
def test_ragcore_answer_rejects_unknown_document_citation(monkeypatch):
provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
monkeypatch.setattr(
provider,
"_client",
lambda: _FakeClient(
post_response=_FakeResponse(
200,
_answers_body(
provider,
answer="A plausible-looking but unmanaged answer.",
citations=[
{
**_citation(provider, "damage-procedure"),
"source_id": str(uuid.uuid4()),
}
],
),
)
),
)
answer = provider.ask("What must I do about damage?", "unknown-answer-citation")
assert answer.evidence_state == "insufficient"
assert answer.answer == ""
assert answer.sources == []
@pytest.mark.parametrize(
("field_name", "bad_value"),
[
("id", "not-a-uuid"),
("document_id", "not-a-uuid"),
("document_version_id", "00000000-0000-0000-0000-000000000000"),
("source_id", "00000000-0000-4000-8000-000000000099"),
("source_uri", "ragcore://source/00000000-0000-4000-8000-000000000099"),
("locator", "another-procedure.md"),
("excerpt_sha256", "0" * 64),
("title", None),
("section", {"not": "a string"}),
],
)
def test_ragcore_rejects_broken_managed_source_provenance(field_name, bad_value):
provider = RAGcoreKnowledgeProvider()
citation = _citation(provider, "damage-procedure")
citation[field_name] = bad_value
assert provider._managed_source(citation, "en-GB") is None
def test_ragcore_rejects_fabricated_excerpt_even_with_matching_hash():
provider = RAGcoreKnowledgeProvider()
citation = _citation(provider, "damage-procedure")
fabricated = "Invent a repair price and promise it to the customer."
citation["excerpt"] = fabricated
citation["excerpt_sha256"] = hashlib.sha256(fabricated.encode("utf-8")).hexdigest()
assert provider._managed_source(citation, "en-GB") is None
def test_ragcore_never_relabels_an_english_source_as_dutch():
provider = RAGcoreKnowledgeProvider()
citation = _citation(provider, "damage-procedure", language="en-GB")
assert provider._managed_source(citation, "nl-BE") is None
def test_knowledge_feedback_is_audited_and_can_be_changed(ops_client): def test_knowledge_feedback_is_audited_and_can_be_changed(ops_client):
@@ -503,6 +725,7 @@ def test_ragcore_provider_not_answerable_is_insufficient_and_never_fabricates(mo
post_response=_FakeResponse( post_response=_FakeResponse(
200, 200,
_answers_body( _answers_body(
provider,
answer="This should never be shown.", answer="This should never be shown.",
answerability="not_answerable", answerability="not_answerable",
citations=[], citations=[],
@@ -524,7 +747,8 @@ def test_ragcore_provider_answerable_without_citations_is_insufficient(monkeypat
"_client", "_client",
lambda: _FakeClient( lambda: _FakeClient(
post_response=_FakeResponse( post_response=_FakeResponse(
200, _answers_body(answerability="answerable", citations=[]) 200,
_answers_body(provider, answerability="answerable", citations=[]),
) )
), ),
) )
@@ -565,19 +789,66 @@ def test_ragcore_provider_malformed_response_is_unavailable(monkeypatch):
assert answer.evidence_state == "unavailable" assert answer.evidence_state == "unavailable"
def _search_body(**overrides) -> dict: def test_ragcore_provider_malformed_top_level_bodies_do_not_escape(monkeypatch):
provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
monkeypatch.setattr(
provider,
"_client",
lambda: _FakeClient(
post_responses={
"/v1/answers": _FakeResponse(200, ["not", "an", "object"]),
"/v1/search": _FakeResponse(200, None),
}
),
)
answer = provider.ask("Anything?", "test-correlation-malformed-top-level")
assert answer.evidence_state == "unavailable"
assert answer.answer == ""
assert answer.sources == []
def test_ragcore_answer_with_malformed_citation_is_insufficient(monkeypatch):
provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
monkeypatch.setattr(
provider,
"_client",
lambda: _FakeClient(
post_response=_FakeResponse(
200,
_answers_body(provider, citations=[None]),
)
),
)
answer = provider.ask("What must I do about damage?", "malformed-answer-citation")
assert answer.evidence_state == "insufficient"
assert answer.answer == ""
assert answer.sources == []
def _search_body(
provider: RAGcoreKnowledgeProvider,
language: str = "en-GB",
**overrides,
) -> dict:
body = { body = {
"retrieval_run_id": str(uuid.uuid4()),
"effective_space_ids": ["00000000-0000-4000-8000-000000000001"],
"results": [ "results": [
{ {
"chunk_id": "chunk-1", "chunk_id": str(uuid.uuid4()),
"citation": { "text": _EXCERPTS[(language, "vehicle-return-procedure")],
"id": "cite-1", "citation": _citation(
"document_id": "doc-1", provider,
"document_version_id": "version-1", "vehicle-return-procedure",
"title": "Vehicle return procedure", language=language,
"section": "Return", section="Return",
"excerpt": "Register the return odometer reading before releasing the vehicle.", ),
},
"rank": 1, "rank": 1,
"scores": {"dense": None, "sparse": None, "fused": 0.5, "rerank": None}, "scores": {"dense": None, "sparse": None, "fused": 0.5, "rerank": None},
} }
@@ -588,6 +859,31 @@ def _search_body(**overrides) -> dict:
return body return body
def test_ragcore_search_ignores_malformed_result_and_citation_bodies(monkeypatch):
provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
malformed_search = _search_body(
provider,
results=[None, {"citation": ["not-an-object"], "scores": "not-an-object"}],
)
monkeypatch.setattr(
provider,
"_client",
lambda: _FakeClient(
post_responses={
"/v1/answers": _FakeResponse(503, {}),
"/v1/search": _FakeResponse(200, malformed_search),
}
),
)
answer = provider.ask("What is the vehicle return procedure?", "malformed-search-items")
assert answer.evidence_state == "insufficient"
assert answer.answer == ""
assert answer.sources == []
def test_ragcore_provider_falls_back_to_search_when_answers_unavailable(monkeypatch): def test_ragcore_provider_falls_back_to_search_when_answers_unavailable(monkeypatch):
"""/v1/answers itself failing (a real RAGcore-side outage in its generation step, """/v1/answers itself failing (a real RAGcore-side outage in its generation step,
not a real 'insufficient evidence' classification) must not silently degrade not a real 'insufficient evidence' classification) must not silently degrade
@@ -601,26 +897,24 @@ def test_ragcore_provider_falls_back_to_search_when_answers_unavailable(monkeypa
lambda: _FakeClient( lambda: _FakeClient(
post_responses={ post_responses={
"/v1/answers": _FakeResponse(503, {"code": "VALIDATION_RETRIES_EXHAUSTED"}), "/v1/answers": _FakeResponse(503, {"code": "VALIDATION_RETRIES_EXHAUSTED"}),
"/v1/search": _FakeResponse(200, _search_body()), "/v1/search": _FakeResponse(200, _search_body(provider)),
} }
), ),
) )
answer = provider.ask("What is the vehicle return procedure?", "test-correlation-fallback") answer = provider.ask("What is the vehicle return procedure?", "test-correlation-fallback")
assert answer.evidence_state == "grounded" assert answer.evidence_state == "grounded"
assert "Register the return odometer reading" in answer.answer assert "Open the active booking and record the ending odometer" in answer.answer
assert "Vehicle return procedure" in answer.answer assert "Vehicle return procedure" in answer.answer
assert len(answer.sources) == 1 assert len(answer.sources) == 1
assert answer.sources[0].title == "Vehicle return procedure" assert answer.sources[0].title == "Vehicle return procedure"
assert answer.sources[0].excerpt == ( assert answer.sources[0].excerpt == _EXCERPTS[("en-GB", "vehicle-return-procedure")]
"Register the return odometer reading before releasing the vehicle."
)
def test_ragcore_answers_circuit_skips_repeated_generation_failure(monkeypatch): def test_ragcore_answers_circuit_skips_repeated_generation_failure(monkeypatch):
provider = RAGcoreKnowledgeProvider() provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1") monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
monkeypatch.setattr(provider._settings, "ragcore_answers_circuit_breaker_seconds", 60.0) monkeypatch.setattr(provider._settings, "ragcore_answers_circuit_breaker_seconds", 60.0)
search_response = _FakeResponse(200, _search_body()) search_response = _FakeResponse(200, _search_body(provider))
monkeypatch.setattr( monkeypatch.setattr(
provider, provider,
"_client", "_client",
@@ -632,8 +926,7 @@ def test_ragcore_answers_circuit_skips_repeated_generation_failure(monkeypatch):
), ),
) )
assert ( assert (
provider.ask("What is the vehicle return procedure?", "first").evidence_state provider.ask("What is the vehicle return procedure?", "first").evidence_state == "grounded"
== "grounded"
) )
monkeypatch.setattr( monkeypatch.setattr(
@@ -654,7 +947,7 @@ def test_ragcore_provider_fallback_answer_is_localized(monkeypatch):
lambda: _FakeClient( lambda: _FakeClient(
post_responses={ post_responses={
"/v1/answers": _FakeResponse(503, {"code": "VALIDATION_RETRIES_EXHAUSTED"}), "/v1/answers": _FakeResponse(503, {"code": "VALIDATION_RETRIES_EXHAUSTED"}),
"/v1/search": _FakeResponse(200, _search_body()), "/v1/search": _FakeResponse(200, _search_body(provider, "nl-BE")),
} }
), ),
) )
@@ -676,7 +969,7 @@ def test_ragcore_provider_fallback_with_no_search_results_is_insufficient(monkey
lambda: _FakeClient( lambda: _FakeClient(
post_responses={ post_responses={
"/v1/answers": _FakeResponse(503, {"code": "VALIDATION_RETRIES_EXHAUSTED"}), "/v1/answers": _FakeResponse(503, {"code": "VALIDATION_RETRIES_EXHAUSTED"}),
"/v1/search": _FakeResponse(200, _search_body(results=[])), "/v1/search": _FakeResponse(200, _search_body(provider, results=[])),
} }
), ),
) )
@@ -695,7 +988,7 @@ def test_ragcore_search_fallback_rejects_out_of_domain_question(monkeypatch):
lambda: _FakeClient( lambda: _FakeClient(
post_responses={ post_responses={
"/v1/answers": _FakeResponse(503, {}), "/v1/answers": _FakeResponse(503, {}),
"/v1/search": _FakeResponse(200, _search_body()), "/v1/search": _FakeResponse(200, _search_body(provider)),
} }
), ),
) )
@@ -707,19 +1000,66 @@ def test_ragcore_search_fallback_rejects_out_of_domain_question(monkeypatch):
assert answer.answer == "" assert answer.answer == ""
def test_ragcore_search_fallback_rejects_unknown_document_citation(monkeypatch):
provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
search = _search_body(provider)
search["results"][0]["citation"]["source_id"] = str(uuid.uuid4())
monkeypatch.setattr(
provider,
"_client",
lambda: _FakeClient(
post_responses={
"/v1/answers": _FakeResponse(503, {}),
"/v1/search": _FakeResponse(200, search),
}
),
)
answer = provider.ask(
"What is the vehicle return procedure?",
"unknown-search-citation",
)
assert answer.evidence_state == "insufficient"
assert answer.answer == ""
assert answer.sources == []
def test_ragcore_search_fallback_rejects_vehicle_colour_despite_high_score(monkeypatch):
provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
search = _search_body(provider)
search["results"][0]["scores"]["fused"] = 1.0
monkeypatch.setattr(
provider,
"_client",
lambda: _FakeClient(
post_responses={
"/v1/answers": _FakeResponse(503, {}),
"/v1/search": _FakeResponse(200, search),
}
),
)
answer = provider.ask("What colour is this vehicle?", "vehicle-colour")
assert answer.evidence_state == "insufficient"
assert answer.answer == ""
def test_ragcore_search_fallback_prefers_damage_procedure(monkeypatch): def test_ragcore_search_fallback_prefers_damage_procedure(monkeypatch):
provider = RAGcoreKnowledgeProvider() provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1") monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
search = _search_body() search = _search_body(provider, "nl-BE", results=[])
search["results"].append( search["results"].append(
{ {
"citation": { "citation": _citation(
"document_id": "damage-procedure", provider,
"document_version_id": "version-2", "damage-procedure",
"title": "damage-procedure.md", language="nl-BE",
"section": "Damage", section="Damage",
"excerpt": "Record damage and keep the vehicle blocked.", ),
},
"rank": 2, "rank": 2,
"scores": {"fused": 0.01}, "scores": {"fused": 0.01},
} }
@@ -743,16 +1083,15 @@ def test_ragcore_search_fallback_prefers_damage_procedure(monkeypatch):
def test_ragcore_search_fallback_accepts_top_ranked_ragcore_damage_evidence(monkeypatch): def test_ragcore_search_fallback_accepts_top_ranked_ragcore_damage_evidence(monkeypatch):
provider = RAGcoreKnowledgeProvider() provider = RAGcoreKnowledgeProvider()
monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1") monkeypatch.setattr(provider._settings, "ragcore_space_id", "space-1")
search = _search_body() search = _search_body(provider, "nl-BE", results=[])
search["results"].append( search["results"].append(
{ {
"citation": { "citation": _citation(
"document_id": "damage-procedure", provider,
"document_version_id": "version-2", "damage-procedure",
"title": "damage-procedure.md", language="nl-BE",
"section": "Damage", section="Damage",
"excerpt": "Record damage and keep the vehicle blocked.", ),
},
"rank": 1, "rank": 1,
# RAGcore uses reciprocal-rank fusion; a genuine rank-one result is about # RAGcore uses reciprocal-rank fusion; a genuine rank-one result is about
# 1 / (60 + 1), not a normalized 0..1 relevance score. # 1 / (60 + 1), not a normalized 0..1 relevance score.
+31
View File
@@ -1,4 +1,12 @@
from app.core.config import get_settings from app.core.config import get_settings
from app.main import app
MCP_ROUTE_ALLOWLIST = {
("GET", "/api/v1/integrations/mcp/operations-summary"),
("GET", "/api/v1/integrations/mcp/attention-vehicles"),
("GET", "/api/v1/integrations/mcp/vehicles/{vehicle_ref}"),
("POST", "/api/v1/integrations/mcp/search-knowledge"),
}
def _headers(token: str | None = None, client_id: str = "test-mcp-client"): def _headers(token: str | None = None, client_id: str = "test-mcp-client"):
@@ -154,3 +162,26 @@ def test_no_write_endpoints_exist_under_mcp_namespace(client):
]: ]:
response = getattr(client, method)(path, headers=_headers()) response = getattr(client, method)(path, headers=_headers())
assert response.status_code in (404, 405) assert response.status_code in (404, 405)
def test_mcp_namespace_matches_exact_route_allowlist_and_rejects_lookalikes(client):
implemented = {
(method.upper(), path)
for path, operations in app.openapi()["paths"].items()
if path.startswith("/api/v1/integrations/mcp/")
for method in operations
if method in {"get", "post", "put", "patch", "delete"}
}
assert implemented == MCP_ROUTE_ALLOWLIST
lookalikes = [
("get", "/api/v1/integrations/mcp-extra/operations-summary"),
("get", "/api/v1/integrations/mcp/operations-summary-extra"),
("get", "/api/v1/integrations/mcp/prefix/attention-vehicles"),
("get", "/api/v1/integrations/mcp/attention-vehicles/suffix"),
("post", "/api/v1/integrations/mcp/search-knowledge-extra"),
("post", "/api/v1/integrations/mcp/prefix/search-knowledge"),
]
for method, path in lookalikes:
response = client.request(method, path, headers=_headers(), json={})
assert response.status_code == 404, path
+18
View File
@@ -63,6 +63,24 @@ def test_eligible_customer_is_irreversibly_anonymized_without_pii_in_audit(ops_c
) )
assert response.status_code == 200 assert response.status_code == 200
assert response.json()["status"] == "anonymized" assert response.json()["status"] == "anonymized"
search = ops_client.get("/api/v1/customers", params={"query": "CUS-PRIVACY"})
assert search.status_code == 200
assert search.json() == []
window = {
"starts_at": "2090-01-01T10:00:00Z",
"ends_at": "2090-01-02T10:00:00Z",
}
available = ops_client.get("/api/v1/bookings/availability", params=window).json()
assert available
booking = ops_client.post(
"/api/v1/bookings",
json={
"customer_ref": "CUS-PRIVACY",
"vehicle_ref": available[0]["public_ref"],
**window,
},
)
assert booking.status_code == 422
with SessionLocal() as db: with SessionLocal() as db:
customer = db.scalar(select(Customer).where(Customer.id == customer_id)) customer = db.scalar(select(Customer).where(Customer.id == customer_id))
assert customer is not None assert customer is not None
+17
View File
@@ -128,6 +128,23 @@ def test_seed_scenario_s4_booking_overlap():
db.close() db.close()
def test_seed_odometer_issues_expose_only_the_regressing_booking_as_correctable():
db = SessionLocal()
try:
reset_and_seed(db)
issue = _by_ref(db, DataQualityIssue, "DQ-0007")
assert issue is not None
assert issue.evidence_json["source_type"] == "return"
assert issue.evidence_json["related_refs"] == [
"INSP-0057",
"BK-H-0007",
"INSP-0007",
]
assert issue.evidence_json["correctable_booking_refs"] == ["BK-H-0007"]
finally:
db.close()
def test_seed_scenario_s5_failed_workflow_run(): def test_seed_scenario_s5_failed_workflow_run():
"""S5: one seeded outbox event is durably 'failed' (terminal, retryable), not merely """S5: one seeded outbox event is durably 'failed' (terminal, retryable), not merely
pending, so the background dispatcher never silently auto-heals it away.""" pending, so the background dispatcher never silently auto-heals it away."""
+73
View File
@@ -0,0 +1,73 @@
from __future__ import annotations
import csv
import subprocess
import sys
from pathlib import Path
REPOSITORY_ROOT = next(
parent
for parent in Path(__file__).resolve().parents
if (parent / "seed" / "generate_seed.py").is_file()
)
SEED_DIRECTORY = REPOSITORY_ROOT / "seed"
def _rows_by_ref(path: Path) -> dict[str, dict[str, str]]:
with path.open(newline="", encoding="utf-8") as handle:
rows = list(csv.DictReader(handle))
assert all(None not in row for row in rows), f"malformed CSV row in {path.name}"
return {row["public_ref"]: row for row in rows}
def test_generator_reproduces_committed_seed_snapshot_byte_for_byte(tmp_path: Path):
subprocess.run(
[
sys.executable,
str(SEED_DIRECTORY / "generate_seed.py"),
"--anchor",
"2026-08-01",
"--seed",
"20260801",
"--out",
str(tmp_path),
],
cwd=REPOSITORY_ROOT,
check=True,
capture_output=True,
text=True,
)
committed_files = sorted(SEED_DIRECTORY.glob("*.csv"))
generated_files = sorted(tmp_path.glob("*.csv"))
assert [path.name for path in generated_files] == [path.name for path in committed_files]
for committed in committed_files:
generated = tmp_path / committed.name
assert generated.read_bytes() == committed.read_bytes(), (
f"{committed.name} differs from the deterministic generated snapshot"
)
bookings = _rows_by_ref(tmp_path / "bookings.csv")
inspections = _rows_by_ref(tmp_path / "inspections.csv")
issues = _rows_by_ref(tmp_path / "data_quality_issues.csv")
assert int(bookings["BK-H-0007"]["end_odometer_km"]) < int(
bookings["BK-H-0057"]["end_odometer_km"]
)
assert int(bookings["BK-H-0010"]["end_odometer_km"]) < int(
bookings["BK-H-0060"]["end_odometer_km"]
)
assert inspections["INSP-0007"]["odometer_km"] == bookings["BK-H-0007"]["end_odometer_km"]
assert inspections["INSP-0010"]["odometer_km"] == bookings["BK-H-0010"]["end_odometer_km"]
assert inspections["INSP-0001"]["completed_at"] == bookings["BK-H-0001"]["ends_at"]
expected_issue_refs = {f"DQ-{index:04d}" for index in range(5, 22)}
assert expected_issue_refs <= issues.keys()
assert all(
issues[public_ref]["evidence"] != "Synthetic deterministic seed issue"
for public_ref in expected_issue_refs
)
assert "INSP-0007" in issues["DQ-0007"]["evidence"]
assert "INSP-0057" in issues["DQ-0007"]["evidence"]
assert "INSP-0010" in issues["DQ-0010"]["evidence"]
assert "INSP-0060" in issues["DQ-0010"]["evidence"]
+19
View File
@@ -23,3 +23,22 @@ def test_manager_can_create_and_update_user(ops_client):
def test_employee_cannot_manage_users(employee_client): def test_employee_cannot_manage_users(employee_client):
assert employee_client.get("/api/v1/users").status_code == 403 assert employee_client.get("/api/v1/users").status_code == 403
assert (
employee_client.post(
"/api/v1/users",
json={
"email": "unauthorised@example.test",
"display_name": "Unauthorised User",
"role": "rental_employee",
"password": "a-secure-demo-password",
},
).status_code
== 403
)
assert (
employee_client.patch(
"/api/v1/users/USR-OPS",
json={"display_name": "Unauthorised Change"},
).status_code
== 403
)
+53
View File
@@ -46,6 +46,12 @@ def test_vehicle_detail_includes_related_records(ops_client):
assert len(body["quality_issues"]) >= 1 assert len(body["quality_issues"]) >= 1
def test_employee_vehicle_detail_never_exposes_quality_evidence(employee_client):
response = employee_client.get("/api/v1/vehicles/MO-016")
assert response.status_code == 200
assert response.json()["quality_issues"] == []
def test_vehicle_detail_404_for_unknown_ref(ops_client): def test_vehicle_detail_404_for_unknown_ref(ops_client):
response = ops_client.get("/api/v1/vehicles/MO-999") response = ops_client.get("/api/v1/vehicles/MO-999")
assert response.status_code == 404 assert response.status_code == 404
@@ -78,6 +84,53 @@ def test_manager_can_record_maintenance_and_release_vehicle(ops_client):
assert released.json()["operational_status"] == "available" assert released.json()["operational_status"] == "available"
def test_low_maintenance_reading_keeps_canonical_and_opens_quality_issue(ops_client):
existing_issues = ops_client.get(
"/api/v1/data-quality/issues",
params={"status": "open", "rule_type": "odometer_regression"},
).json()
unavailable_refs = {issue["entity_ref"] for issue in existing_issues}
vehicles = ops_client.get("/api/v1/vehicles", params={"status": "available"}).json()
vehicle = next(
candidate
for candidate in vehicles
if candidate["odometer_km"] > 0 and candidate["public_ref"] not in unavailable_refs
)
response = ops_client.post(
f"/api/v1/vehicles/{vehicle['public_ref']}/maintenance",
json={
"occurred_at": "2053-01-15T12:00:00Z",
"odometer_km": vehicle["odometer_km"] - 1,
"category": "inspection",
"summary": "Imported workshop reading requires verification.",
"mark_maintenance": True,
},
)
assert response.status_code == 201
detail = ops_client.get(f"/api/v1/vehicles/{vehicle['public_ref']}").json()
assert detail["odometer_km"] == vehicle["odometer_km"]
issues = ops_client.get(
"/api/v1/data-quality/issues",
params={"status": "open", "rule_type": "odometer_regression"},
).json()
issue = next(item for item in issues if item["entity_ref"] == vehicle["public_ref"])
assert issue["evidence"]["source_type"] == "maintenance"
assert issue["evidence"]["correctable_booking_refs"] == []
issue_detail = ops_client.get(f"/api/v1/data-quality/issues/{issue['public_ref']}").json()
assert any(
snapshot["entity_type"] == "maintenance"
and snapshot["public_ref"] == response.json()["public_ref"]
for snapshot in issue_detail["related_snapshots"]
)
retained = ops_client.post(
f"/api/v1/data-quality/issues/{issue['public_ref']}/resolve-odometer-regression",
json={"decision": "retain_canonical"},
)
assert retained.status_code == 200
def test_employee_cannot_record_maintenance(employee_client): def test_employee_cannot_record_maintenance(employee_client):
response = employee_client.post( response = employee_client.post(
"/api/v1/vehicles/MO-001/maintenance", "/api/v1/vehicles/MO-001/maintenance",
+10
View File
@@ -17,6 +17,11 @@ services:
- mobilityops-prometheus:/prometheus - mobilityops-prometheus:/prometheus
ports: ports:
- "127.0.0.1:19090:9090" - "127.0.0.1:19090:9090"
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:9090/-/ready"]
interval: 15s
timeout: 5s
retries: 10
restart: unless-stopped restart: unless-stopped
networks: [mobilityops] networks: [mobilityops]
@@ -61,6 +66,11 @@ services:
- mobilityops-grafana:/var/lib/grafana - mobilityops-grafana:/var/lib/grafana
ports: ports:
- "127.0.0.1:13000:3000" - "127.0.0.1:13000:3000"
healthcheck:
test: ["CMD-SHELL", "wget -q -O - http://127.0.0.1:3000/api/health | grep -Eq '\"database\"[[:space:]]*:[[:space:]]*\"ok\"'"]
interval: 15s
timeout: 5s
retries: 20
depends_on: [prometheus, alertmanager] depends_on: [prometheus, alertmanager]
restart: unless-stopped restart: unless-stopped
networks: [mobilityops] networks: [mobilityops]
+2
View File
@@ -2,6 +2,8 @@ services:
api: api:
image: ${MOBILITYOPS_API_IMAGE:?Set MOBILITYOPS_API_IMAGE to an immutable release image} image: ${MOBILITYOPS_API_IMAGE:?Set MOBILITYOPS_API_IMAGE to an immutable release image}
build: !reset null build: !reset null
environment:
RUN_MIGRATIONS: "false"
web: web:
image: ${MOBILITYOPS_WEB_IMAGE:?Set MOBILITYOPS_WEB_IMAGE to an immutable release image} image: ${MOBILITYOPS_WEB_IMAGE:?Set MOBILITYOPS_WEB_IMAGE to an immutable release image}
+61 -1
View File
@@ -9,13 +9,30 @@ services:
web: web:
restart: unless-stopped restart: unless-stopped
ports: !override ports: !override
- "1236:80" []
healthcheck: healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1/health"] test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1/health"]
interval: 10s interval: 10s
timeout: 5s timeout: 5s
retries: 10 retries: 10
gateway:
image: nginx:1.30.4-alpine@sha256:97d490c12ba55b4946b01546d1c3ed324e8d41ab1c9fcb2a616aa470620e5b46
restart: unless-stopped
ports:
- "1236:80"
volumes:
- ${MOBILITYOPS_GATEWAY_CONFIG:-./deploy/unraid/gateway.conf}:/etc/nginx/conf.d/default.conf:ro
depends_on:
web:
condition: service_healthy
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1/health"]
interval: 10s
timeout: 5s
retries: 10
networks: [mobilityops]
n8n: n8n:
# The review server already has a shared n8n instance. Keep the bundled service # The review server already has a shared n8n instance. Keep the bundled service
# available for an explicit fallback without starting a second instance by default. # available for an explicit fallback without starting a second instance by default.
@@ -67,3 +84,46 @@ services:
retries: 3 retries: 3
start_period: 5m start_period: 5m
networks: [mobilityops] networks: [mobilityops]
offsite-backup:
profiles: ["offsite"]
image: ${MOBILITYOPS_BACKUP_TOOLS_IMAGE:-mobilityops-backup-tools:development}
build:
context: .
dockerfile: deploy/unraid/Dockerfile.backup-tools
args:
VCS_REF: ${MOBILITYOPS_SOURCE_REVISION:-development}
restart: unless-stopped
entrypoint: ["/opt/mobilityops/sync-onedrive-backups.sh"]
environment:
POSTGRES_DB: ${POSTGRES_DB:-mobilityops}
POSTGRES_USER: ${POSTGRES_USER:-mobilityops}
PGPASSWORD: ${POSTGRES_PASSWORD:-mobilityops}
POSTGRES_HOST: db
BACKUP_DESTINATION: /backups
BACKUP_OFFSITE_VERIFY_DIR: /offsite-verify
BACKUP_OFFSITE_INTERVAL_SECONDS: ${BACKUP_OFFSITE_INTERVAL_SECONDS:-900}
BACKUP_RESTORE_DRILL_INTERVAL_SECONDS: ${BACKUP_RESTORE_DRILL_INTERVAL_SECONDS:-604800}
BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-30}
RCLONE_CONFIG: /config/rclone/rclone.conf
RCLONE_ONEDRIVE_REMOTE: ${RCLONE_ONEDRIVE_REMOTE:-onedrive}
RCLONE_ONEDRIVE_PATH: ${RCLONE_ONEDRIVE_PATH:-FleetOps/backups}
volumes:
- ${MOBILITYOPS_BACKUP_DIR:-./backups/postgres}:/backups
- ${MOBILITYOPS_OFFSITE_VERIFY_DIR:-./backups/offsite-verify}:/offsite-verify
- ${MOBILITYOPS_RCLONE_CONFIG_DIR:-./.secrets/rclone}:/config/rclone
- ./deploy/unraid:/opt/mobilityops:ro
depends_on:
db:
condition: service_healthy
healthcheck:
test:
[
"CMD-SHELL",
"find /backups/latest-offsite-success -mmin -1560 -print -quit | grep -q . && find /backups/latest-offsite-restore-drill -mmin -11520 -print -quit | grep -q .",
]
interval: 30m
timeout: 5s
retries: 3
start_period: 10m
networks: [mobilityops]
+2
View File
@@ -107,6 +107,8 @@ services:
N8N_BASIC_AUTH_PASSWORD: ${N8N_BASIC_AUTH_PASSWORD:-change-me} N8N_BASIC_AUTH_PASSWORD: ${N8N_BASIC_AUTH_PASSWORD:-change-me}
N8N_SECURE_COOKIE: "false" N8N_SECURE_COOKIE: "false"
N8N_BLOCK_ENV_ACCESS_IN_NODE: "false" N8N_BLOCK_ENV_ACCESS_IN_NODE: "false"
MOBILITYOPS_ALERT_RECIPIENT: ${MOBILITYOPS_ALERT_RECIPIENT:-alerts@example.test}
MOBILITYOPS_ALERT_SENDER: ${MOBILITYOPS_ALERT_SENDER:-n8n@example.test}
MOBILITYOPS_CALLBACK_TOKEN: ${MOBILITYOPS_CALLBACK_TOKEN:-replace-me-n8n-callback-token} MOBILITYOPS_CALLBACK_TOKEN: ${MOBILITYOPS_CALLBACK_TOKEN:-replace-me-n8n-callback-token}
ports: ports:
- "5678:5678" - "5678:5678"
+9 -7
View File
@@ -2757,10 +2757,6 @@ components:
type: string type: string
format: date-time format: date-time
title: Ends At title: Ends At
requirements_complete:
type: boolean
title: Requirements Complete
default: false
type: object type: object
required: required:
- customer_ref - customer_ref
@@ -4189,11 +4185,13 @@ components:
title: ResolveOverlapRequest title: ResolveOverlapRequest
ReturnCallbackIn: ReturnCallbackIn:
properties: properties:
event_id:
type: string
format: uuid
title: Event Id
correlation_id: correlation_id:
anyOf: type: string
- type: string
format: uuid format: uuid
- type: 'null'
title: Correlation Id title: Correlation Id
follow_up: follow_up:
anyOf: anyOf:
@@ -4208,6 +4206,9 @@ components:
- type: 'null' - type: 'null'
title: Summary title: Summary
type: object type: object
required:
- event_id
- correlation_id
title: ReturnCallbackIn title: ReturnCallbackIn
description: 'Body of the n8n return follow-up callback. description: 'Body of the n8n return follow-up callback.
@@ -4777,6 +4778,7 @@ components:
correlation_id: correlation_id:
anyOf: anyOf:
- type: string - type: string
format: uuid
- type: 'null' - type: 'null'
title: Correlation Id title: Correlation Id
attempt: attempt:
+17
View File
@@ -0,0 +1,17 @@
FROM golang:1.26.6-alpine@sha256:3889b425f035be855a72fb4755265311293b6d414521f0a519d819df32222d83 AS rclone
RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go install -trimpath \
-ldflags '-s -X github.com/rclone/rclone/fs.Version=v1.75.0' \
github.com/rclone/rclone@v1.75.0
FROM postgres:16-alpine@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685
ARG VCS_REF=development
ARG BUILD_DATE=unknown
LABEL org.opencontainers.image.title="MobilityOps backup tools" \
org.opencontainers.image.source="https://github.com/ITWorxBE/MobilityOps" \
org.opencontainers.image.revision="$VCS_REF" \
org.opencontainers.image.created="$BUILD_DATE"
COPY --from=rclone /go/bin/rclone /usr/local/bin/rclone
# The PostgreSQL image's entrypoint uses gosu only when starting the database server;
# this tools-only image always overrides that entrypoint with the sync worker.
RUN rm -f /usr/local/bin/gosu

Some files were not shown because too many files have changed in this diff Show More