M3: implement Data Quality Workbench

Five rule scanners (duplicate customers, missing fields, odometer regression, booking overlap, status conflict) run automatically after seed and via an explicit scan endpoint. Issue defer/reject/merge-customers endpoints with transactional customer merge (booking rewiring, tombstone, audit). Data Quality nav + workbench UI with two-column duplicate comparison and inline (non-native) confirm. Dashboard attention items now link to issues. 35 backend tests passing, ruff clean. Fixed a real false-positive bug in odometer-regression detection found through iteration on seed data, and two TS narrowing errors. Verified end-to-end via browser: S2 merge and S4 overlap scenarios.
This commit is contained in:
NuklearRabbit
2026-08-01 22:11:06 +02:00
parent 0091c57c7f
commit a7cbeaae3b
17 changed files with 1127 additions and 5 deletions
+15 -2
View File
@@ -2,7 +2,7 @@
## Current milestone
M2 — complete. Starting M3 next.
M3 — complete. Starting M4 next.
## Locked decisions
@@ -65,10 +65,23 @@ M2 — complete. Starting M3 next.
- `docker compose up -d --build` (all services) then `docker compose exec api python -m app.cli seed --reset`, then a full browser run of the S1 demo scenario against `BK-DEMO-RETURN`/`MO-024`: submitted 53000 km (below canonical 54820) → result panel showed `INSP-0076`, `resulting_vehicle_status: maintenance` (correctly derived, since canonical 54820 ≥ `next_service_km` 40000), `DQ-RET-0076` created, automation event queued with a real UUID, "no upcoming booking" risk; vehicle detail page confirmed odometer unchanged at 54,820 km and a "Needs attention" badge.
- Both real defects listed above (form disappearing before showing its result; `event_id` reading as `None`) were **found via the browser run, not by pytest** — the test suite asserted on API response shape/values, not on what the UI actually rendered after a status transition. Worth remembering for M3+: UI state-after-mutation bugs need a browser check, not just API tests.
### M3 — Data Quality Workbench
- `app/services/data_quality.py`: `run_scan()` implements all five rules and is called automatically at the end of `seed_loader.reset_and_seed()` (after `db.commit()` of the base seed), plus exposed as `POST /api/v1/data-quality/scan` (Operations Manager only). Idempotency is simplified from the doc's literal `(rule_type, entity_type, entity_id, evidence fingerprint)` to just `(rule_type, entity_type, entity_id)` while an issue is open — see rationale below.
- Router `app/api/routers/data_quality.py`: `GET /issues` (filters status/rule_type/severity), `GET /issues/{ref}` (adds `entity_snapshot`/`related_snapshots` for the UI), `POST /issues/{ref}/defer`, `/reject`, `/merge-customers` (Operations Manager only — enforced via `require_operations_manager`), `POST /scan`.
- **Real bug found and fixed during this milestone, before any browser check**: the first cut of DQ-03 (odometer regression) compared every historical *returned* booking's `end_odometer_km` against the vehicle's *current* `odometer_km`. Since the seed generator assigns `vehicle.odometer_km` independently of booking history (see `seed/generate_seed.py`), this is true for nearly every historical booking by construction (odometer is monotonically increasing over time, so all-but-the-latest reading is "below current") — it produced 51 false-positive issues out of 50 vehicles on first run. Fixed twice: first attempt (compare only the single most-recent booking against canonical) still produced the same problem because canonical itself is disconnected from booking history in this dataset; the working fix compares each vehicle's *own returned-booking sequence* against itself (each booking's end reading vs. the immediately preceding one, chronologically) — a self-consistency check that doesn't depend on the unrelated `vehicle.odometer_km` field at all. Final deterministic seed+scan totals: 15 CSV-seeded + 11 scan-discovered = **26** open/resolved `data_quality_issues` (breakdown: 14 vehicle_status_conflict, 5 missing_required_field, 3 possible_duplicate_customer, 3 odometer_regression, 1 booking_overlap). `tests/test_seed.py`'s exact-count assertion was updated from 15 to 26 accordingly — if the scan logic changes again, update that count.
- Idempotency simplification rationale: the doc's fingerprint-based key would make the scan blind to issues it structurally can't compute a matching fingerprint for against the CSV-seeded rows (which don't carry a fingerprint field), producing duplicate issues for the same real-world problem (e.g. a second `MO-016` overlap issue next to the seeded `DQ-DEMO-OVERLAP`). Using `(rule_type, entity_type, entity_id)` alone while open is a stricter, safe simplification: it can never falsely suppress an issue for a *different* entity, and per-entity there's realistically only one meaningful open issue of a given rule type at a time for this PoC's scope.
- Merge UI intentionally does **not** use `window.confirm()` — a native dialog blocks further automation/testing and isn't screen-reader-distinguishable from page content the same way a rendered `role="alertdialog"` panel is. Built an inline two-step confirm instead (`ReturnForm`-style pattern reused).
- `AttentionItem` gained an `issue_ref` field (dashboard now links attention items straight to `/data-quality/{issue_ref}` instead of only to vehicles); dashboard attention list capped at 8 items (was unbounded, would have shown up to 26 with the richer scan).
- Commands run and verified from this checkout:
- `docker compose run --rm api pytest -q`**35 passed** (new `tests/test_data_quality.py`: all five rule types present, scan idempotent on rerun, scan requires Operations Manager, S2/S4 issue-detail snapshots correct, defer→reject-on-closed 409, merge requires Operations Manager, merge rejects an unrelated survivor ref, full S2 merge scenario asserting rewiring + audit + replay-is-409).
- `docker compose run --rm api ruff check .` — All checks passed.
- `npm run build` — clean (had to fix two `possibly 'null'` TS errors from a closure-narrowing limitation — TS doesn't narrow `const` captured-by-closure across nested function boundaries when the value comes from an index/property expression; fixed by re-binding to explicitly-typed local consts right after the guard).
- Full browser run: Data Quality list (26 open issues, filterable) → `DQ-DEMO-DUPLICATE` two-column compare (CUS-0012 vs CUS-0178, per-field diff highlighting only where they differ) → merge with inline confirm → issue flips to `resolved` → confirmed `customer_merged` audit event with correct actor/entity/correlation → `DQ-DEMO-OVERLAP` (non-duplicate type) renders evidence JSON + defer/reject, no dead compare UI shown for a rule type it doesn't apply to.
## Known blockers
None. External service credentials may be absent; use the documented demo/degraded providers.
## Exact next action
Start M3 (Data Quality Workbench): read `docs/07-data-quality.md`. Implement the five rule scanners (possible_duplicate_customer with weighted scoring, missing_required_field, odometer_regression, booking_overlap, vehicle_status_conflict) as an explicit scan service (idempotent by `(rule_type, entity_type, entity_id, evidence fingerprint)` while open, per the doc's lifecycle section), issue defer/reject/merge-customers endpoints, a transactional customer-merge (rewires bookings, tombstones the loser, audits before/after), the Data Quality nav item + Workbench UI (two-column duplicate comparison, evidence/proposed-resolution for other types), and wire "open quality issues" into the dashboard attention section it already reads from. Seed data already has 15 `data_quality_issues` including the named `DQ-DEMO-*` scenarios (S2 duplicate CUS-0012/CUS-0178, S4 overlap MO-016) to scan/resolve against — remember M2's return workflow also creates ad-hoc `odometer_regression` issues (`DQ-RET-*`) directly, so the scan service's idempotent-fingerprint rule must not double-report those.
Start M4 (n8n automation): read `docs/11-n8n-integration.md`, `n8n/README.md`. Implement the outbox dispatcher (poll pending `outbox_events`, claim with `FOR UPDATE SKIP LOCKED`, POST to n8n's webhook with timeout + exponential backoff + small max-attempt cap, never hold a DB transaction open during the HTTP call), correct/import `n8n/mobilityops-return-processing.json` (credential + the callback route — M2's `vehicle.returned.v1` outbox payload shape is already `contracts/events.schema.json`-compliant, so the workflow should be able to consume it as-is), a narrow MobilityOps callback endpoint the workflow calls back into, an Automation nav item + UI (pending/succeeded/failed list, manual retry button, matches `docs/06-ui-ux.md`'s "Recent automation" concept already on the dashboard), and manual-retry wiring for `POST /api/v1/workflows/{event_id}/retry` per `docs/05-api-contract.md`. The seeded `workflow_runs.csv` already includes one deterministic `failed` row (`event_id 00000000-...-0020`, "Synthetic connection timeout to n8n") for the S5 demo scenario — dashboard/Automation UI must surface it and allow retry. Remember the compose `n8n` service is already up (`http://localhost:5678`, basic auth `admin`/`change-me` from `.env.example`) but the workflow itself hasn't been imported/activated yet this session.
+2
View File
@@ -86,9 +86,11 @@ def get_dashboard(
detail=issue.evidence_json.get("summary", ""),
link_type=link_type,
link_ref=link_ref,
issue_ref=issue.public_ref,
)
)
attention_items.sort(key=lambda item: _SEVERITY_ORDER.get(item.severity, 3))
attention_items = attention_items[:8]
today = _today()
bookings = db.scalars(select(Booking)).all()
+146
View File
@@ -0,0 +1,146 @@
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.api.deps import get_current_user, get_db, require_operations_manager
from app.models.customer import Customer
from app.models.data_quality import DataQualityIssue
from app.models.vehicle import Vehicle
from app.schemas import (
CurrentUser,
DataQualityIssueDetailOut,
DataQualityIssueOut,
MergeCustomersRequest,
MergeCustomersResult,
ScanResultOut,
)
from app.services.data_quality import defer_issue, merge_customers, reject_issue, run_scan
router = APIRouter(prefix="/api/v1/data-quality", tags=["data-quality"])
def _to_out(issue: DataQualityIssue) -> DataQualityIssueOut:
return DataQualityIssueOut(
public_ref=issue.public_ref,
rule_type=issue.rule_type,
entity_type=issue.entity_type,
entity_ref=issue.evidence_json.get("entity_ref", ""),
severity=issue.severity,
status=issue.status,
evidence=issue.evidence_json,
detected_at=issue.detected_at,
resolved_at=issue.resolved_at,
)
@router.get("/issues", response_model=list[DataQualityIssueOut])
def list_issues(
status: str | None = Query(default=None),
rule_type: str | None = Query(default=None),
severity: str | None = Query(default=None),
db: Session = Depends(get_db),
_user: CurrentUser = Depends(get_current_user),
) -> list[DataQualityIssueOut]:
stmt = select(DataQualityIssue).order_by(DataQualityIssue.detected_at.desc())
if status:
stmt = stmt.where(DataQualityIssue.status == status)
if rule_type:
stmt = stmt.where(DataQualityIssue.rule_type == rule_type)
if severity:
stmt = stmt.where(DataQualityIssue.severity == severity)
issues = db.scalars(stmt).all()
return [_to_out(i) for i in issues]
def _snapshot(entity_type: str, ref: str, db: Session) -> dict | None:
if entity_type == "customer":
obj = db.scalar(select(Customer).where(Customer.public_ref == ref))
if obj is None:
return None
return {
"public_ref": obj.public_ref,
"first_name": obj.first_name,
"last_name": obj.last_name,
"email": obj.email,
"phone": obj.phone,
"postal_code": obj.postal_code,
"city": obj.city,
}
obj = db.scalar(select(Vehicle).where(Vehicle.public_ref == ref))
if obj is None:
return None
return {
"public_ref": obj.public_ref,
"make": obj.make,
"model": obj.model,
"location": obj.location,
"operational_status": obj.operational_status,
"odometer_km": obj.odometer_km,
}
@router.get("/issues/{public_ref}", response_model=DataQualityIssueDetailOut)
def get_issue(
public_ref: str,
db: Session = Depends(get_db),
_user: CurrentUser = Depends(get_current_user),
) -> DataQualityIssueDetailOut:
issue = db.scalar(select(DataQualityIssue).where(DataQualityIssue.public_ref == public_ref))
if issue is None:
raise HTTPException(status_code=404, detail="Data quality issue not found")
base = _to_out(issue)
related_refs = issue.evidence_json.get("related_refs", [])
related_entity_type = (
"customer" if issue.rule_type == "possible_duplicate_customer" else "vehicle"
)
return DataQualityIssueDetailOut(
**base.model_dump(),
entity_snapshot=_snapshot(issue.entity_type, base.entity_ref, db),
related_snapshots=[
snap
for ref in related_refs
if (snap := _snapshot(related_entity_type, ref, db)) is not None
],
)
@router.post("/issues/{public_ref}/defer", response_model=DataQualityIssueOut)
def defer(
public_ref: str,
db: Session = Depends(get_db),
user: CurrentUser = Depends(get_current_user),
) -> DataQualityIssueOut:
issue = defer_issue(db, public_ref, user)
return _to_out(issue)
@router.post("/issues/{public_ref}/reject", response_model=DataQualityIssueOut)
def reject(
public_ref: str,
db: Session = Depends(get_db),
user: CurrentUser = Depends(get_current_user),
) -> DataQualityIssueOut:
issue = reject_issue(db, public_ref, user)
return _to_out(issue)
@router.post("/issues/{public_ref}/merge-customers", response_model=MergeCustomersResult)
def merge(
public_ref: str,
body: MergeCustomersRequest,
db: Session = Depends(get_db),
user: CurrentUser = Depends(require_operations_manager),
) -> MergeCustomersResult:
result = merge_customers(db, public_ref, body.survivor_ref, body.field_overrides, user)
return MergeCustomersResult(**result)
@router.post("/scan", response_model=ScanResultOut)
def scan(
db: Session = Depends(get_db),
_user: CurrentUser = Depends(require_operations_manager),
) -> ScanResultOut:
result = run_scan(db)
return ScanResultOut(created=result.created)
+2 -1
View File
@@ -4,7 +4,7 @@ from fastapi import FastAPI, HTTPException, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from app.api.routers import audit, bookings, dashboard, demo, vehicles
from app.api.routers import audit, bookings, dashboard, data_quality, demo, vehicles
from app.core.config import get_settings
from app.core.errors import AppError, error_body
@@ -61,3 +61,4 @@ app.include_router(dashboard.router)
app.include_router(vehicles.router)
app.include_router(bookings.router)
app.include_router(audit.router)
app.include_router(data_quality.router)
+22
View File
@@ -106,6 +106,27 @@ class DataQualityIssueOut(BaseModel):
resolved_at: datetime | None = None
class DataQualityIssueDetailOut(DataQualityIssueOut):
entity_snapshot: dict[str, Any] | None = None
related_snapshots: list[dict[str, Any]] = Field(default_factory=list)
class MergeCustomersRequest(BaseModel):
survivor_ref: str
field_overrides: dict[str, str] | None = None
class MergeCustomersResult(BaseModel):
issue_ref: str
survivor_ref: str
loser_ref: str
rewired_bookings: int
class ScanResultOut(BaseModel):
created: dict[str, int]
class VehicleDetailOut(VehicleOut):
bookings: list[BookingSummaryOut] = Field(default_factory=list)
inspections: list[InspectionOut] = Field(default_factory=list)
@@ -130,6 +151,7 @@ class AttentionItem(BaseModel):
detail: str
link_type: Literal["vehicle", "booking", "customer"]
link_ref: str
issue_ref: str | None = None
class TodayItem(BaseModel):
+4
View File
@@ -260,7 +260,11 @@ def load_seed(db: Session) -> SeedResult:
def reset_and_seed(db: Session) -> SeedResult:
from app.services.data_quality import run_scan
clear_all(db)
result = load_seed(db)
db.commit()
scan = run_scan(db)
result.counts["data_quality_issues"] += sum(scan.created.values())
return result
+426
View File
@@ -0,0 +1,426 @@
from __future__ import annotations
import uuid
from dataclasses import dataclass, field
from datetime import UTC, datetime
from difflib import SequenceMatcher
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.core.errors import AppError
from app.models.booking import Booking
from app.models.customer import Customer
from app.models.data_quality import DataQualityIssue
from app.models.vehicle import Vehicle
from app.schemas import CurrentUser
from app.services.audit import record_audit_event
REQUIRED_CUSTOMER_FIELDS = ("first_name", "last_name")
REQUIRED_VEHICLE_FIELDS = ("registration_number", "make", "model", "location")
DUPLICATE_THRESHOLD = 70
@dataclass
class ScanResult:
created: dict[str, int] = field(default_factory=dict)
def bump(self, rule_type: str) -> None:
self.created[rule_type] = self.created.get(rule_type, 0) + 1
def _normalize(value: str | None) -> str:
return (value or "").strip().lower()
def _has_open_issue(db: Session, rule_type: str, entity_type: str, entity_id: uuid.UUID) -> bool:
return (
db.scalar(
select(DataQualityIssue.id).where(
DataQualityIssue.rule_type == rule_type,
DataQualityIssue.entity_type == entity_type,
DataQualityIssue.entity_id == entity_id,
DataQualityIssue.status == "open",
)
)
is not None
)
def _next_public_ref(db: Session, prefix: str) -> str:
existing = db.execute(select(DataQualityIssue.public_ref)).scalars().all()
numbers = [
int(ref.rsplit("-", 1)[-1])
for ref in existing
if ref.startswith(f"{prefix}-") and ref.rsplit("-", 1)[-1].isdigit()
]
next_number = (max(numbers) + 1) if numbers else 1
return f"{prefix}-{next_number:04d}"
def _open_issue(
db: Session,
scan: ScanResult,
*,
rule_type: str,
entity_type: str,
entity_id: uuid.UUID,
severity: str,
summary: str,
entity_ref: str,
related_refs: list[str],
) -> None:
if _has_open_issue(db, rule_type, entity_type, entity_id):
return
now = datetime.now(UTC)
issue = DataQualityIssue(
public_ref=_next_public_ref(db, "DQ-SCAN"),
rule_type=rule_type,
entity_type=entity_type,
entity_id=entity_id,
severity=severity,
status="open",
evidence_json={
"summary": summary,
"entity_ref": entity_ref,
"related_refs": related_refs,
},
proposed_action_json={},
detected_at=now,
)
db.add(issue)
db.flush()
scan.bump(rule_type)
def _scan_duplicate_customers(db: Session, scan: ScanResult) -> None:
customers = db.scalars(
select(Customer).where(Customer.merged_into_customer_id.is_(None))
).all()
customers.sort(key=lambda c: c.public_ref)
for i, a in enumerate(customers):
for b in customers[i + 1 :]:
score = 0
signals = []
if _normalize(a.email) and _normalize(a.email) == _normalize(b.email):
score += 60
signals.append("exact email")
if _normalize(a.phone) and _normalize(a.phone) == _normalize(b.phone):
score += 50
signals.append("exact phone")
if _normalize(a.postal_code) and _normalize(a.postal_code) == _normalize(b.postal_code):
score += 10
signals.append("exact postal code")
name_a = f"{_normalize(a.first_name)} {_normalize(a.last_name)}"
name_b = f"{_normalize(b.first_name)} {_normalize(b.last_name)}"
ratio = SequenceMatcher(None, name_a, name_b).ratio()
if ratio >= 0.5:
score += round(ratio * 30)
signals.append("similar name")
if score >= DUPLICATE_THRESHOLD:
_open_issue(
db,
scan,
rule_type="possible_duplicate_customer",
entity_type="customer",
entity_id=a.id,
severity="high",
summary="; ".join(signals) + f" (score {score})",
entity_ref=a.public_ref,
related_refs=[b.public_ref],
)
def _scan_missing_required_fields(db: Session, scan: ScanResult) -> None:
for customer in db.scalars(
select(Customer).where(Customer.merged_into_customer_id.is_(None))
).all():
missing = [f for f in REQUIRED_CUSTOMER_FIELDS if not getattr(customer, f)]
if not customer.email and not customer.phone:
missing.append("email_or_phone")
if missing:
_open_issue(
db,
scan,
rule_type="missing_required_field",
entity_type="customer",
entity_id=customer.id,
severity="low",
summary=f"Missing: {', '.join(missing)}",
entity_ref=customer.public_ref,
related_refs=[],
)
for vehicle in db.scalars(select(Vehicle).where(Vehicle.active.is_(True))).all():
missing = [f for f in REQUIRED_VEHICLE_FIELDS if not getattr(vehicle, f)]
if missing:
_open_issue(
db,
scan,
rule_type="missing_required_field",
entity_type="vehicle",
entity_id=vehicle.id,
severity="low",
summary=f"Missing: {', '.join(missing)}",
entity_ref=vehicle.public_ref,
related_refs=[],
)
def _scan_booking_overlaps(db: Session, scan: ScanResult) -> None:
vehicles = db.scalars(select(Vehicle)).all()
bookings_by_vehicle: dict[uuid.UUID, list[Booking]] = {}
for booking in db.scalars(
select(Booking).where(Booking.status.in_(["reserved", "active"]))
).all():
bookings_by_vehicle.setdefault(booking.vehicle_id, []).append(booking)
vehicle_by_id = {v.id: v for v in vehicles}
for vehicle_id, bookings in bookings_by_vehicle.items():
bookings.sort(key=lambda b: b.starts_at)
for i, first in enumerate(bookings):
for second in bookings[i + 1 :]:
if second.starts_at < first.ends_at and first.starts_at < second.ends_at:
vehicle = vehicle_by_id[vehicle_id]
_open_issue(
db,
scan,
rule_type="booking_overlap",
entity_type="vehicle",
entity_id=vehicle_id,
severity="high",
summary=f"Overlapping bookings {first.public_ref} and {second.public_ref}",
entity_ref=vehicle.public_ref,
related_refs=[first.public_ref, second.public_ref],
)
def _scan_vehicle_status_conflicts(db: Session, scan: ScanResult) -> None:
vehicles = db.scalars(select(Vehicle)).all()
active_by_vehicle: dict[uuid.UUID, list[Booking]] = {}
for booking in db.scalars(select(Booking).where(Booking.status == "active")).all():
active_by_vehicle.setdefault(booking.vehicle_id, []).append(booking)
open_high_by_vehicle = {
row[0]
for row in db.execute(
select(DataQualityIssue.entity_id).where(
DataQualityIssue.entity_type == "vehicle",
DataQualityIssue.status == "open",
DataQualityIssue.severity == "high",
)
).all()
}
for vehicle in vehicles:
has_active_booking = vehicle.id in active_by_vehicle
reason = None
if vehicle.operational_status == "available" and has_active_booking:
reason = "marked available while an active booking exists"
elif vehicle.operational_status == "rented" and not has_active_booking:
reason = "marked rented without an active booking"
elif vehicle.operational_status == "available" and vehicle.id in open_high_by_vehicle:
reason = "marked available while a high-severity quality issue is open"
elif vehicle.operational_status == "maintenance" and has_active_booking:
reason = "marked maintenance while an active booking exists"
if reason:
_open_issue(
db,
scan,
rule_type="vehicle_status_conflict",
entity_type="vehicle",
entity_id=vehicle.id,
severity="high",
summary=f"Vehicle {reason}",
entity_ref=vehicle.public_ref,
related_refs=[],
)
def _scan_odometer_regressions(db: Session, scan: ScanResult) -> None:
# The seed dataset's vehicle.odometer_km is generated independently of booking
# history, so comparing every historical booking against it produces near-universal
# false positives. Instead check the booking sequence's own internal consistency:
# each vehicle's completed bookings should show a non-decreasing odometer reading.
vehicles = {v.id: v for v in db.scalars(select(Vehicle)).all()}
bookings_by_vehicle: dict[uuid.UUID, list[Booking]] = {}
for booking in db.scalars(
select(Booking).where(
Booking.status == "returned", Booking.end_odometer_km.is_not(None)
)
).all():
bookings_by_vehicle.setdefault(booking.vehicle_id, []).append(booking)
for vehicle_id, bookings in bookings_by_vehicle.items():
bookings.sort(key=lambda b: b.ends_at)
for earlier, later in zip(bookings, bookings[1:], strict=False):
if later.end_odometer_km < earlier.end_odometer_km:
vehicle = vehicles[vehicle_id]
_open_issue(
db,
scan,
rule_type="odometer_regression",
entity_type="vehicle",
entity_id=vehicle_id,
severity="medium",
summary=(
f"Booking {later.public_ref} recorded {later.end_odometer_km} km, "
f"below the {earlier.end_odometer_km} km recorded by earlier "
f"booking {earlier.public_ref}."
),
entity_ref=vehicle.public_ref,
related_refs=[earlier.public_ref, later.public_ref],
)
break
def run_scan(db: Session) -> ScanResult:
scan = ScanResult()
_scan_duplicate_customers(db, scan)
_scan_missing_required_fields(db, scan)
_scan_odometer_regressions(db, scan)
_scan_booking_overlaps(db, scan)
_scan_vehicle_status_conflicts(db, scan)
db.commit()
return scan
def _load_open_issue(db: Session, public_ref: str) -> DataQualityIssue:
issue = db.scalar(
select(DataQualityIssue).where(DataQualityIssue.public_ref == public_ref)
)
if issue is None:
raise AppError("ISSUE_NOT_FOUND", "Data quality issue not found.", status_code=404)
if issue.status != "open":
raise AppError(
"ISSUE_NOT_OPEN",
f"Issue is '{issue.status}', not 'open'.",
status_code=409,
)
return issue
def defer_issue(db: Session, public_ref: str, actor: CurrentUser) -> DataQualityIssue:
issue = _load_open_issue(db, public_ref)
issue.status = "deferred"
issue.resolved_at = datetime.now(UTC)
issue.resolved_by = actor.display_name
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="data_quality_issue_deferred",
entity_type="data_quality_issue",
entity_id=issue.id,
before={"status": "open"},
after={"status": "deferred"},
)
db.commit()
return issue
def reject_issue(db: Session, public_ref: str, actor: CurrentUser) -> DataQualityIssue:
issue = _load_open_issue(db, public_ref)
issue.status = "rejected"
issue.resolved_at = datetime.now(UTC)
issue.resolved_by = actor.display_name
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="data_quality_issue_rejected",
entity_type="data_quality_issue",
entity_id=issue.id,
before={"status": "open"},
after={"status": "rejected"},
)
db.commit()
return issue
MERGEABLE_FIELDS = ("first_name", "last_name", "email", "phone", "postal_code", "city")
def merge_customers(
db: Session,
public_ref: str,
survivor_ref: str,
field_overrides: dict[str, str] | None,
actor: CurrentUser,
) -> dict:
issue = _load_open_issue(db, public_ref)
if issue.rule_type != "possible_duplicate_customer":
raise AppError(
"NOT_A_DUPLICATE_ISSUE",
"This issue is not a possible-duplicate-customer issue.",
status_code=409,
)
entity_ref = issue.evidence_json.get("entity_ref")
related_refs = issue.evidence_json.get("related_refs", [])
candidate_refs = {entity_ref, *related_refs}
if survivor_ref not in candidate_refs:
raise AppError(
"INVALID_SURVIVOR",
"The survivor reference must be one of the two customers in this issue.",
status_code=422,
details={"candidates": sorted(candidate_refs)},
)
loser_ref = next(ref for ref in candidate_refs if ref != survivor_ref)
survivor = db.scalar(select(Customer).where(Customer.public_ref == survivor_ref))
loser = db.scalar(select(Customer).where(Customer.public_ref == loser_ref))
if survivor is None or loser is None:
raise AppError(
"CUSTOMER_NOT_FOUND", "One of the customers could not be found.", status_code=404
)
before = {
"survivor": {f: getattr(survivor, f) for f in MERGEABLE_FIELDS},
"loser": {f: getattr(loser, f) for f in MERGEABLE_FIELDS},
}
for field_name, value in (field_overrides or {}).items():
if field_name not in MERGEABLE_FIELDS:
raise AppError(
"INVALID_FIELD_OVERRIDE", f"Field '{field_name}' cannot be merged.", status_code=422
)
setattr(survivor, field_name, value)
rewired = db.execute(
Booking.__table__.update()
.where(Booking.customer_id == loser.id)
.values(customer_id=survivor.id)
)
loser.merged_into_customer_id = survivor.id
issue.status = "resolved"
issue.resolved_at = datetime.now(UTC)
issue.resolved_by = actor.display_name
record_audit_event(
db,
actor_type="user",
actor_label=actor.display_name,
action="customer_merged",
entity_type="customer",
entity_id=survivor.id,
before=before,
after={"survivor": {f: getattr(survivor, f) for f in MERGEABLE_FIELDS}},
metadata={
"loser_ref": loser_ref,
"survivor_ref": survivor_ref,
"rewired_bookings": rewired.rowcount,
},
)
db.commit()
return {
"issue_ref": issue.public_ref,
"survivor_ref": survivor_ref,
"loser_ref": loser_ref,
"rewired_bookings": rewired.rowcount,
}
+106
View File
@@ -0,0 +1,106 @@
def test_list_includes_all_five_rule_types(ops_client):
response = ops_client.get("/api/v1/data-quality/issues")
assert response.status_code == 200
issues = response.json()
rule_types = {i["rule_type"] for i in issues}
assert rule_types == {
"possible_duplicate_customer",
"missing_required_field",
"odometer_regression",
"booking_overlap",
"vehicle_status_conflict",
}
def test_scan_is_idempotent_once_seeded(ops_client):
# The session fixture already ran a scan as part of seeding; running again
# must not create duplicate open issues for the same (rule_type, entity).
response = ops_client.post("/api/v1/data-quality/scan")
assert response.status_code == 200
assert response.json()["created"] == {}
def test_scan_requires_operations_manager(employee_client):
response = employee_client.post("/api/v1/data-quality/scan")
assert response.status_code == 403
def test_s2_duplicate_customer_issue_detail(ops_client):
response = ops_client.get("/api/v1/data-quality/issues/DQ-DEMO-DUPLICATE")
assert response.status_code == 200
body = response.json()
assert body["entity_ref"] == "CUS-0012"
assert body["entity_snapshot"]["public_ref"] == "CUS-0012"
assert [s["public_ref"] for s in body["related_snapshots"]] == ["CUS-0178"]
def test_s4_booking_overlap_issue_detail(ops_client):
response = ops_client.get("/api/v1/data-quality/issues/DQ-DEMO-OVERLAP")
assert response.status_code == 200
body = response.json()
assert body["entity_ref"] == "MO-016"
assert set(body["evidence"]["related_refs"]) == {"BK-DEMO-OVERLAP-A", "BK-DEMO-OVERLAP-B"}
def test_defer_then_reject_are_rejected_on_closed_issue(ops_client):
issues = ops_client.get(
"/api/v1/data-quality/issues",
params={"rule_type": "missing_required_field", "status": "open"},
).json()
target = issues[0]["public_ref"]
deferred = ops_client.post(f"/api/v1/data-quality/issues/{target}/defer")
assert deferred.status_code == 200
assert deferred.json()["status"] == "deferred"
again = ops_client.post(f"/api/v1/data-quality/issues/{target}/reject")
assert again.status_code == 409
assert again.json()["error"]["code"] == "ISSUE_NOT_OPEN"
def test_merge_customers_requires_operations_manager(employee_client):
response = employee_client.post(
"/api/v1/data-quality/issues/DQ-DEMO-DUPLICATE/merge-customers",
json={"survivor_ref": "CUS-0012"},
)
assert response.status_code == 403
def test_merge_customers_rejects_unrelated_survivor(ops_client):
response = ops_client.post(
"/api/v1/data-quality/issues/DQ-DEMO-DUPLICATE/merge-customers",
json={"survivor_ref": "CUS-0099"},
)
assert response.status_code == 422
assert response.json()["error"]["code"] == "INVALID_SURVIVOR"
def test_merge_customers_s2_scenario_rewires_and_audits(ops_client):
before_bookings = ops_client.get(
"/api/v1/bookings", params={"vehicle_ref": "MO-001"}
) # warm the client session; irrelevant vehicle, just a cheap authenticated call
assert before_bookings.status_code == 200
response = ops_client.post(
"/api/v1/data-quality/issues/DQ-DEMO-DUPLICATE/merge-customers",
json={"survivor_ref": "CUS-0012", "field_overrides": {"city": "Turnhout"}},
)
assert response.status_code == 200
body = response.json()
assert body["survivor_ref"] == "CUS-0012"
assert body["loser_ref"] == "CUS-0178"
issue = ops_client.get("/api/v1/data-quality/issues/DQ-DEMO-DUPLICATE").json()
assert issue["status"] == "resolved"
audit_events = ops_client.get(
"/api/v1/audit", params={"action": "customer_merged"}
).json()
assert len(audit_events) >= 1
# Already-resolved issue cannot be merged again.
replay = ops_client.post(
"/api/v1/data-quality/issues/DQ-DEMO-DUPLICATE/merge-customers",
json={"survivor_ref": "CUS-0012"},
)
assert replay.status_code == 409
+2 -1
View File
@@ -20,7 +20,8 @@ def test_seed_counts_match_deterministic_dataset():
assert db.scalar(select(func.count()).select_from(Vehicle)) == 50
assert db.scalar(select(func.count()).select_from(Customer)) == 180
assert db.scalar(select(func.count()).select_from(Booking)) == 246
assert db.scalar(select(func.count()).select_from(DataQualityIssue)) == 15
# 15 from the CSV plus a deterministic set discovered by the post-seed scan.
assert db.scalar(select(func.count()).select_from(DataQualityIssue)) == 26
assert db.scalar(select(func.count()).select_from(OutboxEvent)) == 20
assert db.scalar(select(func.count()).select_from(User)) == 2
finally:
+4
View File
@@ -8,6 +8,8 @@ import { Vehicles } from "./pages/Vehicles";
import { VehicleDetail } from "./pages/VehicleDetail";
import { Bookings } from "./pages/Bookings";
import { BookingDetail } from "./pages/BookingDetail";
import { DataQuality } from "./pages/DataQuality";
import { DataQualityIssueDetail } from "./pages/DataQualityIssueDetail";
import { Audit } from "./pages/Audit";
export function App() {
@@ -27,6 +29,8 @@ export function App() {
<Route path="/vehicles/:publicRef" element={<VehicleDetail />} />
<Route path="/bookings" element={<Bookings />} />
<Route path="/bookings/:publicRef" element={<BookingDetail />} />
<Route path="/data-quality" element={<DataQuality />} />
<Route path="/data-quality/:publicRef" element={<DataQualityIssueDetail />} />
<Route path="/audit" element={<Audit />} />
</Route>
<Route path="/" element={<Navigate to="/dashboard" replace />} />
+23
View File
@@ -92,6 +92,7 @@ export interface AttentionItem {
detail: string;
link_type: "vehicle" | "booking" | "customer";
link_ref: string;
issue_ref: string | null;
}
export interface TodayItem {
@@ -144,6 +145,28 @@ export interface RegisterReturnResult {
next_booking_risk: NextBookingRisk | null;
}
export interface EntitySnapshot {
public_ref: string;
[key: string]: unknown;
}
export interface DataQualityIssueDetail extends DataQualityIssue {
entity_snapshot: EntitySnapshot | null;
related_snapshots: EntitySnapshot[];
}
export interface MergeCustomersRequest {
survivor_ref: string;
field_overrides?: Record<string, string>;
}
export interface MergeCustomersResult {
issue_ref: string;
survivor_ref: string;
loser_ref: string;
rewired_bookings: number;
}
export interface AuditEvent {
id: string;
actor_type: string;
+1
View File
@@ -5,6 +5,7 @@ const NAV_ITEMS = [
{ to: "/dashboard", label: "Dashboard" },
{ to: "/vehicles", label: "Vehicles" },
{ to: "/bookings", label: "Bookings" },
{ to: "/data-quality", label: "Data Quality" },
{ to: "/audit", label: "Audit" },
];
+3 -1
View File
@@ -53,7 +53,9 @@ export function Dashboard() {
<SeverityBadge severity={item.severity} />
<div>
<p className="attention-title">
{item.link_type === "vehicle" ? (
{item.issue_ref ? (
<Link to={`/data-quality/${item.issue_ref}`}>{item.title}</Link>
) : item.link_type === "vehicle" ? (
<Link to={`/vehicles/${item.link_ref}`}>{item.title}</Link>
) : (
item.title
+96
View File
@@ -0,0 +1,96 @@
import { useEffect, useState } from "react";
import { Link } from "react-router-dom";
import { api } from "../api/client";
import type { DataQualityIssue } from "../api/types";
import { SeverityBadge, StatusBadge } from "../components/Badge";
const RULE_TYPES = [
"possible_duplicate_customer",
"missing_required_field",
"odometer_regression",
"booking_overlap",
"vehicle_status_conflict",
];
export function DataQuality() {
const [issues, setIssues] = useState<DataQualityIssue[] | null>(null);
const [error, setError] = useState<string | null>(null);
const [status, setStatus] = useState("open");
const [ruleType, setRuleType] = useState("");
useEffect(() => {
const params = new URLSearchParams();
if (status) params.set("status", status);
if (ruleType) params.set("rule_type", ruleType);
api
.get<DataQualityIssue[]>(`/api/v1/data-quality/issues?${params.toString()}`)
.then(setIssues)
.catch(() => setError("Data-quality issues are unavailable right now."));
}, [status, ruleType]);
return (
<div className="page">
<h1>Data Quality</h1>
<form className="filters" aria-label="Filter data-quality issues">
<label>
Status
<select value={status} onChange={(e) => setStatus(e.target.value)}>
<option value="">All statuses</option>
<option value="open">Open</option>
<option value="deferred">Deferred</option>
<option value="resolved">Resolved</option>
<option value="rejected">Rejected</option>
</select>
</label>
<label>
Rule type
<select value={ruleType} onChange={(e) => setRuleType(e.target.value)}>
<option value="">All rule types</option>
{RULE_TYPES.map((r) => (
<option key={r} value={r}>
{r.replace(/_/g, " ")}
</option>
))}
</select>
</label>
</form>
{error && <p className="error" role="alert">{error}</p>}
{!error && !issues && <p>Loading issues</p>}
{issues && issues.length === 0 && <p>No issues match these filters.</p>}
{issues && issues.length > 0 && (
<table className="data-table">
<caption className="visually-hidden">Data-quality issues</caption>
<thead>
<tr>
<th scope="col">Reference</th>
<th scope="col">Rule</th>
<th scope="col">Entity</th>
<th scope="col">Severity</th>
<th scope="col">Status</th>
</tr>
</thead>
<tbody>
{issues.map((i) => (
<tr key={i.public_ref}>
<th scope="row">
<Link to={`/data-quality/${i.public_ref}`}>{i.public_ref}</Link>
</th>
<td>{i.rule_type.replace(/_/g, " ")}</td>
<td>{i.entity_ref}</td>
<td>
<SeverityBadge severity={i.severity} />
</td>
<td>
<StatusBadge status={i.status} />
</td>
</tr>
))}
</tbody>
</table>
)}
</div>
);
}
@@ -0,0 +1,242 @@
import { useCallback, useEffect, useState } from "react";
import { Link, useParams } from "react-router-dom";
import { api, ApiError } from "../api/client";
import type { DataQualityIssueDetail as IssueDetail, EntitySnapshot } from "../api/types";
import { SeverityBadge, StatusBadge } from "../components/Badge";
import { useAuth } from "../context/AuthContext";
const MERGE_FIELDS = ["first_name", "last_name", "email", "phone", "postal_code", "city"];
function DuplicateCustomerPanel({ issue, onResolved }: { issue: IssueDetail; onResolved: () => void }) {
const { user } = useAuth();
const [survivorRef, setSurvivorRef] = useState(issue.entity_snapshot?.public_ref ?? "");
const [fieldChoices, setFieldChoices] = useState<Record<string, "a" | "b">>({});
const [error, setError] = useState<string | null>(null);
const [submitting, setSubmitting] = useState(false);
const [confirming, setConfirming] = useState(false);
if (!issue.entity_snapshot || !issue.related_snapshots[0]) {
return <p className="error">Both customers in this comparison could not be loaded.</p>;
}
const a: EntitySnapshot = issue.entity_snapshot;
const b: EntitySnapshot = issue.related_snapshots[0];
const survivor = survivorRef === a.public_ref ? a : b;
const loser = survivorRef === a.public_ref ? b : a;
async function handleMerge() {
setError(null);
setSubmitting(true);
try {
const overrides: Record<string, string> = {};
for (const field of MERGE_FIELDS) {
const choice = fieldChoices[field];
const chosenSide = choice === "a" ? a : choice === "b" ? b : survivor;
if (chosenSide !== survivor && chosenSide[field]) {
overrides[field] = String(chosenSide[field]);
}
}
await api.post(`/api/v1/data-quality/issues/${issue.public_ref}/merge-customers`, {
survivor_ref: survivor.public_ref,
field_overrides: Object.keys(overrides).length > 0 ? overrides : undefined,
});
onResolved();
} catch (err) {
setError(err instanceof ApiError ? err.message : "Could not merge these customers.");
setConfirming(false);
} finally {
setSubmitting(false);
}
}
if (user?.role !== "operations_manager") {
return (
<p className="panel">
Merging duplicate customers requires the Operations Manager role. Switch role to resolve
this issue.
</p>
);
}
return (
<section className="panel duplicate-compare" aria-labelledby="compare-heading">
<h2 id="compare-heading">Compare and merge</h2>
{error && <p className="error" role="alert">{error}</p>}
<fieldset>
<legend>Keep as survivor</legend>
<label className="checkbox-label">
<input
type="radio"
name="survivor"
checked={survivorRef === a.public_ref}
onChange={() => setSurvivorRef(a.public_ref)}
/>
{a.public_ref}
</label>
<label className="checkbox-label">
<input
type="radio"
name="survivor"
checked={survivorRef === b.public_ref}
onChange={() => setSurvivorRef(b.public_ref)}
/>
{b.public_ref}
</label>
</fieldset>
<table className="data-table compare-table">
<thead>
<tr>
<th scope="col">Field</th>
<th scope="col">{a.public_ref}</th>
<th scope="col">{b.public_ref}</th>
</tr>
</thead>
<tbody>
{MERGE_FIELDS.map((field) => {
const valueA = a[field] ? String(a[field]) : "—";
const valueB = b[field] ? String(b[field]) : "—";
const differ = valueA !== valueB;
return (
<tr key={field}>
<th scope="row">{field.replace(/_/g, " ")}</th>
<td>
{differ ? (
<label className="checkbox-label">
<input
type="radio"
name={`field-${field}`}
checked={(fieldChoices[field] ?? "a") === "a"}
onChange={() => setFieldChoices((c) => ({ ...c, [field]: "a" }))}
/>
{valueA}
</label>
) : (
valueA
)}
</td>
<td>
{differ ? (
<label className="checkbox-label">
<input
type="radio"
name={`field-${field}`}
checked={fieldChoices[field] === "b"}
onChange={() => setFieldChoices((c) => ({ ...c, [field]: "b" }))}
/>
{valueB}
</label>
) : (
valueB
)}
</td>
</tr>
);
})}
</tbody>
</table>
<p>
<strong>{loser.public_ref}</strong> will become a tombstone linked to{" "}
<strong>{survivor.public_ref}</strong>; its bookings will be rewired to the survivor.
</p>
{!confirming && (
<button type="button" onClick={() => setConfirming(true)}>
Merge into {survivor.public_ref}
</button>
)}
{confirming && (
<div className="confirm-bar" role="alertdialog" aria-label="Confirm merge">
<p>
Merge {loser.public_ref} into {survivor.public_ref}? This cannot be undone.
</p>
<button type="button" onClick={handleMerge} disabled={submitting}>
{submitting ? "Merging…" : "Yes, merge"}
</button>
<button type="button" onClick={() => setConfirming(false)} disabled={submitting}>
Cancel
</button>
</div>
)}
</section>
);
}
export function DataQualityIssueDetail() {
const { publicRef } = useParams<{ publicRef: string }>();
const [issue, setIssue] = useState<IssueDetail | null>(null);
const [error, setError] = useState<string | null>(null);
const [actionError, setActionError] = useState<string | null>(null);
const load = useCallback(() => {
if (!publicRef) return;
api
.get<IssueDetail>(`/api/v1/data-quality/issues/${publicRef}`)
.then(setIssue)
.catch(() => setError("This issue could not be found."));
}, [publicRef]);
useEffect(() => {
setIssue(null);
setError(null);
load();
}, [load]);
async function handleAction(action: "defer" | "reject") {
if (!issue) return;
setActionError(null);
try {
await api.post(`/api/v1/data-quality/issues/${issue.public_ref}/${action}`);
load();
} catch (err) {
setActionError(err instanceof ApiError ? err.message : `Could not ${action} this issue.`);
}
}
if (error) return <p className="error" role="alert">{error}</p>;
if (!issue) return <p>Loading issue</p>;
return (
<div className="page">
<p><Link to="/data-quality"> Back to data quality</Link></p>
<h1>{issue.public_ref}</h1>
<p>
<SeverityBadge severity={issue.severity} /> <StatusBadge status={issue.status} />
</p>
<dl className="detail-grid">
<div><dt>Rule</dt><dd>{issue.rule_type.replace(/_/g, " ")}</dd></div>
<div><dt>Entity</dt><dd>{issue.entity_ref}</dd></div>
<div><dt>Evidence</dt><dd>{String(issue.evidence.summary ?? "")}</dd></div>
</dl>
{actionError && <p className="error" role="alert">{actionError}</p>}
{issue.status === "open" && issue.rule_type === "possible_duplicate_customer" && (
<DuplicateCustomerPanel issue={issue} onResolved={load} />
)}
{issue.status === "open" && (
<section className="panel" aria-labelledby="resolution-heading">
<h2 id="resolution-heading">Resolution</h2>
{issue.rule_type !== "possible_duplicate_customer" && (
<>
<p>Evidence for this issue:</p>
<pre className="evidence-block">{JSON.stringify(issue.evidence, null, 2)}</pre>
</>
)}
<p>Defer to review later, or reject if this is not a real issue.</p>
<div className="resolution-actions">
<button type="button" onClick={() => handleAction("defer")}>
Defer
</button>
<button type="button" onClick={() => handleAction("reject")}>
Reject
</button>
</div>
</section>
)}
</div>
);
}
+1
View File
@@ -112,6 +112,7 @@ export function VehicleDetail() {
{vehicle.quality_issues.length === 0 && <li>No quality issues recorded.</li>}
{vehicle.quality_issues.map((q) => (
<li key={q.public_ref}>
<Link to={`/data-quality/${q.public_ref}`}>{q.public_ref}</Link>
<SeverityBadge severity={q.severity} />
<span>{q.rule_type.replace(/_/g, " ")}</span>
<StatusBadge status={q.status} />
+32
View File
@@ -188,6 +188,38 @@ a { color: #1f5c8f; }
.return-form button:disabled { opacity: 0.6; cursor: not-allowed; }
.return-result .error { margin-top: 12px; }
.duplicate-compare fieldset {
border: 1px solid #dce3eb; border-radius: 10px; padding: 12px 16px; margin: 12px 0;
}
.duplicate-compare legend { font-weight: 700; color: #375065; padding: 0 6px; }
.duplicate-compare fieldset label { margin-right: 20px; }
.compare-table th, .compare-table td { vertical-align: top; }
.compare-table label { display: inline-flex; align-items: center; gap: 6px; font-weight: 400; }
.duplicate-compare > button {
padding: 12px 20px; border-radius: 10px; border: none;
background: #14324f; color: white; font-weight: 700; cursor: pointer; margin-top: 12px;
}
.confirm-bar {
margin-top: 12px; padding: 14px; border-radius: 10px;
background: #fdf1de; border: 1px solid #f0d29e;
}
.confirm-bar p { margin: 0 0 10px; font-weight: 600; color: #8a5a10; }
.confirm-bar button {
padding: 10px 16px; border-radius: 8px; border: none; font-weight: 700; cursor: pointer; margin-right: 8px;
}
.confirm-bar button:first-of-type { background: #9a2530; color: white; }
.confirm-bar button:last-of-type { background: white; border: 1px solid #cfd8e2; }
.evidence-block {
background: #f6f8fb; border: 1px solid #dce3eb; border-radius: 10px;
padding: 12px; overflow-x: auto; font-size: 0.85rem;
}
.resolution-actions { display: flex; gap: 10px; margin-top: 12px; }
.resolution-actions button {
padding: 10px 18px; border-radius: 8px; border: 1px solid #cfd8e2;
background: white; font-weight: 700; cursor: pointer;
}
@media (max-width: 700px) {
.app-header { flex-direction: column; align-items: flex-start; }
.user-badge { margin-left: 0; }