M44: harden release integrity and assurance
This commit is contained in:
NuklearRabbit
@@ -1,265 +1,38 @@
|
||||
# MobilityOps — final acceptance audit summary
|
||||
# Fleet Ops release acceptance
|
||||
|
||||
This audit was run after M0–M7 had already been implemented and committed, specifically
|
||||
to independently re-verify the finished system end to end rather than trust the
|
||||
milestone-by-milestone build log. It found and fixed one real category of defect
|
||||
(`mypy` had never been run across the whole build) and confirmed everything else — every
|
||||
user journey, every button/filter/form, both external-dependency degraded modes, secret
|
||||
hygiene, and the clean-checkout path — works as documented.
|
||||
This file is release-scoped evidence, not a timeless claim. Older evidence under
|
||||
`artifacts/evidence/` is historical. Exact commands and production revisions are recorded
|
||||
in `PROJECT_STATE.md`.
|
||||
|
||||
## Final commit
|
||||
## 2026-08-21 release candidate
|
||||
|
||||
This audit's fixes are committed as the commit immediately following
|
||||
`108b5d04fc6f7c5ff9c47009032d6469df29cf3c` ("M7: portfolio polish and final acceptance").
|
||||
Run `git log -1 --format="%H %s"` for the exact hash.
|
||||
- Backend: **271/271** tests passed against an isolated clean PostgreSQL database.
|
||||
- Browser acceptance: **155/155** Chromium tests passed in 4.6 minutes.
|
||||
- Live-safe browser canary: **4/4** passed across Chromium and Firefox against the local
|
||||
deployed stack; unlike the acceptance suite, it never resets or mutates demo records.
|
||||
- Accessibility: the principal login, dashboard, data-quality, knowledge, automation and
|
||||
audit routes have no automated critical/serious WCAG 2 A/AA/2.1 AA violations.
|
||||
- Frontend: TypeScript, ESLint, production build, dependency audit and per-asset JS/CSS
|
||||
budgets passed; committed visual baselines cover the public entry and engineering story.
|
||||
- Contracts: committed OpenAPI, event schema, MCP tools and all five n8n definitions match
|
||||
their code/manifest sources.
|
||||
- Recovery: a custom-format PostgreSQL dump was restored into a disposable database; the
|
||||
Alembic revision and non-zero canonical table counts matched the source database.
|
||||
- Operations: Prometheus/Alertmanager configuration validation passed, including the
|
||||
watchdog and authenticated n8n receiver route.
|
||||
|
||||
## Exact commands executed
|
||||
## Evidence boundary
|
||||
|
||||
Clean-checkout drill (run twice during this audit, most recently against fully wiped
|
||||
Docker volumes):
|
||||
The complete local suite uses the deterministic provider and an isolated database so it is
|
||||
repeatable and safely destructive. Production verification is deliberately smaller and
|
||||
non-destructive; it verifies the real RAGcore/MCP/n8n health surfaces without resetting the
|
||||
shared demo. A successful local result is never presented as proof that an external service
|
||||
was live. The production subsection is added only after the exact committed release is
|
||||
deployed and observed.
|
||||
|
||||
```bash
|
||||
git status # working tree clean before starting
|
||||
docker compose down -v # wipe all volumes — genuinely clean state
|
||||
cp .env.example .env
|
||||
docker compose up --build -d # migrations run automatically (backend/entrypoint.sh)
|
||||
docker compose exec api python -m app.cli seed --reset
|
||||
docker compose run --rm api pytest -q
|
||||
docker compose run --rm api ruff check .
|
||||
docker compose run --rm api mypy app
|
||||
cd frontend && npm run build
|
||||
cd frontend && npx playwright test
|
||||
```
|
||||
## Remaining product boundary
|
||||
|
||||
n8n one-time setup (owner account via browser at `http://localhost:5678/setup`, then):
|
||||
|
||||
```bash
|
||||
docker compose exec n8n n8n import:workflow --input=//imports/mobilityops-return-processing.json
|
||||
docker compose exec n8n n8n publish:workflow --id=mobilityops-return-processing
|
||||
docker compose restart n8n
|
||||
```
|
||||
|
||||
Degraded-mode drills:
|
||||
|
||||
```bash
|
||||
docker compose stop n8n # then register a return via the API — commits, event stays pending
|
||||
docker compose start n8n # dispatcher self-heals, no manual intervention
|
||||
docker compose run --rm -e KNOWLEDGE_PROVIDER=ragcore -e RAGCORE_BASE_URL=http://ragcore-not-reachable:9999 \
|
||||
api python -c "from app.services.knowledge import get_knowledge_provider; ..."
|
||||
```
|
||||
|
||||
## Test and validation results
|
||||
|
||||
| Check | Command | Result |
|
||||
|---|---|---|
|
||||
| Backend unit/integration tests | `docker compose run --rm api pytest -q` | **66 passed**, 0 failed, 0 skipped |
|
||||
| Backend lint | `docker compose run --rm api ruff check .` | **All checks passed** |
|
||||
| Backend type check | `docker compose run --rm api mypy app` | **Success: no issues found in 44 source files** (found and fixed 43 pre-existing errors this audit — see below) |
|
||||
| Frontend build + typecheck | `cd frontend && npm run build` | Clean (`tsc -b && vite build`, zero errors) |
|
||||
| End-to-end (Playwright) | `cd frontend && npx playwright test` | **12 passed** (`demo.spec.ts` — full 9-step demo script; `interactive-elements.spec.ts` — 11 tests covering every nav item, filter, tab, and role boundary) |
|
||||
| Clean-checkout migrations | `docker compose down -v && docker compose up --build -d` | 11 tables created automatically, `alembic current` → `e7b08389f47f (head)`, zero manual step |
|
||||
| Deterministic seed | `docker compose exec api python -m app.cli seed --reset` | `users:2 customers:180 vehicles:50 bookings:246 inspections:75 maintenance:40 data_quality_issues:26 workflow_runs:20` — identical across every reseed this session |
|
||||
| Secret scan | `git ls-files \| grep -x .env`; `git log --all -p -- '*.env'`; history grep for AWS/private-key/`sk-` patterns | No `.env` ever committed; no secrets found in history |
|
||||
|
||||
### mypy defects found and fixed (the one real gap this audit uncovered)
|
||||
|
||||
`mypy` is a declared dev dependency (`backend/pyproject.toml`) but was never added to any
|
||||
milestone's validation loop — only `ruff` was run throughout M0–M7. Running it cold
|
||||
surfaced 43 errors across 10 files. All were triaged and fixed (not suppressed):
|
||||
|
||||
- **Two genuine defensive-programming gaps**, not just type-annotation issues:
|
||||
- `app/services/returns.py`: the vehicle lookup after acquiring the row lock had no
|
||||
`None` guard; a dangling FK would have crashed with an unhandled 500 instead of a
|
||||
clean `404 VEHICLE_NOT_FOUND`. Fixed.
|
||||
- `app/api/routers/bookings.py`: same pattern in `get_booking` for the customer/vehicle
|
||||
lookups — now returns a clean `500` with a message instead of an `AttributeError`.
|
||||
- `app/api/deps.py` / `app/api/routers/demo.py`: `CurrentUser.role` is validated by
|
||||
Pydantic at runtime already, but `get_current_user` now explicitly checks role
|
||||
membership before construction, turning a would-be unhandled `ValidationError` into a
|
||||
clean `401` for a corrupted/tampered session cookie.
|
||||
- Two instances of reusing one variable name for both a `Vehicle` and a `Customer` across
|
||||
branches (`dashboard.py`, `data_quality.py`) — renamed for clarity, not just to satisfy
|
||||
mypy.
|
||||
- `Booking.__table__.update()` / `Customer.__table__.update()` switched to the idiomatic
|
||||
`sqlalchemy.update(Model)` construct (also fixes the type error).
|
||||
- Remainder: deprecated `conint()` → `Annotated[int, Field(...)]`, a `Sequence` vs `list`
|
||||
`.sort()` call, an `assert`-guarded None-narrow after a `WHERE ... IS NOT NULL` filter
|
||||
mypy can't see through, and a couple of narrowly-scoped `# type: ignore[...]` comments
|
||||
for known SQLAlchemy stub gaps (`Result.rowcount`).
|
||||
|
||||
`make lint` now runs `ruff check .` **and** `mypy app`.
|
||||
|
||||
## Application URLs and ports
|
||||
|
||||
| Service | URL | Notes |
|
||||
|---|---|---|
|
||||
| Web (React SPA) | `http://localhost:1228` | nginx-served static build |
|
||||
| API | `http://localhost:8128` | FastAPI, `/health` for liveness |
|
||||
| API docs | `http://localhost:8128/docs` | auto-generated OpenAPI/Swagger UI |
|
||||
| n8n | `http://localhost:5678` | requires one-time owner setup, see below |
|
||||
| PostgreSQL | `localhost:5432` (container-internal only, no host port published) | |
|
||||
|
||||
## Demo users and access method
|
||||
|
||||
No passwords. Two demo-role buttons on `http://localhost:1228/login`:
|
||||
|
||||
- **Open as Operations Manager** → `USR-OPS`, "Amelie De Ridder". Full access: dashboard,
|
||||
data-quality resolution/merge, automation retry, demo reset, MCP/service-token routes
|
||||
are separate (not user-facing).
|
||||
- **Open as Rental Employee** → `USR-EMP`, "Karim Boujaddaine". Can register returns and
|
||||
browse vehicles/bookings/knowledge; Automation page is visible but shows a
|
||||
role-restricted message instead of the delivery table (enforced both in the UI and by
|
||||
the backend's `require_operations_manager` dependency — verified by
|
||||
`test_retry_requires_operations_manager` and the e2e role-restriction test).
|
||||
|
||||
Session is an HMAC-signed, `HttpOnly` cookie (`app/core/security.py`) — a demo mechanism,
|
||||
not a real identity provider (documented as a known limitation).
|
||||
|
||||
## Implemented functionality
|
||||
|
||||
- Operations dashboard with 100% database-backed metrics, attention items linking to the
|
||||
underlying data-quality issue, "today" departures/returns, and recent automation runs.
|
||||
- Vehicle and booking list/detail pages with working filters (status, attention-only) and
|
||||
a tabbed vehicle detail view (overview/bookings/inspections/maintenance/quality).
|
||||
- Full transactional vehicle-return workflow: row-locked, idempotent by
|
||||
`Idempotency-Key`, canonical-odometer regression handling (never silently lowers the
|
||||
canonical value), vehicle status derivation, two audit events, and a schema-compliant
|
||||
outbox event — verified against real concurrent submissions (1×201 + 2×409).
|
||||
- Invalid-mileage rejection: negative values and non-numeric input both correctly
|
||||
rejected with `422` and a precise Pydantic validation message.
|
||||
- Data Quality Workbench: five deterministic rules (duplicate customer via TF-IDF-style
|
||||
weighted signal scoring, missing required field, odometer regression, booking overlap,
|
||||
vehicle status conflict), issue list/detail/defer/reject, and a two-column
|
||||
duplicate-customer compare-and-merge UI with an inline (non-native-dialog) confirmation
|
||||
step, transactional booking rewiring, and audit logging.
|
||||
- Full audit trail: every significant action (login, return, vehicle status change,
|
||||
data-quality issue lifecycle, customer merge, workflow retry, demo reset, n8n
|
||||
callback, MCP tool request, knowledge question) is recorded with actor, correlation ID,
|
||||
and before/after state; filterable by action.
|
||||
- Knowledge Assistant: deterministic TF-IDF-weighted extractive retrieval over the 10
|
||||
procedure documents — never generative, always cites real excerpts, and honestly
|
||||
reports `insufficient`/`unavailable` states rather than fabricating an answer.
|
||||
- n8n automation: background outbox dispatcher (`FOR UPDATE SKIP LOCKED` claim,
|
||||
exponential backoff, no DB transaction held during the HTTP call), a live-verified
|
||||
round trip through an actual n8n workflow, manual retry for failed deliveries, and an
|
||||
Automation page (Operations Manager only) showing all runs with filtering.
|
||||
- Four read-only, service-token-authenticated MCP Hub provider endpoints, each recording
|
||||
its own service-request audit event, with zero write/mutation endpoints anywhere in
|
||||
that namespace.
|
||||
- Responsive UI verified down to 360px width (nav wraps, tables become cards, metric
|
||||
tiles reflow to a 2-column grid, no horizontal overflow) — both by an automated
|
||||
Playwright viewport/overflow assertion and by a captured screenshot.
|
||||
|
||||
## RAGcore integration status: implemented, not live-verified
|
||||
|
||||
The active `KnowledgeProvider` in this environment is `DemoKnowledgeProvider` — fully
|
||||
implemented, fully tested, fully live-verified, and what actually satisfies the
|
||||
knowledge-assistant acceptance criteria. A `RAGcoreKnowledgeProvider` HTTP adapter also
|
||||
exists (`app/services/knowledge/ragcore.py`), targeting a best-effort contract inferred
|
||||
from `contracts/ragcore-contract-assumptions.md` (no live RAGcore API spec was available).
|
||||
Its **unavailable-degradation path is live-verified this audit**: pointed at an
|
||||
unreachable host, it returns `{"evidence_state": "unavailable", "answer": "", "sources":
|
||||
[]}` with no fabrication, exactly as required — but an actual successful round trip
|
||||
against a real RAGcore instance has never been performed, because no such instance was
|
||||
reachable in this environment.
|
||||
|
||||
## MCP Hub integration status: implemented, not live-verified
|
||||
|
||||
All four contracted read-only tools (`mobilityops_get_operations_summary`,
|
||||
`mobilityops_list_attention_vehicles`, `mobilityops_get_vehicle_details`,
|
||||
`mobilityops_search_knowledge`) are implemented as service-token-protected endpoints under
|
||||
`/api/v1/integrations/mcp/`, directly `curl`-verified this audit (auth enforcement,
|
||||
correct data shape, no write methods, service-request audit logging). No live ITWorx MCP
|
||||
Hub instance was reachable in this environment, so an actual Hub-mediated tool call was
|
||||
never performed — only direct calls to MobilityOps's own provider API.
|
||||
|
||||
## n8n integration status: implemented and fully live-verified
|
||||
|
||||
The only external integration with a real, running counterpart service available in this
|
||||
environment. Fully verified this audit, including both success and degraded paths:
|
||||
|
||||
- **Success**: a real return registered via the API was delivered by the background
|
||||
dispatcher to an actual n8n instance (owner account + imported/activated workflow),
|
||||
which called back into MobilityOps and was recorded `succeeded`.
|
||||
- **Degraded mode**: `docker compose stop n8n`, then a return was registered — it
|
||||
**committed successfully** (`201`, booking `status: returned` persisted) exactly as
|
||||
required by the architecture's reliability boundary ("a return command and its outbox
|
||||
event commit in one transaction" and "n8n failure leaves events pending with bounded
|
||||
retries"). The outbox event stayed `pending` with two real `ConnectError`s logged and
|
||||
exponential backoff.
|
||||
- **Self-healing**: restarting n8n required no manual intervention — the background
|
||||
dispatcher picked the pending event back up on its next poll cycle and delivered it to
|
||||
`succeeded` (5 total attempts across the outage).
|
||||
- **Manual retry (S5 scenario)**: a seeded `failed` delivery, retried from the Automation
|
||||
page, moved to `pending` and was delivered to `succeeded` by the live dispatcher within
|
||||
one poll cycle.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- RAGcore and the ITWorx MCP Hub were never reachable in this build/audit environment;
|
||||
both integrations are implemented and tested against inferred/documented contracts but
|
||||
not verified against real instances of those systems (see above).
|
||||
- n8n requires a one-time, per-fresh-environment manual owner-account setup through its
|
||||
own web UI (`http://localhost:5678/setup`) — a property of the n8n 2.x image itself
|
||||
(`N8N_BASIC_AUTH_ACTIVE` no longer gates the UI), not something MobilityOps can bypass.
|
||||
The workflow import/activation itself *is* scripted (`make n8n-setup`).
|
||||
- Demo authentication is an HMAC-signed session cookie tied to two fixed seeded users —
|
||||
appropriate for a PoC, not a production identity provider.
|
||||
- Inspection public references are assigned via a simple `count + 1` sequence, not
|
||||
gap-safe under true concurrent writers (acceptable for this single-tenant demo).
|
||||
- The five data-quality rules use a simplified idempotency key
|
||||
(`rule_type, entity_type, entity_id` while open) rather than the spec's literal
|
||||
evidence-fingerprint scheme — documented rationale in `PROJECT_STATE.md`'s M3 notes.
|
||||
- `npm audit` reports one residual moderate `esbuild`/Vite-8 dev-server-only advisory
|
||||
(fixable only by a Vite major version bump) and one high `react-router` RSC-mode
|
||||
advisory that does not apply to this app (it never uses React Router's RSC/SSR mode).
|
||||
|
||||
## Clean deployment instructions
|
||||
|
||||
```bash
|
||||
git clone <repo> && cd MobilityOps
|
||||
cp .env.example .env
|
||||
make demo # build, start, migrate (automatic), seed
|
||||
```
|
||||
|
||||
One-time n8n setup (only needed for the automation demo path; everything else works
|
||||
without it):
|
||||
|
||||
```bash
|
||||
# open http://localhost:5678/setup in a browser, create any owner account
|
||||
# (8+ chars, 1 number, 1 capital letter — no email verification required)
|
||||
make n8n-setup
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8128/health # {"status":"ok",...}
|
||||
curl -o /dev/null -w "%{http_code}\n" http://localhost:1228/ # 200
|
||||
make test # 66 backend tests
|
||||
make lint # ruff + mypy, zero errors
|
||||
make e2e # 12 Playwright tests (stack must be running)
|
||||
```
|
||||
|
||||
Full detail, recovery expectations, and required operational checks: `docs/17-runbook.md`.
|
||||
|
||||
## Five-minute demonstration flow
|
||||
|
||||
1. Open `http://localhost:1228` → **Open as Operations Manager**.
|
||||
2. **Dashboard**: point out the metrics are live counts (available/rented/cleaning/
|
||||
maintenance/blocked vehicles, open quality issues, pending/failed workflows), and the
|
||||
Attention Required list linking straight to the underlying issues.
|
||||
3. **Vehicles → MO-024** → open the active booking `BK-DEMO-RETURN`, register a return
|
||||
with an odometer reading below MO-024's canonical value → the result panel shows the
|
||||
inspection, the derived vehicle status, the automatically-created data-quality issue,
|
||||
and the queued automation event — canonical odometer is confirmed unchanged.
|
||||
4. **Data Quality → DQ-DEMO-DUPLICATE**: the two-column CUS-0012/CUS-0178 comparison,
|
||||
merge with the inline confirmation step, issue flips to `resolved`.
|
||||
5. **Knowledge**: ask "What must I do when a vehicle returns with damage?" → grounded
|
||||
answer citing both the return and damage-handling procedures with real excerpts.
|
||||
6. **Automation**: filter to `failed`, retry the seeded delivery, watch it succeed within
|
||||
a few seconds via the live n8n instance.
|
||||
7. **Audit**: filter by `return_registered` or `customer_merged` to show every action from
|
||||
this walkthrough is recorded with actor, timestamp, and correlation ID.
|
||||
8. Resize the browser to 360px width to show the responsive layout (nav wraps, tables
|
||||
become cards) — or run `make e2e` and point at the passing responsive assertion.
|
||||
Fleet Ops remains a synthetic single-tenant PoC. It is not a production identity provider,
|
||||
payment system, accounting package or public reservation platform. External RAGcore, MCP
|
||||
Hub and n8n services remain independently operated dependencies and are accessed only
|
||||
through their documented adapters.
|
||||
|
||||
Reference in New Issue
Block a user