feat: scope detection QA to inference coverage
This commit is contained in:
@@ -544,6 +544,15 @@ Sprint 8C makes persisted detections reviewable:
|
||||
- Persisted detection geometries can be returned as GeoJSON FeatureCollections for MapLibre display.
|
||||
- Detection QA compares candidate detection geometries against persisted reference `vector_features`.
|
||||
- QA results reuse `quality_checks` and `metrics`; no parallel QA persistence system is introduced.
|
||||
- Configured-YOLO QA derives its evaluation extent from the persisted tile
|
||||
manifest. Tile bounds are transformed from their explicit source CRS to
|
||||
EPSG:4326, unioned, and used to clip candidate/reference populations before
|
||||
canonical footprint-IoU matching. Reference features wholly outside the
|
||||
imagery presented to the model no longer count as false negatives.
|
||||
- A separate reference-envelope IoU pass is persisted as
|
||||
`box_to_footprint_diagnostics`. It quantifies possible matching artifacts from
|
||||
comparing rectangular detections with irregular building footprints, but is
|
||||
diagnostic only and never changes canonical QA metrics.
|
||||
- Segmentation remains out of scope for Sprint 8C.
|
||||
|
||||
## 2. Tile Metadata
|
||||
|
||||
@@ -872,6 +872,29 @@ Response persists a `quality_check` and `metrics` rows through the existing QA/Q
|
||||
- `false_negatives`
|
||||
- `quality_check_id`
|
||||
|
||||
Configured-YOLO QA automatically reads `tile_manifest_path` from the persisted
|
||||
`AnalysisRun.parameters_json`. Candidate and reference geometries are clipped
|
||||
to the union of the manifest's tile bounds after explicit CRS transformation to
|
||||
EPSG:4326. The response additionally returns:
|
||||
|
||||
- `candidate_feature_count_raw` and `reference_feature_count_raw`;
|
||||
- `coverage`, including raw/evaluated/excluded/boundary-clipped population
|
||||
counts, tile count, source CRS values and coverage mode;
|
||||
- `box_to_footprint_diagnostics`, which compares candidate boxes with reference
|
||||
envelopes at the same IoU threshold.
|
||||
|
||||
The canonical precision, recall, F1 and mean IoU always remain based on
|
||||
candidate geometry versus the persisted reference footprint. Envelope results
|
||||
are explicitly `diagnostic_only` and are persisted in
|
||||
`quality_checks.findings_json`; they never replace or inflate canonical metrics.
|
||||
|
||||
Configured-YOLO QA fails closed with `DETECTION_QA_COVERAGE_UNAVAILABLE` when
|
||||
manifest provenance is absent, `DETECTION_QA_COVERAGE_MISMATCH` when it belongs
|
||||
to another raster, `DETECTION_QA_COVERAGE_INVALID` when bounds/CRS are invalid,
|
||||
or `REFERENCE_FEATURES_OUTSIDE_COVERAGE` when no reference polygons overlap the
|
||||
actual inference coverage. Explicit fixture/legacy runs without a manifest keep
|
||||
the documented unbounded comparison behavior.
|
||||
|
||||
If the reference dataset has no persisted vector features, the endpoint returns `REFERENCE_FEATURES_NOT_FOUND`. It does not calculate fake QA metrics.
|
||||
|
||||
#### Future analysis route: `/api/v1/analysis/building-stats`
|
||||
|
||||
@@ -7556,3 +7556,46 @@ Open:
|
||||
- Clip detection QA reference populations to actual raster/tile coverage and
|
||||
add box-to-building-footprint matching diagnostics before any further model
|
||||
training or promotion decision.
|
||||
|
||||
# Sprint 184 - Detection QA coverage and matching diagnostics
|
||||
|
||||
## Implementation
|
||||
|
||||
- Added a focused detection-QA coverage service that resolves the persisted
|
||||
tile manifest from the analysis run, validates dataset ownership and CRS,
|
||||
transforms each tile footprint to EPSG:4326 and unions the exact inference
|
||||
coverage.
|
||||
- Configured-YOLO QA now fails closed when persisted manifest provenance is
|
||||
absent, invalid or belongs to another raster. Explicit fixture and legacy
|
||||
runs without a manifest retain their existing unbounded QA behavior.
|
||||
- Candidate Detection geometries and persisted reference VectorFeature
|
||||
geometries are clipped to inference coverage before the existing one-to-one
|
||||
IoU matcher runs. Raw, evaluated, excluded-outside and boundary-clipped
|
||||
counts are persisted in `quality_checks.findings_json`.
|
||||
- Canonical precision, recall, F1, mean IoU and metric rows remain strict
|
||||
candidate-polygon versus reference-footprint results. A second
|
||||
candidate-polygon versus reference-envelope pass is persisted and displayed
|
||||
as diagnostic-only evidence; it never replaces or promotes canonical
|
||||
metrics.
|
||||
- Detection Lab now explains which reference population was evaluated and
|
||||
clearly separates possible box-to-footprint artifacts from the canonical
|
||||
scorecard.
|
||||
- Updated the real-data operator smoke assertions, API/AI/database contracts
|
||||
and backend/frontend operator documentation. No migration, request contract,
|
||||
provider fetch or model dependency changed.
|
||||
|
||||
## Local validation
|
||||
|
||||
- `bash scripts/run_readiness_check.sh` passed end to end.
|
||||
- Full backend suite passed: `506` tests.
|
||||
- API contract audit passed with `81` implemented routes and the two documented
|
||||
non-envelope endpoints.
|
||||
- Alembic reports one head: `202606120900`.
|
||||
- Frontend TypeScript checking and production build passed with `85` modules;
|
||||
the dedicated MapLibre chunk remains intact.
|
||||
|
||||
## Next pass
|
||||
|
||||
- Deploy Sprint 184 to Tower, rerun QA for the persisted Mol-center analysis
|
||||
run and verify both persisted coverage evidence and Detection Lab rendering
|
||||
against live PostGIS before making any model-training decision.
|
||||
|
||||
@@ -189,6 +189,13 @@ Metrics may belong to a quality check, an analysis run, or both. Sprint 7A persi
|
||||
|
||||
Quality checks are domain records. Jobs track execution state; quality checks track the persisted QA/QC result; metrics track individual measurements.
|
||||
|
||||
Detection QA coverage and box-to-footprint diagnostics require no schema
|
||||
change. The canonical metric rows remain precision, recall, F1, mean IoU and
|
||||
false-positive/negative counts. Tile coverage population counts and the
|
||||
diagnostic reference-envelope comparison are persisted in the existing
|
||||
`quality_checks.findings_json`; `parameters_json.coverage_policy` records the
|
||||
evaluation policy used for reproducibility.
|
||||
|
||||
### exports
|
||||
|
||||
- `id uuid primary key`
|
||||
|
||||
+2
-2
@@ -26,7 +26,7 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Separate persisted database layers from available analysis-result overlays with an explicit Map content mode.
|
||||
- [x] Persist combined Mol operator evidence under the Unraid storage mount so reports survive all-in-one container replacement.
|
||||
- [x] Visually review Mol Postel and Donk false-positive/false-negative evidence, classify the dominant error modes and only then decide whether another model-training pass is justified.
|
||||
- [ ] Clip detection QA populations to persisted raster/tile coverage and add box-to-footprint matching diagnostics before reconsidering model training.
|
||||
- [x] Clip detection QA populations to persisted raster/tile coverage and add box-to-footprint matching diagnostics before reconsidering model training.
|
||||
- [x] Backend FastAPI foundation, health endpoint and service structure.
|
||||
- [x] React/TypeScript frontend foundation and MapLibre workbench.
|
||||
- [x] Map layer visibility, opacity and feature property inspection.
|
||||
@@ -411,7 +411,7 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Bound populated Project, Dataset, QA, AI and Export panels so long histories do not push core actions thousands of pixels down-page.
|
||||
- [x] Move Data create/upload forms and Map/AI diagnostics into explicit progressive disclosures.
|
||||
- [x] Reorder AI Labs around run controls and Map around layer selection plus the MapLibre canvas.
|
||||
- [ ] Harden detection QA coverage and matching diagnostics before any further model training.
|
||||
- [x] Harden detection QA coverage and matching diagnostics before any further model training.
|
||||
|
||||
- [x] Replace the one-page workflow panel stack with a task-based workbench shell.
|
||||
- [x] Add persistent project/AOI/dataset/layer context.
|
||||
|
||||
Reference in New Issue
Block a user