Add Mol false-negative visual review evidence
This commit is contained in:
@@ -150,6 +150,26 @@ Only records explicitly marked `confirmed_model_false_positive` are emitted by
|
||||
the validator as possible hard-negative review input. The workflow does not
|
||||
train a model, mutate QA persistence, fetch data or infer review decisions.
|
||||
|
||||
### Persisted false-negative visual review
|
||||
|
||||
False negatives require the same manual distinction. A missed reference can be
|
||||
a true model miss, a stale reference, an obscured object, a box-to-footprint
|
||||
matching failure or an object outside the raster actually presented to the
|
||||
model. The read-only false-negative renderer resolves the persisted tile
|
||||
manifest from the fixed-threshold run and overlays:
|
||||
|
||||
- red: the missed GRB/reference footprint;
|
||||
- blue: nearby persisted candidate detections;
|
||||
- green: nearby matched reference footprints.
|
||||
|
||||
The decision contract is `confirmed_model_false_negative`,
|
||||
`reference_gap_or_change`, `qa_alignment_mismatch`,
|
||||
`imagery_obscured_or_uncertain` or `unreviewed`. References outside every
|
||||
persisted inference tile are written to a separate exclusion GeoJSON and are
|
||||
not treated as reviewable model misses. This renderer does not alter persisted
|
||||
QA metrics; coverage-adjusted values remain audit diagnostics until the QA
|
||||
service evaluation population is deliberately hardened.
|
||||
|
||||
### Local model asset catalog
|
||||
|
||||
GeoIntel can list local runtime model files mounted into the backend model
|
||||
|
||||
@@ -7301,3 +7301,37 @@ Open:
|
||||
- Ran the interactive PostGIS AOI query against `vector_features`: `126` GRB buildings returned with no truncation. The guided GIS smoke persisted a derived dataset and GeoJSON export, then produced candidate/reference QA F1 `0.9582` with `126` matches, `0` false positives and `11` false negatives; `263` persisted evidence features rendered back on the map.
|
||||
- Remaining evidence caveat: the dataset QA result reports weak CRS-assumption warnings, so those geometry metrics remain explicitly approximate until CRS provenance handling is reviewed.
|
||||
- Host observation outside GeoIntel: Unraid recovered after reboot and serves the app, but still reports one disabled/invalid array device. Storage administration should resolve that independently of application development.
|
||||
|
||||
# Sprint 179 - Mol Donk/Postel detection evidence diagnosis
|
||||
|
||||
## Implementation
|
||||
|
||||
- Added `render_detection_false_negative_review_contact_sheets.py` as a read-only counterpart to the persisted false-positive review workflow.
|
||||
- Resolved each sample's exact persisted tile manifest from its fixed-threshold run summary; manifests and source tiles are confined to `/app/storage`.
|
||||
- Projected WGS84 missed-reference geometry onto the real inference tiles and rendered nearby persisted candidate detections plus matched-reference context.
|
||||
- Added deterministic AOI/area stratification and an explicit five-state manual decision CSV. No decision is inferred and no training input is exported automatically.
|
||||
- Separated references outside every persisted source tile into `false_negatives_outside_tile_coverage.geojson` instead of hiding them or calling them model misses.
|
||||
- Added focused rendering, manifest, source-coverage and storage-confinement regression tests; wired the script into readiness compilation and the all-in-one image.
|
||||
|
||||
## Live Mol evidence
|
||||
|
||||
- Re-exported the four-zone fixed-threshold portfolio from existing persisted QualityChecks without rerunning inference or mutating application data.
|
||||
- The complete portfolio contains `7,283` evidence features. Donk contributes `424` false positives and `553` false negatives; Postel contributes `43` and `85`.
|
||||
- Rendered and inspected 48 stratified false-positive cases over Donk/Postel. Explicit decisions: `37` QA alignment mismatches, `7` confirmed model false positives, `3` reference gaps/changes and `1` uncertain. The existing validator passed with status `complete`.
|
||||
- Rendered and inspected 48 stratified false-negative cases. Explicit decisions: `27` QA alignment mismatches, `6` confirmed model false negatives, `3` reference gaps/changes and `12` imagery-obscured/uncertain.
|
||||
- Found `41/638` false-negative evidence records outside every persisted inference tile: Donk `28`, Postel `13`. Directionally excluding those records raises Donk recall from `0.5519` to `0.5647` and Postel from `0.3796` to `0.4194`; these are audit diagnostics only and no persisted metric was changed.
|
||||
- The dominant visual mode is rectangle-to-footprint mismatch on large industrial roofs and dense residential blocks, often with a blue persisted candidate already overlapping the red missed GRB footprint. Postel additionally contains many tiny/vegetation-obscured references.
|
||||
- Persistent evidence and the assessment are stored below `/app/storage/operator-evidence/mol-operational-review/20260713`.
|
||||
|
||||
## Decision and next pass
|
||||
|
||||
- **NO-GO for immediate retraining.** Only `7/48` reviewed false positives and `6/48` reviewed false negatives were confirmed model errors; evaluation alignment and coverage defects dominate the selected evidence.
|
||||
- Next harden detection QA to restrict candidate/reference populations to persisted raster/tile coverage and expose best-IoU/overlap/unmatched diagnostics. Rerun Donk/Postel QA against the unchanged persisted detections before deciding whether the confirmed model-error subset justifies curated training.
|
||||
|
||||
## Validation
|
||||
|
||||
- Focused false-positive/false-negative audit and visual-review coverage: `8 passed`.
|
||||
- Ruff passed for the changed Python renderer and regression tests.
|
||||
- `bash scripts/run_readiness_check.sh`: passed with `487` backend tests, the 81-route API contract audit, one Alembic head, frontend typecheck and production build.
|
||||
- `python -m alembic upgrade head --sql` rendered the complete migration chain through `202606120900`; no migration changed.
|
||||
- Local `docker compose config` was unavailable because the Windows workstation has no Docker CLI. The repository-driven Tower deployment remains the required live Docker validation.
|
||||
|
||||
+2
-1
@@ -22,7 +22,8 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Add a Mol multi-zone operational pack with independent positive holdouts, background control and persisted map-ready AOIs.
|
||||
- [x] Execute the Mol pack on live Tower/PostGIS with real orthophotos, GRB references, configured YOLO, persisted QA/QC and a zero-building background control.
|
||||
- [x] Persist combined Mol operator evidence under the Unraid storage mount so reports survive all-in-one container replacement.
|
||||
- [ ] Visually review Mol Postel and Donk false-positive/false-negative evidence, classify the dominant error modes and only then decide whether another model-training pass is justified.
|
||||
- [x] Visually review Mol Postel and Donk false-positive/false-negative evidence, classify the dominant error modes and only then decide whether another model-training pass is justified.
|
||||
- [ ] Clip detection QA populations to persisted raster/tile coverage and add box-to-footprint matching diagnostics before reconsidering model training.
|
||||
- [x] Backend FastAPI foundation, health endpoint and service structure.
|
||||
- [x] React/TypeScript frontend foundation and MapLibre workbench.
|
||||
- [x] Map layer visibility, opacity and feature property inspection.
|
||||
|
||||
Reference in New Issue
Block a user