Add Mol false-negative visual review evidence
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-13 20:22:53 +02:00
parent de4e15f166
commit 50952e18f9
11 changed files with 967 additions and 3 deletions
+20
View File
@@ -150,6 +150,26 @@ Only records explicitly marked `confirmed_model_false_positive` are emitted by
the validator as possible hard-negative review input. The workflow does not
train a model, mutate QA persistence, fetch data or infer review decisions.
### Persisted false-negative visual review
False negatives require the same manual distinction. A missed reference can be
a true model miss, a stale reference, an obscured object, a box-to-footprint
matching failure or an object outside the raster actually presented to the
model. The read-only false-negative renderer resolves the persisted tile
manifest from the fixed-threshold run and overlays:
- red: the missed GRB/reference footprint;
- blue: nearby persisted candidate detections;
- green: nearby matched reference footprints.
The decision contract is `confirmed_model_false_negative`,
`reference_gap_or_change`, `qa_alignment_mismatch`,
`imagery_obscured_or_uncertain` or `unreviewed`. References outside every
persisted inference tile are written to a separate exclusion GeoJSON and are
not treated as reviewable model misses. This renderer does not alter persisted
QA metrics; coverage-adjusted values remain audit diagnostics until the QA
service evaluation population is deliberately hardened.
### Local model asset catalog
GeoIntel can list local runtime model files mounted into the backend model
+34
View File
@@ -7301,3 +7301,37 @@ Open:
- Ran the interactive PostGIS AOI query against `vector_features`: `126` GRB buildings returned with no truncation. The guided GIS smoke persisted a derived dataset and GeoJSON export, then produced candidate/reference QA F1 `0.9582` with `126` matches, `0` false positives and `11` false negatives; `263` persisted evidence features rendered back on the map.
- Remaining evidence caveat: the dataset QA result reports weak CRS-assumption warnings, so those geometry metrics remain explicitly approximate until CRS provenance handling is reviewed.
- Host observation outside GeoIntel: Unraid recovered after reboot and serves the app, but still reports one disabled/invalid array device. Storage administration should resolve that independently of application development.
# Sprint 179 - Mol Donk/Postel detection evidence diagnosis
## Implementation
- Added `render_detection_false_negative_review_contact_sheets.py` as a read-only counterpart to the persisted false-positive review workflow.
- Resolved each sample's exact persisted tile manifest from its fixed-threshold run summary; manifests and source tiles are confined to `/app/storage`.
- Projected WGS84 missed-reference geometry onto the real inference tiles and rendered nearby persisted candidate detections plus matched-reference context.
- Added deterministic AOI/area stratification and an explicit five-state manual decision CSV. No decision is inferred and no training input is exported automatically.
- Separated references outside every persisted source tile into `false_negatives_outside_tile_coverage.geojson` instead of hiding them or calling them model misses.
- Added focused rendering, manifest, source-coverage and storage-confinement regression tests; wired the script into readiness compilation and the all-in-one image.
## Live Mol evidence
- Re-exported the four-zone fixed-threshold portfolio from existing persisted QualityChecks without rerunning inference or mutating application data.
- The complete portfolio contains `7,283` evidence features. Donk contributes `424` false positives and `553` false negatives; Postel contributes `43` and `85`.
- Rendered and inspected 48 stratified false-positive cases over Donk/Postel. Explicit decisions: `37` QA alignment mismatches, `7` confirmed model false positives, `3` reference gaps/changes and `1` uncertain. The existing validator passed with status `complete`.
- Rendered and inspected 48 stratified false-negative cases. Explicit decisions: `27` QA alignment mismatches, `6` confirmed model false negatives, `3` reference gaps/changes and `12` imagery-obscured/uncertain.
- Found `41/638` false-negative evidence records outside every persisted inference tile: Donk `28`, Postel `13`. Directionally excluding those records raises Donk recall from `0.5519` to `0.5647` and Postel from `0.3796` to `0.4194`; these are audit diagnostics only and no persisted metric was changed.
- The dominant visual mode is rectangle-to-footprint mismatch on large industrial roofs and dense residential blocks, often with a blue persisted candidate already overlapping the red missed GRB footprint. Postel additionally contains many tiny/vegetation-obscured references.
- Persistent evidence and the assessment are stored below `/app/storage/operator-evidence/mol-operational-review/20260713`.
## Decision and next pass
- **NO-GO for immediate retraining.** Only `7/48` reviewed false positives and `6/48` reviewed false negatives were confirmed model errors; evaluation alignment and coverage defects dominate the selected evidence.
- Next harden detection QA to restrict candidate/reference populations to persisted raster/tile coverage and expose best-IoU/overlap/unmatched diagnostics. Rerun Donk/Postel QA against the unchanged persisted detections before deciding whether the confirmed model-error subset justifies curated training.
## Validation
- Focused false-positive/false-negative audit and visual-review coverage: `8 passed`.
- Ruff passed for the changed Python renderer and regression tests.
- `bash scripts/run_readiness_check.sh`: passed with `487` backend tests, the 81-route API contract audit, one Alembic head, frontend typecheck and production build.
- `python -m alembic upgrade head --sql` rendered the complete migration chain through `202606120900`; no migration changed.
- Local `docker compose config` was unavailable because the Windows workstation has no Docker CLI. The repository-driven Tower deployment remains the required live Docker validation.
+2 -1
View File
@@ -22,7 +22,8 @@ This file now starts with the current implementation status. Older preparation/b
- [x] Add a Mol multi-zone operational pack with independent positive holdouts, background control and persisted map-ready AOIs.
- [x] Execute the Mol pack on live Tower/PostGIS with real orthophotos, GRB references, configured YOLO, persisted QA/QC and a zero-building background control.
- [x] Persist combined Mol operator evidence under the Unraid storage mount so reports survive all-in-one container replacement.
- [ ] Visually review Mol Postel and Donk false-positive/false-negative evidence, classify the dominant error modes and only then decide whether another model-training pass is justified.
- [x] Visually review Mol Postel and Donk false-positive/false-negative evidence, classify the dominant error modes and only then decide whether another model-training pass is justified.
- [ ] Clip detection QA populations to persisted raster/tile coverage and add box-to-footprint matching diagnostics before reconsidering model training.
- [x] Backend FastAPI foundation, health endpoint and service structure.
- [x] React/TypeScript frontend foundation and MapLibre workbench.
- [x] Map layer visibility, opacity and feature property inspection.