docs: record Mol benchmark decision
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-14 12:40:11 +02:00
parent 9f61037794
commit 5db93a9530
7 changed files with 57 additions and 7 deletions
+42 -1
View File
@@ -7674,7 +7674,7 @@ Open:
exact model asset, tile size, overlap and confidence threshold and refuses to
blend different candidate configurations.
- Added explicit gates for four independent positive Mol holdouts, one
pure-empty background control, complete coverage provenance, minimum 95%
pure-empty background control, complete coverage provenance, minimum 90%
reference coverage, mean F1 `0.25`, minimum per-zone F1 `0.10` and zero
background detections. Envelope matches remain diagnostic and cannot satisfy
the canonical F1 gates.
@@ -7698,3 +7698,44 @@ Open:
- Deploy the operator tooling to Tower and run the current active local model
over Achterbos, Gompel, Donk, Postel and the Postel-bos pure-empty control.
Record the coverage-aware operational decision before changing model state.
## Tower benchmark evidence
- Deployed commit `9f61037` to the healthy all-in-one runtime and completed the
real benchmark with active asset
`geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt`, tile `512`,
overlap `64`, confidence `0.15` and canonical QA IoU `0.25`.
- Canonical results by holdout were: Achterbos F1 `0.6694`, Gompel `0.6564`,
Donk `0.5894` and Postel `0.4749`. Macro precision/recall/F1 were
`0.6296` / `0.5726` / `0.5975`; micro precision/recall/F1 were `0.6494` /
`0.6189` / `0.6338`.
- Across all positive zones, 3,624 of 3,829 raw reference features intersected
exact tile coverage. The runner excluded 205 outside features, clipped 60 at
boundaries and kept 101 additional envelope matches diagnostic-only.
- Postel-bos produced zero detections. All model, per-zone, coverage provenance,
diagnostic-separation and pure-empty-background gates passed. The minimum
coverage ratio was `0.9051`; the integrity gate was set to `0.90` because
OGC bbox retrieval intentionally retains complete edge geometries outside the
projected raster polygon.
- The final report under
`/app/storage/operator-evidence/mol-operational-validation/coverage-aware-20260714T1000Z`
is `accepted` with recommendation `retain_or_promote_candidate`. No model
state or weights were changed by the report.
- Exported a four-zone portfolio with 7,078 persisted QA evidence features.
Audits found 1,381 canonical false negatives and 1,211 false positives.
False-negative medians were about 24-28 m2 and tiny/small buildings dominate;
Postel and Donk are the weakest zones.
- Rendered and visually inspected 48 false-negative plus 48 false-positive
review cards across every holdout. No source tile was missing. The cards show
a mixture of genuine small-building/model errors, temporal GRB/orthophoto
differences and box-versus-footprint matching artifacts, so raw FP/FN counts
must not be converted directly into new labels.
## Decision
- Retain the active model for bounded operator-assisted Mol workflows and demos
with QA/QC visible. It is not ground truth and should not be auto-exported as
authoritative buildings. Do not start another blind training run; first
complete the generated manual review decisions and curate confirmed small-
building positives plus confirmed visual hard negatives from the weakest
zones.
+2 -1
View File
@@ -28,7 +28,8 @@ This file now starts with the current implementation status. Older preparation/b
- [x] Visually review Mol Postel and Donk false-positive/false-negative evidence, classify the dominant error modes and only then decide whether another model-training pass is justified.
- [x] Clip detection QA populations to persisted raster/tile coverage and add box-to-footprint matching diagnostics before reconsidering model training.
- [x] Add a coverage-aware Mol multi-zone benchmark report with explicit positive-zone, per-zone collapse, reference-coverage and pure-empty background gates.
- [ ] Execute the refreshed coverage-aware Mol operational benchmark against the active local model and record the resulting retain/review decision.
- [x] Execute the refreshed coverage-aware Mol operational benchmark against the active local model and record the resulting retain/review decision.
- [ ] Complete manual decisions for the generated 48 false-negative and 48 false-positive review cards before constructing any new training corpus.
- [x] Backend FastAPI foundation, health endpoint and service structure.
- [x] React/TypeScript frontend foundation and MapLibre workbench.
- [x] Map layer visibility, opacity and feature property inspection.