docs: record Mol benchmark decision
This commit is contained in:
@@ -7674,7 +7674,7 @@ Open:
|
||||
exact model asset, tile size, overlap and confidence threshold and refuses to
|
||||
blend different candidate configurations.
|
||||
- Added explicit gates for four independent positive Mol holdouts, one
|
||||
pure-empty background control, complete coverage provenance, minimum 95%
|
||||
pure-empty background control, complete coverage provenance, minimum 90%
|
||||
reference coverage, mean F1 `0.25`, minimum per-zone F1 `0.10` and zero
|
||||
background detections. Envelope matches remain diagnostic and cannot satisfy
|
||||
the canonical F1 gates.
|
||||
@@ -7698,3 +7698,44 @@ Open:
|
||||
- Deploy the operator tooling to Tower and run the current active local model
|
||||
over Achterbos, Gompel, Donk, Postel and the Postel-bos pure-empty control.
|
||||
Record the coverage-aware operational decision before changing model state.
|
||||
|
||||
## Tower benchmark evidence
|
||||
|
||||
- Deployed commit `9f61037` to the healthy all-in-one runtime and completed the
|
||||
real benchmark with active asset
|
||||
`geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt`, tile `512`,
|
||||
overlap `64`, confidence `0.15` and canonical QA IoU `0.25`.
|
||||
- Canonical results by holdout were: Achterbos F1 `0.6694`, Gompel `0.6564`,
|
||||
Donk `0.5894` and Postel `0.4749`. Macro precision/recall/F1 were
|
||||
`0.6296` / `0.5726` / `0.5975`; micro precision/recall/F1 were `0.6494` /
|
||||
`0.6189` / `0.6338`.
|
||||
- Across all positive zones, 3,624 of 3,829 raw reference features intersected
|
||||
exact tile coverage. The runner excluded 205 outside features, clipped 60 at
|
||||
boundaries and kept 101 additional envelope matches diagnostic-only.
|
||||
- Postel-bos produced zero detections. All model, per-zone, coverage provenance,
|
||||
diagnostic-separation and pure-empty-background gates passed. The minimum
|
||||
coverage ratio was `0.9051`; the integrity gate was set to `0.90` because
|
||||
OGC bbox retrieval intentionally retains complete edge geometries outside the
|
||||
projected raster polygon.
|
||||
- The final report under
|
||||
`/app/storage/operator-evidence/mol-operational-validation/coverage-aware-20260714T1000Z`
|
||||
is `accepted` with recommendation `retain_or_promote_candidate`. No model
|
||||
state or weights were changed by the report.
|
||||
- Exported a four-zone portfolio with 7,078 persisted QA evidence features.
|
||||
Audits found 1,381 canonical false negatives and 1,211 false positives.
|
||||
False-negative medians were about 24-28 m2 and tiny/small buildings dominate;
|
||||
Postel and Donk are the weakest zones.
|
||||
- Rendered and visually inspected 48 false-negative plus 48 false-positive
|
||||
review cards across every holdout. No source tile was missing. The cards show
|
||||
a mixture of genuine small-building/model errors, temporal GRB/orthophoto
|
||||
differences and box-versus-footprint matching artifacts, so raw FP/FN counts
|
||||
must not be converted directly into new labels.
|
||||
|
||||
## Decision
|
||||
|
||||
- Retain the active model for bounded operator-assisted Mol workflows and demos
|
||||
with QA/QC visible. It is not ground truth and should not be auto-exported as
|
||||
authoritative buildings. Do not start another blind training run; first
|
||||
complete the generated manual review decisions and curate confirmed small-
|
||||
building positives plus confirmed visual hard negatives from the weakest
|
||||
zones.
|
||||
|
||||
+2
-1
@@ -28,7 +28,8 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Visually review Mol Postel and Donk false-positive/false-negative evidence, classify the dominant error modes and only then decide whether another model-training pass is justified.
|
||||
- [x] Clip detection QA populations to persisted raster/tile coverage and add box-to-footprint matching diagnostics before reconsidering model training.
|
||||
- [x] Add a coverage-aware Mol multi-zone benchmark report with explicit positive-zone, per-zone collapse, reference-coverage and pure-empty background gates.
|
||||
- [ ] Execute the refreshed coverage-aware Mol operational benchmark against the active local model and record the resulting retain/review decision.
|
||||
- [x] Execute the refreshed coverage-aware Mol operational benchmark against the active local model and record the resulting retain/review decision.
|
||||
- [ ] Complete manual decisions for the generated 48 false-negative and 48 false-positive review cards before constructing any new training corpus.
|
||||
- [x] Backend FastAPI foundation, health endpoint and service structure.
|
||||
- [x] React/TypeScript frontend foundation and MapLibre workbench.
|
||||
- [x] Map layer visibility, opacity and feature property inspection.
|
||||
|
||||
Reference in New Issue
Block a user