feat: add coverage-aware Mol benchmark
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-14 12:27:48 +02:00
parent 4f0d2891dd
commit 9f61037794
12 changed files with 682 additions and 10 deletions
+18 -4
View File
@@ -261,9 +261,20 @@ confidence `0.15` and QA IoU `0.25`. Every positive run persists Project, Area,
Dataset, Job, AnalysisRun, Detection, QualityCheck, Metric and Export records.
The background run persists its project, AOI, raster, job, analysis and
detections but intentionally does not invent QA metrics for an empty or sparse
reference context. The combined JSON/Markdown summary reports
`evidence_ready`; this records completed evidence and is not an automatic model
promotion decision.
reference context. Each matrix row now preserves the exact persisted inference
coverage counts and the diagnostic-only reference-envelope comparison beside
the canonical footprint-IoU metrics.
The runner also writes `mol_operational_benchmark_report.json` and `.md`. The
default operational gates require four positive holdouts, one background
control, coverage provenance for every positive run, at least 95% reference
coverage in every zone, mean F1 at least `0.25`, per-zone F1 at least `0.10`
and zero detections in each pure-empty control. Override the numeric gates only
through the documented `MOL_MIN_MEAN_F1`, `MOL_MIN_ZONE_F1`,
`MOL_MIN_REFERENCE_COVERAGE` and `MOL_MAX_BACKGROUND_DETECTIONS` variables.
An `accepted` report records bounded operational evidence; it does not mutate
the active model. A `review_required` report is still a successful benchmark
execution but explicitly blocks a promotion recommendation.
For model-training candidates, prepare a larger operator-only sample manifest so
tile overlap can create meaningful context instead of one tile per source
@@ -344,7 +355,10 @@ logs plus `quality_matrix_summary.json` under
set. The summary ranks `best_by_score`, `best_by_recall` and
`best_by_precision` so the next model decision is based on persisted
`QualityCheck`/`Metric` evidence rather than visual guesses. It does not create
provider data, use fixtures or download model weights.
provider data, use fixtures or download model weights. Coverage-aware rows also
record raw/evaluated/excluded/clipped candidate and reference counts, tile
coverage provenance and the separately labelled box-to-footprint diagnostic
gap.
Run the same matrix across every prepared operator sample: