feat: add coverage-aware Mol benchmark
This commit is contained in:
+18
-4
@@ -261,9 +261,20 @@ confidence `0.15` and QA IoU `0.25`. Every positive run persists Project, Area,
|
||||
Dataset, Job, AnalysisRun, Detection, QualityCheck, Metric and Export records.
|
||||
The background run persists its project, AOI, raster, job, analysis and
|
||||
detections but intentionally does not invent QA metrics for an empty or sparse
|
||||
reference context. The combined JSON/Markdown summary reports
|
||||
`evidence_ready`; this records completed evidence and is not an automatic model
|
||||
promotion decision.
|
||||
reference context. Each matrix row now preserves the exact persisted inference
|
||||
coverage counts and the diagnostic-only reference-envelope comparison beside
|
||||
the canonical footprint-IoU metrics.
|
||||
|
||||
The runner also writes `mol_operational_benchmark_report.json` and `.md`. The
|
||||
default operational gates require four positive holdouts, one background
|
||||
control, coverage provenance for every positive run, at least 95% reference
|
||||
coverage in every zone, mean F1 at least `0.25`, per-zone F1 at least `0.10`
|
||||
and zero detections in each pure-empty control. Override the numeric gates only
|
||||
through the documented `MOL_MIN_MEAN_F1`, `MOL_MIN_ZONE_F1`,
|
||||
`MOL_MIN_REFERENCE_COVERAGE` and `MOL_MAX_BACKGROUND_DETECTIONS` variables.
|
||||
An `accepted` report records bounded operational evidence; it does not mutate
|
||||
the active model. A `review_required` report is still a successful benchmark
|
||||
execution but explicitly blocks a promotion recommendation.
|
||||
|
||||
For model-training candidates, prepare a larger operator-only sample manifest so
|
||||
tile overlap can create meaningful context instead of one tile per source
|
||||
@@ -344,7 +355,10 @@ logs plus `quality_matrix_summary.json` under
|
||||
set. The summary ranks `best_by_score`, `best_by_recall` and
|
||||
`best_by_precision` so the next model decision is based on persisted
|
||||
`QualityCheck`/`Metric` evidence rather than visual guesses. It does not create
|
||||
provider data, use fixtures or download model weights.
|
||||
provider data, use fixtures or download model weights. Coverage-aware rows also
|
||||
record raw/evaluated/excluded/clipped candidate and reference counts, tile
|
||||
coverage provenance and the separately labelled box-to-footprint diagnostic
|
||||
gap.
|
||||
|
||||
Run the same matrix across every prepared operator sample:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user