serve a promoted model at the post-processing it was gated on

scripts/evaluate_belgium_building_candidate.py freezes its post-processing
before the protected test — NMS IoU and a containment threshold selected during
calibration, defaulting to 1.0. The runtime applied a hardcoded 0.85, so a
promoted candidate was served under suppression its evaluation never measured
and dropped detections the gate had counted. Neither report showed the
difference. That constant was mine, added without noticing the evaluation
pipeline already had a tuned value for the same concept.

Containment is now configuration, recorded on every run beside the duplicate
IoU threshold, so an operator can serve a candidate at the value it was gated
at and afterwards see which value produced a given score.

Two runs that post-processed differently produced different candidate sets from
the same model output, so the comparison endpoint refuses to rank them. Runs
recorded before those values were persisted carry none, and absence is not
treated as a difference.

The remaining gap is deliberate and documented rather than closed: the gate
scores the model on its protected test set, the workbench scores the whole
pipeline including coverage clipping and the tile-edge filter. A promoted
candidate will not reproduce its gate F1 exactly, and pretending otherwise
would be the worse answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Jens
2026-08-22 21:15:05 +02:00
co-authored by Claude Opus 5
parent 1a1a9af6e7
commit c8d32a4801
8 changed files with 207 additions and 8 deletions
+9
View File
@@ -92,6 +92,15 @@ runtime source of truth.
`YOLO_SEG_MODEL_PATH`/`SAM_MODEL_PATH` to existing local weights and enables
them explicitly. GeoIntel never downloads segmentation weights automatically;
fixture segmentation remains explicit-only.
- The training gate and the workbench measure different things, deliberately.
`scripts/evaluate_belgium_building_candidate.py` scores the model on frozen
post-processing over its protected test set; the workbench scores the whole
production pipeline, which additionally clips to the persisted inference
coverage and drops boxes truncated by an interior tile edge. A promoted
candidate will therefore not reproduce its gate F1 exactly in the workbench.
What must match is the post-processing: `YOLO_CONTAINMENT_NMS_THRESHOLD` and
`YOLO_DUPLICATE_IOU_THRESHOLD` are recorded with every run, and two runs that
used different values are reported as not comparable.
- Detection QA reports both a strict footprint IoU and an envelope diagnostic.
For an axis-aligned box detector the strict figure has a ceiling below 1 on
rotated or non-rectangular buildings; the response states which geometry mode