serve a promoted model at the post-processing it was gated on
scripts/evaluate_belgium_building_candidate.py freezes its post-processing before the protected test — NMS IoU and a containment threshold selected during calibration, defaulting to 1.0. The runtime applied a hardcoded 0.85, so a promoted candidate was served under suppression its evaluation never measured and dropped detections the gate had counted. Neither report showed the difference. That constant was mine, added without noticing the evaluation pipeline already had a tuned value for the same concept. Containment is now configuration, recorded on every run beside the duplicate IoU threshold, so an operator can serve a candidate at the value it was gated at and afterwards see which value produced a given score. Two runs that post-processed differently produced different candidate sets from the same model output, so the comparison endpoint refuses to rank them. Runs recorded before those values were persisted carry none, and absence is not treated as a difference. The remaining gap is deliberate and documented rather than closed: the gate scores the model on its protected test set, the workbench scores the whole pipeline including coverage clipping and the tile-edge filter. A promoted candidate will not reproduce its gate F1 exactly, and pretending otherwise would be the worse answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -193,6 +193,11 @@ class TestComparingRealRuns:
|
||||
self.dataset_id = dataset_id
|
||||
self.model_name = "yolo-configured"
|
||||
self.parameters_json = {"model_asset_id": asset, "confidence_threshold": threshold}
|
||||
# Both runs post-processed identically, so they stay comparable.
|
||||
self.result_json = {
|
||||
"containment_suppression_threshold": 0.85,
|
||||
"duplicate_iou_threshold": 0.5,
|
||||
}
|
||||
|
||||
runs = {run_a: _Run("liberal", 0.15), run_b: _Run("conservative", 0.45)}
|
||||
scores = {
|
||||
|
||||
Reference in New Issue
Block a user