make QA scoring reproducible and threshold-independent
The greedy IoU matcher gave a reference to whichever candidate was offered first. Row order decided that, and every detection in a run shares one transaction timestamp, so ordering by created_at left the assignment undefined: the same QA run over the same data produced different mean IoU, and the geometry shown to a reviewer as a false positive could be the better of two detections. Candidates are now ranked by confidence with feature identity as tiebreaker, which is also the COCO/PASCAL rule. A single precision/recall/F1 triple describes one operating point, so two models cannot be compared from it: a conservatively calibrated model looks worse at a low confidence cut and better at a high one without detecting anything differently. DetectionMetricsService adds the full curve, average precision and the threshold where F1 actually peaks. Also: - report the population the metrics were computed over, so matches + false_positives equals candidate_feature_count even under an area filter; raw dataset totals move to the _raw fields; - state whether candidates are axis-aligned boxes or footprint polygons. A box can never reach IoU 1 against a rotated building, so the strict score has a ceiling that has nothing to do with detection quality. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -168,24 +168,61 @@ class DetectionQaService:
|
||||
clipped_boundary_count=clipped_boundary_count,
|
||||
)
|
||||
|
||||
# A polygon whose area is within this fraction of its own bounding box is
|
||||
# an axis-aligned rectangle for practical purposes.
|
||||
RECTANGULAR_AREA_RATIO = 0.99
|
||||
|
||||
@staticmethod
|
||||
def candidate_geometry_mode(geometries: list[tuple[dict[str, Any], BaseGeometry]]) -> str:
|
||||
"""Say whether the candidates are detector boxes or true footprints.
|
||||
|
||||
It matters for reading the score. An axis-aligned box can never reach
|
||||
IoU 1 against a rotated or L-shaped building footprint, so a strict
|
||||
footprint IoU understates a box detector by a fixed amount that has
|
||||
nothing to do with whether it found the building.
|
||||
"""
|
||||
|
||||
polygonal = [
|
||||
geometry
|
||||
for _, geometry in geometries
|
||||
if geometry.geom_type in {"Polygon", "MultiPolygon"} and geometry.area > 0
|
||||
]
|
||||
if not polygonal:
|
||||
return "unknown"
|
||||
rectangular = sum(
|
||||
1
|
||||
for geometry in polygonal
|
||||
if geometry.area / geometry.envelope.area >= DetectionQaService.RECTANGULAR_AREA_RATIO
|
||||
)
|
||||
return "axis_aligned_boxes" if rectangular == len(polygonal) else "footprint_polygons"
|
||||
|
||||
@staticmethod
|
||||
def box_to_footprint_diagnostics(
|
||||
strict_evidence: QaMatchEvidence,
|
||||
envelope_evidence: QaMatchEvidence,
|
||||
*,
|
||||
iou_threshold: float,
|
||||
candidate_geometry_mode: str = "unknown",
|
||||
) -> dict[str, Any]:
|
||||
envelope_metrics = DetectionQaService._metrics(envelope_evidence)
|
||||
return {
|
||||
diagnostics = {
|
||||
"diagnostic_only": True,
|
||||
"canonical_method": "candidate_polygon_vs_reference_footprint_iou",
|
||||
"diagnostic_method": "candidate_polygon_vs_reference_envelope_iou",
|
||||
"iou_threshold": iou_threshold,
|
||||
"candidate_geometry_mode": candidate_geometry_mode,
|
||||
"strict_matches": strict_evidence.matches,
|
||||
"envelope_matches": envelope_evidence.matches,
|
||||
"possible_box_to_footprint_mismatch_count": max(0, envelope_evidence.matches - strict_evidence.matches),
|
||||
**envelope_metrics,
|
||||
}
|
||||
if candidate_geometry_mode == "axis_aligned_boxes":
|
||||
diagnostics["interpretation"] = (
|
||||
"Candidates are axis-aligned detector boxes. The strict footprint IoU therefore has a "
|
||||
"ceiling below 1 for rotated or non-rectangular buildings; the envelope figures isolate "
|
||||
"detection quality from that shape mismatch."
|
||||
)
|
||||
return diagnostics
|
||||
|
||||
@staticmethod
|
||||
def _metrics(evidence: QaMatchEvidence) -> dict[str, Any]:
|
||||
|
||||
Reference in New Issue
Block a user