make QA scoring reproducible and threshold-independent
The greedy IoU matcher gave a reference to whichever candidate was offered first. Row order decided that, and every detection in a run shares one transaction timestamp, so ordering by created_at left the assignment undefined: the same QA run over the same data produced different mean IoU, and the geometry shown to a reviewer as a false positive could be the better of two detections. Candidates are now ranked by confidence with feature identity as tiebreaker, which is also the COCO/PASCAL rule. A single precision/recall/F1 triple describes one operating point, so two models cannot be compared from it: a conservatively calibrated model looks worse at a low confidence cut and better at a high one without detecting anything differently. DetectionMetricsService adds the full curve, average precision and the threshold where F1 actually peaks. Also: - report the population the metrics were computed over, so matches + false_positives equals candidate_feature_count even under an area filter; raw dataset totals move to the _raw fields; - state whether candidates are axis-aligned boxes or footprint polygons. A box can never reach IoU 1 against a rotated building, so the strict score has a ceiling that has nothing to do with detection quality. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -19,8 +19,14 @@ class QaProviderComparisonRequest(BaseModel):
|
||||
class QaProviderComparisonResult(BaseModel):
|
||||
status: str
|
||||
warnings: list[str] = Field(default_factory=list)
|
||||
# Counts of the population that was actually matched, so that
|
||||
# ``matches + false_positives == candidate_feature_count`` holds even when
|
||||
# an area filter or an unparseable geometry removed features. The ``_raw``
|
||||
# fields keep the untouched dataset totals visible next to them.
|
||||
candidate_feature_count: int
|
||||
reference_feature_count: int
|
||||
candidate_feature_count_raw: int | None = None
|
||||
reference_feature_count_raw: int | None = None
|
||||
matches: int
|
||||
false_positives: int
|
||||
false_negatives: int
|
||||
|
||||
Reference in New Issue
Block a user