rank model variants on average precision, and say when they are not comparable
The workbench ranks model variants by a stored F1, each measured at that variant's own confidence threshold. A conservatively calibrated detector then looks worse than a liberal one without detecting anything differently: the number says as much about the threshold as about the model. POST /detection/runs/compare ranks on average precision instead, which describes the whole ranking a model produced, and keeps each run's own-threshold F1 visible next to it so the difference between the two readings is auditable. Comparability comes before the ranking. Runs over different source rasters, scored against different references, without a proven inference footprint, or covering a different evaluated population are not alternatives to one another, and no metric makes them so. The report names which of those applies and still returns the numbers — they are simply not a ranking. Each run is scored through the same QA path the workbench uses, so a comparison and the persisted quality checks cannot drift apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -78,6 +78,25 @@ class DetectionQaRequest(BaseModel):
|
||||
calibration_thresholds: list[float] = Field(default_factory=list, max_length=32)
|
||||
|
||||
|
||||
class DetectionComparisonRequest(BaseModel):
|
||||
"""Place several runs side by side against one reference."""
|
||||
|
||||
analysis_run_ids: list[UUID] = Field(min_length=2, max_length=12)
|
||||
reference_dataset_id: UUID
|
||||
iou_threshold: float = Field(default=0.5, ge=0.0, le=1.0)
|
||||
|
||||
|
||||
class DetectionComparisonResponse(BaseModel):
|
||||
reference_dataset_id: UUID
|
||||
iou_threshold: float
|
||||
# Whether these runs answer the same question at all, and why not if they
|
||||
# do not. Numbers from incomparable runs are reported but never ranked as
|
||||
# if they were alternatives.
|
||||
comparability: dict
|
||||
ranking_metric: str
|
||||
rows: list[dict]
|
||||
|
||||
|
||||
class DetectionRunResponse(BaseModel):
|
||||
model_config = ConfigDict(protected_namespaces=())
|
||||
|
||||
|
||||
Reference in New Issue
Block a user