rank model variants on average precision, and say when they are not comparable
The workbench ranks model variants by a stored F1, each measured at that variant's own confidence threshold. A conservatively calibrated detector then looks worse than a liberal one without detecting anything differently: the number says as much about the threshold as about the model. POST /detection/runs/compare ranks on average precision instead, which describes the whole ranking a model produced, and keeps each run's own-threshold F1 visible next to it so the difference between the two readings is auditable. Comparability comes before the ranking. Runs over different source rasters, scored against different references, without a proven inference footprint, or covering a different evaluated population are not alternatives to one another, and no metric makes them so. The report names which of those applies and still returns the numbers — they are simply not a ranking. Each run is scored through the same QA path the workbench uses, so a comparison and the persisted quality checks cannot drift apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -10,6 +10,8 @@ from app.schemas import (
|
||||
AnalysisQaResponse,
|
||||
DetectionListResponse,
|
||||
DetectionModelsResponse,
|
||||
DetectionComparisonRequest,
|
||||
DetectionComparisonResponse,
|
||||
DetectionQaRequest,
|
||||
DetectionRead,
|
||||
DetectionRunListResponse,
|
||||
@@ -22,6 +24,7 @@ from app.schemas import (
|
||||
ModelAssetListResponse,
|
||||
YoloPreflightResponse,
|
||||
)
|
||||
from app.services.detection_comparison_service import DetectionComparisonService
|
||||
from app.services.detection_service import DetectionService
|
||||
from app.services.model_asset_catalog_service import ModelAssetCatalogService
|
||||
from app.services.model_registry_service import ModelRegistryService
|
||||
@@ -232,6 +235,27 @@ def get_dataset_detection_geojson(
|
||||
)
|
||||
|
||||
|
||||
@router.post("/runs/compare", response_model=Envelope[DetectionComparisonResponse])
|
||||
def compare_detection_runs(payload: DetectionComparisonRequest, db: Session = Depends(get_db)) -> dict:
|
||||
"""Rank several runs against one reference on average precision.
|
||||
|
||||
The workbench ranks model variants by a stored F1 measured at each
|
||||
variant's own confidence threshold, which orders the thresholds as much as
|
||||
the models. Average precision describes the whole ranking a model produced.
|
||||
Comparability is reported first: runs over different rasters, different
|
||||
references or different inference coverage are not alternatives.
|
||||
"""
|
||||
|
||||
return envelope(
|
||||
DetectionComparisonService.compare_runs(
|
||||
db,
|
||||
analysis_run_ids=payload.analysis_run_ids,
|
||||
reference_dataset_id=payload.reference_dataset_id,
|
||||
iou_threshold=payload.iou_threshold,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/runs/{analysis_run_id}/qa/reference",
|
||||
response_model=Envelope[AnalysisQaResponse],
|
||||
|
||||
Reference in New Issue
Block a user