Jens/geointel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

rank model variants on average precision, and say when they are not comparable

Browse Source 
The workbench ranks model variants by a stored F1, each measured at that
variant's own confidence threshold. A conservatively calibrated detector then
looks worse than a liberal one without detecting anything differently: the
number says as much about the threshold as about the model. POST
/detection/runs/compare ranks on average precision instead, which describes the
whole ranking a model produced, and keeps each run's own-threshold F1 visible
next to it so the difference between the two readings is auditable.

Comparability comes before the ranking. Runs over different source rasters,
scored against different references, without a proven inference footprint, or
covering a different evaluated population are not alternatives to one another,
and no metric makes them so. The report names which of those applies and still
returns the numbers — they are simply not a ranking.

Each run is scored through the same QA path the workbench uses, so a comparison
and the persisted quality checks cannot drift apart.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Jens
2026-08-22 19:47:14 +02:00
co-authored by Claude Opus 5
parent 2cd2c49389
commit f6eced1b94
6 changed files with 497 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/API_CONTRACTS.md
+22
View File
@@ -1850,6 +1850,28 @@ Each feature includes:
Returns persisted detections for a dataset as a GeoJSON FeatureCollection. Optional filters match the detection list endpoint.
### POST `/api/v1/detection/runs/compare`
Scores several persisted runs against one reference and ranks them on average
precision.
The workbench ranks model variants by a stored F1, each measured at that
variant's own confidence threshold — a figure that says as much about the
threshold as about the model, so a conservatively calibrated detector looks
worse than a liberal one without detecting anything differently. Average
precision describes the whole ranking the model produced. The F1 at each run's
own threshold stays in the response next to it, so the difference between the
two readings is auditable rather than hidden.
Comparability is reported before any ranking. Runs over different source
rasters, scored against different references, without a proven inference
footprint, or covering a different evaluated population are not alternatives to
one another, and `comparability.blocking_reasons` names which of those applies.
The numbers are still returned — they are simply not a ranking.
Each run is scored through the same QA path the workbench uses, so a comparison
and the persisted quality checks cannot drift apart.
### POST `/api/v1/detection/runs/{analysis_run_id}/qa/reference`
Compares persisted detection geometries from an analysis run against persisted `vector_features` from a reference vector dataset.