let an operator's adjudication reach the score
The review vocabulary already separates a model error from a reference gap, because the product's position is that official footprints are not automatically perfect ground truth. Those verdicts were only counted. An operator who inspected forty false positives and established that twelve are buildings the reference simply lacks still saw a precision counting all forty against the model — a number they had personally disproved, on the panel where they disproved it. Applying the verdicts gives an adjudicated score reported next to the raw one, so nothing is quietly improved. Not being able to judge is not evidence in the model's favour, so uncertain and obscured verdicts keep counting, as does a decision from a later release that this runtime does not recognise. Because part of the evidence is usually still unreviewed, the honest form is an interval rather than a single corrected number: pessimistic assumes every unreviewed finding is a model error, optimistic assumes none is, and the headline equals the pessimistic reading so a partly reviewed check never presents as a settled one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1850,6 +1850,32 @@ Each feature includes:
|
||||
|
||||
Returns persisted detections for a dataset as a GeoJSON FeatureCollection. Optional filters match the detection list endpoint.
|
||||
|
||||
### GET `/api/v1/projects/{project_id}/quality-checks/{id}/reviews`
|
||||
|
||||
Returns the evidence queue plus a `summary`, which now carries
|
||||
`reviewed_metrics`: the score with the operator's verdicts applied, next to the
|
||||
raw one.
|
||||
|
||||
The review vocabulary already separates a model error from a reference gap,
|
||||
because official footprints are not automatically perfect ground truth. Those
|
||||
verdicts were only counted, so an operator who established that twelve of forty
|
||||
false positives are buildings the reference simply lacks still saw a precision
|
||||
counting all forty against the model — a number they had personally disproved.
|
||||
|
||||
- `reference_gap_or_change` and `qa_alignment_mismatch` exonerate a finding: the
|
||||
detection, or the missing detection, was not the model's error.
|
||||
- `confirmed_model_false_positive` / `confirmed_model_false_negative` keep it.
|
||||
- `uncertain` and `imagery_obscured_or_uncertain` also keep it. Being unable to
|
||||
judge is not evidence in the model's favour, and treating it as such is how a
|
||||
score drifts upward unearned. A decision from a later release the runtime does
|
||||
not recognise is likewise treated as no judgement.
|
||||
|
||||
Because part of the evidence is usually unreviewed, the result is an interval:
|
||||
`pessimistic` assumes every unreviewed finding is a model error, `optimistic`
|
||||
assumes none is, and `adjudicated` equals the pessimistic reading so a partly
|
||||
reviewed check never presents as a settled one. `review_complete` says whether
|
||||
the interval has collapsed.
|
||||
|
||||
### POST `/api/v1/detection/runs/compare`
|
||||
|
||||
Scores several persisted runs against one reference and ranks them on average
|
||||
|
||||
Reference in New Issue
Block a user