Add detection calibration evidence export
This commit is contained in:
@@ -182,6 +182,20 @@ Results are honest QA/QC evidence from persisted detections and persisted
|
||||
reference `vector_features`; no demo detections, live provider fetches or model
|
||||
downloads are introduced by the calibration tool.
|
||||
|
||||
For visual error inspection, export the persisted QA evidence from a calibration
|
||||
summary:
|
||||
|
||||
```bash
|
||||
CALIBRATION_SUMMARY_PATH=/mnt/user/appdata/geointel/artifacts/detection-calibration/20260707T002103Z/calibration_summary.json \
|
||||
bash scripts/export_detection_calibration_evidence.sh http://192.168.10.150:1202
|
||||
```
|
||||
|
||||
The evidence bundle calls
|
||||
`/api/v1/projects/{project_id}/quality-checks/{quality_check_id}/evidence/geojson`
|
||||
for the persisted quality checks and writes combined GeoJSON plus an HTML/SVG
|
||||
review artifact. This is an inspection aid only; it does not rerun inference or
|
||||
alter stored detections.
|
||||
|
||||
### Sprint 8C detection visualization and QA status
|
||||
|
||||
Sprint 8C makes persisted detections reviewable:
|
||||
|
||||
@@ -1,3 +1,33 @@
|
||||
## Sprint 125 Detection calibration evidence bundle (2026-07-07)
|
||||
|
||||
Changed:
|
||||
- Added `scripts/export_detection_calibration_evidence.sh` to turn a persisted detection calibration summary into visual QA evidence artifacts.
|
||||
- The script reads `calibration_summary.json`, fetches the existing project quality-check evidence GeoJSON endpoint for each `quality_check_id`, enriches features with threshold/score/provenance and writes:
|
||||
- `calibration_evidence.geojson`
|
||||
- `calibration_evidence_summary.json`
|
||||
- `calibration_evidence_review.html`
|
||||
- The HTML review artifact renders a simple SVG overview with distinct roles for `match_candidate`, `match_reference`, `false_positive` and `false_negative`.
|
||||
- Registered the script in `scripts/run_readiness_check.sh` as a syntax check.
|
||||
- Added regression coverage in `backend/tests/test_sprint125_detection_calibration_evidence_bundle.py`.
|
||||
- Updated `scripts/README.md`, `backend/README.md`, `docs/AI_PIPELINES.md`, `docs/TODO.md` and `CHANGELOG.md`.
|
||||
|
||||
Validation:
|
||||
- RED: `python -m pytest backend/tests/test_sprint125_detection_calibration_evidence_bundle.py -q` failed because `scripts/export_detection_calibration_evidence.sh` did not exist.
|
||||
- `python -m pytest backend/tests/test_sprint125_detection_calibration_evidence_bundle.py -q` passed.
|
||||
- `bash -n scripts/export_detection_calibration_evidence.sh` passed.
|
||||
- `bash scripts/export_detection_calibration_evidence.sh --help` passed.
|
||||
- Missing-input guard printed usage and did not fetch evidence.
|
||||
|
||||
Open:
|
||||
- Pull the script to Tower and export evidence for the latest Geel calibration summary.
|
||||
|
||||
Limitations:
|
||||
- This is operator tooling only. It does not rerun inference, change API behavior, add UI behavior, fetch providers, seed demo data or download models.
|
||||
- The SVG review is a lightweight geometry overview, not a replacement for full MapLibre evidence review in the workbench.
|
||||
|
||||
Next recommended pass:
|
||||
- Run the evidence export on Tower, inspect the role distribution and use the artifacts to decide whether the current model is worth further threshold tuning or should be replaced.
|
||||
|
||||
## Sprint 124 Detection calibration sweep tooling (2026-07-07)
|
||||
|
||||
Changed:
|
||||
|
||||
@@ -96,6 +96,7 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Fix configured-YOLO mixed-case class labels so `Building` model output matches `building` domain filters.
|
||||
- [x] Persist CRS metadata in raster tile manifests so AI detections can be transformed to WGS84 GeoJSON correctly.
|
||||
- [x] Add real-data detection calibration sweep tooling for confidence-threshold and QA/QC metric comparison.
|
||||
- [x] Add calibration QA evidence export tooling for false-positive/false-negative inspection artifacts.
|
||||
- [ ] Calibrate confidence, IoU and model selection against persisted Geel detections and additional local orthophoto/reference samples.
|
||||
|
||||
## Sprint 8 status
|
||||
|
||||
Reference in New Issue
Block a user