Add persistent false negative evidence audit
This commit is contained in:
@@ -684,6 +684,41 @@ QA evidence GeoJSON endpoint, writes `calibration_evidence.geojson`,
|
||||
matched references, false positives and false negatives. Set
|
||||
`CALIBRATION_EVIDENCE_MODE=best` to export only the `best_by_score` run.
|
||||
|
||||
Build fixed-threshold portfolio inputs when two model runs must be compared at
|
||||
the same confidence threshold across every AOI:
|
||||
|
||||
```bash
|
||||
python scripts/build_fixed_threshold_evidence_portfolio_inputs.py \
|
||||
--multi-sample-summary artifacts/detection-quality-matrix/multi-sample/<run>/multi_sample_quality_summary.json \
|
||||
--threshold 0.35 \
|
||||
--model-asset-id geointel-building-yolov8s-aoi1024bg512r3e50-pt \
|
||||
--model-sha256 e0980572aac90e7efc514608eb16d7de5bfbf27a4bbec04e7bc1bc8c02f9601f \
|
||||
--tile-size 512 \
|
||||
--tile-overlap 64 \
|
||||
--output-dir artifacts/detection-false-negative-review/active-inputs
|
||||
```
|
||||
|
||||
The builder selects exactly one persisted QA run per AOI and refuses ambiguous
|
||||
model/tile/threshold matches. Pass its emitted manifest to
|
||||
`assemble_detection_calibration_evidence_portfolio.sh` with
|
||||
`CALIBRATION_EVIDENCE_MODE=all`; each filtered summary contains one run.
|
||||
|
||||
Compare two or more downloaded evidence portfolios with geodetic WGS84 areas:
|
||||
|
||||
```bash
|
||||
python scripts/audit_detection_false_negative_evidence.py \
|
||||
--portfolio active=artifacts/detection-false-negative-review/active/calibration_evidence_portfolio.json \
|
||||
--portfolio candidate=artifacts/detection-false-negative-review/candidate/calibration_evidence_portfolio.json \
|
||||
--output-dir artifacts/detection-false-negative-review/audit
|
||||
```
|
||||
|
||||
The audit reports false-negative rates and area buckets per AOI/model, plus
|
||||
reference buildings missed by every compared portfolio. Stable
|
||||
`source_feature_id` values are preferred; a normalized geometry fingerprint is
|
||||
used only when source IDs are absent. Invalid or missing geometry fails the
|
||||
audit instead of being silently skipped. The tools do not run inference,
|
||||
create QA records, mutate model defaults or download data/models.
|
||||
|
||||
Docker images install only the GIS runtime by default. To build a local/Tower
|
||||
image with PyTorch/Ultralytics available for the configured-YOLO preflight and
|
||||
runtime path, set:
|
||||
|
||||
Reference in New Issue
Block a user