Add persistent false negative evidence audit
GeoIntel CI / docs-smoke (push) Has been cancelled
GeoIntel CI / contract-smoke (push) Has been cancelled

This commit is contained in:
Codex
2026-07-12 23:30:52 +02:00
parent 1a8e54e21a
commit 53cd38a5b2
6 changed files with 775 additions and 0 deletions
+35
View File
@@ -684,6 +684,41 @@ QA evidence GeoJSON endpoint, writes `calibration_evidence.geojson`,
matched references, false positives and false negatives. Set
`CALIBRATION_EVIDENCE_MODE=best` to export only the `best_by_score` run.
Build fixed-threshold portfolio inputs when two model runs must be compared at
the same confidence threshold across every AOI:
```bash
python scripts/build_fixed_threshold_evidence_portfolio_inputs.py \
--multi-sample-summary artifacts/detection-quality-matrix/multi-sample/<run>/multi_sample_quality_summary.json \
--threshold 0.35 \
--model-asset-id geointel-building-yolov8s-aoi1024bg512r3e50-pt \
--model-sha256 e0980572aac90e7efc514608eb16d7de5bfbf27a4bbec04e7bc1bc8c02f9601f \
--tile-size 512 \
--tile-overlap 64 \
--output-dir artifacts/detection-false-negative-review/active-inputs
```
The builder selects exactly one persisted QA run per AOI and refuses ambiguous
model/tile/threshold matches. Pass its emitted manifest to
`assemble_detection_calibration_evidence_portfolio.sh` with
`CALIBRATION_EVIDENCE_MODE=all`; each filtered summary contains one run.
Compare two or more downloaded evidence portfolios with geodetic WGS84 areas:
```bash
python scripts/audit_detection_false_negative_evidence.py \
--portfolio active=artifacts/detection-false-negative-review/active/calibration_evidence_portfolio.json \
--portfolio candidate=artifacts/detection-false-negative-review/candidate/calibration_evidence_portfolio.json \
--output-dir artifacts/detection-false-negative-review/audit
```
The audit reports false-negative rates and area buckets per AOI/model, plus
reference buildings missed by every compared portfolio. Stable
`source_feature_id` values are preferred; a normalized geometry fingerprint is
used only when source IDs are absent. Invalid or missing geometry fails the
audit instead of being silently skipped. The tools do not run inference,
create QA records, mutate model defaults or download data/models.
Docker images install only the GIS runtime by default. To build a local/Tower
image with PyTorch/Ultralytics available for the configured-YOLO preflight and
runtime path, set: