Add operator hard-negative detection matrix
This commit is contained in:
+18
-1
@@ -215,7 +215,24 @@ and writes `quality_matrix_summary.json` with detection count, QA score,
|
||||
precision, recall, F1, mean IoU and false-positive/false-negative counts. The
|
||||
rankings `best_by_score`, `best_by_recall` and `best_by_precision` are operator
|
||||
decision aids only; GeoIntel still does not download models, seed fixture
|
||||
detections or treat AI detections as ground truth without QA/QC.
|
||||
detections or treat AI detections as ground truth without QA/QC. The same
|
||||
candidate should also pass the background false-positive matrix before it is
|
||||
considered as a default:
|
||||
|
||||
```bash
|
||||
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
|
||||
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos" \
|
||||
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
|
||||
QUALITY_TILE_SIZES="640" \
|
||||
QUALITY_TILE_OVERLAPS="64" \
|
||||
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
|
||||
bash scripts/run_operator_hard_negative_detection_matrix.sh http://192.168.10.150:1202
|
||||
```
|
||||
|
||||
The hard-negative matrix uploads only background rasters and counts detections
|
||||
as false-positive pressure. It does not run QA/QC or invent reference metrics
|
||||
for empty/sparse background AOIs. The first expanded local model improved dense
|
||||
AOI F1, but Kasterlee-bos false positives block default promotion.
|
||||
|
||||
To compare the same model/tile/threshold grid across all prepared operator
|
||||
samples, use:
|
||||
|
||||
@@ -1,3 +1,40 @@
|
||||
## Sprint 132 Operator hard-negative detection matrix (2026-07-07)
|
||||
|
||||
Changed:
|
||||
- Added `scripts/run_operator_hard_negative_detection_matrix.sh`.
|
||||
- The script reads `operator_samples_manifest.json`, selects samples marked `background_candidate` or `allow_empty_reference`, uploads only the raster, generates a tile manifest, checks configured-YOLO preflight, runs `POST /api/v1/detection/run` and counts persisted detections.
|
||||
- It intentionally does not upload reference vectors and does not call detection QA/QC endpoints, because background AOIs have no meaningful reference target.
|
||||
- Added readiness shell-syntax coverage for the new script.
|
||||
- Added regression coverage in `backend/tests/test_sprint132_operator_hard_negative_matrix.py`.
|
||||
- Updated `scripts/README.md`, `docs/TODO.md` and `CHANGELOG.md`.
|
||||
|
||||
Tested:
|
||||
- RED: `python -m pytest backend\tests\test_sprint132_operator_hard_negative_matrix.py -q` failed while `scripts/run_operator_hard_negative_detection_matrix.sh` did not exist.
|
||||
- `python -m pytest backend\tests\test_sprint132_operator_hard_negative_matrix.py -q` passed.
|
||||
- `bash -n scripts/run_operator_hard_negative_detection_matrix.sh` passed.
|
||||
- Live Tower 27-run hard-negative matrix completed:
|
||||
- output: `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/expanded160e50-live/hard_negative_matrix_summary.json`
|
||||
- samples: Postel-bos, Lommel-heide and Kasterlee-bos
|
||||
- models: `geointel-building-yolov8n-expanded160e50-pt`, `geointel-building-yolov8n-tile30-pt`, `yolov8s-building-segmentation-pt`
|
||||
- tile size: `640`
|
||||
- overlap: `64`
|
||||
- thresholds: `0.25`, `0.15`, `0.05`
|
||||
- Live false-positive pressure results:
|
||||
- Postel-bos: expanded160e50 produced 0 detections at `0.25`/`0.15`, 1 at `0.05`; tile30 produced 0/0/1; yolov8s produced 0/3/6.
|
||||
- Lommel-heide: expanded160e50 produced 0 detections at `0.25`/`0.15`, 10 at `0.05`; tile30 produced 0/0/3; yolov8s produced 0/0/0.
|
||||
- Kasterlee-bos: expanded160e50 produced 38/46/76 detections at `0.25`/`0.15`/`0.05`; tile30 produced 15/22/42; yolov8s produced 5/6/7.
|
||||
|
||||
Open:
|
||||
- None for the hard-negative matrix tooling itself.
|
||||
|
||||
Limitations:
|
||||
- Background matrix scores false-positive pressure from detection counts only. It does not calculate precision/recall/F1 because background candidates intentionally do not provide a full reference target.
|
||||
- Kasterlee-bos still has 7 GRB features and is best interpreted as a sparse/hard-negative AOI, not a purely empty background tile.
|
||||
- `geointel-building-yolov8n-expanded160e50-pt` should not be promoted to default model while Kasterlee-bos false-positive pressure remains high.
|
||||
|
||||
Next recommended pass:
|
||||
- Train a hard-negative-balanced candidate: oversample sparse/background tiles, lower the dense-AOI max-detection bias, and rerun both dense QA matrix and hard-negative matrix before changing any default model selection.
|
||||
|
||||
## Sprint 131 Operator sample expansion and negative-tile YOLO candidate (2026-07-07)
|
||||
|
||||
Changed:
|
||||
|
||||
+2
-1
@@ -105,7 +105,8 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Train and benchmark the first tile-level local YOLO candidate on Tower through the persisted QA/QC matrix.
|
||||
- [x] Calibrate confidence, IoU and model selection against additional local orthophoto/reference samples beyond Geel/Mol/Turnhout.
|
||||
- [x] Add negative/background AOIs to the operator sample corpus and train an expanded local tile-level YOLO candidate.
|
||||
- [ ] Add a hard-negative model-quality pass with more sparse/background AOIs, balanced tile export and explicit false-positive scoring.
|
||||
- [x] Add a hard-negative model-quality pass with sparse/background AOIs and explicit false-positive scoring.
|
||||
- [ ] Train a hard-negative-balanced YOLO candidate and rerun dense QA plus background false-positive matrices.
|
||||
- [ ] Find or train a materially stronger aerial/Kempen building model candidate; `geointel-building-yolov8n-expanded160e50-pt` is the best current dense-AOI candidate but still too weak and too noisy for a V1 default.
|
||||
|
||||
## Sprint 8 status
|
||||
|
||||
Reference in New Issue
Block a user