Expand operator samples for YOLO hard negatives

2026-07-07 21:42:23 +02:00
parent e8d79fccbb
commit 89c5729d33
8 changed files with 288 additions and 23 deletions
@@ -1,3 +1,54 @@
+## Sprint 131 Operator sample expansion and negative-tile YOLO candidate (2026-07-07)
+
+Changed:
+- Extended `scripts/prepare_operator_real_data_samples.py` with `sample_role` and `allow_empty_reference`.
+- Added reference AOIs for Herentals, Balen, Retie and Westerlo.
+- Added background-candidate AOIs for Postel-bos, Lommel-heide and Kasterlee-bos. Background candidates can persist empty GRB FeatureCollections for negative-tile training, while normal reference samples still fail on empty GRB results.
+- Added regression coverage in `backend/tests/test_sprint131_operator_sample_expansion.py`.
+- Updated `scripts/README.md`, `backend/README.md`, `docs/AI_PIPELINES.md`, `docs/TODO.md` and `CHANGELOG.md`.
+
+Tested:
+- RED: `python -m pytest backend\tests\test_sprint131_operator_sample_expansion.py -q` failed before the new sample metadata and background candidates existed.
+- `python -m pytest backend\tests\test_sprint131_operator_sample_expansion.py -q` passed.
+- `python -m py_compile scripts\prepare_operator_real_data_samples.py` passed.
+- `python scripts\prepare_operator_real_data_samples.py --help` passed.
+- Live Tower operator sample preparation passed:
+  - manifest: `/app/storage/operator-data/operator_samples_manifest.json`
+  - samples: Geel `617`, Mol `374`, Turnhout `773`, Herentals `665`, Balen `309`, Retie `592`, Westerlo `334`, Postel-bos `0`, Lommel-heide `0`, Kasterlee-bos `7` reference features.
+- Live Tower expanded tile export passed:
+  - dataset: `/app/storage/operator-data/yolo-building-tile-expanded160`
+  - tile size: `160`
+  - stride: `80`
+  - exported tiles: `360`
+  - positive tiles: `260`
+  - negative tiles: `100`
+  - labels: `11213`
+  - train tiles: `252`
+  - validation tiles: `108`
+- Live Tower 50-epoch CPU training passed:
+  - output model: `/app/models/geointel-building-yolov8n-expanded160e50.pt`
+  - catalog asset: `geointel-building-yolov8n-expanded160e50-pt`
+  - SHA256: `bf6a5e8d25a62d784ee53764ea11d7ce89c4e7aeeac7588010e497b8d7dafb2b`
+  - final validation: precision `0.428`, recall `0.389`, mAP50 `0.318`, mAP50-95 `0.106`
+- Live API preflight passed for `geointel-building-yolov8n-expanded160e50-pt` with `status=ready`, `model_load_ok=true`, `manifest_valid=true`, `tile_paths_exist=true`, `will_download_models=false` and `will_run_inference=false`.
+- Live 45-run multi-sample QA matrix completed:
+  - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/expanded160e50-live/multi_sample_quality_summary.json`
+  - command compared `geointel-building-yolov8n-expanded160e50-pt`, `geointel-building-yolov8n-tile30-pt` and `yolov8s-building-segmentation-pt` over Geel, Mol, Turnhout, Retie and Kasterlee-bos with tile `640`, overlap `64`, thresholds `0.25`/`0.15`/`0.05`.
+  - best overall score and recall: Geel, `geointel-building-yolov8n-expanded160e50-pt`, tile `640`, threshold `0.05`, 300 detections, 91 matches, 209 false positives, 526 false negatives, precision `0.30333333333333334`, recall `0.14748784440842788`, F1 `0.1984732824427481`.
+  - dense-sample score winners: Geel, Mol, Turnhout and Retie all selected `geointel-building-yolov8n-expanded160e50-pt`.
+  - hard-negative/sparse-sample winner: Kasterlee-bos selected `yolov8s-building-segmentation-pt`, threshold `0.25`, F1 `0.16666666666666666`; the expanded local model produced too many false positives there.
+
+Open:
+- None for the sample-preparation and expanded-training runtime proof itself.
+
+Limitations:
+- This remains operator tooling only. It does not add Training Studio, browser training controls, provider fetching, fake detections, model auto-provisioning or API contract changes.
+- `geointel-building-yolov8n-expanded160e50-pt` is the best tested candidate on dense operator AOIs, but it is still experimental and should not become the V1 default until hard-negative false positives improve.
+- The next model pass should add more sparse/background AOIs, tune confidence/NMS/max-detection settings and compare a stronger architecture or longer run against the same persisted QA matrix.
+
+Next recommended pass:
+- Build a hard-negative model-quality pass: expand sparse/background AOIs, export a balanced tile dataset, train a stronger candidate, and rerun the multi-sample QA matrix with dense and background samples scored separately.
+
 ## Sprint 130 Operator YOLO tile-level dataset tooling (2026-07-07)

 Changed: