Record unique hard-negative YOLO candidate gate

2026-07-09 03:54:48 +02:00
parent afd2cba7c8
commit 4e9fe6ccb1
6 changed files with 202 additions and 7 deletions
@@ -1,3 +1,51 @@
+## Sprint 146 Unique hard-negative YOLOv8s candidate gate (2026-07-09)
+
+Changed:
+- Fixed the all-in-one Docker runtime so `/app/scripts/train_operator_yolo_detector.sh` is copied into the image and made executable.
+- Added regression coverage in `backend/tests/test_docker_runtime_config.py` so the all-in-one image must include the operator YOLO training wrapper.
+- Hardened `scripts/build_detection_model_promotion_report.py` with explicit `--default-positive-tile-size` and `--default-positive-tile-overlap` options for older positive evidence portfolios that record the model at portfolio level but omit per-run tile provenance.
+- Added regression coverage in `backend/tests/test_sprint143_detection_model_promotion_report.py` for portfolio-level model fallback plus explicit positive tile defaults.
+
+Tested:
+- Red step: `python -m pytest backend\tests\test_docker_runtime_config.py::test_all_in_one_dockerfile_copies_operator_scripts_for_runtime_use -q` failed because the all-in-one Dockerfile did not copy the training wrapper.
+- `python -m pytest backend\tests\test_docker_runtime_config.py -q` (`22 passed`)
+- `bash scripts/run_readiness_check.sh` (`423 passed`; frontend typecheck/build passed; Alembic head `202606120900`)
+- Pushed commit `afd2cba` and redeployed Tower with `.\scripts\deploy_tower.ps1 -InstallAi true`; browser runtime verification and live migration smoke passed.
+- Verified the wrapper in the live container: `/app/scripts/train_operator_yolo_detector.sh` exists and is executable.
+- Trained `geointel-building-yolov8s-uniquehardneg160e50-pt` on Tower from `/app/storage/operator-data/yolo-building-tile-uniquehardneg160/dataset.yaml` using local `/app/models/yolov8s.pt`, 50 CPU epochs, image size `160`, batch `8`.
+- Training completed with final validation precision `0.38`, recall `0.365`, mAP50 `0.278`, mAP50-95 `0.0803`.
+- Live model asset:
+  - `/app/models/geointel-building-yolov8s-uniquehardneg160e50.pt`
+  - model asset id `geointel-building-yolov8s-uniquehardneg160e50-pt`
+  - SHA256 `c4e480273d3da5fc27532cd8bdc3fa7786582e06848ea11b56714ab8cb1750b3`
+- Ran 7-AOI positive calibration sweeps for Geel, Mol, Turnhout, Herentals, Balen, Retie and Westerlo:
+  - output root `/mnt/user/appdata/geointel/artifacts/detection-calibration/uniquehardneg160e50-positive`
+  - best observed AOI result: Westerlo threshold `0.25`, F1 `0.384180790960452`, precision `0.5177664974619289`, recall `0.30538922155688625`, detections `197`
+  - other positive AOIs remained weak, with best F1 roughly `0.10` to `0.17`.
+- Assembled positive evidence portfolio:
+  - output `/mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/uniquehardneg160e50-positive/calibration_evidence_portfolio.json`
+  - sample count `7`
+  - evidence features `17008`
+  - role counts `false_negative=9710`, `false_positive=4734`, `match_candidate=1282`, `match_reference=1282`
+- Ran 9-sample hard-negative/background matrix:
+  - output `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/uniquehardneg160e50/hard_negative_matrix_summary.json`
+  - threshold `0.25`: total background detections `98`, max sample detections `58`
+  - threshold `0.15`: total background detections `137`, max sample detections `85`
+  - threshold `0.05`: total background detections `276`, max sample detections `172`
+- Red step: `python -m pytest backend\tests\test_sprint143_detection_model_promotion_report.py -q` failed because the promotion report could not yet accept explicit positive tile defaults.
+- `python -m pytest backend\tests\test_sprint143_detection_model_promotion_report.py -q` (`2 passed`)
+- Rebuilt the promotion report using explicit positive tile defaults:
+  - output `/mnt/user/appdata/geointel/artifacts/detection-model-promotion/uniquehardneg160e50-positive-vs-hard-negative-v2/detection_model_promotion_report.json`
+  - threshold `0.05`: rejected for `positive_mean_f1_below_gate` and `background_false_positive_pressure`, mean F1 `0.15797188547918842`, max background detections `172`
+  - threshold `0.15`: rejected for the same reasons, mean F1 `0.1598974047548654`, max background detections `85`
+  - threshold `0.25`: rejected for the same reasons, mean F1 `0.15967766715169612`, max background detections `58`
+  - recommended candidate `none`
+
+Open:
+- Do not activate `geointel-building-yolov8s-uniquehardneg160e50-pt` as the V1 default.
+- The candidate improves some individual AOIs but still combines low mean positive F1 with unacceptable false-positive pressure on background samples.
+- Next model work should focus on materially better training data/model strategy, not another default activation attempt with this artifact.
+
 ## Sprint 145 YOLOv8s hardneg r8 e60 full candidate evaluation (2026-07-08)

 Changed: