From 9bef905da8c65f0e80ae39aaf0bceab8c5f12a6b Mon Sep 17 00:00:00 2001 From: Codex Date: Thu, 9 Jul 2026 04:19:48 +0200 Subject: [PATCH] Record AOI512 YOLO candidate gate --- CHANGELOG.md | 9 ++++++ docs/CODEX_EXECUTION_LOG.md | 58 +++++++++++++++++++++++++++++++++++++ docs/TODO.md | 2 ++ 3 files changed, 69 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f815f017..0bd3fe5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ # Changelog +## Sprint 147 AOI512 YOLOv8s scale-match candidate gate (2026-07-09) + +- Built and audited an AOI-scale YOLO dataset at `512px` tile size to test whether the previous `160px` training scale was the main quality blocker. +- Trained Tower-local model asset `geointel-building-yolov8s-aoi512e80-pt` from `/app/storage/operator-data/yolo-building-aoi512-uniquehardneg`. +- Ran 7 positive AOI sweeps, a 17,156-feature evidence portfolio, a 9-sample hard-negative/background matrix and a promotion report. +- Result: the candidate is rejected. The best threshold `0.25` reached mean positive F1 `0.13511851520077328` and still produced max background detections `56`. +- Conclusion: scale-match training helps the training validation curve but does not solve operational persisted QA/QC quality. The next model pass needs better positive AOI coverage and label strategy, not only more epochs or another threshold. +- No API contract, migration, frontend behavior, provider fetching, model download or active model configuration changed. + ## Sprint 146 Unique hard-negative YOLOv8s candidate gate (2026-07-09) - Fixed the all-in-one Docker image so the operator YOLO training wrapper is available at `/app/scripts/train_operator_yolo_detector.sh`. diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index df2be3e7..42bb5ae3 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -1,3 +1,61 @@ +## Sprint 147 AOI512 YOLOv8s scale-match candidate gate (2026-07-09) + +Changed: +- Built an alternate operator YOLO dataset at AOI scale instead of small 160px tiles: + - output `/app/storage/operator-data/yolo-building-aoi512-uniquehardneg` + - tile size `512`, stride `512` + - validation samples `turnhout`, `retie`, `westerlo`, `arendonk_heide` + - 16 total images, 13 positive images, 3 negative images, 3172 labels +- Audited the AOI512 dataset: + - output `/mnt/user/appdata/geointel/artifacts/operator-yolo-dataset-audits/aoi512-uniquehardneg/operator_yolo_dataset_quality_audit.json` + - status `ok` + - no missing label files + - no invalid label rows + - 13 positive samples + - 3 positive validation samples + - median normalized box area `0.000793456875` +- Trained a Tower-local YOLOv8s scale-match candidate: + - dataset `/app/storage/operator-data/yolo-building-aoi512-uniquehardneg/dataset.yaml` + - base model `/app/models/yolov8s.pt` + - 80 CPU epochs + - image size `512` + - batch `4` + - artifact `/app/models/geointel-building-yolov8s-aoi512e80.pt` + - model asset id `geointel-building-yolov8s-aoi512e80-pt` + - SHA256 `b796284a13358498c296fa41270dd234a81718a76cdafe62490a5206bb86ac5b` +- The training validation improved versus the previous 160px candidate but remained weak: + - precision `0.404` + - recall `0.270` + - mAP50 `0.163` + - mAP50-95 `0.0505` + +Tested: +- Ran 7-AOI positive calibration sweeps for Geel, Mol, Turnhout, Herentals, Balen, Retie and Westerlo: + - output root `/mnt/user/appdata/geointel/artifacts/detection-calibration/aoi512e80-positive` + - best sample: Westerlo threshold `0.25`, F1 `0.23509933774834438`, precision `0.26296296296296295`, recall `0.2125748502994012`, detections `270` + - most other AOIs remained around F1 `0.10` to `0.14` +- Ran 9-sample hard-negative/background matrix: + - output `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/aoi512e80/hard_negative_matrix_summary.json` + - threshold `0.25`: total background detections `105`, max sample detections `56` + - threshold `0.15`: total background detections `181`, max sample detections `100` + - threshold `0.05`: total background detections `503`, max sample detections `278` +- Assembled positive evidence portfolio: + - output `/mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/aoi512e80-positive/calibration_evidence_portfolio.json` + - sample count `7` + - evidence features `17156` +- Ran promotion report: + - output `/mnt/user/appdata/geointel/artifacts/detection-model-promotion/aoi512e80-positive-vs-hard-negative/detection_model_promotion_report.json` + - threshold `0.05`: rejected for `positive_mean_f1_below_gate` and `background_false_positive_pressure`, mean F1 `0.1335249717919908`, max background detections `278` + - threshold `0.15`: rejected for the same reasons, mean F1 `0.1335249717919908`, max background detections `100` + - threshold `0.25`: rejected for the same reasons, mean F1 `0.13511851520077328`, max background detections `56` + - recommended candidate `none` + +Open: +- Do not activate `geointel-building-yolov8s-aoi512e80-pt` as the V1 default. +- AOI-scale training improves the Ultralytics validation curve but does not improve persisted QA/QC enough for operational use. +- The recurring failure mode is overproduction near the 300-detection cap with low recall and too many false positives. +- Next recommended pass: add/export more diverse positive AOIs and improve label quality/geometry-to-box strategy before training another higher-capacity model. A pure scale or epoch change is not enough. + ## Sprint 146 Unique hard-negative YOLOv8s candidate gate (2026-07-09) Changed: diff --git a/docs/TODO.md b/docs/TODO.md index f940187f..c6e5d0cb 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -109,8 +109,10 @@ This file now starts with the current implementation status. Older preparation/b - [x] Train a hard-negative-balanced YOLO candidate and rerun dense QA plus background false-positive matrices. - [x] Benchmark an external remote-sensing YOLOv8l building candidate as an explicit local model asset. - [x] Train and gate the `uniquehardneg160e50` YOLOv8s candidate through 7 positive AOIs and 9 hard-negative/background samples. +- [x] Train and gate an AOI-scale `aoi512e80` YOLOv8s candidate to test the 160px training-scale hypothesis. - [ ] Find or train a materially stronger aerial/Kempen building model candidate; `geointel-building-yolov8n-expanded160e50-pt` is the best current dense-AOI candidate but still too weak and too noisy for a V1 default. - [ ] Train a higher-capacity local aerial-building detector with stronger positive recall while preserving the hard-negative false-positive gate. +- [ ] Add more diverse positive AOIs and revisit geometry-to-box label strategy before the next default-model training attempt. ## Sprint 8 status