diff --git a/CHANGELOG.md b/CHANGELOG.md index 1222a8fc..70cbd317 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,13 @@ # Changelog +## Sprint 145 YOLOv8s hardneg r8 e60 full candidate evaluation (2026-07-08) + +- Completed the Tower-local YOLOv8s hard-negative r8 training run through 60 CPU epochs and published local model asset `geointel-building-yolov8s-hardneg160r8e60-pt`. +- Ran the full 7-AOI positive matrix, hard-negative matrix, evidence portfolio and promotion report for the completed e60 artifact. +- Result: the model is rejected. It reduces Kasterlee-bos background detections versus `expanded160e50` at threshold `0.15` (`11` versus `46`), but mean positive F1 remains too low (`0.07870592446136859` at threshold `0.15`). +- No backend API, migration, frontend runtime, model download, provider-fetching behavior or active model configuration changed. + ## Sprint 144 YOLOv8s hardneg r8 partial candidate evaluation (2026-07-08) - Started a Tower-local YOLOv8s training run on the `yolo-building-tile-hardneg160r8` dataset with requested 60 epochs. diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index d6ae30df..0befb732 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -1,3 +1,44 @@ +## Sprint 145 YOLOv8s hardneg r8 e60 full candidate evaluation (2026-07-08) + +Changed: +- Verified that the Tower-local YOLOv8s hardneg r8 training run completed all 60 requested CPU epochs. +- Finalized the completed model artifact: + - `/app/models/geointel-building-yolov8s-hardneg160r8e60.pt` + - model asset id `geointel-building-yolov8s-hardneg160r8e60-pt` + - SHA256 `d7daea04bd51a54a06944f0d4bf1961fe453dceb1fb34ef51daa73f6901fca81` +- Wrote `/app/storage/training/operator-yolo/geointel-building-yolov8s-hardneg160r8e60/training_summary.json`. + +Tested: +- Training summary reports `status=ok`, requested epochs `60`, completed epochs `60`. +- Ran 7-AOI positive matrix: + - output `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/yolov8s-hardneg160r8e60-positive-20260708/multi_sample_quality_summary.json` + - sample count `7` + - run count `21` + - best result: Westerlo threshold `0.15`, F1/score `0.20588235294117646`, precision `0.26666666666666666`, recall `0.16766467065868262`, detections `210`, false positives `154`, false negatives `278`. +- Ran hard-negative matrix: + - output `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/yolov8s-hardneg160r8e60-live/hard_negative_matrix_summary.json` + - Postel-bos detections `0/0/0` at thresholds `0.05/0.15/0.25` + - Lommel-heide detections `0/0/0` + - Kasterlee-bos detections `30/11/9` +- Assembled evidence portfolio: + - output `/mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/yolov8s-hardneg160r8e60-positive-20260708/output/calibration_evidence_portfolio.json` + - sample count `7` + - evidence features `15714` + - role counts `false_negative=10468`, `false_positive=4198`, `match_candidate=524`, `match_reference=524` +- Ran promotion report: + - output `/mnt/user/appdata/geointel/artifacts/detection-model-promotion/yolov8s-hardneg160r8e60-20260708/detection_model_promotion_report.json` + - evaluated `3` candidate thresholds + - recommended candidate `none` + - threshold `0.05`: mean F1 `0.07825931710275633`, max background detections `30`, rejected for `positive_mean_f1_below_gate` and `background_false_positive_pressure` + - threshold `0.15`: mean F1 `0.07870592446136859`, max background detections `11`, rejected for the same reasons + - threshold `0.25`: mean F1 `0.06111741186503092`, max background detections `9`, rejected for the same reasons +- Comparison baseline: `geointel-building-yolov8n-expanded160e50-pt|640|64|0.15` had mean F1 `0.19718007234060198` and max background detections `46`. + +Open: +- The e60 YOLOv8s hardneg r8 candidate improves hard-negative behavior at threshold `0.15` compared with `expanded160e50`, but its positive-AOI mean F1 is too low for operational use. +- Do not activate this model as default. +- Next model pass should focus on improving positive recall/fit without losing the hard-negative gains, likely through better labels, more positive AOIs, or a different architecture/training strategy rather than simply extending this same run. + ## Sprint 144 YOLOv8s hardneg r8 partial candidate evaluation (2026-07-08) Changed: diff --git a/docs/TODO.md b/docs/TODO.md index fd6e1739..dd3cd486 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -432,6 +432,7 @@ This file now starts with the current implementation status. Older preparation/b - [x] Prevent same-threshold calibration evidence responses from overwriting each other in multi-model portfolios. - [x] Add a model promotion decision report that combines positive-AOI score with hard-negative false-positive pressure. - [x] Train and reject a YOLOv8s hard-negative r8 partial candidate after 12 CPU epochs through the full positive/background promotion gate. +- [x] Finish and reject the full YOLOv8s hard-negative r8 e60 candidate through the same promotion gate. - [ ] Add more AOIs after the tile-level baseline so the next local model attempt is not limited to Geel/Mol/Turnhout. - [ ] Add negative/background AOIs so the next tile dataset is not all positive tiles. -- [ ] Finish or resume a long-running higher-capacity YOLO training pass outside the 1-hour interactive command window, then rerun the same promotion gate. +- [ ] Improve positive training coverage/label quality before the next higher-capacity model attempt; simply extending the same hardneg r8 run is not enough.