From e0ab504c8161242a764333f19fb71bf0cbc7b3f3 Mon Sep 17 00:00:00 2001 From: Codex Date: Wed, 8 Jul 2026 18:32:09 +0200 Subject: [PATCH] Record YOLOv8s partial candidate evaluation --- CHANGELOG.md | 8 +++++++ docs/CODEX_EXECUTION_LOG.md | 45 +++++++++++++++++++++++++++++++++++++ docs/TODO.md | 2 ++ 3 files changed, 55 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1abae8c7..1222a8fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,14 @@ # Changelog +## Sprint 144 YOLOv8s hardneg r8 partial candidate evaluation (2026-07-08) + +- Started a Tower-local YOLOv8s training run on the `yolo-building-tile-hardneg160r8` dataset with requested 60 epochs. +- Preserved the 12-epoch `best.pt` artifact as explicit partial model asset `geointel-building-yolov8s-hardneg160r8e12partial-pt` after the CPU training command reached the 1-hour command limit. +- Ran the partial candidate through the 7-AOI positive matrix, hard-negative matrix, evidence portfolio and promotion report. +- Result: the partial candidate is rejected. Best positive F1 was Westerlo at `0.14826498422712936`, mean positive F1 at threshold `0.05` was `0.05026994383963278`, and Kasterlee-bos still produced 18 detections at threshold `0.05`. +- No backend API, migration, frontend runtime, model download, provider-fetching behavior or active model configuration changed. + ## Sprint 143 Detection model promotion decision report (2026-07-08) - Added `scripts/build_detection_model_promotion_report.py` for operator-only model promotion review. diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index 6b93f2ff..d6ae30df 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -1,3 +1,48 @@ +## Sprint 144 YOLOv8s hardneg r8 partial candidate evaluation (2026-07-08) + +Changed: +- Started a Tower-local YOLOv8s training run using: + - dataset `/app/storage/operator-data/yolo-building-tile-hardneg160r8/dataset.yaml` + - base model `/app/models/yolov8s.pt` + - requested epochs `60` + - image size `640` + - batch `2` + - device `cpu` +- The Codex command reached its 1-hour timeout after 12 completed epochs; the run had produced `weights/best.pt` and `weights/last.pt`. +- Preserved the partial best artifact as `/app/models/geointel-building-yolov8s-hardneg160r8e12partial.pt`. +- Wrote `/app/storage/training/operator-yolo/geointel-building-yolov8s-hardneg160r8e60/training_summary_partial_e12.json`. +- Treated the artifact explicitly as a partial evaluation candidate, not as a completed 60-epoch model. + +Tested: +- Live model catalog listed `geointel-building-yolov8s-hardneg160r8e12partial-pt` with SHA256 `0246202cddc47eb994a0afc9ee10d56b72298bd1cdc0b72b75e12b28e2202330`. +- Ran 7-AOI positive matrix: + - output `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/yolov8s-hardneg160r8e12partial-positive-20260708/multi_sample_quality_summary.json` + - sample count `7` + - run count `21` + - best result: Westerlo threshold `0.05`, F1/score `0.14826498422712936`, precision `0.15666666666666668`, recall `0.1407185628742515`, detections `300`, false positives `253`, false negatives `287`. +- Ran hard-negative matrix: + - output `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/yolov8s-hardneg160r8e12partial-live/hard_negative_matrix_summary.json` + - Postel-bos detections `0/0/0` at thresholds `0.05/0.15/0.25` + - Lommel-heide detections `0/0/0` + - Kasterlee-bos detections `18/1/0` +- Assembled evidence portfolio: + - output `/mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/yolov8s-hardneg160r8e12partial-positive-20260708/output/calibration_evidence_portfolio.json` + - sample count `7` + - evidence features `12908` + - role counts `false_negative=10861`, `false_positive=1785`, `match_candidate=131`, `match_reference=131` +- Ran promotion report: + - output `/mnt/user/appdata/geointel/artifacts/detection-model-promotion/yolov8s-hardneg160r8e12partial-20260708/detection_model_promotion_report.json` + - evaluated `3` candidate thresholds + - recommended candidate `none` + - threshold `0.05` rejected for `positive_mean_f1_below_gate` and `background_false_positive_pressure`, with mean F1 `0.05026994383963278` and max background detections `18` + - threshold `0.15` rejected for the same reasons, with mean F1 `0.0022606965174129354` and max background detections `1` + - threshold `0.25` rejected for insufficient positive evidence and positive F1 below gate. + +Open: +- The partial YOLOv8s r8 candidate is materially worse than the existing `expanded160e50` positive-AOI baseline and must not be activated. +- CPU-only training is too slow for a complete 60-epoch YOLOv8s pass inside a 1-hour interactive command window. +- Next pass should either resume/finish long-running training outside the interactive timeout or use GPU/accelerated runtime; only then rerun the same positive, hard-negative, evidence portfolio and promotion gates. + ## Sprint 143 Detection model promotion decision report (2026-07-08) Changed: diff --git a/docs/TODO.md b/docs/TODO.md index 02b03a1a..fd6e1739 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -431,5 +431,7 @@ This file now starts with the current implementation status. Older preparation/b - [x] Preserve model/tile provenance in calibration evidence bundle summaries. - [x] Prevent same-threshold calibration evidence responses from overwriting each other in multi-model portfolios. - [x] Add a model promotion decision report that combines positive-AOI score with hard-negative false-positive pressure. +- [x] Train and reject a YOLOv8s hard-negative r8 partial candidate after 12 CPU epochs through the full positive/background promotion gate. - [ ] Add more AOIs after the tile-level baseline so the next local model attempt is not limited to Geel/Mol/Turnhout. - [ ] Add negative/background AOIs so the next tile dataset is not all positive tiles. +- [ ] Finish or resume a long-running higher-capacity YOLO training pass outside the 1-hour interactive command window, then rerun the same promotion gate.