From e8d79fccbbc6a35047f7f0868cba3d32999b819e Mon Sep 17 00:00:00 2001 From: Codex Date: Tue, 7 Jul 2026 21:15:58 +0200 Subject: [PATCH] Record operator tile model benchmark results --- CHANGELOG.md | 5 +++++ docs/CODEX_EXECUTION_LOG.md | 28 ++++++++++++++++++++++++++-- docs/TODO.md | 9 ++++++--- 3 files changed, 37 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0294c848..6fd0f490 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,11 @@ - Added readiness coverage for the tile exporter Python compile check. - Added regression coverage in `backend/tests/test_sprint130_operator_yolo_tile_dataset.py` for script contract, help behavior without GIS imports, edge-covering tile windows and deterministic negative-tile selection. - Updated operator documentation for tile-level dataset export and reuse of the existing local training wrapper. +- Live Tower tile export produced `/app/storage/operator-data/yolo-building-tile-dataset` with 75 overlapping tiles and 5321 clipped building labels from the Geel/Mol/Turnhout operator samples. +- Live Tower 30-epoch CPU training produced `/app/models/geointel-building-yolov8n-tile30.pt`; the model catalog exposes it as `geointel-building-yolov8n-tile30-pt` with SHA256 `b9e228202500d7c85836d12a72e320f4f2f0cef24cbb1b5bf7fa78a6778390af`. +- Live YOLO preflight loaded `geointel-building-yolov8n-tile30-pt` successfully with `status=ready`, `model_load_ok=true`, `manifest_valid=true`, `tile_paths_exist=true`, `will_download_models=false` and `will_run_inference=false`. +- Live 48-run Geel/Mol/Turnhout QA matrix compared `geointel-building-yolov8n-tile30-pt` with `yolov8s-building-segmentation-pt`; best overall score was Mol with the tile model, tile `640`, threshold `0.15`, precision `0.13602941176470587`, recall `0.09893048128342247`, F1 `0.11455108359133127`. +- Result decision: the tile-trained local model is now the best tested candidate on Geel/Mol and best overall, but remains experimental and should not become the V1 default until more AOIs and negative/background samples materially improve recall and false-positive behavior. - No Training Studio UI, API contract change, provider fetching, model auto-provisioning or app-side model training behavior was introduced. ## Sprint 129 Operator YOLO training dataset tooling (2026-07-07) diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index cd1d3435..875b41e9 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -13,15 +13,39 @@ Tested: - `python -m pytest backend\tests\test_sprint130_operator_yolo_tile_dataset.py -q` passed. - `python scripts\export_operator_yolo_tile_dataset.py --help` passed without requiring local GIS dependencies. - `python -m py_compile scripts\export_operator_yolo_tile_dataset.py` passed. +- Live Tower tile export passed: + - dataset: `/app/storage/operator-data/yolo-building-tile-dataset` + - samples: Geel, Mol and Turnhout + - tile size: `192` + - stride: `96` + - exported tiles: `75` + - positive tiles: `75` + - labels: `5321` + - validation split: Turnhout +- Live Tower 30-epoch CPU tile training passed: + - output model: `/app/models/geointel-building-yolov8n-tile30.pt` + - catalog asset: `geointel-building-yolov8n-tile30-pt` + - SHA256: `b9e228202500d7c85836d12a72e320f4f2f0cef24cbb1b5bf7fa78a6778390af` + - final validation: precision `0.208`, recall `0.271`, mAP50 `0.122`, mAP50-95 `0.0308` +- Live API preflight passed for `geointel-building-yolov8n-tile30-pt` with `status=ready`, `model_load_ok=true`, `manifest_valid=true`, `tile_paths_exist=true`, `will_download_models=false` and `will_run_inference=false`. +- Live 48-run multi-sample QA matrix completed: + - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T190720Z/multi_sample_quality_summary.json` + - command compared `geointel-building-yolov8n-tile30-pt` with `yolov8s-building-segmentation-pt` over Geel, Mol and Turnhout, tile sizes `512`/`640`, overlap `64`, thresholds `0.50`/`0.25`/`0.15`/`0.05`. + - best overall score: Mol, `geointel-building-yolov8n-tile30-pt`, tile `640`, threshold `0.15`, 272 detections, 37 matches, 235 false positives, 337 false negatives, precision `0.13602941176470587`, recall `0.09893048128342247`, F1 `0.11455108359133127`. + - best overall recall: Mol, `geointel-building-yolov8n-tile30-pt`, tile `512`, threshold `0.05`, 544 detections, 44 matches, 500 false positives, 330 false negatives, precision `0.08088235294117647`, recall `0.11764705882352941`, F1 `0.09586056644880174`. + - best overall precision: Turnhout, `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.25`, precision `0.4`, recall `0.01034928848641656`, F1 `0.0201765447667087`. + - per-sample score winners: Geel `geointel-building-yolov8n-tile30-pt` F1 `0.09671179883945842`; Mol `geointel-building-yolov8n-tile30-pt` F1 `0.11455108359133127`; Turnhout `yolov8s-building-segmentation-pt` F1 `0.09971777986829727`. Open: -- Run the tile exporter inside the AI-enabled Tower runtime, train a local tile-level model, and benchmark it through the existing multi-sample Detection + QA matrix. +- None for the tile exporter/training runtime proof itself. Limitations: - This remains operator tooling only. It does not add Training Studio, browser training controls, provider fetching, fake detections, model auto-provisioning or API contract changes. +- `geointel-building-yolov8n-tile30-pt` is an experimental local candidate, not a V1 default. It improves the operator-trained baseline materially but still has low recall and many false positives on the current 3-sample corpus. +- The current corpus is too small and all exported tiles were positive; the next model pass needs more AOIs and deliberate negative/background tiles. Next recommended pass: -- Export the tile-level dataset on Tower with `tile-size=192`, `stride=96`, train a longer local model, and compare it against the current `yolov8s-building-segmentation-pt` baseline. +- Expand the operator sample corpus beyond Geel/Mol/Turnhout, include negative/background AOIs, regenerate the tile dataset, then train a longer/larger local model candidate and rerun the same persisted QA matrix. ## Sprint 129 Operator YOLO training dataset tooling (2026-07-07) diff --git a/docs/TODO.md b/docs/TODO.md index 3c4fbc88..02a1e664 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -101,8 +101,10 @@ This file now starts with the current implementation status. Older preparation/b - [x] Add reproducible Geel/Mol/Turnhout operator sample preparation and multi-sample quality matrix tooling. - [x] Run first Geel/Mol/Turnhout persisted detection quality baseline. - [x] Add and benchmark a stronger `yolov8s` building-segmentation runtime model candidate. -- [ ] Calibrate confidence, IoU and model selection against persisted Geel/Mol/Turnhout detections and any additional local orthophoto/reference samples. -- [ ] Find or train a materially stronger aerial/Kempen building model candidate; current `yolov8s` recall remains too low for a V1 default. +- [x] Add operator-only tile-level YOLO dataset export with overlapping windows and deterministic negative tile retention. +- [x] Train and benchmark the first tile-level local YOLO candidate on Tower through the persisted QA/QC matrix. +- [ ] Calibrate confidence, IoU and model selection against additional local orthophoto/reference samples beyond Geel/Mol/Turnhout. +- [ ] Find or train a materially stronger aerial/Kempen building model candidate; `geointel-building-yolov8n-tile30-pt` is the best current overall candidate but still too weak for a V1 default. ## Sprint 8 status @@ -407,5 +409,6 @@ This file now starts with the current implementation status. Older preparation/b - [x] Add operator-only YOLO dataset export and local training-smoke wrapper for real sample calibration. - [x] Train/evaluate a small local GeoIntel building-detector smoke from the current operator samples and reject it because QA/QC did not improve. - [x] Add operator-only tile-level YOLO dataset export with overlapping windows and deterministic negative tile retention. -- [ ] Run tile-level training on Tower and accept/reject the resulting local model through the persisted QA/QC matrix. +- [x] Run tile-level training on Tower and accept/reject the resulting local model through the persisted QA/QC matrix. - [ ] Add more AOIs after the tile-level baseline so the next local model attempt is not limited to Geel/Mol/Turnhout. +- [ ] Add negative/background AOIs so the next tile dataset is not all positive tiles.