Document operator YOLO training benchmark

CHANGELOG.md

@@ -15,6 +15,10 @@
 - Added readiness coverage for the exporter Python compile check and training wrapper shell syntax.
 - Added regression coverage in `backend/tests/test_sprint129_operator_yolo_training_dataset.py`.
 - Updated operator documentation for dataset export, training smoke usage and the requirement to benchmark any trained model through the existing real-data Detection + QA matrix before treating it as useful.
+- Live Tower export produced a YOLO dataset with 3 operator samples and 1427 labels; a clean 8-epoch CPU training smoke produced `/app/models/geointel-building-yolov8n-operator8.pt`.
+- Live preflight loaded `geointel-building-yolov8n-operator8-pt` successfully with `will_download_models=false` and `will_run_inference=false`.
+- Live multi-sample QA matrix showed the 8-epoch operator model is not useful yet: it produced zero detections at thresholds `0.15`-`0.50`, and the low-threshold `0.01` run produced mostly false positives with best F1 `0.003798670465337132`.
+- `yolov8s-building-segmentation-pt` remains the best tested model, with best overall F1 `0.04195804195804196` on Mol at tile `640`, threshold `0.15`; still not sufficient for V1 default extraction.
 - No Training Studio UI, API contract change, provider fetching, model auto-provisioning or app-side model training behavior was introduced.
 
 ## Sprint 128 Stronger building model runtime benchmark (2026-07-07)

docs/CODEX_EXECUTION_LOG.md

@@ -24,15 +24,34 @@ Tested:
 - Live Tower export passed: `/app/storage/operator-data/yolo-building-dataset` contains 3 images and 1427 labels from the current Geel/Mol/Turnhout samples.
 - Live Tower training smoke passed with the existing `/app/models/yolov8n.pt` base model and wrote `/app/models/geointel-building-yolov8n-smoke.pt`.
 - The first live smoke showed Ultralytics fetching an auxiliary plot font. The wrapper now sets `plots=False` and seeds `Arial.ttf` from an existing local system font before importing Ultralytics so the operator smoke path does not invoke plot/font network behavior.
+- Full local readiness passed after the final wrapper hardening: `bash scripts/run_readiness_check.sh` ran 396 backend tests, frontend typecheck/build, Alembic head and smoke syntax checks.
+- Pushed commits through `3936768` to Gitea and reset Tower `/mnt/user/appdata/geointel` to the same commit.
+- Live Tower no-font smoke passed after deleting `/app/storage/ultralytics/Ultralytics/Arial.ttf`; the wrapper seeded the font locally and the 1-epoch run completed without the previous font fetch line.
+- Live Tower final 8-epoch training smoke passed:
+  - output model: `/app/models/geointel-building-yolov8n-operator8.pt`
+  - catalog asset: `geointel-building-yolov8n-operator8-pt`
+  - SHA256: `071d64c89a8fd28f915f7a5a553e2d164942721292693a1612c725284c6e2e1e`
+  - training summary: `/app/storage/training/operator-yolo/geointel-building-yolov8n-operator8/training_summary.json`.
+- Live API preflight passed for `geointel-building-yolov8n-operator8-pt` with `status=ready`, `model_load_ok=true`, `manifest_valid=true`, `tile_paths_exist=true`, `will_download_models=false` and `will_run_inference=false`.
+- Live 36-run multi-sample QA matrix completed:
+  - command compared `geointel-building-yolov8n-operator8-pt` with `yolov8s-building-segmentation-pt` over Geel, Mol and Turnhout, tile sizes `512`/`640`, thresholds `0.50`/`0.25`/`0.15`.
+  - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T033048Z/multi_sample_quality_summary.json`.
+  - `geointel-building-yolov8n-operator8-pt` produced zero detections for every sample/tile combination at thresholds `0.15` through `0.50`.
+  - best overall remained `yolov8s-building-segmentation-pt` on Mol, tile `640`, threshold `0.15`: 55 detections, 9 matches, 46 false positives, 365 false negatives, precision `0.16363636363636364`, recall `0.02406417112299465`, F1 `0.04195804195804196`.
+- Live low-threshold operator8 matrix completed:
+  - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T033230Z/multi_sample_quality_summary.json`.
+  - at threshold `0.05`, operator8 still produced zero detections on all samples.
+  - at threshold `0.01`, operator8 produced many false positives and almost no matches; best case was Turnhout tile `512` with 280 detections, 2 matches, 278 false positives, precision `0.007142857142857143`, recall `0.00258732212160414`, F1 `0.003798670465337132`.
 
 Open:
-- Run the exporter and training smoke inside the AI-enabled Tower runtime, then benchmark the trained artifact through the existing multi-sample Detection + QA matrix.
+- None for the operator training/export tooling itself.
 
 Limitations:
 - This is operator tooling only. It does not add Training Studio, browser training controls, provider fetching, fake detections, model auto-provisioning or API contract changes.
+- The 3-sample/8-epoch operator model is not production-useful and should not be activated as the default building detector.
 
 Next recommended pass:
-- Generate the local YOLO dataset from the current Geel/Mol/Turnhout samples, train a small local model smoke from `yolov8n.pt`, and compare it against the current `yolov8s-building-segmentation-pt` benchmark.
+- Improve the training dataset before more model work: generate tile-level YOLO labels with enough positive/negative tiles, add more AOIs, then train a longer local model and rerun the same persisted QA matrix.
 
 ## Sprint 128 Stronger building model runtime benchmark (2026-07-07)
 

docs/TODO.md

@@ -405,4 +405,5 @@ This file now starts with the current implementation status. Older preparation/b
 - [x] Add safe local YOLO model env configuration helper for Unraid/Tower runtime activation.
 - [x] Harden configured YOLO inference for single-band raster tiles and wrapped runtime errors.
 - [x] Add operator-only YOLO dataset export and local training-smoke wrapper for real sample calibration.
-- [ ] Train/evaluate a local GeoIntel building detector from the operator samples and only activate it after QA/QC matrix improvement.
+- [x] Train/evaluate a small local GeoIntel building-detector smoke from the current operator samples and reject it because QA/QC did not improve.
+- [ ] Build a larger tile-level training dataset with more AOIs, positive/negative tiles and validation splits before the next local model attempt.