diff --git a/CHANGELOG.md b/CHANGELOG.md index 62575ec0..199c4df7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ # Changelog +## Sprint 128 Stronger building model runtime benchmark (2026-07-07) + +- Added `keremberke/yolov8s-building-segmentation` as an explicit Tower runtime model asset at `/mnt/user/appdata/geointel/models/yolov8s-building-segmentation.pt`; the file is not committed to Git. +- Verified the live model catalog exposes `yolov8s-building-segmentation-pt` with `will_download_models=false` and SHA256 `a27af31654c6a4edbdc85581c33d93c13986b5919de7de410f8d85d801b3bb34`. +- YOLO preflight loaded the model locally with `model_load_ok=true` and no automatic download. +- Ran a 36-run Geel/Mol/Turnhout matrix comparing `yolov8n-building-segmentation-pt` and `yolov8s-building-segmentation-pt` across tile sizes `512`/`640` and thresholds `0.50`/`0.25`/`0.15`. +- Best overall score was Mol with `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.15`: 55 detections, 9 matches, 46 false positives, 365 false negatives, precision `0.16363636363636364`, recall `0.02406417112299465`, F1 `0.04195804195804196`. +- Conclusion: `yolov8s` is cleaner than `yolov8n` on some samples, but still misses most GRB buildings; it is not a sufficient V1 default. + ## Sprint 127 Multi-sample detection quality calibration tooling (2026-07-07) - Added `scripts/prepare_operator_real_data_samples.py` to prepare documented Geel, Mol and Turnhout orthophoto/GRB GBG building sample pairs as explicit runtime artifacts. diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index d985e5a8..2846b91c 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -1,3 +1,47 @@ +## Sprint 128 Stronger building model runtime benchmark (2026-07-07) + +Changed: +- Added `keremberke/yolov8s-building-segmentation` as an explicit runtime model asset on Tower: + - path: `/mnt/user/appdata/geointel/models/yolov8s-building-segmentation.pt` + - source: `https://huggingface.co/keremberke/yolov8s-building-segmentation/resolve/main/best.pt` + - SHA256: `a27af31654c6a4edbdc85581c33d93c13986b5919de7de410f8d85d801b3bb34` +- Did not commit model weights to Git and did not add app-side model download behavior. +- Ran a stronger-candidate multi-sample matrix comparing: + - `yolov8n-building-segmentation-pt` + - `yolov8s-building-segmentation-pt` + - samples: Geel, Mol, Turnhout + - tile sizes: `512`, `640` + - overlap: `64` + - thresholds: `0.50`, `0.25`, `0.15` + +Tested: +- Hugging Face API metadata check showed `keremberke/yolov8s-building-segmentation` is public, uses the Ultralytics library, exposes `best.pt`, and reports validation mAP@0.5 values for the `keremberke/satellite-building-segmentation` dataset. +- Live model asset catalog reported `yolov8s-building-segmentation-pt`, `size_bytes=23814274`, `will_download_models=false`, SHA256 `a27af31654c6a4edbdc85581c33d93c13986b5919de7de410f8d85d801b3bb34`. +- Live preflight passed with `model_asset_id=yolov8s-building-segmentation-pt`, `check_model_load=true`, `model_file_exists=true`, `model_load_ok=true`, `will_download_models=false` and `will_run_inference=false`. +- Tower 36-run matrix passed: + - command: `OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8s-building-segmentation-pt" QUALITY_TILE_SIZES="512 640" QUALITY_TILE_OVERLAPS="64" QUALITY_THRESHOLDS="0.50 0.25 0.15" bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202` + - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/multi_sample_quality_summary.json` + - best overall score: Mol, `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.15`, 55 detections, 9 matches, 46 false positives, 365 false negatives, precision `0.16363636363636364`, recall `0.02406417112299465`, F1 `0.04195804195804196`. + - best overall recall: Mol, `yolov8s-building-segmentation-pt`, tile `512`, threshold `0.15`, 77 detections, 9 matches, 68 false positives, 365 false negatives, precision `0.11688311688311688`, recall `0.02406417112299465`, F1 `0.03991130820399113`. + - best overall precision: Turnhout, `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.25`, 20 detections, 8 matches, 12 false positives, 765 false negatives, precision `0.4`, recall `0.01034928848641656`, F1 `0.0201765447667087`. + - best Geel score: `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.15`, F1 `0.0187207488299532`. + - best Mol score: `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.15`, F1 `0.04195804195804196`. + - best Turnhout score remained `yolov8n-building-segmentation-pt`, tile `512`, threshold `0.15`, F1 `0.03934426229508197`. +- Best-score evidence export passed for every sample in the new matrix: + - Geel evidence: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/geel/calibration_evidence_review.html`, 641 features, 611 false negatives, 18 false positives, 6 matched detections and 6 matched references. + - Mol evidence: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/mol/calibration_evidence_review.html`, 429 features, 365 false negatives, 46 false positives, 9 matched detections and 9 matched references. + - Turnhout evidence: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/turnhout/calibration_evidence_review.html`, 915 features, 755 false negatives, 124 false positives, 18 matched detections and 18 matched references. + +Open: +- None for adding and benchmarking the `yolov8s` candidate. + +Limitations: +- `yolov8s` improves precision and F1 on Geel/Mol but still misses most reference buildings. Best recall is about 2.4%, which is not sufficient for a usable V1 building extraction default. +- The model is a runtime artifact and remains outside Git. + +Next recommended pass: +- Source or train a materially stronger aerial-building model. The GeoIntel pipeline is now good enough to benchmark candidates quickly, but the current public YOLO building-segmentation candidates are still too weak for the target Flemish orthophoto/GRB workflow. + ## Sprint 127 Multi-sample detection quality calibration tooling (2026-07-07) Changed: diff --git a/docs/TODO.md b/docs/TODO.md index aa8cbd17..89c285e6 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -100,7 +100,9 @@ This file now starts with the current implementation status. Older preparation/b - [x] Add real-data detection quality matrix tooling for model/tile/threshold comparison. - [x] Add reproducible Geel/Mol/Turnhout operator sample preparation and multi-sample quality matrix tooling. - [x] Run first Geel/Mol/Turnhout persisted detection quality baseline. +- [x] Add and benchmark a stronger `yolov8s` building-segmentation runtime model candidate. - [ ] Calibrate confidence, IoU and model selection against persisted Geel/Mol/Turnhout detections and any additional local orthophoto/reference samples. +- [ ] Find or train a materially stronger aerial/Kempen building model candidate; current `yolov8s` recall remains too low for a V1 default. ## Sprint 8 status