Document stronger building model benchmark

2026-07-07 05:03:44 +02:00
parent 3c400d6167
commit 306fcd1b24
3 changed files with 55 additions and 0 deletions
@@ -1,3 +1,47 @@
+## Sprint 128 Stronger building model runtime benchmark (2026-07-07)
+
+Changed:
+- Added `keremberke/yolov8s-building-segmentation` as an explicit runtime model asset on Tower:
+  - path: `/mnt/user/appdata/geointel/models/yolov8s-building-segmentation.pt`
+  - source: `https://huggingface.co/keremberke/yolov8s-building-segmentation/resolve/main/best.pt`
+  - SHA256: `a27af31654c6a4edbdc85581c33d93c13986b5919de7de410f8d85d801b3bb34`
+- Did not commit model weights to Git and did not add app-side model download behavior.
+- Ran a stronger-candidate multi-sample matrix comparing:
+  - `yolov8n-building-segmentation-pt`
+  - `yolov8s-building-segmentation-pt`
+  - samples: Geel, Mol, Turnhout
+  - tile sizes: `512`, `640`
+  - overlap: `64`
+  - thresholds: `0.50`, `0.25`, `0.15`
+
+Tested:
+- Hugging Face API metadata check showed `keremberke/yolov8s-building-segmentation` is public, uses the Ultralytics library, exposes `best.pt`, and reports validation mAP@0.5 values for the `keremberke/satellite-building-segmentation` dataset.
+- Live model asset catalog reported `yolov8s-building-segmentation-pt`, `size_bytes=23814274`, `will_download_models=false`, SHA256 `a27af31654c6a4edbdc85581c33d93c13986b5919de7de410f8d85d801b3bb34`.
+- Live preflight passed with `model_asset_id=yolov8s-building-segmentation-pt`, `check_model_load=true`, `model_file_exists=true`, `model_load_ok=true`, `will_download_models=false` and `will_run_inference=false`.
+- Tower 36-run matrix passed:
+  - command: `OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8s-building-segmentation-pt" QUALITY_TILE_SIZES="512 640" QUALITY_TILE_OVERLAPS="64" QUALITY_THRESHOLDS="0.50 0.25 0.15" bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202`
+  - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/multi_sample_quality_summary.json`
+  - best overall score: Mol, `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.15`, 55 detections, 9 matches, 46 false positives, 365 false negatives, precision `0.16363636363636364`, recall `0.02406417112299465`, F1 `0.04195804195804196`.
+  - best overall recall: Mol, `yolov8s-building-segmentation-pt`, tile `512`, threshold `0.15`, 77 detections, 9 matches, 68 false positives, 365 false negatives, precision `0.11688311688311688`, recall `0.02406417112299465`, F1 `0.03991130820399113`.
+  - best overall precision: Turnhout, `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.25`, 20 detections, 8 matches, 12 false positives, 765 false negatives, precision `0.4`, recall `0.01034928848641656`, F1 `0.0201765447667087`.
+  - best Geel score: `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.15`, F1 `0.0187207488299532`.
+  - best Mol score: `yolov8s-building-segmentation-pt`, tile `640`, threshold `0.15`, F1 `0.04195804195804196`.
+  - best Turnhout score remained `yolov8n-building-segmentation-pt`, tile `512`, threshold `0.15`, F1 `0.03934426229508197`.
+- Best-score evidence export passed for every sample in the new matrix:
+  - Geel evidence: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/geel/calibration_evidence_review.html`, 641 features, 611 false negatives, 18 false positives, 6 matched detections and 6 matched references.
+  - Mol evidence: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/mol/calibration_evidence_review.html`, 429 features, 365 false negatives, 46 false positives, 9 matched detections and 9 matched references.
+  - Turnhout evidence: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/turnhout/calibration_evidence_review.html`, 915 features, 755 false negatives, 124 false positives, 18 matched detections and 18 matched references.
+
+Open:
+- None for adding and benchmarking the `yolov8s` candidate.
+
+Limitations:
+- `yolov8s` improves precision and F1 on Geel/Mol but still misses most reference buildings. Best recall is about 2.4%, which is not sufficient for a usable V1 building extraction default.
+- The model is a runtime artifact and remains outside Git.
+
+Next recommended pass:
+- Source or train a materially stronger aerial-building model. The GeoIntel pipeline is now good enough to benchmark candidates quickly, but the current public YOLO building-segmentation candidates are still too weak for the target Flemish orthophoto/GRB workflow.
+
 ## Sprint 127 Multi-sample detection quality calibration tooling (2026-07-07)

 Changed: