Document remote-sensing YOLO candidate benchmark

2026-07-07 23:13:36 +02:00
parent 9bd6752128
commit 4d2ba1ea5c
4 changed files with 75 additions and 0 deletions
@@ -1,3 +1,45 @@
+## Sprint 134 External remote-sensing YOLO candidate benchmark (2026-07-07)
+
+Changed:
+- Added no production code and no repository-stored model weights.
+- Evaluated the Hugging Face `agademer/yolo-remote-sensing-photovoltaic` model card as a candidate because it explicitly targets remote-sensing imagery and includes `building` as a class.
+- Downloaded the YOLOv8l detection checkpoint manually as an operator/runtime asset:
+  - source model: `agademer/yolo-remote-sensing-photovoltaic`
+  - source file: `yolo-remote-sensing-photovoltaic-v8l-solar-farms-and-cities-v20260331-detect-1000_epochs.pt`
+  - runtime file: `/app/models/yolo-remote-sensing-photovoltaic-v8l-detect-1000.pt`
+  - catalog asset: `yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt`
+  - SHA256: `242ff4ab889569278f0eb9fcd22eb2c4bf2a52e48d05d89cc7cfa7941165d203`
+- Updated `scripts/README.md`, `docs/TODO.md`, `docs/CODEX_EXECUTION_LOG.md` and `CHANGELOG.md` with benchmark evidence and the decision not to promote this model.
+
+Tested:
+- Live API preflight passed for `yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt` with `status=ready`, `model_load_ok=true`, `manifest_valid=true`, `tile_paths_exist=true`, `will_download_models=false` and `will_run_inference=false`.
+- Live 45-run multi-sample QA matrix completed:
+  - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/remote-sensing-v8l1000-live/multi_sample_quality_summary.json`
+  - command compared `yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt`, `geointel-building-yolov8n-expanded160e50-pt` and `geointel-building-yolov8n-hardneg160r8e40-pt` over Geel, Mol, Turnhout, Retie and Kasterlee-bos with tile `640`, overlap `64`, thresholds `0.25`/`0.15`/`0.05`.
+  - dense-sample results for the external candidate:
+    - Geel: best F1 `0.0`
+    - Mol: best F1 `0.010582010582010581`
+    - Turnhout: best F1 `0.019070321811680575`
+    - Retie: best F1 `0.0`
+  - Kasterlee-bos sparse result: best F1 `0.25` with 1 detection, 1 match and 0 false positives at thresholds `0.25`/`0.15`.
+  - dense-sample winner remained `geointel-building-yolov8n-expanded160e50-pt`.
+- Live 27-run hard-negative matrix completed:
+  - output: `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/remote-sensing-v8l1000-live/hard_negative_matrix_summary.json`
+  - Postel-bos: external candidate produced 0/1/3 detections at thresholds `0.25`/`0.15`/`0.05`.
+  - Lommel-heide: external candidate produced 0/0/0 detections.
+  - Kasterlee-bos: external candidate produced 1/2/5 detections, cleaner than expanded160e50 and hardneg160r8e40 on that sparse AOI.
+
+Open:
+- None for this benchmark pass.
+
+Limitations:
+- The external model is very conservative on the current Kempen orthophoto/GRB benchmark and misses most dense buildings.
+- It may be useful as evidence for a high-precision/sparse-review mode, but it is not a viable V1 default extraction model.
+- The runtime `.pt` remains outside Git and must be managed as an operator asset.
+
+Next recommended pass:
+- Train a higher-capacity local detector from a stronger base model using the existing operator tile dataset, then gate it with both dense QA and hard-negative matrices before any model default changes.
+
 ## Sprint 133 Hard-negative-balanced YOLO candidate (2026-07-07)

 Changed: