diff --git a/CHANGELOG.md b/CHANGELOG.md index 1056d51b..104a9976 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ # Changelog +## Sprint 134 External remote-sensing YOLO candidate benchmark (2026-07-07) + +- Evaluated the Hugging Face `agademer/yolo-remote-sensing-photovoltaic` YOLOv8l detection checkpoint as an explicit operator-provided runtime model asset. +- Downloaded `yolo-remote-sensing-photovoltaic-v8l-solar-farms-and-cities-v20260331-detect-1000_epochs.pt` to the Tower runtime as `/app/models/yolo-remote-sensing-photovoltaic-v8l-detect-1000.pt`; the model file is not committed to Git. +- The model catalog exposes it as `yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt` with SHA256 `242ff4ab889569278f0eb9fcd22eb2c4bf2a52e48d05d89cc7cfa7941165d203`. +- Live YOLO preflight loaded the model successfully with `status=ready`, `model_load_ok=true`, `manifest_valid=true`, `tile_paths_exist=true`, `will_download_models=false` and `will_run_inference=false`. +- Live 45-run dense QA matrix compared the external YOLOv8l candidate with `geointel-building-yolov8n-expanded160e50-pt` and `geointel-building-yolov8n-hardneg160r8e40-pt` across Geel, Mol, Turnhout, Retie and Kasterlee-bos. +- Result: the external candidate was very conservative and missed most dense GRB buildings. It scored F1 `0.0` on Geel, `0.010582010582010581` on Mol, `0.019070321811680575` on Turnhout and `0.0` on Retie, while expanded160e50 remained the dense-AOI winner. +- Live 27-run background matrix showed the external candidate was cleaner on Kasterlee-bos than local YOLOv8n candidates, with 1/2/5 detections at thresholds `0.25`/`0.15`/`0.05`, but it leaked 0/1/3 detections on Postel-bos and was therefore not uniformly cleaner than hardneg160r8e40. +- Decision: keep the model as runtime evidence only. It should not become the V1 default because recall is too low for operational extraction. The next pass should train a higher-capacity local model, starting from a stronger base and using the existing dense plus hard-negative benchmark gates. +- No API contract change, provider fetching, fake detections, model auto-provisioning, repository-stored weights or app-side model training behavior was introduced. + ## Sprint 133 Hard-negative-balanced YOLO candidate (2026-07-07) - Added `--background-negative-repeat` / `OPERATOR_YOLO_BACKGROUND_NEGATIVE_REPEAT` support to `scripts/export_operator_yolo_tile_dataset.py` so train-split background-candidate negative tiles can be repeated deterministically without duplicating validation tiles. diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index 01ef9fb7..c7c77608 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -1,3 +1,45 @@ +## Sprint 134 External remote-sensing YOLO candidate benchmark (2026-07-07) + +Changed: +- Added no production code and no repository-stored model weights. +- Evaluated the Hugging Face `agademer/yolo-remote-sensing-photovoltaic` model card as a candidate because it explicitly targets remote-sensing imagery and includes `building` as a class. +- Downloaded the YOLOv8l detection checkpoint manually as an operator/runtime asset: + - source model: `agademer/yolo-remote-sensing-photovoltaic` + - source file: `yolo-remote-sensing-photovoltaic-v8l-solar-farms-and-cities-v20260331-detect-1000_epochs.pt` + - runtime file: `/app/models/yolo-remote-sensing-photovoltaic-v8l-detect-1000.pt` + - catalog asset: `yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt` + - SHA256: `242ff4ab889569278f0eb9fcd22eb2c4bf2a52e48d05d89cc7cfa7941165d203` +- Updated `scripts/README.md`, `docs/TODO.md`, `docs/CODEX_EXECUTION_LOG.md` and `CHANGELOG.md` with benchmark evidence and the decision not to promote this model. + +Tested: +- Live API preflight passed for `yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt` with `status=ready`, `model_load_ok=true`, `manifest_valid=true`, `tile_paths_exist=true`, `will_download_models=false` and `will_run_inference=false`. +- Live 45-run multi-sample QA matrix completed: + - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/remote-sensing-v8l1000-live/multi_sample_quality_summary.json` + - command compared `yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt`, `geointel-building-yolov8n-expanded160e50-pt` and `geointel-building-yolov8n-hardneg160r8e40-pt` over Geel, Mol, Turnhout, Retie and Kasterlee-bos with tile `640`, overlap `64`, thresholds `0.25`/`0.15`/`0.05`. + - dense-sample results for the external candidate: + - Geel: best F1 `0.0` + - Mol: best F1 `0.010582010582010581` + - Turnhout: best F1 `0.019070321811680575` + - Retie: best F1 `0.0` + - Kasterlee-bos sparse result: best F1 `0.25` with 1 detection, 1 match and 0 false positives at thresholds `0.25`/`0.15`. + - dense-sample winner remained `geointel-building-yolov8n-expanded160e50-pt`. +- Live 27-run hard-negative matrix completed: + - output: `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/remote-sensing-v8l1000-live/hard_negative_matrix_summary.json` + - Postel-bos: external candidate produced 0/1/3 detections at thresholds `0.25`/`0.15`/`0.05`. + - Lommel-heide: external candidate produced 0/0/0 detections. + - Kasterlee-bos: external candidate produced 1/2/5 detections, cleaner than expanded160e50 and hardneg160r8e40 on that sparse AOI. + +Open: +- None for this benchmark pass. + +Limitations: +- The external model is very conservative on the current Kempen orthophoto/GRB benchmark and misses most dense buildings. +- It may be useful as evidence for a high-precision/sparse-review mode, but it is not a viable V1 default extraction model. +- The runtime `.pt` remains outside Git and must be managed as an operator asset. + +Next recommended pass: +- Train a higher-capacity local detector from a stronger base model using the existing operator tile dataset, then gate it with both dense QA and hard-negative matrices before any model default changes. + ## Sprint 133 Hard-negative-balanced YOLO candidate (2026-07-07) Changed: diff --git a/docs/TODO.md b/docs/TODO.md index bbc63f9d..3496ced9 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -107,7 +107,9 @@ This file now starts with the current implementation status. Older preparation/b - [x] Add negative/background AOIs to the operator sample corpus and train an expanded local tile-level YOLO candidate. - [x] Add a hard-negative model-quality pass with sparse/background AOIs and explicit false-positive scoring. - [x] Train a hard-negative-balanced YOLO candidate and rerun dense QA plus background false-positive matrices. +- [x] Benchmark an external remote-sensing YOLOv8l building candidate as an explicit local model asset. - [ ] Find or train a materially stronger aerial/Kempen building model candidate; `geointel-building-yolov8n-expanded160e50-pt` is the best current dense-AOI candidate but still too weak and too noisy for a V1 default. +- [ ] Train a higher-capacity local aerial-building detector with stronger positive recall while preserving the hard-negative false-positive gate. ## Sprint 8 status diff --git a/scripts/README.md b/scripts/README.md index f3182d19..abd81a6a 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -416,6 +416,25 @@ Lommel-heide across all tested thresholds. It also regressed dense-AOI F1 against `geointel-building-yolov8n-expanded160e50-pt`, so it is useful model quality evidence but not a V1 default. +An external remote-sensing YOLOv8l candidate was also benchmarked as an +operator-provided local model asset: + +```bash +mkdir -p models +curl -L --fail \ + -o models/yolo-remote-sensing-photovoltaic-v8l-detect-1000.pt \ + https://huggingface.co/agademer/yolo-remote-sensing-photovoltaic/resolve/main/yolo-remote-sensing-photovoltaic-v8l-solar-farms-and-cities-v20260331-detect-1000_epochs.pt +``` + +GeoIntel exposed the file as +`yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt` with SHA256 +`242ff4ab889569278f0eb9fcd22eb2c4bf2a52e48d05d89cc7cfa7941165d203`, and +YOLO preflight loaded it without downloads. On the live dense matrix it missed +most buildings and scored far below `geointel-building-yolov8n-expanded160e50-pt` +on Geel, Mol, Turnhout and Retie. On Kasterlee-bos it was clean and precise, +but that sparse-AOI behavior is not enough for V1 extraction. Keep it as +benchmark evidence only, not as a default model. + Export calibration QA evidence for visual review: ```bash