diff --git a/CHANGELOG.md b/CHANGELOG.md index 3eac9d8b..43005df8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,16 @@ # Changelog +## Sprint 153 AOI1024 clean-label YOLOv8s candidate gate (2026-07-09) + +- Audited AOI1024 label-quality variants after paged GRB reference regeneration and selected `/app/storage/operator-data/yolo-building-aoi1024-visible050-minpx8` for training because it passed the dataset audit while keeping the 512px runtime scale. +- Trained inactive local model asset `geointel-building-yolov8s-aoi1024clean512e50-pt` from the cleaned 512px tile dataset. The Tower catalog reports SHA256 `7cfadb684dd56623d2e35ebd65593211d3051908c87121bef438b231d3e47cce`, `status=available`, `active=false` and `will_download_models=false`. +- Ultralytics validation for the completed 50-epoch CPU run ended at approximately precision `0.441`, recall `0.380`, mAP50 `0.265` and mAP50-95 `0.096`. +- Ran the full seven-reference AOI1024 persisted QA matrix at tile `512`, overlap `64` and thresholds `0.25`, `0.15` and `0.05`. Mean positive F1 improved materially: `0.4723513253430784` at threshold `0.05`, `0.4753322215541376` at `0.15` and `0.31021575247724875` at `0.25`. +- Ran the full nine-sample hard-negative/background matrix. Background false-positive pressure still blocks default promotion: max detections were `137` at threshold `0.05`, `75` at `0.15` and `55` at `0.25`. +- Generated the promotion report at `artifacts/detection-model-promotion/aoi1024clean512e50-full/detection_model_promotion_report.md`; recommendation remains `none` because every threshold fails `background_false_positive_pressure`. +- No API contract, migration, provider fetching, fake detection data, model download behavior or active model default changed. + ## Sprint 152 GRB reference paging for operator samples (2026-07-09) - Fixed the operator real-data sample preparer so GRB GBG reference GeoJSON is fetched through OGC API `rel=next` pagination links instead of stopping at the first `limit=1000` page. diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index 6b698d80..d8d1c164 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -6041,3 +6041,58 @@ Open: ## Next recommended pass - Fix dense GRB reference completeness first: add provider-side paging or split dense AOIs so reference exports do not cap at 1000 features, then regenerate AOI1024 labels before another training run. + +# Sprint 153 - AOI1024 clean-label YOLOv8s candidate gate + +## What changed + +- Built and audited AOI1024 label-quality candidate exports on Tower after the paged GRB reference regeneration: + - baseline `yolo-building-aoi1024-visible025`: 144 tiles, 117 positive, 27 negative, 29,170 labels, audit `needs_attention`, median normalized box area `0.000762939453125`, small-box share `0.34744600617072335`. + - `yolo-building-aoi1024-visible050`: 144 tiles, 117 positive, 27 negative, 28,552 labels, audit `needs_attention`, small-box share `0.34463435135892406`. + - `yolo-building-aoi1024-visible025-tile640`: 45,529 labels, audit `needs_attention`, small-box share `0.4952667530584902`. + - `yolo-building-aoi1024-visible025-tile384`: 400 tiles, 311 positive, 89 negative, 46,423 labels, audit `ok`, small-box share `0.215281218361588`. + - `yolo-building-aoi1024-visible050-minpx8`: 144 tiles, 117 positive, 27 negative, 21,530 labels, audit `ok`, median normalized box area `0.0010299684375`, small-box share `0.14681839294008361`. +- Selected `/app/storage/operator-data/yolo-building-aoi1024-visible050-minpx8` for the next controlled candidate because it passes the label audit while preserving the 512px runtime scale. +- Trained Tower-local inactive model asset: + - model path: `/app/models/geointel-building-yolov8s-aoi1024clean512e50.pt` + - asset id: `geointel-building-yolov8s-aoi1024clean512e50-pt` + - SHA256: `7cfadb684dd56623d2e35ebd65593211d3051908c87121bef438b231d3e47cce` + - training summary: `/app/storage/training/operator-yolo/geointel-building-yolov8s-aoi1024clean512e50/training_summary.json` + - base model: `/app/models/yolov8s.pt` + - epochs: 50 + - image size: 512 + - batch size: 2 + - device: CPU + - final Ultralytics validation on `best.pt`: precision `0.441`, recall `0.380`, mAP50 `0.265`, mAP50-95 `0.096`. +- Verified the live model catalog reports the candidate as `status=available`, `active=false` and `will_download_models=false`. + +## What was tested + +- Tower training command: + - `docker exec -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-aoi1024-visible050-minpx8 -e YOLO_BASE_MODEL_PATH=/app/models/yolov8s.pt -e TRAIN_OUTPUT_DIR=/app/storage/training/operator-yolo -e TRAIN_RUN_NAME=geointel-building-yolov8s-aoi1024clean512e50 -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-yolov8s-aoi1024clean512e50.pt -e TRAIN_EPOCHS=50 -e TRAIN_IMGSZ=512 -e TRAIN_BATCH=2 -e TRAIN_WORKERS=0 -e TRAIN_DEVICE=cpu geointel bash /app/scripts/train_operator_yolo_detector.sh` +- Full seven-reference AOI1024 persisted positive QA matrix: + - command used `scripts/run_multi_sample_detection_quality_matrix.sh` with manifest `/mnt/user/appdata/geointel/storage/operator-data/operator-samples-1024/operator_samples_manifest.json`, samples `geel mol turnhout herentals balen retie westerlo`, model `geointel-building-yolov8s-aoi1024clean512e50-pt`, tile `512`, overlap `64`, thresholds `0.25 0.15 0.05` and IoU threshold `0.25`. + - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024clean512e50-full/multi_sample_quality_summary.json` + - threshold `0.05`: 7 positive samples, mean F1 `0.4723513253430784`, mean precision `0.44861169484563357`, mean recall `0.5039340859731523`, min F1 `0.4442082890541977`. + - threshold `0.15`: 7 positive samples, mean F1 `0.4753322215541376`, mean precision `0.6673270868402211`, mean recall `0.37207511021815726`, min F1 `0.4055555555555556`. + - threshold `0.25`: 7 positive samples, mean F1 `0.31021575247724875`, mean precision `0.8163316419273472`, mean recall `0.19608909809008596`, min F1 `0.16430903155603915`. +- Full nine-sample hard-negative/background matrix: + - command used `scripts/run_operator_hard_negative_detection_matrix.sh` with the same AOI1024 manifest, model `geointel-building-yolov8s-aoi1024clean512e50-pt`, tile `512`, overlap `64` and thresholds `0.25 0.15 0.05`. + - output: `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/aoi1024clean512e50-full/hard_negative_matrix_summary.json` + - threshold `0.05`: total background detections `488`, max sample detections `137`. + - threshold `0.15`: total background detections `269`, max sample detections `75`. + - threshold `0.25`: total background detections `205`, max sample detections `55`. +- Promotion report: + - command used `scripts/build_detection_model_promotion_report.py` with `--min-positive-samples 7`, `--min-background-samples 9`, `--min-mean-f1 0.25` and `--max-background-detections-per-sample 0`. + - output: `/mnt/user/appdata/geointel/artifacts/detection-model-promotion/aoi1024clean512e50-full/detection_model_promotion_report.md` + - recommendation: `none`. + +## Known limitations + +- The clean-label model is materially stronger on positive reference AOIs than the previous AOI1024-visible025 candidate, but it is still not safe as a V1 default because hard-negative/background false positives remain high in mixed wooded AOIs. +- The likely bottleneck has shifted from label-file integrity to discriminating buildings from visually similar background structures and wooded-edge artifacts. +- The model remains inactive. No default model, API contract, migration, provider fetching or download behavior changed. + +## Next recommended pass + +- Build a background-aware training pass instead of another positive-only clean-label pass: combine the `visible050-minpx8` positives with stronger unique hard-negative/background retention, then re-run the same seven-positive/nine-background promotion gate before any activation. diff --git a/docs/TODO.md b/docs/TODO.md index 6e501ba9..28d77911 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -116,10 +116,12 @@ This file now starts with the current implementation status. Older preparation/b - [x] Rerun live dense-AOI calibration after redeploy with `YOLO_DUPLICATE_IOU_THRESHOLD=0.5`; Westerlo 0.25 improved to F1 `0.2537313432835821` and Turnhout 0.25 improved to F1 `0.14114114114114112`, but the candidate remains rejected. - [x] Add `OPERATOR_YOLO_MIN_LABEL_VISIBLE_RATIO` so the next overlapping-tile dataset can drop tiny clipped edge-fragment labels. - [x] Add operator-only larger-AOI sample prep flags so the next training dataset is not limited to one 512x512 tile per documented sample. +- [x] Regenerate Tower AOI1024 operator samples with paged GRB references and verify no dense reference sample is capped at 1000 features. +- [x] Export and audit AOI1024 clean-label variants; select `yolo-building-aoi1024-visible050-minpx8` as the first audit-passing 512px training candidate. +- [x] Train and gate `geointel-building-yolov8s-aoi1024clean512e50-pt` through seven positive AOIs and nine hard-negative/background AOIs. - [ ] Find or train a materially stronger aerial/Kempen building model candidate; `geointel-building-yolov8n-expanded160e50-pt` is the best current dense-AOI candidate but still too weak and too noisy for a V1 default. -- [ ] Train a higher-capacity local aerial-building detector with stronger positive recall while preserving the hard-negative false-positive gate. -- [ ] Prepare `/app/storage/operator-data/operator-samples-1024` on Tower, then export and audit `yolo-building-aoi1024-visible025` before the next default-model training attempt. -- [ ] Build the next candidate gate around better positive AOI coverage, label strategy and hard-negative retention. +- [ ] Train a background-aware local aerial-building detector with stronger positive recall while reducing hard-negative false-positive pressure. +- [ ] Build the next candidate gate around cleaner positive labels plus stronger unique hard-negative retention. ## Sprint 8 status