diff --git a/CHANGELOG.md b/CHANGELOG.md index 43005df8..ff4a44d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ # Changelog +## Sprint 154 Background-aware AOI1024 YOLOv8s candidate gate (2026-07-09) + +- Exported and audited background-aware AOI1024 training dataset `/app/storage/operator-data/yolo-building-aoi1024-bgaware512r3`; the audit passed with 162 tiles, 117 positive tiles, 45 negative tiles, 21,530 labels and no warnings. +- Trained inactive local model asset `geointel-building-yolov8s-aoi1024bg512r3e50-pt` from the background-aware dataset. The Tower catalog reports SHA256 `e0980572aac90e7efc514608eb16d7de5bfbf27a4bbec04e7bc1bc8c02f9601f`, `status=available`, `active=false` and `will_download_models=false`. +- Ultralytics validation for the completed 50-epoch CPU run ended at approximately precision `0.440`, recall `0.362`, mAP50 `0.251` and mAP50-95 `0.0878`. +- Ran the seven-reference AOI1024 persisted QA matrix at tile `512`, overlap `64` and thresholds `0.25`, `0.15` and `0.05`. Mean positive F1 improved to `0.4908049127242224` at threshold `0.05`, `0.5074022485589402` at `0.15` and `0.44378879337957716` at `0.25`. +- Ran an additional conservative-threshold positive matrix at thresholds `0.35`, `0.45` and `0.60`. Threshold `0.35` produced mean F1 `0.32086574003576274`, mean precision `0.840006` and mean recall `0.202135`. +- Ran full hard-negative/background matrices. The candidate still fails automatic default promotion because mixed background-candidate AOIs keep false-positive pressure: max detections were `184` at threshold `0.05`, `103` at `0.15`, `75` at `0.25`, `55` at `0.35`, `38` at `0.45` and `18` at `0.60`. +- Generated promotion reports at `artifacts/detection-model-promotion/aoi1024bg512r3e50-full/detection_model_promotion_report.md` and `artifacts/detection-model-promotion/aoi1024bg512r3e50-high-threshold/detection_model_promotion_report.md`; recommendation remains `none`. +- No API contract, migration, provider fetching, fake detection data, model download behavior or active model default changed. + ## Sprint 153 AOI1024 clean-label YOLOv8s candidate gate (2026-07-09) - Audited AOI1024 label-quality variants after paged GRB reference regeneration and selected `/app/storage/operator-data/yolo-building-aoi1024-visible050-minpx8` for training because it passed the dataset audit while keeping the 512px runtime scale. diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index d8d1c164..b7ffdeb6 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -6096,3 +6096,75 @@ Open: ## Next recommended pass - Build a background-aware training pass instead of another positive-only clean-label pass: combine the `visible050-minpx8` positives with stronger unique hard-negative/background retention, then re-run the same seven-positive/nine-background promotion gate before any activation. + +# Sprint 154 - Background-aware AOI1024 YOLOv8s candidate gate + +## What changed + +- Built a background-aware AOI1024 tile dataset on Tower from the clean positive labels plus explicit background retention: + - dataset path: `/app/storage/operator-data/yolo-building-aoi1024-bgaware512r3` + - audit path: `/mnt/user/appdata/geointel/storage/operator-data/yolo-building-aoi1024-bgaware512r3-audit/operator_yolo_dataset_quality_audit.json` + - tile count: `162` + - positive tiles: `117` + - negative tiles: `45` + - label count: `21530` + - background negative repeat: `3` + - audit status: `ok` + - warnings: `[]` +- Trained Tower-local inactive model asset: + - model path: `/app/models/geointel-building-yolov8s-aoi1024bg512r3e50.pt` + - asset id: `geointel-building-yolov8s-aoi1024bg512r3e50-pt` + - SHA256: `e0980572aac90e7efc514608eb16d7de5bfbf27a4bbec04e7bc1bc8c02f9601f` + - training summary: `/app/storage/training/operator-yolo/geointel-building-yolov8s-aoi1024bg512r3e50/training_summary.json` + - base model: `/app/models/yolov8s.pt` + - epochs: `50` + - image size: `512` + - batch size: `2` + - device: CPU + - final Ultralytics validation on `best.pt`: precision `0.440`, recall `0.362`, mAP50 `0.251`, mAP50-95 `0.0878`. +- Verified the live model catalog reports the candidate as `status=available`, `active=false` and `will_download_models=false`. +- No API contract, migration, provider fetching, fake detection data, model download behavior or active model default changed. + +## What was tested + +- Tower training command: + - `docker exec -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-aoi1024-bgaware512r3 -e YOLO_BASE_MODEL_PATH=/app/models/yolov8s.pt -e TRAIN_OUTPUT_DIR=/app/storage/training/operator-yolo -e TRAIN_RUN_NAME=geointel-building-yolov8s-aoi1024bg512r3e50 -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-yolov8s-aoi1024bg512r3e50.pt -e TRAIN_EPOCHS=50 -e TRAIN_IMGSZ=512 -e TRAIN_BATCH=2 -e TRAIN_WORKERS=0 -e TRAIN_DEVICE=cpu geointel bash /app/scripts/train_operator_yolo_detector.sh` +- Full seven-reference AOI1024 persisted positive QA matrix: + - command used `scripts/run_multi_sample_detection_quality_matrix.sh` with manifest `/mnt/user/appdata/geointel/storage/operator-data/operator-samples-1024/operator_samples_manifest.json`, samples `geel mol turnhout herentals balen retie westerlo`, model `geointel-building-yolov8s-aoi1024bg512r3e50-pt`, tile `512`, overlap `64`, thresholds `0.25 0.15 0.05` and IoU threshold `0.25`. + - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-full/multi_sample_quality_summary.json` + - threshold `0.05`: 7 positive samples, mean F1 `0.4908049127242224`, mean precision `0.468604`, mean recall `0.520016`, total detections `15277`. + - threshold `0.15`: 7 positive samples, mean F1 `0.5074022485589402`, mean precision `0.636639`, mean recall `0.424258`, total detections `9186`. + - threshold `0.25`: 7 positive samples, mean F1 `0.44378879337957716`, mean precision `0.762683`, mean recall `0.316314`, total detections `5584`. +- Conservative seven-reference AOI1024 persisted positive QA matrix: + - command used the same manifest/model/tile/overlap with thresholds `0.35 0.45 0.60`. + - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-high-threshold/multi_sample_quality_summary.json` + - threshold `0.35`: 7 positive samples, mean F1 `0.32086574003576274`, mean precision `0.840006`, mean recall `0.202135`, total detections `3106`. + - threshold `0.45`: 7 positive samples, mean F1 `0.20093433626447627`, mean precision `0.885043`, mean recall `0.116348`, total detections `1592`. + - threshold `0.60`: 7 positive samples, mean F1 `0.06640476239173489`, mean precision `0.871611`, mean recall `0.035083`, total detections `418`. +- Full nine-sample hard-negative/background matrix: + - command used `scripts/run_operator_hard_negative_detection_matrix.sh` with the AOI1024 manifest, model `geointel-building-yolov8s-aoi1024bg512r3e50-pt`, tile `512`, overlap `64` and thresholds `0.25 0.15 0.05`. + - output: `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/aoi1024bg512r3e50-full/hard_negative_matrix_summary.json` + - threshold `0.05`: total background detections `638`, max sample detections `184`. + - threshold `0.15`: total background detections `390`, max sample detections `103`. + - threshold `0.25`: total background detections `278`, max sample detections `75`. +- Conservative nine-sample hard-negative/background matrix: + - output: `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/aoi1024bg512r3e50-high-threshold/hard_negative_matrix_summary.json` + - threshold `0.35`: total background detections `198`, max sample detections `55`. + - threshold `0.45`: total background detections `148`, max sample detections `38`. + - threshold `0.60`: total background detections `76`, max sample detections `18`. +- Promotion reports: + - balanced thresholds report: `/mnt/user/appdata/geointel/artifacts/detection-model-promotion/aoi1024bg512r3e50-full/detection_model_promotion_report.md` + - conservative thresholds report: `/mnt/user/appdata/geointel/artifacts/detection-model-promotion/aoi1024bg512r3e50-high-threshold/detection_model_promotion_report.md` + - recommendation: `none`. + +## Known limitations + +- The background-aware model is the strongest positive-AOI candidate so far and is usable as an explicit review/demo candidate, especially at threshold `0.15` for balance or `0.35` for conservative high-precision review. +- It is still not safe as a V1 default because the full background-candidate gate fails `background_false_positive_pressure`. +- Some `background_candidate` AOIs contain GRB reference buildings and should be split into pure-empty negatives versus sparse-positive contextual samples before the next gate; otherwise the hard-negative score remains intentionally conservative but not perfectly diagnostic. +- The model remains inactive. Operators can select it explicitly from the local model asset catalog. + +## Next recommended pass + +- Build a V1 operator detection profile layer in the UI/docs: expose `balanced` (`threshold=0.15`) and `conservative review` (`threshold=0.35`) as explicit choices for local model assets, with clear warning that the model is not a default-approved detector. +- Clean the background corpus classification: separate pure-empty AOIs from sparse-building contextual AOIs, then retrain or recalibrate against that cleaner gate. diff --git a/docs/TODO.md b/docs/TODO.md index 28d77911..6043371d 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -119,9 +119,10 @@ This file now starts with the current implementation status. Older preparation/b - [x] Regenerate Tower AOI1024 operator samples with paged GRB references and verify no dense reference sample is capped at 1000 features. - [x] Export and audit AOI1024 clean-label variants; select `yolo-building-aoi1024-visible050-minpx8` as the first audit-passing 512px training candidate. - [x] Train and gate `geointel-building-yolov8s-aoi1024clean512e50-pt` through seven positive AOIs and nine hard-negative/background AOIs. -- [ ] Find or train a materially stronger aerial/Kempen building model candidate; `geointel-building-yolov8n-expanded160e50-pt` is the best current dense-AOI candidate but still too weak and too noisy for a V1 default. -- [ ] Train a background-aware local aerial-building detector with stronger positive recall while reducing hard-negative false-positive pressure. -- [ ] Build the next candidate gate around cleaner positive labels plus stronger unique hard-negative retention. +- [x] Train and gate background-aware `geointel-building-yolov8s-aoi1024bg512r3e50-pt`; it is the strongest positive-AOI candidate so far but remains inactive because full background-candidate false-positive pressure still blocks default promotion. +- [ ] Add explicit operator detection profiles for local model assets: balanced review around threshold `0.15` and conservative high-precision review around threshold `0.35`, both clearly marked as non-default-approved until promotion gates pass. +- [ ] Split the background corpus into pure-empty negatives and sparse-building contextual AOIs, then retrain or recalibrate against the cleaner gate. +- [ ] Promote a V1 default building detector only after it passes seven positive AOIs, clean hard-negative/background gates and persisted QA/QC evidence without fake detections or model downloads. ## Sprint 8 status