Record background-aware AOI1024 YOLO gate
This commit is contained in:
Codex
@@ -6096,3 +6096,75 @@ Open:
|
||||
## Next recommended pass
|
||||
|
||||
- Build a background-aware training pass instead of another positive-only clean-label pass: combine the `visible050-minpx8` positives with stronger unique hard-negative/background retention, then re-run the same seven-positive/nine-background promotion gate before any activation.
|
||||
|
||||
# Sprint 154 - Background-aware AOI1024 YOLOv8s candidate gate
|
||||
|
||||
## What changed
|
||||
|
||||
- Built a background-aware AOI1024 tile dataset on Tower from the clean positive labels plus explicit background retention:
|
||||
- dataset path: `/app/storage/operator-data/yolo-building-aoi1024-bgaware512r3`
|
||||
- audit path: `/mnt/user/appdata/geointel/storage/operator-data/yolo-building-aoi1024-bgaware512r3-audit/operator_yolo_dataset_quality_audit.json`
|
||||
- tile count: `162`
|
||||
- positive tiles: `117`
|
||||
- negative tiles: `45`
|
||||
- label count: `21530`
|
||||
- background negative repeat: `3`
|
||||
- audit status: `ok`
|
||||
- warnings: `[]`
|
||||
- Trained Tower-local inactive model asset:
|
||||
- model path: `/app/models/geointel-building-yolov8s-aoi1024bg512r3e50.pt`
|
||||
- asset id: `geointel-building-yolov8s-aoi1024bg512r3e50-pt`
|
||||
- SHA256: `e0980572aac90e7efc514608eb16d7de5bfbf27a4bbec04e7bc1bc8c02f9601f`
|
||||
- training summary: `/app/storage/training/operator-yolo/geointel-building-yolov8s-aoi1024bg512r3e50/training_summary.json`
|
||||
- base model: `/app/models/yolov8s.pt`
|
||||
- epochs: `50`
|
||||
- image size: `512`
|
||||
- batch size: `2`
|
||||
- device: CPU
|
||||
- final Ultralytics validation on `best.pt`: precision `0.440`, recall `0.362`, mAP50 `0.251`, mAP50-95 `0.0878`.
|
||||
- Verified the live model catalog reports the candidate as `status=available`, `active=false` and `will_download_models=false`.
|
||||
- No API contract, migration, provider fetching, fake detection data, model download behavior or active model default changed.
|
||||
|
||||
## What was tested
|
||||
|
||||
- Tower training command:
|
||||
- `docker exec -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-aoi1024-bgaware512r3 -e YOLO_BASE_MODEL_PATH=/app/models/yolov8s.pt -e TRAIN_OUTPUT_DIR=/app/storage/training/operator-yolo -e TRAIN_RUN_NAME=geointel-building-yolov8s-aoi1024bg512r3e50 -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-yolov8s-aoi1024bg512r3e50.pt -e TRAIN_EPOCHS=50 -e TRAIN_IMGSZ=512 -e TRAIN_BATCH=2 -e TRAIN_WORKERS=0 -e TRAIN_DEVICE=cpu geointel bash /app/scripts/train_operator_yolo_detector.sh`
|
||||
- Full seven-reference AOI1024 persisted positive QA matrix:
|
||||
- command used `scripts/run_multi_sample_detection_quality_matrix.sh` with manifest `/mnt/user/appdata/geointel/storage/operator-data/operator-samples-1024/operator_samples_manifest.json`, samples `geel mol turnhout herentals balen retie westerlo`, model `geointel-building-yolov8s-aoi1024bg512r3e50-pt`, tile `512`, overlap `64`, thresholds `0.25 0.15 0.05` and IoU threshold `0.25`.
|
||||
- output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-full/multi_sample_quality_summary.json`
|
||||
- threshold `0.05`: 7 positive samples, mean F1 `0.4908049127242224`, mean precision `0.468604`, mean recall `0.520016`, total detections `15277`.
|
||||
- threshold `0.15`: 7 positive samples, mean F1 `0.5074022485589402`, mean precision `0.636639`, mean recall `0.424258`, total detections `9186`.
|
||||
- threshold `0.25`: 7 positive samples, mean F1 `0.44378879337957716`, mean precision `0.762683`, mean recall `0.316314`, total detections `5584`.
|
||||
- Conservative seven-reference AOI1024 persisted positive QA matrix:
|
||||
- command used the same manifest/model/tile/overlap with thresholds `0.35 0.45 0.60`.
|
||||
- output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-high-threshold/multi_sample_quality_summary.json`
|
||||
- threshold `0.35`: 7 positive samples, mean F1 `0.32086574003576274`, mean precision `0.840006`, mean recall `0.202135`, total detections `3106`.
|
||||
- threshold `0.45`: 7 positive samples, mean F1 `0.20093433626447627`, mean precision `0.885043`, mean recall `0.116348`, total detections `1592`.
|
||||
- threshold `0.60`: 7 positive samples, mean F1 `0.06640476239173489`, mean precision `0.871611`, mean recall `0.035083`, total detections `418`.
|
||||
- Full nine-sample hard-negative/background matrix:
|
||||
- command used `scripts/run_operator_hard_negative_detection_matrix.sh` with the AOI1024 manifest, model `geointel-building-yolov8s-aoi1024bg512r3e50-pt`, tile `512`, overlap `64` and thresholds `0.25 0.15 0.05`.
|
||||
- output: `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/aoi1024bg512r3e50-full/hard_negative_matrix_summary.json`
|
||||
- threshold `0.05`: total background detections `638`, max sample detections `184`.
|
||||
- threshold `0.15`: total background detections `390`, max sample detections `103`.
|
||||
- threshold `0.25`: total background detections `278`, max sample detections `75`.
|
||||
- Conservative nine-sample hard-negative/background matrix:
|
||||
- output: `/mnt/user/appdata/geointel/artifacts/detection-hard-negatives/aoi1024bg512r3e50-high-threshold/hard_negative_matrix_summary.json`
|
||||
- threshold `0.35`: total background detections `198`, max sample detections `55`.
|
||||
- threshold `0.45`: total background detections `148`, max sample detections `38`.
|
||||
- threshold `0.60`: total background detections `76`, max sample detections `18`.
|
||||
- Promotion reports:
|
||||
- balanced thresholds report: `/mnt/user/appdata/geointel/artifacts/detection-model-promotion/aoi1024bg512r3e50-full/detection_model_promotion_report.md`
|
||||
- conservative thresholds report: `/mnt/user/appdata/geointel/artifacts/detection-model-promotion/aoi1024bg512r3e50-high-threshold/detection_model_promotion_report.md`
|
||||
- recommendation: `none`.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- The background-aware model is the strongest positive-AOI candidate so far and is usable as an explicit review/demo candidate, especially at threshold `0.15` for balance or `0.35` for conservative high-precision review.
|
||||
- It is still not safe as a V1 default because the full background-candidate gate fails `background_false_positive_pressure`.
|
||||
- Some `background_candidate` AOIs contain GRB reference buildings and should be split into pure-empty negatives versus sparse-positive contextual samples before the next gate; otherwise the hard-negative score remains intentionally conservative but not perfectly diagnostic.
|
||||
- The model remains inactive. Operators can select it explicitly from the local model asset catalog.
|
||||
|
||||
## Next recommended pass
|
||||
|
||||
- Build a V1 operator detection profile layer in the UI/docs: expose `balanced` (`threshold=0.15`) and `conservative review` (`threshold=0.35`) as explicit choices for local model assets, with clear warning that the model is not a default-approved detector.
|
||||
- Clean the background corpus classification: separate pure-empty AOIs from sparse-building contextual AOIs, then retrain or recalibrate against that cleaner gate.
|
||||
|
||||
Reference in New Issue
Block a user