Record clean AOI1024 model rejection
GeoIntel CI / docs-smoke (push) Has been cancelled
GeoIntel CI / contract-smoke (push) Has been cancelled

This commit is contained in:
Codex
2026-07-11 17:05:03 +02:00
parent b3bd34c384
commit 2f60a41679
3 changed files with 117 additions and 1 deletions
+107
View File
@@ -6640,3 +6640,110 @@ Open:
## Next recommended pass
- Add/curate more high-quality positive AOIs and cleaner building labels, then rerun the multi-AOI calibration and promotion gate before changing default operator thresholds.
# Sprint 165 - Clean AOI1024 YOLO dataset and promotion rejection
## What changed
- Hardened `scripts/export_operator_yolo_tile_dataset.py` so legacy operator manifests without explicit `background_category` now derive background categories consistently with the split-background evaluator:
- `reference_feature_count == 0` -> `pure_empty_negative`
- `reference_feature_count > 0` -> `sparse_building_context`
- normal reference samples -> `reference_aoi`
- Added regression coverage in:
- `backend/tests/test_sprint130_operator_yolo_tile_dataset.py`
- `backend/tests/test_sprint156_background_corpus_classification.py`
- Documented the clean AOI1024 export/audit profile in `scripts/README.md`.
- Updated `docs/TODO.md` with the clean dataset and rejected candidate state.
- No API contract, migration, default model activation, provider fetch path, model download behavior or fake detection path changed.
## Local validation
- RED: `python -m pytest backend/tests/test_sprint130_operator_yolo_tile_dataset.py::test_background_category_is_derived_for_legacy_operator_manifests -q` failed before the exporter helper existed.
- GREEN: same targeted test passed after adding `background_category_for_sample`.
- Ran `python -m pytest backend/tests/test_sprint130_operator_yolo_tile_dataset.py backend/tests/test_sprint146_operator_yolo_dataset_quality_audit.py backend/tests/test_sprint157_background_split_matrix_runner.py -q`: 12 passed.
- Ran `python -m pytest backend/tests/test_sprint130_operator_yolo_tile_dataset.py backend/tests/test_sprint156_background_corpus_classification.py -q`: 11 passed.
- Ran `bash scripts/run_readiness_check.sh`: 457 backend tests passed, frontend typecheck passed, frontend build passed, readiness passed.
## Tower runtime evidence
- Pushed commit `b3bd34c` and redeployed the all-in-one Tower runtime at `http://192.168.10.150:1202`.
- Deploy validation passed:
- live migration smoke passed;
- browser runtime verification passed;
- container exposed `0.0.0.0:1202->80/tcp`.
- Regenerated clean AOI1024 tile dataset:
- dataset: `/app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035`
- tile count: `144`
- positive tiles: `114`
- negative tiles: `30`
- labels: `14632`
- `min_label_px=12`
- `min_label_visible_ratio=0.35`
- background categories in tile summary:
- `pure_empty_negative=27`
- `sparse_building_context=54`
- `reference_aoi=63`
- Dataset audit:
- report: `/app/artifacts/operator-yolo-dataset-audit/aoi1024-cleanpx12vis035/operator_yolo_dataset_quality_audit.json`
- status: `ok`
- positive sample count: `13`
- background sample count: `9`
- invalid labels: `0`
- missing label files: `0`
- median normalized box area: `0.001373291015625`
- small-box share: `0.0`
## Training and evaluation
- Trained a new inactive local model:
- model asset id: `geointel-building-yolov8s-aoi1024cleanpx12vis035e50-pt`
- model file: `/app/models/geointel-building-yolov8s-aoi1024cleanpx12vis035e50.pt`
- SHA256: `4863b27717cb7fd3126ccf86a988b3c1863b5fb9271cf35370ca2473b9ed10f4`
- base model: `/app/models/yolov8s.pt`
- dataset: `/app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035/dataset.yaml`
- epochs: `50`
- image size: `512`
- batch: `4`
- device: `cpu`
- active runtime model: unchanged
- Ran positive multi-AOI matrix:
- output: `artifacts/detection-quality-matrix/multi-sample/aoi1024cleanpx12vis035e50-positive/multi_sample_quality_summary.json`
- samples: `geel`, `mol`, `turnhout`, `herentals`, `balen`, `retie`, `westerlo`
- thresholds: `0.35`, `0.25`, `0.15`
- Ran split-background promotion workflow:
- background split: `artifacts/detection-hard-negatives/background-split/aoi1024cleanpx12vis035e50-split/background_corpus_split_summary.json`
- promotion report: `artifacts/detection-model-promotion/split-aware/aoi1024cleanpx12vis035e50-split/detection_model_promotion_report.json`
- recommended candidate: `none`
## Promotion result
- `geointel-building-yolov8s-aoi1024cleanpx12vis035e50-pt|512|64|0.15`
- rejected: `positive_mean_f1_below_gate`, `background_false_positive_pressure`
- positive samples: `7`
- background samples: `3`
- mean F1: `0.1542209371995884`
- mean precision: `0.18801639524786692`
- mean recall: `0.13146097412721683`
- max pure-empty detections: `2`
- `geointel-building-yolov8s-aoi1024cleanpx12vis035e50-pt|512|64|0.25`
- rejected: `positive_mean_f1_below_gate`
- mean F1: `0.14443976458069788`
- mean precision: `0.24254741138809843`
- mean recall: `0.10383903038875668`
- max pure-empty detections: `0`
- `geointel-building-yolov8s-aoi1024cleanpx12vis035e50-pt|512|64|0.35`
- rejected: `positive_mean_f1_below_gate`
- mean F1: `0.11851668674424971`
- mean precision: `0.30588130541017644`
- mean recall: `0.07481388046665544`
- max pure-empty detections: `0`
## Known limitations
- The clean label filter fixed the small-box audit problem but did not improve promotion-quality detection metrics.
- The new model remains useful evidence only and must stay inactive.
- The current limiting factor is not script/runtime wiring; it is training data representativeness and label geometry quality for small/dense Belgian building footprints.
## Next recommended pass
- Add building-size/visibility diagnostics per AOI and use them to choose or generate better positive samples before another training run. Do not spend more CPU on the current cleanpx dataset without changing the sample/label strategy.