Add multi-sample detection quality calibration

docs/AI_PIPELINES.md

@@ -167,6 +167,17 @@ download model weights. A zero detection count is valid as runtime evidence only
 when the selected model genuinely returns no usable detections after canonical
 class filtering; it does not prove the model is useful for the target imagery.
 
+Documented operator samples can be prepared inside the all-in-one runtime
+container:
+
+```bash
+docker exec -it geointel python /app/scripts/prepare_operator_real_data_samples.py --samples geel,mol,turnhout
+```
+
+The helper fetches explicit Digitaal Vlaanderen orthophoto/GRB GBG sample pairs
+for the documented AOIs only and writes `operator_samples_manifest.json`. The
+application itself still does not perform live provider fetching.
+
 For confidence-threshold calibration, use the sweep wrapper:
 
 ```bash
@@ -201,6 +212,23 @@ rankings `best_by_score`, `best_by_recall` and `best_by_precision` are operator
 decision aids only; GeoIntel still does not download models, seed fixture
 detections or treat AI detections as ground truth without QA/QC.
 
+To compare the same model/tile/threshold grid across all prepared operator
+samples, use:
+
+```bash
+OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
+QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
+QUALITY_TILE_SIZES="512 640" \
+QUALITY_TILE_OVERLAPS="64" \
+QUALITY_THRESHOLDS="0.50 0.15" \
+bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202
+```
+
+The multi-sample summary exposes `best_overall_by_score`,
+`best_overall_by_recall`, `best_overall_by_precision` and `best_by_sample` so
+model-quality decisions are based on repeated persisted QA/QC evidence rather
+than one AOI.
+
 For visual error inspection, export the persisted QA evidence from a calibration
 summary:
 

docs/CODEX_EXECUTION_LOG.md

@@ -1,3 +1,37 @@
+## Sprint 127 Multi-sample detection quality calibration tooling (2026-07-07)
+
+Changed:
+- Added `scripts/prepare_operator_real_data_samples.py` as an explicit operator/runtime helper for documented Geel, Mol and Turnhout real-data samples.
+- The helper downloads small Digitaal Vlaanderen OMWRGBMRVL WMS `Ortho` GeoTIFFs and GRB OGC API Features `GBG` building GeoJSON references for the documented AOIs only.
+- The helper writes `operator_samples_manifest.json`, sample metadata, source URLs and attribution under the runtime operator-data directory and reuses existing files by default.
+- Added `scripts/run_multi_sample_detection_quality_matrix.sh` to run `scripts/run_detection_quality_matrix.sh` once per manifest sample.
+- The multi-sample wrapper combines per-sample `quality_matrix_summary.json` files into `multi_sample_quality_summary.json` with `best_overall_by_score`, `best_overall_by_recall`, `best_overall_by_precision` and `best_by_sample`.
+- Added readiness checks for Python compile and shell syntax.
+- Added regression coverage in `backend/tests/test_sprint127_operator_sample_quality_matrix.py`.
+- Updated `scripts/README.md`, `backend/README.md`, `docs/AI_PIPELINES.md`, `docs/TODO.md` and `CHANGELOG.md`.
+
+Tested:
+- RED: `python -m pytest backend\tests\test_sprint127_operator_sample_quality_matrix.py -q` failed because the sample-preparation and multi-sample scripts did not exist.
+- RED: `python -m pytest backend\tests\test_sprint127_operator_sample_quality_matrix.py::test_prepare_operator_real_data_samples_help_does_not_require_gis_dependencies -q` failed because `--help` required missing GIS dependencies.
+- `python -m pytest backend\tests\test_sprint127_operator_sample_quality_matrix.py -q` passed.
+- `python scripts\prepare_operator_real_data_samples.py --help` passed without requiring local GIS dependencies.
+- `python -m py_compile scripts\prepare_operator_real_data_samples.py` passed.
+- `bash -n scripts/run_multi_sample_detection_quality_matrix.sh` passed.
+- `python -m pytest backend\tests\test_sprint127_operator_sample_quality_matrix.py backend\tests\test_sprint126_detection_quality_matrix.py backend\tests\test_sprint125_detection_calibration_evidence_bundle.py backend\tests\test_sprint124_detection_calibration_sweep.py -q` passed.
+- `python scripts\smoke_docs.py` passed.
+- `git diff --check` passed.
+- `bash scripts/run_readiness_check.sh` passed: 393 backend tests, frontend typecheck/build, Alembic head `202606120900`, live smoke syntax checks and the new sample/multi-sample checks.
+
+Open:
+- Live Tower sample preparation and multi-sample matrix run still needed.
+
+Limitations:
+- This is operator tooling only. It does not add a live GRB provider, live orthophoto provider, application endpoint, migration, frontend feature, model download or fixture inference path.
+- The prepared sample files are runtime artifacts under appdata/storage and remain excluded from Git.
+
+Next recommended pass:
+- Run the sample-preparation helper in the Tower all-in-one container, then run the multi-sample matrix from the Tower checkout and document the combined quality baseline.
+
 ## Sprint 126 Detection quality matrix tooling (2026-07-07)
 
 Changed:

docs/TODO.md

@@ -98,7 +98,8 @@ This file now starts with the current implementation status. Older preparation/b
 - [x] Add real-data detection calibration sweep tooling for confidence-threshold and QA/QC metric comparison.
 - [x] Add calibration QA evidence export tooling for false-positive/false-negative inspection artifacts.
 - [x] Add real-data detection quality matrix tooling for model/tile/threshold comparison.
-- [ ] Calibrate confidence, IoU and model selection against persisted Geel detections and additional local orthophoto/reference samples.
+- [x] Add reproducible Geel/Mol/Turnhout operator sample preparation and multi-sample quality matrix tooling.
+- [ ] Calibrate confidence, IoU and model selection against persisted Geel/Mol/Turnhout detections and any additional local orthophoto/reference samples.
 
 ## Sprint 8 status