Page GRB references for operator samples

docs/AI_PIPELINES.md

@@ -196,10 +196,15 @@ The helper fetches explicit Digitaal Vlaanderen orthophoto/GRB GBG sample pairs
 for the documented AOIs only and writes `operator_samples_manifest.json`. The
 default corpus includes dense reference AOIs for Geel, Mol, Turnhout, Herentals,
 Balen, Retie and Westerlo plus explicitly marked background candidates for
-Postel-bos, Lommel-heide and Kasterlee-bos. Background candidates may persist
-empty GRB FeatureCollections for negative-tile training; normal reference AOIs
-still fail on empty GRB responses. The application itself still does not perform
-live provider fetching.
+Postel-bos, Lommel-heide, Kasterlee-bos, Dessel-heide, Ravels-bos,
+Meerhout-bos, Geel-Bel, Arendonk-heide and Herenthout-bos. Background
+candidates may persist empty GRB FeatureCollections for negative-tile training;
+normal reference AOIs still fail on empty GRB responses. Dense GRB references
+are fetched through OGC API `rel=next` pagination links instead of trusting only
+the first 1000-feature page. Generated reference GeoJSON records
+`reference_pages_fetched`, `reference_truncated`, `reference_page_limit`,
+`reference_max_features` and `source_urls` for auditability. The application
+itself still does not perform live provider fetching.
 
 For confidence-threshold calibration, use the sweep wrapper:
 

docs/CODEX_EXECUTION_LOG.md

@@ -5938,6 +5938,32 @@ Open:
 
 - Use `yolo-building-tile-uniquehardneg160` as the next safer hard-negative training dataset candidate. Benchmark after training before changing defaults.
 
+# Sprint 152 - GRB reference paging for operator samples
+
+## What changed
+
+- Fixed `scripts/prepare_operator_real_data_samples.py` so GRB GBG reference exports follow OGC API `rel=next` pagination links instead of silently trusting only the first `limit=1000` page.
+- Added operator controls:
+  - `--reference-page-limit` / `OPERATOR_GRB_PAGE_LIMIT`, default `1000`.
+  - `--reference-max-features` / `OPERATOR_GRB_MAX_FEATURES`, default `100000`.
+- Generated reference GeoJSON now records `source_urls`, `reference_pages_fetched`, `reference_truncated`, `reference_page_limit` and `reference_max_features`.
+- Kept the change operator-only: no GeoIntel API route calls this helper, no product provider endpoint changed, no live GRB/OSM import was added, no migration changed and no YOLO model was activated.
+
+## What was tested
+
+- RED: `python -m pytest backend\tests\test_sprint131_operator_sample_expansion.py::test_fetch_reference_follows_grb_next_links_until_complete -q` failed because only the first GRB page was persisted.
+- GREEN: `python -m pytest backend\tests\test_sprint131_operator_sample_expansion.py::test_fetch_reference_follows_grb_next_links_until_complete -q`
+- `python -m pytest backend\tests\test_sprint127_operator_sample_quality_matrix.py backend\tests\test_sprint131_operator_sample_expansion.py -q`
+
+## Known limitations
+
+- Existing Tower AOI1024 artifacts were generated before this fix. Regenerate the operator samples with `--force` before retraining or re-auditing dense AOI labels.
+- This does not make the Sprint 7B GRB provider a live product importer; it only fixes explicit operator sample preparation.
+
+## Next recommended pass
+
+- Regenerate `/app/storage/operator-data/operator-samples-1024` on Tower with the paged script, re-export the AOI1024 YOLO tile dataset, rerun the dataset audit and only then consider another inactive training candidate.
+
 # Sprint 151 - Runtime GIS upload and AOI1024 YOLO candidate
 
 ## What changed

docs/TODO.md

@@ -472,5 +472,6 @@ This file now starts with the current implementation status. Older preparation/b
 - [x] Run AOI1024 background/hard-negative matrix for `geointel-building-yolov8s-aoi1024visible025e50-pt`.
 - [x] Fix promotion-report parsing for `multi_sample_quality_summary.json` inputs.
 - [x] Generate AOI1024 promotion report and keep recommended candidate as `none`.
-- [ ] Add GRB paging or smaller dense AOI sampling before trusting 1000-feature-capped dense reference exports as full ground truth.
+- [x] Add GRB paging before trusting dense reference exports as full ground truth.
+- [ ] Regenerate Tower AOI1024 operator samples with paged GRB references, then re-export and audit labels before any new training attempt.
 - [ ] Keep every local YOLO candidate inactive until positive-AOI and hard-negative promotion reports recommend default activation.