Balance YOLO label QA coverage

CHANGELOG.md

@@ -13,7 +13,9 @@
 - Preserved Turnhout, Retie, Westerlo and Arendonk-heide as manifest-backed validation holdouts and made the tile exporter reject unknown samples or holdout leakage.
 - Added `recommended_split` provenance to generated sample/reference/tile metadata and recorded the validation split in dataset summaries.
 - Hardened persistent false-negative comparison so portfolios with different reference feature identities cannot be compared.
-- Documented the expanded low-minimum-label dataset flow; no model was activated and no product API or migration changed.
+- Generated and audited the 20-source expanded dataset on Tower: 171 retained tiles, 45,892 valid labels, 9 low-variance negatives removed and no structural audit warnings.
+- Balanced visual label QA by source sample before selecting repeated dense tiles; the live pass covered all 19 retained sources without invalid labels, missing images or blank selections.
+- No model was activated and no product API or migration changed.
 
 ## Sprint 170 Persistent false-negative evidence audit (2026-07-12)
 

backend/tests/test_sprint167_operator_yolo_label_qa_contact_sheets.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import importlib.util
 import json
 import subprocess
 import sys
@@ -11,6 +12,48 @@ from PIL import Image, ImageDraw
 ROOT = Path(__file__).resolve().parents[2]
 
 
+def load_renderer():
+    script_path = ROOT / "scripts" / "render_operator_yolo_label_qa_contact_sheets.py"
+    spec = importlib.util.spec_from_file_location("operator_label_qa_renderer", script_path)
+    assert spec is not None
+    assert spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def test_contact_sheet_selection_balances_source_samples_before_dense_repeats() -> None:
+    module = load_renderer()
+    tiles = [
+        {
+            "sample_slug": "dense",
+            "split": "train",
+            "tile_index": index,
+            "label_count": 100 - index,
+            "is_negative": False,
+            "kept": True,
+        }
+        for index in range(5)
+    ]
+    tiles.extend(
+        [
+            {"sample_slug": "medium", "split": "train", "tile_index": 0, "label_count": 20, "is_negative": False, "kept": True},
+            {"sample_slug": "small", "split": "val", "tile_index": 0, "label_count": 5, "is_negative": False, "kept": True},
+            {"sample_slug": "background", "split": "train", "tile_index": 0, "label_count": 0, "is_negative": True, "kept": True},
+        ]
+    )
+
+    selected = module.select_tiles(tiles, max_tiles=4)
+
+    assert {tile["sample_slug"] for tile in selected} == {
+        "dense",
+        "medium",
+        "small",
+        "background",
+    }
+
+
 def write_patterned_image(path: Path, color: tuple[int, int, int]) -> None:
     image = Image.new("RGB", (64, 64), color=color)
     draw = ImageDraw.Draw(image)

docs/CODEX_EXECUTION_LOG.md

@@ -6991,11 +6991,27 @@ Open:
 - Official GRB OGC API probes returned building features at all four new AOI centers.
 - Local sample generation was attempted but correctly stopped because the workstation Python lacks the existing GIS runtime extras; the all-in-one Tower runtime is the supported execution environment.
 
+## Tower runtime evidence
+
+- Pushed `0f49c98` and redeployed the all-in-one runtime at `http://192.168.10.150:1202`; live migration and browser proxy verification passed.
+- Refreshed `/app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json` to schema version 2 with 20 sources.
+- Newly fetched real GRB reference counts: Olen 1,952, Lille 1,839, Oud-Turnhout 2,691 and Kasterlee 1,831; all are `recommended_split=train`.
+- Exported `/app/storage/operator-data/yolo-building-aoi1024-expanded-minpx4vis035`:
+  - 171 retained tiles;
+  - 153 positive and 18 negative tiles;
+  - 45,892 labels;
+  - 144 train and 27 validation tiles;
+  - 9 low-variance negatives skipped;
+  - validation holdouts Turnhout, Retie, Westerlo and Arendonk-heide recorded in summary provenance.
+- Dataset audit status `ok`: no warnings, invalid labels or missing files; median normalized box area `0.000694274766`, small-box share `0.3832694151486098`.
+- Improved the visual contact-sheet selector after the first live sheet overrepresented dense AOIs. The balanced rerun selected 40 tiles across all 19 retained source samples with zero invalid labels, missing images, missing label files or low-variance selections.
+- Started one 50-epoch CPU YOLOv8s candidate as inactive runtime evidence: `geointel-building-yolov8s-aoi1024expandedminpx4vis035e50.pt`.
+
 ## Known limitations
 
 - No new model has been trained or activated.
-- The four AOIs still require full orthophoto/reference preparation, tile export, structural audit and visual contact-sheet review on Tower.
+- The inactive training run must finish and pass positive-AOI plus split-background promotion gates before it can be considered for activation.
 
 ## Next recommended pass
 
-- Redeploy Tower, refresh the existing AOI1024 manifest so only missing AOIs are downloaded, then compare a `min-label-px=4` expanded dataset against the rejected `min-label-px=12` baseline before deciding whether training is justified.
+- Finish the inactive candidate, run fixed-threshold positive evidence and split pure-empty/sparse-context background matrices, and preserve the current production default unless the promotion report passes every gate.

docs/TODO.md

@@ -501,7 +501,7 @@ This file now starts with the current implementation status. Older preparation/b
 - [x] Add explicit Olen, Lille, Oud-Turnhout and Kasterlee positive training AOIs.
 - [x] Preserve Turnhout, Retie, Westerlo and Arendonk-heide as manifest-backed validation holdouts.
 - [x] Make the tile exporter reject unknown validation samples and holdout leakage.
-- [ ] Refresh the full AOI1024 operator manifest on Tower and fetch only missing AOIs.
-- [ ] Export and audit a low-minimum-label dataset without changing the active model.
-- [ ] Render and inspect a label contact sheet before training.
-- [ ] Train only when the expanded dataset passes structural and visual review.
+- [x] Refresh the full AOI1024 operator manifest on Tower and fetch only missing AOIs.
+- [x] Export and audit a low-minimum-label dataset without changing the active model.
+- [x] Render and inspect a sample-balanced label contact sheet before training.
+- [ ] Finish the inactive expanded-minpx4 candidate and run the full promotion gate.

scripts/README.md

@@ -488,7 +488,10 @@ The renderer writes `operator_yolo_label_qa_summary.json`,
 existing YOLO labels on existing tile images only; it does not run inference,
 train a model, fetch providers or create fake detections. Missing image files,
 missing label files, invalid YOLO rows and low-variance/blank-looking rendered
-tiles are reported in the JSON/Markdown artifacts.
+tiles are reported in the JSON/Markdown artifacts. Selection is balanced by
+`sample_slug` before taking additional high-density tiles, so one dense urban
+AOI cannot hide the other source samples from visual review. The report records
+`selected_sample_count` and `selected_sample_slugs` for coverage evidence.
 
 Current Tower audit status:
 
@@ -518,6 +521,15 @@ Current Tower audit status:
   available but inactive. The split-aware promotion report rejected all tested
   thresholds: `0.25` and `0.35` passed the pure-empty background gate but had
   mean F1 below `0.25`; `0.15` also failed the pure-empty false-positive gate.
+- `yolo-building-aoi1024-expanded-minpx4vis035`: expanded small-building
+  recovery dataset with 20 source AOIs, 171 retained tiles, 45,892 labels,
+  144 train tiles, 27 validation tiles and 9 low-variance negatives removed.
+  Its configured audit passed with no warnings, median normalized box area
+  `0.000694274766`, small-box share `0.3832694151486098`, no invalid labels and
+  no missing label files. The balanced visual pass rendered 40 tiles across all
+  19 source samples that retained at least one tile, with no invalid labels,
+  missing images or low-variance selections. A new candidate may be trained,
+  but remains inactive until positive and split-background promotion gates pass.
 
 After rebuilding the all-in-one image, the operator scripts are available inside
 the container at `/app/scripts/...`. Before rebuilding, use the host checkout or

scripts/render_operator_yolo_label_qa_contact_sheets.py

@@ -120,6 +120,33 @@ def tile_sort_key(tile: dict[str, Any]) -> tuple[int, str, str, int, int]:
     )
 
 
+def balanced_tiles_by_sample(tiles: list[dict[str, Any]], limit: int) -> list[dict[str, Any]]:
+    if limit <= 0 or not tiles:
+        return []
+    grouped: dict[str, list[dict[str, Any]]] = {}
+    for tile in tiles:
+        grouped.setdefault(str(tile.get("sample_slug") or "unknown"), []).append(tile)
+    for sample_tiles in grouped.values():
+        sample_tiles.sort(key=tile_sort_key)
+    sample_order = sorted(grouped, key=lambda slug: tile_sort_key(grouped[slug][0]))
+
+    selected: list[dict[str, Any]] = []
+    depth = 0
+    while len(selected) < limit:
+        added = False
+        for slug in sample_order:
+            sample_tiles = grouped[slug]
+            if depth < len(sample_tiles):
+                selected.append(sample_tiles[depth])
+                added = True
+                if len(selected) == limit:
+                    break
+        if not added:
+            break
+        depth += 1
+    return selected
+
+
 def select_tiles(tiles: list[dict[str, Any]], max_tiles: int) -> list[dict[str, Any]]:
     if max_tiles <= 0:
         raise ValueError("max_tiles must be positive")
@@ -135,7 +162,8 @@ def select_tiles(tiles: list[dict[str, Any]], max_tiles: int) -> list[dict[str,
     )
 
     negative_slots = min(len(negatives), max(1, max_tiles // 5)) if negatives and max_tiles > 1 else 0
-    selected = positives[: max_tiles - negative_slots] + negatives[:negative_slots]
+    selected = balanced_tiles_by_sample(positives, max_tiles - negative_slots)
+    selected.extend(balanced_tiles_by_sample(negatives, negative_slots))
 
     if len(selected) < max_tiles:
         selected_ids = {id(tile) for tile in selected}
@@ -302,6 +330,12 @@ def build_report(summary: dict[str, Any], summary_path: Path, args: argparse.Nam
             "columns": args.columns,
             "thumb_size": args.thumb_size,
             "selected_tile_count": len(selected_tiles),
+            "selected_sample_count": len(
+                {str(tile.get("sample_slug") or "unknown") for tile in selected_tiles}
+            ),
+            "selected_sample_slugs": sorted(
+                {str(tile.get("sample_slug") or "unknown") for tile in selected_tiles}
+            ),
             "rendered_tile_count": len(rendered_cards),
             "missing_image_count": missing_image_count,
             "missing_label_file_count": missing_label_file_count,
@@ -322,6 +356,7 @@ def write_markdown(report: dict[str, Any], output_dir: Path) -> None:
         "",
         f"- status: `{report['status']}`",
         f"- selected tiles: {report['selected_tile_count']}",
+        f"- selected source samples: {report['selected_sample_count']}",
         f"- rendered tiles: {report['rendered_tile_count']}",
         f"- missing images: {report['missing_image_count']}",
         f"- missing label files: {report['missing_label_file_count']}",