Filter low variance YOLO negative tiles

docs/CODEX_EXECUTION_LOG.md

@@ -6866,3 +6866,41 @@ Open:
 ## Next recommended pass
 
 - Add no-data/low-variance filtering to the operator YOLO tile export path, regenerate the clean AOI1024 dataset, rerun the contact-sheet QA, and only then consider another training attempt.
+
+# Sprint 168 - Operator YOLO low-variance negative filtering
+
+## What changed
+
+- Added opt-in low-variance negative filtering to `scripts/export_operator_yolo_tile_dataset.py`.
+- Added CLI/env controls:
+  - `--drop-low-variance-negatives` / `OPERATOR_YOLO_DROP_LOW_VARIANCE_NEGATIVES`;
+  - `--blank-range-threshold` / `OPERATOR_YOLO_BLANK_RANGE_THRESHOLD`.
+- The filter evaluates the rendered raster tile image and skips only negative tiles when enabled.
+- Positive/labeled tiles are never removed by this variance gate.
+- Kept tile records now include `low_visual_variance`.
+- Skipped blank/no-data negative records use `skip_reason="low_visual_variance_negative"`.
+- Dataset summaries now include:
+  - `drop_low_variance_negatives`;
+  - `blank_range_threshold`;
+  - `skipped_low_variance_negative_tile_count`.
+- Updated operator documentation with the refreshed AOI1024 cleanpx export command.
+- Added design and execution plan docs under `docs/superpowers/`.
+
+## Local validation
+
+- RED: `python -m pytest backend/tests/test_sprint130_operator_yolo_tile_dataset.py::test_export_can_skip_low_variance_negative_tiles -q` failed because `export_sample_tiles()` did not accept `drop_low_variance_negatives`.
+- GREEN: same targeted test passed after adding the filter.
+- Ran `python -m pytest backend/tests/test_sprint130_operator_yolo_tile_dataset.py -q`: 8 passed.
+- Ran `python -m pytest backend/tests/test_sprint130_operator_yolo_tile_dataset.py backend/tests/test_sprint167_operator_yolo_label_qa_contact_sheets.py backend/tests/test_docker_runtime_config.py::test_all_in_one_dockerfile_copies_operator_scripts_for_runtime_use backend/tests/test_docker_runtime_config.py::test_all_in_one_dockerfile_copies_operator_scripts_after_dependency_install -q`: 11 passed.
+- Ran `python scripts/export_operator_yolo_tile_dataset.py --help`: the CLI exposes `--drop-low-variance-negatives`, `--no-drop-low-variance-negatives` and `--blank-range-threshold` without loading GIS dependencies.
+- Ran `bash scripts/run_readiness_check.sh`: 460 backend tests passed, frontend typecheck passed, frontend build passed, readiness passed.
+
+## Known limitations
+
+- The low-variance gate is deliberately simple and only identifies visually blank/no-data-looking negative tiles.
+- It is opt-in to avoid silently changing historical dataset exports.
+- Operator visual contact-sheet review remains required before any new training run.
+
+## Next recommended pass
+
+- Redeploy Tower, regenerate the AOI1024 cleanpx dataset with `--drop-low-variance-negatives`, rerun dataset audit and contact-sheet QA, then decide whether the filtered dataset is suitable for another training run.

docs/TODO.md

@@ -131,7 +131,8 @@ This file now starts with the current implementation status. Older preparation/b
 - [x] Add guarded promoted-candidate activation helper requiring a promotion report path and exact candidate key before `.env` can be changed.
 - [x] Add per-sample YOLO dataset audit diagnostics for parsed labels, median box area, small-box share and AOI-specific warning codes.
 - [x] Add deterministic visual YOLO label QA contact sheets before spending more CPU on another training run.
-- [ ] Filter no-data/low-variance pure-empty negative tiles from operator YOLO exports before the next training run.
+- [x] Filter no-data/low-variance pure-empty negative tiles from operator YOLO exports before the next training run.
+- [ ] Regenerate the AOI1024 cleanpx YOLO dataset with low-variance negative filtering and rerun visual contact-sheet QA before training.
 - [ ] Apply promoted V1 default building detector only after explicit operator review of the emitted `.env` updates, followed by rebuild/restart and browser/runtime smoke.
 
 ## Sprint 8 status

docs/superpowers/plans/2026-07-11-yolo-low-variance-negative-filter.md

@@ -0,0 +1,56 @@
+# YOLO Low-Variance Negative Filter Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add opt-in filtering for blank/low-variance negative tiles in the operator YOLO tile exporter.
+
+**Architecture:** Keep the behavior inside `scripts/export_operator_yolo_tile_dataset.py` because this is operator-only dataset construction, not application inference. Compute low-variance from the raster window image array, skip only negative tiles when explicitly enabled, and expose all decisions in `yolo_tile_dataset_summary.json`.
+
+**Tech Stack:** Python, pytest, rasterio/Pillow runtime helpers already used by the exporter.
+
+---
+
+### Task 1: Regression Test
+
+**Files:**
+- Modify: `backend/tests/test_sprint130_operator_yolo_tile_dataset.py`
+
+- [x] Add a test that monkeypatches `image_array_from_raster_window` and verifies `export_sample_tiles` skips only low-variance negative tiles when `drop_low_variance_negatives=True`.
+- [x] Run the targeted test and confirm it fails because the exporter does not yet accept/report the new filter fields.
+
+### Task 2: Exporter Implementation
+
+**Files:**
+- Modify: `scripts/export_operator_yolo_tile_dataset.py`
+
+- [x] Add CLI/env options:
+  - `--drop-low-variance-negatives`
+  - `--blank-range-threshold`
+- [x] Add a helper to detect low visual variance from an image array.
+- [x] Thread the options into `export_sample_tiles`.
+- [x] Skip only negative low-variance tiles when the option is enabled.
+- [x] Add `low_visual_variance` to kept tile records.
+- [x] Add skipped records with `skip_reason="low_visual_variance_negative"`.
+- [x] Add summary fields for the option, threshold and skipped count.
+
+### Task 3: Documentation
+
+**Files:**
+- Modify: `scripts/README.md`
+- Modify: `docs/CODEX_EXECUTION_LOG.md`
+- Modify: `docs/TODO.md`
+
+- [x] Document the new operator export flags and intended Tower command.
+- [x] Record local validation and known limitation.
+- [x] Mark the low-variance export filter item as implemented after validation.
+
+### Task 4: Validation and Handoff
+
+**Files:**
+- No additional file changes expected.
+
+- [x] Run targeted pytest for the exporter/contact-sheet tests.
+- [x] Run `bash scripts/run_readiness_check.sh`.
+- [ ] Commit and push.
+- [ ] Rebuild/deploy Tower if code changed.
+- [ ] Regenerate the AOI1024 dataset with the new filter enabled, then render contact sheets and record the result.

docs/superpowers/specs/2026-07-11-yolo-low-variance-negative-filter-design.md

@@ -0,0 +1,34 @@
+# YOLO Low-Variance Negative Filter Design
+
+## Goal
+
+Prevent blank/no-data pure-empty negative tiles from entering the next operator YOLO training dataset while keeping the exporter conservative and auditable.
+
+## Scope
+
+This pass changes only operator-side YOLO tile export tooling. It does not change backend APIs, database schema, frontend behavior, model activation, training behavior, provider fetching or inference.
+
+## Approach
+
+Add an opt-in filter to `scripts/export_operator_yolo_tile_dataset.py` that evaluates raster tile image variance before writing negative tiles. The filter applies only after labels are computed and only when a tile is negative. Positive tiles are never dropped by this gate, even if visually low-variance, because dropping labeled data silently would be a worse failure mode.
+
+The filter will use the same simple max-min grayscale range heuristic as the contact-sheet QA script. A tile with range at or below `OPERATOR_YOLO_BLANK_RANGE_THRESHOLD` is treated as low-variance. When `--drop-low-variance-negatives` is enabled, that negative tile is skipped and recorded in summary fields instead of being written into `images/` and `labels/`.
+
+## Reporting
+
+The dataset summary must include:
+
+- `drop_low_variance_negatives`
+- `blank_range_threshold`
+- `skipped_low_variance_negative_tile_count`
+- skipped tile records with `skip_reason="low_visual_variance_negative"`
+
+Kept tile records should include `low_visual_variance` so downstream contact-sheet and audit tooling can expose the signal.
+
+## Validation
+
+Add a regression test that constructs a blank negative raster window and a patterned negative raster window, enables the filter, and verifies that only the blank negative is skipped for `low_visual_variance_negative`.
+
+## Known Limitation
+
+The heuristic is intentionally simple. It identifies blank/no-data tiles, not semantic quality. Operator visual QA remains required before another training run.