Prepare unique hard-negative training dataset

CHANGELOG.md

@@ -1415,3 +1415,4 @@ Added:
 - Added pytest coverage and readiness syntax checking for the new operator YOLO dataset audit script.
 - Recorded live Tower audit results showing `yolo-building-tile-expanded160` as the clean current baseline and r4/r8 hard-negative datasets as repeat-heavy evidence sets that need more unique background AOIs before further hard-negative training.
 - Expanded the explicit operator background-candidate AOI registry from 3 to 9 unique hard-negative locations and added tests for diversity/spread before further YOLO training.
+- Fixed the all-in-one Dockerfile so documented operator scripts are copied into `/app/scripts/`, then prepared and audited the new Tower `yolo-building-tile-uniquehardneg160` dataset as the next training candidate.

backend/tests/test_docker_runtime_config.py

@@ -55,6 +55,14 @@ def test_all_in_one_dockerfile_can_opt_into_ai_dependencies_without_base_install
     assert "libglib2.0-0" in dockerfile
 
 
+def test_all_in_one_dockerfile_copies_operator_scripts_for_runtime_use() -> None:
+    dockerfile = (ROOT / "deploy" / "unraid" / "Dockerfile.all-in-one").read_text(encoding="utf-8")
+
+    assert "COPY scripts/prepare_operator_real_data_samples.py /app/scripts/prepare_operator_real_data_samples.py" in dockerfile
+    assert "COPY scripts/export_operator_yolo_tile_dataset.py /app/scripts/export_operator_yolo_tile_dataset.py" in dockerfile
+    assert "COPY scripts/audit_operator_yolo_dataset_quality.py /app/scripts/audit_operator_yolo_dataset_quality.py" in dockerfile
+
+
 def test_compose_does_not_require_missing_root_env_file() -> None:
     compose = (ROOT / "docker-compose.yml").read_text(encoding="utf-8")
 

deploy/unraid/Dockerfile.all-in-one

@@ -46,6 +46,9 @@ WORKDIR /app
 
 COPY backend/ /app/
 COPY fixtures/ /app/fixtures/
+COPY scripts/prepare_operator_real_data_samples.py /app/scripts/prepare_operator_real_data_samples.py
+COPY scripts/export_operator_yolo_tile_dataset.py /app/scripts/export_operator_yolo_tile_dataset.py
+COPY scripts/audit_operator_yolo_dataset_quality.py /app/scripts/audit_operator_yolo_dataset_quality.py
 COPY deploy/unraid/nginx-all-in-one.conf /etc/nginx/conf.d/default.conf
 COPY deploy/unraid/all-in-one-start.sh /usr/local/bin/geointel-all-in-one-start
 COPY --from=frontend-build /frontend/dist/ /usr/share/nginx/html/

docs/CODEX_EXECUTION_LOG.md

@@ -5679,16 +5679,29 @@ Open:
 - Background candidates remain operator/runtime samples only. They are not product providers, not fixtures and not automatic app fetches.
 - Added sample-registry test coverage for minimum unique background count, unique centers and regional spread.
 - Updated operator documentation with the expanded default corpus and the next required Tower regeneration step.
+- Fixed the all-in-one Dockerfile so the documented operator scripts are copied into `/app/scripts/` during normal rebuilds.
+- Prepared the expanded Tower operator manifest and exported `yolo-building-tile-uniquehardneg160`.
 
 ## What was tested
 
 - `python -m pytest backend\tests\test_sprint131_operator_sample_expansion.py -q`
+- `python -m pytest backend\tests\test_sprint127_operator_sample_quality_matrix.py backend\tests\test_sprint131_operator_sample_expansion.py -q`
+- `bash scripts/run_readiness_check.sh`
+- Tower live operator sample prep:
+  - manifest samples: 16 total, 7 reference and 9 background candidates.
+  - background candidate GRB feature counts: Postel-bos 0, Lommel-heide 0, Kasterlee-bos 7, Dessel-heide 30, Ravels-bos 3, Meerhout-bos 20, Geel-Bel 17, Arendonk-heide 0, Herenthout-bos 90.
+- Tower live tile export and audit for `yolo-building-tile-uniquehardneg160`:
+  - status `ok`
+  - 576 tiles, 346 positive, 230 negative
+  - 16 samples, 13 positive samples, 9 background samples
+  - 11,757 labels, 0 missing label files, 0 invalid label rows
+  - repeated background negative share 0.0
 
 ## Known limitations
 
-- The new AOIs still need to be prepared on Tower before they affect the live operator YOLO tile exports.
-- GRB may return sparse buildings in some background candidates; they remain valid hard-negative candidates only after the manifest and tile audit confirm their actual labels.
+- Some background candidates contain real GRB buildings. They are still useful as mixed rural/background samples, but the pure negative pressure currently comes mostly from Postel-bos, Lommel-heide and Arendonk-heide plus empty tiles inside sparse candidates.
+- The running Tower container was updated via temporary `docker cp` for live data prep before the Dockerfile copy fix existed; a normal rebuild is needed for `/app/scripts/prepare_operator_real_data_samples.py` to exist inside the image automatically.
 
 ## Next recommended pass
 
-- Pull this commit on Tower, rerun `prepare_operator_real_data_samples.py`, export a new hard-negative tile dataset without excessive repeat pressure and audit it before training.
+- Rebuild the Tower all-in-one image, then use `yolo-building-tile-uniquehardneg160` as the next safer hard-negative training dataset candidate. Benchmark after training before changing defaults.

docs/TODO.md

@@ -444,11 +444,13 @@ This file now starts with the current implementation status. Older preparation/b
 - [x] Use live audit output to decide whether the next model pass needs more positive AOIs, label cleanup or unique hard negatives.
 - [x] Add more unique background/hard-negative AOIs before repeating hard-negative-balanced YOLO training.
 - [ ] Keep `yolo-building-tile-expanded160` as the clean current training baseline; avoid promoting r4/r8 repeat-heavy datasets as defaults.
-- [ ] Regenerate Tower operator samples, export a new unique-hard-negative tile dataset and rerun the dataset audit before training.
+- [x] Regenerate Tower operator samples, export a new unique-hard-negative tile dataset and rerun the dataset audit before training.
 
 # Sprint 147 - Unique hard-negative AOI expansion
 
 - [x] Expand the documented operator background candidates from 3 to 9 unique AOIs.
 - [x] Keep every new background AOI explicit, `allow_empty_reference=True`, and `sample_role='background_candidate'`.
 - [x] Add test coverage for minimum background candidate count, unique centers and regional spread.
-- [ ] Prepare the new samples on Tower and build a fresh hard-negative tile dataset.
+- [x] Prepare the new samples on Tower and build a fresh hard-negative tile dataset.
+- [ ] Rebuild Tower all-in-one image so the newly copied operator scripts are available inside `/app/scripts` without `docker cp`.
+- [ ] Train a new candidate from `yolo-building-tile-uniquehardneg160` and run the positive/background promotion gates before activating it.

scripts/README.md

@@ -354,6 +354,13 @@ Current Tower audit status:
   background AOIs before training another hard-negative-balanced candidate.
 - Regenerate `operator_samples_manifest.json` after pulling Sprint 147+ so the
   expanded unique background AOI set is available for the next tile export.
+- `yolo-building-tile-uniquehardneg160`: clean expanded-background baseline;
+  576 tiles, 346 positive, 230 negative, 11,757 labels, 0 invalid labels and
+  0 repeated background negatives in the first Tower audit.
+
+After rebuilding the all-in-one image, the operator scripts are available inside
+the container at `/app/scripts/...`. Before rebuilding, use the host checkout or
+temporarily copy scripts into the running container for one-off data prep.
 
 For hard-negative-balanced experiments, repeat only train-split negative tiles
 from samples marked `sample_role=background_candidate`: