Record split background promotion runtime evidence

2026-07-11 00:34:26 +02:00
parent a499c5fbfa
commit 340be960e9
2 changed files with 73 additions and 0 deletions
@@ -7,6 +7,14 @@

 # Changelog

+## Sprint 162 Split-background promotion runtime pass (2026-07-11)
+
+- Hardened split-background preflight compatibility for legacy operator manifests by deriving missing background categories from `reference_feature_count`.
+- Ran Tower split-background promotion evidence against `http://192.168.10.150:1202`.
+- Produced a high-threshold promote candidate: `geointel-building-yolov8s-aoi1024bg512r3e50-pt|512|64|0.35`.
+- Verified the strict pure-empty background gate: 3 samples, 9 runs, 0 detections.
+- Preserved the current model default; no automatic activation, provider fetching, fake outputs, model downloads, API contracts or migrations changed.
+
 ## Sprint 161 Widescreen workbench support (2026-07-10)

 - Added dedicated `1800px` and `2200px` frontend layout breakpoints for wide and ultrawide monitors.
@@ -6467,3 +6467,68 @@ Open:

 - Manually refresh the already-open `http://192.168.10.150:1202` browser tab on the actual widescreen monitor to observe the new layout.
 - Continue with the split-background promotion preflight/matrix workflow now that Tower is rebuilt with the latest frontend and AI runtime.
+
+# Sprint 162 - Split-background promotion runtime pass
+
+## What changed
+
+- Hardened `scripts/run_split_background_promotion_workflow.sh` preflight so legacy operator manifests without explicit `background_category` are handled consistently with `run_operator_hard_negative_detection_matrix.sh`.
+- Missing background categories are now derived from `reference_feature_count`: `0` becomes `pure_empty_negative`, and background samples with references become `sparse_building_context`.
+- Added a regression test for the derived-category preflight path.
+- Updated operator docs and changelog. No API contract, migration, provider fetching, model default, fake output or model download behavior changed.
+
+## What was tested locally
+
+- Ran `python -m pytest tests/test_sprint159_split_promotion_workflow.py -q`: 4 passed.
+- Ran `python -m pytest tests/test_sprint157_background_split_matrix_runner.py tests/test_sprint158_promotion_report_split_background.py tests/test_sprint159_split_promotion_workflow.py -q`: 9 passed.
+- Ran `bash -n scripts/run_split_background_promotion_workflow.sh`.
+- Ran `python -m compileall backend/app`.
+- Ran `python -m pytest` in `backend`: 450 passed, 17 existing Pydantic protected-namespace warnings.
+- Ran `npm run typecheck` in `frontend`.
+- Ran `npm run build` in `frontend`.
+- Ran `bash scripts/run_readiness_check.sh`: passed.
+
+## Tower runtime evidence
+
+- Pushed commit `a499c5f` and fast-forwarded the Tower checkout.
+- Ran split-background preflight against `http://192.168.10.150:1202` with:
+  - `PROMOTION_POSITIVE_PORTFOLIO_PATH=artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-full/multi_sample_quality_summary.json`
+  - `OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator-samples-1024/operator_samples_manifest.json`
+  - result: passed.
+- Ran the full low/mid-threshold split workflow into:
+  - background split: `artifacts/detection-hard-negatives/background-split/aoi1024bg512r3e50-split-20260710T222641Z`
+  - promotion report: `artifacts/detection-model-promotion/split-aware/aoi1024bg512r3e50-split-20260710T222641Z/detection_model_promotion_report.json`
+  - result: no recommended candidate because `0.15` produced 46 pure-empty detections on `postel_bos`, while `0.35` had no matching positive evidence in that portfolio.
+- Ran high-threshold preflight and full split workflow using:
+  - `PROMOTION_POSITIVE_PORTFOLIO_PATH=artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-high-threshold/multi_sample_quality_summary.json`
+  - `QUALITY_THRESHOLDS="0.35 0.45 0.6"`
+  - background split: `artifacts/detection-hard-negatives/background-split/aoi1024bg512r3e50-high-threshold-split-20260710T222934Z`
+  - promotion report: `artifacts/detection-model-promotion/split-aware/aoi1024bg512r3e50-high-threshold-split-20260710T222934Z/detection_model_promotion_report.json`
+- High-threshold strict pure-empty gate passed:
+  - 3 pure-empty samples: `arendonk_heide`, `lommel_heide`, `postel_bos`
+  - 9 pure-empty runs
+  - total detections: `0`
+  - max detections per sample/threshold run: `0`
+- Recommended candidate:
+  - `geointel-building-yolov8s-aoi1024bg512r3e50-pt|512|64|0.35`
+  - positive samples: `7`
+  - pure-empty background samples: `3`
+  - mean F1: `0.32086574003576274`
+  - mean precision: `0.8400057773951873`
+  - mean recall: `0.20213514285308795`
+  - max pure-empty detections: `0`
+  - promotion status: `promote_candidate`
+- Sparse-context review remained review-only:
+  - 6 sparse-context samples
+  - 18 sparse-context runs
+  - total detections: `422`
+  - max sparse-context detections: `55`
+
+## Known limitations
+
+- The recommended candidate has conservative recall (`0.2021`) at threshold `0.35`; it is suitable as an operator-review candidate, not as an automatically activated production default.
+- The model default was not changed. Activation should remain a separate explicit operator decision after reviewing the high-threshold report.
+
+## Next recommended pass
+
+- Add a guarded model activation/operator-selection workflow that can mark a promoted candidate as active only after the report artifact and candidate key are explicitly supplied.