Use split background reports in promotion gate
This commit is contained in:
@@ -260,6 +260,24 @@ The split runner writes `background_corpus_split_summary.json` and Markdown
|
||||
handoff output with a strict `pure_empty_negative` gate and a separate
|
||||
review-only `sparse_building_context` block.
|
||||
|
||||
Use that split summary directly in the model promotion report:
|
||||
|
||||
```bash
|
||||
python scripts/build_detection_model_promotion_report.py \
|
||||
--positive-portfolio artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
|
||||
--background-split-summary artifacts/detection-hard-negatives/aoi1024bg512r3e50-split/background_corpus_split_summary.json \
|
||||
--output-dir artifacts/detection-model-promotion/aoi1024bg512r3e50-split-aware \
|
||||
--min-positive-samples 7 \
|
||||
--min-background-samples 2 \
|
||||
--min-mean-f1 0.25 \
|
||||
--max-background-detections-per-sample 0
|
||||
```
|
||||
|
||||
The promotion report follows the split contract: `pure_empty_negative` is the
|
||||
only strict background gate for default promotion, while
|
||||
`sparse_building_context` remains review-only evidence in the report. This keeps
|
||||
contextual buildings from being treated as empty-background false positives.
|
||||
|
||||
The underlying single-category matrix remains available:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -6275,3 +6275,40 @@ Open:
|
||||
|
||||
- Rebuild/redeploy the runtime, regenerate the operator manifest if needed, then run `scripts/run_background_corpus_split_matrix.sh` against `http://192.168.10.150:1202`.
|
||||
- Use the emitted split report to decide whether to retrain, recalibrate thresholds or keep the AOI1024 candidate operator-only.
|
||||
|
||||
# Sprint 158 - Split-aware promotion report
|
||||
|
||||
## What changed
|
||||
|
||||
- Added `--background-split-summary` support to `scripts/build_detection_model_promotion_report.py`.
|
||||
- The promotion report now resolves a split report's `pure_empty_negative` source summary as the strict default-promotion background gate.
|
||||
- The same report records `sparse_building_context` as review-only evidence, including source path and detection-pressure context, without counting it as a default-promotion blocker.
|
||||
- Kept direct `--hard-negative-summary` support unchanged for older operator workflows.
|
||||
- Updated operator pipeline docs, TODO and changelog.
|
||||
|
||||
## What was tested
|
||||
|
||||
- Red step: `python -m pytest tests/test_sprint158_promotion_report_split_background.py -q` failed because the promotion report required `--hard-negative-summary` and did not yet accept `--background-split-summary`.
|
||||
- Ran `python -m pytest tests/test_sprint158_promotion_report_split_background.py -q`: 1 passed.
|
||||
- Ran `python -m pytest tests/test_sprint143_detection_model_promotion_report.py tests/test_sprint157_background_split_matrix_runner.py -q`: 7 passed.
|
||||
- Ran `python -m py_compile scripts/build_detection_model_promotion_report.py scripts/build_background_corpus_split_report.py`.
|
||||
- Ran `python -m pytest tests/test_sprint158_promotion_report_split_background.py tests/test_sprint143_detection_model_promotion_report.py tests/test_sprint157_background_split_matrix_runner.py -q`: 8 passed.
|
||||
- Ran `python -m compileall backend/app`.
|
||||
- Ran `python -m pytest` in `backend`: 444 passed, 17 existing Pydantic namespace warnings.
|
||||
- Ran `cd frontend && npm run typecheck`.
|
||||
- Ran `cd frontend && npm run build`.
|
||||
- Ran `bash scripts/run_readiness_check.sh`: passed.
|
||||
- Ran `cd backend && python -m alembic heads`: `202606120900 (head)`.
|
||||
- Ran `cd backend && python -m alembic upgrade head --sql`.
|
||||
- Ran `bash -n scripts/live_migration_smoke.sh`.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- This pass is report/tooling only. It does not run live split matrices on Tower, retrain YOLO, change a model default, change API contracts, change migrations, fetch providers, download model weights or create fake detections.
|
||||
- The AOI1024 background-aware local model remains explicit operator-review only until the split matrices plus positive-AOI QA/QC evidence pass the documented gates.
|
||||
|
||||
## Next recommended pass
|
||||
|
||||
- After redeploy, run `scripts/run_background_corpus_split_matrix.sh` on Tower for `geointel-building-yolov8s-aoi1024bg512r3e50-pt`.
|
||||
- Feed the generated `background_corpus_split_summary.json` into `scripts/build_detection_model_promotion_report.py --background-split-summary` together with the seven-AOI positive summary.
|
||||
- If pure-empty false-positive pressure still fails, retrain or recalibrate before any default activation. If pure-empty passes, inspect sparse-context review evidence before deciding whether to keep the model operator-only or prepare a guarded default-candidate decision.
|
||||
|
||||
@@ -123,6 +123,7 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Add explicit operator detection profiles for local model assets: balanced review around threshold `0.15` and conservative high-precision review around threshold `0.35`, both clearly marked as non-default-approved until promotion gates pass.
|
||||
- [x] Add pure-empty versus sparse-building contextual background corpus classification to operator manifests, hard-negative matrix filters and YOLO tile provenance.
|
||||
- [x] Add a split background-corpus matrix runner and report builder that runs pure-empty and sparse-context matrices separately.
|
||||
- [x] Teach the model promotion report to consume split background summaries so only `pure_empty_negative` blocks default promotion and `sparse_building_context` stays review-only.
|
||||
- [ ] Rerun split background matrices on Tower after rebuild, then retrain or recalibrate against the cleaner pure-empty gate plus separate sparse-context inspection matrix.
|
||||
- [ ] Promote a V1 default building detector only after it passes seven positive AOIs, clean hard-negative/background gates and persisted QA/QC evidence without fake detections or model downloads.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user