Jens/geointel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add detection quality matrix
GeoIntel CI / docs-smoke (push) Has been cancelled
Details
GeoIntel CI / contract-smoke (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Codex
2026-07-07 04:29:30 +02:00
parent e1586727d2
commit e728f7004f
9 changed files with 513 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CODEX_EXECUTION_LOG.md
+31
View File
@@ -1,3 +1,34 @@
## Sprint 126 Detection quality matrix tooling (2026-07-07)
Changed:
- Added `scripts/run_detection_quality_matrix.sh` as an operator-facing model/tile/threshold matrix for the configured-YOLO real-data path.
- The matrix reuses `scripts/verify_real_data_detection_qa_workflow.sh` for each row so every result is backed by persisted Project, Dataset, AnalysisRun, Detection, QualityCheck, Metric and Export records.
- The script accepts `QUALITY_MODEL_ASSET_IDS`, `QUALITY_TILE_SIZES`, `QUALITY_TILE_OVERLAPS` and `QUALITY_THRESHOLDS`, writes per-run logs and produces `quality_matrix_summary.json`.
- The summary reports model asset, tile size, overlap, confidence threshold, detection count, QA score, precision, recall, F1, mean IoU, matches, false positives and false negatives.
- Added `best_by_score`, `best_by_recall` and `best_by_precision` rankings for operator model-quality decisions.
- Added readiness syntax coverage and regression coverage in `backend/tests/test_sprint126_detection_quality_matrix.py`.
- Updated `scripts/README.md`, `backend/README.md`, `docs/AI_PIPELINES.md`, `docs/TODO.md` and `CHANGELOG.md`.
Tested:
- RED: `python -m pytest backend\tests\test_sprint126_detection_quality_matrix.py -q` failed because `scripts/run_detection_quality_matrix.sh` did not exist.
- `python -m pytest backend\tests\test_sprint126_detection_quality_matrix.py -q` passed.
- `python -m pytest backend\tests\test_sprint126_detection_quality_matrix.py backend\tests\test_sprint124_detection_calibration_sweep.py backend\tests\test_sprint125_detection_calibration_evidence_bundle.py -q` passed.
- `bash -n scripts/run_detection_quality_matrix.sh` passed.
- `bash scripts/run_detection_quality_matrix.sh --help` passed.
- `python scripts\smoke_docs.py` passed.
- `git diff --check` passed.
- `bash scripts/run_readiness_check.sh` passed: 390 backend tests, frontend typecheck/build, Alembic head `202606120900`, live smoke syntax checks and the new matrix syntax check.
Open:
- Live Tower matrix run still needed against the current Geel operator sample.
Limitations:
- This is operator benchmarking tooling only. It does not change inference behavior, add model downloads, seed fixture detections, fetch providers, change API contracts or change migrations.
- A single Geel sample is not enough to declare a production V1 building-extraction baseline; additional orthophoto/reference samples are still needed before picking defaults.
Next recommended pass:
- Run the matrix on Tower for the current Geel operator sample, then decide whether the active building model should remain the default evaluation model or be replaced.
## Sprint 125 Detection calibration evidence bundle (2026-07-07)
Changed: