diff --git a/CHANGELOG.md b/CHANGELOG.md index cf9faa06..3d4a2f4b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ - Added `scripts/run_detection_calibration_sweep.sh` to run the existing real-data detection + QA workflow across multiple configured-YOLO confidence thresholds. - The sweep writes per-threshold logs and a `calibration_summary.json` with persisted detection count, QA score, precision, recall, F1, mean IoU, matches, false positives and false negatives. - Added readiness syntax coverage and static regression coverage for the calibration sweep contract. +- Ran the sweep on Tower against the Geel operator sample; threshold `0.15` ranked best among `0.50`, `0.35`, `0.25` and `0.15`, but recall remained below 1%, confirming the next problem is model/data calibration rather than runtime availability. - No new model dependencies, provider fetching, fake detections, API contracts or product UI behavior were introduced. ## Sprint 123 YOLO class and tile CRS normalization (2026-07-07) diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index d9e9ebcd..c3a6c3d6 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -14,16 +14,29 @@ Validation: - `bash -n scripts/run_detection_calibration_sweep.sh` passed. - `bash scripts/run_detection_calibration_sweep.sh --help` passed. - Missing-input guard printed usage and did not start a live workflow. +- `bash scripts/run_readiness_check.sh` passed: 388 backend tests, Alembic head check, frontend typecheck/build and shell syntax checks. +- Tower pulled commit `590e597` with `git pull --ff-only origin main` and ran the default Geel calibration sweep. +- The first Tower sweep exposed that the summary script looked for `f1_score` while persisted metrics use `f1`; the script was patched with a `metrics.get("f1")` fallback and regression coverage. +- RED: `python -m pytest backend/tests/test_sprint124_detection_calibration_sweep.py -q` failed while the script lacked the `f1` fallback. +- `python -m pytest backend/tests/test_sprint124_detection_calibration_sweep.py -q` passed after the fallback. +- Tower pulled commit `e8cd463` and reran the Geel calibration sweep successfully: + - output: `artifacts/detection-calibration/20260707T002103Z/calibration_summary.json` + - threshold `0.50`: 4 detections, score/F1 `0.0032206119162640897`, precision `0.25`, recall `0.0016207455429497568`, 1 match, 3 false positives, 616 false negatives + - threshold `0.35`: 9 detections, score/F1 `0.003194888178913738`, precision `0.1111111111111111`, recall `0.0016207455429497568`, 1 match, 8 false positives, 616 false negatives + - threshold `0.25`: 20 detections, score/F1 `0.0031397174254317113`, precision `0.05`, recall `0.0016207455429497568`, 1 match, 19 false positives, 616 false negatives + - threshold `0.15`: 54 detections, score/F1 `0.014903129657228018`, precision `0.09259259259259259`, recall `0.008103727714748784`, 5 matches, 49 false positives, 612 false negatives + - best by score for this sample: threshold `0.15` Open: -- Run the sweep on Tower with the Geel operator sample and record the observed threshold metrics. +- None for calibration tooling. Limitations: - The sweep is intentionally mutating and creates one real workflow run per threshold. - It is calibration tooling only; it does not change inference, add model downloads, fetch providers, seed demo detections or change API/UI behavior. +- The current active building model still performs poorly on the Geel validation sample. Threshold `0.15` is best among the tested values, but recall remains under 1%; this is model/data-quality evidence, not a production-ready extraction baseline. Next recommended pass: -- Run the Tower sweep for the current Geel sample, then repeat on additional operator-provided orthophoto/reference samples before choosing V1 default confidence/IoU guidance. +- Inspect false-positive/false-negative evidence for the Geel runs, then add at least two more local orthophoto/reference samples before choosing V1 default confidence/IoU guidance. ## Sprint 123 YOLO class and tile CRS normalization (2026-07-07)