diff --git a/CHANGELOG.md b/CHANGELOG.md index 01d93ec5..b8f75e01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ - The script writes per-run logs and a `quality_matrix_summary.json` with detection count, QA score, precision, recall, F1, mean IoU, matches, false positives and false negatives. - The summary ranks `best_by_score`, `best_by_recall` and `best_by_precision` for operator model-quality decisions. - Added readiness syntax coverage and static regression coverage for the quality matrix contract. +- Ran the matrix on Tower against the Geel operator sample: `yolov8n-building-segmentation-pt` with tile `512`, overlap `64` and threshold `0.15` ranked best by score/recall with 80 detections, 6 matches, 74 false positives, 611 false negatives and F1 `0.017216642754662843`; generic `yolov8n-pt` produced zero building detections. ## Sprint 125 Detection calibration evidence bundle (2026-07-07) diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index 2d6019e7..d89b3a60 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -18,16 +18,29 @@ Tested: - `python scripts\smoke_docs.py` passed. - `git diff --check` passed. - `bash scripts/run_readiness_check.sh` passed: 390 backend tests, frontend typecheck/build, Alembic head `202606120900`, live smoke syntax checks and the new matrix syntax check. +- Pushed commit `e728f70` to Gitea and fast-forwarded Tower `/mnt/user/appdata/geointel` to the same commit. +- Tower live matrix passed against the Geel operator sample: + - command: `QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" QUALITY_TILE_SIZES="512 640" QUALITY_TILE_OVERLAPS="64" QUALITY_THRESHOLDS="0.50 0.15" bash scripts/run_detection_quality_matrix.sh http://192.168.10.150:1202` + - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/20260707T023000Z/quality_matrix_summary.json` + - run count: 8 + - best by score and recall: `yolov8n-building-segmentation-pt`, tile `512`, overlap `64`, threshold `0.15`, 80 detections, 6 matches, 74 false positives, 611 false negatives, precision `0.075`, recall `0.009724473257698542`, F1 `0.017216642754662843`. + - best by precision: `yolov8n-building-segmentation-pt`, tile `640`, overlap `64`, threshold `0.50`, 4 detections, 1 match, 3 false positives, 616 false negatives, precision `0.25`, recall `0.0016207455429497568`, F1 `0.0032206119162640897`. + - generic `yolov8n-pt` produced zero building detections for all tested tile/threshold combinations. +- Tower best-run evidence export passed from the matrix summary: + - `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/20260707T023000Z/calibration_evidence.geojson` + - `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/20260707T023000Z/calibration_evidence_summary.json` + - `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/20260707T023000Z/calibration_evidence_review.html` + - evidence features: 697 total, 611 false negatives, 74 false positives, 6 matched detections and 6 matched references. Open: -- Live Tower matrix run still needed against the current Geel operator sample. +- None for matrix tooling. Limitations: - This is operator benchmarking tooling only. It does not change inference behavior, add model downloads, seed fixture detections, fetch providers, change API contracts or change migrations. - A single Geel sample is not enough to declare a production V1 building-extraction baseline; additional orthophoto/reference samples are still needed before picking defaults. Next recommended pass: -- Run the matrix on Tower for the current Geel operator sample, then decide whether the active building model should remain the default evaluation model or be replaced. +- Add at least two more local orthophoto/reference samples and run the same quality matrix before choosing V1 defaults; the current Geel evidence says runtime plumbing works, but the active model is still not extraction-quality. ## Sprint 125 Detection calibration evidence bundle (2026-07-07)