Document detection quality matrix results
This commit is contained in:
@@ -13,6 +13,7 @@
|
|||||||
- The script writes per-run logs and a `quality_matrix_summary.json` with detection count, QA score, precision, recall, F1, mean IoU, matches, false positives and false negatives.
|
- The script writes per-run logs and a `quality_matrix_summary.json` with detection count, QA score, precision, recall, F1, mean IoU, matches, false positives and false negatives.
|
||||||
- The summary ranks `best_by_score`, `best_by_recall` and `best_by_precision` for operator model-quality decisions.
|
- The summary ranks `best_by_score`, `best_by_recall` and `best_by_precision` for operator model-quality decisions.
|
||||||
- Added readiness syntax coverage and static regression coverage for the quality matrix contract.
|
- Added readiness syntax coverage and static regression coverage for the quality matrix contract.
|
||||||
|
- Ran the matrix on Tower against the Geel operator sample: `yolov8n-building-segmentation-pt` with tile `512`, overlap `64` and threshold `0.15` ranked best by score/recall with 80 detections, 6 matches, 74 false positives, 611 false negatives and F1 `0.017216642754662843`; generic `yolov8n-pt` produced zero building detections.
|
||||||
|
|
||||||
## Sprint 125 Detection calibration evidence bundle (2026-07-07)
|
## Sprint 125 Detection calibration evidence bundle (2026-07-07)
|
||||||
|
|
||||||
|
|||||||
@@ -18,16 +18,29 @@ Tested:
|
|||||||
- `python scripts\smoke_docs.py` passed.
|
- `python scripts\smoke_docs.py` passed.
|
||||||
- `git diff --check` passed.
|
- `git diff --check` passed.
|
||||||
- `bash scripts/run_readiness_check.sh` passed: 390 backend tests, frontend typecheck/build, Alembic head `202606120900`, live smoke syntax checks and the new matrix syntax check.
|
- `bash scripts/run_readiness_check.sh` passed: 390 backend tests, frontend typecheck/build, Alembic head `202606120900`, live smoke syntax checks and the new matrix syntax check.
|
||||||
|
- Pushed commit `e728f70` to Gitea and fast-forwarded Tower `/mnt/user/appdata/geointel` to the same commit.
|
||||||
|
- Tower live matrix passed against the Geel operator sample:
|
||||||
|
- command: `QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" QUALITY_TILE_SIZES="512 640" QUALITY_TILE_OVERLAPS="64" QUALITY_THRESHOLDS="0.50 0.15" bash scripts/run_detection_quality_matrix.sh http://192.168.10.150:1202`
|
||||||
|
- output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/20260707T023000Z/quality_matrix_summary.json`
|
||||||
|
- run count: 8
|
||||||
|
- best by score and recall: `yolov8n-building-segmentation-pt`, tile `512`, overlap `64`, threshold `0.15`, 80 detections, 6 matches, 74 false positives, 611 false negatives, precision `0.075`, recall `0.009724473257698542`, F1 `0.017216642754662843`.
|
||||||
|
- best by precision: `yolov8n-building-segmentation-pt`, tile `640`, overlap `64`, threshold `0.50`, 4 detections, 1 match, 3 false positives, 616 false negatives, precision `0.25`, recall `0.0016207455429497568`, F1 `0.0032206119162640897`.
|
||||||
|
- generic `yolov8n-pt` produced zero building detections for all tested tile/threshold combinations.
|
||||||
|
- Tower best-run evidence export passed from the matrix summary:
|
||||||
|
- `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/20260707T023000Z/calibration_evidence.geojson`
|
||||||
|
- `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/20260707T023000Z/calibration_evidence_summary.json`
|
||||||
|
- `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/20260707T023000Z/calibration_evidence_review.html`
|
||||||
|
- evidence features: 697 total, 611 false negatives, 74 false positives, 6 matched detections and 6 matched references.
|
||||||
|
|
||||||
Open:
|
Open:
|
||||||
- Live Tower matrix run still needed against the current Geel operator sample.
|
- None for matrix tooling.
|
||||||
|
|
||||||
Limitations:
|
Limitations:
|
||||||
- This is operator benchmarking tooling only. It does not change inference behavior, add model downloads, seed fixture detections, fetch providers, change API contracts or change migrations.
|
- This is operator benchmarking tooling only. It does not change inference behavior, add model downloads, seed fixture detections, fetch providers, change API contracts or change migrations.
|
||||||
- A single Geel sample is not enough to declare a production V1 building-extraction baseline; additional orthophoto/reference samples are still needed before picking defaults.
|
- A single Geel sample is not enough to declare a production V1 building-extraction baseline; additional orthophoto/reference samples are still needed before picking defaults.
|
||||||
|
|
||||||
Next recommended pass:
|
Next recommended pass:
|
||||||
- Run the matrix on Tower for the current Geel operator sample, then decide whether the active building model should remain the default evaluation model or be replaced.
|
- Add at least two more local orthophoto/reference samples and run the same quality matrix before choosing V1 defaults; the current Geel evidence says runtime plumbing works, but the active model is still not extraction-quality.
|
||||||
|
|
||||||
## Sprint 125 Detection calibration evidence bundle (2026-07-07)
|
## Sprint 125 Detection calibration evidence bundle (2026-07-07)
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user