From 3c400d6167907393768a6440186825d94a61663a Mon Sep 17 00:00:00 2001 From: Codex Date: Tue, 7 Jul 2026 04:55:47 +0200 Subject: [PATCH] Document multi-sample detection quality baseline --- CHANGELOG.md | 1 + docs/CODEX_EXECUTION_LOG.md | 25 +++++++++++++++++++++++-- docs/TODO.md | 1 + 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6292c899..62575ec0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ - Added `scripts/run_multi_sample_detection_quality_matrix.sh` to run the existing real-data quality matrix for every prepared sample and combine the results. - The combined summary writes `multi_sample_quality_summary.json` with overall score/recall/precision rankings and per-sample best configurations. - Added readiness coverage and regression tests for the sample-preparation and multi-sample matrix contracts. +- Ran the full 24-run Tower matrix for Geel, Mol and Turnhout. Best overall score/recall was Turnhout with `yolov8n-building-segmentation-pt`, tile `512`, overlap `64`, threshold `0.15`, 142 detections, 18 matches, 124 false positives, 755 false negatives and F1 `0.03934426229508197`; generic `yolov8n-pt` produced zero building detections across all samples. ## Sprint 126 Detection quality matrix tooling (2026-07-07) diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index c10a9488..d985e5a8 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -21,16 +21,37 @@ Tested: - `python scripts\smoke_docs.py` passed. - `git diff --check` passed. - `bash scripts/run_readiness_check.sh` passed: 393 backend tests, frontend typecheck/build, Alembic head `202606120900`, live smoke syntax checks and the new sample/multi-sample checks. +- Pushed commit `06dfc5f` to Gitea and fast-forwarded Tower `/mnt/user/appdata/geointel` to the same commit. +- Tower sample preparation passed by running the new helper in the live `geointel` all-in-one container through stdin: + - Geel reused existing runtime files: 617 GRB GBG reference features. + - Mol was newly prepared: 374 GRB GBG reference features. + - Turnhout was newly prepared: 773 GRB GBG reference features. + - Manifest: `/mnt/user/appdata/geointel/storage/operator-data/operator_samples_manifest.json`. +- Tower multi-sample quality matrix passed: + - command: `OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" QUALITY_TILE_SIZES="512 640" QUALITY_TILE_OVERLAPS="64" QUALITY_THRESHOLDS="0.50 0.15" bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202` + - output: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025303Z/multi_sample_quality_summary.json` + - run count: 24 real persisted workflows across 3 samples. + - best overall score/recall: Turnhout, `yolov8n-building-segmentation-pt`, tile `512`, overlap `64`, threshold `0.15`, 142 detections, 18 matches, 124 false positives, 755 false negatives, precision `0.1267605633802817`, recall `0.02328589909443726`, F1 `0.03934426229508197`. + - best overall precision: Geel, `yolov8n-building-segmentation-pt`, tile `640`, overlap `64`, threshold `0.50`, 4 detections, 1 match, 3 false positives, 616 false negatives, precision `0.25`, recall `0.0016207455429497568`, F1 `0.0032206119162640897`. + - best Geel score: tile `512`, threshold `0.15`, 80 detections, 6 matches, 74 false positives, 611 false negatives, F1 `0.017216642754662843`. + - best Mol score: tile `640`, threshold `0.15`, 77 detections, 7 matches, 70 false positives, 367 false negatives, F1 `0.03104212860310421`. + - best Turnhout score: tile `512`, threshold `0.15`, 142 detections, 18 matches, 124 false positives, 755 false negatives, F1 `0.03934426229508197`. + - generic `yolov8n-pt` produced zero building detections across every sample, tile size and threshold. +- Tower best-score evidence export passed for every sample: + - Geel: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025303Z/geel/calibration_evidence_review.html`, 697 features, 611 false negatives, 74 false positives, 6 matched detections and 6 matched references. + - Mol: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025303Z/mol/calibration_evidence_review.html`, 451 features, 367 false negatives, 70 false positives, 7 matched detections and 7 matched references. + - Turnhout: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025303Z/turnhout/calibration_evidence_review.html`, 915 features, 755 false negatives, 124 false positives, 18 matched detections and 18 matched references. Open: -- Live Tower sample preparation and multi-sample matrix run still needed. +- None for multi-sample tooling. Limitations: - This is operator tooling only. It does not add a live GRB provider, live orthophoto provider, application endpoint, migration, frontend feature, model download or fixture inference path. - The prepared sample files are runtime artifacts under appdata/storage and remain excluded from Git. +- The three-sample benchmark confirms the current building-segmentation evaluation model is not extraction-quality for V1: best recall is only about 2.3% and false negatives dominate every sample. Next recommended pass: -- Run the sample-preparation helper in the Tower all-in-one container, then run the multi-sample matrix from the Tower checkout and document the combined quality baseline. +- Replace or add a stronger aerial/building model candidate and rerun the same multi-sample matrix; optionally add IoU-threshold sweeps after a model produces materially better candidate detections. ## Sprint 126 Detection quality matrix tooling (2026-07-07) diff --git a/docs/TODO.md b/docs/TODO.md index 43e1f458..aa8cbd17 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -99,6 +99,7 @@ This file now starts with the current implementation status. Older preparation/b - [x] Add calibration QA evidence export tooling for false-positive/false-negative inspection artifacts. - [x] Add real-data detection quality matrix tooling for model/tile/threshold comparison. - [x] Add reproducible Geel/Mol/Turnhout operator sample preparation and multi-sample quality matrix tooling. +- [x] Run first Geel/Mol/Turnhout persisted detection quality baseline. - [ ] Calibrate confidence, IoU and model selection against persisted Geel/Mol/Turnhout detections and any additional local orthophoto/reference samples. ## Sprint 8 status