Document live multi-AOI calibration portfolio

2026-07-08 14:13:58 +02:00
parent 28a49bd0d5
commit 8e1e80e22e
3 changed files with 48 additions and 0 deletions
@@ -7,6 +7,14 @@

 # Changelog

+## Sprint 140 Live multi-AOI calibration portfolio run (2026-07-08)
+
+- Ran the new multi-AOI calibration evidence portfolio assembler on Tower against existing persisted quality-matrix summaries for Geel, Mol and Turnhout.
+- Produced a real operator handoff under `/mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/live-20260708/output/` with portfolio JSON, Markdown and per-AOI evidence GeoJSON/HTML review artifacts.
+- Portfolio evidence contains 5,509 persisted QA evidence features across 3 AOIs: 5,239 false negatives, 164 false positives, 53 matched detections and 53 matched references.
+- Result: the evidence pipeline works, but the evaluated model/threshold set should not be promoted because recall remains very low across the AOIs.
+- No app rebuild, API change, migration, inference rerun, model training, model download, provider fetch or frontend runtime change was introduced.
+
 ## Sprint 139 Multi-AOI calibration evidence portfolio (2026-07-08)

 - Added `scripts/assemble_detection_calibration_evidence_portfolio.sh` to package multiple AOI calibration summaries and their persisted QA evidence bundles into one model-review portfolio.
@@ -1,3 +1,42 @@
+## Sprint 140 Live multi-AOI calibration portfolio run (2026-07-08)
+
+Changed:
+- Created a Tower-local `calibration-evidence-portfolio-manifest.json` for existing persisted multi-sample quality matrix summaries:
+  - Geel: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/geel/quality_matrix_summary.json`
+  - Mol: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/mol/quality_matrix_summary.json`
+  - Turnhout: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/turnhout/quality_matrix_summary.json`
+- Ran `scripts/assemble_detection_calibration_evidence_portfolio.sh` against the live Tower app at `http://127.0.0.1:1202`.
+- Produced the first real multi-AOI calibration evidence handoff:
+  - `/mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/live-20260708/output/calibration_evidence_portfolio.json`
+  - `/mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/live-20260708/output/calibration_evidence_portfolio.md`
+  - per-sample `calibration_evidence.geojson`, `calibration_evidence_summary.json` and `calibration_evidence_review.html` artifacts for Geel, Mol and Turnhout.
+
+Tested:
+- Tower manifest validation passed with `python3 -m json.tool`.
+- Live portfolio assembly passed against persisted QA evidence from the running app.
+- Output validation passed: portfolio JSON/Markdown and all per-sample evidence GeoJSON/summary/review files exist and are non-empty.
+- Tower runtime remained healthy on `0.0.0.0:1202->80/tcp`.
+
+Evidence:
+- Portfolio sample count: `3`.
+- Total evidence features: `5509`.
+- Combined role counts: `false_negative=5239`, `false_positive=164`, `match_candidate=53`, `match_reference=53`.
+- Best sample by score: `mol`.
+- Best per-sample scores:
+  - Geel: threshold `0.15`, F1/score `0.0187207488299532`, precision `0.25`, recall `0.009724473257698542`, evidence features `1885`.
+  - Mol: threshold `0.15`, F1/score `0.04195804195804196`, precision `0.16363636363636364`, recall `0.02406417112299465`, evidence features `1223`.
+  - Turnhout: threshold `0.15`, F1/score `0.03597122302158273`, precision `0.2459016393442623`, recall `0.019404915912031046`, evidence features `2401`.
+
+Open:
+- The live portfolio confirms the evidence pipeline works, but the evaluated model/threshold set is not yet operational-quality for building extraction due to very low recall and a dominant false-negative count.
+
+Limitations:
+- This pass produced operator artifacts only. It did not rerun inference, mutate application data, change database state, add endpoints, change frontend runtime behavior, download models, train models or rebuild the Docker app.
+- The portfolio input came from the existing `20260707T025932Z` quality matrix; the next model review should use fresh matrices when new AOIs or model candidates are added.
+
+Next recommended pass:
+- Add more positive and hard-negative AOIs, then run a fresh multi-sample quality matrix and portfolio for the next model candidate. Do not promote the current evaluated model as a V1 default.
+
 ## Sprint 139 Multi-AOI calibration evidence portfolio (2026-07-08)

 Changed:
@@ -426,5 +426,6 @@ This file now starts with the current implementation status. Older preparation/b
 - [x] Allow the evidence bundle script to consume Detection Lab calibration summary exports.
 - [x] Add a local browser-summary QA evidence bundle smoke using mocked canonical evidence responses.
 - [x] Add a multi-AOI calibration evidence portfolio convention for model-review handoff.
+- [x] Run the first live multi-AOI calibration evidence portfolio on Tower for Geel, Mol and Turnhout.
 - [ ] Add more AOIs after the tile-level baseline so the next local model attempt is not limited to Geel/Mol/Turnhout.
 - [ ] Add negative/background AOIs so the next tile dataset is not all positive tiles.