Jens/geointel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document live multi-AOI calibration portfolio
GeoIntel CI / docs-smoke (push) Has been cancelled
Details
GeoIntel CI / contract-smoke (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Codex
2026-07-08 14:13:58 +02:00
parent 28a49bd0d5
commit 8e1e80e22e
3 changed files with 48 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CODEX_EXECUTION_LOG.md
+39
View File
@@ -1,3 +1,42 @@
## Sprint 140 Live multi-AOI calibration portfolio run (2026-07-08)
Changed:
- Created a Tower-local `calibration-evidence-portfolio-manifest.json` for existing persisted multi-sample quality matrix summaries:
- Geel: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/geel/quality_matrix_summary.json`
- Mol: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/mol/quality_matrix_summary.json`
- Turnhout: `/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/20260707T025932Z/turnhout/quality_matrix_summary.json`
- Ran `scripts/assemble_detection_calibration_evidence_portfolio.sh` against the live Tower app at `http://127.0.0.1:1202`.
- Produced the first real multi-AOI calibration evidence handoff:
- `/mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/live-20260708/output/calibration_evidence_portfolio.json`
- `/mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/live-20260708/output/calibration_evidence_portfolio.md`
- per-sample `calibration_evidence.geojson`, `calibration_evidence_summary.json` and `calibration_evidence_review.html` artifacts for Geel, Mol and Turnhout.
Tested:
- Tower manifest validation passed with `python3 -m json.tool`.
- Live portfolio assembly passed against persisted QA evidence from the running app.
- Output validation passed: portfolio JSON/Markdown and all per-sample evidence GeoJSON/summary/review files exist and are non-empty.
- Tower runtime remained healthy on `0.0.0.0:1202->80/tcp`.
Evidence:
- Portfolio sample count: `3`.
- Total evidence features: `5509`.
- Combined role counts: `false_negative=5239`, `false_positive=164`, `match_candidate=53`, `match_reference=53`.
- Best sample by score: `mol`.
- Best per-sample scores:
- Geel: threshold `0.15`, F1/score `0.0187207488299532`, precision `0.25`, recall `0.009724473257698542`, evidence features `1885`.
- Mol: threshold `0.15`, F1/score `0.04195804195804196`, precision `0.16363636363636364`, recall `0.02406417112299465`, evidence features `1223`.
- Turnhout: threshold `0.15`, F1/score `0.03597122302158273`, precision `0.2459016393442623`, recall `0.019404915912031046`, evidence features `2401`.
Open:
- The live portfolio confirms the evidence pipeline works, but the evaluated model/threshold set is not yet operational-quality for building extraction due to very low recall and a dominant false-negative count.
Limitations:
- This pass produced operator artifacts only. It did not rerun inference, mutate application data, change database state, add endpoints, change frontend runtime behavior, download models, train models or rebuild the Docker app.
- The portfolio input came from the existing `20260707T025932Z` quality matrix; the next model review should use fresh matrices when new AOIs or model candidates are added.
Next recommended pass:
- Add more positive and hard-negative AOIs, then run a fresh multi-sample quality matrix and portfolio for the next model candidate. Do not promote the current evaluated model as a V1 default.
## Sprint 139 Multi-AOI calibration evidence portfolio (2026-07-08)
Changed: