docs: record fail-closed accuracy challenger

2026-07-15 05:42:17 +02:00
parent f82a7f456a
commit ecc9f77732
11 changed files with 218 additions and 22 deletions
@@ -0,0 +1,77 @@
+# Reviewed accuracy challenger - 2026-07-15
+
+## Scope
+
+This pass closes the evidence requirement from the preceding FP/FN review. It
+adds complete, training-only orthophoto/GRB AOIs outside all protected
+operational holdouts, trains one inactive local challenger and compares both
+models through the current persisted GeoIntel workflow. No model was
+downloaded and the active runtime configuration was not changed.
+
+- Active model: `geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt`
+- Active SHA256: `a9088b8491dfae36694b53e9e9406cb4e3511d334a5712fa34f75078a47759c1`
+- Challenger: `geointel-building-yolov8s-reviewedexp6-minpx3-img640-ft20-pt`
+- Challenger SHA256: `038f1f97a6afd534f29e1f392a730a58207b928ca01e31ab8d8fed6106705820`
+- Fixed profile: tile `512`, overlap `64`, confidence `0.15`, QA IoU `0.25`
+
+## Data and training
+
+Six new training-only centers were added in Arendonk, Dessel, Meerhout,
+Laakdal, Nijlen and Hulshout. Every center is in its documented municipality,
+uses a complete paged GRB GBG export and remains at least 2 km from every
+protected positive holdout.
+
+- New GRB references: 9,964 features
+- Corpus: 252 tiles, 234 positive and 18 negative
+- Split: 234 training and 18 validation tiles
+- Labels: 79,192
+- Dataset audit: `ok`, zero invalid or missing labels
+- Visual audit: 64 tiles across 28 retained sources, zero invalid or
+  low-variance selections
+- Fine-tuning: 20 CPU epochs from the active local model, image size `640`
+- Best checkpoint: epoch 16, validation precision `0.700`, recall `0.406`,
+  mAP50 `0.369`, mAP50-95 `0.159`
+
+## Coverage-aligned result
+
+Both assets were rerun after the current coverage-aware QA logic was deployed.
+Each zone comparison uses the same evaluated reference population; older runs
+with different edge coverage are not used for the decision.
+
+| Zone | Active F1 | Challenger F1 | Delta |
+| --- | ---: | ---: | ---: |
+| Mol Achterbos | 0.6694 | 0.6850 | +0.0156 |
+| Mol Donk | 0.5894 | 0.6259 | +0.0365 |
+| Mol Gompel | 0.6564 | 0.6694 | +0.0131 |
+| Mol Postel | 0.4749 | 0.4753 | +0.0005 |
+| Retie | 0.6325 | 0.6491 | +0.0166 |
+| Turnhout | 0.5919 | 0.5988 | +0.0069 |
+| Westerlo | 0.6336 | 0.6700 | +0.0364 |
+
+| Aggregate | Active | Challenger |
+| --- | ---: | ---: |
+| Mean precision | 0.6141 | 0.6251 |
+| Mean recall | 0.6062 | 0.6293 |
+| Mean F1 | 0.6069 | 0.6248 |
+| Matches | 5,711 | 5,919 |
+| False positives | 3,568 | 3,506 |
+| False negatives | 3,416 | 3,208 |
+
+## Promotion decision
+
+| Pure-empty control | Active detections | Challenger detections |
+| --- | ---: | ---: |
+| Arendonk-heide | 0 | 0 |
+| Lommel-heide | 0 | 0 |
+| Postel-bos | 0 | 2 |
+
+Decision: **retain the active model**. The challenger improves every positive
+zone but violates the zero-detection Postel-bos gate. The formal promotion
+report therefore recommends the existing active key
+`geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt|512|64|0.15`.
+Positive-score gains do not override a failed fail-closed background control.
+
+Persistent evidence is stored below
+`/app/storage/operator-data/model-review/reviewed-accuracy-expansion/` and
+`/app/storage/operator-evidence/mol-operational-validation/`. Generated rasters,
+weights and large evidence JSON files remain outside Git.