diff --git a/CHANGELOG.md b/CHANGELOG.md index 69cee555..6eb3d9d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,25 @@ # Changelog +## Sprint 199 Reviewed accuracy expansion (2026-07-15) + +- Added six leakage-free training AOIs in Arendonk, Dessel, Meerhout, Laakdal, + Nijlen and Hulshout, backed by 9,964 paged GRB building references. +- Exported and audited a 252-tile, 79,192-label corpus; the configured audit and + balanced 64-tile visual review found no invalid, missing or low-variance + selections. +- Fine-tuned one inactive local YOLOv8s challenger for 20 CPU epochs without + downloads. Its best checkpoint SHA256 is + `038f1f97a6afd534f29e1f392a730a58207b928ca01e31ab8d8fed6106705820`. +- Re-ran active and challenger models through the same current persisted QA/QC + pipeline on four Mol and three regional holdouts. The challenger improved + mean F1 from `0.6069` to `0.6248` and improved every zone. +- Retained the active model because the challenger produced two detections in + empty Postel-bos; the active profile remained zero across all three empty + controls. No runtime model or `.env` setting was changed. +- Replaced stale Detection Lab profile averages with coverage-aligned active + evidence: precision `0.6141`, recall `0.6062`, F1 `0.6069`. + ## Sprint 198 Evidence-closed model review (2026-07-15) - Completed visual and geometric review of 48 persisted false-positive and 48 diff --git a/backend/README.md b/backend/README.md index ac6372ed..3ef0728d 100644 --- a/backend/README.md +++ b/backend/README.md @@ -519,6 +519,13 @@ false-negative counts, but has lower precision than the previous balanced model; operators must review and persist QA/QC rather than treating detections as ground truth. +The latest coverage-aligned rerun of this exact profile measured mean precision +`0.6141`, recall `0.6062` and F1 `0.6069` over Mol Achterbos, Donk, Gompel and +Postel plus Retie, Turnhout and Westerlo. The three pure-empty controls remained +at zero detections. A reviewed six-AOI fine-tuning challenger reached mean F1 +`0.6248` but remained inactive because it produced two false detections in the +Postel-bos empty control. + For model-quality calibration, run the confidence sweep wrapper: ```bash diff --git a/backend/tests/test_sprint155_detection_operator_profiles.py b/backend/tests/test_sprint155_detection_operator_profiles.py index 467c0538..c19a6fc3 100644 --- a/backend/tests/test_sprint155_detection_operator_profiles.py +++ b/backend/tests/test_sprint155_detection_operator_profiles.py @@ -20,12 +20,14 @@ def test_detection_operator_profiles_define_explicit_yolo_candidates_and_promote assert "defaultApproved: true" in source assert "promotionRecommendation: 'promote_candidate'" in source assert "positiveSampleCount: 7" in source - assert "f1: 0.5824578631584316" in source + assert "precision: 0.6140895327792112" in source + assert "recall: 0.6062221049337548" in source + assert "f1: 0.6068607646002744" in source assert "f1: 0.5432865390636915" in source assert "maxBackgroundDetections: 0" in source assert "lege-achtergrondtest is geslaagd" in source - assert "1.571 minder gemiste gebouwen" in source - assert "onterecht gevonden objecten" in source + assert "Postel blijft met 47,5% F1" in source + assert "controlekandidaat en niet als grondwaarheid" in source def test_detection_lab_surfaces_profiles_as_deliberate_operator_actions() -> None: diff --git a/backend/tests/test_sprint193_end_user_workbench.py b/backend/tests/test_sprint193_end_user_workbench.py index 79d4e0b6..82d3dac6 100644 --- a/backend/tests/test_sprint193_end_user_workbench.py +++ b/backend/tests/test_sprint193_end_user_workbench.py @@ -67,7 +67,8 @@ def test_visible_ai_and_quality_labels_are_end_user_facing() -> None: providers = read("frontend/src/components/providers/ProviderPanel.tsx") assert "Aanbevolen controleprofiel kleine gebouwen" in profiles - assert "Controleer wel extra op onterecht gevonden objecten" in profiles + assert "Postel blijft met 47,5% F1" in profiles + assert "controlekandidaat en niet als grondwaarheid" in profiles assert "qualityStatusLabel" in quality assert "nog niet uitgevoerd" in quality assert "Nog geen bestand gekozen." in export_preview diff --git a/docs/AI_PIPELINES.md b/docs/AI_PIPELINES.md index d4de20c5..d320b405 100644 --- a/docs/AI_PIPELINES.md +++ b/docs/AI_PIPELINES.md @@ -421,9 +421,10 @@ false positives block default promotion. The focused small-building local model asset, `geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt`, is the current recommended Detection Lab operator profile. Use tile size `512`, overlap `64` -and confidence threshold `0.15`. Persisted QA/QC at match IoU `0.25` across -seven positive AOIs measured mean precision `0.5898`, recall `0.5770` and F1 -`0.5825`; minimum per-AOI F1 was `0.5528`. The strict three-sample pure-empty +and confidence threshold `0.15`. Its original promotion evidence at match IoU +`0.25` across seven positive AOIs measured mean precision `0.5898`, recall +`0.5770` and F1 `0.5825`; minimum per-AOI F1 was `0.5528`. The strict +three-sample pure-empty background gate produced zero detections. Compared with the previous balanced profile, the same persisted reference populations contain 1,571 fewer false negatives, including 745 fewer misses in the 25-100 m2 bucket and 181 fewer @@ -443,6 +444,24 @@ produced zero detections in the pure-empty Postel forest control. The active confidence therefore remains `0.15`. This result does not claim production perfection and does not justify another model-training run by itself. +A coverage-aligned July 2026 rerun supersedes the older displayed profile +averages above without changing the active model or threshold. On the exact +current pipeline, the seven independent Mol/Kempen zones measured mean +precision `0.6141`, recall `0.6062` and F1 `0.6069`; the minimum zone F1 was +`0.4749` in Mol Postel. The active model again produced zero detections in +Postel-bos, Lommel-heide and Arendonk-heide. These are the values shown in the +Detection Lab operator profile. + +The reviewed-accuracy experiment added six training-only AOIs from Arendonk, +Dessel, Meerhout, Laakdal, Nijlen and Hulshout. The paged GRB export contained +9,964 complete reference features. Its audited `512`-tile corpus retained 252 +tiles and 79,192 labels with no invalid or missing labels. The inactive +`geointel-building-yolov8s-reviewedexp6-minpx3-img640-ft20-pt` challenger +improved mean seven-zone F1 to `0.6248`, but produced two detections in the +explicitly empty Postel-bos control. The formal fail-closed promotion report +therefore retained the current active model. Positive-score gains never +override a failed pure-empty background gate. + False-positive and false-negative evidence from persisted detection QA can be classified through `detection_reviews`. The queue derives from quality-check evidence ids and resolves persisted Detection and reference VectorFeature rows. diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index 39563742..718bad61 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -8247,3 +8247,49 @@ Next: - Collect a new training-only small-building/background evidence pack outside all operational holdouts, then train an inactive candidate only if the pack passes label, leakage and sample-volume audits. + +## Sprint 199 - Reviewed accuracy expansion and fail-closed challenger (2026-07-15) + +Implemented: +- Added six training-only reference AOIs for Arendonk, Dessel, Meerhout, + Laakdal, Nijlen and Hulshout. Tests enforce explicit training role, unique + municipality/center pairs and at least 2 km separation from every protected + Mol, Turnhout, Retie, Westerlo, Vosselaar and Grobbendonk holdout. +- Prepared 1024 px orthophoto/GRB pairs on Tower. The paged GRB exports contain + 9,964 features in total, use EPSG:31370 rasters and have no truncated, + invalid or empty reference geometry. +- Exported `yolo-building-aoi1024-reviewedexp6-minpx3vis035`: 252 tiles, 234 + positive, 18 negative, 234 train, 18 validation and 79,192 labels. The + dataset audit returned `ok`; the 64-tile visual review covered 28 retained + sources with no invalid, missing or low-variance selections. +- Fine-tuned the active local YOLOv8s asset for 20 CPU epochs at image size + `640`. The best checkpoint came from epoch 16 and was copied as inactive + `geointel-building-yolov8s-reviewedexp6-minpx3-img640-ft20-pt`, SHA256 + `038f1f97a6afd534f29e1f392a730a58207b928ca01e31ab8d8fed6106705820`. +- Re-ran both active and challenger assets through the exact current API, + persistence and coverage-aware QA/QC path at tile `512`, overlap `64`, + confidence `0.15` and match IoU `0.25`. The evaluated reference populations + are equal per zone; older unequal-coverage runs were excluded. +- The active model measured mean precision `0.6141`, recall `0.6062` and F1 + `0.6069`; the challenger measured `0.6251`, `0.6293` and `0.6248`. Challenger + F1 improved in all seven zones and reduced false negatives from 3,416 to + 3,208. +- The challenger produced two detections in explicitly empty Postel-bos while + the active model remained at zero across Postel-bos, Lommel-heide and + Arendonk-heide. The formal promotion report recommended the existing active + key. No `.env`, active model path or runtime threshold was changed. +- Updated the approved Detection Lab profile to the current coverage-aligned + active evidence and retained the explicit Postel limitation. + +Validation evidence: +- Full readiness after the documentation/profile update passed 598 backend + tests, backend compilation, one Alembic head, frontend typecheck/build and + the complete shell syntax gate. +- Persistent promotion evidence lives under + `/app/storage/operator-data/model-review/reviewed-accuracy-expansion/` and + `/app/storage/operator-evidence/mol-operational-validation/`. + +Next: +- Convert the two confirmed Postel-bos challenger errors into a complete, + leakage-free hard-negative training sample, add independent empty controls, + then train a new inactive candidate through the same fail-closed gate. diff --git a/docs/TODO.md b/docs/TODO.md index 76a9b24f..2b280ef5 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -506,9 +506,9 @@ This file now starts with the current implementation status. Older preparation/b - [x] Add a model promotion decision report that combines positive-AOI score with hard-negative false-positive pressure. - [x] Train and reject a YOLOv8s hard-negative r8 partial candidate after 12 CPU epochs through the full positive/background promotion gate. - [x] Finish and reject the full YOLOv8s hard-negative r8 e60 candidate through the same promotion gate. -- [ ] Add more AOIs after the tile-level baseline so the next local model attempt is not limited to Geel/Mol/Turnhout. -- [ ] Add negative/background AOIs so the next tile dataset is not all positive tiles. -- [ ] Improve positive training coverage/label quality before the next higher-capacity model attempt; simply extending the same hardneg r8 run is not enough. +- [x] Add more AOIs after the tile-level baseline so the next local model attempt is not limited to Geel/Mol/Turnhout. +- [x] Add negative/background AOIs so the next tile dataset is not all positive tiles. +- [x] Improve positive training coverage/label quality before the next higher-capacity model attempt; simply extending the same hardneg r8 run is not enough. # Sprint 146 - Operator YOLO dataset quality audit - [x] Add a dataset/label-quality audit for generated operator YOLO tile datasets. @@ -516,7 +516,7 @@ This file now starts with the current implementation status. Older preparation/b - [x] Wire the audit script into the readiness syntax gate. - [x] Use live audit output to decide whether the next model pass needs more positive AOIs, label cleanup or unique hard negatives. - [x] Add more unique background/hard-negative AOIs before repeating hard-negative-balanced YOLO training. -- [ ] Keep `yolo-building-tile-expanded160` as the clean current training baseline; avoid promoting r4/r8 repeat-heavy datasets as defaults. +- [x] Supersede `yolo-building-tile-expanded160` only with audited AOI1024 corpora; no repeat-heavy r4/r8 dataset became a default. - [x] Regenerate Tower operator samples, export a new unique-hard-negative tile dataset and rerun the dataset audit before training. # Sprint 147 - Unique hard-negative AOI expansion @@ -550,7 +550,7 @@ This file now starts with the current implementation status. Older preparation/b - [x] Prefer the complete municipality workspace on a fresh session while keeping the lightweight boundary as the initially rendered layer. - [x] Make GeoJSON bounds calculation safe for municipality-scale layers. -- [ ] Acquire and tile a georeferenced raster only for an explicitly selected +- [x] Acquire and tile a georeferenced raster only for an explicitly selected Mol analysis zone before running the next configured-YOLO validation. # Sprint 182 - Municipality viewport delivery and bounded AI handoff @@ -561,7 +561,7 @@ This file now starts with the current implementation status. Older preparation/b - [x] Preserve complete loading and auto-fit behavior for small vectors and persisted AI/QA overlays. - [x] Allow the real raster/detection/QA operator smoke to reuse the definitive Mol project. - [x] Link operator raster/reference uploads to the persisted analysis Area and keep their names distinct from municipality-wide layers. -- [ ] Execute the bounded Mol-Centrum configured-YOLO/QA workflow in the deployed runtime and retain its persisted ids as release evidence. +- [x] Execute the bounded Mol-Centrum configured-YOLO/QA workflow in the deployed runtime and retain its persisted ids as release evidence. # Sprint 171 - Positive AOI expansion and small-building recovery @@ -575,3 +575,15 @@ This file now starts with the current implementation status. Older preparation/b - [x] Finish the inactive expanded-minpx4 candidate and run the full promotion gate. - [x] Export the focused 23-sample minpx3 corpus with independent Vosselaar/Grobbendonk validation and external Turnhout/Retie/Westerlo holdouts. - [x] Train, audit and guarded-activate the focused small-building candidate only after all persisted promotion gates passed. + +# Sprint 199 - Reviewed accuracy expansion and fail-closed challenger + +- [x] Add six complete training-only AOIs outside every protected Mol/Kempen holdout. +- [x] Page and audit 9,964 new GRB building references with municipality and CRS provenance. +- [x] Export and visually inspect the 252-tile, 79,192-label reviewed-accuracy corpus. +- [x] Train one inactive 20-epoch YOLOv8s challenger from the active local model without downloads. +- [x] Rerun active and challenger models on identical coverage-aware reference populations. +- [x] Confirm positive-zone F1 improvement in all seven holdouts. +- [x] Reject challenger activation because Postel-bos produced two detections while the active model remained at zero. +- [x] Update the recommended UI profile with the current active model's coverage-aligned evidence. +- [ ] Collect new training-only hard negatives matching the two Postel-bos errors before another challenger attempt. diff --git a/docs/reviews/2026-07-15-reviewed-accuracy-challenger.md b/docs/reviews/2026-07-15-reviewed-accuracy-challenger.md new file mode 100644 index 00000000..9ab65b29 --- /dev/null +++ b/docs/reviews/2026-07-15-reviewed-accuracy-challenger.md @@ -0,0 +1,77 @@ +# Reviewed accuracy challenger - 2026-07-15 + +## Scope + +This pass closes the evidence requirement from the preceding FP/FN review. It +adds complete, training-only orthophoto/GRB AOIs outside all protected +operational holdouts, trains one inactive local challenger and compares both +models through the current persisted GeoIntel workflow. No model was +downloaded and the active runtime configuration was not changed. + +- Active model: `geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt` +- Active SHA256: `a9088b8491dfae36694b53e9e9406cb4e3511d334a5712fa34f75078a47759c1` +- Challenger: `geointel-building-yolov8s-reviewedexp6-minpx3-img640-ft20-pt` +- Challenger SHA256: `038f1f97a6afd534f29e1f392a730a58207b928ca01e31ab8d8fed6106705820` +- Fixed profile: tile `512`, overlap `64`, confidence `0.15`, QA IoU `0.25` + +## Data and training + +Six new training-only centers were added in Arendonk, Dessel, Meerhout, +Laakdal, Nijlen and Hulshout. Every center is in its documented municipality, +uses a complete paged GRB GBG export and remains at least 2 km from every +protected positive holdout. + +- New GRB references: 9,964 features +- Corpus: 252 tiles, 234 positive and 18 negative +- Split: 234 training and 18 validation tiles +- Labels: 79,192 +- Dataset audit: `ok`, zero invalid or missing labels +- Visual audit: 64 tiles across 28 retained sources, zero invalid or + low-variance selections +- Fine-tuning: 20 CPU epochs from the active local model, image size `640` +- Best checkpoint: epoch 16, validation precision `0.700`, recall `0.406`, + mAP50 `0.369`, mAP50-95 `0.159` + +## Coverage-aligned result + +Both assets were rerun after the current coverage-aware QA logic was deployed. +Each zone comparison uses the same evaluated reference population; older runs +with different edge coverage are not used for the decision. + +| Zone | Active F1 | Challenger F1 | Delta | +| --- | ---: | ---: | ---: | +| Mol Achterbos | 0.6694 | 0.6850 | +0.0156 | +| Mol Donk | 0.5894 | 0.6259 | +0.0365 | +| Mol Gompel | 0.6564 | 0.6694 | +0.0131 | +| Mol Postel | 0.4749 | 0.4753 | +0.0005 | +| Retie | 0.6325 | 0.6491 | +0.0166 | +| Turnhout | 0.5919 | 0.5988 | +0.0069 | +| Westerlo | 0.6336 | 0.6700 | +0.0364 | + +| Aggregate | Active | Challenger | +| --- | ---: | ---: | +| Mean precision | 0.6141 | 0.6251 | +| Mean recall | 0.6062 | 0.6293 | +| Mean F1 | 0.6069 | 0.6248 | +| Matches | 5,711 | 5,919 | +| False positives | 3,568 | 3,506 | +| False negatives | 3,416 | 3,208 | + +## Promotion decision + +| Pure-empty control | Active detections | Challenger detections | +| --- | ---: | ---: | +| Arendonk-heide | 0 | 0 | +| Lommel-heide | 0 | 0 | +| Postel-bos | 0 | 2 | + +Decision: **retain the active model**. The challenger improves every positive +zone but violates the zero-detection Postel-bos gate. The formal promotion +report therefore recommends the existing active key +`geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt|512|64|0.15`. +Positive-score gains do not override a failed fail-closed background control. + +Persistent evidence is stored below +`/app/storage/operator-data/model-review/reviewed-accuracy-expansion/` and +`/app/storage/operator-evidence/mol-operational-validation/`. Generated rasters, +weights and large evidence JSON files remain outside Git. diff --git a/frontend/README.md b/frontend/README.md index c09862cf..275a6077 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -183,6 +183,7 @@ Before creating tiles, the guided action inspects raster dimensions and estimate - When `yolo-configured` is selected, users can provide an existing raster tile manifest path. - Detection Lab lists local model assets from `GET /api/v1/detection/model-assets` so operators can choose an existing mounted model file instead of editing only one hidden `YOLO_MODEL_PATH` slot. - Detection Lab exposes explicit operator profiles for mounted local building detectors. The focused small-building model is the recommended recall-balanced `0.15` profile; the previous expanded-AOI `0.15` model remains available for higher precision, and the background-aware `0.35` model remains the conservative review choice. Applying a profile never downloads weights, changes runtime environment or starts inference automatically. +- The recommended profile metrics are coverage-aligned with the current QA pipeline: mean precision `0.6141`, recall `0.6062` and F1 `0.6069` over seven independent Mol/Kempen zones, with zero detections in all three pure-empty controls. A higher-positive-F1 challenger remains hidden from approved profiles because it failed the Postel-bos empty-control gate. - Applying a profile deliberately selects the local model asset and threshold for the browser-run request; runtime default activation remains a separate guarded `.env` operation through `scripts/activate_promoted_yolo_candidate.py`. - Detection Lab includes a read-only YOLO runtime preflight panel with backend status, dependency visibility, local model configuration, `torch`/`ultralytics` versions, CUDA state and `YOLO_CONFIG_DIR`. - The UI still does not download models or create fake detections; backend status and error codes remain the source of truth. diff --git a/frontend/src/components/detection/detectionProfiles.ts b/frontend/src/components/detection/detectionProfiles.ts index 210fd3a6..95d495b6 100644 --- a/frontend/src/components/detection/detectionProfiles.ts +++ b/frontend/src/components/detection/detectionProfiles.ts @@ -22,15 +22,15 @@ export const DETECTION_OPERATOR_PROFILES: DetectionOperatorProfile[] = [ confidenceThreshold: 0.15, defaultApproved: true, promotionRecommendation: 'promote_candidate', - precision: 0.5898197517793451, - recall: 0.576992100419565, - f1: 0.5824578631584316, + precision: 0.6140895327792112, + recall: 0.6062221049337548, + f1: 0.6068607646002744, positiveSampleCount: 7, maxBackgroundDetections: 0, description: - 'Aanbevolen profiel met een evenwicht tussen gevonden en gemiste kleine gebouwen, gemeten over zeven testgebieden in de Kempen.', + 'Aanbevolen profiel met een evenwicht tussen gevonden en gemiste kleine gebouwen, opnieuw gemeten over zeven onafhankelijke testgebieden in Mol en de Kempen.', limitationMessage: - 'De lege-achtergrondtest is geslaagd en de kwaliteitsmeting vond 1.571 minder gemiste gebouwen dan het vorige profiel. Controleer wel extra op onterecht gevonden objecten.', + 'De drie lege-achtergrondtests zijn geslaagd. Postel blijft met 47,5% F1 het moeilijkste testgebied; behandel elke detectie als een controlekandidaat en niet als grondwaarheid.', }, { id: 'expanded-balanced-review', diff --git a/scripts/README.md b/scripts/README.md index 8ebbe287..3afedac4 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -624,10 +624,22 @@ Current Tower audit status: review contained no missing, invalid or low-variance selections. The trained `geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt` candidate passed seven positive-AOI and three pure-empty background gates at tile `512`, - overlap `64`, threshold `0.15` and QA match IoU `0.25`. Mean F1 is `0.5825` - and all pure-empty samples remain at zero detections. Persisted comparison - found 1,571 fewer false negatives than the previous balanced model, with a - lower mean precision and therefore a higher operator review load. + overlap `64`, threshold `0.15` and QA match IoU `0.25`. Original promotion + evidence measured mean F1 `0.5825`; the later coverage-aligned rerun measured + `0.6069`. All pure-empty samples remain at zero detections. The original + persisted comparison found 1,571 fewer false negatives than the previous + balanced model, with a lower mean precision and therefore a higher operator + review load. +- `yolo-building-aoi1024-reviewedexp6-minpx3vis035`: leakage-free accuracy + expansion of the focused corpus with new training-only AOIs in Arendonk, + Dessel, Meerhout, Laakdal, Nijlen and Hulshout. The paged GRB preparation + added 9,964 reference features. The export retained 252 tiles (234 positive, + 18 negative) and 79,192 labels; its configured audit and 64-tile visual + review found no invalid, missing or low-variance selections. The inactive + `geointel-building-yolov8s-reviewedexp6-minpx3-img640-ft20-pt` model improved + coverage-aligned seven-zone mean F1 from `0.6069` to `0.6248`, but its two + detections in empty Postel-bos failed the strict background gate. The active + focused model remains unchanged. Use `--samples` or `OPERATOR_YOLO_SAMPLES` to make an experimental corpus membership explicit. Dataset summaries preserve the complete manifest count,