84 lines
4.4 KiB
Markdown
84 lines
4.4 KiB
Markdown
# Belgian building detector: closed training loop
|
|
|
|
## Meaning of complete
|
|
|
|
`100% trained` means that every frozen release gate below passes. It does not
|
|
mean a fabricated 100% precision, recall or mAP score. A model that memorises a
|
|
small test set is not complete.
|
|
|
|
The loop is:
|
|
|
|
1. provision new, spatially independent AOIs from governed official services;
|
|
2. freeze imagery, labels, metadata and checksums into a new corpus version;
|
|
3. reject invalid, duplicate and sub-resolution labels and run spatial-leakage
|
|
checks;
|
|
4. train only on the train split with CUDA on the Tower NVIDIA GPU;
|
|
5. use validation for early stopping and calibration only for threshold choice;
|
|
6. evaluate the fixed threshold once on regional test and background-test data;
|
|
7. attribute false positives and false negatives to a region, AOI and context;
|
|
8. add new training-only examples for the observed failure modes and repeat;
|
|
9. stop only when every objective gate passes; request human review afterward.
|
|
|
|
Protected calibration, test and background-test AOIs never become training
|
|
data. A new iteration adds independent training AOIs instead.
|
|
|
|
## Frozen release gates
|
|
|
|
| Area | Gate |
|
|
| --- | --- |
|
|
| Runtime | CUDA required; NVIDIA device visible; no CPU fallback |
|
|
| Corpus | Immutable manifest and artifacts with SHA-256 evidence |
|
|
| Geographic composition | Each land region has at least 15 train, 2 val, 3 calibration, 3 test and 2 background-test AOIs |
|
|
| Contexts | Dense urban, suburban, rural, industrial and difficult negative contexts represented |
|
|
| Leakage | No intersecting AOIs across protected split roles |
|
|
| Label integrity | No malformed rows; sub-resolution labels explicitly rejected |
|
|
| Temporal truth | Unknown per-pixel dates remain unknown; acquisition dates may not masquerade as observation dates |
|
|
| Threshold selection | Calibration set only; maximise the worst regional F1 before aggregate F1 |
|
|
| Test aggregate | F1 at least 0.55 at the frozen footprint/detection match IoU 0.25 contract |
|
|
| Regional test | Every region: F1 at least 0.45, precision at least 0.50 and recall at least 0.40 |
|
|
| Pure background | Zero detections on every pure-empty tile at the selected threshold |
|
|
| Production | Exact candidate checksum and fail-closed promotion report required |
|
|
| Final review | Human accepts every queued AOI contact sheet after all automated gates pass |
|
|
|
|
These are minimum release gates, not performance targets. Raising a confidence
|
|
threshold until detections disappear cannot pass because regional recall is a
|
|
simultaneous gate.
|
|
|
|
## Current gap inventory
|
|
|
|
The v4 dated corpus closes basic composition, leakage and temporal-identity
|
|
gaps with 75 independent AOIs and 13,765 accepted building labels. It adds
|
|
coastal, ribbon-development, farmland, park, forest and additional urban
|
|
contexts. Its remaining known gaps are:
|
|
|
|
- dated 2025 imagery is used for Flanders and Brussels and the dated 2024 SPW
|
|
campaign for Wallonia. Exact flight days remain a later metadata refinement,
|
|
but all corpus relations are now measured periods rather than download dates;
|
|
- GRB/PICC/UrbIS describe ground footprints, whereas visible roofs can remain
|
|
displaced. The existing detector QA contract therefore uses IoU 0.25; the
|
|
threshold is frozen and cannot be relaxed per candidate;
|
|
- the first loop candidate generalises poorly in Flanders and Wallonia,
|
|
especially Mechelen, Sint-Niklaas, Leuven, Mons and dense PICC areas;
|
|
- sparse hard contexts pass the empty-image test more easily than dense urban
|
|
recall, so both gates must remain independent;
|
|
- building boxes are a valid first detector contract, but footprint-perfect
|
|
geometry ultimately requires a separately validated segmentation model.
|
|
|
|
The active production model remains unchanged while any gate fails.
|
|
|
|
## Reproducible evidence
|
|
|
|
- corpus assembler: `scripts/assemble_belgium_building_corpus.py`;
|
|
- corpus auditor: `scripts/audit_belgium_building_corpus.py`;
|
|
- tile exporter/auditor/contact sheets: the `operator_yolo` scripts;
|
|
- per-AOI evaluator: `scripts/evaluate_belgium_building_candidate.py`;
|
|
- calibration-only selection and release gates:
|
|
`scripts/assess_belgium_building_training_iteration.py`.
|
|
- checkpointed CUDA orchestration:
|
|
`scripts/run_belgium_building_training_loop.py`.
|
|
|
|
Every failed assessment returns `continue_training_loop`. Only a report with
|
|
`training_complete` may proceed to final human review and guarded activation.
|
|
The orchestrator refuses to start unless the frozen dataset audit is `ok` and
|
|
contains zero blank/low-variance positive tiles.
|