geointel/docs/BELGIUM_BUILDING_TRAINING_LOOP.md

Belgian building detector: closed training loop

Meaning of complete

100% trained means that every frozen release gate below passes. It does not mean a fabricated 100% precision, recall or mAP score. A model that memorises a small test set is not complete.

The loop is:

1. provision new, spatially independent AOIs from governed official services;
2. freeze imagery, labels, metadata and checksums into a new corpus version;
3. reject invalid, duplicate and sub-resolution labels and run spatial-leakage checks;
4. train only on the train split with CUDA on the Tower NVIDIA GPU;
5. use validation for early stopping and calibration only for threshold choice;
6. evaluate the fixed threshold once on regional test and background-test data;
7. attribute false positives and false negatives to a region, AOI and context;
8. add new training-only examples for the observed failure modes and repeat;
9. stop only when every objective gate passes; request human review afterward.

Protected calibration, test and background-test AOIs never become training data. A new iteration adds independent training AOIs instead.

Frozen release gates

Area	Gate
Runtime	CUDA required; NVIDIA device visible; no CPU fallback
Corpus	Immutable manifest and artifacts with SHA-256 evidence
Geographic composition	Each land region has at least 15 train, 2 val, 3 calibration, 3 test and 2 background-test AOIs
Contexts	Dense urban, suburban, rural, industrial and difficult negative contexts represented
Leakage	No intersecting AOIs across protected split roles
Label integrity	No malformed rows; sub-resolution labels explicitly rejected
Temporal truth	Unknown per-pixel dates remain unknown; acquisition dates may not masquerade as observation dates
Threshold selection	Calibration set only; maximise the worst regional F1 before aggregate F1
Test aggregate	F1 at least 0.55 at the frozen footprint/detection match IoU 0.25 contract
Dense-tile capacity	Retain up to 1000 detections per tile; the library default of 300 is below observed Belgian urban label density
Regional test	Every region: F1 at least 0.45, precision at least 0.50 and recall at least 0.40
Pure background	Zero detections on every pure-empty tile at the selected threshold
Production	Exact candidate checksum and fail-closed promotion report required
Final review	Human accepts every queued AOI contact sheet after all automated gates pass

These are minimum release gates, not performance targets. Raising a confidence threshold until detections disappear cannot pass because regional recall is a simultaneous gate.

Current gap inventory

The v4 dated corpus closes basic composition, leakage and temporal-identity gaps with 75 independent AOIs and 13,765 accepted building labels. It adds coastal, ribbon-development, farmland, park, forest and additional urban contexts. Its remaining known gaps are:

• dated 2025 imagery is used for Flanders and Brussels and the dated 2024 SPW campaign for Wallonia. Exact flight days remain a later metadata refinement, but all corpus relations are now measured periods rather than download dates;
• GRB/PICC/UrbIS describe ground footprints, whereas visible roofs can remain displaced. The existing detector QA contract therefore uses IoU 0.25; the threshold is frozen and cannot be relaxed per candidate;
• the first loop candidate generalises poorly in Flanders and Wallonia, especially Mechelen, Sint-Niklaas, Leuven, Mons and dense PICC areas;
• sparse hard contexts pass the empty-image test more easily than dense urban recall, so both gates must remain independent;
• building boxes are a valid first detector contract, but footprint-perfect geometry ultimately requires a separately validated segmentation model.

The active production model remains unchanged while any gate fails.

Reproducible evidence

• corpus assembler: scripts/assemble_belgium_building_corpus.py;
• corpus auditor: scripts/audit_belgium_building_corpus.py;
• tile exporter/auditor/contact sheets: the operator_yolo scripts;
• per-AOI evaluator: scripts/evaluate_belgium_building_candidate.py;
• calibration-only selection and release gates: scripts/assess_belgium_building_training_iteration.py.
• checkpointed CUDA orchestration: scripts/run_belgium_building_training_loop.py.

Every failed assessment returns continue_training_loop. Only a report with training_complete may proceed to final human review and guarded activation. Optimizer, initial learning rate, image size and geometric augmentation are explicit loop inputs. This permits a conservative aerial-imagery finetune (for example AdamW with mosaic disabled) without changing calibration, test or release gates.

After a failed assessment, scripts/build_failure_driven_yolo_sampling.py creates a checksummed, train-only sampling manifest. Positive tiles from regions that fail F1 or recall are repeated, while true negative train tiles are repeated when a regional precision gate or the pure-background gate fails. Calibration, test, background-test and validation AOIs are excluded by their frozen corpus split; the generated evidence records that no protected sample entered training. The checkpointed orchestrator invokes this builder after every rejected iteration, stores its checksum in training-loop-state.json, and uses the resulting dataset YAML for the next checkpoint. A restart resumes both the candidate weights and that exact failure-driven training input. The orchestrator refuses to start unless every automated frozen-dataset gate passes and the corpus contains zero blank/low-variance positive tiles. The audit status may remain needs_human_review while training and objective evaluation continue: final human sign-off is deliberately the last gate and can never be interpreted as model promotion approval in advance. For dated imagery, GRB BEGINDATUM and PICC DATE_CREAT are compared with the end of the imagery period. A feature created afterward is retained in the audit but excluded from training as created_after_imagery_period. UrbIS does not expose an equivalent feature creation field in this acquisition contract, so its remaining temporal relation stays an explicit sample-level limitation.

An opt-in visible-roof experiment can dissolve source footprints that truly touch or overlap; separated footprints are never bridged. The audit retains every contributing native feature identifier and reports both source-feature and visible-instance counts. This mode is not the default: the Belgium v8 experiment showed that unconditional touching-footprint dissolve can merge whole urban blocks and therefore must pass the same independent gates before it can replace native instances.

The compact-roof variant therefore merges a connected group only when it has at most 12 source footprints and fills at least 55% of its axis-aligned envelope. Larger or irregular connected groups retain their native instances and are marked native_instance_complex_touch_group. These fixed criteria prevent administrative row-house chains from becoming one ambiguous detector box while keeping the experiment deterministic and auditable.