geointel/docs/BELGIUM_BUILDING_TRAINING_LOOP.md

Belgian building detector: closed training loop

Meaning of complete

100% trained means that every frozen release gate below passes. It does not mean a fabricated 100% precision, recall or mAP score. A model that memorises a small test set is not complete.

The loop is:

1. provision new, spatially independent AOIs from governed official services;
2. freeze imagery, labels, metadata and checksums into a new corpus version;
3. reject invalid, duplicate and sub-resolution labels and run spatial-leakage checks;
4. train only on the train split with CUDA on the Tower NVIDIA GPU;
5. use validation for early stopping and calibration only for threshold choice;
6. evaluate the fixed threshold once on regional test and background-test data;
7. attribute false positives and false negatives to a region, AOI and context;
8. add new training-only examples for the observed failure modes and repeat;
9. stop only when every objective gate passes; request human review afterward.

Protected calibration, test and background-test AOIs never become training data. A new iteration adds independent training AOIs instead.

Frozen release gates

Area	Gate
Runtime	CUDA required; NVIDIA device visible; no CPU fallback
Corpus	Immutable manifest and artifacts with SHA-256 evidence
Geographic composition	Each land region has at least 15 train, 2 val, 3 calibration, 3 test and 2 background-test AOIs
Contexts	Dense urban, suburban, rural, industrial and difficult negative contexts represented
Leakage	No intersecting AOIs across protected split roles
Label integrity	No malformed rows; sub-resolution labels explicitly rejected
Temporal truth	Unknown per-pixel dates remain unknown; acquisition dates may not masquerade as observation dates
Threshold selection	Calibration set only; maximise the worst regional F1 before aggregate F1
Test aggregate	F1 at least 0.55 at the frozen footprint/detection match IoU 0.25 contract
Dense-tile capacity	Retain up to 1000 detections per tile; the library default of 300 is below observed Belgian urban label density
Regional test	Every region: F1 at least 0.45, precision at least 0.50 and recall at least 0.40
Pure background	Zero detections on every pure-empty tile at the selected threshold
Production	Exact candidate checksum and fail-closed promotion report required
Final review	Human accepts every queued AOI contact sheet after all automated gates pass

These are minimum release gates, not performance targets. Raising a confidence threshold until detections disappear cannot pass because regional recall is a simultaneous gate.

Current gap inventory

The v4 dated corpus closes basic composition, leakage and temporal-identity gaps with 75 independent AOIs and 13,765 accepted building labels. It adds coastal, ribbon-development, farmland, park, forest and additional urban contexts. Its remaining known gaps are:

• dated 2025 imagery is used for Flanders and Brussels and the dated 2024 SPW campaign for Wallonia. Exact flight days remain a later metadata refinement, but all corpus relations are now measured periods rather than download dates;
• GRB/PICC/UrbIS describe ground footprints, whereas visible roofs can remain displaced. The existing detector QA contract therefore uses IoU 0.25; the threshold is frozen and cannot be relaxed per candidate;
• the first loop candidate generalises poorly in Flanders and Wallonia, especially Mechelen, Sint-Niklaas, Leuven, Mons and dense PICC areas;
• sparse hard contexts pass the empty-image test more easily than dense urban recall, so both gates must remain independent;
• building boxes are a valid first detector contract, but footprint-perfect geometry ultimately requires a separately validated segmentation model.

The active production model remains unchanged while any gate fails.

The v31 expansion adds eighteen independent train-only AOIs after the v30 rotated-holdout calibration isolated the remaining domain gaps. Nine Flemish AOIs cover industrial roofs, ribbon development and coastal urban fabric. Nine Walloon AOIs add dense urban, rural-town, regional-architecture and difficult industrial/rail/quarry negatives. Existing validation, calibration, test and background-test AOIs remain frozen. The provisioner can load a short-lived operator session from a mode-0600 token file, allowing governed acquisition on an authenticated runtime without exposing credentials in process arguments.

Reproducible evidence

• corpus assembler: scripts/assemble_belgium_building_corpus.py;
• corpus auditor: scripts/audit_belgium_building_corpus.py;
• tile exporter/auditor/contact sheets: the operator_yolo scripts;
• per-AOI evaluator: scripts/evaluate_belgium_building_candidate.py;
• calibration-only selection and release gates: scripts/assess_belgium_building_training_iteration.py.
• checkpointed CUDA orchestration: scripts/run_belgium_building_training_loop.py.

Every failed assessment returns continue_training_loop. Only a report with training_complete may proceed to final human review and guarded activation. Optimizer, initial learning rate, image size and geometric augmentation are explicit loop inputs. This permits a conservative aerial-imagery finetune (for example AdamW with mosaic disabled) without changing calibration, test or release gates.

After a failed assessment, scripts/build_failure_driven_yolo_sampling.py creates a checksummed, train-only sampling manifest. Positive tiles from regions that fail F1 or recall are repeated, while true negative train tiles are repeated when a regional precision gate or the pure-background gate fails. Calibration, test, background-test and validation AOIs are excluded by their frozen corpus split; the generated evidence records that no protected sample entered training. Per-AOI calibration evidence also identifies failed region/context pairs. Train-only AOIs with the same governed context receive a stronger repeat factor than the remaining failed region, so correction rounds target distinct failure modes without copying a protected AOI into training. The sampling evidence records both context sets and repeat factors. When no matching train context exists, regional sampling remains active and the missing context becomes a concrete input for the next immutable corpus expansion. The checkpointed orchestrator invokes this builder after every rejected iteration, stores its checksum in training-loop-state.json, and uses the resulting dataset YAML for the next checkpoint. A restart resumes both the candidate weights and that exact failure-driven training input. An already completed out-of-band checkpoint enters the same contract with --evaluate-initial-model: the first iteration skips fitting, copies and hashes the checkpoint, and begins at calibration. A rejection then follows the identical failure-driven CUDA path and cannot open protected test evidence early. Tower's v37 supervisor binds the completed results.png artifact to a versioned JSON argv list. The handoff starts the orchestrator detached exactly once; shell strings are not accepted. Subsequent iterations retain the frozen 180-degree aerial rotation, vertical/horizontal flip, scale and translation parameters rather than silently reverting to generic augmentation defaults. Before fitting a state-pending iteration, the orchestrator checks its canonical run directory for weights/last.pt. If present, it resumes that exact CUDA checkpoint instead of restarting from the prior candidate. A host-side parent supervisor monitors the exact loop-process marker and persisted loop state; on container recreation it restores the current orchestrator script and launches the versioned JSON argv command. Invalid state and bounded launch exhaustion fail closed. The orchestrator refuses to start unless every automated frozen-dataset gate passes and the corpus contains zero blank/low-variance positive tiles. The audit status may remain needs_human_review while training and objective evaluation continue: final human sign-off is deliberately the last gate and can never be interpreted as model promotion approval in advance. For dated imagery, GRB BEGINDATUM and PICC DATE_CREAT are compared with the end of the imagery period. A feature created afterward is retained in the audit but excluded from training as created_after_imagery_period. UrbIS does not expose an equivalent feature creation field in this acquisition contract, so its remaining temporal relation stays an explicit sample-level limitation.

An opt-in visible-roof experiment can dissolve source footprints that truly touch or overlap; separated footprints are never bridged. The audit retains every contributing native feature identifier and reports both source-feature and visible-instance counts. This mode is not the default: the Belgium v8 experiment showed that unconditional touching-footprint dissolve can merge whole urban blocks and therefore must pass the same independent gates before it can replace native instances.

The compact-roof variant therefore merges a connected group only when it has at most 12 source footprints and fills at least 55% of its axis-aligned envelope. Larger or irregular connected groups retain their native instances and are marked native_instance_complex_touch_group. These fixed criteria prevent administrative row-house chains from becoming one ambiguous detector box while keeping the experiment deterministic and auditable.