geointel/docs/BELGIUM_BUILDING_TRAINING_LOOP.md

# Belgian building detector: closed training loop

## Meaning of complete

`100% trained` means that every frozen release gate below passes. It does not
mean a fabricated 100% precision, recall or mAP score. A model that memorises a
small test set is not complete.

The loop is:

1. provision new, spatially independent AOIs from governed official services;
2. freeze imagery, labels, metadata and checksums into a new corpus version;
3. reject invalid, duplicate and sub-resolution labels and run spatial-leakage
   checks;
4. train only on the train split with CUDA on the Tower NVIDIA GPU;
5. use validation for early stopping and calibration only for threshold choice;
6. evaluate the fixed threshold once on regional test and background-test data;
7. attribute false positives and false negatives to a region, AOI and context;
8. add new training-only examples for the observed failure modes and repeat;
9. stop only when every objective gate passes; request human review afterward.

Protected calibration, test and background-test AOIs never become training
data. A new iteration adds independent training AOIs instead.

## Frozen release gates

| Area | Gate |
| --- | --- |
| Runtime | CUDA required; NVIDIA device visible; no CPU fallback |
| Corpus | Immutable manifest and artifacts with SHA-256 evidence |
| Geographic composition | Each land region has at least 15 train, 2 val, 3 calibration, 3 test and 2 background-test AOIs |
| Contexts | Dense urban, suburban, rural, industrial and difficult negative contexts represented |
| Leakage | No intersecting AOIs across protected split roles |
| Label integrity | No malformed rows; sub-resolution labels explicitly rejected |
| Temporal truth | Unknown per-pixel dates remain unknown; acquisition dates may not masquerade as observation dates |
| Threshold selection | Calibration set only; maximise the worst regional F1 before aggregate F1 |
| Test aggregate | F1 at least 0.55 at the frozen footprint/detection match IoU 0.25 contract |
| Dense-tile capacity | Retain up to 1000 detections per tile; the library default of 300 is below observed Belgian urban label density |
| Regional test | Every region: F1 at least 0.45, precision at least 0.50 and recall at least 0.40 |
| Pure background | Zero detections on every pure-empty tile at the selected threshold |
| Production | Exact candidate checksum and fail-closed promotion report required |
| Final review | Human accepts every queued AOI contact sheet after all automated gates pass |

These are minimum release gates, not performance targets. Raising a confidence
threshold until detections disappear cannot pass because regional recall is a
simultaneous gate.

## Current gap inventory

The v4 dated corpus closes basic composition, leakage and temporal-identity
gaps with 75 independent AOIs and 13,765 accepted building labels. It adds
coastal, ribbon-development, farmland, park, forest and additional urban
contexts. Its remaining known gaps are:

- dated 2025 imagery is used for Flanders and Brussels and the dated 2024 SPW
  campaign for Wallonia. Exact flight days remain a later metadata refinement,
  but all corpus relations are now measured periods rather than download dates;
- GRB/PICC/UrbIS describe ground footprints, whereas visible roofs can remain
  displaced. The existing detector QA contract therefore uses IoU 0.25; the
  threshold is frozen and cannot be relaxed per candidate;
- the first loop candidate generalises poorly in Flanders and Wallonia,
  especially Mechelen, Sint-Niklaas, Leuven, Mons and dense PICC areas;
- sparse hard contexts pass the empty-image test more easily than dense urban
  recall, so both gates must remain independent;
- building boxes are a valid first detector contract, but footprint-perfect
  geometry ultimately requires a separately validated segmentation model.

The active production model remains unchanged while any gate fails.

## Reproducible evidence

- corpus assembler: `scripts/assemble_belgium_building_corpus.py`;
- corpus auditor: `scripts/audit_belgium_building_corpus.py`;
- tile exporter/auditor/contact sheets: the `operator_yolo` scripts;
- per-AOI evaluator: `scripts/evaluate_belgium_building_candidate.py`;
- calibration-only selection and release gates:
  `scripts/assess_belgium_building_training_iteration.py`.
- checkpointed CUDA orchestration:
  `scripts/run_belgium_building_training_loop.py`.

Every failed assessment returns `continue_training_loop`. Only a report with
`training_complete` may proceed to final human review and guarded activation.
Optimizer, initial learning rate, image size and geometric augmentation are
explicit loop inputs. This permits a conservative aerial-imagery finetune
(for example AdamW with mosaic disabled) without changing calibration, test
or release gates.

After a failed assessment,
`scripts/build_failure_driven_yolo_sampling.py` creates a checksummed,
train-only sampling manifest. Positive tiles from regions that fail F1 or
recall are repeated, while true negative train tiles are repeated when a
regional precision gate or the pure-background gate fails. Calibration, test,
background-test and validation AOIs are excluded by their frozen corpus split;
the generated evidence records that no protected sample entered training.
The checkpointed orchestrator invokes this builder after every rejected
iteration, stores its checksum in `training-loop-state.json`, and uses the
resulting dataset YAML for the next checkpoint. A restart resumes both the
candidate weights and that exact failure-driven training input.
An already completed out-of-band checkpoint enters the same contract with
`--evaluate-initial-model`: the first iteration skips fitting, copies and
hashes the checkpoint, and begins at calibration. A rejection then follows
the identical failure-driven CUDA path and cannot open protected test evidence
early.
Tower's v37 supervisor binds the completed `results.png` artifact to a
versioned JSON argv list. The handoff starts the orchestrator detached exactly
once; shell strings are not accepted. Subsequent iterations retain the frozen
180-degree aerial rotation, vertical/horizontal flip, scale and translation
parameters rather than silently reverting to generic augmentation defaults.
The orchestrator refuses to start unless every automated frozen-dataset gate
passes and the corpus contains zero blank/low-variance positive tiles. The
audit status may remain `needs_human_review` while training and objective
evaluation continue: final human sign-off is deliberately the last gate and
can never be interpreted as model promotion approval in advance.
For dated imagery, GRB `BEGINDATUM` and PICC `DATE_CREAT` are compared with the
end of the imagery period. A feature created afterward is retained in the
audit but excluded from training as `created_after_imagery_period`. UrbIS does
not expose an equivalent feature creation field in this acquisition contract,
so its remaining temporal relation stays an explicit sample-level limitation.

An opt-in visible-roof experiment can dissolve source footprints that truly
touch or overlap; separated footprints are never bridged. The audit retains
every contributing native feature identifier and reports both source-feature
and visible-instance counts. This mode is not the default: the Belgium v8
experiment showed that unconditional touching-footprint dissolve can merge
whole urban blocks and therefore must pass the same independent gates before
it can replace native instances.

The compact-roof variant therefore merges a connected group only when it has
at most 12 source footprints and fills at least 55% of its axis-aligned
envelope. Larger or irregular connected groups retain their native instances
and are marked `native_instance_complex_touch_group`. These fixed criteria
prevent administrative row-house chains from becoming one ambiguous detector
box while keeping the experiment deterministic and auditable.