148 lines
8.7 KiB
Markdown
148 lines
8.7 KiB
Markdown
# Belgian building detector: closed training loop
|
|
|
|
## Meaning of complete
|
|
|
|
`100% trained` means that every frozen release gate below passes. It does not
|
|
mean a fabricated 100% precision, recall or mAP score. A model that memorises a
|
|
small test set is not complete.
|
|
|
|
The loop is:
|
|
|
|
1. provision new, spatially independent AOIs from governed official services;
|
|
2. freeze imagery, labels, metadata and checksums into a new corpus version;
|
|
3. reject invalid, duplicate and sub-resolution labels and run spatial-leakage
|
|
checks;
|
|
4. train only on the train split with CUDA on the Tower NVIDIA GPU;
|
|
5. use validation for early stopping and calibration only for threshold choice;
|
|
6. evaluate the fixed threshold once on regional test and background-test data;
|
|
7. attribute false positives and false negatives to a region, AOI and context;
|
|
8. add new training-only examples for the observed failure modes and repeat;
|
|
9. stop only when every objective gate passes; request human review afterward.
|
|
|
|
Protected calibration, test and background-test AOIs never become training
|
|
data. A new iteration adds independent training AOIs instead.
|
|
|
|
## Frozen release gates
|
|
|
|
| Area | Gate |
|
|
| --- | --- |
|
|
| Runtime | CUDA required; NVIDIA device visible; no CPU fallback |
|
|
| Corpus | Immutable manifest and artifacts with SHA-256 evidence |
|
|
| Geographic composition | Each land region has at least 15 train, 2 val, 3 calibration, 3 test and 2 background-test AOIs |
|
|
| Contexts | Dense urban, suburban, rural, industrial and difficult negative contexts represented |
|
|
| Leakage | No intersecting AOIs across protected split roles |
|
|
| Label integrity | No malformed rows; sub-resolution labels explicitly rejected |
|
|
| Temporal truth | Unknown per-pixel dates remain unknown; acquisition dates may not masquerade as observation dates |
|
|
| Threshold selection | Calibration set only; maximise the worst regional F1 before aggregate F1 |
|
|
| Test aggregate | F1 at least 0.55 at the frozen footprint/detection match IoU 0.25 contract |
|
|
| Dense-tile capacity | Retain up to 1000 detections per tile; the library default of 300 is below observed Belgian urban label density |
|
|
| Regional test | Every region: F1 at least 0.45, precision at least 0.50 and recall at least 0.40 |
|
|
| Pure background | Zero detections on every pure-empty tile at the selected threshold |
|
|
| Production | Exact candidate checksum and fail-closed promotion report required |
|
|
| Final review | Human accepts every queued AOI contact sheet after all automated gates pass |
|
|
|
|
These are minimum release gates, not performance targets. Raising a confidence
|
|
threshold until detections disappear cannot pass because regional recall is a
|
|
simultaneous gate.
|
|
|
|
## Current gap inventory
|
|
|
|
The v4 dated corpus closes basic composition, leakage and temporal-identity
|
|
gaps with 75 independent AOIs and 13,765 accepted building labels. It adds
|
|
coastal, ribbon-development, farmland, park, forest and additional urban
|
|
contexts. Its remaining known gaps are:
|
|
|
|
- dated 2025 imagery is used for Flanders and Brussels and the dated 2024 SPW
|
|
campaign for Wallonia. Exact flight days remain a later metadata refinement,
|
|
but all corpus relations are now measured periods rather than download dates;
|
|
- GRB/PICC/UrbIS describe ground footprints, whereas visible roofs can remain
|
|
displaced. The existing detector QA contract therefore uses IoU 0.25; the
|
|
threshold is frozen and cannot be relaxed per candidate;
|
|
- the first loop candidate generalises poorly in Flanders and Wallonia,
|
|
especially Mechelen, Sint-Niklaas, Leuven, Mons and dense PICC areas;
|
|
- sparse hard contexts pass the empty-image test more easily than dense urban
|
|
recall, so both gates must remain independent;
|
|
- building boxes are a valid first detector contract, but footprint-perfect
|
|
geometry ultimately requires a separately validated segmentation model.
|
|
|
|
The active production model remains unchanged while any gate fails.
|
|
|
|
## Reproducible evidence
|
|
|
|
- corpus assembler: `scripts/assemble_belgium_building_corpus.py`;
|
|
- corpus auditor: `scripts/audit_belgium_building_corpus.py`;
|
|
- tile exporter/auditor/contact sheets: the `operator_yolo` scripts;
|
|
- per-AOI evaluator: `scripts/evaluate_belgium_building_candidate.py`;
|
|
- calibration-only selection and release gates:
|
|
`scripts/assess_belgium_building_training_iteration.py`.
|
|
- checkpointed CUDA orchestration:
|
|
`scripts/run_belgium_building_training_loop.py`.
|
|
|
|
Every failed assessment returns `continue_training_loop`. Only a report with
|
|
`training_complete` may proceed to final human review and guarded activation.
|
|
Optimizer, initial learning rate, image size and geometric augmentation are
|
|
explicit loop inputs. This permits a conservative aerial-imagery finetune
|
|
(for example AdamW with mosaic disabled) without changing calibration, test
|
|
or release gates.
|
|
|
|
After a failed assessment,
|
|
`scripts/build_failure_driven_yolo_sampling.py` creates a checksummed,
|
|
train-only sampling manifest. Positive tiles from regions that fail F1 or
|
|
recall are repeated, while true negative train tiles are repeated when a
|
|
regional precision gate or the pure-background gate fails. Calibration, test,
|
|
background-test and validation AOIs are excluded by their frozen corpus split;
|
|
the generated evidence records that no protected sample entered training.
|
|
Per-AOI calibration evidence also identifies failed region/context pairs.
|
|
Train-only AOIs with the same governed context receive a stronger repeat factor
|
|
than the remaining failed region, so correction rounds target distinct failure
|
|
modes without copying a protected AOI into training. The sampling evidence
|
|
records both context sets and repeat factors. When no matching train context
|
|
exists, regional sampling remains active and the missing context becomes a
|
|
concrete input for the next immutable corpus expansion.
|
|
The checkpointed orchestrator invokes this builder after every rejected
|
|
iteration, stores its checksum in `training-loop-state.json`, and uses the
|
|
resulting dataset YAML for the next checkpoint. A restart resumes both the
|
|
candidate weights and that exact failure-driven training input.
|
|
An already completed out-of-band checkpoint enters the same contract with
|
|
`--evaluate-initial-model`: the first iteration skips fitting, copies and
|
|
hashes the checkpoint, and begins at calibration. A rejection then follows
|
|
the identical failure-driven CUDA path and cannot open protected test evidence
|
|
early.
|
|
Tower's v37 supervisor binds the completed `results.png` artifact to a
|
|
versioned JSON argv list. The handoff starts the orchestrator detached exactly
|
|
once; shell strings are not accepted. Subsequent iterations retain the frozen
|
|
180-degree aerial rotation, vertical/horizontal flip, scale and translation
|
|
parameters rather than silently reverting to generic augmentation defaults.
|
|
Before fitting a state-pending iteration, the orchestrator checks its canonical
|
|
run directory for `weights/last.pt`. If present, it resumes that exact CUDA
|
|
checkpoint instead of restarting from the prior candidate. A host-side parent
|
|
supervisor monitors the exact loop-process marker and persisted loop state; on
|
|
container recreation it restores the current orchestrator script and launches
|
|
the versioned JSON argv command. Invalid state and bounded launch exhaustion
|
|
fail closed.
|
|
The orchestrator refuses to start unless every automated frozen-dataset gate
|
|
passes and the corpus contains zero blank/low-variance positive tiles. The
|
|
audit status may remain `needs_human_review` while training and objective
|
|
evaluation continue: final human sign-off is deliberately the last gate and
|
|
can never be interpreted as model promotion approval in advance.
|
|
For dated imagery, GRB `BEGINDATUM` and PICC `DATE_CREAT` are compared with the
|
|
end of the imagery period. A feature created afterward is retained in the
|
|
audit but excluded from training as `created_after_imagery_period`. UrbIS does
|
|
not expose an equivalent feature creation field in this acquisition contract,
|
|
so its remaining temporal relation stays an explicit sample-level limitation.
|
|
|
|
An opt-in visible-roof experiment can dissolve source footprints that truly
|
|
touch or overlap; separated footprints are never bridged. The audit retains
|
|
every contributing native feature identifier and reports both source-feature
|
|
and visible-instance counts. This mode is not the default: the Belgium v8
|
|
experiment showed that unconditional touching-footprint dissolve can merge
|
|
whole urban blocks and therefore must pass the same independent gates before
|
|
it can replace native instances.
|
|
|
|
The compact-roof variant therefore merges a connected group only when it has
|
|
at most 12 source footprints and fills at least 55% of its axis-aligned
|
|
envelope. Larger or irregular connected groups retain their native instances
|
|
and are marked `native_instance_complex_touch_group`. These fixed criteria
|
|
prevent administrative row-house chains from becoming one ambiguous detector
|
|
box while keeping the experiment deterministic and auditable.
|