geointel/docs/PYTORCH_TRAINING_ROADMAP_BELGIUM.md

GeoIntel PyTorch training roadmap — Belgium

Status: active programme
Owner: AI pipeline, with GIS and QA gates
Runtime: Tower NVIDIA GeForce RTX 4080 SUPER, PyTorch CUDA
Companion contract: docs/PYTORCH_MODEL_PROGRAM.md

1. Programme outcome

GeoIntel has a small portfolio of task-specific image models, each constrained to proven classes, input imagery, geography and acquisition periods. A model is called fully trained only when its dataset, split isolation, label review, CUDA training, independent evaluation, calibration, deployment and monitoring gates all pass. Completing an epoch count is not completion.

The programme does not train neural replacements for official GIS sources. Road length, water area, land use, terrain, flood depth, bathymetry and source-to-source change remain deterministic analyses.

2. Target portfolio

Priority	Model contract	Output	Initial geographic claim	Current state
P0	building-detect-be	building bounding boxes	Belgium land, per validated region	active model is building-only and Mol/Kempen-scoped
P1	building-segment-be	building footprint masks	no claim until mask QA passes	not_configured
P2	solar-panel-detect-be	photovoltaic installation boxes/polygons	no claim until reviewed labels exist	unvalidated asset; not_configured

Additional classes require a written use case, a distinct label contract and the same intake gates. They are not appended opportunistically to an existing model.

3. Non-model capabilities

The following stay outside PyTorch unless a later RFC demonstrates a material benefit over the authoritative source:

• roads, parcels, waterways and building footprints from GRB/PICC/UrbIS;
• land-cover and land-use classes from governed thematic rasters;
• elevation, slope, flood, bathymetry and raster indices;
• vector/raster temporal comparison and QA/QC metrics;
• the grounded assistant, which remains an Ollama language-model integration.

4. Definition of fully trained

A model/version is fully trained for one declared scope only when:

1. its immutable dataset manifest identifies every image, label source, observation date, CRS, licence and checksum;
2. train, validation, calibration and test AOIs are spatially disjoint, including a buffer larger than the maximum tile context;
3. automated label checks pass and sampled labels have completed human review;
4. at least one reproducible CUDA training run and one reproducibility rerun produce checksum-bound evidence;
5. evaluation covers Flanders, Wallonia and Brussels separately, plus urban, suburban, rural, industrial and pure-background contexts where applicable;
6. it beats the incumbent on the frozen portfolio without a material regional or background regression;
7. confidence threshold, tiling and duplicate suppression are calibrated on a calibration split, never on the final test split;
8. the model card, limitation text, rollback asset and production preflight are complete;
9. a post-deployment shadow run confirms the offline results before scope is widened.

5. Dataset topology

Geographic strata

• Flanders: dense urban, ribbon development, villages, farms, industry and forest/heath negatives.
• Wallonia: Liège/Charleroi urban fabric, villages, dispersed rural buildings, industrial valleys, quarries and forest negatives.
• Brussels: dense blocks, detached residential, industrial/logistics sites, parks and rail contexts.
• Boundary controls: language boundary and cross-provider seams.
• Hold-outs: complete municipalities or non-overlapping large AOIs never used for training or hyperparameter selection.

Source bindings

Region	Imagery	Building reference	Required evidence
Flanders	governed Digitaal Vlaanderen orthophoto	GRB GBG	product edition and acquisition mismatch report
Wallonia	governed SPW orthophoto	SPW PICC buildings	WMS/catalog identity, CC BY attribution, campaign and PICC snapshot
Brussels	governed Paradigm/UrbIS orthophoto	UrbIS buildings	WMS layer identity, licence, edition and vector snapshot

Split contract

• train: model fitting only;
• val: epoch/model selection only;
• calibration: threshold, tile size, overlap and NMS tuning only;
• test: opened once per candidate family for the promotion decision;
• background-test: pure-empty and difficult contextual negatives;
• no source image, overlapping tile, building instance or buffered AOI may cross split boundaries.

6. Work packages

WP0 — Governance and frozen contracts

• distinguish PyTorch tasks from deterministic GIS functions;
• require CUDA in production training and record runtime evidence;
• parameterize single-class export by class/source/layer;
• assign stable model IDs, semantic versions and model-card schema;
• replace weak historical promotion defaults with task-specific approved gates while preserving old reports as historical evidence.

Exit: schema tests pass and no runtime can advertise an unproven class/scope.

WP1 — National imagery acquisition

• verify official SPW and Brussels imagery services and usage conditions;
• implement bounded SPW orthophoto acquisition with the existing AOI worker;
• implement bounded UrbIS orthophoto acquisition with the existing AOI worker;
• expose provider edition, resolution, time, CRS, attribution and checksum;
• reuse retry/cache/size limits and verify both live WMS providers;
• retain explicit partial/not_configured status when a service is down.

Exit: one checksum-bound RGB GeoTIFF per target stratum can be acquired without manual URL editing and without crossing provider coverage zones.

WP2 — Reference-label normalization

• normalize GRB, PICC and UrbIS building semantics to canonical building;
• retain original feature IDs and source classes;
• quantify invalid geometry repairs, duplicates and temporal mismatches;
• define exclusion rules for ruins, canopies, underground structures and objects below the resolvable pixel size;
• generate overlay/contact-sheet review packs.

Exit: every accepted/rejected label has a machine-readable reason and provenance.

WP3 — Frozen Belgium building corpus

• select and persist the geographic strata and split assignments;
• acquire imagery and references through governed services;
• export tiles at candidate resolutions with deterministic negative sampling;
• run leakage, checksum, class, label-size and visual-variance audits;
• complete representative human label review;
• freeze dataset version building-be-v1 and make it immutable.

Minimum composition is set by coverage, not one global tile count: every region and context stratum must contain positive instances and appropriate negatives. Any sparse stratum blocks the national claim.

Exit: corpus audit passes with zero leakage and no unreviewed critical findings.

WP4 — Building detector training

• define a bounded experiment matrix: architecture size, initialization, image size and augmentation policy;
• train all planned candidates on cuda:0 with deterministic seeds where supported, early stopping and immutable summaries;
• rerun the best configuration from a clean run directory;
• retain training curves, environment, dataset/model hashes and peak VRAM;
• do not overwrite the active production asset.

Initial matrix:

Family	Initialization	Image sizes	Purpose
YOLO small	local generic yolov8s.pt	640, 960	reproducible baseline
Active-model fine-tune	current promoted building model	640	controlled national-domain extension
Larger challenger	locally approved medium model, if VRAM preflight passes	640	accuracy/latency trade-off

Exit: all candidates have valid artifacts; failed/OOM runs remain recorded and cannot be selected silently.

WP5 — Independent evaluation and calibration

• run fixed candidate inference on frozen validation/calibration/test sets;
• calculate precision, recall, F1, AP50/AP50-95 and spatial QA at multiple object sizes;
• report each region/context independently and macro-average them;
• audit false positives and false negatives with map/contact-sheet evidence;
• tune confidence/tile/overlap only on calibration data;
• compare against the active model with paired AOI-level results.

Provisional promotion gates (to be frozen before the test set is opened):

• zero detections in approved pure-empty background AOIs;
• no region below its approved minimum F1;
• macro precision and recall both meet the approved floor;
• candidate improves the primary macro metric or offers a documented operational advantage without statistically/materially worse regional performance;
• inference fits RTX 4080 SUPER VRAM and bounded runtime limits;
• no unresolved high-severity visual-review findings.

Numeric floors are frozen from reviewed baseline distributions before final testing; they are not relaxed after seeing candidate test results.

WP6 — Promotion and deployment

• generate model card, promotion report and operator decision;
• copy, never overwrite, the immutable promoted .pt asset;
• configure exact class list and validated geographic scopes;
• rebuild the immutable image/runtime configuration;
• verify Docker GPU exposure, CUDA preflight and bounded live inference;
• run shadow comparison before widening the public scope;
• preserve one-command rollback to the previous asset/configuration.

Exit: API/UI show the exact supported class, regions and limitations derived from evidence, not from file presence.

WP7 — Building segmentation

• decide whether official footprint polygons and imagery alignment support a trustworthy mask corpus;
• implement polygon-to-mask export with edge/occlusion rules;
• train a dedicated segmentation model; do not reuse detection boxes as masks;
• evaluate mask IoU, boundary quality, area bias and topology;
• keep not_configured unless segmentation-specific gates pass.

Exit: georeferenced persisted masks/polygons pass segmentation QA and export contracts on independent regional samples.

WP8 — Solar-panel intake

• write the user-facing analytical use case and canonical class ontology;
• secure authoritative or reviewed annotations with explicit licensing;
• create roof, industrial, ground-mounted and hard-negative strata;
• benchmark the existing local external model as a challenger only;
• train a Belgium-specific detector/segmenter if the data supports it;
• apply the same WP3–WP6 promotion path.

Exit: solar capability remains not_configured until its own evidence passes; building evidence cannot promote it.

WP9 — Monitoring and controlled learning loop

• capture inference/QA drift by region, imagery edition and object size;
• maintain operator review queues without feeding decisions directly into training labels;
• require reviewed label releases for every retraining cycle;
• retrain on a schedule only when drift/data evidence justifies it;
• retain previous datasets, models and promotion reports for rollback/audit.

Exit: no automatic self-training or silent production promotion exists.

7. Verification matrix

Each model release must pass:

• unit tests for transforms, labels, splits, hashes and class contracts;
• integration tests for persisted Dataset/Job/AnalysisRun/Detection or Segmentation records;
• geospatial tests for CRS, tile transforms, clipping and duplicate suppression;
• GPU tests for Docker DeviceRequest, CUDA availability and fail-closed behavior;
• offline evaluation on frozen positive/background portfolios;
• live bounded smoke on Tower;
• frontend loading, progress, empty, error, limitation and export states;
• upgrade, restart, rollback and corrupted/missing-model failure tests.

8. GPU execution policy

• production training device: cuda:0;
• required flag: TRAIN_REQUIRE_CUDA=true;
• inference remains available only within the active validated scope while a challenger trains;
• schedule long training outside operational validation windows;
• preflight free VRAM and choose batch size conservatively;
• never run two unbounded training jobs concurrently;
• persist partial-run evidence on interruption and restart as a new run unless checkpoint resume is explicitly checksum-validated.

9. Artefact layout

/app/storage/operator-data/model-program/<task>/<dataset-version>/
  source-manifest.json
  split-manifest.json
  dataset.yaml
  dataset-audit.json
  visual-review/

/app/storage/training/<task>/<run-id>/
  args.yaml
  results.csv
  weights/best.pt
  training_summary.json

/app/storage/operator-data/model-review/<task>/<candidate-version>/
  positive-portfolio/
  background-portfolio/
  calibration/
  promotion-report.json
  model-card.json

Large imagery, labels and weights remain outside Git. Code, schemas, fixture contracts and documentation remain versioned in the repository.

10. Delivery sequence

1. WP0 contract freeze.
2. WP1 regional imagery providers.
3. WP2 source normalization.
4. WP3 frozen Belgium building corpus.
5. WP4–WP5 training and independent evaluation.
6. WP6 building promotion and controlled scope widening.
7. WP7 segmentation only after the detector/data foundation is stable.
8. WP8 solar only after its label intake is approved.
9. WP9 continuously after the first national model release.

WP1–WP3 are the current critical path. More epochs on the existing Kempen-only corpus cannot satisfy the Belgian claim and are therefore not the next action.

11. Programme completion

The programme is complete when every target portfolio row is either:

• production-configured with immutable, independently verified evidence for its declared classes and geography; or
• explicitly not_configured with the missing data/evidence named.

“Everything works” never means one universal model. It means every GeoIntel capability uses the correct deterministic source or a task-specific, validated model, and the platform communicates that distinction accurately.