diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index 3ad32b3e..460832a8 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -1,3 +1,9 @@ +## 2026-07-26 - Complete Belgium PyTorch training roadmap + +- Added `docs/PYTORCH_TRAINING_ROADMAP_BELGIUM.md` as the executable programme board. +- Defined the target model portfolio, national dataset strata, authority bindings, spatially isolated train/validation/calibration/test splits, nine work packages, promotion/rollback gates, GPU scheduling, artefact layout and the evidence-based definition of `fully trained`. +- Kept official GIS measurements outside neural training and made the current critical path explicit: regional orthophoto acquisition, label normalization and a frozen Belgium building corpus precede further full training. + ## 2026-07-26 - PyTorch programme clarification and training hardening - Clarified that PyTorch governs trainable imagery models and does not replace authoritative terrain, flood, land-use, road, water or change analyses. diff --git a/docs/PYTORCH_MODEL_PROGRAM.md b/docs/PYTORCH_MODEL_PROGRAM.md index 7b7f6d56..8f8e270f 100644 --- a/docs/PYTORCH_MODEL_PROGRAM.md +++ b/docs/PYTORCH_MODEL_PROGRAM.md @@ -1,5 +1,8 @@ # PyTorch model programme +The complete execution plan, work packages and release gates are maintained in +`docs/PYTORCH_TRAINING_ROADMAP_BELGIUM.md`. + ## Principle PyTorch is the governed CUDA runtime for trainable image models. It is not a diff --git a/docs/PYTORCH_TRAINING_ROADMAP_BELGIUM.md b/docs/PYTORCH_TRAINING_ROADMAP_BELGIUM.md new file mode 100644 index 00000000..7ccd63a9 --- /dev/null +++ b/docs/PYTORCH_TRAINING_ROADMAP_BELGIUM.md @@ -0,0 +1,318 @@ +# GeoIntel PyTorch training roadmap — Belgium + +Status: active programme +Owner: AI pipeline, with GIS and QA gates +Runtime: Tower NVIDIA GeForce RTX 4080 SUPER, PyTorch CUDA +Companion contract: `docs/PYTORCH_MODEL_PROGRAM.md` + +## 1. Programme outcome + +GeoIntel has a small portfolio of task-specific image models, each constrained +to proven classes, input imagery, geography and acquisition periods. A model is +called **fully trained** only when its dataset, split isolation, label review, +CUDA training, independent evaluation, calibration, deployment and monitoring +gates all pass. Completing an epoch count is not completion. + +The programme does not train neural replacements for official GIS sources. +Road length, water area, land use, terrain, flood depth, bathymetry and +source-to-source change remain deterministic analyses. + +## 2. Target portfolio + +| Priority | Model contract | Output | Initial geographic claim | Current state | +|---|---|---|---|---| +| P0 | `building-detect-be` | building bounding boxes | Belgium land, per validated region | active model is building-only and Mol/Kempen-scoped | +| P1 | `building-segment-be` | building footprint masks | no claim until mask QA passes | `not_configured` | +| P2 | `solar-panel-detect-be` | photovoltaic installation boxes/polygons | no claim until reviewed labels exist | unvalidated asset; `not_configured` | + +Additional classes require a written use case, a distinct label contract and +the same intake gates. They are not appended opportunistically to an existing +model. + +## 3. Non-model capabilities + +The following stay outside PyTorch unless a later RFC demonstrates a material +benefit over the authoritative source: + +- roads, parcels, waterways and building footprints from GRB/PICC/UrbIS; +- land-cover and land-use classes from governed thematic rasters; +- elevation, slope, flood, bathymetry and raster indices; +- vector/raster temporal comparison and QA/QC metrics; +- the grounded assistant, which remains an Ollama language-model integration. + +## 4. Definition of fully trained + +A model/version is fully trained for one declared scope only when: + +1. its immutable dataset manifest identifies every image, label source, + observation date, CRS, licence and checksum; +2. train, validation, calibration and test AOIs are spatially disjoint, including + a buffer larger than the maximum tile context; +3. automated label checks pass and sampled labels have completed human review; +4. at least one reproducible CUDA training run and one reproducibility rerun + produce checksum-bound evidence; +5. evaluation covers Flanders, Wallonia and Brussels separately, plus urban, + suburban, rural, industrial and pure-background contexts where applicable; +6. it beats the incumbent on the frozen portfolio without a material regional + or background regression; +7. confidence threshold, tiling and duplicate suppression are calibrated on a + calibration split, never on the final test split; +8. the model card, limitation text, rollback asset and production preflight are + complete; +9. a post-deployment shadow run confirms the offline results before scope is + widened. + +## 5. Dataset topology + +### Geographic strata + +- Flanders: dense urban, ribbon development, villages, farms, industry and + forest/heath negatives. +- Wallonia: Liège/Charleroi urban fabric, villages, dispersed rural buildings, + industrial valleys, quarries and forest negatives. +- Brussels: dense blocks, detached residential, industrial/logistics sites, + parks and rail contexts. +- Boundary controls: language boundary and cross-provider seams. +- Hold-outs: complete municipalities or non-overlapping large AOIs never used + for training or hyperparameter selection. + +### Source bindings + +| Region | Imagery | Building reference | Required evidence | +|---|---|---|---| +| Flanders | governed Digitaal Vlaanderen orthophoto | GRB GBG | product edition and acquisition mismatch report | +| Wallonia | governed SPW orthophoto | SPW PICC buildings | WMS/catalog identity, CC BY attribution, campaign and PICC snapshot | +| Brussels | governed Paradigm/UrbIS orthophoto | UrbIS buildings | WMS layer identity, licence, edition and vector snapshot | + +### Split contract + +- `train`: model fitting only; +- `val`: epoch/model selection only; +- `calibration`: threshold, tile size, overlap and NMS tuning only; +- `test`: opened once per candidate family for the promotion decision; +- `background-test`: pure-empty and difficult contextual negatives; +- no source image, overlapping tile, building instance or buffered AOI may cross + split boundaries. + +## 6. Work packages + +### WP0 — Governance and frozen contracts + +- [x] distinguish PyTorch tasks from deterministic GIS functions; +- [x] require CUDA in production training and record runtime evidence; +- [x] parameterize single-class export by class/source/layer; +- [ ] assign stable model IDs, semantic versions and model-card schema; +- [ ] replace weak historical promotion defaults with task-specific approved + gates while preserving old reports as historical evidence. + +Exit: schema tests pass and no runtime can advertise an unproven class/scope. + +### WP1 — National imagery acquisition + +- [x] verify official SPW and Brussels imagery services and usage conditions; +- [ ] implement bounded SPW orthophoto acquisition with the existing AOI worker; +- [ ] implement bounded UrbIS orthophoto acquisition with the existing AOI worker; +- [ ] expose provider edition, resolution, time, CRS, attribution and checksum; +- [ ] add retry/cache/size limits and live provider preflights; +- [ ] retain explicit `partial`/`not_configured` status when a service is down. + +Exit: one checksum-bound RGB GeoTIFF per target stratum can be acquired without +manual URL editing and without crossing provider coverage zones. + +### WP2 — Reference-label normalization + +- [ ] normalize GRB, PICC and UrbIS building semantics to canonical `building`; +- [ ] retain original feature IDs and source classes; +- [ ] quantify invalid geometry repairs, duplicates and temporal mismatches; +- [ ] define exclusion rules for ruins, canopies, underground structures and + objects below the resolvable pixel size; +- [ ] generate overlay/contact-sheet review packs. + +Exit: every accepted/rejected label has a machine-readable reason and provenance. + +### WP3 — Frozen Belgium building corpus + +- [ ] select and persist the geographic strata and split assignments; +- [ ] acquire imagery and references through governed services; +- [ ] export tiles at candidate resolutions with deterministic negative sampling; +- [ ] run leakage, checksum, class, label-size and visual-variance audits; +- [ ] complete representative human label review; +- [ ] freeze dataset version `building-be-v1` and make it immutable. + +Minimum composition is set by coverage, not one global tile count: every region +and context stratum must contain positive instances and appropriate negatives. +Any sparse stratum blocks the national claim. + +Exit: corpus audit passes with zero leakage and no unreviewed critical findings. + +### WP4 — Building detector training + +- [ ] define a bounded experiment matrix: architecture size, initialization, + image size and augmentation policy; +- [ ] train all planned candidates on `cuda:0` with deterministic seeds where + supported, early stopping and immutable summaries; +- [ ] rerun the best configuration from a clean run directory; +- [ ] retain training curves, environment, dataset/model hashes and peak VRAM; +- [ ] do not overwrite the active production asset. + +Initial matrix: + +| Family | Initialization | Image sizes | Purpose | +|---|---|---|---| +| YOLO small | local generic `yolov8s.pt` | 640, 960 | reproducible baseline | +| Active-model fine-tune | current promoted building model | 640 | controlled national-domain extension | +| Larger challenger | locally approved medium model, if VRAM preflight passes | 640 | accuracy/latency trade-off | + +Exit: all candidates have valid artifacts; failed/OOM runs remain recorded and +cannot be selected silently. + +### WP5 — Independent evaluation and calibration + +- [ ] run fixed candidate inference on frozen validation/calibration/test sets; +- [ ] calculate precision, recall, F1, AP50/AP50-95 and spatial QA at multiple + object sizes; +- [ ] report each region/context independently and macro-average them; +- [ ] audit false positives and false negatives with map/contact-sheet evidence; +- [ ] tune confidence/tile/overlap only on calibration data; +- [ ] compare against the active model with paired AOI-level results. + +Provisional promotion gates (to be frozen before the test set is opened): + +- zero detections in approved pure-empty background AOIs; +- no region below its approved minimum F1; +- macro precision and recall both meet the approved floor; +- candidate improves the primary macro metric or offers a documented operational + advantage without statistically/materially worse regional performance; +- inference fits RTX 4080 SUPER VRAM and bounded runtime limits; +- no unresolved high-severity visual-review findings. + +Numeric floors are frozen from reviewed baseline distributions before final +testing; they are not relaxed after seeing candidate test results. + +### WP6 — Promotion and deployment + +- [ ] generate model card, promotion report and operator decision; +- [ ] copy, never overwrite, the immutable promoted `.pt` asset; +- [ ] configure exact class list and validated geographic scopes; +- [ ] rebuild the immutable image/runtime configuration; +- [ ] verify Docker GPU exposure, CUDA preflight and bounded live inference; +- [ ] run shadow comparison before widening the public scope; +- [ ] preserve one-command rollback to the previous asset/configuration. + +Exit: API/UI show the exact supported class, regions and limitations derived +from evidence, not from file presence. + +### WP7 — Building segmentation + +- [ ] decide whether official footprint polygons and imagery alignment support a + trustworthy mask corpus; +- [ ] implement polygon-to-mask export with edge/occlusion rules; +- [ ] train a dedicated segmentation model; do not reuse detection boxes as masks; +- [ ] evaluate mask IoU, boundary quality, area bias and topology; +- [ ] keep `not_configured` unless segmentation-specific gates pass. + +Exit: georeferenced persisted masks/polygons pass segmentation QA and export +contracts on independent regional samples. + +### WP8 — Solar-panel intake + +- [ ] write the user-facing analytical use case and canonical class ontology; +- [ ] secure authoritative or reviewed annotations with explicit licensing; +- [ ] create roof, industrial, ground-mounted and hard-negative strata; +- [ ] benchmark the existing local external model as a challenger only; +- [ ] train a Belgium-specific detector/segmenter if the data supports it; +- [ ] apply the same WP3–WP6 promotion path. + +Exit: solar capability remains `not_configured` until its own evidence passes; +building evidence cannot promote it. + +### WP9 — Monitoring and controlled learning loop + +- [ ] capture inference/QA drift by region, imagery edition and object size; +- [ ] maintain operator review queues without feeding decisions directly into + training labels; +- [ ] require reviewed label releases for every retraining cycle; +- [ ] retrain on a schedule only when drift/data evidence justifies it; +- [ ] retain previous datasets, models and promotion reports for rollback/audit. + +Exit: no automatic self-training or silent production promotion exists. + +## 7. Verification matrix + +Each model release must pass: + +- unit tests for transforms, labels, splits, hashes and class contracts; +- integration tests for persisted Dataset/Job/AnalysisRun/Detection or + Segmentation records; +- geospatial tests for CRS, tile transforms, clipping and duplicate suppression; +- GPU tests for Docker DeviceRequest, CUDA availability and fail-closed behavior; +- offline evaluation on frozen positive/background portfolios; +- live bounded smoke on Tower; +- frontend loading, progress, empty, error, limitation and export states; +- upgrade, restart, rollback and corrupted/missing-model failure tests. + +## 8. GPU execution policy + +- production training device: `cuda:0`; +- required flag: `TRAIN_REQUIRE_CUDA=true`; +- inference remains available only within the active validated scope while a + challenger trains; +- schedule long training outside operational validation windows; +- preflight free VRAM and choose batch size conservatively; +- never run two unbounded training jobs concurrently; +- persist partial-run evidence on interruption and restart as a new run unless + checkpoint resume is explicitly checksum-validated. + +## 9. Artefact layout + +```text +/app/storage/operator-data/model-program/// + source-manifest.json + split-manifest.json + dataset.yaml + dataset-audit.json + visual-review/ + +/app/storage/training/// + args.yaml + results.csv + weights/best.pt + training_summary.json + +/app/storage/operator-data/model-review/// + positive-portfolio/ + background-portfolio/ + calibration/ + promotion-report.json + model-card.json +``` + +Large imagery, labels and weights remain outside Git. Code, schemas, fixture +contracts and documentation remain versioned in the repository. + +## 10. Delivery sequence + +1. WP0 contract freeze. +2. WP1 regional imagery providers. +3. WP2 source normalization. +4. WP3 frozen Belgium building corpus. +5. WP4–WP5 training and independent evaluation. +6. WP6 building promotion and controlled scope widening. +7. WP7 segmentation only after the detector/data foundation is stable. +8. WP8 solar only after its label intake is approved. +9. WP9 continuously after the first national model release. + +WP1–WP3 are the current critical path. More epochs on the existing Kempen-only +corpus cannot satisfy the Belgian claim and are therefore not the next action. + +## 11. Programme completion + +The programme is complete when every target portfolio row is either: + +- production-configured with immutable, independently verified evidence for its + declared classes and geography; or +- explicitly `not_configured` with the missing data/evidence named. + +“Everything works” never means one universal model. It means every GeoIntel +capability uses the correct deterministic source or a task-specific, validated +model, and the platform communicates that distinction accurately. diff --git a/docs/TODO.md b/docs/TODO.md index d0276f7e..a3aa800e 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -2,6 +2,8 @@ ## PyTorch-modelprogramma +Uitvoeringsbord: `docs/PYTORCH_TRAINING_ROADMAP_BELGIUM.md`. + - [x] Scheid trainbare beeldtaken van deterministische GIS-analyses. - [x] Generaliseer de single-class tile-export op klasse, referentiebron en referentielaag. - [x] Laat productietraining fail-closed stoppen wanneer vereiste CUDA ontbreekt.