Federate official Belgium data sources
GeoIntel release gates / Compile, test, contracts and builds (push) Canceled after 0s
GeoIntel release gates / Python and npm vulnerability policy (push) Canceled after 0s
GeoIntel release gates / GIS image, SBOM and container scan (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-19 01:34:39 +02:00
parent 33bcd0f3bd
commit 50897c3473
37 changed files with 2179 additions and 186 deletions
+28 -7
View File
@@ -2264,13 +2264,21 @@ return an empty, honest result. No provider request occurs during analysis.
Future provider output continues to use DatasetService and, for vectors,
VectorFeatureService. Arbitrary service URLs, browser-side fetches, insecure
TLS bypasses and startup downloads remain forbidden.
## Governed official nature and soil acquisition
## Governed regional official-vector acquisition
### GET `/api/v1/projects/{project_id}/datasets/official-vector/products`
Returns the fixed official vector registry in the canonical `{ "data": ... }`
envelope. The allowlist contains `bwk_natura2000_2025` and `dov_soil_types`;
arbitrary collection names or URLs are never accepted.
envelope. Every item includes its theme, geometry contract, authority,
licence/attribution, provider-native collection, query mode and
`coverage_zones`. The fixed allowlist contains:
- Flanders: `bwk_natura2000_2025` and `dov_soil_types`;
- Wallonia: `spw_picc_buildings`, `spw_picc_roads`,
`spw_picc_waterways` and `spw_picc_water_surfaces`;
- Brussels: `urbis_buildings` and `urbis_cadastral_parcels`.
Arbitrary collection names or URLs are never accepted.
### POST `/api/v1/projects/{project_id}/datasets/official-vector/acquire`
@@ -2293,16 +2301,29 @@ Request:
The synchronous `vector.official.acquire` Job validates the metric request
size, intersects `bbox` with the persisted Area, retrieves every bounded page,
clips polygon geometry in EPSG:31370 and persists EPSG:4326 features through
`DatasetService`. A repeated exact request can reuse the 24-hour cache.
Provider errors, unstable or incomplete WFS pagination and safety-limit violations
fail without persisting a truncated Dataset.
clips source geometry in the provider-native metric CRS and persists
EPSG:4326 features through `DatasetService`. A repeated exact request can
reuse the 24-hour cache. Provider errors, unstable/incomplete pagination and
safety-limit violations fail without persisting a truncated Dataset.
Walloon and Brussels products additionally require a persisted exact regional
coverage Area (`Wallonia` or `Brussels-Capital Region`). SPW/PICC uses stable
`OBJECTID` ArcGIS REST paging; UrbIS uses WFS 2.0 paging ordered by
`INSPIRE_ID`. A caller cannot make a Flemish product cover Wallonia, or merge
different regional authorities into one Dataset.
`bwk_natura2000_2025` preserves BWK `EVAL`, `EENH*`, `HAB*` and `PHAB*`
semantics. `dov_soil_types` preserves mapped soil, texture, drainage, profile
and substrate classes and is dated as the 1949-1971 survey period. Neither
contract accepts a caller-supplied endpoint.
PICC building footprints, road axes and hydrographic axes/surfaces retain
their source identifiers and report source-appropriate counts, footprint/
surface hectares or line kilometres. UrbIS building footprints and cadastral
parcels remain separate products with their own Paradigm/FPS Finance licence
notes. These products do not claim semantic parity with GRB and do not expose
water volume.
## Governed Landgebruik Vlaanderen forest and agriculture
The existing
+23
View File
@@ -1,5 +1,28 @@
## Autonomous RC program for Belgium and the Belgian North Sea (2026-07-17)
### Post-RC national data federation (2026-07-19)
- Added explicit administrative and maritime themes for persisted NGI/RBINS
reference data and made readiness depend on analyzable themes rather than a
raw Dataset count.
- Generalized the governed Statbel population scripts to the Belgium boundary
and complete municipality inventory while retaining all release-promotion
safety gates.
- Added allowlisted bounded SPW/PICC ArcGIS REST and UrbIS WFS adapters with
source-specific coverage, CRS, identity, paging, geometry, licence,
provenance and metric contracts. Output still enters PostGIS only through
`DatasetService.import_vector_bytes`.
- Made the frontend resolve actual coverage zones before choosing GRB,
SPW/PICC or UrbIS and retain separate source results for mixed-zone
selections.
- Added SPW/UrbIS runtime controls to both Compose variants and the editable
Unraid/Dockerman deployment path.
- Rechecked MDK bathymetry. Strict hostname validation still fails because the
configured official hostname presents a `*.l27powered.eu` certificate, so
acquisition remains truthfully disabled.
- Focused backend and frontend suites passed before full release validation;
full local and Tower evidence follows in the final P5 gate.
### RC-4 national and maritime coverage foundation
- Added an independent coverage registry so the exact Sprint 7
+35 -7
View File
@@ -751,9 +751,37 @@ inferred from these plan zones.
The coverage resolver reports a theme `operational` only when a matching
`ready` Dataset exists in the selected project. Implemented acquisition
without project materialization is `partial`; Walloon, Brussels, Statbel and
MDK families remain `not_configured` until their own bounded adapters and
source-specific metric contracts are implemented.
without project materialization is `partial`. Statbel is an operator-managed
national source. SPW/PICC and UrbIS are bounded API sources for their explicit
themes and zones. MDK remains `not_configured` while its strict-TLS readiness
probe fails.
The governed Statbel operator supports `belgium` as a first-class scope. It
uses the retained NGI Belgium boundary, retains every statistical-sector row
and requires the existing plan -> stage -> named review -> checksum-confirmed
apply process. Flanders, Wallonia, Brussels and bounded municipality
population queries therefore derive from one reviewed national edition rather
than mutually incompatible regional imports. No edition is downloaded or
promoted during application startup.
## Wallonia PICC and Brussels UrbIS
The official-vector registry exposes bounded, source-specific regional
baselines:
- SPW/PICC MapServer layers 11 (buildings), 21 (road axes), 28
(hydrographic axes) and 30 (hydrographic surfaces) for Wallonia;
- Paradigm UrbIS `Buildings` and `CadastralParcels` WFS collections for the
Brussels-Capital Region.
SPW queries use stable `OBJECTID` paging and request EPSG:4326 output while
retaining EPSG:3812 as the native source CRS. UrbIS responses use EPSG:31370,
stable `INSPIRE_ID` paging and are transformed only after exact regional
clipping. Both adapters enforce bbox dimensions, page/feature/response limits,
strict HTTPS host/path validation, geometry contracts and persisted coverage
Areas. They persist only through `DatasetService`; the browser never contacts
either provider directly. Cross-zone selections remain split by authority and
metric semantics.
## Bathymetry, inland profiles and maritime scope
@@ -768,10 +796,10 @@ The following sources are audited but not yet operational:
- MDK Dieptemodel Belgisch Continentaal Plat/Noordzee: 20 x 20 m continuous
raster in LAT, exposed through WCS/WMTS. GeoIntel now has a strict-TLS,
read-only GetCapabilities probe. On 2026-07-17 the official metadata
endpoint presented a certificate for another hostname and returned no
usable capabilities path; status therefore remains `tls_error` and raster
acquisition is disabled.
read-only GetCapabilities probe. Revalidation on 2026-07-19 still found that
`bathy.agentschapmdk.be` presents a certificate for `*.l27powered.eu`.
Hostname validation therefore fails with `tls_error`; raster acquisition
remains disabled and TLS verification cannot be bypassed.
- SPW bathymetry of navigable waterways and reservoir lakes: 0.5 m bed
elevation and XYZ data in mDNG.
- Port of Antwerp-Bruges periodic soundings: catalog candidate pending a
@@ -0,0 +1,236 @@
# Post-RC Data Coverage Roadmap: Belgium And The Belgian North Sea
## Status and purpose
This is the active autonomous implementation board after `v1.0.0-rc.1`.
It is not an RC-12 phase. The signed RC remains immutable release evidence;
all work below targets the next version and must preserve the frozen API,
persistence, provenance and no-fake-data rules.
The goal is to make the national workbench useful with real data:
1. choose an understandable theme;
2. select a bounded area anywhere in Belgium or the Belgian North Sea;
3. acquire or reuse only governed data that covers that selection;
4. calculate source-appropriate metrics;
5. expose authority, time, licence, coverage and limitations;
6. keep unsupported or unavailable results visibly unavailable.
Mol and the Kempen remain mandatory regression areas. They do not constrain
the geographic implementation.
## Autonomous execution rules
- Read `docs/CODEX_BOOTSTRAP_PROMPT.md` and
`docs/RC_SCOPE_FREEZE_BELGIUM_NORTH_SEA.md` before every resumed pass.
- Never run an unbounded browser-side or application-startup provider fetch.
- Every external endpoint must be allowlisted, strict-TLS and size/time limited.
- Provider output must be staged, validated and persisted through
`DatasetService` and `VectorFeatureService`.
- No source may become `operational` from catalogue evidence alone.
- Do not merge semantically different regional sources into one value.
- Do not create water volume, bathymetry, population, AI or historical values
when the required evidence is absent.
- A failed source or validation gate stops only that source. Other independent
phases may continue.
- Update `docs/CODEX_EXECUTION_LOG.md`, `docs/TODO.md`, relevant source/API
docs and this roadmap after every completed gate.
## Global validation gate
Run after every implementation phase:
```bash
python -m compileall backend/app
cd backend && python -m pytest
cd frontend && npm run test:unit
cd frontend && npm run typecheck
cd frontend && npm run build
bash scripts/run_readiness_check.sh
cd backend && python -m alembic heads
cd backend && python -m alembic upgrade head --sql
bash -n scripts/live_migration_smoke.sh
```
When Docker/Tower is available:
```bash
docker compose config
curl --fail http://192.168.10.150:1202/health/ready
bash scripts/run_rc8_release_journeys.sh \
http://192.168.10.150:1202 \
artifacts/post-rc-release-journeys \
artifacts/post-rc-golden-areas.json
```
No phase is complete when compile, tests, typecheck, build, migration or the
applicable live bounded journey fails.
## P0 - Existing national layers become usable
**State: implemented locally; live release evidence pending under P5.**
### Work
- Add understandable map themes for administrative context, maritime planning
and marine reporting/legal context.
- Match persisted `ngi_adminvector`, `rbins_marine_reporting_units` and
`rbins_msp_2026` datasets explicitly; do not rely on filename substrings.
- Automatically select the first operational theme when the previous/default
theme has no matching dataset in the active workspace.
- Add source-appropriate selection summaries. Administrative polygon area may
be reported in hectares; overlapping marine use zones must not be summed as
unique sea area.
- Make workbench readiness distinguish stored context/reference layers from
actually analyzable themes.
- Add frontend and backend regression tests and verify the live national and
North Sea workspaces without console errors.
### Exit
- Belgium land opens with a usable NGI administrative theme.
- The Belgian North Sea opens with usable legal/reporting and marine-plan
themes.
- Drawing or selecting a bounded area produces persisted-source results with
provenance.
- An unavailable buildings/population/bathymetry source remains unavailable.
## P1 - National Statbel population
**State: implementation complete; reviewed live national materialization pending.**
### Work
- Extend the existing governed Statbel plan-stage-review-apply flow with a
national scope that uses the persisted Belgium boundary.
- Keep the 2025 REDEGEO old/new geometries separate.
- Validate archive checksums, schema, CRS, sector joins, `ZZZZ` accounting and
national totals before persistence.
- Materialize one immutable national edition through `DatasetService`; bounded
map queries continue to run in PostGIS.
- Expose inhabitants as an area-weighted estimate only when a selection cuts
statistical sectors, with an explicit warning and compatible time-series
keys.
- Never download or replace a release automatically.
### Exit
- A named reviewed plan is required for live apply.
- Belgium, Flanders, Wallonia and Brussels bounded population selections work
from the same authoritative Statbel edition.
- Mol remains within its documented baseline tolerance.
## P2 - Wallonia bounded topographic baseline
**State: implemented locally; live Wallonia journey pending under P5.**
### Governed source
SPW PICC ArcGIS REST:
```text
https://geoservices.wallonie.be/arcgis/rest/services/TOPOGRAPHIE/PICC_VDIFF/MapServer
```
Initial allowlisted layers:
- `11`: building footprints;
- `21`: road axes;
- `28`: hydrographic axes;
- `30`: hydrographic surfaces.
### Work
- Add a bounded ArcGIS REST adapter using an EPSG:4326 selection transformed by
the official service to EPSG:3812/4326.
- Enforce maximum selection dimensions, paging, response size, feature count,
strict content type, stable layer identity and source metadata.
- Normalize geometry to EPSG:4326 while retaining provider-native attributes.
- Persist buildings, roads and water as separate immutable datasets.
- Add hectare/kilometre metrics without claiming semantic parity with GRB.
- Expose the products only for Walloon intersections and keep mixed-zone
results split.
### Exit
- The Wallonia golden area can explicitly acquire and analyse all allowlisted
PICC themes.
- Provider failures and record-limit truncation fail closed.
- No direct provider-to-`vector_features` write exists.
## P3 - Brussels bounded UrbIS baseline
**State: implemented locally; live Brussels journey pending under P5.**
### Governed source
Paradigm UrbIS parcels and buildings WFS, discovered and pinned from the
official datastore metadata. The exact service URL and type names must pass a
capabilities probe before implementation is marked operational.
### Work
- Add bounded WFS acquisition for building footprints and cadastral parcels.
- Retain the separate Paradigm and FPS Finance authorities/licences where
applicable.
- Enforce bbox, paging, response size, CRS and geometry validation.
- Persist separate immutable building and parcel datasets.
- Add source-appropriate area/count metrics and Brussels-only coverage.
### Exit
- The Brussels golden area can explicitly acquire and analyse buildings and
parcels.
- Catalogue-only or failed capabilities evidence remains `not_configured`.
## P4 - Maritime and bathymetry hardening
**State: complete with MDK acquisition visibly blocked by strict-TLS evidence.**
### Work
- Keep the current RBINS marine legal/reporting and 2026-2034 planning layers
operational and map-queryable.
- Probe the MDK 20 m LAT product only through strict TLS and retained metadata.
- Implement bounded GeoTIFF acquisition only after CRS, nodata, pixel
semantics, edition, licence and LAT evidence pass.
- Keep depth relative to LAT separate from water-surface elevation and volume.
- SPW mDNG bathymetry remains a separate Walloon inland/navigation source.
### Exit
- Maritime plan and legal/reporting queries are operational.
- Bathymetry is either a governed analytical raster with vertical datum and
uncertainty evidence, or remains visibly `not_configured`.
- No water-volume metric exists without compatible bed and water-surface data.
## P5 - Federation, UX and release evidence
**State: in progress.**
### Work
- Make on-demand products depend on the intersected zone, not on a manually
selected technical project.
- Keep cross-region selections split by source and metric semantics.
- Present loaded, available-on-demand, partial, not-configured and unsupported
states in ordinary Dutch language.
- Run land, language-boundary, Brussels, Wallonia, coast and North Sea golden
journeys against live PostGIS.
- Capture source checksums, selected bounds, result metrics, provenance,
performance, browser console/network and responsive screenshots.
- Build, push and deploy one immutable post-RC image only after every applicable
gate is green.
### Final acceptance
- Existing RC behavior remains green.
- At least one real bounded analytical theme is operational for Belgium,
Flanders, Wallonia, Brussels and the Belgian North Sea.
- Population is nationally operational or explicitly blocked by failed source
evidence.
- Buildings/roads/water are operational through the competent regional source
where implemented; no parity claim is made.
- No unavailable theme is presented as measured data.
- The live browser can select an area, acquire/reuse data, analyse it, inspect
provenance and export it without entering a technical project workflow.
+15
View File
@@ -1,5 +1,20 @@
# GeoIntel TODO
## Actieve post-RC datadekkingsfase
`docs/POST_RC_DATA_COVERAGE_ROADMAP_BELGIUM_NORTH_SEA.md` is het actieve
autonome uitvoeringsbord na `v1.0.0-rc.1`. Dit is geen RC-12. De ondertekende
RC blijft immutable bewijs; deze fase maakt echte, begrensde nationale en
maritieme databronnen bruikbaar in de volgende versie.
- [x] P0: bestaande NGI/RBINS-lagen als kaartthema en selectieanalyse ontsluiten.
- [x] P0: readiness baseren op bruikbare analysethema's, niet alleen datasetaantal.
- [ ] P1: nationale Statbel-populatieflow veilig operationaliseren.
- [x] P2: bounded SPW PICC voor Waalse gebouwen, wegen en hydrographie.
- [x] P3: bounded UrbIS voor Brusselse gebouwen en percelen.
- [x] P4: maritieme thema's harden en bathymetrie fail-closed onderzoeken.
- [ ] P5: zonefederatie, UX, golden journeys en immutable deployment.
## Actief RC-programma: Belgie en de Belgische Noordzee
`docs/RC_ROADMAP_BELGIUM_NORTH_SEA.md` is vanaf 2026-07-17 het enige actieve