Federate official Belgium data sources
This commit is contained in:
+28
-7
@@ -2264,13 +2264,21 @@ return an empty, honest result. No provider request occurs during analysis.
|
||||
Future provider output continues to use DatasetService and, for vectors,
|
||||
VectorFeatureService. Arbitrary service URLs, browser-side fetches, insecure
|
||||
TLS bypasses and startup downloads remain forbidden.
|
||||
## Governed official nature and soil acquisition
|
||||
## Governed regional official-vector acquisition
|
||||
|
||||
### GET `/api/v1/projects/{project_id}/datasets/official-vector/products`
|
||||
|
||||
Returns the fixed official vector registry in the canonical `{ "data": ... }`
|
||||
envelope. The allowlist contains `bwk_natura2000_2025` and `dov_soil_types`;
|
||||
arbitrary collection names or URLs are never accepted.
|
||||
envelope. Every item includes its theme, geometry contract, authority,
|
||||
licence/attribution, provider-native collection, query mode and
|
||||
`coverage_zones`. The fixed allowlist contains:
|
||||
|
||||
- Flanders: `bwk_natura2000_2025` and `dov_soil_types`;
|
||||
- Wallonia: `spw_picc_buildings`, `spw_picc_roads`,
|
||||
`spw_picc_waterways` and `spw_picc_water_surfaces`;
|
||||
- Brussels: `urbis_buildings` and `urbis_cadastral_parcels`.
|
||||
|
||||
Arbitrary collection names or URLs are never accepted.
|
||||
|
||||
### POST `/api/v1/projects/{project_id}/datasets/official-vector/acquire`
|
||||
|
||||
@@ -2293,16 +2301,29 @@ Request:
|
||||
|
||||
The synchronous `vector.official.acquire` Job validates the metric request
|
||||
size, intersects `bbox` with the persisted Area, retrieves every bounded page,
|
||||
clips polygon geometry in EPSG:31370 and persists EPSG:4326 features through
|
||||
`DatasetService`. A repeated exact request can reuse the 24-hour cache.
|
||||
Provider errors, unstable or incomplete WFS pagination and safety-limit violations
|
||||
fail without persisting a truncated Dataset.
|
||||
clips source geometry in the provider-native metric CRS and persists
|
||||
EPSG:4326 features through `DatasetService`. A repeated exact request can
|
||||
reuse the 24-hour cache. Provider errors, unstable/incomplete pagination and
|
||||
safety-limit violations fail without persisting a truncated Dataset.
|
||||
|
||||
Walloon and Brussels products additionally require a persisted exact regional
|
||||
coverage Area (`Wallonia` or `Brussels-Capital Region`). SPW/PICC uses stable
|
||||
`OBJECTID` ArcGIS REST paging; UrbIS uses WFS 2.0 paging ordered by
|
||||
`INSPIRE_ID`. A caller cannot make a Flemish product cover Wallonia, or merge
|
||||
different regional authorities into one Dataset.
|
||||
|
||||
`bwk_natura2000_2025` preserves BWK `EVAL`, `EENH*`, `HAB*` and `PHAB*`
|
||||
semantics. `dov_soil_types` preserves mapped soil, texture, drainage, profile
|
||||
and substrate classes and is dated as the 1949-1971 survey period. Neither
|
||||
contract accepts a caller-supplied endpoint.
|
||||
|
||||
PICC building footprints, road axes and hydrographic axes/surfaces retain
|
||||
their source identifiers and report source-appropriate counts, footprint/
|
||||
surface hectares or line kilometres. UrbIS building footprints and cadastral
|
||||
parcels remain separate products with their own Paradigm/FPS Finance licence
|
||||
notes. These products do not claim semantic parity with GRB and do not expose
|
||||
water volume.
|
||||
|
||||
## Governed Landgebruik Vlaanderen forest and agriculture
|
||||
|
||||
The existing
|
||||
|
||||
@@ -1,5 +1,28 @@
|
||||
## Autonomous RC program for Belgium and the Belgian North Sea (2026-07-17)
|
||||
|
||||
### Post-RC national data federation (2026-07-19)
|
||||
|
||||
- Added explicit administrative and maritime themes for persisted NGI/RBINS
|
||||
reference data and made readiness depend on analyzable themes rather than a
|
||||
raw Dataset count.
|
||||
- Generalized the governed Statbel population scripts to the Belgium boundary
|
||||
and complete municipality inventory while retaining all release-promotion
|
||||
safety gates.
|
||||
- Added allowlisted bounded SPW/PICC ArcGIS REST and UrbIS WFS adapters with
|
||||
source-specific coverage, CRS, identity, paging, geometry, licence,
|
||||
provenance and metric contracts. Output still enters PostGIS only through
|
||||
`DatasetService.import_vector_bytes`.
|
||||
- Made the frontend resolve actual coverage zones before choosing GRB,
|
||||
SPW/PICC or UrbIS and retain separate source results for mixed-zone
|
||||
selections.
|
||||
- Added SPW/UrbIS runtime controls to both Compose variants and the editable
|
||||
Unraid/Dockerman deployment path.
|
||||
- Rechecked MDK bathymetry. Strict hostname validation still fails because the
|
||||
configured official hostname presents a `*.l27powered.eu` certificate, so
|
||||
acquisition remains truthfully disabled.
|
||||
- Focused backend and frontend suites passed before full release validation;
|
||||
full local and Tower evidence follows in the final P5 gate.
|
||||
|
||||
### RC-4 national and maritime coverage foundation
|
||||
|
||||
- Added an independent coverage registry so the exact Sprint 7
|
||||
|
||||
+35
-7
@@ -751,9 +751,37 @@ inferred from these plan zones.
|
||||
|
||||
The coverage resolver reports a theme `operational` only when a matching
|
||||
`ready` Dataset exists in the selected project. Implemented acquisition
|
||||
without project materialization is `partial`; Walloon, Brussels, Statbel and
|
||||
MDK families remain `not_configured` until their own bounded adapters and
|
||||
source-specific metric contracts are implemented.
|
||||
without project materialization is `partial`. Statbel is an operator-managed
|
||||
national source. SPW/PICC and UrbIS are bounded API sources for their explicit
|
||||
themes and zones. MDK remains `not_configured` while its strict-TLS readiness
|
||||
probe fails.
|
||||
|
||||
The governed Statbel operator supports `belgium` as a first-class scope. It
|
||||
uses the retained NGI Belgium boundary, retains every statistical-sector row
|
||||
and requires the existing plan -> stage -> named review -> checksum-confirmed
|
||||
apply process. Flanders, Wallonia, Brussels and bounded municipality
|
||||
population queries therefore derive from one reviewed national edition rather
|
||||
than mutually incompatible regional imports. No edition is downloaded or
|
||||
promoted during application startup.
|
||||
|
||||
## Wallonia PICC and Brussels UrbIS
|
||||
|
||||
The official-vector registry exposes bounded, source-specific regional
|
||||
baselines:
|
||||
|
||||
- SPW/PICC MapServer layers 11 (buildings), 21 (road axes), 28
|
||||
(hydrographic axes) and 30 (hydrographic surfaces) for Wallonia;
|
||||
- Paradigm UrbIS `Buildings` and `CadastralParcels` WFS collections for the
|
||||
Brussels-Capital Region.
|
||||
|
||||
SPW queries use stable `OBJECTID` paging and request EPSG:4326 output while
|
||||
retaining EPSG:3812 as the native source CRS. UrbIS responses use EPSG:31370,
|
||||
stable `INSPIRE_ID` paging and are transformed only after exact regional
|
||||
clipping. Both adapters enforce bbox dimensions, page/feature/response limits,
|
||||
strict HTTPS host/path validation, geometry contracts and persisted coverage
|
||||
Areas. They persist only through `DatasetService`; the browser never contacts
|
||||
either provider directly. Cross-zone selections remain split by authority and
|
||||
metric semantics.
|
||||
|
||||
## Bathymetry, inland profiles and maritime scope
|
||||
|
||||
@@ -768,10 +796,10 @@ The following sources are audited but not yet operational:
|
||||
|
||||
- MDK Dieptemodel Belgisch Continentaal Plat/Noordzee: 20 x 20 m continuous
|
||||
raster in LAT, exposed through WCS/WMTS. GeoIntel now has a strict-TLS,
|
||||
read-only GetCapabilities probe. On 2026-07-17 the official metadata
|
||||
endpoint presented a certificate for another hostname and returned no
|
||||
usable capabilities path; status therefore remains `tls_error` and raster
|
||||
acquisition is disabled.
|
||||
read-only GetCapabilities probe. Revalidation on 2026-07-19 still found that
|
||||
`bathy.agentschapmdk.be` presents a certificate for `*.l27powered.eu`.
|
||||
Hostname validation therefore fails with `tls_error`; raster acquisition
|
||||
remains disabled and TLS verification cannot be bypassed.
|
||||
- SPW bathymetry of navigable waterways and reservoir lakes: 0.5 m bed
|
||||
elevation and XYZ data in mDNG.
|
||||
- Port of Antwerp-Bruges periodic soundings: catalog candidate pending a
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
# Post-RC Data Coverage Roadmap: Belgium And The Belgian North Sea
|
||||
|
||||
## Status and purpose
|
||||
|
||||
This is the active autonomous implementation board after `v1.0.0-rc.1`.
|
||||
It is not an RC-12 phase. The signed RC remains immutable release evidence;
|
||||
all work below targets the next version and must preserve the frozen API,
|
||||
persistence, provenance and no-fake-data rules.
|
||||
|
||||
The goal is to make the national workbench useful with real data:
|
||||
|
||||
1. choose an understandable theme;
|
||||
2. select a bounded area anywhere in Belgium or the Belgian North Sea;
|
||||
3. acquire or reuse only governed data that covers that selection;
|
||||
4. calculate source-appropriate metrics;
|
||||
5. expose authority, time, licence, coverage and limitations;
|
||||
6. keep unsupported or unavailable results visibly unavailable.
|
||||
|
||||
Mol and the Kempen remain mandatory regression areas. They do not constrain
|
||||
the geographic implementation.
|
||||
|
||||
## Autonomous execution rules
|
||||
|
||||
- Read `docs/CODEX_BOOTSTRAP_PROMPT.md` and
|
||||
`docs/RC_SCOPE_FREEZE_BELGIUM_NORTH_SEA.md` before every resumed pass.
|
||||
- Never run an unbounded browser-side or application-startup provider fetch.
|
||||
- Every external endpoint must be allowlisted, strict-TLS and size/time limited.
|
||||
- Provider output must be staged, validated and persisted through
|
||||
`DatasetService` and `VectorFeatureService`.
|
||||
- No source may become `operational` from catalogue evidence alone.
|
||||
- Do not merge semantically different regional sources into one value.
|
||||
- Do not create water volume, bathymetry, population, AI or historical values
|
||||
when the required evidence is absent.
|
||||
- A failed source or validation gate stops only that source. Other independent
|
||||
phases may continue.
|
||||
- Update `docs/CODEX_EXECUTION_LOG.md`, `docs/TODO.md`, relevant source/API
|
||||
docs and this roadmap after every completed gate.
|
||||
|
||||
## Global validation gate
|
||||
|
||||
Run after every implementation phase:
|
||||
|
||||
```bash
|
||||
python -m compileall backend/app
|
||||
cd backend && python -m pytest
|
||||
cd frontend && npm run test:unit
|
||||
cd frontend && npm run typecheck
|
||||
cd frontend && npm run build
|
||||
bash scripts/run_readiness_check.sh
|
||||
cd backend && python -m alembic heads
|
||||
cd backend && python -m alembic upgrade head --sql
|
||||
bash -n scripts/live_migration_smoke.sh
|
||||
```
|
||||
|
||||
When Docker/Tower is available:
|
||||
|
||||
```bash
|
||||
docker compose config
|
||||
curl --fail http://192.168.10.150:1202/health/ready
|
||||
bash scripts/run_rc8_release_journeys.sh \
|
||||
http://192.168.10.150:1202 \
|
||||
artifacts/post-rc-release-journeys \
|
||||
artifacts/post-rc-golden-areas.json
|
||||
```
|
||||
|
||||
No phase is complete when compile, tests, typecheck, build, migration or the
|
||||
applicable live bounded journey fails.
|
||||
|
||||
## P0 - Existing national layers become usable
|
||||
|
||||
**State: implemented locally; live release evidence pending under P5.**
|
||||
|
||||
### Work
|
||||
|
||||
- Add understandable map themes for administrative context, maritime planning
|
||||
and marine reporting/legal context.
|
||||
- Match persisted `ngi_adminvector`, `rbins_marine_reporting_units` and
|
||||
`rbins_msp_2026` datasets explicitly; do not rely on filename substrings.
|
||||
- Automatically select the first operational theme when the previous/default
|
||||
theme has no matching dataset in the active workspace.
|
||||
- Add source-appropriate selection summaries. Administrative polygon area may
|
||||
be reported in hectares; overlapping marine use zones must not be summed as
|
||||
unique sea area.
|
||||
- Make workbench readiness distinguish stored context/reference layers from
|
||||
actually analyzable themes.
|
||||
- Add frontend and backend regression tests and verify the live national and
|
||||
North Sea workspaces without console errors.
|
||||
|
||||
### Exit
|
||||
|
||||
- Belgium land opens with a usable NGI administrative theme.
|
||||
- The Belgian North Sea opens with usable legal/reporting and marine-plan
|
||||
themes.
|
||||
- Drawing or selecting a bounded area produces persisted-source results with
|
||||
provenance.
|
||||
- An unavailable buildings/population/bathymetry source remains unavailable.
|
||||
|
||||
## P1 - National Statbel population
|
||||
|
||||
**State: implementation complete; reviewed live national materialization pending.**
|
||||
|
||||
### Work
|
||||
|
||||
- Extend the existing governed Statbel plan-stage-review-apply flow with a
|
||||
national scope that uses the persisted Belgium boundary.
|
||||
- Keep the 2025 REDEGEO old/new geometries separate.
|
||||
- Validate archive checksums, schema, CRS, sector joins, `ZZZZ` accounting and
|
||||
national totals before persistence.
|
||||
- Materialize one immutable national edition through `DatasetService`; bounded
|
||||
map queries continue to run in PostGIS.
|
||||
- Expose inhabitants as an area-weighted estimate only when a selection cuts
|
||||
statistical sectors, with an explicit warning and compatible time-series
|
||||
keys.
|
||||
- Never download or replace a release automatically.
|
||||
|
||||
### Exit
|
||||
|
||||
- A named reviewed plan is required for live apply.
|
||||
- Belgium, Flanders, Wallonia and Brussels bounded population selections work
|
||||
from the same authoritative Statbel edition.
|
||||
- Mol remains within its documented baseline tolerance.
|
||||
|
||||
## P2 - Wallonia bounded topographic baseline
|
||||
|
||||
**State: implemented locally; live Wallonia journey pending under P5.**
|
||||
|
||||
### Governed source
|
||||
|
||||
SPW PICC ArcGIS REST:
|
||||
|
||||
```text
|
||||
https://geoservices.wallonie.be/arcgis/rest/services/TOPOGRAPHIE/PICC_VDIFF/MapServer
|
||||
```
|
||||
|
||||
Initial allowlisted layers:
|
||||
|
||||
- `11`: building footprints;
|
||||
- `21`: road axes;
|
||||
- `28`: hydrographic axes;
|
||||
- `30`: hydrographic surfaces.
|
||||
|
||||
### Work
|
||||
|
||||
- Add a bounded ArcGIS REST adapter using an EPSG:4326 selection transformed by
|
||||
the official service to EPSG:3812/4326.
|
||||
- Enforce maximum selection dimensions, paging, response size, feature count,
|
||||
strict content type, stable layer identity and source metadata.
|
||||
- Normalize geometry to EPSG:4326 while retaining provider-native attributes.
|
||||
- Persist buildings, roads and water as separate immutable datasets.
|
||||
- Add hectare/kilometre metrics without claiming semantic parity with GRB.
|
||||
- Expose the products only for Walloon intersections and keep mixed-zone
|
||||
results split.
|
||||
|
||||
### Exit
|
||||
|
||||
- The Wallonia golden area can explicitly acquire and analyse all allowlisted
|
||||
PICC themes.
|
||||
- Provider failures and record-limit truncation fail closed.
|
||||
- No direct provider-to-`vector_features` write exists.
|
||||
|
||||
## P3 - Brussels bounded UrbIS baseline
|
||||
|
||||
**State: implemented locally; live Brussels journey pending under P5.**
|
||||
|
||||
### Governed source
|
||||
|
||||
Paradigm UrbIS parcels and buildings WFS, discovered and pinned from the
|
||||
official datastore metadata. The exact service URL and type names must pass a
|
||||
capabilities probe before implementation is marked operational.
|
||||
|
||||
### Work
|
||||
|
||||
- Add bounded WFS acquisition for building footprints and cadastral parcels.
|
||||
- Retain the separate Paradigm and FPS Finance authorities/licences where
|
||||
applicable.
|
||||
- Enforce bbox, paging, response size, CRS and geometry validation.
|
||||
- Persist separate immutable building and parcel datasets.
|
||||
- Add source-appropriate area/count metrics and Brussels-only coverage.
|
||||
|
||||
### Exit
|
||||
|
||||
- The Brussels golden area can explicitly acquire and analyse buildings and
|
||||
parcels.
|
||||
- Catalogue-only or failed capabilities evidence remains `not_configured`.
|
||||
|
||||
## P4 - Maritime and bathymetry hardening
|
||||
|
||||
**State: complete with MDK acquisition visibly blocked by strict-TLS evidence.**
|
||||
|
||||
### Work
|
||||
|
||||
- Keep the current RBINS marine legal/reporting and 2026-2034 planning layers
|
||||
operational and map-queryable.
|
||||
- Probe the MDK 20 m LAT product only through strict TLS and retained metadata.
|
||||
- Implement bounded GeoTIFF acquisition only after CRS, nodata, pixel
|
||||
semantics, edition, licence and LAT evidence pass.
|
||||
- Keep depth relative to LAT separate from water-surface elevation and volume.
|
||||
- SPW mDNG bathymetry remains a separate Walloon inland/navigation source.
|
||||
|
||||
### Exit
|
||||
|
||||
- Maritime plan and legal/reporting queries are operational.
|
||||
- Bathymetry is either a governed analytical raster with vertical datum and
|
||||
uncertainty evidence, or remains visibly `not_configured`.
|
||||
- No water-volume metric exists without compatible bed and water-surface data.
|
||||
|
||||
## P5 - Federation, UX and release evidence
|
||||
|
||||
**State: in progress.**
|
||||
|
||||
### Work
|
||||
|
||||
- Make on-demand products depend on the intersected zone, not on a manually
|
||||
selected technical project.
|
||||
- Keep cross-region selections split by source and metric semantics.
|
||||
- Present loaded, available-on-demand, partial, not-configured and unsupported
|
||||
states in ordinary Dutch language.
|
||||
- Run land, language-boundary, Brussels, Wallonia, coast and North Sea golden
|
||||
journeys against live PostGIS.
|
||||
- Capture source checksums, selected bounds, result metrics, provenance,
|
||||
performance, browser console/network and responsive screenshots.
|
||||
- Build, push and deploy one immutable post-RC image only after every applicable
|
||||
gate is green.
|
||||
|
||||
### Final acceptance
|
||||
|
||||
- Existing RC behavior remains green.
|
||||
- At least one real bounded analytical theme is operational for Belgium,
|
||||
Flanders, Wallonia, Brussels and the Belgian North Sea.
|
||||
- Population is nationally operational or explicitly blocked by failed source
|
||||
evidence.
|
||||
- Buildings/roads/water are operational through the competent regional source
|
||||
where implemented; no parity claim is made.
|
||||
- No unavailable theme is presented as measured data.
|
||||
- The live browser can select an area, acquire/reuse data, analyse it, inspect
|
||||
provenance and export it without entering a technical project workflow.
|
||||
@@ -1,5 +1,20 @@
|
||||
# GeoIntel TODO
|
||||
|
||||
## Actieve post-RC datadekkingsfase
|
||||
|
||||
`docs/POST_RC_DATA_COVERAGE_ROADMAP_BELGIUM_NORTH_SEA.md` is het actieve
|
||||
autonome uitvoeringsbord na `v1.0.0-rc.1`. Dit is geen RC-12. De ondertekende
|
||||
RC blijft immutable bewijs; deze fase maakt echte, begrensde nationale en
|
||||
maritieme databronnen bruikbaar in de volgende versie.
|
||||
|
||||
- [x] P0: bestaande NGI/RBINS-lagen als kaartthema en selectieanalyse ontsluiten.
|
||||
- [x] P0: readiness baseren op bruikbare analysethema's, niet alleen datasetaantal.
|
||||
- [ ] P1: nationale Statbel-populatieflow veilig operationaliseren.
|
||||
- [x] P2: bounded SPW PICC voor Waalse gebouwen, wegen en hydrographie.
|
||||
- [x] P3: bounded UrbIS voor Brusselse gebouwen en percelen.
|
||||
- [x] P4: maritieme thema's harden en bathymetrie fail-closed onderzoeken.
|
||||
- [ ] P5: zonefederatie, UX, golden journeys en immutable deployment.
|
||||
|
||||
## Actief RC-programma: Belgie en de Belgische Noordzee
|
||||
|
||||
`docs/RC_ROADMAP_BELGIUM_NORTH_SEA.md` is vanaf 2026-07-17 het enige actieve
|
||||
|
||||
Reference in New Issue
Block a user