Add governed SPW bathymetry analysis

2026-07-19 04:07:01 +02:00
parent ae3d3dc634
commit 70c5e34ecf
37 changed files with 1852 additions and 102 deletions
@@ -2192,9 +2192,9 @@ sets an explicit Ollama context window and returns

 Returns the governed bathymetry source registry in the canonical envelope.
 VHA inland profiles are `operational`. MDK Belgian Continental Shelf is
-`probe_only`; SPW Walloon bathymetry remains `available_not_integrated`.
-Neither source can be acquired until its raster/download and vertical-datum
-flow passes live validation.
+`probe_only`. The pinned SPW Walloon bathymetry archive is `operational`
+through a bounded, explicit operator import. There is no browser-side source
+fetch and no arbitrary source URL.

 ### GET `/api/v1/projects/{project_id}/datasets/bathymetry/sources/mdk_bcp_bathymetry/readiness`

@@ -2261,6 +2261,31 @@ request limit, while `total_feature_count` and all configured depth/width
 metrics are calculated across the complete spatial result. Empty municipalities
 return an empty, honest result. No provider request occurs during analysis.

+### POST `/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/bathymetry/select`
+
+Runs a bounded selection against a ready persisted
+`source_name=spw_bathymetry` raster. The request contains an EPSG:4326 bbox
+and optional persisted `area_id`, matching the other governed raster-selection
+contracts.
+
+The response reports:
+
+- mean, minimum, maximum, p10 and p90 waterbed elevation in `m mDNG`;
+- exact raster-cell surface with surveyed bed values in hectares;
+- source coverage percentage inside the selected geometry;
+- source resolution, survey period `2019-2022`, vertical reference and
+  explicit unsupported metrics.
+
+`current_water_depth_m`, `water_volume_m3` and vertical-datum conversion remain
+unsupported. The endpoint reads the persisted EPSG:3812 geometry-aligned
+raster; it does not query SPW and does not synthesize missing cells.
+
+### GET `/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/bathymetry/image`
+
+Returns a transparent PNG rendering of the persisted bathymetry COG for the
+MapLibre image-overlay path. Rendering changes presentation only. Analytical
+values always come from the stored Float32 source cells.
+
 Future provider output continues to use DatasetService and, for vectors,
 VectorFeatureService. Arbitrary service URLs, browser-side fetches, insecure
 TLS bypasses and startup downloads remain forbidden.
@@ -8,10 +8,11 @@ GeoIntel must distinguish three different questions:
 2. What is the continuous elevation of the bed at a specific survey epoch?
 3. What is the water depth or volume at a specific moment?

-Only the first question is operational for Mol through the VHA cross-section
-profile layer. A bed model does not provide water depth without a compatible
-water-surface elevation. Flood-hazard maximum depth is a scenario result and
-must not be reused as current water level.
+The first question is operational for Flanders through VHA cross-section
+profiles. The second is now operational for bounded, surveyed Walloon
+waterways through the official SPW raster. A bed model does not provide water
+depth without a compatible water-surface elevation. Flood-hazard maximum depth
+is a scenario result and must not be reused as current water level.

 ## Governed source matrix

@@ -19,7 +20,7 @@ must not be reused as current water level.
 | --- | --- | --- | --- | --- |
 | VMM VHA Digital Atlas | Flanders | Point locations, structured profile fields, PDF evidence | Document-specific | Operational, bounded vector acquisition |
 | MDK Belgian Continental Shelf model | Belgian North Sea | Continuous 20 x 20 m bathymetric raster | LAT | Available, not integrated |
-| SPW navigable waterways and reservoir lakes | Wallonia | 0.5 m bed-elevation raster and XYZ cloud | mDNG | Available, not integrated |
+| SPW navigable waterways and reservoir lakes | Wallonia | 0.5 m bed-elevation raster and XYZ cloud | mDNG | Operational, pinned operator archive and bounded COG |
 | Port of Antwerp-Bruges publications | Port survey areas | Periodic soundings | Product-specific | Catalog candidate |

 VHA contains approximately 129,643 profile points across Flanders at the
@@ -135,7 +136,10 @@ bed evolution.
   **Probe implemented. Acquisition blocked because the live endpoint fails
   strict hostname validation and does not expose usable capabilities.**
 4. Implement SPW download staging and vertical-datum metadata validation.
+   **Done: pinned official ZIP, safe `/vsizip/` access, bounded COG,
+   DatasetService persistence, map overlay and mDNG selection metrics.**
 5. Add maritime boundaries as separate authoritative scope layers.
+   **Done for the Belgian territorial sea, EEZ and continental shelf.**
 6. Add cross-source vertical-datum transformation only with authoritative
   grids/parameters and uncertainty tests.
 7. Add volume only after a compatible measured or modeled water-surface source
@@ -1,53 +1,69 @@
 # GeoIntel Build Status

-Updated: 2026-07-17
+Updated: 2026-07-19

 ## Current state

-GeoIntel is an implemented map-first GeoAI workbench running as an all-in-one
-Unraid container with embedded PostGIS, FastAPI, React/MapLibre, local
-Ollama integration and optional local YOLO/PyTorch inference.
+GeoIntel `v1.0.0-rc.1` is an accepted map-first GeoAI workbench for Belgium
+and the legally distinct Belgian maritime scopes. It runs as an immutable
+all-in-one Unraid image with PostGIS, FastAPI, React/MapLibre, local Ollama
+integration and optional local YOLO/PyTorch inference.

-The active release program is
-`docs/RC_ROADMAP_BELGIUM_NORTH_SEA.md`.
+Mol and the Kempen are deep regression areas, not the product boundary. The
+national workbench also has deterministic golden journeys for Wallonia,
+Brussels, a language-boundary selection, the coast and the Belgian North Sea.

-## Product scope
+## Release status

- Target: all Belgian land and the separately labelled Belgian territorial
-  sea, EEZ and continental shelf.
- Existing deep regression references: Mol and the Kempen transport region.
- Required new golden areas: Wallonia, Brussels, a cross-region area, coast
-  and Belgian North Sea.
- Architecture: federated authoritative providers with one coverage matrix;
-  no assumption that GRB, PICC, UrbIS and NGI are semantically identical.
+There are no open release blockers for the signed `v1.0.0-rc.1` evidence set.
+RC-0 through RC-11 are complete. Backup/restore, fresh install, upgrade,
+rollback, fail-closed readiness, one Alembic head, API contracts, supply-chain
+policy, responsive browser journeys and live PostGIS acceptance are proven.

-## Implemented foundation
+The active post-RC candidate extends real source coverage. Its repository gate
+passes 1,052 backend tests, 22 frontend tests, frontend typecheck/build,
+one Alembic head and the complete readiness script. Its remaining acceptance
+step is live SPW bathymetry persistence and browser verification before a new
+immutable image is called releasable.

- Projects, Areas, Datasets, versions, PostGIS vector features and raster
-  artifacts.
- Map selection, semantic metrics, historical comparison and exports.
- Governed Flemish sources for buildings, context, orthophotos, terrain,
-  flooding, soil, nature, agriculture, population and policy rasters.
- Detection persistence, local configured YOLO/PyTorch path, QA/QC and review
-  evidence.
- Segmentation persistence foundation without fake production inference.
- Local Ollama assistant grounded in persisted evidence.
- Single-container Unraid deployment on port 1202.
+## Operational product loop

-## Release blockers
+- Open one national map without selecting a technical project.
+- Select an understandable theme and draw or reuse a bounded Area.
+- Resolve governed coverage per Belgian jurisdiction.
+- Analyse persisted PostGIS vectors or bounded raster artifacts.
+- Show source-appropriate metrics, provenance, time and limitations.
+- Compare compatible Statbel 2021-2025 and other governed historical series.
+- Export persisted evidence or question it through the local Ollama assistant.
+- Run configured local YOLO detection with persisted QA and explicit human
+  review limitations.

- Current production backup and isolated restore proof.
- Fail-closed readiness and truthful runtime capability reporting.
- Stale job/run reconciliation and correlated exception logging.
- Temporal compatibility guard for detection QA.
- Production secret/configuration parity, immutable images and rollback.
- Full CI, reproducible dependencies and supply-chain evidence.
- Critical response-model typing and real frontend/browser E2E coverage.
- National and maritime scope/providers/golden areas.
+## Explicit non-blocking boundaries

-## Latest evidence
+- MDK analytical North Sea bathymetry remains `not_configured`: the public
+  endpoint fails strict hostname validation and the official low-resolution
+  product is currently request-based. TLS and source-integrity checks are not
+  bypassed.
+- SPW supplies governed Walloon waterbed elevation in mDNG, not current water
+  depth. Water volume remains unsupported until a compatible water-surface,
+  time, vertical-datum and uncertainty contract exists.
+- Building detection is an assisted review workflow. The retained benchmark
+  still requires independent false-negative review before any claim of
+  autonomous production accuracy or another training pass.
+- Real SAM/YOLO-seg inference, a training studio, LiDAR, production
+  multi-user authentication and real-time monitoring remain post-V1 scope.
+- Additional official datasets may improve coverage, but they are not V1
+  completion blockers when the coverage matrix reports absence honestly.

-Run:
+## Reproduce current repository evidence
+
+```bash
+bash scripts/run_readiness_check.sh
+cd backend && python -m alembic upgrade head --sql
+bash -n scripts/live_migration_smoke.sh
+```
+
+Live release evidence:

 ```bash
 python scripts/capture_release_evidence.py \
@@ -55,8 +71,3 @@ python scripts/capture_release_evidence.py \
  --release-id rc-belgium-north-sea \
  --live-base-url http://192.168.10.150:1202
 ```
-
-The evidence manifest is runtime output and is deliberately ignored by Git.
-The first captured baseline reported one Alembic head (`202607160001`) and
-reachable live health/capability routes. Health truthfulness is an open RC-2
-blocker until readiness becomes fail-closed.
@@ -2,6 +2,28 @@

 ### Post-RC national data federation (2026-07-19)

+- Continued the national federation pass with official history and
+  bathymetry. Statbel's real 2021 geometry release exposed two documented
+  packaging differences: the ZIP member omits the repeated `31370` token and
+  `dt_situation` uses `YYYY/MM/DD`. The preflight now normalizes only those
+  forms and retains every existing fail-closed content check.
+- The corrected Tower fetch-only pass validated national 2021-2024 snapshots,
+  each with 19,795 retained sector geometries, before any API persistence.
+- Downloaded the fixed official SPW Geoportail bathymetry release
+  (`0a544b42-0b30-4c8e-85e7-38149b99eae0`) to persistent operator evidence.
+  SHA-256 is
+  `04122a1c5cecc7b77be025b580995d024545e82d8310e158b4c112f7fa5f8a6e`.
+  GDAL verified a 456,893 by 179,437 Float32 EPSG:3812 source at 0.5 m,
+  nodata `-9999`, with mDNG waterbed elevations.
+- Added `import_spw_bathymetry.py`, bounded COG persistence, source-specific
+  coverage resolution, raster selection/image APIs and frontend Waterbodem
+  integration. The contract exposes bed elevation, measured-cell hectares and
+  coverage only; current depth, volume and datum conversion remain blocked.
+- Added the bathymetry PNG route to the explicit non-envelope API audit set
+  after the first full gate correctly rejected its untracked binary response.
+- The complete repository gate passed 1,052 backend tests, 22 frontend tests,
+  frontend typecheck/build, one Alembic head and all readiness checks before
+  live deployment.
 - Added explicit administrative and maritime themes for persisted NGI/RBINS
  reference data and made readiness depend on analyzable themes rather than a
  raw Dataset count.
@@ -291,6 +291,15 @@ which creates the ordinary annual `datasets`, `dataset_versions` and
 year remains idempotent and previous annual snapshots are never updated or
 deleted.

+SPW bathymetry also requires no migration. The immutable official ZIP and
+bounded operator COG are filesystem artifacts. A successful API upload creates
+an ordinary raster `Dataset` and `DatasetVersion`; raster values stay in the
+versioned GeoTIFF rather than PostGIS. `source_metadata` retains EPSG:3812,
+mDNG, survey period, source checksum and bounded extent, while
+`provenance_metadata` retains the official URL, archive member, derived
+checksum and selection geometry/hash. Analysis never writes metric rows and
+never converts the vertical datum implicitly.
+
 Definitive ALZ release management uses the same persistence boundary and adds
 no migration or release table. Catalog planning, staged archive/GeoJSON,
 crop-code evidence and named review live on the filesystem. Only an approved
@@ -764,6 +764,13 @@ population queries therefore derive from one reviewed national edition rather
 than mutually incompatible regional imports. No edition is downloaded or
 promoted during application startup.

+The live national workspace contains one consistent Statbel population series
+for 2021-2025. The 2021-2024 geometry archives use the official
+`YYYY/MM/DD` situation-date notation and archive member names without a
+repeated CRS token; the fail-closed preflight normalizes only those documented
+format variants and still validates EPSG:31370, release year, schema, joins,
+totals, topology repairs and checksums.
+
 ## Wallonia PICC and Brussels UrbIS

 The official-vector registry exposes bounded, source-specific regional
@@ -792,7 +799,7 @@ profile numbers, measurement dates, available structured depth/width values
 and official document URLs. Scanned documents remain evidence; missing fields
 are not filled by fabricated OCR output.

-The following sources are audited but not yet operational:
+The following sources are audited:

 - MDK Dieptemodel Belgisch Continentaal Plat/Noordzee: 20 x 20 m continuous
  raster in LAT, exposed through WCS/WMTS. GeoIntel now has a strict-TLS,
@@ -800,8 +807,13 @@ The following sources are audited but not yet operational:
  `bathy.agentschapmdk.be` presents a certificate for `*.l27powered.eu`.
  Hostname validation therefore fails with `tls_error`; raster acquisition
  remains disabled and TLS verification cannot be bypassed.
- SPW bathymetry of navigable waterways and reservoir lakes: 0.5 m bed
-  elevation and XYZ data in mDNG.
+- SPW bathymetry of navigable waterways and reservoir lakes: the official
+  2023-05-23 GeoTIFF release is operational through
+  `scripts/import_spw_bathymetry.py`. The operator pins SHA-256
+  `04122a1c5cecc7b77be025b580995d024545e82d8310e158b4c112f7fa5f8a6e`,
+  validates the EPSG:3812/Float32/0.5 m/-9999 contract, creates only a bounded
+  COG and persists it through the canonical dataset upload API. Values are
+  bed elevations in mDNG from surveys composed over 2019-2022.
 - Port of Antwerp-Bruges periodic soundings: catalog candidate pending a
  stable public machine contract.

@@ -97,7 +97,7 @@ applicable live bounded journey fails.

 ## P1 - National Statbel population

-**State: complete; reviewed 2025 national edition materialized live.**
+**State: complete; reviewed 2021-2025 national series materialized live.**

 ### Work

@@ -122,10 +122,17 @@ applicable live bounded journey fails.

 ### Live evidence

- Reviewed dataset: `ee0a46e5-6139-4b0a-93b5-9dabb3b3dfc7`.
- 20,781 persisted sectors and all 565 municipalities are represented.
- National reconciliation: 11,825,551 inhabitants, including 7,654 explicitly
-  retained as unlocated rather than silently assigned to geometry.
+- Reviewed immutable datasets:
+  - 2021: `27b2aaba-6ec4-42d8-91c3-9085456b5e7a`;
+  - 2022: `312e8278-5552-4dd0-abf7-7e1f17d2562a`;
+  - 2023: `72bfb8a7-9ecb-462d-a622-f8cf693a3cfb`;
+  - 2024: `e7846fc6-85d1-4120-9442-e62a5b1cd346`;
+  - 2025: `ee0a46e5-6139-4b0a-93b5-9dabb3b3dfc7`.
+- The 2021-2024 editions each retain 19,795 sector geometries. The 2025
+  REDEGEO edition retains 20,781 sectors and represents all 565 municipalities.
+- The 2025 national reconciliation reports 11,825,551 inhabitants, including
+  7,654 explicitly retained as unlocated rather than silently assigned to
+  geometry.
 - A bounded Mol selection returns an explicitly labelled area-weighted estimate.

 ## P2 - Wallonia bounded topographic baseline
@@ -207,7 +214,8 @@ capabilities probe before implementation is marked operational.

 ## P4 - Maritime and bathymetry hardening

-**State: complete with MDK acquisition visibly blocked by strict-TLS evidence.**
+**State: repository-complete; SPW live acceptance is the final candidate gate
+and MDK acquisition remains visibly blocked by strict-TLS evidence.**

 ### Work

@@ -218,6 +226,9 @@ capabilities probe before implementation is marked operational.
  semantics, edition, licence and LAT evidence pass.
 - Keep depth relative to LAT separate from water-surface elevation and volume.
 - SPW mDNG bathymetry remains a separate Walloon inland/navigation source.
+- The bounded SPW operator stages the fixed official GeoTIFF archive, validates
+  its checksum and raster identity, writes a clipped COG through the canonical
+  Dataset flow and reports only waterbed elevation and surveyed coverage.

 ### Exit

@@ -142,6 +142,23 @@ Dataset/DatasetVersion plus PostGIS `vector_features` through the canonical
 persistence services. Temporary standalone preflight output does not create a
 database record and may be removed explicitly after operator review.

+SPW bathymetry follows the same immutable-source/derived-dataset separation:
+
+```text
+storage/operator-evidence/spw-bathymetry/2023-05-23/
+  raw/BATHY_50CM_ALTITUDE_DNG_GEOTIFF_3812.zip
+  derived/spw_bathymetry_{selection_hash}_3812.tif
+```
+
+The raw ZIP is pinned by official URL and SHA-256 and is never extracted as an
+8+ GiB working tree. Rasterio/GDAL reads the official GeoTIFF through
+`/vsizip/`; the operator writes a bounded compressed COG and uploads that COG
+through DatasetService. The queryable file therefore remains an ordinary
+versioned raster Dataset. Source URL, archive/member checksum, selection
+geometry/hash, survey period, EPSG:3812 and mDNG semantics are retained in
+source/provenance metadata. PNG map overlays are derived responses and are not
+authoritative storage.
+
 Governed release-decision evidence is separate from the source artifacts:

 ```text
@@ -100,9 +100,11 @@ geen open productroadmap meer.

 Bewuste, niet-blokkerende grenzen:

- Watervolume blijft niet beschikbaar zolang geen gekoppelde bathymetrie,
-  bodemhoogte en onzekerheidsmodel bestaan. Dit is geospatiale correctheid,
-  geen onafgewerkte UI.
+- De begrensde SPW-rasterflow maakt Waalse waterbodemhoogte in mDNG
+  analyseerbaar. Watervolume blijft niet beschikbaar zolang geen compatibele
+  wateroppervlaktehoogte, tijdskoppeling, datumtransformatie en
+  onzekerheidsmodel bestaan. Dit is geospatiale correctheid, geen onafgewerkte
+  UI.
 - Gebouwdetectie is operationeel maar blijft een controlekandidaat; verdere
  training gebeurt pas na nieuwe, onafhankelijke reviewdata.
 - Een echt SAM/YOLO-seg-model en een trainingsstudio blijven post-V1. De
@@ -129,7 +131,10 @@ Bewuste, niet-blokkerende grenzen:
 - [x] Expand the modern 2013-2025 land-use operator with water, built-function and transport surfaces from the retained official raster.
 - [x] Add a source inventory that separates loaded data from audited official follow-up sources.
 - [x] Add a local Ollama question window grounded in persisted GeoIntel metrics and installed server models.
- [ ] Add a governed depth/bathymetry source before exposing water volume; never infer volume from 2D GRB water geometry.
+- [x] Add a governed bounded SPW bed-elevation source with explicit mDNG and
+  survey-period semantics; never infer depth or volume from 2D water geometry.
+- [ ] Add a compatible water-surface, time, vertical-datum and uncertainty
+  contract before exposing water volume.
 - [x] Integrate governed Waterinfo/VMM annual station series without presenting point measurements as area-wide water volume.
 - [x] Add bounded historical orthophoto acquisition for official 1971-2025 products with a map overlay and no current-GRB QA on old imagery.
 - [x] Add BWK/Natura 2000 through an explicit provider/operator contract.
@@ -811,9 +816,9 @@ This file now starts with the current implementation status. Older preparation/b
 - [ ] Add bounded MDK GeoTIFF acquisition only after live CRS, LAT, nodata and
  response-limit validation. Current official endpoint is blocked by TLS
  hostname mismatch and unavailable capabilities; never bypass verification.
- [ ] Add SPW staged-download adapter with mDNG metadata and survey-epoch
+- [x] Add SPW staged-download adapter with mDNG metadata and survey-epoch
  coverage validation.
- [ ] Add authoritative territorial-sea, EEZ and continental-shelf boundary
+- [x] Add authoritative territorial-sea, EEZ and continental-shelf boundary
  layers with legally accurate labels.
 - [ ] Add vertical-datum conversion only when authoritative transforms and
  uncertainty tests exist; never merge TAW, LAT and mDNG implicitly.