Add governed SPW bathymetry analysis
This commit is contained in:
@@ -7,6 +7,29 @@
|
|||||||
|
|
||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## National history and governed SPW bathymetry (2026-07-19)
|
||||||
|
|
||||||
|
- Accepted the two documented official Statbel geometry-archive variants:
|
||||||
|
internal member names with or without the repeated `31370` token and
|
||||||
|
situation dates in ISO or `YYYY/MM/DD` notation. CRS, release-year, schema,
|
||||||
|
join, topology, total and checksum validation remain fail-closed.
|
||||||
|
- Prepared the official Statbel 2021-2024 national snapshots so the existing
|
||||||
|
2025 Belgium population layer can become one consistent five-edition
|
||||||
|
evolution series.
|
||||||
|
- Added a pinned SPW bathymetry operator for the official 2023-05-23
|
||||||
|
0.5-m GeoTIFF release. It validates SHA-256, archive safety, EPSG:3812,
|
||||||
|
Float32, nodata and value bounds, reads through `/vsizip/`, creates only a
|
||||||
|
bounded COG and persists through the canonical Dataset API.
|
||||||
|
- Added persisted SPW bathymetry selection and PNG routes, with waterbed-height
|
||||||
|
percentiles in mDNG, surveyed hectares and coverage. Current water depth,
|
||||||
|
volume and vertical-datum conversion remain explicitly unsupported.
|
||||||
|
- Integrated ready SPW rasters into the Waterbodem map theme, MapLibre image
|
||||||
|
overlay, rectangle analysis and source portfolio while excluding analytical
|
||||||
|
bathymetry rasters from detection imagery.
|
||||||
|
- Kept MDK North Sea acquisition blocked behind strict TLS and official
|
||||||
|
low-resolution data-request prerequisites; no certificate bypass or
|
||||||
|
synthesized North Sea depth was introduced.
|
||||||
|
|
||||||
## Post-RC Belgium data federation (2026-07-19)
|
## Post-RC Belgium data federation (2026-07-19)
|
||||||
|
|
||||||
- Made persisted NGI administrative, RBINS marine reporting and Belgian
|
- Made persisted NGI administrative, RBINS marine reporting and Belgian
|
||||||
|
|||||||
+25
-1
@@ -1806,7 +1806,8 @@ matched name with `--show-names`.
|
|||||||
performs a bounded official VHA ArcGIS query, exact persisted-Area clipping,
|
performs a bounded official VHA ArcGIS query, exact persisted-Area clipping,
|
||||||
watercourse-name normalization and ordinary Dataset/VectorFeature persistence.
|
watercourse-name normalization and ordinary Dataset/VectorFeature persistence.
|
||||||
`GET /api/v1/projects/{project_id}/datasets/bathymetry/sources` reports VHA as
|
`GET /api/v1/projects/{project_id}/datasets/bathymetry/sources` reports VHA as
|
||||||
operational and the audited MDK/SPW candidates as unavailable for acquisition.
|
operational, MDK as probe-only and the pinned SPW raster operator as
|
||||||
|
operational.
|
||||||
|
|
||||||
Runtime controls are `BATHYMETRY_PROFILES_ENABLED`,
|
Runtime controls are `BATHYMETRY_PROFILES_ENABLED`,
|
||||||
`BATHYMETRY_PROFILES_LAYER_URL`, `BATHYMETRY_WATERCOURSE_LAYER_URL`,
|
`BATHYMETRY_PROFILES_LAYER_URL`, `BATHYMETRY_WATERCOURSE_LAYER_URL`,
|
||||||
@@ -1854,6 +1855,29 @@ readiness state such as TLS or endpoint failure. Runtime controls are
|
|||||||
`MDK_BATHYMETRY_PROBE_TIMEOUT_SECONDS` and
|
`MDK_BATHYMETRY_PROBE_TIMEOUT_SECONDS` and
|
||||||
`MDK_BATHYMETRY_PROBE_MAX_RESPONSE_MB`. TLS verification cannot be disabled.
|
`MDK_BATHYMETRY_PROBE_MAX_RESPONSE_MB`. TLS verification cannot be disabled.
|
||||||
|
|
||||||
|
### SPW waterbed raster
|
||||||
|
|
||||||
|
The official 2023-05-23 SPW bathymetry ZIP is integrated only through the
|
||||||
|
bounded operator. Stage the immutable ZIP under persistent storage and run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec geointel python /app/scripts/import_spw_bathymetry.py \
|
||||||
|
--base-url http://127.0.0.1:8000 \
|
||||||
|
--project-name "Belgium and North Sea Workbench" \
|
||||||
|
--area "RC Golden - Wallonia urban-rural" \
|
||||||
|
--bbox 4.85,50.45,4.87,50.47 \
|
||||||
|
--raw-zip /app/storage/operator-evidence/spw-bathymetry/2023-05-23/raw/BATHY_50CM_ALTITUDE_DNG_GEOTIFF_3812.zip \
|
||||||
|
--output-dir /app/storage/operator-evidence/spw-bathymetry/2023-05-23/derived
|
||||||
|
```
|
||||||
|
|
||||||
|
The script validates the pinned official checksum, safe archive members,
|
||||||
|
EPSG:3812, one Float32 band, approximately 0.5 m cells and nodata `-9999`.
|
||||||
|
It then creates a bounded COG and uploads it through `/datasets/upload`.
|
||||||
|
`POST .../raster/bathymetry/select` returns waterbed elevation in mDNG,
|
||||||
|
surveyed surface and coverage. Current depth, volume and datum conversion stay
|
||||||
|
unavailable without a compatible water-surface source. Selection analysis is
|
||||||
|
bounded by `BATHYMETRY_RASTER_MAX_PIXELS` (30 million by default).
|
||||||
|
|
||||||
## Governed regional official-vector acquisition
|
## Governed regional official-vector acquisition
|
||||||
|
|
||||||
The thematic raster registry includes forest and agricultural land-use masks
|
The thematic raster registry includes forest and agricultural land-use masks
|
||||||
|
|||||||
@@ -48,6 +48,8 @@ from app.schemas import (
|
|||||||
FloodHazardSelectionRequest,
|
FloodHazardSelectionRequest,
|
||||||
BathymetryPartitionFinalizeRequest,
|
BathymetryPartitionFinalizeRequest,
|
||||||
BathymetryProfileAcquireRequest,
|
BathymetryProfileAcquireRequest,
|
||||||
|
BathymetryRasterSelectionRequest,
|
||||||
|
BathymetryRasterSelectionResponse,
|
||||||
ThematicRasterAcquireRequest,
|
ThematicRasterAcquireRequest,
|
||||||
ThematicRasterProductRead,
|
ThematicRasterProductRead,
|
||||||
ThematicRasterSelectionResponse,
|
ThematicRasterSelectionResponse,
|
||||||
@@ -89,6 +91,7 @@ from app.services.terrain_analysis_service import TerrainAnalysisService
|
|||||||
from app.services.flood_hazard_acquisition_service import FloodHazardAcquisitionService
|
from app.services.flood_hazard_acquisition_service import FloodHazardAcquisitionService
|
||||||
from app.services.flood_hazard_analysis_service import FloodHazardAnalysisService
|
from app.services.flood_hazard_analysis_service import FloodHazardAnalysisService
|
||||||
from app.services.bathymetry_profile_acquisition_service import BathymetryProfileAcquisitionService
|
from app.services.bathymetry_profile_acquisition_service import BathymetryProfileAcquisitionService
|
||||||
|
from app.services.bathymetry_raster_analysis_service import BathymetryRasterAnalysisService
|
||||||
from app.services.mdk_bathymetry_probe_service import MdkBathymetryProbeService
|
from app.services.mdk_bathymetry_probe_service import MdkBathymetryProbeService
|
||||||
from app.services.thematic_raster_acquisition_service import ThematicRasterAcquisitionService
|
from app.services.thematic_raster_acquisition_service import ThematicRasterAcquisitionService
|
||||||
from app.services.thematic_raster_analysis_service import ThematicRasterAnalysisService
|
from app.services.thematic_raster_analysis_service import ThematicRasterAnalysisService
|
||||||
@@ -888,6 +891,33 @@ def raster_terrain_image(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/datasets/{dataset_id}/raster/bathymetry/select",
|
||||||
|
response_model=Envelope[BathymetryRasterSelectionResponse],
|
||||||
|
)
|
||||||
|
def raster_bathymetry_selection(
|
||||||
|
project_id: UUID,
|
||||||
|
dataset_id: UUID,
|
||||||
|
payload: BathymetryRasterSelectionRequest,
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
):
|
||||||
|
return envelope(BathymetryRasterAnalysisService.analyze(db, project_id, dataset_id, payload))
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/datasets/{dataset_id}/raster/bathymetry/image")
|
||||||
|
def raster_bathymetry_image(
|
||||||
|
project_id: UUID,
|
||||||
|
dataset_id: UUID,
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
):
|
||||||
|
content = BathymetryRasterAnalysisService.render_png(db, project_id, dataset_id)
|
||||||
|
return Response(
|
||||||
|
content=content,
|
||||||
|
media_type="image/png",
|
||||||
|
headers={"Cache-Control": "private, max-age=86400"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.post(
|
@router.post(
|
||||||
"/datasets/{dataset_id}/raster/flood-hazard/select",
|
"/datasets/{dataset_id}/raster/flood-hazard/select",
|
||||||
response_model=Envelope[FloodHazardSelectionResponse],
|
response_model=Envelope[FloodHazardSelectionResponse],
|
||||||
|
|||||||
@@ -220,6 +220,11 @@ class Settings(BaseSettings):
|
|||||||
le=256,
|
le=256,
|
||||||
validation_alias="BATHYMETRY_PROFILES_MAX_RESPONSE_MB",
|
validation_alias="BATHYMETRY_PROFILES_MAX_RESPONSE_MB",
|
||||||
)
|
)
|
||||||
|
bathymetry_raster_max_pixels: int = Field(
|
||||||
|
default=30_000_000,
|
||||||
|
ge=1,
|
||||||
|
validation_alias="BATHYMETRY_RASTER_MAX_PIXELS",
|
||||||
|
)
|
||||||
mdk_bathymetry_probe_enabled: bool = Field(default=True, validation_alias="MDK_BATHYMETRY_PROBE_ENABLED")
|
mdk_bathymetry_probe_enabled: bool = Field(default=True, validation_alias="MDK_BATHYMETRY_PROBE_ENABLED")
|
||||||
mdk_bathymetry_wcs_url: str = Field(
|
mdk_bathymetry_wcs_url: str = Field(
|
||||||
default="https://bathy.agentschapmdk.be/spatialfusionserver/services/ows/wcs/EL_wcs",
|
default="https://bathy.agentschapmdk.be/spatialfusionserver/services/ows/wcs/EL_wcs",
|
||||||
|
|||||||
@@ -93,6 +93,10 @@ from .bathymetry import (
|
|||||||
BathymetryPartitionFinalizationResult,
|
BathymetryPartitionFinalizationResult,
|
||||||
BathymetryProfileAcquireRequest,
|
BathymetryProfileAcquireRequest,
|
||||||
BathymetryProfileAcquisitionResult,
|
BathymetryProfileAcquisitionResult,
|
||||||
|
BathymetryRasterMetric,
|
||||||
|
BathymetryRasterSelectionRequest,
|
||||||
|
BathymetryRasterSelectionResponse,
|
||||||
|
BathymetryRasterSelectionSummary,
|
||||||
BathymetrySourceProbeRead,
|
BathymetrySourceProbeRead,
|
||||||
BathymetrySourceRead,
|
BathymetrySourceRead,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -109,3 +109,43 @@ class BathymetrySourceProbeRead(BaseModel):
|
|||||||
checked_at: datetime
|
checked_at: datetime
|
||||||
message: str
|
message: str
|
||||||
limitation_message: str
|
limitation_message: str
|
||||||
|
|
||||||
|
|
||||||
|
class BathymetryRasterSelectionRequest(BaseModel):
|
||||||
|
bbox: VectorSelectionBBox
|
||||||
|
area_id: UUID | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class BathymetryRasterMetric(BaseModel):
|
||||||
|
metric_key: str
|
||||||
|
metric_label: str
|
||||||
|
metric_value: float
|
||||||
|
metric_unit: str
|
||||||
|
aggregation_method: str
|
||||||
|
is_estimate: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class BathymetryRasterSelectionSummary(BaseModel):
|
||||||
|
metric_label: str
|
||||||
|
metric_value: float
|
||||||
|
metric_unit: str
|
||||||
|
aggregation_method: str
|
||||||
|
primary_metric_key: str
|
||||||
|
metrics: list[BathymetryRasterMetric]
|
||||||
|
|
||||||
|
|
||||||
|
class BathymetryRasterSelectionResponse(BaseModel):
|
||||||
|
dataset_id: UUID
|
||||||
|
product_key: str
|
||||||
|
selection_bbox: VectorSelectionBBox
|
||||||
|
selection_area_id: UUID | None = None
|
||||||
|
selected_cell_count: int = Field(ge=1)
|
||||||
|
valid_cell_count: int = Field(ge=1)
|
||||||
|
coverage_ratio: float = Field(ge=0, le=1)
|
||||||
|
resolution_m: float = Field(gt=0)
|
||||||
|
vertical_reference: str
|
||||||
|
survey_period: str
|
||||||
|
summary: BathymetryRasterSelectionSummary
|
||||||
|
unsupported_metrics: list[str]
|
||||||
|
limitation_message: str
|
||||||
|
generated_at: str
|
||||||
|
|||||||
@@ -91,20 +91,21 @@ class BathymetryProfileAcquisitionService:
|
|||||||
"authority_level": "authoritative",
|
"authority_level": "authoritative",
|
||||||
"geographic_coverage": "Waalse bevaarbare waterwegen en stuwmeren met uitgevoerde opmetingen",
|
"geographic_coverage": "Waalse bevaarbare waterwegen en stuwmeren met uitgevoerde opmetingen",
|
||||||
"data_kind": "bodemhoogteraster en XYZ-puntenwolk",
|
"data_kind": "bodemhoogteraster en XYZ-puntenwolk",
|
||||||
"query_modes": ["download", "arcgis_map_service"],
|
"query_modes": ["operator_archive", "bounded_raster", "arcgis_map_service"],
|
||||||
"vertical_reference": "mDNG",
|
"vertical_reference": "mDNG",
|
||||||
"horizontal_crs": "EPSG:3812; visualisatieservice kan EPSG:31370 aanbieden",
|
"horizontal_crs": "EPSG:3812; visualisatieservice kan EPSG:31370 aanbieden",
|
||||||
"native_resolution": "0,5 m",
|
"native_resolution": "0,5 m",
|
||||||
"integration_status": "available_not_integrated",
|
"integration_status": "operational",
|
||||||
"acquisition_supported": False,
|
"acquisition_supported": True,
|
||||||
"configured": False,
|
"configured": True,
|
||||||
"service_url": "https://geoservices.wallonie.be/arcgis/rest/services/EAU/BATHY/MapServer",
|
"service_url": "https://geoservices.wallonie.be/arcgis/rest/services/EAU/BATHY/MapServer",
|
||||||
"catalog_url": "https://geoportail.wallonie.be/catalogue/c450c28f-d357-48af-8423-62d524632cf9.html",
|
"catalog_url": "https://geoportail.wallonie.be/catalogue/0a544b42-0b30-4c8e-85e7-38149b99eae0.html",
|
||||||
"attribution": "Service public de Wallonie",
|
"attribution": "Service public de Wallonie",
|
||||||
"license_note": "CC BY 4.0 volgens de officiële Geoportail-metadata.",
|
"license_note": "CC BY 4.0 volgens de officiële Geoportail-metadata.",
|
||||||
"limitation_message": (
|
"limitation_message": (
|
||||||
"Dekking en meetjaar verschillen per vaarweg of reservoir. Integratie vereist een beheerde "
|
"De gepinde officiële release kan begrensd als raster worden geïmporteerd via de operator. "
|
||||||
"download- en mosaïekstroom plus expliciete omzetting van mDNG."
|
"Dekking verschilt per vaarweg; de waarden zijn bodemhoogtes in mDNG uit 2019-2022, "
|
||||||
|
"zonder stilzwijgende datumconversie of afleiding van actuele waterdiepte."
|
||||||
),
|
),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -0,0 +1,359 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import math
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from uuid import UUID
|
||||||
|
|
||||||
|
from geoalchemy2.shape import to_shape
|
||||||
|
from pyproj import Transformer
|
||||||
|
from shapely.geometry import box, mapping
|
||||||
|
from shapely.ops import transform as shapely_transform
|
||||||
|
|
||||||
|
from app.core.config import Settings, get_settings
|
||||||
|
from app.core.errors import AppError
|
||||||
|
from app.models import Area, Dataset
|
||||||
|
from app.schemas.bathymetry import (
|
||||||
|
BathymetryRasterMetric,
|
||||||
|
BathymetryRasterSelectionRequest,
|
||||||
|
BathymetryRasterSelectionResponse,
|
||||||
|
BathymetryRasterSelectionSummary,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class BathymetryRasterAnalysisService:
|
||||||
|
SOURCE_NAME = "spw_bathymetry"
|
||||||
|
PRODUCT_KEY = "spw_bathymetry_50cm_mdng"
|
||||||
|
UNSUPPORTED_METRICS = [
|
||||||
|
"current_water_depth_m",
|
||||||
|
"water_volume_m3",
|
||||||
|
"vertical_datum_conversion",
|
||||||
|
]
|
||||||
|
LIMITATION = (
|
||||||
|
"De rasterwaarden zijn waterbodemhoogtes in mDNG uit een samengestelde SPW-opmeting "
|
||||||
|
"(2019-2022). Zonder een gelijktijdig waterpeil zijn actuele waterdiepte en watervolume "
|
||||||
|
"niet berekenbaar. mDNG wordt niet stilzwijgend naar TAW, LAT of een ander verticaal datum omgezet."
|
||||||
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _load_dataset(db, project_id: UUID, dataset_id: UUID) -> Dataset:
|
||||||
|
dataset = db.get(Dataset, dataset_id)
|
||||||
|
if not dataset or dataset.project_id != project_id:
|
||||||
|
raise AppError(code="DATASET_NOT_FOUND", message="Dataset not found", status_code=404)
|
||||||
|
if dataset.dataset_type != "raster" or dataset.source_name != BathymetryRasterAnalysisService.SOURCE_NAME:
|
||||||
|
raise AppError(
|
||||||
|
code="INVALID_BATHYMETRY_RASTER_DATASET",
|
||||||
|
message="Bathymetry analysis requires a governed SPW bathymetry raster dataset",
|
||||||
|
status_code=400,
|
||||||
|
)
|
||||||
|
if dataset.status != "ready" or not dataset.storage_path or not Path(dataset.storage_path).is_file():
|
||||||
|
raise AppError(
|
||||||
|
code="DATASET_FILE_MISSING",
|
||||||
|
message="Persisted bathymetry raster file is unavailable",
|
||||||
|
status_code=404,
|
||||||
|
)
|
||||||
|
return dataset
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _metadata(dataset: Dataset) -> dict:
|
||||||
|
metadata = dataset.source_metadata if isinstance(dataset.source_metadata, dict) else {}
|
||||||
|
if (
|
||||||
|
metadata.get("product_key") != BathymetryRasterAnalysisService.PRODUCT_KEY
|
||||||
|
or metadata.get("theme") != "bathymetry"
|
||||||
|
or metadata.get("value_semantics") != "bed_elevation"
|
||||||
|
or metadata.get("vertical_reference") != "mDNG"
|
||||||
|
or metadata.get("source_crs") != "EPSG:3812"
|
||||||
|
):
|
||||||
|
raise AppError(
|
||||||
|
code="INVALID_BATHYMETRY_RASTER_METADATA",
|
||||||
|
message="Bathymetry raster provenance or value semantics are incomplete",
|
||||||
|
status_code=409,
|
||||||
|
)
|
||||||
|
return metadata
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _selection_geometry(db, project_id: UUID, payload: BathymetryRasterSelectionRequest):
|
||||||
|
selection = box(payload.bbox.min_x, payload.bbox.min_y, payload.bbox.max_x, payload.bbox.max_y)
|
||||||
|
if payload.area_id is None:
|
||||||
|
return selection
|
||||||
|
area = db.get(Area, payload.area_id)
|
||||||
|
if not area:
|
||||||
|
raise AppError(code="AREA_NOT_FOUND", message="Area not found", status_code=404)
|
||||||
|
if area.project_id != project_id:
|
||||||
|
raise AppError(
|
||||||
|
code="INVALID_DATASET_SCOPE",
|
||||||
|
message="Area does not belong to this project",
|
||||||
|
status_code=400,
|
||||||
|
)
|
||||||
|
selection = selection.intersection(to_shape(area.geometry))
|
||||||
|
if selection.is_empty or selection.area <= 0:
|
||||||
|
raise AppError(
|
||||||
|
code="BATHYMETRY_SELECTION_OUTSIDE_AREA",
|
||||||
|
message="Selection does not overlap the selected work area",
|
||||||
|
status_code=422,
|
||||||
|
)
|
||||||
|
return selection
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def analyze(
|
||||||
|
db,
|
||||||
|
project_id: UUID,
|
||||||
|
dataset_id: UUID,
|
||||||
|
payload: BathymetryRasterSelectionRequest,
|
||||||
|
*,
|
||||||
|
settings: Settings | None = None,
|
||||||
|
) -> dict:
|
||||||
|
resolved_settings = settings or get_settings()
|
||||||
|
dataset = BathymetryRasterAnalysisService._load_dataset(db, project_id, dataset_id)
|
||||||
|
source_metadata = BathymetryRasterAnalysisService._metadata(dataset)
|
||||||
|
selection_4326 = BathymetryRasterAnalysisService._selection_geometry(db, project_id, payload)
|
||||||
|
try:
|
||||||
|
import numpy as np
|
||||||
|
import rasterio
|
||||||
|
from rasterio.features import geometry_mask
|
||||||
|
from rasterio.mask import mask
|
||||||
|
except ImportError as exc:
|
||||||
|
raise AppError(
|
||||||
|
code="RASTER_PROCESSING_UNAVAILABLE",
|
||||||
|
message="Rasterio and numpy are required for bathymetry analysis",
|
||||||
|
status_code=503,
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
try:
|
||||||
|
with rasterio.open(dataset.storage_path) as source:
|
||||||
|
if source.crs is None or source.crs.to_epsg() != 3812:
|
||||||
|
raise AppError(
|
||||||
|
code="INVALID_DATASET_CRS",
|
||||||
|
message="SPW bathymetry raster CRS must be EPSG:3812",
|
||||||
|
status_code=409,
|
||||||
|
)
|
||||||
|
if source.count != 1:
|
||||||
|
raise AppError(
|
||||||
|
code="INVALID_BATHYMETRY_RASTER_BANDS",
|
||||||
|
message="SPW bathymetry requires one bed-elevation band",
|
||||||
|
status_code=409,
|
||||||
|
)
|
||||||
|
transformer = Transformer.from_crs("EPSG:4326", source.crs, always_xy=True)
|
||||||
|
selection_metric = shapely_transform(transformer.transform, selection_4326)
|
||||||
|
analysis_geometry = selection_metric.intersection(box(*source.bounds))
|
||||||
|
if analysis_geometry.is_empty or analysis_geometry.area <= 0:
|
||||||
|
raise AppError(
|
||||||
|
code="BATHYMETRY_SELECTION_OUTSIDE_DATASET",
|
||||||
|
message="Selection does not overlap the persisted bathymetry raster",
|
||||||
|
status_code=422,
|
||||||
|
)
|
||||||
|
min_x, min_y, max_x, max_y = analysis_geometry.bounds
|
||||||
|
expected_cells = math.ceil((max_x - min_x) / abs(source.res[0])) * math.ceil(
|
||||||
|
(max_y - min_y) / abs(source.res[1])
|
||||||
|
)
|
||||||
|
if expected_cells > resolved_settings.bathymetry_raster_max_pixels:
|
||||||
|
raise AppError(
|
||||||
|
code="BATHYMETRY_SELECTION_TOO_LARGE",
|
||||||
|
message="Bathymetry analysis exceeds the configured raster cell limit",
|
||||||
|
details={
|
||||||
|
"pixel_count": expected_cells,
|
||||||
|
"max_pixels": resolved_settings.bathymetry_raster_max_pixels,
|
||||||
|
},
|
||||||
|
status_code=422,
|
||||||
|
)
|
||||||
|
clipped, clipped_transform = mask(
|
||||||
|
source,
|
||||||
|
[mapping(analysis_geometry)],
|
||||||
|
crop=True,
|
||||||
|
filled=False,
|
||||||
|
indexes=[1],
|
||||||
|
)
|
||||||
|
band = np.ma.asarray(clipped[0], dtype="float64")
|
||||||
|
raw = band.filled(np.nan)
|
||||||
|
selected_cells = geometry_mask(
|
||||||
|
[mapping(analysis_geometry)],
|
||||||
|
out_shape=band.shape,
|
||||||
|
transform=clipped_transform,
|
||||||
|
invert=True,
|
||||||
|
)
|
||||||
|
valid_cells = selected_cells & ~np.ma.getmaskarray(band) & np.isfinite(raw)
|
||||||
|
if source.nodata is not None:
|
||||||
|
valid_cells &= ~np.isclose(raw, float(source.nodata))
|
||||||
|
values = raw[valid_cells]
|
||||||
|
if values.size == 0:
|
||||||
|
raise AppError(
|
||||||
|
code="BATHYMETRY_NO_VALID_DATA",
|
||||||
|
message="No surveyed waterbed cells occur in this selection",
|
||||||
|
status_code=422,
|
||||||
|
)
|
||||||
|
resolution_x = abs(float(source.res[0]))
|
||||||
|
resolution_y = abs(float(source.res[1]))
|
||||||
|
cell_area_m2 = resolution_x * resolution_y
|
||||||
|
except AppError:
|
||||||
|
raise
|
||||||
|
except Exception as exc:
|
||||||
|
raise AppError(
|
||||||
|
code="BATHYMETRY_ANALYSIS_FAILED",
|
||||||
|
message="The persisted bathymetry raster could not be analysed",
|
||||||
|
details={"reason": str(exc)},
|
||||||
|
status_code=500,
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
def metric(key: str, label: str, value: float, unit: str, method: str) -> BathymetryRasterMetric:
|
||||||
|
return BathymetryRasterMetric(
|
||||||
|
metric_key=key,
|
||||||
|
metric_label=label,
|
||||||
|
metric_value=round(float(value), 4),
|
||||||
|
metric_unit=unit,
|
||||||
|
aggregation_method=method,
|
||||||
|
)
|
||||||
|
|
||||||
|
selected_cell_count = int(selected_cells.sum())
|
||||||
|
valid_cell_count = int(values.size)
|
||||||
|
vertical_unit = str(source_metadata["vertical_reference"])
|
||||||
|
coverage_ratio = valid_cell_count / max(1, selected_cell_count)
|
||||||
|
metrics = [
|
||||||
|
metric(
|
||||||
|
"bed_elevation_mean_m",
|
||||||
|
"Gemiddelde waterbodemhoogte",
|
||||||
|
values.mean(),
|
||||||
|
f"m {vertical_unit}",
|
||||||
|
"mean_valid_source_cells",
|
||||||
|
),
|
||||||
|
metric(
|
||||||
|
"bed_elevation_min_m",
|
||||||
|
"Laagste waterbodemhoogte",
|
||||||
|
values.min(),
|
||||||
|
f"m {vertical_unit}",
|
||||||
|
"minimum_valid_source_cells",
|
||||||
|
),
|
||||||
|
metric(
|
||||||
|
"bed_elevation_max_m",
|
||||||
|
"Hoogste waterbodemhoogte",
|
||||||
|
values.max(),
|
||||||
|
f"m {vertical_unit}",
|
||||||
|
"maximum_valid_source_cells",
|
||||||
|
),
|
||||||
|
metric(
|
||||||
|
"bed_elevation_p10_m",
|
||||||
|
"10e percentiel waterbodemhoogte",
|
||||||
|
np.percentile(values, 10),
|
||||||
|
f"m {vertical_unit}",
|
||||||
|
"percentile_10_valid_source_cells",
|
||||||
|
),
|
||||||
|
metric(
|
||||||
|
"bed_elevation_p90_m",
|
||||||
|
"90e percentiel waterbodemhoogte",
|
||||||
|
np.percentile(values, 90),
|
||||||
|
f"m {vertical_unit}",
|
||||||
|
"percentile_90_valid_source_cells",
|
||||||
|
),
|
||||||
|
metric(
|
||||||
|
"surveyed_bed_surface_ha",
|
||||||
|
"Oppervlakte met gemeten waterbodem",
|
||||||
|
valid_cell_count * cell_area_m2 / 10_000.0,
|
||||||
|
"ha",
|
||||||
|
"valid_source_cells_times_cell_area",
|
||||||
|
),
|
||||||
|
metric(
|
||||||
|
"bathymetry_coverage_pct",
|
||||||
|
"Dekking waterbodemmeting",
|
||||||
|
coverage_ratio * 100.0,
|
||||||
|
"%",
|
||||||
|
"valid_source_cells_divided_by_selected_cells",
|
||||||
|
),
|
||||||
|
]
|
||||||
|
primary = metrics[0]
|
||||||
|
response = BathymetryRasterSelectionResponse(
|
||||||
|
dataset_id=dataset.id,
|
||||||
|
product_key=BathymetryRasterAnalysisService.PRODUCT_KEY,
|
||||||
|
selection_bbox=payload.bbox,
|
||||||
|
selection_area_id=payload.area_id,
|
||||||
|
selected_cell_count=selected_cell_count,
|
||||||
|
valid_cell_count=valid_cell_count,
|
||||||
|
coverage_ratio=round(coverage_ratio, 6),
|
||||||
|
resolution_m=round(max(resolution_x, resolution_y), 4),
|
||||||
|
vertical_reference=vertical_unit,
|
||||||
|
survey_period=str(source_metadata.get("survey_period") or "2019-2022"),
|
||||||
|
summary=BathymetryRasterSelectionSummary(
|
||||||
|
metric_label=primary.metric_label,
|
||||||
|
metric_value=primary.metric_value,
|
||||||
|
metric_unit=primary.metric_unit,
|
||||||
|
aggregation_method=primary.aggregation_method,
|
||||||
|
primary_metric_key=primary.metric_key,
|
||||||
|
metrics=metrics,
|
||||||
|
),
|
||||||
|
unsupported_metrics=BathymetryRasterAnalysisService.UNSUPPORTED_METRICS,
|
||||||
|
limitation_message=BathymetryRasterAnalysisService.LIMITATION,
|
||||||
|
generated_at=datetime.now(UTC).isoformat(),
|
||||||
|
)
|
||||||
|
return response.model_dump(mode="json")
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def render_png(db, project_id: UUID, dataset_id: UUID, *, max_dimension: int = 1800) -> bytes:
|
||||||
|
dataset = BathymetryRasterAnalysisService._load_dataset(db, project_id, dataset_id)
|
||||||
|
BathymetryRasterAnalysisService._metadata(dataset)
|
||||||
|
try:
|
||||||
|
import numpy as np
|
||||||
|
import rasterio
|
||||||
|
from PIL import Image
|
||||||
|
from rasterio.enums import Resampling
|
||||||
|
except ImportError as exc:
|
||||||
|
raise AppError(
|
||||||
|
code="RASTER_PROCESSING_UNAVAILABLE",
|
||||||
|
message="Rasterio, numpy and Pillow are required for bathymetry rendering",
|
||||||
|
status_code=503,
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
try:
|
||||||
|
with rasterio.open(dataset.storage_path) as source:
|
||||||
|
scale = min(1.0, max_dimension / max(source.width, source.height))
|
||||||
|
width = max(1, round(source.width * scale))
|
||||||
|
height = max(1, round(source.height * scale))
|
||||||
|
data = source.read(
|
||||||
|
1,
|
||||||
|
out_shape=(height, width),
|
||||||
|
masked=True,
|
||||||
|
resampling=Resampling.bilinear,
|
||||||
|
)
|
||||||
|
values = np.asarray(data.filled(np.nan), dtype="float64")
|
||||||
|
valid = np.isfinite(values) & ~np.ma.getmaskarray(data)
|
||||||
|
if source.nodata is not None:
|
||||||
|
valid &= ~np.isclose(values, float(source.nodata))
|
||||||
|
if not valid.any():
|
||||||
|
raise AppError(
|
||||||
|
code="BATHYMETRY_NO_VALID_DATA",
|
||||||
|
message="Bathymetry raster contains no renderable cells",
|
||||||
|
status_code=422,
|
||||||
|
)
|
||||||
|
low, high = np.percentile(values[valid], [2, 98])
|
||||||
|
if high <= low:
|
||||||
|
high = low + 1.0
|
||||||
|
normalized = np.clip((values - low) / (high - low), 0.0, 1.0)
|
||||||
|
normalized = np.where(valid, normalized, 0.0)
|
||||||
|
stops = np.asarray([0.0, 0.35, 0.7, 1.0])
|
||||||
|
colors = np.asarray(
|
||||||
|
[
|
||||||
|
[8, 47, 73],
|
||||||
|
[15, 118, 140],
|
||||||
|
[103, 190, 170],
|
||||||
|
[236, 224, 163],
|
||||||
|
],
|
||||||
|
dtype="float64",
|
||||||
|
)
|
||||||
|
rgba = np.zeros((height, width, 4), dtype="uint8")
|
||||||
|
for channel in range(3):
|
||||||
|
rgba[:, :, channel] = np.interp(
|
||||||
|
normalized,
|
||||||
|
stops,
|
||||||
|
colors[:, channel],
|
||||||
|
).astype("uint8")
|
||||||
|
rgba[:, :, 3] = np.where(valid, 220, 0).astype("uint8")
|
||||||
|
output = io.BytesIO()
|
||||||
|
Image.fromarray(rgba).save(output, format="PNG", optimize=True)
|
||||||
|
return output.getvalue()
|
||||||
|
except AppError:
|
||||||
|
raise
|
||||||
|
except Exception as exc:
|
||||||
|
raise AppError(
|
||||||
|
code="BATHYMETRY_PREVIEW_FAILED",
|
||||||
|
message="The persisted bathymetry raster could not be rendered",
|
||||||
|
details={"reason": str(exc)},
|
||||||
|
status_code=500,
|
||||||
|
) from exc
|
||||||
@@ -233,11 +233,11 @@ SOURCE_DEFINITIONS = (
|
|||||||
attribution="Service public de Wallonie",
|
attribution="Service public de Wallonie",
|
||||||
license_note="Consult the license of each Geoportail Wallonie product.",
|
license_note="Consult the license of each Geoportail Wallonie product.",
|
||||||
limitation_message=(
|
limitation_message=(
|
||||||
"Bounded PICC buildings, road axes and hydrography are operational; "
|
"Bounded PICC buildings, road axes, hydrography and operator-imported SPW bathymetry are operational; "
|
||||||
"other Walloon themes remain unavailable until separately governed."
|
"other Walloon themes remain unavailable until separately governed."
|
||||||
),
|
),
|
||||||
materialized_source_names=("spw_picc",),
|
materialized_source_names=("spw_picc", "spw_bathymetry"),
|
||||||
operational_themes=("buildings", "roads", "surface_water"),
|
operational_themes=("buildings", "roads", "surface_water", "bathymetry"),
|
||||||
),
|
),
|
||||||
_contract(
|
_contract(
|
||||||
source_name="urbis",
|
source_name="urbis",
|
||||||
@@ -348,15 +348,16 @@ FLANDERS_THEME_DATASETS: dict[str, dict[str, tuple[str, ...]]] = {
|
|||||||
"flood_climate": {"vmm_flood_hazard": ()},
|
"flood_climate": {"vmm_flood_hazard": ()},
|
||||||
}
|
}
|
||||||
|
|
||||||
REGIONAL_THEME_DATASETS: dict[str, dict[str, tuple[str, ...]]] = {
|
REGIONAL_THEME_DATASETS: dict[str, dict[str, dict[str, tuple[str, ...]]]] = {
|
||||||
"spw_geoportail": {
|
"spw_geoportail": {
|
||||||
"buildings": ("buildings",),
|
"buildings": {"spw_picc": ("buildings",)},
|
||||||
"roads": ("roads",),
|
"roads": {"spw_picc": ("roads",)},
|
||||||
"surface_water": ("water",),
|
"surface_water": {"spw_picc": ("water",)},
|
||||||
|
"bathymetry": {"spw_bathymetry": ()},
|
||||||
},
|
},
|
||||||
"urbis": {
|
"urbis": {
|
||||||
"buildings": ("buildings",),
|
"buildings": {"urbis": ("buildings",)},
|
||||||
"parcels": ("parcels",),
|
"parcels": {"urbis": ("parcels",)},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -437,10 +438,10 @@ class CoverageRegistryService:
|
|||||||
continue
|
continue
|
||||||
layer_names = theme_sources[dataset.source_name]
|
layer_names = theme_sources[dataset.source_name]
|
||||||
elif definition.contract.source_name in REGIONAL_THEME_DATASETS:
|
elif definition.contract.source_name in REGIONAL_THEME_DATASETS:
|
||||||
theme_layers = REGIONAL_THEME_DATASETS[definition.contract.source_name].get(theme)
|
theme_sources = REGIONAL_THEME_DATASETS[definition.contract.source_name].get(theme, {})
|
||||||
if theme_layers is None:
|
if dataset.source_name not in theme_sources:
|
||||||
continue
|
continue
|
||||||
layer_names = theme_layers
|
layer_names = theme_sources[dataset.source_name]
|
||||||
metadata = dataset.source_metadata if isinstance(dataset.source_metadata, dict) else {}
|
metadata = dataset.source_metadata if isinstance(dataset.source_metadata, dict) else {}
|
||||||
coverage_zones = metadata.get("coverage_zones") or metadata.get("coverage_zone") or []
|
coverage_zones = metadata.get("coverage_zones") or metadata.get("coverage_zone") or []
|
||||||
if isinstance(coverage_zones, str):
|
if isinstance(coverage_zones, str):
|
||||||
|
|||||||
@@ -221,6 +221,53 @@ def test_bounded_api_materialization_only_covers_its_persisted_bbox() -> None:
|
|||||||
assert outside.items[0].materialized_dataset_ids == []
|
assert outside.items[0].materialized_dataset_ids == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_spw_bathymetry_materialization_is_source_specific() -> None:
|
||||||
|
project_id = uuid4()
|
||||||
|
scope = [
|
||||||
|
scope_area("Belgium land", box(2.5, 49.5, 6.4, 51.5)),
|
||||||
|
scope_area("Wallonia", box(2.5, 49.5, 6.4, 50.8)),
|
||||||
|
]
|
||||||
|
selection = CoverageBBox(minx=4.851, miny=50.451, maxx=4.869, maxy=50.469)
|
||||||
|
spw_picc = SimpleNamespace(
|
||||||
|
id=uuid4(),
|
||||||
|
status="ready",
|
||||||
|
source_name="spw_picc",
|
||||||
|
reference_layer_name="buildings",
|
||||||
|
source_metadata={
|
||||||
|
"coverage_zones": ["wallonia"],
|
||||||
|
"bbox_epsg4326": [4.85, 50.45, 4.87, 50.47],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
without_bathymetry = CoverageRegistryService.resolve(
|
||||||
|
FakeSession(project=SimpleNamespace(id=project_id), areas=scope, datasets=[spw_picc]),
|
||||||
|
project_id,
|
||||||
|
selection,
|
||||||
|
["bathymetry"],
|
||||||
|
)
|
||||||
|
assert without_bathymetry.items[0].status == "partial"
|
||||||
|
assert without_bathymetry.items[0].materialized_dataset_ids == []
|
||||||
|
|
||||||
|
bathymetry_id = uuid4()
|
||||||
|
bathymetry = SimpleNamespace(
|
||||||
|
id=bathymetry_id,
|
||||||
|
status="ready",
|
||||||
|
source_name="spw_bathymetry",
|
||||||
|
reference_layer_name=None,
|
||||||
|
source_metadata={
|
||||||
|
"coverage_zones": ["wallonia"],
|
||||||
|
"bbox_epsg4326": [4.85, 50.45, 4.87, 50.47],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
with_bathymetry = CoverageRegistryService.resolve(
|
||||||
|
FakeSession(project=SimpleNamespace(id=project_id), areas=scope, datasets=[spw_picc, bathymetry]),
|
||||||
|
project_id,
|
||||||
|
selection,
|
||||||
|
["bathymetry"],
|
||||||
|
)
|
||||||
|
assert with_bathymetry.items[0].status == "operational"
|
||||||
|
assert with_bathymetry.items[0].materialized_dataset_ids == [bathymetry_id]
|
||||||
|
|
||||||
|
|
||||||
def test_mixed_land_and_north_sea_selection_remains_split() -> None:
|
def test_mixed_land_and_north_sea_selection_remains_split() -> None:
|
||||||
project_id = uuid4()
|
project_id = uuid4()
|
||||||
project = SimpleNamespace(id=project_id)
|
project = SimpleNamespace(id=project_id)
|
||||||
|
|||||||
@@ -0,0 +1,270 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import io
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
import zipfile
|
||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
import rasterio
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from pyproj import Transformer
|
||||||
|
from rasterio.io import MemoryFile
|
||||||
|
from rasterio.transform import from_origin
|
||||||
|
from shapely.geometry import shape
|
||||||
|
|
||||||
|
from app.core.config import Settings
|
||||||
|
from app.core.errors import AppError
|
||||||
|
from app.db.session import get_db
|
||||||
|
from app.main import app
|
||||||
|
from app.models import Dataset
|
||||||
|
from app.schemas.bathymetry import BathymetryRasterSelectionRequest
|
||||||
|
from app.services.bathymetry_raster_analysis_service import BathymetryRasterAnalysisService
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[2]
|
||||||
|
SCRIPT_PATH = ROOT / "scripts" / "import_spw_bathymetry.py"
|
||||||
|
|
||||||
|
|
||||||
|
def load_operator():
|
||||||
|
name = "test_import_spw_bathymetry_sprint241"
|
||||||
|
spec = importlib.util.spec_from_file_location(name, SCRIPT_PATH)
|
||||||
|
assert spec is not None
|
||||||
|
assert spec.loader is not None
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
sys.modules[name] = module
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
OPERATOR = load_operator()
|
||||||
|
|
||||||
|
|
||||||
|
class FakeSession:
|
||||||
|
def __init__(self, rows):
|
||||||
|
self.rows = rows
|
||||||
|
|
||||||
|
def get(self, model, row_id):
|
||||||
|
return self.rows.get((model, row_id))
|
||||||
|
|
||||||
|
|
||||||
|
def bathymetry_tiff(*, nodata_only: bool = False) -> bytes:
|
||||||
|
values = np.linspace(72.0, 80.0, 400, dtype="float32").reshape(20, 20)
|
||||||
|
values[:, :5] = -9999.0
|
||||||
|
if nodata_only:
|
||||||
|
values[:] = -9999.0
|
||||||
|
with MemoryFile() as memory:
|
||||||
|
with memory.open(
|
||||||
|
driver="GTiff",
|
||||||
|
width=20,
|
||||||
|
height=20,
|
||||||
|
count=1,
|
||||||
|
dtype="float32",
|
||||||
|
crs="EPSG:3812",
|
||||||
|
transform=from_origin(684_000, 629_000, 0.5, 0.5),
|
||||||
|
nodata=-9999.0,
|
||||||
|
) as output:
|
||||||
|
output.write(values, 1)
|
||||||
|
return memory.read()
|
||||||
|
|
||||||
|
|
||||||
|
def selection_payload() -> BathymetryRasterSelectionRequest:
|
||||||
|
transformer = Transformer.from_crs("EPSG:3812", "EPSG:4326", always_xy=True)
|
||||||
|
min_x, min_y = transformer.transform(684_000, 628_990)
|
||||||
|
max_x, max_y = transformer.transform(684_010, 629_000)
|
||||||
|
return BathymetryRasterSelectionRequest(
|
||||||
|
bbox={
|
||||||
|
"min_x": min(min_x, max_x),
|
||||||
|
"min_y": min(min_y, max_y),
|
||||||
|
"max_x": max(min_x, max_x),
|
||||||
|
"max_y": max(min_y, max_y),
|
||||||
|
"crs": "EPSG:4326",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def persisted_dataset(path: Path, *, metadata: dict | None = None) -> Dataset:
|
||||||
|
path.write_bytes(bathymetry_tiff())
|
||||||
|
return Dataset(
|
||||||
|
id=uuid4(),
|
||||||
|
project_id=uuid4(),
|
||||||
|
name="spw_bathymetry_test_3812.tif",
|
||||||
|
dataset_type="raster",
|
||||||
|
source="SPW official operator archive",
|
||||||
|
source_name="spw_bathymetry",
|
||||||
|
source_metadata=metadata
|
||||||
|
or {
|
||||||
|
"product_key": "spw_bathymetry_50cm_mdng",
|
||||||
|
"theme": "bathymetry",
|
||||||
|
"value_semantics": "bed_elevation",
|
||||||
|
"vertical_reference": "mDNG",
|
||||||
|
"source_crs": "EPSG:3812",
|
||||||
|
"survey_period": "2019-2022",
|
||||||
|
},
|
||||||
|
storage_path=str(path),
|
||||||
|
status="ready",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_bathymetry_analysis_returns_real_bed_elevation_and_surface_metrics(tmp_path: Path) -> None:
|
||||||
|
dataset = persisted_dataset(tmp_path / "bathymetry.tif")
|
||||||
|
db = FakeSession({(Dataset, dataset.id): dataset})
|
||||||
|
|
||||||
|
result = BathymetryRasterAnalysisService.analyze(
|
||||||
|
db,
|
||||||
|
dataset.project_id,
|
||||||
|
dataset.id,
|
||||||
|
selection_payload(),
|
||||||
|
settings=Settings(_env_file=None, bathymetry_raster_max_pixels=10_000),
|
||||||
|
)
|
||||||
|
|
||||||
|
metrics = {item["metric_key"]: item for item in result["summary"]["metrics"]}
|
||||||
|
assert result["product_key"] == "spw_bathymetry_50cm_mdng"
|
||||||
|
assert result["vertical_reference"] == "mDNG"
|
||||||
|
assert result["survey_period"] == "2019-2022"
|
||||||
|
assert result["selected_cell_count"] == 400
|
||||||
|
assert result["valid_cell_count"] == 300
|
||||||
|
assert result["coverage_ratio"] == pytest.approx(0.75)
|
||||||
|
assert metrics["bed_elevation_mean_m"]["metric_unit"] == "m mDNG"
|
||||||
|
assert metrics["surveyed_bed_surface_ha"]["metric_value"] == pytest.approx(0.0075)
|
||||||
|
assert metrics["bathymetry_coverage_pct"]["metric_value"] == pytest.approx(75.0)
|
||||||
|
assert result["unsupported_metrics"] == [
|
||||||
|
"current_water_depth_m",
|
||||||
|
"water_volume_m3",
|
||||||
|
"vertical_datum_conversion",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_bathymetry_analysis_fails_closed_for_metadata_size_and_empty_cells(tmp_path: Path) -> None:
|
||||||
|
dataset = persisted_dataset(tmp_path / "bathymetry.tif")
|
||||||
|
db = FakeSession({(Dataset, dataset.id): dataset})
|
||||||
|
|
||||||
|
with pytest.raises(AppError) as size_error:
|
||||||
|
BathymetryRasterAnalysisService.analyze(
|
||||||
|
db,
|
||||||
|
dataset.project_id,
|
||||||
|
dataset.id,
|
||||||
|
selection_payload(),
|
||||||
|
settings=Settings(_env_file=None, bathymetry_raster_max_pixels=100),
|
||||||
|
)
|
||||||
|
assert size_error.value.code == "BATHYMETRY_SELECTION_TOO_LARGE"
|
||||||
|
|
||||||
|
dataset.source_metadata = {"theme": "bathymetry"}
|
||||||
|
with pytest.raises(AppError) as metadata_error:
|
||||||
|
BathymetryRasterAnalysisService.analyze(
|
||||||
|
db,
|
||||||
|
dataset.project_id,
|
||||||
|
dataset.id,
|
||||||
|
selection_payload(),
|
||||||
|
)
|
||||||
|
assert metadata_error.value.code == "INVALID_BATHYMETRY_RASTER_METADATA"
|
||||||
|
|
||||||
|
dataset.source_metadata = {
|
||||||
|
"product_key": "spw_bathymetry_50cm_mdng",
|
||||||
|
"theme": "bathymetry",
|
||||||
|
"value_semantics": "bed_elevation",
|
||||||
|
"vertical_reference": "mDNG",
|
||||||
|
"source_crs": "EPSG:3812",
|
||||||
|
}
|
||||||
|
Path(dataset.storage_path).write_bytes(bathymetry_tiff(nodata_only=True))
|
||||||
|
with pytest.raises(AppError) as empty_error:
|
||||||
|
BathymetryRasterAnalysisService.analyze(
|
||||||
|
db,
|
||||||
|
dataset.project_id,
|
||||||
|
dataset.id,
|
||||||
|
selection_payload(),
|
||||||
|
)
|
||||||
|
assert empty_error.value.code == "BATHYMETRY_NO_VALID_DATA"
|
||||||
|
|
||||||
|
|
||||||
|
def test_bathymetry_image_and_route_use_persisted_raster_and_canonical_envelope(tmp_path: Path) -> None:
|
||||||
|
dataset = persisted_dataset(tmp_path / "bathymetry.tif")
|
||||||
|
db = FakeSession({(Dataset, dataset.id): dataset})
|
||||||
|
image = BathymetryRasterAnalysisService.render_png(db, dataset.project_id, dataset.id)
|
||||||
|
|
||||||
|
assert image.startswith(b"\x89PNG\r\n\x1a\n")
|
||||||
|
|
||||||
|
app.dependency_overrides[get_db] = lambda: db
|
||||||
|
try:
|
||||||
|
response = TestClient(app).post(
|
||||||
|
f"/api/v1/projects/{dataset.project_id}/datasets/{dataset.id}/raster/bathymetry/select",
|
||||||
|
json=selection_payload().model_dump(mode="json"),
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
assert response.status_code == 200
|
||||||
|
payload = response.json()
|
||||||
|
assert set(payload) == {"data"}
|
||||||
|
assert payload["data"]["dataset_id"] == str(dataset.id)
|
||||||
|
assert payload["data"]["summary"]["primary_metric_key"] == "bed_elevation_mean_m"
|
||||||
|
|
||||||
|
|
||||||
|
def test_operator_validates_pinned_archive_and_rejects_unsafe_members(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
safe_path = tmp_path / "safe.zip"
|
||||||
|
with zipfile.ZipFile(safe_path, "w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||||
|
archive.writestr(OPERATOR.SOURCE_MEMBER, bathymetry_tiff())
|
||||||
|
monkeypatch.setattr(OPERATOR, "SOURCE_SHA256", OPERATOR.sha256_file(safe_path))
|
||||||
|
|
||||||
|
member = OPERATOR.validate_archive(safe_path)
|
||||||
|
|
||||||
|
assert member.filename == OPERATOR.SOURCE_MEMBER
|
||||||
|
|
||||||
|
unsafe_path = tmp_path / "unsafe.zip"
|
||||||
|
with zipfile.ZipFile(unsafe_path, "w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||||
|
archive.writestr("../escape.txt", "unsafe")
|
||||||
|
archive.writestr(OPERATOR.SOURCE_MEMBER, bathymetry_tiff())
|
||||||
|
monkeypatch.setattr(OPERATOR, "SOURCE_SHA256", OPERATOR.sha256_file(unsafe_path))
|
||||||
|
with pytest.raises(OPERATOR.SpwBathymetryImportError, match="unsafe member"):
|
||||||
|
OPERATOR.validate_archive(unsafe_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_operator_crops_zip_member_to_valid_cog(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
archive_path = tmp_path / "source.zip"
|
||||||
|
with zipfile.ZipFile(archive_path, "w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||||
|
archive.writestr(OPERATOR.SOURCE_MEMBER, bathymetry_tiff())
|
||||||
|
monkeypatch.setattr(OPERATOR, "SOURCE_SHA256", OPERATOR.sha256_file(archive_path))
|
||||||
|
member = OPERATOR.validate_archive(archive_path)
|
||||||
|
output_path = tmp_path / "bounded.tif"
|
||||||
|
|
||||||
|
diagnostics = OPERATOR.crop_source(
|
||||||
|
archive_path,
|
||||||
|
member,
|
||||||
|
shape(
|
||||||
|
{
|
||||||
|
"type": "Polygon",
|
||||||
|
"coordinates": [[
|
||||||
|
[selection_payload().bbox.min_x, selection_payload().bbox.min_y],
|
||||||
|
[selection_payload().bbox.max_x, selection_payload().bbox.min_y],
|
||||||
|
[selection_payload().bbox.max_x, selection_payload().bbox.max_y],
|
||||||
|
[selection_payload().bbox.min_x, selection_payload().bbox.max_y],
|
||||||
|
[selection_payload().bbox.min_x, selection_payload().bbox.min_y],
|
||||||
|
]],
|
||||||
|
}
|
||||||
|
),
|
||||||
|
output_path,
|
||||||
|
max_pixels=10_000,
|
||||||
|
)
|
||||||
|
|
||||||
|
with rasterio.open(output_path) as output:
|
||||||
|
assert output.crs.to_epsg() == 3812
|
||||||
|
assert output.driver == "GTiff"
|
||||||
|
assert output.nodata == -9999.0
|
||||||
|
assert output.profile["tiled"] is True
|
||||||
|
assert diagnostics["valid_cell_count"] == 300
|
||||||
|
assert len(diagnostics["output_sha256"]) == 64
|
||||||
|
|
||||||
|
|
||||||
|
def test_operator_is_api_only_and_does_not_claim_depth_or_volume() -> None:
|
||||||
|
source = SCRIPT_PATH.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
assert "/datasets/upload" in source
|
||||||
|
assert "water_depth_available" in source
|
||||||
|
assert '"water_volume_available": False' in source
|
||||||
|
assert "SessionLocal" not in source
|
||||||
|
assert "db.add(" not in source
|
||||||
@@ -100,6 +100,7 @@ COPY scripts/provision_mol_bathymetry_profiles.py /app/scripts/provision_mol_bat
|
|||||||
COPY scripts/provision_flanders_geographic_scope.py /app/scripts/provision_flanders_geographic_scope.py
|
COPY scripts/provision_flanders_geographic_scope.py /app/scripts/provision_flanders_geographic_scope.py
|
||||||
COPY scripts/provision_flanders_bathymetry_profiles.py /app/scripts/provision_flanders_bathymetry_profiles.py
|
COPY scripts/provision_flanders_bathymetry_profiles.py /app/scripts/provision_flanders_bathymetry_profiles.py
|
||||||
COPY scripts/probe_mdk_bathymetry.py /app/scripts/probe_mdk_bathymetry.py
|
COPY scripts/probe_mdk_bathymetry.py /app/scripts/probe_mdk_bathymetry.py
|
||||||
|
COPY scripts/import_spw_bathymetry.py /app/scripts/import_spw_bathymetry.py
|
||||||
COPY scripts/provision_mol_bwk_natura2000.py /app/scripts/provision_mol_bwk_natura2000.py
|
COPY scripts/provision_mol_bwk_natura2000.py /app/scripts/provision_mol_bwk_natura2000.py
|
||||||
COPY scripts/provision_regional_bwk_natura2000.py /app/scripts/provision_regional_bwk_natura2000.py
|
COPY scripts/provision_regional_bwk_natura2000.py /app/scripts/provision_regional_bwk_natura2000.py
|
||||||
COPY scripts/provision_agricultural_parcel_history.py /app/scripts/provision_agricultural_parcel_history.py
|
COPY scripts/provision_agricultural_parcel_history.py /app/scripts/provision_agricultural_parcel_history.py
|
||||||
|
|||||||
+28
-3
@@ -2192,9 +2192,9 @@ sets an explicit Ollama context window and returns
|
|||||||
|
|
||||||
Returns the governed bathymetry source registry in the canonical envelope.
|
Returns the governed bathymetry source registry in the canonical envelope.
|
||||||
VHA inland profiles are `operational`. MDK Belgian Continental Shelf is
|
VHA inland profiles are `operational`. MDK Belgian Continental Shelf is
|
||||||
`probe_only`; SPW Walloon bathymetry remains `available_not_integrated`.
|
`probe_only`. The pinned SPW Walloon bathymetry archive is `operational`
|
||||||
Neither source can be acquired until its raster/download and vertical-datum
|
through a bounded, explicit operator import. There is no browser-side source
|
||||||
flow passes live validation.
|
fetch and no arbitrary source URL.
|
||||||
|
|
||||||
### GET `/api/v1/projects/{project_id}/datasets/bathymetry/sources/mdk_bcp_bathymetry/readiness`
|
### GET `/api/v1/projects/{project_id}/datasets/bathymetry/sources/mdk_bcp_bathymetry/readiness`
|
||||||
|
|
||||||
@@ -2261,6 +2261,31 @@ request limit, while `total_feature_count` and all configured depth/width
|
|||||||
metrics are calculated across the complete spatial result. Empty municipalities
|
metrics are calculated across the complete spatial result. Empty municipalities
|
||||||
return an empty, honest result. No provider request occurs during analysis.
|
return an empty, honest result. No provider request occurs during analysis.
|
||||||
|
|
||||||
|
### POST `/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/bathymetry/select`
|
||||||
|
|
||||||
|
Runs a bounded selection against a ready persisted
|
||||||
|
`source_name=spw_bathymetry` raster. The request contains an EPSG:4326 bbox
|
||||||
|
and optional persisted `area_id`, matching the other governed raster-selection
|
||||||
|
contracts.
|
||||||
|
|
||||||
|
The response reports:
|
||||||
|
|
||||||
|
- mean, minimum, maximum, p10 and p90 waterbed elevation in `m mDNG`;
|
||||||
|
- exact raster-cell surface with surveyed bed values in hectares;
|
||||||
|
- source coverage percentage inside the selected geometry;
|
||||||
|
- source resolution, survey period `2019-2022`, vertical reference and
|
||||||
|
explicit unsupported metrics.
|
||||||
|
|
||||||
|
`current_water_depth_m`, `water_volume_m3` and vertical-datum conversion remain
|
||||||
|
unsupported. The endpoint reads the persisted EPSG:3812 geometry-aligned
|
||||||
|
raster; it does not query SPW and does not synthesize missing cells.
|
||||||
|
|
||||||
|
### GET `/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/bathymetry/image`
|
||||||
|
|
||||||
|
Returns a transparent PNG rendering of the persisted bathymetry COG for the
|
||||||
|
MapLibre image-overlay path. Rendering changes presentation only. Analytical
|
||||||
|
values always come from the stored Float32 source cells.
|
||||||
|
|
||||||
Future provider output continues to use DatasetService and, for vectors,
|
Future provider output continues to use DatasetService and, for vectors,
|
||||||
VectorFeatureService. Arbitrary service URLs, browser-side fetches, insecure
|
VectorFeatureService. Arbitrary service URLs, browser-side fetches, insecure
|
||||||
TLS bypasses and startup downloads remain forbidden.
|
TLS bypasses and startup downloads remain forbidden.
|
||||||
|
|||||||
@@ -8,10 +8,11 @@ GeoIntel must distinguish three different questions:
|
|||||||
2. What is the continuous elevation of the bed at a specific survey epoch?
|
2. What is the continuous elevation of the bed at a specific survey epoch?
|
||||||
3. What is the water depth or volume at a specific moment?
|
3. What is the water depth or volume at a specific moment?
|
||||||
|
|
||||||
Only the first question is operational for Mol through the VHA cross-section
|
The first question is operational for Flanders through VHA cross-section
|
||||||
profile layer. A bed model does not provide water depth without a compatible
|
profiles. The second is now operational for bounded, surveyed Walloon
|
||||||
water-surface elevation. Flood-hazard maximum depth is a scenario result and
|
waterways through the official SPW raster. A bed model does not provide water
|
||||||
must not be reused as current water level.
|
depth without a compatible water-surface elevation. Flood-hazard maximum depth
|
||||||
|
is a scenario result and must not be reused as current water level.
|
||||||
|
|
||||||
## Governed source matrix
|
## Governed source matrix
|
||||||
|
|
||||||
@@ -19,7 +20,7 @@ must not be reused as current water level.
|
|||||||
| --- | --- | --- | --- | --- |
|
| --- | --- | --- | --- | --- |
|
||||||
| VMM VHA Digital Atlas | Flanders | Point locations, structured profile fields, PDF evidence | Document-specific | Operational, bounded vector acquisition |
|
| VMM VHA Digital Atlas | Flanders | Point locations, structured profile fields, PDF evidence | Document-specific | Operational, bounded vector acquisition |
|
||||||
| MDK Belgian Continental Shelf model | Belgian North Sea | Continuous 20 x 20 m bathymetric raster | LAT | Available, not integrated |
|
| MDK Belgian Continental Shelf model | Belgian North Sea | Continuous 20 x 20 m bathymetric raster | LAT | Available, not integrated |
|
||||||
| SPW navigable waterways and reservoir lakes | Wallonia | 0.5 m bed-elevation raster and XYZ cloud | mDNG | Available, not integrated |
|
| SPW navigable waterways and reservoir lakes | Wallonia | 0.5 m bed-elevation raster and XYZ cloud | mDNG | Operational, pinned operator archive and bounded COG |
|
||||||
| Port of Antwerp-Bruges publications | Port survey areas | Periodic soundings | Product-specific | Catalog candidate |
|
| Port of Antwerp-Bruges publications | Port survey areas | Periodic soundings | Product-specific | Catalog candidate |
|
||||||
|
|
||||||
VHA contains approximately 129,643 profile points across Flanders at the
|
VHA contains approximately 129,643 profile points across Flanders at the
|
||||||
@@ -135,7 +136,10 @@ bed evolution.
|
|||||||
**Probe implemented. Acquisition blocked because the live endpoint fails
|
**Probe implemented. Acquisition blocked because the live endpoint fails
|
||||||
strict hostname validation and does not expose usable capabilities.**
|
strict hostname validation and does not expose usable capabilities.**
|
||||||
4. Implement SPW download staging and vertical-datum metadata validation.
|
4. Implement SPW download staging and vertical-datum metadata validation.
|
||||||
|
**Done: pinned official ZIP, safe `/vsizip/` access, bounded COG,
|
||||||
|
DatasetService persistence, map overlay and mDNG selection metrics.**
|
||||||
5. Add maritime boundaries as separate authoritative scope layers.
|
5. Add maritime boundaries as separate authoritative scope layers.
|
||||||
|
**Done for the Belgian territorial sea, EEZ and continental shelf.**
|
||||||
6. Add cross-source vertical-datum transformation only with authoritative
|
6. Add cross-source vertical-datum transformation only with authoritative
|
||||||
grids/parameters and uncertainty tests.
|
grids/parameters and uncertainty tests.
|
||||||
7. Add volume only after a compatible measured or modeled water-surface source
|
7. Add volume only after a compatible measured or modeled water-surface source
|
||||||
|
|||||||
+52
-41
@@ -1,53 +1,69 @@
|
|||||||
# GeoIntel Build Status
|
# GeoIntel Build Status
|
||||||
|
|
||||||
Updated: 2026-07-17
|
Updated: 2026-07-19
|
||||||
|
|
||||||
## Current state
|
## Current state
|
||||||
|
|
||||||
GeoIntel is an implemented map-first GeoAI workbench running as an all-in-one
|
GeoIntel `v1.0.0-rc.1` is an accepted map-first GeoAI workbench for Belgium
|
||||||
Unraid container with embedded PostGIS, FastAPI, React/MapLibre, local
|
and the legally distinct Belgian maritime scopes. It runs as an immutable
|
||||||
Ollama integration and optional local YOLO/PyTorch inference.
|
all-in-one Unraid image with PostGIS, FastAPI, React/MapLibre, local Ollama
|
||||||
|
integration and optional local YOLO/PyTorch inference.
|
||||||
|
|
||||||
The active release program is
|
Mol and the Kempen are deep regression areas, not the product boundary. The
|
||||||
`docs/RC_ROADMAP_BELGIUM_NORTH_SEA.md`.
|
national workbench also has deterministic golden journeys for Wallonia,
|
||||||
|
Brussels, a language-boundary selection, the coast and the Belgian North Sea.
|
||||||
|
|
||||||
## Product scope
|
## Release status
|
||||||
|
|
||||||
- Target: all Belgian land and the separately labelled Belgian territorial
|
There are no open release blockers for the signed `v1.0.0-rc.1` evidence set.
|
||||||
sea, EEZ and continental shelf.
|
RC-0 through RC-11 are complete. Backup/restore, fresh install, upgrade,
|
||||||
- Existing deep regression references: Mol and the Kempen transport region.
|
rollback, fail-closed readiness, one Alembic head, API contracts, supply-chain
|
||||||
- Required new golden areas: Wallonia, Brussels, a cross-region area, coast
|
policy, responsive browser journeys and live PostGIS acceptance are proven.
|
||||||
and Belgian North Sea.
|
|
||||||
- Architecture: federated authoritative providers with one coverage matrix;
|
|
||||||
no assumption that GRB, PICC, UrbIS and NGI are semantically identical.
|
|
||||||
|
|
||||||
## Implemented foundation
|
The active post-RC candidate extends real source coverage. Its repository gate
|
||||||
|
passes 1,052 backend tests, 22 frontend tests, frontend typecheck/build,
|
||||||
|
one Alembic head and the complete readiness script. Its remaining acceptance
|
||||||
|
step is live SPW bathymetry persistence and browser verification before a new
|
||||||
|
immutable image is called releasable.
|
||||||
|
|
||||||
- Projects, Areas, Datasets, versions, PostGIS vector features and raster
|
## Operational product loop
|
||||||
artifacts.
|
|
||||||
- Map selection, semantic metrics, historical comparison and exports.
|
|
||||||
- Governed Flemish sources for buildings, context, orthophotos, terrain,
|
|
||||||
flooding, soil, nature, agriculture, population and policy rasters.
|
|
||||||
- Detection persistence, local configured YOLO/PyTorch path, QA/QC and review
|
|
||||||
evidence.
|
|
||||||
- Segmentation persistence foundation without fake production inference.
|
|
||||||
- Local Ollama assistant grounded in persisted evidence.
|
|
||||||
- Single-container Unraid deployment on port 1202.
|
|
||||||
|
|
||||||
## Release blockers
|
- Open one national map without selecting a technical project.
|
||||||
|
- Select an understandable theme and draw or reuse a bounded Area.
|
||||||
|
- Resolve governed coverage per Belgian jurisdiction.
|
||||||
|
- Analyse persisted PostGIS vectors or bounded raster artifacts.
|
||||||
|
- Show source-appropriate metrics, provenance, time and limitations.
|
||||||
|
- Compare compatible Statbel 2021-2025 and other governed historical series.
|
||||||
|
- Export persisted evidence or question it through the local Ollama assistant.
|
||||||
|
- Run configured local YOLO detection with persisted QA and explicit human
|
||||||
|
review limitations.
|
||||||
|
|
||||||
- Current production backup and isolated restore proof.
|
## Explicit non-blocking boundaries
|
||||||
- Fail-closed readiness and truthful runtime capability reporting.
|
|
||||||
- Stale job/run reconciliation and correlated exception logging.
|
|
||||||
- Temporal compatibility guard for detection QA.
|
|
||||||
- Production secret/configuration parity, immutable images and rollback.
|
|
||||||
- Full CI, reproducible dependencies and supply-chain evidence.
|
|
||||||
- Critical response-model typing and real frontend/browser E2E coverage.
|
|
||||||
- National and maritime scope/providers/golden areas.
|
|
||||||
|
|
||||||
## Latest evidence
|
- MDK analytical North Sea bathymetry remains `not_configured`: the public
|
||||||
|
endpoint fails strict hostname validation and the official low-resolution
|
||||||
|
product is currently request-based. TLS and source-integrity checks are not
|
||||||
|
bypassed.
|
||||||
|
- SPW supplies governed Walloon waterbed elevation in mDNG, not current water
|
||||||
|
depth. Water volume remains unsupported until a compatible water-surface,
|
||||||
|
time, vertical-datum and uncertainty contract exists.
|
||||||
|
- Building detection is an assisted review workflow. The retained benchmark
|
||||||
|
still requires independent false-negative review before any claim of
|
||||||
|
autonomous production accuracy or another training pass.
|
||||||
|
- Real SAM/YOLO-seg inference, a training studio, LiDAR, production
|
||||||
|
multi-user authentication and real-time monitoring remain post-V1 scope.
|
||||||
|
- Additional official datasets may improve coverage, but they are not V1
|
||||||
|
completion blockers when the coverage matrix reports absence honestly.
|
||||||
|
|
||||||
Run:
|
## Reproduce current repository evidence
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/run_readiness_check.sh
|
||||||
|
cd backend && python -m alembic upgrade head --sql
|
||||||
|
bash -n scripts/live_migration_smoke.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Live release evidence:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python scripts/capture_release_evidence.py \
|
python scripts/capture_release_evidence.py \
|
||||||
@@ -55,8 +71,3 @@ python scripts/capture_release_evidence.py \
|
|||||||
--release-id rc-belgium-north-sea \
|
--release-id rc-belgium-north-sea \
|
||||||
--live-base-url http://192.168.10.150:1202
|
--live-base-url http://192.168.10.150:1202
|
||||||
```
|
```
|
||||||
|
|
||||||
The evidence manifest is runtime output and is deliberately ignored by Git.
|
|
||||||
The first captured baseline reported one Alembic head (`202607160001`) and
|
|
||||||
reachable live health/capability routes. Health truthfulness is an open RC-2
|
|
||||||
blocker until readiness becomes fail-closed.
|
|
||||||
|
|||||||
@@ -2,6 +2,28 @@
|
|||||||
|
|
||||||
### Post-RC national data federation (2026-07-19)
|
### Post-RC national data federation (2026-07-19)
|
||||||
|
|
||||||
|
- Continued the national federation pass with official history and
|
||||||
|
bathymetry. Statbel's real 2021 geometry release exposed two documented
|
||||||
|
packaging differences: the ZIP member omits the repeated `31370` token and
|
||||||
|
`dt_situation` uses `YYYY/MM/DD`. The preflight now normalizes only those
|
||||||
|
forms and retains every existing fail-closed content check.
|
||||||
|
- The corrected Tower fetch-only pass validated national 2021-2024 snapshots,
|
||||||
|
each with 19,795 retained sector geometries, before any API persistence.
|
||||||
|
- Downloaded the fixed official SPW Geoportail bathymetry release
|
||||||
|
(`0a544b42-0b30-4c8e-85e7-38149b99eae0`) to persistent operator evidence.
|
||||||
|
SHA-256 is
|
||||||
|
`04122a1c5cecc7b77be025b580995d024545e82d8310e158b4c112f7fa5f8a6e`.
|
||||||
|
GDAL verified a 456,893 by 179,437 Float32 EPSG:3812 source at 0.5 m,
|
||||||
|
nodata `-9999`, with mDNG waterbed elevations.
|
||||||
|
- Added `import_spw_bathymetry.py`, bounded COG persistence, source-specific
|
||||||
|
coverage resolution, raster selection/image APIs and frontend Waterbodem
|
||||||
|
integration. The contract exposes bed elevation, measured-cell hectares and
|
||||||
|
coverage only; current depth, volume and datum conversion remain blocked.
|
||||||
|
- Added the bathymetry PNG route to the explicit non-envelope API audit set
|
||||||
|
after the first full gate correctly rejected its untracked binary response.
|
||||||
|
- The complete repository gate passed 1,052 backend tests, 22 frontend tests,
|
||||||
|
frontend typecheck/build, one Alembic head and all readiness checks before
|
||||||
|
live deployment.
|
||||||
- Added explicit administrative and maritime themes for persisted NGI/RBINS
|
- Added explicit administrative and maritime themes for persisted NGI/RBINS
|
||||||
reference data and made readiness depend on analyzable themes rather than a
|
reference data and made readiness depend on analyzable themes rather than a
|
||||||
raw Dataset count.
|
raw Dataset count.
|
||||||
|
|||||||
@@ -291,6 +291,15 @@ which creates the ordinary annual `datasets`, `dataset_versions` and
|
|||||||
year remains idempotent and previous annual snapshots are never updated or
|
year remains idempotent and previous annual snapshots are never updated or
|
||||||
deleted.
|
deleted.
|
||||||
|
|
||||||
|
SPW bathymetry also requires no migration. The immutable official ZIP and
|
||||||
|
bounded operator COG are filesystem artifacts. A successful API upload creates
|
||||||
|
an ordinary raster `Dataset` and `DatasetVersion`; raster values stay in the
|
||||||
|
versioned GeoTIFF rather than PostGIS. `source_metadata` retains EPSG:3812,
|
||||||
|
mDNG, survey period, source checksum and bounded extent, while
|
||||||
|
`provenance_metadata` retains the official URL, archive member, derived
|
||||||
|
checksum and selection geometry/hash. Analysis never writes metric rows and
|
||||||
|
never converts the vertical datum implicitly.
|
||||||
|
|
||||||
Definitive ALZ release management uses the same persistence boundary and adds
|
Definitive ALZ release management uses the same persistence boundary and adds
|
||||||
no migration or release table. Catalog planning, staged archive/GeoJSON,
|
no migration or release table. Catalog planning, staged archive/GeoJSON,
|
||||||
crop-code evidence and named review live on the filesystem. Only an approved
|
crop-code evidence and named review live on the filesystem. Only an approved
|
||||||
|
|||||||
+15
-3
@@ -764,6 +764,13 @@ population queries therefore derive from one reviewed national edition rather
|
|||||||
than mutually incompatible regional imports. No edition is downloaded or
|
than mutually incompatible regional imports. No edition is downloaded or
|
||||||
promoted during application startup.
|
promoted during application startup.
|
||||||
|
|
||||||
|
The live national workspace contains one consistent Statbel population series
|
||||||
|
for 2021-2025. The 2021-2024 geometry archives use the official
|
||||||
|
`YYYY/MM/DD` situation-date notation and archive member names without a
|
||||||
|
repeated CRS token; the fail-closed preflight normalizes only those documented
|
||||||
|
format variants and still validates EPSG:31370, release year, schema, joins,
|
||||||
|
totals, topology repairs and checksums.
|
||||||
|
|
||||||
## Wallonia PICC and Brussels UrbIS
|
## Wallonia PICC and Brussels UrbIS
|
||||||
|
|
||||||
The official-vector registry exposes bounded, source-specific regional
|
The official-vector registry exposes bounded, source-specific regional
|
||||||
@@ -792,7 +799,7 @@ profile numbers, measurement dates, available structured depth/width values
|
|||||||
and official document URLs. Scanned documents remain evidence; missing fields
|
and official document URLs. Scanned documents remain evidence; missing fields
|
||||||
are not filled by fabricated OCR output.
|
are not filled by fabricated OCR output.
|
||||||
|
|
||||||
The following sources are audited but not yet operational:
|
The following sources are audited:
|
||||||
|
|
||||||
- MDK Dieptemodel Belgisch Continentaal Plat/Noordzee: 20 x 20 m continuous
|
- MDK Dieptemodel Belgisch Continentaal Plat/Noordzee: 20 x 20 m continuous
|
||||||
raster in LAT, exposed through WCS/WMTS. GeoIntel now has a strict-TLS,
|
raster in LAT, exposed through WCS/WMTS. GeoIntel now has a strict-TLS,
|
||||||
@@ -800,8 +807,13 @@ The following sources are audited but not yet operational:
|
|||||||
`bathy.agentschapmdk.be` presents a certificate for `*.l27powered.eu`.
|
`bathy.agentschapmdk.be` presents a certificate for `*.l27powered.eu`.
|
||||||
Hostname validation therefore fails with `tls_error`; raster acquisition
|
Hostname validation therefore fails with `tls_error`; raster acquisition
|
||||||
remains disabled and TLS verification cannot be bypassed.
|
remains disabled and TLS verification cannot be bypassed.
|
||||||
- SPW bathymetry of navigable waterways and reservoir lakes: 0.5 m bed
|
- SPW bathymetry of navigable waterways and reservoir lakes: the official
|
||||||
elevation and XYZ data in mDNG.
|
2023-05-23 GeoTIFF release is operational through
|
||||||
|
`scripts/import_spw_bathymetry.py`. The operator pins SHA-256
|
||||||
|
`04122a1c5cecc7b77be025b580995d024545e82d8310e158b4c112f7fa5f8a6e`,
|
||||||
|
validates the EPSG:3812/Float32/0.5 m/-9999 contract, creates only a bounded
|
||||||
|
COG and persists it through the canonical dataset upload API. Values are
|
||||||
|
bed elevations in mDNG from surveys composed over 2019-2022.
|
||||||
- Port of Antwerp-Bruges periodic soundings: catalog candidate pending a
|
- Port of Antwerp-Bruges periodic soundings: catalog candidate pending a
|
||||||
stable public machine contract.
|
stable public machine contract.
|
||||||
|
|
||||||
|
|||||||
@@ -97,7 +97,7 @@ applicable live bounded journey fails.
|
|||||||
|
|
||||||
## P1 - National Statbel population
|
## P1 - National Statbel population
|
||||||
|
|
||||||
**State: complete; reviewed 2025 national edition materialized live.**
|
**State: complete; reviewed 2021-2025 national series materialized live.**
|
||||||
|
|
||||||
### Work
|
### Work
|
||||||
|
|
||||||
@@ -122,10 +122,17 @@ applicable live bounded journey fails.
|
|||||||
|
|
||||||
### Live evidence
|
### Live evidence
|
||||||
|
|
||||||
- Reviewed dataset: `ee0a46e5-6139-4b0a-93b5-9dabb3b3dfc7`.
|
- Reviewed immutable datasets:
|
||||||
- 20,781 persisted sectors and all 565 municipalities are represented.
|
- 2021: `27b2aaba-6ec4-42d8-91c3-9085456b5e7a`;
|
||||||
- National reconciliation: 11,825,551 inhabitants, including 7,654 explicitly
|
- 2022: `312e8278-5552-4dd0-abf7-7e1f17d2562a`;
|
||||||
retained as unlocated rather than silently assigned to geometry.
|
- 2023: `72bfb8a7-9ecb-462d-a622-f8cf693a3cfb`;
|
||||||
|
- 2024: `e7846fc6-85d1-4120-9442-e62a5b1cd346`;
|
||||||
|
- 2025: `ee0a46e5-6139-4b0a-93b5-9dabb3b3dfc7`.
|
||||||
|
- The 2021-2024 editions each retain 19,795 sector geometries. The 2025
|
||||||
|
REDEGEO edition retains 20,781 sectors and represents all 565 municipalities.
|
||||||
|
- The 2025 national reconciliation reports 11,825,551 inhabitants, including
|
||||||
|
7,654 explicitly retained as unlocated rather than silently assigned to
|
||||||
|
geometry.
|
||||||
- A bounded Mol selection returns an explicitly labelled area-weighted estimate.
|
- A bounded Mol selection returns an explicitly labelled area-weighted estimate.
|
||||||
|
|
||||||
## P2 - Wallonia bounded topographic baseline
|
## P2 - Wallonia bounded topographic baseline
|
||||||
@@ -207,7 +214,8 @@ capabilities probe before implementation is marked operational.
|
|||||||
|
|
||||||
## P4 - Maritime and bathymetry hardening
|
## P4 - Maritime and bathymetry hardening
|
||||||
|
|
||||||
**State: complete with MDK acquisition visibly blocked by strict-TLS evidence.**
|
**State: repository-complete; SPW live acceptance is the final candidate gate
|
||||||
|
and MDK acquisition remains visibly blocked by strict-TLS evidence.**
|
||||||
|
|
||||||
### Work
|
### Work
|
||||||
|
|
||||||
@@ -218,6 +226,9 @@ capabilities probe before implementation is marked operational.
|
|||||||
semantics, edition, licence and LAT evidence pass.
|
semantics, edition, licence and LAT evidence pass.
|
||||||
- Keep depth relative to LAT separate from water-surface elevation and volume.
|
- Keep depth relative to LAT separate from water-surface elevation and volume.
|
||||||
- SPW mDNG bathymetry remains a separate Walloon inland/navigation source.
|
- SPW mDNG bathymetry remains a separate Walloon inland/navigation source.
|
||||||
|
- The bounded SPW operator stages the fixed official GeoTIFF archive, validates
|
||||||
|
its checksum and raster identity, writes a clipped COG through the canonical
|
||||||
|
Dataset flow and reports only waterbed elevation and surveyed coverage.
|
||||||
|
|
||||||
### Exit
|
### Exit
|
||||||
|
|
||||||
|
|||||||
@@ -142,6 +142,23 @@ Dataset/DatasetVersion plus PostGIS `vector_features` through the canonical
|
|||||||
persistence services. Temporary standalone preflight output does not create a
|
persistence services. Temporary standalone preflight output does not create a
|
||||||
database record and may be removed explicitly after operator review.
|
database record and may be removed explicitly after operator review.
|
||||||
|
|
||||||
|
SPW bathymetry follows the same immutable-source/derived-dataset separation:
|
||||||
|
|
||||||
|
```text
|
||||||
|
storage/operator-evidence/spw-bathymetry/2023-05-23/
|
||||||
|
raw/BATHY_50CM_ALTITUDE_DNG_GEOTIFF_3812.zip
|
||||||
|
derived/spw_bathymetry_{selection_hash}_3812.tif
|
||||||
|
```
|
||||||
|
|
||||||
|
The raw ZIP is pinned by official URL and SHA-256 and is never extracted as an
|
||||||
|
8+ GiB working tree. Rasterio/GDAL reads the official GeoTIFF through
|
||||||
|
`/vsizip/`; the operator writes a bounded compressed COG and uploads that COG
|
||||||
|
through DatasetService. The queryable file therefore remains an ordinary
|
||||||
|
versioned raster Dataset. Source URL, archive/member checksum, selection
|
||||||
|
geometry/hash, survey period, EPSG:3812 and mDNG semantics are retained in
|
||||||
|
source/provenance metadata. PNG map overlays are derived responses and are not
|
||||||
|
authoritative storage.
|
||||||
|
|
||||||
Governed release-decision evidence is separate from the source artifacts:
|
Governed release-decision evidence is separate from the source artifacts:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
|||||||
+11
-6
@@ -100,9 +100,11 @@ geen open productroadmap meer.
|
|||||||
|
|
||||||
Bewuste, niet-blokkerende grenzen:
|
Bewuste, niet-blokkerende grenzen:
|
||||||
|
|
||||||
- Watervolume blijft niet beschikbaar zolang geen gekoppelde bathymetrie,
|
- De begrensde SPW-rasterflow maakt Waalse waterbodemhoogte in mDNG
|
||||||
bodemhoogte en onzekerheidsmodel bestaan. Dit is geospatiale correctheid,
|
analyseerbaar. Watervolume blijft niet beschikbaar zolang geen compatibele
|
||||||
geen onafgewerkte UI.
|
wateroppervlaktehoogte, tijdskoppeling, datumtransformatie en
|
||||||
|
onzekerheidsmodel bestaan. Dit is geospatiale correctheid, geen onafgewerkte
|
||||||
|
UI.
|
||||||
- Gebouwdetectie is operationeel maar blijft een controlekandidaat; verdere
|
- Gebouwdetectie is operationeel maar blijft een controlekandidaat; verdere
|
||||||
training gebeurt pas na nieuwe, onafhankelijke reviewdata.
|
training gebeurt pas na nieuwe, onafhankelijke reviewdata.
|
||||||
- Een echt SAM/YOLO-seg-model en een trainingsstudio blijven post-V1. De
|
- Een echt SAM/YOLO-seg-model en een trainingsstudio blijven post-V1. De
|
||||||
@@ -129,7 +131,10 @@ Bewuste, niet-blokkerende grenzen:
|
|||||||
- [x] Expand the modern 2013-2025 land-use operator with water, built-function and transport surfaces from the retained official raster.
|
- [x] Expand the modern 2013-2025 land-use operator with water, built-function and transport surfaces from the retained official raster.
|
||||||
- [x] Add a source inventory that separates loaded data from audited official follow-up sources.
|
- [x] Add a source inventory that separates loaded data from audited official follow-up sources.
|
||||||
- [x] Add a local Ollama question window grounded in persisted GeoIntel metrics and installed server models.
|
- [x] Add a local Ollama question window grounded in persisted GeoIntel metrics and installed server models.
|
||||||
- [ ] Add a governed depth/bathymetry source before exposing water volume; never infer volume from 2D GRB water geometry.
|
- [x] Add a governed bounded SPW bed-elevation source with explicit mDNG and
|
||||||
|
survey-period semantics; never infer depth or volume from 2D water geometry.
|
||||||
|
- [ ] Add a compatible water-surface, time, vertical-datum and uncertainty
|
||||||
|
contract before exposing water volume.
|
||||||
- [x] Integrate governed Waterinfo/VMM annual station series without presenting point measurements as area-wide water volume.
|
- [x] Integrate governed Waterinfo/VMM annual station series without presenting point measurements as area-wide water volume.
|
||||||
- [x] Add bounded historical orthophoto acquisition for official 1971-2025 products with a map overlay and no current-GRB QA on old imagery.
|
- [x] Add bounded historical orthophoto acquisition for official 1971-2025 products with a map overlay and no current-GRB QA on old imagery.
|
||||||
- [x] Add BWK/Natura 2000 through an explicit provider/operator contract.
|
- [x] Add BWK/Natura 2000 through an explicit provider/operator contract.
|
||||||
@@ -811,9 +816,9 @@ This file now starts with the current implementation status. Older preparation/b
|
|||||||
- [ ] Add bounded MDK GeoTIFF acquisition only after live CRS, LAT, nodata and
|
- [ ] Add bounded MDK GeoTIFF acquisition only after live CRS, LAT, nodata and
|
||||||
response-limit validation. Current official endpoint is blocked by TLS
|
response-limit validation. Current official endpoint is blocked by TLS
|
||||||
hostname mismatch and unavailable capabilities; never bypass verification.
|
hostname mismatch and unavailable capabilities; never bypass verification.
|
||||||
- [ ] Add SPW staged-download adapter with mDNG metadata and survey-epoch
|
- [x] Add SPW staged-download adapter with mDNG metadata and survey-epoch
|
||||||
coverage validation.
|
coverage validation.
|
||||||
- [ ] Add authoritative territorial-sea, EEZ and continental-shelf boundary
|
- [x] Add authoritative territorial-sea, EEZ and continental-shelf boundary
|
||||||
layers with legally accurate labels.
|
layers with legally accurate labels.
|
||||||
- [ ] Add vertical-datum conversion only when authoritative transforms and
|
- [ ] Add vertical-datum conversion only when authoritative transforms and
|
||||||
uncertainty tests exist; never merge TAW, LAT and mDNG implicitly.
|
uncertainty tests exist; never merge TAW, LAT and mDNG implicitly.
|
||||||
|
|||||||
+6
-2
@@ -695,8 +695,12 @@ The UI always labels the points as historical cross-sections. It does not
|
|||||||
present them as a continuous bed raster and does not calculate water volume.
|
present them as a continuous bed raster and does not calculate water volume.
|
||||||
Municipality partitions remain hidden for a regional Area until their backend
|
Municipality partitions remain hidden for a regional Area until their backend
|
||||||
manifest reports complete coverage. The Sources workspace lists MDK North Sea
|
manifest reports complete coverage. The Sources workspace lists MDK North Sea
|
||||||
as read-only probe-only and SPW Walloon bathymetry as planned until bounded
|
as read-only probe-only. Ready `spw_bathymetry` rasters activate the same
|
||||||
raster/download acquisition is operational.
|
`Waterbodem` theme for Wallonia with a bounded MapLibre image overlay and
|
||||||
|
selection metrics for mDNG waterbed height, measured surface and source
|
||||||
|
coverage. SPW import remains an explicit backend operator workflow; the
|
||||||
|
browser never downloads the source archive. Missing water level, depth,
|
||||||
|
volume and vertical-datum conversion remain clearly unsupported.
|
||||||
|
|
||||||
For a complete regional manifest, the theme card totals all data-bearing
|
For a complete regional manifest, the theme card totals all data-bearing
|
||||||
municipality partitions instead of displaying one representative partition.
|
municipality partitions instead of displaying one representative partition.
|
||||||
|
|||||||
+16
-6
@@ -212,7 +212,10 @@ function App(): JSX.Element {
|
|||||||
const thematicRasters = datasets.filter(
|
const thematicRasters = datasets.filter(
|
||||||
(dataset) => dataset.dataset_type === 'raster' && dataset.source_name === 'department_omgeving_thematic_raster' && dataset.status === 'ready',
|
(dataset) => dataset.dataset_type === 'raster' && dataset.source_name === 'department_omgeving_thematic_raster' && dataset.status === 'ready',
|
||||||
)
|
)
|
||||||
return [...vectors, ...terrain, ...floodHazards, ...thematicRasters]
|
const bathymetryRasters = datasets.filter(
|
||||||
|
(dataset) => dataset.dataset_type === 'raster' && dataset.source_name === 'spw_bathymetry' && dataset.status === 'ready',
|
||||||
|
)
|
||||||
|
return [...vectors, ...terrain, ...floodHazards, ...thematicRasters, ...bathymetryRasters]
|
||||||
},
|
},
|
||||||
[datasets],
|
[datasets],
|
||||||
)
|
)
|
||||||
@@ -694,12 +697,17 @@ function App(): JSX.Element {
|
|||||||
? FLANDERS_WORKSPACE_LABEL
|
? FLANDERS_WORKSPACE_LABEL
|
||||||
: selectedProject?.name ?? 'Geen werkruimte'
|
: selectedProject?.name ?? 'Geen werkruimte'
|
||||||
const areaContextLabel = selectedArea?.name ?? (areas.length > 0 ? 'Kies een gebied' : 'Geen gebied')
|
const areaContextLabel = selectedArea?.name ?? (areas.length > 0 ? 'Kies een gebied' : 'Geen gebied')
|
||||||
const bathymetryContextActive = Boolean(
|
const bathymetryProfileContextActive = Boolean(
|
||||||
activeWorkspace === 'map'
|
activeWorkspace === 'map'
|
||||||
&& selectedDataset?.source_name === 'vmm_vha_bathymetry_profiles',
|
&& selectedDataset?.source_name === 'vmm_vha_bathymetry_profiles',
|
||||||
)
|
)
|
||||||
|
const bathymetryRasterContextActive = Boolean(
|
||||||
|
activeWorkspace === 'map'
|
||||||
|
&& selectedDataset?.source_name === 'spw_bathymetry',
|
||||||
|
)
|
||||||
|
const bathymetryContextActive = bathymetryProfileContextActive || bathymetryRasterContextActive
|
||||||
const regionalBathymetryContextActive = Boolean(
|
const regionalBathymetryContextActive = Boolean(
|
||||||
bathymetryContextActive
|
bathymetryProfileContextActive
|
||||||
&& selectedDataset?.source_metadata?.regional_partitions_complete === true
|
&& selectedDataset?.source_metadata?.regional_partitions_complete === true
|
||||||
&& selectedArea
|
&& selectedArea
|
||||||
&& !/^Gemeente\s/i.test(selectedArea.name),
|
&& !/^Gemeente\s/i.test(selectedArea.name),
|
||||||
@@ -744,9 +752,11 @@ function App(): JSX.Element {
|
|||||||
const layerContextLabel = activeWorkspace === 'map' && mapContextLayerLabel
|
const layerContextLabel = activeWorkspace === 'map' && mapContextLayerLabel
|
||||||
? mapContextLayerLabel
|
? mapContextLayerLabel
|
||||||
: bathymetryContextActive
|
: bathymetryContextActive
|
||||||
? regionalBathymetryContextActive
|
? bathymetryRasterContextActive
|
||||||
? `${bathymetryProfileCount.toLocaleString('nl-BE')} profielen · ${regionalBathymetryPartitions.length} gemeenten`
|
? `${Number(selectedDataset?.source_metadata?.analysis_resolution_m ?? 0.5).toLocaleString('nl-BE')} m raster · mDNG · ${String(selectedDataset?.source_metadata?.survey_period ?? '2019-2022')}`
|
||||||
: `${bathymetryProfileCount.toLocaleString('nl-BE')} profielen`
|
: regionalBathymetryContextActive
|
||||||
|
? `${bathymetryProfileCount.toLocaleString('nl-BE')} profielen · ${regionalBathymetryPartitions.length} gemeenten`
|
||||||
|
: `${bathymetryProfileCount.toLocaleString('nl-BE')} profielen`
|
||||||
: mapFeatureCollection
|
: mapFeatureCollection
|
||||||
? `${mapFeatureCount.toLocaleString('nl-BE')} objecten`
|
? `${mapFeatureCount.toLocaleString('nl-BE')} objecten`
|
||||||
: workspaceDataLoading
|
: workspaceDataLoading
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import { TemporalTrendChart } from './TemporalTrendChart'
|
|||||||
import { terrainImageUrl } from '../../lib/terrainImage'
|
import { terrainImageUrl } from '../../lib/terrainImage'
|
||||||
import { floodHazardImageUrl } from '../../lib/floodHazardImage'
|
import { floodHazardImageUrl } from '../../lib/floodHazardImage'
|
||||||
import { thematicRasterImageUrl } from '../../lib/thematicRaster'
|
import { thematicRasterImageUrl } from '../../lib/thematicRaster'
|
||||||
|
import { bathymetryRasterImageUrl } from '../../lib/bathymetryRaster'
|
||||||
import { FLANDERS_WORKSPACE_PROJECT_NAME } from '../../config/primaryFocus'
|
import { FLANDERS_WORKSPACE_PROJECT_NAME } from '../../config/primaryFocus'
|
||||||
import {
|
import {
|
||||||
MAP_ANALYSIS_BUDGET_MS,
|
MAP_ANALYSIS_BUDGET_MS,
|
||||||
@@ -306,6 +307,11 @@ function datasetAvailabilityLabel(
|
|||||||
const resolution = Number(dataset.source_metadata?.['analysis_resolution_m'])
|
const resolution = Number(dataset.source_metadata?.['analysis_resolution_m'])
|
||||||
return `${Number.isFinite(resolution) ? `${resolution.toLocaleString('nl-BE')} m` : 'Raster'} overstromingsscenario${regionalSuffix}`
|
return `${Number.isFinite(resolution) ? `${resolution.toLocaleString('nl-BE')} m` : 'Raster'} overstromingsscenario${regionalSuffix}`
|
||||||
}
|
}
|
||||||
|
if (dataset.dataset_type === 'raster' && dataset.source_name === 'spw_bathymetry') {
|
||||||
|
const resolution = Number(dataset.source_metadata?.['analysis_resolution_m'])
|
||||||
|
const period = String(dataset.source_metadata?.['survey_period'] ?? '2019-2022')
|
||||||
|
return `${Number.isFinite(resolution) ? `${resolution.toLocaleString('nl-BE')} m` : 'Raster'} waterbodemhoogte · ${period}`
|
||||||
|
}
|
||||||
if (dataset.source_name === 'vmm_vha_bathymetry_profiles') {
|
if (dataset.source_name === 'vmm_vha_bathymetry_profiles') {
|
||||||
const profiles = partitions.reduce(
|
const profiles = partitions.reduce(
|
||||||
(total, item) => total + (item.feature_count ?? item.vector_summary?.feature_count ?? 0),
|
(total, item) => total + (item.feature_count ?? item.vector_summary?.feature_count ?? 0),
|
||||||
@@ -362,6 +368,9 @@ function datasetMatchesTheme(dataset: DatasetCreateResponse, theme: DataTheme):
|
|||||||
if (dataset.source_name === 'vmm_vha_bathymetry_profiles') {
|
if (dataset.source_name === 'vmm_vha_bathymetry_profiles') {
|
||||||
return theme.id === 'bathymetry'
|
return theme.id === 'bathymetry'
|
||||||
}
|
}
|
||||||
|
if (dataset.source_name === 'spw_bathymetry') {
|
||||||
|
return theme.id === 'bathymetry'
|
||||||
|
}
|
||||||
if (dataset.source_name === 'digitaal_vlaanderen_dhmv') {
|
if (dataset.source_name === 'digitaal_vlaanderen_dhmv') {
|
||||||
return theme.id === 'elevation'
|
return theme.id === 'elevation'
|
||||||
}
|
}
|
||||||
@@ -492,6 +501,7 @@ function pickThemeDataset(
|
|||||||
(dataset.source_name === 'digitaal_vlaanderen_dhmv' ? 5_000_000 : 0) +
|
(dataset.source_name === 'digitaal_vlaanderen_dhmv' ? 5_000_000 : 0) +
|
||||||
(dataset.source_name === 'vmm_flood_hazard' ? 5_000_000 : 0) +
|
(dataset.source_name === 'vmm_flood_hazard' ? 5_000_000 : 0) +
|
||||||
(dataset.source_name === 'vmm_vha_bathymetry_profiles' ? 5_000_000 : 0) +
|
(dataset.source_name === 'vmm_vha_bathymetry_profiles' ? 5_000_000 : 0) +
|
||||||
|
(dataset.source_name === 'spw_bathymetry' ? 5_100_000 : 0) +
|
||||||
(dataset.source_metadata?.['product_key'] === 'dtm_1m' ? 1_000_000 : 0) +
|
(dataset.source_metadata?.['product_key'] === 'dtm_1m' ? 1_000_000 : 0) +
|
||||||
(dataset.source_metadata?.['product_key'] === 'pluviaal_current_t100' ? 1_000_000 : 0) +
|
(dataset.source_metadata?.['product_key'] === 'pluviaal_current_t100' ? 1_000_000 : 0) +
|
||||||
(dataset.dataset_role === 'reference' ? 10_000 : 0)
|
(dataset.dataset_role === 'reference' ? 10_000 : 0)
|
||||||
@@ -594,6 +604,9 @@ function formatDatasetObservation(dataset: DatasetCreateResponse): string {
|
|||||||
? `historische profielen ${firstMeasurement}-${lastMeasurement}`
|
? `historische profielen ${firstMeasurement}-${lastMeasurement}`
|
||||||
: 'historische profielmetingen'
|
: 'historische profielmetingen'
|
||||||
}
|
}
|
||||||
|
if (dataset.source_name === 'spw_bathymetry') {
|
||||||
|
return `samengestelde waterbodemmeting ${String(dataset.source_metadata?.['survey_period'] ?? '2019-2022')} · mDNG`
|
||||||
|
}
|
||||||
if (dataset.source_name === 'department_omgeving_thematic_raster') {
|
if (dataset.source_name === 'department_omgeving_thematic_raster') {
|
||||||
const observationYear = Number(dataset.source_metadata?.['observation_year'])
|
const observationYear = Number(dataset.source_metadata?.['observation_year'])
|
||||||
if (Number.isFinite(observationYear)) {
|
if (Number.isFinite(observationYear)) {
|
||||||
@@ -1206,17 +1219,33 @@ export function MapWorkspace({
|
|||||||
: [],
|
: [],
|
||||||
[activeThemeDataset, selectedProjectId, thematicRasterBounds],
|
[activeThemeDataset, selectedProjectId, thematicRasterBounds],
|
||||||
)
|
)
|
||||||
|
const bathymetryRasterBounds = activeThemeDataset?.source_name === 'spw_bathymetry'
|
||||||
|
? activeThemeDataset.source_metadata?.['bbox_epsg4326']
|
||||||
|
: null
|
||||||
|
const bathymetryRasterImageOverlays = useMemo(
|
||||||
|
() => activeThemeDataset?.source_name === 'spw_bathymetry' && selectedProjectId && Array.isArray(bathymetryRasterBounds) && bathymetryRasterBounds.length === 4
|
||||||
|
? [{
|
||||||
|
url: bathymetryRasterImageUrl(selectedProjectId, activeThemeDataset.id),
|
||||||
|
bbox: bathymetryRasterBounds.map(Number) as [number, number, number, number],
|
||||||
|
label: 'Waterbodemhoogte in mDNG',
|
||||||
|
opacity: 0.86,
|
||||||
|
}]
|
||||||
|
: [],
|
||||||
|
[activeThemeDataset, bathymetryRasterBounds, selectedProjectId],
|
||||||
|
)
|
||||||
const thematicLegendMin = String(activeThemeDataset?.source_metadata?.['legend_min_label'] ?? 'Lagere waarde')
|
const thematicLegendMin = String(activeThemeDataset?.source_metadata?.['legend_min_label'] ?? 'Lagere waarde')
|
||||||
const thematicLegendMax = String(activeThemeDataset?.source_metadata?.['legend_max_label'] ?? 'Hogere waarde')
|
const thematicLegendMax = String(activeThemeDataset?.source_metadata?.['legend_max_label'] ?? 'Hogere waarde')
|
||||||
const activeImageOverlays = useMemo(
|
const activeImageOverlays = useMemo(
|
||||||
() => thematicRasterImageOverlays.length > 0
|
() => bathymetryRasterImageOverlays.length > 0
|
||||||
? thematicRasterImageOverlays
|
? bathymetryRasterImageOverlays
|
||||||
: floodHazardImageOverlays.length > 0
|
: thematicRasterImageOverlays.length > 0
|
||||||
|
? thematicRasterImageOverlays
|
||||||
|
: floodHazardImageOverlays.length > 0
|
||||||
? floodHazardImageOverlays
|
? floodHazardImageOverlays
|
||||||
: terrainImageOverlays.length > 0
|
: terrainImageOverlays.length > 0
|
||||||
? terrainImageOverlays
|
? terrainImageOverlays
|
||||||
: orthophotoImageOverlay ? [orthophotoImageOverlay] : [],
|
: orthophotoImageOverlay ? [orthophotoImageOverlay] : [],
|
||||||
[floodHazardImageOverlays, orthophotoImageOverlay, terrainImageOverlays, thematicRasterImageOverlays],
|
[bathymetryRasterImageOverlays, floodHazardImageOverlays, orthophotoImageOverlay, terrainImageOverlays, thematicRasterImageOverlays],
|
||||||
)
|
)
|
||||||
const municipalityAreaCount = areas.filter((area) => /^Gemeente\s/i.test(area.name)).length
|
const municipalityAreaCount = areas.filter((area) => /^Gemeente\s/i.test(area.name)).length
|
||||||
const themeTemporalSeriesMap = useMemo(
|
const themeTemporalSeriesMap = useMemo(
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import type { DatasetCreateResponse, VectorSelectionBBox, VectorSelectionRespons
|
|||||||
import { terrainSelectionToMapSelection } from '../lib/terrainSelection'
|
import { terrainSelectionToMapSelection } from '../lib/terrainSelection'
|
||||||
import { floodHazardSelectionToMapSelection } from '../lib/floodHazardSelection'
|
import { floodHazardSelectionToMapSelection } from '../lib/floodHazardSelection'
|
||||||
import { thematicRasterSelectionToMapSelection } from '../lib/thematicRaster'
|
import { thematicRasterSelectionToMapSelection } from '../lib/thematicRaster'
|
||||||
|
import { bathymetryRasterSelectionToMapSelection } from '../lib/bathymetryRaster'
|
||||||
|
|
||||||
interface MapSelectionExtractOptions {
|
interface MapSelectionExtractOptions {
|
||||||
selectedProjectId: string | null
|
selectedProjectId: string | null
|
||||||
@@ -44,7 +45,8 @@ export function useMapSelectionExtract({
|
|||||||
const terrainDataset = selectedDataset.dataset_type === 'raster' && selectedDataset.source_name === 'digitaal_vlaanderen_dhmv'
|
const terrainDataset = selectedDataset.dataset_type === 'raster' && selectedDataset.source_name === 'digitaal_vlaanderen_dhmv'
|
||||||
const floodHazardDataset = selectedDataset.dataset_type === 'raster' && selectedDataset.source_name === 'vmm_flood_hazard'
|
const floodHazardDataset = selectedDataset.dataset_type === 'raster' && selectedDataset.source_name === 'vmm_flood_hazard'
|
||||||
const thematicRasterDataset = selectedDataset.dataset_type === 'raster' && selectedDataset.source_name === 'department_omgeving_thematic_raster'
|
const thematicRasterDataset = selectedDataset.dataset_type === 'raster' && selectedDataset.source_name === 'department_omgeving_thematic_raster'
|
||||||
if (!isVectorDatasetType(selectedDataset.dataset_type) && !terrainDataset && !floodHazardDataset && !thematicRasterDataset) {
|
const bathymetryRasterDataset = selectedDataset.dataset_type === 'raster' && selectedDataset.source_name === 'spw_bathymetry'
|
||||||
|
if (!isVectorDatasetType(selectedDataset.dataset_type) && !terrainDataset && !floodHazardDataset && !thematicRasterDataset && !bathymetryRasterDataset) {
|
||||||
setMapSelectionError('Gebiedsanalyse ondersteunt een vectorlaag of een beheerd thematisch raster.')
|
setMapSelectionError('Gebiedsanalyse ondersteunt een vectorlaag of een beheerd thematisch raster.')
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
@@ -70,6 +72,11 @@ export function useMapSelectionExtract({
|
|||||||
bbox: { ...bbox, crs: 'EPSG:4326' },
|
bbox: { ...bbox, crs: 'EPSG:4326' },
|
||||||
area_id: areaId,
|
area_id: areaId,
|
||||||
}))
|
}))
|
||||||
|
: bathymetryRasterDataset
|
||||||
|
? bathymetryRasterSelectionToMapSelection(await datasetsApi.selectBathymetryRaster(selectedProjectId, selectedDataset.id, {
|
||||||
|
bbox: { ...bbox, crs: 'EPSG:4326' },
|
||||||
|
area_id: areaId,
|
||||||
|
}))
|
||||||
: await datasetsApi.selectVectorFeatures(selectedProjectId, selectedDataset.id, {
|
: await datasetsApi.selectVectorFeatures(selectedProjectId, selectedDataset.id, {
|
||||||
bbox: { ...bbox, crs: 'EPSG:4326' },
|
bbox: { ...bbox, crs: 'EPSG:4326' },
|
||||||
area_id: areaId,
|
area_id: areaId,
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import type { DatasetCreateResponse, VectorSelectionBBox, VectorSelectionRespons
|
|||||||
import { terrainSelectionToMapSelection } from '../lib/terrainSelection'
|
import { terrainSelectionToMapSelection } from '../lib/terrainSelection'
|
||||||
import { floodHazardSelectionToMapSelection } from '../lib/floodHazardSelection'
|
import { floodHazardSelectionToMapSelection } from '../lib/floodHazardSelection'
|
||||||
import { thematicRasterSelectionToMapSelection } from '../lib/thematicRaster'
|
import { thematicRasterSelectionToMapSelection } from '../lib/thematicRaster'
|
||||||
|
import { bathymetryRasterSelectionToMapSelection } from '../lib/bathymetryRaster'
|
||||||
|
|
||||||
export type MapThemeAcquisitionKind = 'thematic_raster' | 'dhmv' | 'flood_hazard' | 'grb' | 'official_vector'
|
export type MapThemeAcquisitionKind = 'thematic_raster' | 'dhmv' | 'flood_hazard' | 'grb' | 'official_vector'
|
||||||
|
|
||||||
@@ -148,6 +149,11 @@ export function useMapThemeSelectionInsights<TThemeId extends string>(
|
|||||||
bbox,
|
bbox,
|
||||||
area_id: areaId,
|
area_id: areaId,
|
||||||
}))
|
}))
|
||||||
|
: dataset.dataset_type === 'raster' && dataset.source_name === 'spw_bathymetry'
|
||||||
|
? bathymetryRasterSelectionToMapSelection(await datasetsApi.selectBathymetryRaster(selectedProjectId, dataset.id, {
|
||||||
|
bbox,
|
||||||
|
area_id: areaId,
|
||||||
|
}))
|
||||||
: dataset.source_name === 'vmm_vha_bathymetry_profiles' && partitioned
|
: dataset.source_name === 'vmm_vha_bathymetry_profiles' && partitioned
|
||||||
? await datasetsApi.selectBathymetryProfilePartitions(selectedProjectId, {
|
? await datasetsApi.selectBathymetryProfilePartitions(selectedProjectId, {
|
||||||
bbox,
|
bbox,
|
||||||
|
|||||||
@@ -0,0 +1,52 @@
|
|||||||
|
import { describe, expect, it } from 'vitest'
|
||||||
|
import {
|
||||||
|
bathymetryRasterImageUrl,
|
||||||
|
bathymetryRasterSelectionToMapSelection,
|
||||||
|
} from './bathymetryRaster'
|
||||||
|
|
||||||
|
describe('bathymetry raster mapping', () => {
|
||||||
|
it('uses the governed image endpoint', () => {
|
||||||
|
expect(bathymetryRasterImageUrl('project-1', 'dataset-1')).toBe(
|
||||||
|
'/api/v1/projects/project-1/datasets/dataset-1/raster/bathymetry/image',
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('maps persisted bed-elevation metrics without turning them into object counts', () => {
|
||||||
|
const mapped = bathymetryRasterSelectionToMapSelection({
|
||||||
|
dataset_id: 'dataset-1',
|
||||||
|
product_key: 'spw_bathymetry_50cm_mdng',
|
||||||
|
selection_bbox: { min_x: 4.85, min_y: 50.45, max_x: 4.87, max_y: 50.47, crs: 'EPSG:4326' },
|
||||||
|
selected_cell_count: 100,
|
||||||
|
valid_cell_count: 25,
|
||||||
|
coverage_ratio: 0.25,
|
||||||
|
resolution_m: 0.5,
|
||||||
|
vertical_reference: 'mDNG',
|
||||||
|
survey_period: '2019-2022',
|
||||||
|
summary: {
|
||||||
|
metric_label: 'Gemiddelde waterbodemhoogte',
|
||||||
|
metric_value: 74.5,
|
||||||
|
metric_unit: 'm mDNG',
|
||||||
|
aggregation_method: 'mean_valid_source_cells',
|
||||||
|
primary_metric_key: 'bed_elevation_mean_m',
|
||||||
|
metrics: [{
|
||||||
|
metric_key: 'bed_elevation_mean_m',
|
||||||
|
metric_label: 'Gemiddelde waterbodemhoogte',
|
||||||
|
metric_value: 74.5,
|
||||||
|
metric_unit: 'm mDNG',
|
||||||
|
aggregation_method: 'mean_valid_source_cells',
|
||||||
|
is_estimate: false,
|
||||||
|
}],
|
||||||
|
},
|
||||||
|
unsupported_metrics: ['current_water_depth_m', 'water_volume_m3'],
|
||||||
|
limitation_message: 'Geen gelijktijdig waterpeil.',
|
||||||
|
generated_at: '2026-07-19T00:00:00Z',
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(mapped.feature_count).toBe(25)
|
||||||
|
expect(mapped.geojson.features).toEqual([])
|
||||||
|
expect(mapped.summary?.primary_metric_key).toBe('bed_elevation_mean_m')
|
||||||
|
expect(mapped.summary?.metric_unit).toBe('m mDNG')
|
||||||
|
expect(mapped.summary?.is_estimate).toBe(false)
|
||||||
|
expect(mapped.summary?.warning).toContain('waterpeil')
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
import type { BathymetryRasterSelectionResponse, VectorSelectionResponse } from '../types'
|
||||||
|
|
||||||
|
export function bathymetryRasterImageUrl(projectId: string, datasetId: string): string {
|
||||||
|
return `/api/v1/projects/${projectId}/datasets/${datasetId}/raster/bathymetry/image`
|
||||||
|
}
|
||||||
|
|
||||||
|
export function bathymetryRasterSelectionToMapSelection(
|
||||||
|
result: BathymetryRasterSelectionResponse,
|
||||||
|
): VectorSelectionResponse {
|
||||||
|
return {
|
||||||
|
selection_bbox: result.selection_bbox,
|
||||||
|
selection_area_id: result.selection_area_id,
|
||||||
|
feature_count: result.valid_cell_count,
|
||||||
|
total_feature_count: result.valid_cell_count,
|
||||||
|
limit: 0,
|
||||||
|
truncated: false,
|
||||||
|
geojson: { type: 'FeatureCollection', features: [] },
|
||||||
|
summary: {
|
||||||
|
...result.summary,
|
||||||
|
feature_count: result.valid_cell_count,
|
||||||
|
is_estimate: false,
|
||||||
|
warning: result.limitation_message,
|
||||||
|
metrics: result.summary.metrics.map((metric) => ({
|
||||||
|
...metric,
|
||||||
|
is_estimate: false,
|
||||||
|
})),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -4,6 +4,7 @@ const NON_IMAGERY_RASTER_SOURCES = new Set([
|
|||||||
'department_omgeving_thematic_raster',
|
'department_omgeving_thematic_raster',
|
||||||
'digitaal_vlaanderen_dhmv',
|
'digitaal_vlaanderen_dhmv',
|
||||||
'vmm_flood_hazard',
|
'vmm_flood_hazard',
|
||||||
|
'spw_bathymetry',
|
||||||
])
|
])
|
||||||
|
|
||||||
export function isDetectionImageryDataset(dataset: DatasetCreateResponse): boolean {
|
export function isDetectionImageryDataset(dataset: DatasetCreateResponse): boolean {
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ const DATASET_SOURCE_LABELS: Record<string, string> = {
|
|||||||
waterinfo: 'Waterinfo Vlaanderen',
|
waterinfo: 'Waterinfo Vlaanderen',
|
||||||
vmm_flood_hazard: 'Vlaamse Milieumaatschappij',
|
vmm_flood_hazard: 'Vlaamse Milieumaatschappij',
|
||||||
vmm_vha_bathymetry_profiles: 'VMM / Vlaamse Hydrografische Atlas',
|
vmm_vha_bathymetry_profiles: 'VMM / Vlaamse Hydrografische Atlas',
|
||||||
|
spw_bathymetry: 'Service public de Wallonie',
|
||||||
department_omgeving_thematic_raster: 'Departement Omgeving',
|
department_omgeving_thematic_raster: 'Departement Omgeving',
|
||||||
dov_soil_map: 'Databank Ondergrond Vlaanderen',
|
dov_soil_map: 'Databank Ondergrond Vlaanderen',
|
||||||
}
|
}
|
||||||
@@ -57,6 +58,9 @@ export function getDatasetDisplayName(dataset: DatasetCreateResponse): string {
|
|||||||
if (dataset.source_name === 'vmm_vha_bathymetry_profiles') {
|
if (dataset.source_name === 'vmm_vha_bathymetry_profiles') {
|
||||||
return 'VHA-dwarsprofielen waterbodem'
|
return 'VHA-dwarsprofielen waterbodem'
|
||||||
}
|
}
|
||||||
|
if (dataset.source_name === 'spw_bathymetry') {
|
||||||
|
return 'SPW waterbodemhoogte 0,5 m'
|
||||||
|
}
|
||||||
if (dataset.source_name === 'department_omgeving_thematic_raster') {
|
if (dataset.source_name === 'department_omgeving_thematic_raster') {
|
||||||
const productName = dataset.source_metadata?.['product_display_name']
|
const productName = dataset.source_metadata?.['product_display_name']
|
||||||
return typeof productName === 'string' && productName.trim() ? productName : 'Officieel Vlaams themaraster'
|
return typeof productName === 'string' && productName.trim() ? productName : 'Officieel Vlaams themaraster'
|
||||||
|
|||||||
@@ -414,11 +414,11 @@ export const OFFICIAL_SOURCE_PORTFOLIO: OfficialSourceDefinition[] = [
|
|||||||
name: 'Waalse vaarweg- en stuwmeerbathymetrie',
|
name: 'Waalse vaarweg- en stuwmeerbathymetrie',
|
||||||
owner: 'Service public de Wallonie',
|
owner: 'Service public de Wallonie',
|
||||||
coverage: 'Gemeten Waalse vaarwegen en stuwmeren, 0,5 m in mDNG',
|
coverage: 'Gemeten Waalse vaarwegen en stuwmeren, 0,5 m in mDNG',
|
||||||
value: 'Hoogwaardige bodemrasters voor federale uitbreiding buiten Vlaanderen.',
|
value: 'Officiële waterbodemhoogtes voor begrensde analyse buiten Vlaanderen.',
|
||||||
metricExamples: 'bodemhoogte, diepteprofiel en vergelijkbare meetcampagnes',
|
metricExamples: 'bodemhoogte, gemeten oppervlakte en brondekking',
|
||||||
priority: 'planned',
|
priority: 'next',
|
||||||
url: 'https://geoportail.wallonie.be/catalogue/c450c28f-d357-48af-8423-62d524632cf9.html',
|
url: 'https://geoportail.wallonie.be/catalogue/0a544b42-0b30-4c8e-85e7-38149b99eae0.html',
|
||||||
matches: (dataset) => sourceNameIs(dataset, 'spw_walloon_waterway_bathymetry'),
|
matches: (dataset) => sourceNameIs(dataset, 'spw_bathymetry'),
|
||||||
},
|
},
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ import type {
|
|||||||
FloodHazardProductRead,
|
FloodHazardProductRead,
|
||||||
FloodHazardSelectionResponse,
|
FloodHazardSelectionResponse,
|
||||||
BathymetryProfileAcquireRequest,
|
BathymetryProfileAcquireRequest,
|
||||||
|
BathymetryRasterSelectionResponse,
|
||||||
BathymetrySourceProbeRead,
|
BathymetrySourceProbeRead,
|
||||||
BathymetrySourceRead,
|
BathymetrySourceRead,
|
||||||
DhmvProductRead,
|
DhmvProductRead,
|
||||||
@@ -206,6 +207,15 @@ export const datasetsApi = {
|
|||||||
`/api/v1/projects/${projectId}/datasets/bathymetry/profiles/partitions/select`,
|
`/api/v1/projects/${projectId}/datasets/bathymetry/profiles/partitions/select`,
|
||||||
payload,
|
payload,
|
||||||
),
|
),
|
||||||
|
selectBathymetryRaster: (
|
||||||
|
projectId: string,
|
||||||
|
datasetId: string,
|
||||||
|
payload: { bbox: VectorSelectionRequest['bbox']; area_id?: string },
|
||||||
|
): Promise<BathymetryRasterSelectionResponse> =>
|
||||||
|
apiPost<BathymetryRasterSelectionResponse>(
|
||||||
|
`/api/v1/projects/${projectId}/datasets/${datasetId}/raster/bathymetry/select`,
|
||||||
|
payload,
|
||||||
|
),
|
||||||
acquireThematicRaster: (projectId: string, payload: ThematicRasterAcquireRequest): Promise<JobRead> =>
|
acquireThematicRaster: (projectId: string, payload: ThematicRasterAcquireRequest): Promise<JobRead> =>
|
||||||
apiPost<JobRead>(`/api/v1/projects/${projectId}/datasets/thematic-raster/acquire`, payload),
|
apiPost<JobRead>(`/api/v1/projects/${projectId}/datasets/thematic-raster/acquire`, payload),
|
||||||
listThematicRasterProducts: (projectId: string): Promise<{ items: ThematicRasterProductRead[]; total: number }> =>
|
listThematicRasterProducts: (projectId: string): Promise<{ items: ThematicRasterProductRead[]; total: number }> =>
|
||||||
|
|||||||
@@ -542,6 +542,30 @@ export interface BathymetrySourceProbeRead {
|
|||||||
limitation_message: string
|
limitation_message: string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface BathymetryRasterSelectionResponse {
|
||||||
|
dataset_id: string
|
||||||
|
product_key: 'spw_bathymetry_50cm_mdng'
|
||||||
|
selection_bbox: VectorSelectionBBox
|
||||||
|
selection_area_id?: string | null
|
||||||
|
selected_cell_count: number
|
||||||
|
valid_cell_count: number
|
||||||
|
coverage_ratio: number
|
||||||
|
resolution_m: number
|
||||||
|
vertical_reference: 'mDNG'
|
||||||
|
survey_period: string
|
||||||
|
summary: {
|
||||||
|
metric_label: string
|
||||||
|
metric_value: number
|
||||||
|
metric_unit: string
|
||||||
|
aggregation_method: string
|
||||||
|
primary_metric_key: string
|
||||||
|
metrics: VectorSelectionMetric[]
|
||||||
|
}
|
||||||
|
unsupported_metrics: string[]
|
||||||
|
limitation_message: string
|
||||||
|
generated_at: string
|
||||||
|
}
|
||||||
|
|
||||||
export interface ThematicRasterAcquireRequest {
|
export interface ThematicRasterAcquireRequest {
|
||||||
bbox: VectorSelectionBBox
|
bbox: VectorSelectionBBox
|
||||||
area_id?: string | null
|
area_id?: string | null
|
||||||
|
|||||||
@@ -1984,6 +1984,28 @@ docker exec geointel python /app/scripts/probe_mdk_bathymetry.py
|
|||||||
|
|
||||||
This performs only `GetCapabilities`, keeps strict TLS verification enabled
|
This performs only `GetCapabilities`, keeps strict TLS verification enabled
|
||||||
and returns exit code `2` for an honest non-ready source.
|
and returns exit code `2` for an honest non-ready source.
|
||||||
|
|
||||||
|
## Bounded SPW bathymetry raster
|
||||||
|
|
||||||
|
Download or stage only the official SPW ZIP documented in
|
||||||
|
`docs/DATA_SOURCES.md`. The operator pins its SHA-256, validates its archive
|
||||||
|
and raster contract, and imports a bounded COG through the public GeoIntel API:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec geointel python /app/scripts/import_spw_bathymetry.py \
|
||||||
|
--base-url http://127.0.0.1:8000 \
|
||||||
|
--project-name "Belgium and North Sea Workbench" \
|
||||||
|
--area "RC Golden - Wallonia urban-rural" \
|
||||||
|
--bbox 4.85,50.45,4.87,50.47 \
|
||||||
|
--raw-zip /app/storage/operator-evidence/spw-bathymetry/2023-05-23/raw/BATHY_50CM_ALTITUDE_DNG_GEOTIFF_3812.zip \
|
||||||
|
--output-dir /app/storage/operator-evidence/spw-bathymetry/2023-05-23/derived
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `--dry-run` to validate identity and scope without cropping or
|
||||||
|
persistence. Use `--download` only when the requested raw path does not yet
|
||||||
|
exist; download remains strict-TLS and fixed to the official source URL.
|
||||||
|
Selections above `--max-pixels` fail before raster materialization.
|
||||||
|
|
||||||
# Release backup and restore proof
|
# Release backup and restore proof
|
||||||
|
|
||||||
The release-candidate safety path is host-operated against the running
|
The release-candidate safety path is host-operated against the running
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ ALLOWED_NON_ENVELOPE_ENDPOINTS = {
|
|||||||
("GET", "/api/v1/exports/{export_id}/download"),
|
("GET", "/api/v1/exports/{export_id}/download"),
|
||||||
("GET", "/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/image"),
|
("GET", "/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/image"),
|
||||||
("GET", "/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/terrain/image"),
|
("GET", "/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/terrain/image"),
|
||||||
|
("GET", "/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/bathymetry/image"),
|
||||||
("GET", "/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/flood-hazard/image"),
|
("GET", "/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/flood-hazard/image"),
|
||||||
("GET", "/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/thematic/image"),
|
("GET", "/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/thematic/image"),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,629 @@
|
|||||||
|
"""Validate, crop and import the official SPW bathymetry release.
|
||||||
|
|
||||||
|
The operator is intentionally bounded and fail-closed. It accepts one pinned
|
||||||
|
official archive, keeps the source checksum in provenance, creates a compact
|
||||||
|
COG for a requested EPSG:4326 scope and persists it through DatasetService's
|
||||||
|
canonical upload API. It never writes directly to PostGIS or dataset storage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from hashlib import sha256
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import mimetypes
|
||||||
|
import os
|
||||||
|
from pathlib import Path, PurePosixPath
|
||||||
|
import tempfile
|
||||||
|
from typing import Any
|
||||||
|
from urllib.error import HTTPError, URLError
|
||||||
|
from urllib.request import Request, urlopen
|
||||||
|
import uuid
|
||||||
|
import zipfile
|
||||||
|
|
||||||
|
from pyproj import Transformer
|
||||||
|
import rasterio
|
||||||
|
from rasterio.features import geometry_mask
|
||||||
|
from rasterio.mask import mask
|
||||||
|
from rasterio.shutil import copy as raster_copy
|
||||||
|
from shapely.geometry import box, mapping, shape
|
||||||
|
from shapely.ops import transform as shapely_transform
|
||||||
|
|
||||||
|
|
||||||
|
SOURCE_URL = (
|
||||||
|
"https://geoservices.wallonie.be/geotraitement/spwdatadownload/results/"
|
||||||
|
"0a544b42-0b30-4c8e-85e7-38149b99eae0/"
|
||||||
|
"BATHY_50CM_ALTITUDE_DNG_GEOTIFF_3812.zip"
|
||||||
|
)
|
||||||
|
CATALOG_URL = (
|
||||||
|
"https://geoportail.wallonie.be/catalogue/"
|
||||||
|
"0a544b42-0b30-4c8e-85e7-38149b99eae0.html"
|
||||||
|
)
|
||||||
|
SOURCE_SHA256 = "04122a1c5cecc7b77be025b580995d024545e82d8310e158b4c112f7fa5f8a6e"
|
||||||
|
SOURCE_MEMBER = "BATHY_50CM_ALTITUDE_DNG.tif"
|
||||||
|
SOURCE_VERSION = "SPW bathymetry 2023-05-23"
|
||||||
|
SURVEY_PERIOD = "2019-2022"
|
||||||
|
SOURCE_CRS = "EPSG:3812"
|
||||||
|
VERTICAL_REFERENCE = "mDNG"
|
||||||
|
NODATA = -9999.0
|
||||||
|
MAX_ARCHIVE_BYTES = 600 * 1024 * 1024
|
||||||
|
MAX_UNCOMPRESSED_BYTES = 10 * 1024 * 1024 * 1024
|
||||||
|
MAX_ARCHIVE_MEMBERS = 32
|
||||||
|
MAX_COMPRESSION_RATIO = 100.0
|
||||||
|
DEFAULT_MAX_PIXELS = 30_000_000
|
||||||
|
DEFAULT_PROJECT_NAME = "Belgium and North Sea Workbench"
|
||||||
|
DEFAULT_API_URL = "http://127.0.0.1:8000"
|
||||||
|
|
||||||
|
|
||||||
|
class SpwBathymetryImportError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def sha256_file(path: Path) -> str:
|
||||||
|
digest = sha256()
|
||||||
|
with path.open("rb") as stream:
|
||||||
|
for chunk in iter(lambda: stream.read(8 * 1024 * 1024), b""):
|
||||||
|
digest.update(chunk)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def download_source(path: Path, *, timeout: int) -> None:
|
||||||
|
request = Request(
|
||||||
|
SOURCE_URL,
|
||||||
|
headers={
|
||||||
|
"Accept": "application/zip",
|
||||||
|
"User-Agent": "GeoIntel/1.0 governed-spw-bathymetry-operator",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
temporary = path.with_suffix(path.suffix + ".partial")
|
||||||
|
digest = sha256()
|
||||||
|
size = 0
|
||||||
|
try:
|
||||||
|
with urlopen(request, timeout=timeout) as response, temporary.open("wb") as output:
|
||||||
|
if "zip" not in str(response.headers.get("Content-Type") or "").lower():
|
||||||
|
raise SpwBathymetryImportError("SPW response is not an official ZIP artifact")
|
||||||
|
for chunk in iter(lambda: response.read(8 * 1024 * 1024), b""):
|
||||||
|
size += len(chunk)
|
||||||
|
if size > MAX_ARCHIVE_BYTES:
|
||||||
|
raise SpwBathymetryImportError("SPW source archive exceeds the configured size limit")
|
||||||
|
output.write(chunk)
|
||||||
|
digest.update(chunk)
|
||||||
|
except (HTTPError, URLError, TimeoutError, OSError) as exc:
|
||||||
|
temporary.unlink(missing_ok=True)
|
||||||
|
raise SpwBathymetryImportError(f"Official SPW source download failed: {exc}") from exc
|
||||||
|
if digest.hexdigest() != SOURCE_SHA256:
|
||||||
|
temporary.unlink(missing_ok=True)
|
||||||
|
raise SpwBathymetryImportError("Downloaded SPW artifact checksum does not match the pinned release")
|
||||||
|
temporary.replace(path)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_archive(path: Path) -> zipfile.ZipInfo:
|
||||||
|
if not path.is_file():
|
||||||
|
raise SpwBathymetryImportError(f"SPW source archive does not exist: {path}")
|
||||||
|
if path.stat().st_size > MAX_ARCHIVE_BYTES:
|
||||||
|
raise SpwBathymetryImportError("SPW source archive exceeds the configured size limit")
|
||||||
|
actual_sha256 = sha256_file(path)
|
||||||
|
if actual_sha256 != SOURCE_SHA256:
|
||||||
|
raise SpwBathymetryImportError(
|
||||||
|
f"SPW source checksum mismatch: expected {SOURCE_SHA256}, received {actual_sha256}"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with zipfile.ZipFile(path) as archive:
|
||||||
|
members = archive.infolist()
|
||||||
|
if len(members) > MAX_ARCHIVE_MEMBERS:
|
||||||
|
raise SpwBathymetryImportError("SPW archive contains too many members")
|
||||||
|
total_size = 0
|
||||||
|
selected: zipfile.ZipInfo | None = None
|
||||||
|
for member in members:
|
||||||
|
member_path = PurePosixPath(member.filename)
|
||||||
|
if member.is_dir():
|
||||||
|
continue
|
||||||
|
if member_path.is_absolute() or ".." in member_path.parts:
|
||||||
|
raise SpwBathymetryImportError("SPW archive contains an unsafe member path")
|
||||||
|
total_size += member.file_size
|
||||||
|
compressed = max(1, member.compress_size)
|
||||||
|
if member.file_size / compressed > MAX_COMPRESSION_RATIO:
|
||||||
|
raise SpwBathymetryImportError("SPW archive contains an unsafe compression ratio")
|
||||||
|
if member_path.name == SOURCE_MEMBER:
|
||||||
|
selected = member
|
||||||
|
if total_size > MAX_UNCOMPRESSED_BYTES:
|
||||||
|
raise SpwBathymetryImportError("SPW archive expands beyond the configured size limit")
|
||||||
|
if selected is None:
|
||||||
|
raise SpwBathymetryImportError(f"SPW archive does not contain {SOURCE_MEMBER}")
|
||||||
|
return selected
|
||||||
|
except zipfile.BadZipFile as exc:
|
||||||
|
raise SpwBathymetryImportError("SPW source artifact is not a valid ZIP archive") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def parse_bbox(value: str):
|
||||||
|
try:
|
||||||
|
coordinates = [float(item.strip()) for item in value.split(",")]
|
||||||
|
except ValueError as exc:
|
||||||
|
raise argparse.ArgumentTypeError("bbox must contain four numeric EPSG:4326 coordinates") from exc
|
||||||
|
if len(coordinates) != 4:
|
||||||
|
raise argparse.ArgumentTypeError("bbox must contain min_x,min_y,max_x,max_y")
|
||||||
|
min_x, min_y, max_x, max_y = coordinates
|
||||||
|
if (
|
||||||
|
not all(math.isfinite(item) for item in coordinates)
|
||||||
|
or min_x >= max_x
|
||||||
|
or min_y >= max_y
|
||||||
|
or min_x < -180
|
||||||
|
or max_x > 180
|
||||||
|
or min_y < -90
|
||||||
|
or max_y > 90
|
||||||
|
):
|
||||||
|
raise argparse.ArgumentTypeError("bbox is not a valid EPSG:4326 extent")
|
||||||
|
return box(min_x, min_y, max_x, max_y)
|
||||||
|
|
||||||
|
|
||||||
|
def unwrap(payload: Any, status_code: int) -> Any:
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise SpwBathymetryImportError(f"GeoIntel returned an invalid JSON envelope (HTTP {status_code})")
|
||||||
|
if status_code >= 400 or payload.get("error"):
|
||||||
|
error = payload.get("error") if isinstance(payload.get("error"), dict) else {}
|
||||||
|
message = error.get("message") or f"GeoIntel request failed with HTTP {status_code}"
|
||||||
|
raise SpwBathymetryImportError(str(message))
|
||||||
|
if "data" not in payload:
|
||||||
|
raise SpwBathymetryImportError("GeoIntel response does not use the canonical data envelope")
|
||||||
|
return payload["data"]
|
||||||
|
|
||||||
|
|
||||||
|
def api_json(
|
||||||
|
base_url: str,
|
||||||
|
path: str,
|
||||||
|
*,
|
||||||
|
timeout: int,
|
||||||
|
method: str = "GET",
|
||||||
|
payload: dict[str, Any] | None = None,
|
||||||
|
) -> Any:
|
||||||
|
content = json.dumps(payload).encode("utf-8") if payload is not None else None
|
||||||
|
request = Request(
|
||||||
|
f"{base_url.rstrip('/')}{path}",
|
||||||
|
data=content,
|
||||||
|
method=method,
|
||||||
|
headers={
|
||||||
|
"Accept": "application/json",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"User-Agent": "GeoIntel/1.0 governed-spw-bathymetry-operator",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urlopen(request, timeout=timeout) as response:
|
||||||
|
status_code = int(response.status)
|
||||||
|
response_content = response.read()
|
||||||
|
except HTTPError as exc:
|
||||||
|
status_code = exc.code
|
||||||
|
response_content = exc.read()
|
||||||
|
except (URLError, TimeoutError, OSError) as exc:
|
||||||
|
raise SpwBathymetryImportError(f"GeoIntel API is unavailable: {exc}") from exc
|
||||||
|
try:
|
||||||
|
response_payload = json.loads(response_content.decode("utf-8"))
|
||||||
|
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||||
|
raise SpwBathymetryImportError(f"GeoIntel returned non-JSON HTTP {status_code}") from exc
|
||||||
|
return unwrap(response_payload, status_code)
|
||||||
|
|
||||||
|
|
||||||
|
def paged_items(base_url: str, path: str, *, timeout: int) -> list[dict[str, Any]]:
|
||||||
|
items: list[dict[str, Any]] = []
|
||||||
|
offset = 0
|
||||||
|
while True:
|
||||||
|
separator = "&" if "?" in path else "?"
|
||||||
|
page = api_json(
|
||||||
|
base_url,
|
||||||
|
f"{path}{separator}limit=200&offset={offset}",
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
rows = list(page.get("items") or [])
|
||||||
|
items.extend(item for item in rows if isinstance(item, dict))
|
||||||
|
total = int(page.get("total") or 0)
|
||||||
|
if not rows or len(items) >= total:
|
||||||
|
return items
|
||||||
|
offset += len(rows)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_project(
|
||||||
|
base_url: str,
|
||||||
|
project_name: str,
|
||||||
|
*,
|
||||||
|
project_id: str | None,
|
||||||
|
timeout: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
if project_id:
|
||||||
|
return api_json(base_url, f"/api/v1/projects/{project_id}", timeout=timeout)
|
||||||
|
projects = paged_items(base_url, "/api/v1/projects?status=active", timeout=timeout)
|
||||||
|
matches = [item for item in projects if str(item.get("name") or "").casefold() == project_name.casefold()]
|
||||||
|
if len(matches) != 1:
|
||||||
|
raise SpwBathymetryImportError(
|
||||||
|
f"Expected one active project named {project_name!r}, found {len(matches)}"
|
||||||
|
)
|
||||||
|
return matches[0]
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_area(
|
||||||
|
base_url: str,
|
||||||
|
project_id: str,
|
||||||
|
*,
|
||||||
|
area_id: str | None,
|
||||||
|
area_name: str | None,
|
||||||
|
timeout: int,
|
||||||
|
) -> dict[str, Any] | None:
|
||||||
|
if not area_id and not area_name:
|
||||||
|
return None
|
||||||
|
areas = paged_items(base_url, f"/api/v1/projects/{project_id}/areas", timeout=timeout)
|
||||||
|
if area_id:
|
||||||
|
matches = [item for item in areas if str(item.get("id")) == area_id]
|
||||||
|
else:
|
||||||
|
matches = [
|
||||||
|
item
|
||||||
|
for item in areas
|
||||||
|
if str(area_name).casefold() in str(item.get("name") or "").casefold()
|
||||||
|
]
|
||||||
|
if len(matches) != 1:
|
||||||
|
raise SpwBathymetryImportError(f"Expected exactly one matching persisted Area, found {len(matches)}")
|
||||||
|
if not isinstance(matches[0].get("geometry"), dict):
|
||||||
|
raise SpwBathymetryImportError("Selected Area has no persisted geometry")
|
||||||
|
return matches[0]
|
||||||
|
|
||||||
|
|
||||||
|
def source_path(archive_path: Path, member: zipfile.ZipInfo) -> str:
|
||||||
|
return f"/vsizip/{archive_path.resolve().as_posix()}/{member.filename}"
|
||||||
|
|
||||||
|
|
||||||
|
def selection_hash(selection) -> str:
|
||||||
|
payload = json.dumps(mapping(selection), sort_keys=True, separators=(",", ":"))
|
||||||
|
return sha256(payload.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def crop_source(
|
||||||
|
archive_path: Path,
|
||||||
|
member: zipfile.ZipInfo,
|
||||||
|
selection_4326,
|
||||||
|
output_path: Path,
|
||||||
|
*,
|
||||||
|
max_pixels: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
with rasterio.open(source_path(archive_path, member)) as source:
|
||||||
|
if source.crs is None or source.crs.to_epsg() != 3812:
|
||||||
|
raise SpwBathymetryImportError("SPW bathymetry source CRS must be EPSG:3812")
|
||||||
|
if source.count != 1 or source.dtypes[0] != "float32":
|
||||||
|
raise SpwBathymetryImportError("SPW bathymetry source must contain one Float32 band")
|
||||||
|
if source.nodata is None or not math.isclose(float(source.nodata), NODATA):
|
||||||
|
raise SpwBathymetryImportError("SPW bathymetry source nodata value must be -9999")
|
||||||
|
if not (
|
||||||
|
math.isclose(abs(source.res[0]), 0.5, rel_tol=0.01)
|
||||||
|
and math.isclose(abs(source.res[1]), 0.5, rel_tol=0.01)
|
||||||
|
):
|
||||||
|
raise SpwBathymetryImportError("SPW bathymetry source resolution must be approximately 0.5 m")
|
||||||
|
transformer = Transformer.from_crs("EPSG:4326", source.crs, always_xy=True)
|
||||||
|
selection_metric = shapely_transform(transformer.transform, selection_4326)
|
||||||
|
selection_metric = selection_metric.intersection(box(*source.bounds))
|
||||||
|
if selection_metric.is_empty or selection_metric.area <= 0:
|
||||||
|
raise SpwBathymetryImportError("Requested scope does not overlap the official SPW bathymetry")
|
||||||
|
min_x, min_y, max_x, max_y = selection_metric.bounds
|
||||||
|
expected_pixels = math.ceil((max_x - min_x) / abs(source.res[0])) * math.ceil(
|
||||||
|
(max_y - min_y) / abs(source.res[1])
|
||||||
|
)
|
||||||
|
if expected_pixels > max_pixels:
|
||||||
|
raise SpwBathymetryImportError(
|
||||||
|
f"Requested scope requires {expected_pixels:,} cells; limit is {max_pixels:,}"
|
||||||
|
)
|
||||||
|
clipped, transform = mask(
|
||||||
|
source,
|
||||||
|
[mapping(selection_metric)],
|
||||||
|
crop=True,
|
||||||
|
filled=True,
|
||||||
|
nodata=NODATA,
|
||||||
|
indexes=[1],
|
||||||
|
)
|
||||||
|
profile = source.profile.copy()
|
||||||
|
profile.update(
|
||||||
|
driver="GTiff",
|
||||||
|
width=clipped.shape[2],
|
||||||
|
height=clipped.shape[1],
|
||||||
|
count=1,
|
||||||
|
transform=transform,
|
||||||
|
nodata=NODATA,
|
||||||
|
compress="DEFLATE",
|
||||||
|
predictor=3,
|
||||||
|
tiled=True,
|
||||||
|
blockxsize=512,
|
||||||
|
blockysize=512,
|
||||||
|
BIGTIFF="IF_SAFER",
|
||||||
|
)
|
||||||
|
except SpwBathymetryImportError:
|
||||||
|
raise
|
||||||
|
except Exception as exc:
|
||||||
|
raise SpwBathymetryImportError(f"SPW bathymetry raster could not be read: {exc}") from exc
|
||||||
|
|
||||||
|
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with tempfile.NamedTemporaryFile(suffix=".tif", dir=output_path.parent, delete=False) as temporary:
|
||||||
|
temporary_path = Path(temporary.name)
|
||||||
|
try:
|
||||||
|
with rasterio.open(temporary_path, "w", **profile) as output:
|
||||||
|
output.write(clipped)
|
||||||
|
raster_copy(
|
||||||
|
temporary_path,
|
||||||
|
output_path,
|
||||||
|
driver="COG",
|
||||||
|
compress="DEFLATE",
|
||||||
|
predictor=3,
|
||||||
|
blocksize=512,
|
||||||
|
overview_resampling="average",
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
temporary_path.unlink(missing_ok=True)
|
||||||
|
|
||||||
|
with rasterio.open(output_path) as output:
|
||||||
|
values = output.read(1, masked=True)
|
||||||
|
valid = geometry_mask(
|
||||||
|
[mapping(selection_metric)],
|
||||||
|
out_shape=values.shape,
|
||||||
|
transform=output.transform,
|
||||||
|
invert=True,
|
||||||
|
) & ~values.mask
|
||||||
|
valid_values = values.data[valid]
|
||||||
|
if valid_values.size == 0:
|
||||||
|
output_path.unlink(missing_ok=True)
|
||||||
|
raise SpwBathymetryImportError("Selected scope contains no surveyed SPW waterbed cells")
|
||||||
|
if not bool(((valid_values > -500.0) & (valid_values < 1000.0)).all()):
|
||||||
|
output_path.unlink(missing_ok=True)
|
||||||
|
raise SpwBathymetryImportError("SPW bathymetry contains values outside the governed mDNG range")
|
||||||
|
return {
|
||||||
|
"width": output.width,
|
||||||
|
"height": output.height,
|
||||||
|
"resolution_m": max(abs(output.res[0]), abs(output.res[1])),
|
||||||
|
"valid_cell_count": int(valid_values.size),
|
||||||
|
"valid_value_min": float(valid_values.min()),
|
||||||
|
"valid_value_max": float(valid_values.max()),
|
||||||
|
"valid_value_mean": float(valid_values.mean()),
|
||||||
|
"output_sha256": sha256_file(output_path),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def multipart_upload(
|
||||||
|
base_url: str,
|
||||||
|
project_id: str,
|
||||||
|
output_path: Path,
|
||||||
|
fields: dict[str, str],
|
||||||
|
*,
|
||||||
|
timeout: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
boundary = f"geointel-{uuid.uuid4().hex}"
|
||||||
|
body = bytearray()
|
||||||
|
for name, value in fields.items():
|
||||||
|
body.extend(f"--{boundary}\r\n".encode())
|
||||||
|
body.extend(f'Content-Disposition: form-data; name="{name}"\r\n\r\n'.encode())
|
||||||
|
body.extend(value.encode("utf-8"))
|
||||||
|
body.extend(b"\r\n")
|
||||||
|
body.extend(f"--{boundary}\r\n".encode())
|
||||||
|
body.extend(
|
||||||
|
(
|
||||||
|
f'Content-Disposition: form-data; name="file"; filename="{output_path.name}"\r\n'
|
||||||
|
f"Content-Type: {mimetypes.guess_type(output_path.name)[0] or 'image/tiff'}\r\n\r\n"
|
||||||
|
).encode()
|
||||||
|
)
|
||||||
|
body.extend(output_path.read_bytes())
|
||||||
|
body.extend(f"\r\n--{boundary}--\r\n".encode())
|
||||||
|
request = Request(
|
||||||
|
f"{base_url.rstrip('/')}/api/v1/projects/{project_id}/datasets/upload",
|
||||||
|
data=bytes(body),
|
||||||
|
method="POST",
|
||||||
|
headers={
|
||||||
|
"Accept": "application/json",
|
||||||
|
"Content-Type": f"multipart/form-data; boundary={boundary}",
|
||||||
|
"User-Agent": "GeoIntel/1.0 governed-spw-bathymetry-operator",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urlopen(request, timeout=timeout) as response:
|
||||||
|
status_code = int(response.status)
|
||||||
|
response_content = response.read()
|
||||||
|
except HTTPError as exc:
|
||||||
|
status_code = exc.code
|
||||||
|
response_content = exc.read()
|
||||||
|
except (URLError, TimeoutError, OSError) as exc:
|
||||||
|
raise SpwBathymetryImportError(f"GeoIntel upload failed: {exc}") from exc
|
||||||
|
try:
|
||||||
|
response_payload = json.loads(response_content.decode("utf-8"))
|
||||||
|
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||||
|
raise SpwBathymetryImportError(f"GeoIntel upload returned non-JSON HTTP {status_code}") from exc
|
||||||
|
return unwrap(response_payload, status_code)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description="Import a bounded official SPW bathymetry raster.")
|
||||||
|
parser.add_argument("--base-url", default=os.environ.get("GEOINTEL_INTERNAL_API_URL", DEFAULT_API_URL))
|
||||||
|
parser.add_argument("--project-name", default=DEFAULT_PROJECT_NAME)
|
||||||
|
parser.add_argument("--project-id")
|
||||||
|
parser.add_argument("--area")
|
||||||
|
parser.add_argument("--area-id")
|
||||||
|
parser.add_argument("--bbox", required=True, type=parse_bbox, help="min_x,min_y,max_x,max_y in EPSG:4326")
|
||||||
|
parser.add_argument("--raw-zip", required=True, type=Path)
|
||||||
|
parser.add_argument("--download", action="store_true", help="Download the pinned official ZIP when --raw-zip is absent.")
|
||||||
|
parser.add_argument("--output-dir", type=Path, default=Path("storage/operator-evidence/spw-bathymetry/derived"))
|
||||||
|
parser.add_argument("--max-pixels", type=int, default=DEFAULT_MAX_PIXELS)
|
||||||
|
parser.add_argument("--timeout", type=int, default=900)
|
||||||
|
parser.add_argument("--force-refresh", action="store_true")
|
||||||
|
parser.add_argument("--dry-run", action="store_true")
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = parse_args()
|
||||||
|
archive_path = args.raw_zip.resolve()
|
||||||
|
if not archive_path.exists() and args.download:
|
||||||
|
download_source(archive_path, timeout=args.timeout)
|
||||||
|
member = validate_archive(archive_path)
|
||||||
|
project = resolve_project(
|
||||||
|
args.base_url,
|
||||||
|
args.project_name,
|
||||||
|
project_id=args.project_id,
|
||||||
|
timeout=args.timeout,
|
||||||
|
)
|
||||||
|
project_id = str(project["id"])
|
||||||
|
area = resolve_area(
|
||||||
|
args.base_url,
|
||||||
|
project_id,
|
||||||
|
area_id=args.area_id,
|
||||||
|
area_name=args.area,
|
||||||
|
timeout=args.timeout,
|
||||||
|
)
|
||||||
|
selection = args.bbox
|
||||||
|
if area is not None:
|
||||||
|
selection = selection.intersection(shape(area["geometry"]))
|
||||||
|
if selection.is_empty or selection.area <= 0:
|
||||||
|
raise SpwBathymetryImportError("Requested bbox does not overlap the selected persisted Area")
|
||||||
|
|
||||||
|
scope_hash = selection_hash(selection)
|
||||||
|
output_name = f"spw_bathymetry_{scope_hash[:16]}_3812.tif"
|
||||||
|
output_path = args.output_dir.resolve() / output_name
|
||||||
|
raw_sha256 = sha256_file(archive_path)
|
||||||
|
datasets = paged_items(
|
||||||
|
args.base_url,
|
||||||
|
f"/api/v1/projects/{project_id}/datasets",
|
||||||
|
timeout=args.timeout,
|
||||||
|
)
|
||||||
|
existing = next(
|
||||||
|
(
|
||||||
|
item
|
||||||
|
for item in datasets
|
||||||
|
if item.get("status") == "ready"
|
||||||
|
and item.get("source_name") == "spw_bathymetry"
|
||||||
|
and (item.get("source_metadata") or {}).get("source_artifact_sha256") == raw_sha256
|
||||||
|
and (item.get("source_metadata") or {}).get("selection_hash") == scope_hash
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if existing and not args.force_refresh:
|
||||||
|
print(json.dumps({"status": "reused", "dataset_id": existing["id"]}, ensure_ascii=False))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"status": "validated",
|
||||||
|
"project_id": project_id,
|
||||||
|
"area_id": area.get("id") if area else None,
|
||||||
|
"bbox_epsg4326": list(selection.bounds),
|
||||||
|
"source_sha256": raw_sha256,
|
||||||
|
"source_member": member.filename,
|
||||||
|
"output_path": str(output_path),
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
diagnostics = crop_source(
|
||||||
|
archive_path,
|
||||||
|
member,
|
||||||
|
selection,
|
||||||
|
output_path,
|
||||||
|
max_pixels=args.max_pixels,
|
||||||
|
)
|
||||||
|
source_metadata = {
|
||||||
|
"product_key": "spw_bathymetry_50cm_mdng",
|
||||||
|
"product_display_name": "SPW waterbodemhoogte 0,5 m",
|
||||||
|
"theme": "bathymetry",
|
||||||
|
"value_semantics": "bed_elevation",
|
||||||
|
"vertical_reference": VERTICAL_REFERENCE,
|
||||||
|
"source_crs": SOURCE_CRS,
|
||||||
|
"native_resolution_m": 0.5,
|
||||||
|
"analysis_resolution_m": diagnostics["resolution_m"],
|
||||||
|
"nodata_value": NODATA,
|
||||||
|
"survey_period": SURVEY_PERIOD,
|
||||||
|
"published_on": "2023-05-23",
|
||||||
|
"bbox_epsg4326": list(selection.bounds),
|
||||||
|
"coverage_zones": ["wallonia"],
|
||||||
|
"coverage_scope": "bounded_selection",
|
||||||
|
"selection_hash": scope_hash,
|
||||||
|
"source_artifact_sha256": raw_sha256,
|
||||||
|
"valid_cell_count": diagnostics["valid_cell_count"],
|
||||||
|
"valid_value_min": diagnostics["valid_value_min"],
|
||||||
|
"valid_value_max": diagnostics["valid_value_max"],
|
||||||
|
"valid_value_mean": diagnostics["valid_value_mean"],
|
||||||
|
"attribution": "Service public de Wallonie (SPW)",
|
||||||
|
"license_note": "CC BY 4.0",
|
||||||
|
"catalog_url": CATALOG_URL,
|
||||||
|
"limitation_message": (
|
||||||
|
"Waterbodemhoogte in mDNG uit opmetingen 2019-2022; geen actuele waterdiepte of watervolume."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
provenance_metadata = {
|
||||||
|
"operator": "import_spw_bathymetry.py",
|
||||||
|
"official_source_url": SOURCE_URL,
|
||||||
|
"official_catalog_url": CATALOG_URL,
|
||||||
|
"source_artifact_sha256": raw_sha256,
|
||||||
|
"source_archive_member": member.filename,
|
||||||
|
"derived_artifact_sha256": diagnostics["output_sha256"],
|
||||||
|
"selection_geometry_epsg4326": mapping(selection),
|
||||||
|
"selection_hash": scope_hash,
|
||||||
|
"processed_at": datetime.now(UTC).isoformat(),
|
||||||
|
"water_depth_available": False,
|
||||||
|
"water_volume_available": False,
|
||||||
|
"vertical_datum_conversion_applied": False,
|
||||||
|
}
|
||||||
|
fields = {
|
||||||
|
"dataset_type": "raster",
|
||||||
|
"source": "SPW official operator archive",
|
||||||
|
"dataset_role": "source",
|
||||||
|
"source_name": "spw_bathymetry",
|
||||||
|
"source_metadata_json": json.dumps(source_metadata, separators=(",", ":")),
|
||||||
|
"provenance_metadata_json": json.dumps(provenance_metadata, separators=(",", ":")),
|
||||||
|
"valid_from": "2019-01-01T00:00:00Z",
|
||||||
|
"valid_to": "2022-12-31T23:59:59Z",
|
||||||
|
"temporal_granularity": "period",
|
||||||
|
"source_version": SOURCE_VERSION,
|
||||||
|
}
|
||||||
|
if area is not None:
|
||||||
|
fields["area_id"] = str(area["id"])
|
||||||
|
created = multipart_upload(
|
||||||
|
args.base_url,
|
||||||
|
project_id,
|
||||||
|
output_path,
|
||||||
|
fields,
|
||||||
|
timeout=args.timeout,
|
||||||
|
)
|
||||||
|
analysis = api_json(
|
||||||
|
args.base_url,
|
||||||
|
f"/api/v1/projects/{project_id}/datasets/{created['id']}/raster/bathymetry/select",
|
||||||
|
timeout=args.timeout,
|
||||||
|
method="POST",
|
||||||
|
payload={
|
||||||
|
"bbox": {
|
||||||
|
"min_x": selection.bounds[0],
|
||||||
|
"min_y": selection.bounds[1],
|
||||||
|
"max_x": selection.bounds[2],
|
||||||
|
"max_y": selection.bounds[3],
|
||||||
|
"crs": "EPSG:4326",
|
||||||
|
},
|
||||||
|
"area_id": area.get("id") if area else None,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"status": "imported",
|
||||||
|
"dataset_id": created["id"],
|
||||||
|
"output_path": str(output_path),
|
||||||
|
"source_sha256": raw_sha256,
|
||||||
|
"derived_sha256": diagnostics["output_sha256"],
|
||||||
|
"summary": analysis["summary"],
|
||||||
|
"coverage_ratio": analysis["coverage_ratio"],
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
raise SystemExit(main())
|
||||||
|
except SpwBathymetryImportError as exc:
|
||||||
|
raise SystemExit(f"SPW_BATHYMETRY_IMPORT_FAILED: {exc}") from exc
|
||||||
@@ -80,6 +80,7 @@ ${PYTHON_BIN} -m py_compile scripts/provision_waterinfo_station_history.py
|
|||||||
${PYTHON_BIN} -m py_compile scripts/provision_flanders_geographic_scope.py
|
${PYTHON_BIN} -m py_compile scripts/provision_flanders_geographic_scope.py
|
||||||
${PYTHON_BIN} -m py_compile scripts/provision_flanders_bathymetry_profiles.py
|
${PYTHON_BIN} -m py_compile scripts/provision_flanders_bathymetry_profiles.py
|
||||||
${PYTHON_BIN} -m py_compile scripts/probe_mdk_bathymetry.py
|
${PYTHON_BIN} -m py_compile scripts/probe_mdk_bathymetry.py
|
||||||
|
${PYTHON_BIN} -m py_compile scripts/import_spw_bathymetry.py
|
||||||
${PYTHON_BIN} -m py_compile scripts/provision_mol_bwk_natura2000.py
|
${PYTHON_BIN} -m py_compile scripts/provision_mol_bwk_natura2000.py
|
||||||
${PYTHON_BIN} -m py_compile scripts/provision_regional_bwk_natura2000.py
|
${PYTHON_BIN} -m py_compile scripts/provision_regional_bwk_natura2000.py
|
||||||
${PYTHON_BIN} -m py_compile scripts/provision_agricultural_parcel_history.py
|
${PYTHON_BIN} -m py_compile scripts/provision_agricultural_parcel_history.py
|
||||||
|
|||||||
Reference in New Issue
Block a user