feat: add governed hydrology and historical imagery
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-15 12:17:45 +02:00
parent 5b1156e989
commit fb38eb3e91
32 changed files with 1646 additions and 55 deletions
+25 -5
View File
@@ -191,15 +191,24 @@ Response: `DatasetRead` with extracted metadata if supported.
Vector uploads remain stored as original files and are also persisted into `vector_features` as queryable PostGIS state.
### GET `/api/v1/projects/{project_id}/datasets/orthophoto/products`
Return the governed Digitaal Vlaanderen orthophoto product allowlist in the
canonical envelope. Every product reports its key, display/observation label,
temporal granularity, native resolution, colour mode, catalogue URL,
limitations and whether current configured-YOLO detection is allowed.
### POST `/api/v1/projects/{project_id}/datasets/orthophoto/acquire`
Explicitly acquire a bounded most-recent winter orthophoto selection from the
official Digitaal Vlaanderen `OMWRGBMRVL` WMS `Ortho` layer.
Explicitly acquire a bounded orthophoto selection from a governed official
Digitaal Vlaanderen WMS product. Arbitrary WMS URLs and layer names are not
accepted.
```json
{
"bbox": {"min_x": 5.10, "min_y": 51.17, "max_x": 5.11, "max_y": 51.18, "crs": "EPSG:4326"},
"area_id": "optional-project-area-uuid",
"product_key": "most_recent",
"force_refresh": false
}
```
@@ -207,7 +216,8 @@ official Digitaal Vlaanderen `OMWRGBMRVL` WMS `Ortho` layer.
The canonical envelope contains a synchronous Job. Its `output_dataset_id`
identifies the raster Dataset; `result_json` contains provider, layer, pixel
dimensions, EPSG:4326/EPSG:31370 bounds, sampling resolution, attribution,
cache reuse and limitation text.
cache reuse and limitation text. Historical products also persist their
observation/validity period and a spatially scoped temporal-series key.
Safety contract:
@@ -218,8 +228,11 @@ Safety contract:
reuse;
- WMS bytes are georeferenced to EPSG:31370 and persisted only through
`DatasetService`; no fetch runs on startup;
- this is the latest mosaic available at request time, not a historical
observation date for every pixel.
- only `most_recent` can enter the current configured-YOLO plus GRB-QA path;
historical products are visual evidence and are never validated against the
current GRB state;
- product periods such as `1979_1990` remain explicitly multi-year and are not
presented as exact annual observations.
### GET `/api/v1/projects/{project_id}/datasets`
@@ -274,6 +287,13 @@ If preview dependencies are unavailable:
- code: `RASTER_PROCESSING_UNAVAILABLE`
- message: `Raster preview unavailable...`
### GET `/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/image`
Return a persisted orthophoto Dataset as a bounded browser-safe PNG. This is an
explicit binary non-envelope endpoint used by the MapLibre image source. It
accepts only ready datasets from the governed orthophoto provider and never
reads arbitrary filesystem paths.
### POST `/api/v1/projects/{project_id}/datasets/{dataset_id}/raster/clip`
Clip raster by selected area. Returns a `202`-style accepted job payload through the job wrapper (`jobs` create/read flow).
+30
View File
@@ -8438,3 +8438,33 @@ Next:
- Integrate Waterinfo/VMM station observations as point time series and add
historical orthophoto acquisition, while preserving their spatial and
methodological limitations.
## Sprint 203 - Governed Waterinfo history and historical orthophotos (2026-07-15)
Implemented:
- Added `provision_waterinfo_station_history.py` using the documented Waterinfo
KiWIS annual water-level/discharge groups. The operator filters station
points against the exact persisted Area, retains raw JSON plus SHA256
manifests and writes only through the canonical upload API.
- Persisted one Point Dataset per station/year and kept station identities in
separate temporal series. Added backend `mean` aggregation without combining
stations or inferring area-wide water level/volume.
- Added a governed orthophoto allowlist for the most-recent product, annual
winter mosaics 2012-2025, older winter periods, RGB 1979-1990 and
panchromatic 1971. Arbitrary WMS URLs/layers remain impossible.
- Added raster temporal metadata and a constrained PNG rendering endpoint for
persisted orthophoto datasets. The map can select and display official
historical imagery over the same bounded rectangle.
- Historical products explicitly bypass configured-YOLO and current-GRB QA;
only `most_recent` retains that path.
- Updated source inventory behavior and wrote a governed source backlog with
acceptance criteria for BWK/Natura 2000, agricultural parcels, Buildings
Register, DHMV and bathymetry.
Validation evidence:
- Focused Waterinfo/orthophoto/semantic metric tests and frontend typecheck/build
passed before the full repository gate.
Remaining operational step:
- Deploy the all-in-one image, provision the real Mol Waterinfo station series
and verify one historical image acquisition/overlay through the LAN browser.
+29 -6
View File
@@ -214,9 +214,6 @@ These sources are available from their public authorities but are not silently
treated as loaded GeoIntel data. The Source inventory labels them separately
until a governed operator import, provenance record and validation pass exist.
- Historical orthophotos (Digitaal Vlaanderen): 1971 and 1979-1990 through
the `OKZ` WMS, with additional dated mosaics as separate products. Suitable
for visual/image evolution after a bounded acquisition contract is added.
- Biologische Waarderingskaart / Natura 2000 (INBO), state 2025: suitable for
habitat, biotope and ecological-value analysis, not a continuous annual
series.
@@ -229,9 +226,35 @@ until a governed operator import, provenance record and validation pass exist.
- DHMV II DTM/DSM (Digitaal Vlaanderen): 1 m/5 m elevation based on 2013-2015
LiDAR, suitable for elevation, slope and drainage. It does not provide water
depth.
- Waterinfo/VMM: station time series for water level, flow and precipitation.
These can describe hydrological state, but do not provide area-wide water
volume without a compatible bottom profile/bathymetry model.
## Governed Waterinfo station history
`scripts/provision_waterinfo_station_history.py` uses the public Waterinfo
KiWIS query service only after an explicit operator command. It discovers the
documented annual water-level (`192784`) and discharge (`192895`) groups,
filters station points against the exact persisted Area and retains the raw
station/value JSON plus SHA256 manifests.
Each station and year becomes one immutable reference Dataset through the
normal upload API. Series keys include the station identity; different stations
are never averaged into one municipal value. Selection aggregation is a numeric
mean over the selected station records and remains labelled as a point
measurement. Water level or discharge does not establish area-wide water
volume without compatible depth, profile and coverage data.
## Governed historical orthophotos
The bounded map acquisition registry includes the official annual winter
mosaics for 2012-2025, period products for 2000-2003, 2005-2007 and 2008-2011,
the RGB 1979-1990 summer mosaic and the panchromatic 1971 mosaic. Requests stay
within the configured 128-1,024 m safety envelope and are persisted as
EPSG:31370 raster Datasets with explicit product, layer, observation period,
attribution and request hashes.
Historical mosaics are for visual comparison only in this phase. Current YOLO
building detection and current GRB QA remain restricted to `most_recent`, since
validating an old image against today's building state would produce dishonest
quality metrics.
## OSM
+15
View File
@@ -198,6 +198,21 @@ Metadata:
- nodata
- resolution
Temporal orthophoto rasters additionally require a governed product key, WMS
layer, observation label, `observed_at`, optional `valid_from`/`valid_to`,
temporal granularity, request/spatial hash, attribution and a limitation that
states whether the product is annual, multi-year or merely most recent.
### Hydrological station observations
Waterinfo observations are persisted as EPSG:4326 Point features, one station
and one observation year per immutable Dataset. Required properties are the
station/timeseries identity, measurement type, numeric annual value, reported
unit, observation year, provider/owner and attribution. Different station
series may not be merged into one area-wide value. Point water level and
discharge may not be converted to water volume without governed compatible
depth/profile data.
### Vector
Ondersteund:
+10
View File
@@ -99,6 +99,16 @@ Mask files are provenance/debug artifacts. QA, map display and GeoJSON output mu
Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.
## Official temporal source artifacts
Waterinfo raw station layers, timeseries responses and checksum manifests live
under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
evidence; queryable annual Point snapshots are normal Dataset/vector_feature
records. Bounded orthophotos are normal raster Dataset files. Their WMS URL,
product/layer, request/spatial hash, temporal validity and limitations are held
in source/provenance metadata. Browser PNG rendering is derived on request and
does not replace the stored GeoTIFF.
Offline demo export artifacts can be inspected and cleaned with:
```bash
+45 -2
View File
@@ -22,12 +22,55 @@
- [x] Add a source inventory that separates loaded data from audited official follow-up sources.
- [x] Add a local Ollama question window grounded in persisted GeoIntel metrics and installed server models.
- [ ] Add a governed depth/bathymetry source before exposing water volume; never infer volume from 2D GRB water geometry.
- [ ] Integrate one governed hydrology source (Waterinfo/VMM station series) without presenting point measurements as area-wide water volume.
- [ ] Add bounded historical orthophoto acquisition and visual change analysis after validating layer/year coverage.
- [x] Integrate governed Waterinfo/VMM annual station series without presenting point measurements as area-wide water volume.
- [x] Add bounded historical orthophoto acquisition for official 1971-2025 products with a map overlay and no current-GRB QA on old imagery.
- [ ] Add BWK/Natura 2000 and annual agricultural-use parcels through explicit provider/operator contracts.
- [ ] Extend the official 1778/1873/1969 historical buildings, water and roads series from Mol to the approved regional scope with partitioned source audits.
- [x] Connect a drawn rectangle to bounded official orthophoto acquisition, local configured-YOLO detection and persisted GRB QA.
## Governed source expansion backlog
Implement these in order. Every source must use an explicit operator/provider
contract, retain raw checksummed evidence, persist through DatasetService and
pass live Mol validation before regional expansion.
### P1 - Biologische Waarderingskaart / Natura 2000
- [ ] Confirm the official downloadable service/layer/version and licence for the 2025 state.
- [ ] Define stable habitat/biotope/value fields and keep BWK valuation separate from Natura 2000 habitat classification.
- [ ] Clip in EPSG:31370 to Mol, transform to EPSG:4326 and persist valid polygonal `vector_features` with source feature ids.
- [ ] Add hectare metrics per governed class plus unknown/unmapped-class diagnostics; do not invent an annual trend from one state.
- [ ] Add source manifest, unit tests, live row/count/geometry audit and map/source-inventory presentation.
### P2 - Annual agricultural-use parcels
- [ ] Inventory official annual editions and document schema/code-list changes before selecting a comparable year range.
- [ ] Retain annual source files and crop code lists; normalize only fields whose meaning is stable across editions.
- [ ] Persist separate annual Datasets with hectares by crop/use class and an explicit unstable-parcel-identity limitation.
- [ ] Compare area/category totals over time; do not claim parcel lineage where identifiers or boundaries changed.
- [ ] Validate Mol first, then partition all 28 Kempen municipalities with completeness/checksum manifests.
### P3 - Buildings and Addresses Register
- [ ] Define a governed snapshot operator and relation mapping between building unit, building object, address and GRB geometry.
- [ ] Keep register lifecycle/status semantics separate from GRB footprint geometry and document match confidence/unmatched rows.
- [ ] Add current building-status/address metrics without exposing personal data or treating addresses as households/population.
- [ ] Add Mol reconciliation tests and a live mismatch report before regional import.
### P4 - Digitaal Hoogtemodel Vlaanderen
- [ ] Select official DTM/DSM product, service and native resolution; retain acquisition date and vertical reference.
- [ ] Add bounded raster acquisition/clip storage with nodata, CRS, resolution and checksum validation.
- [ ] Implement governed elevation, relief and slope statistics in metres/degrees; keep drainage interpretation explicitly derived.
- [ ] Do not label terrain/surface height as water depth and do not enable volume from DHMV alone.
### P5 - Water depth / bathymetry
- [ ] Identify an authoritative source with compatible spatial coverage, vertical datum, date and uncertainty; otherwise keep volume unavailable.
- [ ] Define waterbody linkage, surface elevation, bottom elevation and uncertainty propagation before adding any volume metric.
- [ ] Validate coverage gaps and prohibit extrapolation outside measured/profiled waterbodies.
- [ ] Add independent GIS review and golden-volume fixtures before exposing the result to users or Ollama.
This file now starts with the current implementation status. Older preparation/backlog sections are preserved below as historical planning context and should not be treated as the live sprint board without checking `docs/CODEX_EXECUTION_LOG.md`.
## Release hardening status