geointel/docs/DATA_SPECIFICATION.md

# Data Specification v1.0

## 1. Principes

GeoIntel werkt documentatiegedreven met expliciete data-contracten.

Elke dataset krijgt:

- bron
- type
- formaat
- CRS
- bounds
- resolutie of schaal
- licentie
- updatefrequentie
- gebruiksdoel
- cache-strategie
- kwaliteitsstatus

## 2. Kernbronnen

## GRB — Basiskaart Vlaanderen

### Rol

GRB is een primaire referentiebron voor Vlaanderen. Voor GeoIntel is GRB belangrijker dan OSM wanneer officiële referentiegeometrieën nodig zijn.

### Gebruik

- referentiegebouwen
- wegen en terreinobjecten
- QA/QC tegenover AI-detecties
- validatie van gebouwdetectie
- vergelijking met eigen detecties of segmentaties

### Type

Vector.

### Mogelijke toegang

- WFS per gebied
- downloadpakketten
- periodieke PostGIS-cache

### Cache-strategie

Voor V1 is een gebiedsgerichte cache het beste:

1. gebruiker selecteert gebied
2. backend vraagt relevante GRB-lagen op
3. geometrieën worden opgeslagen in PostGIS
4. analysis runs verwijzen naar de gecachete versie

### Belangrijk voor QA/QC

GRB-gebouwpolygonen worden gebruikt als ground-truth proxy. Niet absoluut perfect, maar zeer bruikbaar als officiële referentie.

## Gebouwenregister

### Rol

Authoritative register lifecycle and relation metadata attached to a dated
building snapshot. It complements GRB; it does not replace the separate GRB
footprint evidence or its QA role.

### Gebruik

- stable building object and version identity
- official lifecycle status and geometry method
- aggregate registered building-unit counts by lifecycle status
- aggregate address counts by register status
- classified reconciliation against persisted GRB geometry

### Type

Official OGC API Features input, normalized polygon GeoJSON upload and
PostGIS `vector_features` output.

### Persisted contract

- `dataset_role=reference`
- `source_name=digitaal_vlaanderen_buildings_addresses_register`
- `reference_layer_name=building_registry`
- `temporal_granularity=snapshot`
- stable building identity within the register, but no fabricated historical
  observation between snapshots
- polygon geometry clipped in EPSG:31370 and persisted as EPSG:4326
- selection metrics: exact footprint hectares, building lifecycle counts,
  unit counts, address-status counts and confirmed GRB match counts

### Privacy and semantics

Queryable properties may contain register object ids, lifecycle status,
geometry method, aggregate unit/address counts and GRB reconciliation evidence.
They must not contain `VolledigAdres`, `Straatnaam`, `Huisnummer`,
`HuisnummerLabel` or `Busnummer`. Address and unit counts may not be labelled
as population, residents, households or dwellings. Building footprint is
ground area, not floor area, height or volume.

## OpenStreetMap

### Rol

Snelle open referentiedata, nuttig als fallback en voor POI's.

### Gebruik

- gebouwen indien GRB nog niet beschikbaar is
- wegen
- water
- landuse
- POI's
- demo- en fallbackdata

### Type

Vector.

### Toegang

- Overpass API
- pyrosm/osmnx
- lokale extracten

### Cache-strategie

Opslaan per project/area in PostGIS.

## Sentinel-2

### Rol

Remote sensing bron voor vegetatie, water en bebouwingsindices.

### Gebruik

- NDVI
- NDWI
- NDBI
- seizoensvergelijking
- trendanalyse
- change detection

### Type

Raster multispectral.

### Prioriteit

V2.

## DHMV / Hoogtemodel Vlaanderen

### Rol

Hoogtedata voor terrein- en watergevoeligheidsanalyse.

### Gebruik

- DTM-maaiveldhoogte in meter TAW
- DSM-oppervlaktehoogte in meter TAW
- reliëf en helling in graden uit geldige rastercellen
- laagste punten als terreinindicator
- gebouwhoogte-inschatting alleen in een latere, gevalideerde DSM-DTM/gebouwketen

### Governed product

- Digitaal Vlaanderen DHMV II, DTM/DSM raster 1 m
- WCS coverages `DHMVII_DTM_1m` en `DHMVII_DSM_1m`
- EPSG:31370, Float32, nodata `-9999`, verticale referentie TAW
- opnameperiode 2013-2015
- standaard GeoIntel-analysekopie 5 m, exact geclipt op Area

Waterdiepte, waterinhoud, actuele toestand en afstroming zijn geen directe
DHMV-metingen. Zij blijven onbeschikbaar tot een afzonderlijke bron en
gevalideerde methode bestaan.

### Type

Raster / afgeleid van LiDAR.

### Prioriteit

P4 operationeel voor Mol; regionale uitrol volgt dezelfde operatorgrenzen.

## VMM overstromingsgevaarkaarten waterdiepte

### Rol

Scenarioanalyse voor gemodelleerde overstroming door intense neerslag
(`pluviaal`) en vanuit waterlopen (`fluviaal`). Dit is een afzonderlijk thema
en geen verdieping van de gewone GRB-waterlaag.

### Governed products

- VMM OGRK WCS 1.1, twaalf vaste coverages
- huidig klimaat en klimaatprojectie 2050
- grote, middelgrote en kleine kans: T10, T100 en T1000
- native raster 2 m, EPSG:31370, bronwaarden in centimeter
- GeoIntel normaliseert positieve dieptes naar meter en bronnullen naar
  transparant `-9999` nodata
- standaard analysekopie 5 m, exact geclipt op de persisted Area

### Toegestane metingen

- gemodelleerd overstroomd oppervlak in hectare
- aandeel van de selectie met positieve gemodelleerde diepte
- gemiddelde, P90 en maximale lokale gemodelleerde maximumdiepte in meter
- diepte-oppervlakte-integraal in m3, uitsluitend onder die exacte naam en met
  de vaste waarschuwing dat lokale maxima niet noodzakelijk gelijktijdig zijn

De laag meet geen huidige waterstand en bevat geen bodemhoogte. Bathymetrie,
permanente inhoud van waterlichamen en gelijktijdig overstromingsvolume blijven
onbeschikbaar. Scenario's zijn alternatieve modelcondities, geen historische
meetmomenten en daarom geen GeoIntel-temporale reeks.

### Type

Raster / gemodelleerd overstromingsgevaar.

### Prioriteit

P5 overstromingsscenario's operationeel. VHA-dwarsprofielen zijn als
historische puntmetingen operationeel voor Mol en partitioneerbaar voor alle
officiële Vlaamse gemeente-Areas. Ze blijven strikt gescheiden van dit
scenario-raster en leveren geen gebiedsdekkende bathymetrie of volume.

## Bathymetry partition metadata

Een VHA-profiel-Dataset behoudt `partition_area_id`,
`partition_area_name`, `municipality`, meetdatumbereik en exacte aantallen.
Regionale Vlaamse dekking is alleen geldig wanneer de server een volledig
manifest voor de actuele officiële gemeente-inventaris heeft gevalideerd.
Daarna bevatten Dataset en DatasetVersion:

- `partition_scope_key`
- `partition_count`, `data_partition_count` en `no_profile_partition_count`
- `partition_manifest_sha256`
- `partitioned_source_audit=true`
- `regional_partitions_complete=true`

Een no-profile partition is bronbewijs dat de begrensde VHA-query nul punten
opleverde. Er worden nooit lege of kunstmatige profielobjecten aangemaakt.

MDK Noordzee blijft `probe_only`: WCS-capabilities kunnen veilig worden
gecontroleerd, maar er bestaat nog geen toegestane rasteracquisitie.

## LAS/LAZ LiDAR

### Rol

Advanced module voor puntenwolkanalyse.

### Gebruik

- puntenwolk metadata
- clipping
- classificatie
- DEM/DSM generatie
- hoogteprofielen

### Tools

- PDAL
- laspy

### Prioriteit

V4/V5.

## 3. Dataset Types

### Raster

Ondersteund:

- GeoTIFF
- TIFF
- JPEG2000 later
- PNG/JPG met worldfile later

Metadata:

- CRS
- transform
- bounds
- width/height
- band count
- dtype
- nodata
- resolution

Temporal orthophoto rasters additionally require a governed product key, WMS
layer, observation label, `observed_at`, optional `valid_from`/`valid_to`,
temporal granularity, request/spatial hash, attribution and a limitation that
states whether the product is annual, multi-year or merely most recent.

A governed rolling `most_recent` release import must also retain the exact official
`YYYY.NN` edition and the preflight identities for WMS capabilities, WCS
coverage description, selected EPSG:31370 domain and sampled flight year. A
legacy `most_recent_at_<date>` value is acquisition timing, not an official
edition, and cannot be compared as if it were one. The read-only preflight
creates no Dataset. Stage retains the exact source response, normalized
three-band EPSG:31370 GeoTIFF and review preview. Apply requires exact plan and
review hashes, persists sampled official flight dates as the temporal evidence
range and creates a new Dataset/DatasetVersion through DatasetService. It does
not retroactively rewrite or delete legacy rows.

WALOUS land-cover rasters retain EPSG:3812, their native 1 m source resolution,
the derived analysis resolution, source checksum and exact observation range.
The governed class domain is `{1,2,3,4,5,6,7,8,9,80,90}`. Codes 80 and 90 are
valid official low-woody-cover classes; `10` and `11` are not substitutes.
Bounded derivatives use `uint8`, nodata 255 and nearest-neighbour resampling.
Area metrics group forest/tree cover as `{8,9,80,90}`, water as `{5}`,
artificial cover as `{1,2,3}`, rotating herbaceous cover as `{6}`, continuous
herbaceous cover as `{7}` and bare soil as `{4}`.

The 2018 WALOUS edition additionally retains the original stacked source codes
and the complete official `Classe vue` crosswalk in provenance. Code `62`
(greenhouses) is normalized to canonical construction class `2`; this and the
2018 production-method difference must be displayed as a temporal comparison
limitation. Source value `0` is explicit implicit-background/nodata and is
never counted as a land-cover class.

The Wallonia MNT source retains EPSG:3812, native 1 m resolution, Float terrain
values, source checksum, acquisition period 2021-02-19/2022-03-05 and vertical
reference EPSG:5710 (DNG). Bounded derivatives use Float32, nodata `-9999`,
bilinear resampling and the selected Area as an exact mask. DNG and TAW are not
interchanged.

### Hydrological station observations

Waterinfo observations are persisted as EPSG:4326 Point features, one station
and one observation year per immutable Dataset. Required properties are the
station/timeseries identity, measurement type, numeric annual value, reported
unit, observation year, provider/owner and attribution. Different station
series may not be merged into one area-wide value. Point water level and
discharge may not be converted to water volume without governed compatible
depth/profile data.

### BWK and Natura 2000 polygons

The governed state-2025 import uses `BWK:Bwkhab` polygons. Geometry is fetched
in EPSG:4326, validated and clipped against the persisted Mol boundary in
EPSG:31370, then persisted in EPSG:4326. Required retained fields include
`EVAL`, `EENH1..8`, `V1..3`, `HERK`, `BWKLABEL`, `HAB1..5`, `PHAB1..5`,
`HERKHAB`, `HERKPHAB` and `HABLEGENDE`.

BWK value classes remain independent. Natura 2000 codes, regionally important
biotope codes and uncertain `ohab` knowledge gaps are also independent. PHAB
shares can be theoretical source allocations; derived hectares therefore keep
an estimate warning, especially for partial rectangle intersections. State
2025 is a product edition and must not be presented as a uniform 2025 field
survey or an annual time series.

### Annual agricultural-use parcels

Definitive Agentschap Landbouw en Zeevisserij editions from 2008 through 2025
are stored as separate annual vector Datasets. The retained source is the
official ZIP archive containing a Belgian Lambert 72 GeoPackage. Queryable
geometry is clipped against the persisted Area in EPSG:31370 and normalized to
EPSG:4326 before canonical `vector_features` persistence.

A future campaign becomes historical input only when the official publication
contract labels it `v3`. Early `v1` and `v2` snapshots are provisional and may
not create annual GeoIntel Datasets. The governed release plan binds the exact
archive publication date, schema, CRS, crop-code list, scope totals and deltas
against the previous definitive edition before named review and apply.

Stable source fields include `agpakey`, parcel number, declared source area,
reference id, spring crop, main crop, official main-crop group, production
method and source municipality. All annual source properties remain available,
but only `maincropgroup_title` is normalized to a controlled comparison key.
Detailed crop codes and titles can change meaning or wording between editions;
their complete annual code list is therefore retained with the source archive.

Historical analysis compares intersected hectares for the complete declaration
and official main-crop groups. It must not compare individual parcel lineage:
identifiers, boundaries and declarations are not stable between campaign years.
The declaration can include water, hedges, buildings and infrastructure, so its
total is not a cadastral ownership area or cultivated-crop area.

### Vector

Ondersteund:

- GeoJSON
- Shapefile
- GPKG
- WFS-resultaten

### National and maritime coverage contracts

Belgium-wide and maritime reference Datasets use the same Dataset and
VectorFeature entities as every other vector source. No parallel national
schema is introduced.

Required source metadata includes:

```yaml
authority_level: authoritative|official_context|contextual
coverage_zones:
  - belgium|flanders|wallonia|brussels
  - belgian_north_sea|territorial_sea|exclusive_economic_zone|continental_shelf
source_url: https://...
attribution: text
license_note: text
reference_layer_name: normalized_layer_key
```

EEZ and continental-shelf records may have equal horizontal geometry while
their `legal_domain` differs. They must remain distinct records. A selected
bbox crossing regions or the coast produces zone-specific coverage items;
only metrics with compatible source meaning, unit, time and method may be
merged.

Metadata:

- CRS
- bounds
- geometry type
- feature count
- attributes
- validity summary

### Model outputs

Ondersteund:

- bounding boxes
- polygons
- masks
- confidence scores
- class labels
- source tile reference

## 4. Data Quality Requirements

Elke dataset moet bij intake minimaal controleren:

- bestand leesbaar
- CRS aanwezig of expliciet onbekend
- geometrieën geldig of herstelbaar
- bounds binnen verwacht gebied
- raster heeft bruikbare resolutie
- geen lege lagen
- opslagpad bestaat

## 5. Data Lineage

Elke output moet kunnen terugwijzen naar:

- bronbestand
- processing step
- analysis run
- modelversie
- parameters
- tijdstip
- eventuele referentielaag

## Sprint 7B Provider Contracts

Sprint 7B makes provider architecture operationally visible without performing live external fetches.

### GRB

- provider_name: `grb`
- authority_level: `authoritative`
- configured: `false`
- status: `not_configured`
- supported_layers: `buildings`, `roads`, `parcels`
- supported_geometry_types: `Polygon`, `MultiPolygon`, `LineString`, `MultiLineString`
- supported_query_modes: `area`
- dataset mapping: `dataset_role=reference`, `source_name=grb`
- write path: future provider output must flow through `DatasetService` and `VectorFeatureService`
- limitation: no live WFS call, no download, no fake GRB data in Sprint 7B

### OSM

- provider_name: `osm`
- authority_level: `contextual`
- configured: `false`
- status: `not_configured`
- supported_layers: `buildings`, `roads`, `water`, `landuse`
- supported_geometry_types: `Polygon`, `MultiPolygon`, `LineString`, `MultiLineString`
- supported_query_modes: `area`
- default dataset mapping: `dataset_role=source`, `source_name=osm`
- explicit reference mapping: `dataset_role=reference`, `source_name=osm`
- write path: future provider output must flow through `DatasetService` and `VectorFeatureService`
- limitation: no live Overpass call, no download, no fake OSM data in Sprint 7B

### Manual

- provider_name: `manual`
- authority_level: `manual`
- configured: `true`
- status: `configured`
- dataset mapping: `dataset_role=reference`, `source_name=manual`
- write path: existing dataset upload/reference flow

### Fixture

- provider_name: `fixture`
- authority_level: `fixture`
- configured: `true`
- status: `configured`
- dataset mapping: `dataset_role=reference`, `source_name=fixture`
- write path: checked-in demo/test fixture flow

## Bathymetry profile vector contract

The operational VHA layer is an EPSG:4326 point FeatureCollection. Required
normalized properties are:

- `provider_record_id` and `source_feature_id`;
- `watercourse_vhag`, `watercourse_name` and alternative names;
- `profile_number` and `measurement_date`;
- nullable `recorded_depth_m`, `recorded_crown_width_m` and
  `recorded_floor_width_m`;
- nullable allowlisted `source_document_url`;
- `document_available`, `structured_depth_available`;
- `measurement_semantics=historical_cross_section_profile_point`;
- `vertical_reference=document-specific`.

Null means the provider did not expose a structured value. It is never
converted to zero. Dataset metadata records exact counts, measurement range,
scope, attribution and `volume_supported=false`.
# Operational Flanders theme specification

## Land-use forest and agriculture masks

The governed 2025 Landgebruik Vlaanderen v3 coverage is categorical. GeoIntel
derives two product-specific binary rasters only after validating integer
source classes: forest class 12, and agricultural land-use classes 13 and 14.
The measurement is intersected grid area in hectares at 10 m resolution.
Forest volume, tree count, legal forest status, agricultural crop declaration,
ownership and zoning are unsupported. The agricultural mask does not replace
the temporal ALZ parcel series.

## BWK and Natura 2000 polygons

The `bwk_natura2000_2025` product follows stable allowlisted WFS 2.0 pagination,
clips geometries in Lambert 72 and persists through DatasetService. Official
`EVAL`, `EENH1..8`, `HAB1..5`, `PHAB1..5`, `HERK`, `HERKHAB`,
`HERKPHAB` and `HABLEGENDE` remain available. BWK class areas are exact
intersections; PHAB-derived habitat areas remain estimates.

## DOV soil polygons

The `dov_soil_types` product persists polygon geometry as EPSG:4326 and
clips/measures in EPSG:31370. Soil type, unified type, series, generalized
legend, texture, drainage, profile, substrate and region remain source
attributes. The observation is the 1949-1971 field-survey period and the
digital edition is June 2017. It is an authoritative historical baseline at
1:20,000, not evidence of current drainage or parcel-level site conditions.