Initial public release

docs/DATA_SPECIFICATION.md

@@ -0,0 +1,557 @@
+# Data Specification v1.0
+
+## 1. Principes
+
+GeoIntel werkt documentatiegedreven met expliciete data-contracten.
+
+Elke dataset krijgt:
+
+- bron
+- type
+- formaat
+- CRS
+- bounds
+- resolutie of schaal
+- licentie
+- updatefrequentie
+- gebruiksdoel
+- cache-strategie
+- kwaliteitsstatus
+
+## 2. Kernbronnen
+
+## GRB — Basiskaart Vlaanderen
+
+### Rol
+
+GRB is een primaire referentiebron voor Vlaanderen. Voor GeoIntel is GRB belangrijker dan OSM wanneer officiële referentiegeometrieën nodig zijn.
+
+### Gebruik
+
+- referentiegebouwen
+- wegen en terreinobjecten
+- QA/QC tegenover AI-detecties
+- validatie van gebouwdetectie
+- vergelijking met eigen detecties of segmentaties
+
+### Type
+
+Vector.
+
+### Mogelijke toegang
+
+- WFS per gebied
+- downloadpakketten
+- periodieke PostGIS-cache
+
+### Cache-strategie
+
+Voor V1 is een gebiedsgerichte cache het beste:
+
+1. gebruiker selecteert gebied
+2. backend vraagt relevante GRB-lagen op
+3. geometrieën worden opgeslagen in PostGIS
+4. analysis runs verwijzen naar de gecachete versie
+
+### Belangrijk voor QA/QC
+
+GRB-gebouwpolygonen worden gebruikt als ground-truth proxy. Niet absoluut perfect, maar zeer bruikbaar als officiële referentie.
+
+## Gebouwenregister
+
+### Rol
+
+Authoritative register lifecycle and relation metadata attached to a dated
+building snapshot. It complements GRB; it does not replace the separate GRB
+footprint evidence or its QA role.
+
+### Gebruik
+
+- stable building object and version identity
+- official lifecycle status and geometry method
+- aggregate registered building-unit counts by lifecycle status
+- aggregate address counts by register status
+- classified reconciliation against persisted GRB geometry
+
+### Type
+
+Official OGC API Features input, normalized polygon GeoJSON upload and
+PostGIS `vector_features` output.
+
+### Persisted contract
+
+- `dataset_role=reference`
+- `source_name=digitaal_vlaanderen_buildings_addresses_register`
+- `reference_layer_name=building_registry`
+- `temporal_granularity=snapshot`
+- stable building identity within the register, but no fabricated historical
+  observation between snapshots
+- polygon geometry clipped in EPSG:31370 and persisted as EPSG:4326
+- selection metrics: exact footprint hectares, building lifecycle counts,
+  unit counts, address-status counts and confirmed GRB match counts
+
+### Privacy and semantics
+
+Queryable properties may contain register object ids, lifecycle status,
+geometry method, aggregate unit/address counts and GRB reconciliation evidence.
+They must not contain `VolledigAdres`, `Straatnaam`, `Huisnummer`,
+`HuisnummerLabel` or `Busnummer`. Address and unit counts may not be labelled
+as population, residents, households or dwellings. Building footprint is
+ground area, not floor area, height or volume.
+
+## OpenStreetMap
+
+### Rol
+
+Snelle open referentiedata, nuttig als fallback en voor POI's.
+
+### Gebruik
+
+- gebouwen indien GRB nog niet beschikbaar is
+- wegen
+- water
+- landuse
+- POI's
+- demo- en fallbackdata
+
+### Type
+
+Vector.
+
+### Toegang
+
+- Overpass API
+- pyrosm/osmnx
+- lokale extracten
+
+### Cache-strategie
+
+Opslaan per project/area in PostGIS.
+
+## Sentinel-2
+
+### Rol
+
+Remote sensing bron voor vegetatie, water en bebouwingsindices.
+
+### Gebruik
+
+- NDVI
+- NDWI
+- NDBI
+- seizoensvergelijking
+- trendanalyse
+- change detection
+
+### Type
+
+Raster multispectral.
+
+### Prioriteit
+
+V2.
+
+## DHMV / Hoogtemodel Vlaanderen
+
+### Rol
+
+Hoogtedata voor terrein- en watergevoeligheidsanalyse.
+
+### Gebruik
+
+- DTM-maaiveldhoogte in meter TAW
+- DSM-oppervlaktehoogte in meter TAW
+- reliëf en helling in graden uit geldige rastercellen
+- laagste punten als terreinindicator
+- gebouwhoogte-inschatting alleen in een latere, gevalideerde DSM-DTM/gebouwketen
+
+### Governed product
+
+- Digitaal Vlaanderen DHMV II, DTM/DSM raster 1 m
+- WCS coverages `DHMVII_DTM_1m` en `DHMVII_DSM_1m`
+- EPSG:31370, Float32, nodata `-9999`, verticale referentie TAW
+- opnameperiode 2013-2015
+- standaard GeoIntel-analysekopie 5 m, exact geclipt op Area
+
+Waterdiepte, waterinhoud, actuele toestand en afstroming zijn geen directe
+DHMV-metingen. Zij blijven onbeschikbaar tot een afzonderlijke bron en
+gevalideerde methode bestaan.
+
+### Type
+
+Raster / afgeleid van LiDAR.
+
+### Prioriteit
+
+P4 operationeel voor Mol; regionale uitrol volgt dezelfde operatorgrenzen.
+
+## VMM overstromingsgevaarkaarten waterdiepte
+
+### Rol
+
+Scenarioanalyse voor gemodelleerde overstroming door intense neerslag
+(`pluviaal`) en vanuit waterlopen (`fluviaal`). Dit is een afzonderlijk thema
+en geen verdieping van de gewone GRB-waterlaag.
+
+### Governed products
+
+- VMM OGRK WCS 1.1, twaalf vaste coverages
+- huidig klimaat en klimaatprojectie 2050
+- grote, middelgrote en kleine kans: T10, T100 en T1000
+- native raster 2 m, EPSG:31370, bronwaarden in centimeter
+- GeoIntel normaliseert positieve dieptes naar meter en bronnullen naar
+  transparant `-9999` nodata
+- standaard analysekopie 5 m, exact geclipt op de persisted Area
+
+### Toegestane metingen
+
+- gemodelleerd overstroomd oppervlak in hectare
+- aandeel van de selectie met positieve gemodelleerde diepte
+- gemiddelde, P90 en maximale lokale gemodelleerde maximumdiepte in meter
+- diepte-oppervlakte-integraal in m3, uitsluitend onder die exacte naam en met
+  de vaste waarschuwing dat lokale maxima niet noodzakelijk gelijktijdig zijn
+
+De laag meet geen huidige waterstand en bevat geen bodemhoogte. Bathymetrie,
+permanente inhoud van waterlichamen en gelijktijdig overstromingsvolume blijven
+onbeschikbaar. Scenario's zijn alternatieve modelcondities, geen historische
+meetmomenten en daarom geen GeoIntel-temporale reeks.
+
+### Type
+
+Raster / gemodelleerd overstromingsgevaar.
+
+### Prioriteit
+
+P5 overstromingsscenario's operationeel. VHA-dwarsprofielen zijn als
+historische puntmetingen operationeel voor Mol en partitioneerbaar voor alle
+officiële Vlaamse gemeente-Areas. Ze blijven strikt gescheiden van dit
+scenario-raster en leveren geen gebiedsdekkende bathymetrie of volume.
+
+## Bathymetry partition metadata
+
+Een VHA-profiel-Dataset behoudt `partition_area_id`,
+`partition_area_name`, `municipality`, meetdatumbereik en exacte aantallen.
+Regionale Vlaamse dekking is alleen geldig wanneer de server een volledig
+manifest voor de actuele officiële gemeente-inventaris heeft gevalideerd.
+Daarna bevatten Dataset en DatasetVersion:
+
+- `partition_scope_key`
+- `partition_count`, `data_partition_count` en `no_profile_partition_count`
+- `partition_manifest_sha256`
+- `partitioned_source_audit=true`
+- `regional_partitions_complete=true`
+
+Een no-profile partition is bronbewijs dat de begrensde VHA-query nul punten
+opleverde. Er worden nooit lege of kunstmatige profielobjecten aangemaakt.
+
+MDK Noordzee blijft `probe_only`: WCS-capabilities kunnen veilig worden
+gecontroleerd, maar er bestaat nog geen toegestane rasteracquisitie.
+
+## LAS/LAZ LiDAR
+
+### Rol
+
+Advanced module voor puntenwolkanalyse.
+
+### Gebruik
+
+- puntenwolk metadata
+- clipping
+- classificatie
+- DEM/DSM generatie
+- hoogteprofielen
+
+### Tools
+
+- PDAL
+- laspy
+
+### Prioriteit
+
+V4/V5.
+
+## 3. Dataset Types
+
+### Raster
+
+Ondersteund:
+
+- GeoTIFF
+- TIFF
+- JPEG2000 later
+- PNG/JPG met worldfile later
+
+Metadata:
+
+- CRS
+- transform
+- bounds
+- width/height
+- band count
+- dtype
+- nodata
+- resolution
+
+Temporal orthophoto rasters additionally require a governed product key, WMS
+layer, observation label, `observed_at`, optional `valid_from`/`valid_to`,
+temporal granularity, request/spatial hash, attribution and a limitation that
+states whether the product is annual, multi-year or merely most recent.
+
+A governed rolling `most_recent` release import must also retain the exact official
+`YYYY.NN` edition and the preflight identities for WMS capabilities, WCS
+coverage description, selected EPSG:31370 domain and sampled flight year. A
+legacy `most_recent_at_<date>` value is acquisition timing, not an official
+edition, and cannot be compared as if it were one. The read-only preflight
+creates no Dataset. Stage retains the exact source response, normalized
+three-band EPSG:31370 GeoTIFF and review preview. Apply requires exact plan and
+review hashes, persists sampled official flight dates as the temporal evidence
+range and creates a new Dataset/DatasetVersion through DatasetService. It does
+not retroactively rewrite or delete legacy rows.
+
+WALOUS land-cover rasters retain EPSG:3812, their native 1 m source resolution,
+the derived analysis resolution, source checksum and exact observation range.
+The governed class domain is `{1,2,3,4,5,6,7,8,9,80,90}`. Codes 80 and 90 are
+valid official low-woody-cover classes; `10` and `11` are not substitutes.
+Bounded derivatives use `uint8`, nodata 255 and nearest-neighbour resampling.
+Area metrics group forest/tree cover as `{8,9,80,90}`, water as `{5}`,
+artificial cover as `{1,2,3}`, rotating herbaceous cover as `{6}`, continuous
+herbaceous cover as `{7}` and bare soil as `{4}`.
+
+The 2018 WALOUS edition additionally retains the original stacked source codes
+and the complete official `Classe vue` crosswalk in provenance. Code `62`
+(greenhouses) is normalized to canonical construction class `2`; this and the
+2018 production-method difference must be displayed as a temporal comparison
+limitation. Source value `0` is explicit implicit-background/nodata and is
+never counted as a land-cover class.
+
+The Wallonia MNT source retains EPSG:3812, native 1 m resolution, Float terrain
+values, source checksum, acquisition period 2021-02-19/2022-03-05 and vertical
+reference EPSG:5710 (DNG). Bounded derivatives use Float32, nodata `-9999`,
+bilinear resampling and the selected Area as an exact mask. DNG and TAW are not
+interchanged.
+
+### Hydrological station observations
+
+Waterinfo observations are persisted as EPSG:4326 Point features, one station
+and one observation year per immutable Dataset. Required properties are the
+station/timeseries identity, measurement type, numeric annual value, reported
+unit, observation year, provider/owner and attribution. Different station
+series may not be merged into one area-wide value. Point water level and
+discharge may not be converted to water volume without governed compatible
+depth/profile data.
+
+### BWK and Natura 2000 polygons
+
+The governed state-2025 import uses `BWK:Bwkhab` polygons. Geometry is fetched
+in EPSG:4326, validated and clipped against the persisted Mol boundary in
+EPSG:31370, then persisted in EPSG:4326. Required retained fields include
+`EVAL`, `EENH1..8`, `V1..3`, `HERK`, `BWKLABEL`, `HAB1..5`, `PHAB1..5`,
+`HERKHAB`, `HERKPHAB` and `HABLEGENDE`.
+
+BWK value classes remain independent. Natura 2000 codes, regionally important
+biotope codes and uncertain `ohab` knowledge gaps are also independent. PHAB
+shares can be theoretical source allocations; derived hectares therefore keep
+an estimate warning, especially for partial rectangle intersections. State
+2025 is a product edition and must not be presented as a uniform 2025 field
+survey or an annual time series.
+
+### Annual agricultural-use parcels
+
+Definitive Agentschap Landbouw en Zeevisserij editions from 2008 through 2025
+are stored as separate annual vector Datasets. The retained source is the
+official ZIP archive containing a Belgian Lambert 72 GeoPackage. Queryable
+geometry is clipped against the persisted Area in EPSG:31370 and normalized to
+EPSG:4326 before canonical `vector_features` persistence.
+
+A future campaign becomes historical input only when the official publication
+contract labels it `v3`. Early `v1` and `v2` snapshots are provisional and may
+not create annual GeoIntel Datasets. The governed release plan binds the exact
+archive publication date, schema, CRS, crop-code list, scope totals and deltas
+against the previous definitive edition before named review and apply.
+
+Stable source fields include `agpakey`, parcel number, declared source area,
+reference id, spring crop, main crop, official main-crop group, production
+method and source municipality. All annual source properties remain available,
+but only `maincropgroup_title` is normalized to a controlled comparison key.
+Detailed crop codes and titles can change meaning or wording between editions;
+their complete annual code list is therefore retained with the source archive.
+
+Historical analysis compares intersected hectares for the complete declaration
+and official main-crop groups. It must not compare individual parcel lineage:
+identifiers, boundaries and declarations are not stable between campaign years.
+The declaration can include water, hedges, buildings and infrastructure, so its
+total is not a cadastral ownership area or cultivated-crop area.
+
+### Vector
+
+Ondersteund:
+
+- GeoJSON
+- Shapefile
+- GPKG
+- WFS-resultaten
+
+### National and maritime coverage contracts
+
+Belgium-wide and maritime reference Datasets use the same Dataset and
+VectorFeature entities as every other vector source. No parallel national
+schema is introduced.
+
+Required source metadata includes:
+
+```yaml
+authority_level: authoritative|official_context|contextual
+coverage_zones:
+  - belgium|flanders|wallonia|brussels
+  - belgian_north_sea|territorial_sea|exclusive_economic_zone|continental_shelf
+source_url: https://...
+attribution: text
+license_note: text
+reference_layer_name: normalized_layer_key
+```
+
+EEZ and continental-shelf records may have equal horizontal geometry while
+their `legal_domain` differs. They must remain distinct records. A selected
+bbox crossing regions or the coast produces zone-specific coverage items;
+only metrics with compatible source meaning, unit, time and method may be
+merged.
+
+Metadata:
+
+- CRS
+- bounds
+- geometry type
+- feature count
+- attributes
+- validity summary
+
+### Model outputs
+
+Ondersteund:
+
+- bounding boxes
+- polygons
+- masks
+- confidence scores
+- class labels
+- source tile reference
+
+## 4. Data Quality Requirements
+
+Elke dataset moet bij intake minimaal controleren:
+
+- bestand leesbaar
+- CRS aanwezig of expliciet onbekend
+- geometrieën geldig of herstelbaar
+- bounds binnen verwacht gebied
+- raster heeft bruikbare resolutie
+- geen lege lagen
+- opslagpad bestaat
+
+## 5. Data Lineage
+
+Elke output moet kunnen terugwijzen naar:
+
+- bronbestand
+- processing step
+- analysis run
+- modelversie
+- parameters
+- tijdstip
+- eventuele referentielaag
+
+## Sprint 7B Provider Contracts
+
+Sprint 7B makes provider architecture operationally visible without performing live external fetches.
+
+### GRB
+
+- provider_name: `grb`
+- authority_level: `authoritative`
+- configured: `false`
+- status: `not_configured`
+- supported_layers: `buildings`, `roads`, `parcels`
+- supported_geometry_types: `Polygon`, `MultiPolygon`, `LineString`, `MultiLineString`
+- supported_query_modes: `area`
+- dataset mapping: `dataset_role=reference`, `source_name=grb`
+- write path: future provider output must flow through `DatasetService` and `VectorFeatureService`
+- limitation: no live WFS call, no download, no fake GRB data in Sprint 7B
+
+### OSM
+
+- provider_name: `osm`
+- authority_level: `contextual`
+- configured: `false`
+- status: `not_configured`
+- supported_layers: `buildings`, `roads`, `water`, `landuse`
+- supported_geometry_types: `Polygon`, `MultiPolygon`, `LineString`, `MultiLineString`
+- supported_query_modes: `area`
+- default dataset mapping: `dataset_role=source`, `source_name=osm`
+- explicit reference mapping: `dataset_role=reference`, `source_name=osm`
+- write path: future provider output must flow through `DatasetService` and `VectorFeatureService`
+- limitation: no live Overpass call, no download, no fake OSM data in Sprint 7B
+
+### Manual
+
+- provider_name: `manual`
+- authority_level: `manual`
+- configured: `true`
+- status: `configured`
+- dataset mapping: `dataset_role=reference`, `source_name=manual`
+- write path: existing dataset upload/reference flow
+
+### Fixture
+
+- provider_name: `fixture`
+- authority_level: `fixture`
+- configured: `true`
+- status: `configured`
+- dataset mapping: `dataset_role=reference`, `source_name=fixture`
+- write path: checked-in demo/test fixture flow
+
+## Bathymetry profile vector contract
+
+The operational VHA layer is an EPSG:4326 point FeatureCollection. Required
+normalized properties are:
+
+- `provider_record_id` and `source_feature_id`;
+- `watercourse_vhag`, `watercourse_name` and alternative names;
+- `profile_number` and `measurement_date`;
+- nullable `recorded_depth_m`, `recorded_crown_width_m` and
+  `recorded_floor_width_m`;
+- nullable allowlisted `source_document_url`;
+- `document_available`, `structured_depth_available`;
+- `measurement_semantics=historical_cross_section_profile_point`;
+- `vertical_reference=document-specific`.
+
+Null means the provider did not expose a structured value. It is never
+converted to zero. Dataset metadata records exact counts, measurement range,
+scope, attribution and `volume_supported=false`.
+# Operational Flanders theme specification
+
+## Land-use forest and agriculture masks
+
+The governed 2025 Landgebruik Vlaanderen v3 coverage is categorical. GeoIntel
+derives two product-specific binary rasters only after validating integer
+source classes: forest class 12, and agricultural land-use classes 13 and 14.
+The measurement is intersected grid area in hectares at 10 m resolution.
+Forest volume, tree count, legal forest status, agricultural crop declaration,
+ownership and zoning are unsupported. The agricultural mask does not replace
+the temporal ALZ parcel series.
+
+## BWK and Natura 2000 polygons
+
+The `bwk_natura2000_2025` product follows stable allowlisted WFS 2.0 pagination,
+clips geometries in Lambert 72 and persists through DatasetService. Official
+`EVAL`, `EENH1..8`, `HAB1..5`, `PHAB1..5`, `HERK`, `HERKHAB`,
+`HERKPHAB` and `HABLEGENDE` remain available. BWK class areas are exact
+intersections; PHAB-derived habitat areas remain estimates.
+
+## DOV soil polygons
+
+The `dov_soil_types` product persists polygon geometry as EPSG:4326 and
+clips/measures in EPSG:31370. Soil type, unified type, series, generalized
+legend, texture, drainage, profile, substrate and region remain source
+attributes. The observation is the 1949-1971 field-survey period and the
+digital edition is June 2017. It is an authoritative historical baseline at
+1:20,000, not evidence of current drainage or parcel-level site conditions.