Initial public release

docs/DATASET_STRATEGY.md

@@ -0,0 +1,247 @@
+# Dataset Strategy
+
+## Goal
+Define exactly how GeoIntel obtains, stores, caches and uses source data. This prevents implementation drift during autonomous Codex passes.
+
+## Source priority
+| Priority | Source | Role | V1 status |
+|---:|---|---|---|
+| 1 | GRB / Basiskaart Vlaanderen | Professional reference vector data for Flemish features | Required foundation |
+| 2 | OSM | Fallback and supplementary vector context | Required foundation |
+| 3 | User-uploaded raster/vector data | Portfolio demo data and controlled test input | Required foundation |
+| 4 | Sentinel-2 L2A | Remote-sensing indices | V2 |
+| 5 | DHMV / DEM / DSM | Height analysis | V3 |
+| 6 | LAS / LAZ LiDAR | Point-cloud workbench | V4 |
+
+## GRB strategy
+GRB is treated as the most important V1 external reference dataset.
+
+### Why
+- It is the Flemish large-scale reference map.
+- It provides official topographic reference geometry for professional GIS workflows.
+- It is ideal for QA/QC of AI-generated building detections and other extracted features.
+
+### Access strategy
+Preferred V1 strategy:
+1. Implement `ReferenceLayerService` with a provider abstraction.
+2. First provider: WFS/remote fetch if service endpoint configuration is available.
+3. Second provider: local GeoPackage/GeoJSON/shapefile import for offline demos.
+4. Cache fetched features in PostGIS with a dataset version record.
+
+This dual approach avoids blocking the project on one external API while still aligning with the real GRB workflow.
+
+### V1 GRB feature classes
+Minimum required:
+- buildings / building ground geometry
+- roads or road-related reference geometry if easily available
+- water features if easily available
+
+V1 building QA only needs the building reference layer to be fully functional.
+
+### Cache policy
+- Cache by project area, layer name, provider and fetch date.
+- Store original provider metadata.
+- Store normalized geometry in PostGIS.
+- Do not mutate cached reference features during analysis.
+
+## OSM strategy
+OSM is used for fast, broad, fallback context.
+
+### Access options
+- Overpass API for small selected areas.
+- Local test fixtures for repeatable tests.
+- Later: Geofabrik extracts if performance becomes an issue.
+
+### V1 OSM layers
+- buildings
+- roads
+- water
+- green/landuse where available
+
+### OSM caveat
+OSM is community-maintained and may be incomplete. UI and reports must describe it as contextual/fallback data, not official ground truth.
+
+## Official modern land-use strategy
+
+The Departement Omgeving version 3 land-use maps are the authoritative modern
+source series for categorical land use. GeoIntel retrieves only explicit
+boundary subsets from MercatorNet WCS; no whole-Flanders raster is fetched at
+startup or during an interactive map query.
+
+- Source years: 2013, 2016, 2019, 2022 and 2025.
+- Native representation: one integer category per 10 m cell in EPSG:31370.
+- Preserved evidence: raw GeoTIFF, WCS request, catalogue URL, checksum, class
+  histogram and processing manifest.
+- Operational representation: source cells polygonized in EPSG:31370, clipped
+  to the requested boundary and normalized to EPSG:4326 `vector_features`.
+- Boundary behavior: retain every source cell touching the boundary, then
+  perform the exact metric geometry intersection; exclude and report NoData.
+- Selection metric: exact polygon intersection area within the source's 10 m
+  representation, reported in hectares with a non-cadastral warning.
+- Identity: raster polygons are not stable source objects; temporal object
+  lineage is unavailable.
+
+Modern land-use series must remain separate from the 1778, 1873 and 1969
+historical cartographic series. Comparisons are valid within each persisted
+series and must retain the source methodology warning. Extension from Mol to
+Kempen reuses the same operator with an explicit approved boundary, project,
+area and scope key; an ambiguous regional label is never converted into an
+invented boundary.
+
+## Operational Kempen scope
+
+The canonical regional workspace uses the official 28-municipality Vlaamse
+`Vervoerregio Kempen` policy boundary. `scripts/geographic_scopes.py` is the
+operator source of truth for its members and current NIS codes. The resulting
+regional geometry is a union of current authoritative VRBG municipality
+boundaries.
+
+- Project: `Kempen Regional Workbench`.
+- Regional Area: `Vervoerregio Kempen - officiële operationele grens`.
+- Member Areas: one current official boundary for every municipality.
+- Scope key: `kempen-transport-region`.
+- Provenance: authority URL, VRBG request URL, member codes, fetch timestamp,
+  artifact checksums and the policy-boundary limitation.
+- Scale policy: thematic providers are partitioned and validated separately;
+  no interactive request or application startup may download the entire
+  region or silently substitute missing data.
+
+### Regional building partition policy
+
+The current GRB building snapshot is fetched in one resumable partition per
+registered municipality. Partitions are source/provenance artifacts, not 28
+independent user-facing layers. A building intersecting more than one member
+boundary is owned by the member with the largest intersection area; the lower
+NIS code resolves exact ties. The retained geometry is clipped to the regional
+union, preserving complete in-scope coverage while guaranteeing one source ID
+per regional dataset.
+
+After all partitions pass validation, one combined immutable GeoJSON artifact
+is copied into managed storage and indexed in bounded batches through the
+normal DatasetService/VectorFeatureService ownership boundary. The resulting
+reference dataset uses `source_name=grb`, `reference_layer_name=buildings`,
+`coverage_scope=kempen-transport-region` and an exact `feature_count`
+selection aggregation. Existing viewport and bbox selection APIs remain the
+only interactive delivery path.
+
+### Regional GRB context partition policy
+
+Roads, water and parcels reuse the building operator's 28 resumable source
+partitions and one-dataset-per-theme persistence model. Polygon collections
+use maximum intersection area for cross-boundary ownership; line collections
+use maximum intersection length. Exact ties use the lower NIS code. Source
+identities are collection-qualified because the water theme combines `WTZ`,
+`WLAS` and `WGR`.
+
+The user-facing datasets use `source_name=grb`, `dataset_role=reference`,
+`coverage_scope=kempen-transport-region` and respectively
+`reference_layer_name=roads`, `water` or `parcels`. They are current source
+snapshots with immutable observation dates, not temporal series inferred from
+successive operator runs. Feature counts retain source-object semantics and
+must not be relabelled as road length, water area or legal parcel counts.
+
+Only `scripts/provision_regional_grb_context.py` may fetch these bounded
+artifacts. Application startup, map requests and the public provider registry
+remain fetch-free. All combined artifacts enter PostGIS through the existing
+DatasetService/VectorFeatureService boundary and existing bbox/Area selection
+contracts.
+
+## User-uploaded raster strategy
+V1 must support controlled local datasets because public raster access and model compatibility can be difficult.
+
+### Required upload types
+- GeoTIFF / TIFF where raster georeferencing is available.
+- JPG/PNG for non-georeferenced demo inference, with clear warning that outputs cannot be geospatially exported unless georeferencing is supplied.
+
+### Raster metadata required
+- CRS if available
+- bounds if available
+- transform if available
+- resolution if available
+- band count
+- nodata value if available
+- width/height
+- dtype
+
+## User-uploaded vector strategy
+Required formats:
+- GeoJSON
+- zipped shapefile
+- GeoPackage later if easy
+
+Required normalization:
+- validate geometry
+- determine CRS
+- reproject to canonical project CRS when necessary
+- store original metadata
+
+## Sentinel-2 strategy
+Not required in first build, but architecture must prepare for it.
+
+### Preferred discovery
+Use STAC-style catalogue access for Sentinel-2 L2A scenes once implemented.
+
+### V2 indices
+- NDVI = `(NIR - Red) / (NIR + Red)`
+- NDWI = `(Green - NIR) / (Green + NIR)`
+- NDBI = `(SWIR - NIR) / (SWIR + NIR)`
+
+### Cloud handling
+V2 should include cloud percentage filtering and warn when cloud contamination may affect outputs.
+
+## DHMV / height strategy
+Not V1. Store architecture placeholders only.
+
+Future uses:
+- elevation statistics
+- slope
+- low-point detection
+- water sensitivity proxy
+- building height if DSM and building footprints are available
+
+## Canonical CRS strategy
+- Store project areas and vector outputs in PostGIS with SRID metadata.
+- Use EPSG:4326 for API interchange when practical.
+- Use a metric projected CRS for area/length calculations. For Flanders, prefer Belgian Lambert 72 / EPSG:31370 or another documented metric CRS.
+- Never calculate area or distance on raw EPSG:4326 geometries.
+
+## Dataset versioning
+Every dataset must have a version record:
+- source name
+- provider
+- fetch/import date
+- original path or endpoint
+- CRS
+- bounds
+- checksum where applicable
+- processing status
+
+## V1 acceptance
+V1 dataset strategy is complete when:
+- a user can upload one raster and one vector dataset;
+- metadata is extracted and persisted;
+- an area can request/cache a reference building layer;
+- detection outputs can be compared with that reference layer;
+- exports include source metadata.
+
+## Temporal snapshots and evolution
+
+- A historical observation is one persisted dataset snapshot. Existing
+  datasets are not overwritten and features are not hidden in job JSON.
+- Related observations share a stable `temporal_series_key`; `observed_at`
+  records when the source describes reality, while `imported_at` records
+  ingestion time.
+- The latest-state map uses the latest available observation but must not call
+  an old source edition current reality.
+- Evolution metrics use the same selection geometry, aggregation and units for
+  both snapshots.
+- Partial-sector population is an area-weighted estimate. GeoIntel must not
+  imply address-level distribution when only sector totals are available.
+- Historical cartographic classes can change meaning between editions. Source
+  classes and processing notes remain provenance, and object changes require
+  explicit stable source identity.
+- Canonical `vector_features.geometry` is 2D EPSG:4326. Valid source Z values
+  are removed only from the query index, while the original upload remains
+  unchanged and `z_dimension_feature_count` records that normalization.
+- Dataset, version and vector-feature rows are committed atomically. Failed
+  geometry indexing rolls back all rows and removes the newly stored upload.