geointel/docs/DATASET_STRATEGY.md

Dataset Strategy

Goal

Define exactly how GeoIntel obtains, stores, caches and uses source data. This prevents implementation drift during autonomous Codex passes.

Source priority

Priority	Source	Role	V1 status
1	GRB / Basiskaart Vlaanderen	Professional reference vector data for Flemish features	Required foundation
2	OSM	Fallback and supplementary vector context	Required foundation
3	User-uploaded raster/vector data	Portfolio demo data and controlled test input	Required foundation
4	Sentinel-2 L2A	Remote-sensing indices	V2
5	DHMV / DEM / DSM	Height analysis	V3
6	LAS / LAZ LiDAR	Point-cloud workbench	V4

GRB strategy

GRB is treated as the most important V1 external reference dataset.

Why

• It is the Flemish large-scale reference map.
• It provides official topographic reference geometry for professional GIS workflows.
• It is ideal for QA/QC of AI-generated building detections and other extracted features.

Access strategy

Preferred V1 strategy:

1. Implement ReferenceLayerService with a provider abstraction.
2. First provider: WFS/remote fetch if service endpoint configuration is available.
3. Second provider: local GeoPackage/GeoJSON/shapefile import for offline demos.
4. Cache fetched features in PostGIS with a dataset version record.

This dual approach avoids blocking the project on one external API while still aligning with the real GRB workflow.

V1 GRB feature classes

Minimum required:

• buildings / building ground geometry
• roads or road-related reference geometry if easily available
• water features if easily available

V1 building QA only needs the building reference layer to be fully functional.

Cache policy

• Cache by project area, layer name, provider and fetch date.
• Store original provider metadata.
• Store normalized geometry in PostGIS.
• Do not mutate cached reference features during analysis.

OSM strategy

OSM is used for fast, broad, fallback context.

Access options

• Overpass API for small selected areas.
• Local test fixtures for repeatable tests.
• Later: Geofabrik extracts if performance becomes an issue.

V1 OSM layers

• buildings
• roads
• water
• green/landuse where available

OSM caveat

OSM is community-maintained and may be incomplete. UI and reports must describe it as contextual/fallback data, not official ground truth.

Official modern land-use strategy

The Departement Omgeving version 3 land-use maps are the authoritative modern source series for categorical land use. GeoIntel retrieves only explicit boundary subsets from MercatorNet WCS; no whole-Flanders raster is fetched at startup or during an interactive map query.

• Source years: 2013, 2016, 2019, 2022 and 2025.
• Native representation: one integer category per 10 m cell in EPSG:31370.
• Preserved evidence: raw GeoTIFF, WCS request, catalogue URL, checksum, class histogram and processing manifest.
• Operational representation: source cells polygonized in EPSG:31370, clipped to the requested boundary and normalized to EPSG:4326 vector_features.
• Boundary behavior: retain every source cell touching the boundary, then perform the exact metric geometry intersection; exclude and report NoData.
• Selection metric: exact polygon intersection area within the source's 10 m representation, reported in hectares with a non-cadastral warning.
• Identity: raster polygons are not stable source objects; temporal object lineage is unavailable.

Modern land-use series must remain separate from the 1778, 1873 and 1969 historical cartographic series. Comparisons are valid within each persisted series and must retain the source methodology warning. Extension from Mol to Kempen reuses the same operator with an explicit approved boundary, project, area and scope key; an ambiguous regional label is never converted into an invented boundary.

User-uploaded raster strategy

V1 must support controlled local datasets because public raster access and model compatibility can be difficult.

Required upload types

• GeoTIFF / TIFF where raster georeferencing is available.
• JPG/PNG for non-georeferenced demo inference, with clear warning that outputs cannot be geospatially exported unless georeferencing is supplied.

Raster metadata required

• CRS if available
• bounds if available
• transform if available
• resolution if available
• band count
• nodata value if available
• width/height
• dtype

User-uploaded vector strategy

Required formats:

• GeoJSON
• zipped shapefile
• GeoPackage later if easy

Required normalization:

• validate geometry
• determine CRS
• reproject to canonical project CRS when necessary
• store original metadata

Sentinel-2 strategy

Not required in first build, but architecture must prepare for it.

Preferred discovery

Use STAC-style catalogue access for Sentinel-2 L2A scenes once implemented.

V2 indices

• NDVI = (NIR - Red) / (NIR + Red)
• NDWI = (Green - NIR) / (Green + NIR)
• NDBI = (SWIR - NIR) / (SWIR + NIR)

Cloud handling

V2 should include cloud percentage filtering and warn when cloud contamination may affect outputs.

DHMV / height strategy

Not V1. Store architecture placeholders only.

Future uses:

• elevation statistics
• slope
• low-point detection
• water sensitivity proxy
• building height if DSM and building footprints are available

Canonical CRS strategy

• Store project areas and vector outputs in PostGIS with SRID metadata.
• Use EPSG:4326 for API interchange when practical.
• Use a metric projected CRS for area/length calculations. For Flanders, prefer Belgian Lambert 72 / EPSG:31370 or another documented metric CRS.
• Never calculate area or distance on raw EPSG:4326 geometries.

Dataset versioning

Every dataset must have a version record:

• source name
• provider
• fetch/import date
• original path or endpoint
• CRS
• bounds
• checksum where applicable
• processing status

V1 acceptance

V1 dataset strategy is complete when:

• a user can upload one raster and one vector dataset;
• metadata is extracted and persisted;
• an area can request/cache a reference building layer;
• detection outputs can be compared with that reference layer;
• exports include source metadata.

Temporal snapshots and evolution

• A historical observation is one persisted dataset snapshot. Existing datasets are not overwritten and features are not hidden in job JSON.
• Related observations share a stable temporal_series_key; observed_at records when the source describes reality, while imported_at records ingestion time.
• The latest-state map uses the latest available observation but must not call an old source edition current reality.
• Evolution metrics use the same selection geometry, aggregation and units for both snapshots.
• Partial-sector population is an area-weighted estimate. GeoIntel must not imply address-level distribution when only sector totals are available.
• Historical cartographic classes can change meaning between editions. Source classes and processing notes remain provenance, and object changes require explicit stable source identity.
• Canonical vector_features.geometry is 2D EPSG:4326. Valid source Z values are removed only from the query index, while the original upload remains unchanged and z_dimension_feature_count records that normalization.
• Dataset, version and vector-feature rows are committed atomically. Failed geometry indexing rolls back all rows and removes the newly stored upload.