geointel/docs/DATASET_STRATEGY.md

Dataset Strategy

Goal

Define exactly how GeoIntel obtains, stores, caches and uses source data. This prevents implementation drift during autonomous Codex passes.

Source priority

Priority	Source	Role	V1 status
1	GRB / Basiskaart Vlaanderen	Professional reference vector data for Flemish features	Required foundation
2	OSM	Fallback and supplementary vector context	Required foundation
3	User-uploaded raster/vector data	Portfolio demo data and controlled test input	Required foundation
4	Sentinel-2 L2A	Remote-sensing indices	V2
5	DHMV / DEM / DSM	Height analysis	V3
6	LAS / LAZ LiDAR	Point-cloud workbench	V4

GRB strategy

GRB is treated as the most important V1 external reference dataset.

Why

• It is the Flemish large-scale reference map.
• It provides official topographic reference geometry for professional GIS workflows.
• It is ideal for QA/QC of AI-generated building detections and other extracted features.

Access strategy

Preferred V1 strategy:

1. Implement ReferenceLayerService with a provider abstraction.
2. First provider: WFS/remote fetch if service endpoint configuration is available.
3. Second provider: local GeoPackage/GeoJSON/shapefile import for offline demos.
4. Cache fetched features in PostGIS with a dataset version record.

This dual approach avoids blocking the project on one external API while still aligning with the real GRB workflow.

V1 GRB feature classes

Minimum required:

• buildings / building ground geometry
• roads or road-related reference geometry if easily available
• water features if easily available

V1 building QA only needs the building reference layer to be fully functional.

Cache policy

• Cache by project area, layer name, provider and fetch date.
• Store original provider metadata.
• Store normalized geometry in PostGIS.
• Do not mutate cached reference features during analysis.

OSM strategy

OSM is used for fast, broad, fallback context.

Access options

• Overpass API for small selected areas.
• Local test fixtures for repeatable tests.
• Later: Geofabrik extracts if performance becomes an issue.

V1 OSM layers

• buildings
• roads
• water
• green/landuse where available

OSM caveat

OSM is community-maintained and may be incomplete. UI and reports must describe it as contextual/fallback data, not official ground truth.

User-uploaded raster strategy

V1 must support controlled local datasets because public raster access and model compatibility can be difficult.

Required upload types

• GeoTIFF / TIFF where raster georeferencing is available.
• JPG/PNG for non-georeferenced demo inference, with clear warning that outputs cannot be geospatially exported unless georeferencing is supplied.

Raster metadata required

• CRS if available
• bounds if available
• transform if available
• resolution if available
• band count
• nodata value if available
• width/height
• dtype

User-uploaded vector strategy

Required formats:

• GeoJSON
• zipped shapefile
• GeoPackage later if easy

Required normalization:

• validate geometry
• determine CRS
• reproject to canonical project CRS when necessary
• store original metadata

Sentinel-2 strategy

Not required in first build, but architecture must prepare for it.

Preferred discovery

Use STAC-style catalogue access for Sentinel-2 L2A scenes once implemented.

V2 indices

• NDVI = (NIR - Red) / (NIR + Red)
• NDWI = (Green - NIR) / (Green + NIR)
• NDBI = (SWIR - NIR) / (SWIR + NIR)

Cloud handling

V2 should include cloud percentage filtering and warn when cloud contamination may affect outputs.

DHMV / height strategy

Not V1. Store architecture placeholders only.

Future uses:

• elevation statistics
• slope
• low-point detection
• water sensitivity proxy
• building height if DSM and building footprints are available

Canonical CRS strategy

• Store project areas and vector outputs in PostGIS with SRID metadata.
• Use EPSG:4326 for API interchange when practical.
• Use a metric projected CRS for area/length calculations. For Flanders, prefer Belgian Lambert 72 / EPSG:31370 or another documented metric CRS.
• Never calculate area or distance on raw EPSG:4326 geometries.

Dataset versioning

Every dataset must have a version record:

• source name
• provider
• fetch/import date
• original path or endpoint
• CRS
• bounds
• checksum where applicable
• processing status

V1 acceptance

V1 dataset strategy is complete when:

• a user can upload one raster and one vector dataset;
• metadata is extracted and persisted;
• an area can request/cache a reference building layer;
• detection outputs can be compared with that reference layer;
• exports include source metadata.

Temporal snapshots and evolution

• A historical observation is one persisted dataset snapshot. Existing datasets are not overwritten and features are not hidden in job JSON.
• Related observations share a stable temporal_series_key; observed_at records when the source describes reality, while imported_at records ingestion time.
• The latest-state map uses the latest available observation but must not call an old source edition current reality.
• Evolution metrics use the same selection geometry, aggregation and units for both snapshots.
• Partial-sector population is an area-weighted estimate. GeoIntel must not imply address-level distribution when only sector totals are available.
• Historical cartographic classes can change meaning between editions. Source classes and processing notes remain provenance, and object changes require explicit stable source identity.
• Canonical vector_features.geometry is 2D EPSG:4326. Valid source Z values are removed only from the query index, while the original upload remains unchanged and z_dimension_feature_count records that normalization.
• Dataset, version and vector-feature rows are committed atomically. Failed geometry indexing rolls back all rows and removes the newly stored upload.