# Dataset Strategy

## Goal
Define exactly how GeoIntel obtains, stores, caches and uses source data. This prevents implementation drift during autonomous Codex passes.

## Source priority
| Priority | Source | Role | V1 status |
|---:|---|---|---|
| 1 | GRB / Basiskaart Vlaanderen | Professional reference vector data for Flemish features | Required foundation |
| 2 | OSM | Fallback and supplementary vector context | Required foundation |
| 3 | User-uploaded raster/vector data | Portfolio demo data and controlled test input | Required foundation |
| 4 | Sentinel-2 L2A | Remote-sensing indices | V2 |
| 5 | DHMV / DEM / DSM | Height analysis | V3 |
| 6 | LAS / LAZ LiDAR | Point-cloud workbench | V4 |

## GRB strategy
GRB is treated as the most important V1 external reference dataset.

### Why
- It is the Flemish large-scale reference map.
- It provides official topographic reference geometry for professional GIS workflows.
- It is ideal for QA/QC of AI-generated building detections and other extracted features.

### Access strategy
Preferred V1 strategy:
1. Implement `ReferenceLayerService` with a provider abstraction.
2. First provider: WFS/remote fetch if service endpoint configuration is available.
3. Second provider: local GeoPackage/GeoJSON/shapefile import for offline demos.
4. Cache fetched features in PostGIS with a dataset version record.

This dual approach avoids blocking the project on one external API while still aligning with the real GRB workflow.

### V1 GRB feature classes
Minimum required:
- buildings / building ground geometry
- roads or road-related reference geometry if easily available
- water features if easily available

V1 building QA only needs the building reference layer to be fully functional.

### Cache policy
- Cache by project area, layer name, provider and fetch date.
- Store original provider metadata.
- Store normalized geometry in PostGIS.
- Do not mutate cached reference features during analysis.

## OSM strategy
OSM is used for fast, broad, fallback context.

### Access options
- Overpass API for small selected areas.
- Local test fixtures for repeatable tests.
- Later: Geofabrik extracts if performance becomes an issue.

### V1 OSM layers
- buildings
- roads
- water
- green/landuse where available

### OSM caveat
OSM is community-maintained and may be incomplete. UI and reports must describe it as contextual/fallback data, not official ground truth.

## User-uploaded raster strategy
V1 must support controlled local datasets because public raster access and model compatibility can be difficult.

### Required upload types
- GeoTIFF / TIFF where raster georeferencing is available.
- JPG/PNG for non-georeferenced demo inference, with clear warning that outputs cannot be geospatially exported unless georeferencing is supplied.

### Raster metadata required
- CRS if available
- bounds if available
- transform if available
- resolution if available
- band count
- nodata value if available
- width/height
- dtype

## User-uploaded vector strategy
Required formats:
- GeoJSON
- zipped shapefile
- GeoPackage later if easy

Required normalization:
- validate geometry
- determine CRS
- reproject to canonical project CRS when necessary
- store original metadata

## Sentinel-2 strategy
Not required in first build, but architecture must prepare for it.

### Preferred discovery
Use STAC-style catalogue access for Sentinel-2 L2A scenes once implemented.

### V2 indices
- NDVI = `(NIR - Red) / (NIR + Red)`
- NDWI = `(Green - NIR) / (Green + NIR)`
- NDBI = `(SWIR - NIR) / (SWIR + NIR)`

### Cloud handling
V2 should include cloud percentage filtering and warn when cloud contamination may affect outputs.

## DHMV / height strategy
Not V1. Store architecture placeholders only.

Future uses:
- elevation statistics
- slope
- low-point detection
- water sensitivity proxy
- building height if DSM and building footprints are available

## Canonical CRS strategy
- Store project areas and vector outputs in PostGIS with SRID metadata.
- Use EPSG:4326 for API interchange when practical.
- Use a metric projected CRS for area/length calculations. For Flanders, prefer Belgian Lambert 72 / EPSG:31370 or another documented metric CRS.
- Never calculate area or distance on raw EPSG:4326 geometries.

## Dataset versioning
Every dataset must have a version record:
- source name
- provider
- fetch/import date
- original path or endpoint
- CRS
- bounds
- checksum where applicable
- processing status

## V1 acceptance
V1 dataset strategy is complete when:
- a user can upload one raster and one vector dataset;
- metadata is extracted and persisted;
- an area can request/cache a reference building layer;
- detection outputs can be compared with that reference layer;
- exports include source metadata.