141 lines
4.7 KiB
Markdown
141 lines
4.7 KiB
Markdown
# Dataset Strategy
|
|
|
|
## Goal
|
|
Define exactly how GeoIntel obtains, stores, caches and uses source data. This prevents implementation drift during autonomous Codex passes.
|
|
|
|
## Source priority
|
|
| Priority | Source | Role | V1 status |
|
|
|---:|---|---|---|
|
|
| 1 | GRB / Basiskaart Vlaanderen | Professional reference vector data for Flemish features | Required foundation |
|
|
| 2 | OSM | Fallback and supplementary vector context | Required foundation |
|
|
| 3 | User-uploaded raster/vector data | Portfolio demo data and controlled test input | Required foundation |
|
|
| 4 | Sentinel-2 L2A | Remote-sensing indices | V2 |
|
|
| 5 | DHMV / DEM / DSM | Height analysis | V3 |
|
|
| 6 | LAS / LAZ LiDAR | Point-cloud workbench | V4 |
|
|
|
|
## GRB strategy
|
|
GRB is treated as the most important V1 external reference dataset.
|
|
|
|
### Why
|
|
- It is the Flemish large-scale reference map.
|
|
- It provides official topographic reference geometry for professional GIS workflows.
|
|
- It is ideal for QA/QC of AI-generated building detections and other extracted features.
|
|
|
|
### Access strategy
|
|
Preferred V1 strategy:
|
|
1. Implement `ReferenceLayerService` with a provider abstraction.
|
|
2. First provider: WFS/remote fetch if service endpoint configuration is available.
|
|
3. Second provider: local GeoPackage/GeoJSON/shapefile import for offline demos.
|
|
4. Cache fetched features in PostGIS with a dataset version record.
|
|
|
|
This dual approach avoids blocking the project on one external API while still aligning with the real GRB workflow.
|
|
|
|
### V1 GRB feature classes
|
|
Minimum required:
|
|
- buildings / building ground geometry
|
|
- roads or road-related reference geometry if easily available
|
|
- water features if easily available
|
|
|
|
V1 building QA only needs the building reference layer to be fully functional.
|
|
|
|
### Cache policy
|
|
- Cache by project area, layer name, provider and fetch date.
|
|
- Store original provider metadata.
|
|
- Store normalized geometry in PostGIS.
|
|
- Do not mutate cached reference features during analysis.
|
|
|
|
## OSM strategy
|
|
OSM is used for fast, broad, fallback context.
|
|
|
|
### Access options
|
|
- Overpass API for small selected areas.
|
|
- Local test fixtures for repeatable tests.
|
|
- Later: Geofabrik extracts if performance becomes an issue.
|
|
|
|
### V1 OSM layers
|
|
- buildings
|
|
- roads
|
|
- water
|
|
- green/landuse where available
|
|
|
|
### OSM caveat
|
|
OSM is community-maintained and may be incomplete. UI and reports must describe it as contextual/fallback data, not official ground truth.
|
|
|
|
## User-uploaded raster strategy
|
|
V1 must support controlled local datasets because public raster access and model compatibility can be difficult.
|
|
|
|
### Required upload types
|
|
- GeoTIFF / TIFF where raster georeferencing is available.
|
|
- JPG/PNG for non-georeferenced demo inference, with clear warning that outputs cannot be geospatially exported unless georeferencing is supplied.
|
|
|
|
### Raster metadata required
|
|
- CRS if available
|
|
- bounds if available
|
|
- transform if available
|
|
- resolution if available
|
|
- band count
|
|
- nodata value if available
|
|
- width/height
|
|
- dtype
|
|
|
|
## User-uploaded vector strategy
|
|
Required formats:
|
|
- GeoJSON
|
|
- zipped shapefile
|
|
- GeoPackage later if easy
|
|
|
|
Required normalization:
|
|
- validate geometry
|
|
- determine CRS
|
|
- reproject to canonical project CRS when necessary
|
|
- store original metadata
|
|
|
|
## Sentinel-2 strategy
|
|
Not required in first build, but architecture must prepare for it.
|
|
|
|
### Preferred discovery
|
|
Use STAC-style catalogue access for Sentinel-2 L2A scenes once implemented.
|
|
|
|
### V2 indices
|
|
- NDVI = `(NIR - Red) / (NIR + Red)`
|
|
- NDWI = `(Green - NIR) / (Green + NIR)`
|
|
- NDBI = `(SWIR - NIR) / (SWIR + NIR)`
|
|
|
|
### Cloud handling
|
|
V2 should include cloud percentage filtering and warn when cloud contamination may affect outputs.
|
|
|
|
## DHMV / height strategy
|
|
Not V1. Store architecture placeholders only.
|
|
|
|
Future uses:
|
|
- elevation statistics
|
|
- slope
|
|
- low-point detection
|
|
- water sensitivity proxy
|
|
- building height if DSM and building footprints are available
|
|
|
|
## Canonical CRS strategy
|
|
- Store project areas and vector outputs in PostGIS with SRID metadata.
|
|
- Use EPSG:4326 for API interchange when practical.
|
|
- Use a metric projected CRS for area/length calculations. For Flanders, prefer Belgian Lambert 72 / EPSG:31370 or another documented metric CRS.
|
|
- Never calculate area or distance on raw EPSG:4326 geometries.
|
|
|
|
## Dataset versioning
|
|
Every dataset must have a version record:
|
|
- source name
|
|
- provider
|
|
- fetch/import date
|
|
- original path or endpoint
|
|
- CRS
|
|
- bounds
|
|
- checksum where applicable
|
|
- processing status
|
|
|
|
## V1 acceptance
|
|
V1 dataset strategy is complete when:
|
|
- a user can upload one raster and one vector dataset;
|
|
- metadata is extracted and persisted;
|
|
- an area can request/cache a reference building layer;
|
|
- detection outputs can be compared with that reference layer;
|
|
- exports include source metadata.
|