Initial GeoIntel V1 foundation

2026-06-16 23:36:32 +02:00
commit 6ea3586a3e
605 changed files with 45284 additions and 0 deletions
@@ -0,0 +1,140 @@
+# Dataset Strategy
+
+## Goal
+Define exactly how GeoIntel obtains, stores, caches and uses source data. This prevents implementation drift during autonomous Codex passes.
+
+## Source priority
+| Priority | Source | Role | V1 status |
+|---:|---|---|---|
+| 1 | GRB / Basiskaart Vlaanderen | Professional reference vector data for Flemish features | Required foundation |
+| 2 | OSM | Fallback and supplementary vector context | Required foundation |
+| 3 | User-uploaded raster/vector data | Portfolio demo data and controlled test input | Required foundation |
+| 4 | Sentinel-2 L2A | Remote-sensing indices | V2 |
+| 5 | DHMV / DEM / DSM | Height analysis | V3 |
+| 6 | LAS / LAZ LiDAR | Point-cloud workbench | V4 |
+
+## GRB strategy
+GRB is treated as the most important V1 external reference dataset.
+
+### Why
+- It is the Flemish large-scale reference map.
+- It provides official topographic reference geometry for professional GIS workflows.
+- It is ideal for QA/QC of AI-generated building detections and other extracted features.
+
+### Access strategy
+Preferred V1 strategy:
+1. Implement `ReferenceLayerService` with a provider abstraction.
+2. First provider: WFS/remote fetch if service endpoint configuration is available.
+3. Second provider: local GeoPackage/GeoJSON/shapefile import for offline demos.
+4. Cache fetched features in PostGIS with a dataset version record.
+
+This dual approach avoids blocking the project on one external API while still aligning with the real GRB workflow.
+
+### V1 GRB feature classes
+Minimum required:
+- buildings / building ground geometry
+- roads or road-related reference geometry if easily available
+- water features if easily available
+
+V1 building QA only needs the building reference layer to be fully functional.
+
+### Cache policy
+- Cache by project area, layer name, provider and fetch date.
+- Store original provider metadata.
+- Store normalized geometry in PostGIS.
+- Do not mutate cached reference features during analysis.
+
+## OSM strategy
+OSM is used for fast, broad, fallback context.
+
+### Access options
+- Overpass API for small selected areas.
+- Local test fixtures for repeatable tests.
+- Later: Geofabrik extracts if performance becomes an issue.
+
+### V1 OSM layers
+- buildings
+- roads
+- water
+- green/landuse where available
+
+### OSM caveat
+OSM is community-maintained and may be incomplete. UI and reports must describe it as contextual/fallback data, not official ground truth.
+
+## User-uploaded raster strategy
+V1 must support controlled local datasets because public raster access and model compatibility can be difficult.
+
+### Required upload types
+- GeoTIFF / TIFF where raster georeferencing is available.
+- JPG/PNG for non-georeferenced demo inference, with clear warning that outputs cannot be geospatially exported unless georeferencing is supplied.
+
+### Raster metadata required
+- CRS if available
+- bounds if available
+- transform if available
+- resolution if available
+- band count
+- nodata value if available
+- width/height
+- dtype
+
+## User-uploaded vector strategy
+Required formats:
+- GeoJSON
+- zipped shapefile
+- GeoPackage later if easy
+
+Required normalization:
+- validate geometry
+- determine CRS
+- reproject to canonical project CRS when necessary
+- store original metadata
+
+## Sentinel-2 strategy
+Not required in first build, but architecture must prepare for it.
+
+### Preferred discovery
+Use STAC-style catalogue access for Sentinel-2 L2A scenes once implemented.
+
+### V2 indices
+- NDVI = `(NIR - Red) / (NIR + Red)`
+- NDWI = `(Green - NIR) / (Green + NIR)`
+- NDBI = `(SWIR - NIR) / (SWIR + NIR)`
+
+### Cloud handling
+V2 should include cloud percentage filtering and warn when cloud contamination may affect outputs.
+
+## DHMV / height strategy
+Not V1. Store architecture placeholders only.
+
+Future uses:
+- elevation statistics
+- slope
+- low-point detection
+- water sensitivity proxy
+- building height if DSM and building footprints are available
+
+## Canonical CRS strategy
+- Store project areas and vector outputs in PostGIS with SRID metadata.
+- Use EPSG:4326 for API interchange when practical.
+- Use a metric projected CRS for area/length calculations. For Flanders, prefer Belgian Lambert 72 / EPSG:31370 or another documented metric CRS.
+- Never calculate area or distance on raw EPSG:4326 geometries.
+
+## Dataset versioning
+Every dataset must have a version record:
+- source name
+- provider
+- fetch/import date
+- original path or endpoint
+- CRS
+- bounds
+- checksum where applicable
+- processing status
+
+## V1 acceptance
+V1 dataset strategy is complete when:
+- a user can upload one raster and one vector dataset;
+- metadata is extracted and persisted;
+- an area can request/cache a reference building layer;
+- detection outputs can be compared with that reference layer;
+- exports include source metadata.