Initial public release

docs/DATA_CATALOG.md

@@ -0,0 +1,227 @@
+# GeoIntel Kempen — Data Catalog v1.0
+
+This catalog is the implementation reference for every dataset that GeoIntel may ingest, cache, analyse, validate against, or export. Codex must treat this file as source-of-truth when implementing data ingestion and analysis modules.
+
+## Data strategy
+
+GeoIntel is built around a professional reference-data-first strategy:
+
+1. Use official Flemish geodata where available.
+2. Use OpenStreetMap as a supplemental and fallback source.
+3. Use raster/satellite/aerial imagery for AI and remote-sensing pipelines.
+4. Cache all expensive or external requests in PostGIS and/or filesystem storage.
+5. Store provenance metadata for every derived result.
+
+## Priority sources
+
+| Priority | Source | Role | Data type | V1 status |
+|---|---|---|---|---|
+| P0 | GRB / Basiskaart Vlaanderen | official reference geometry | vector | required as primary reference target |
+| P0 | User uploaded GeoTIFF / imagery | model input | raster | required |
+| P0 | User uploaded GeoJSON/Shapefile/GPKG | vector input/reference | vector | required |
+| P1 | OpenStreetMap | fallback and additional context | vector | required in V1 if GRB connector is not ready |
+| P1 | Gebouwenregister | building identifiers and metadata | vector/API | prepare architecture |
+| P2 | Sentinel-2 | NDVI/NDWI/NDBI and temporal analysis | raster | roadmap |
+| P2 | DHMV / height products | DEM/DSM/slope/height analysis | raster/point cloud | roadmap |
+| P3 | LAS/LAZ point clouds | LiDAR workbench | point cloud | roadmap |
+
+## GRB — Basiskaart Vlaanderen
+
+### Purpose
+GRB is the main official reference layer for GeoIntel Kempen. It is used for QA/QC, validation, feature comparison, and map context. It contains accurately measured reference objects such as buildings, parcels, roads and road inrichting, watercourses, railway beds and road networks. It is a cost-free authentic source managed by Digitaal Vlaanderen.
+
+### Implementation role
+GRB is not just a background map. It must become a validation and reference backbone:
+
+- Compare AI building detections with official building footprints.
+- Identify false positives and false negatives.
+- Create QA/QC dashboards.
+- Provide high-quality vector context for map overlays.
+- Support change-detection workflows where official snapshots are available.
+
+### Required layer groups
+The exact service layer names must be discovered during implementation through the GRB WFS/WMS capabilities endpoint or downloaded package metadata. The application-level canonical layer groups are:
+
+| Canonical group | Expected geometry | Usage |
+|---|---|---|
+| `grb_buildings` | polygon | building reference, QA/QC, footprint analysis |
+| `grb_roads` | line/polygon | road/infrastructure context, pressure metrics |
+| `grb_water` | line/polygon | water context, water proximity, hydro overlays |
+| `grb_railways` | line/polygon | infrastructure barrier/context |
+| `grb_parcels` | polygon | optional parcel context, not required for V1 |
+
+### Required normalized fields
+For every imported GRB feature, normalize at least:
+
+```yaml
+id: internal UUID
+source: "GRB"
+source_layer: original layer name
+source_feature_id: original feature identifier when available
+canonical_group: grb_buildings | grb_roads | grb_water | grb_railways | grb_parcels
+geometry: PostGIS geometry
+geometry_type: Polygon | MultiPolygon | LineString | MultiLineString
+crs_original: original CRS
+crs_storage: EPSG:31370 or EPSG:4326 depending DB policy
+attributes_json: raw attributes
+fetched_at: timestamp
+source_updated_at: timestamp when available
+bbox: calculated bounds
+area_m2: calculated for polygon features
+length_m: calculated for linear features
+```
+
+### Access strategy
+V1 may implement either:
+
+1. WFS area-of-interest fetch, preferred for targeted analysis.
+2. Downloaded package import for cached offline analysis.
+3. OSM fallback when GRB access is not implemented yet.
+
+Codex may build an abstraction `ReferenceDataProvider` so GRB and OSM can both satisfy the same downstream analysis contracts.
+
+## User uploaded raster imagery
+
+### Supported formats
+
+- GeoTIFF `.tif`, `.tiff`
+- Cloud Optimized GeoTIFF if possible
+- JPEG/PNG only if accompanied by georeferencing metadata or used as non-geospatial demo input
+
+### Required metadata
+
+```yaml
+dataset_id
+file_path
+file_size_bytes
+raster_driver
+width
+height
+band_count
+crs
+transform
+bounds
+resolution_x
+resolution_y
+nodata_values
+dtype_per_band
+statistics_per_band
+color_interpretation
+is_georeferenced
+created_at
+```
+
+### Required uses
+
+- Map overlay preview.
+- Raster metadata inspection.
+- Clipping to area of interest.
+- Tiling for object detection and segmentation.
+- Remote sensing index calculations when bands support it.
+
+## User uploaded vector data
+
+### Supported formats
+
+- GeoJSON
+- Shapefile ZIP
+- GeoPackage `.gpkg`
+- KML/KMZ later
+
+### Required metadata
+
+```yaml
+dataset_id
+source_filename
+format
+layer_names
+feature_count
+geometry_types
+crs
+bounds
+attributes_schema
+created_at
+```
+
+### Required uses
+
+- Reference data.
+- Manual annotations.
+- User-supplied areas of interest.
+- Comparison layers for QA/QC.
+
+## OpenStreetMap
+
+### Purpose
+OSM is used for fast open-data enrichment and fallback when official sources are not yet implemented.
+
+### Canonical groups
+
+- `osm_buildings`
+- `osm_roads`
+- `osm_water`
+- `osm_green`
+- `osm_landuse`
+- `osm_poi`
+
+### Required fields
+
+```yaml
+osm_id
+tags_json
+canonical_group
+geometry
+area_m2
+length_m
+fetched_at
+```
+
+## Sentinel-2
+
+### Purpose
+Sentinel-2 supports remote sensing indices and temporal change analysis.
+
+### Required indices later
+
+- NDVI = `(NIR - Red) / (NIR + Red)`
+- NDWI = `(Green - NIR) / (Green + NIR)`
+- NDBI = `(SWIR - NIR) / (SWIR + NIR)`
+
+### V1 rule
+Prepare architecture, but do not block V1 on a full Sentinel downloader.
+
+## DHMV / DEM / DSM
+
+### Purpose
+Height intelligence: slope, low points, terrain context, DSM minus DEM for object height approximations.
+
+### Required outputs later
+
+- `min_elevation_m`
+- `max_elevation_m`
+- `mean_elevation_m`
+- `slope_mean_deg`
+- `low_point_polygons`
+- `height_profile_samples`
+
+## Data provenance requirements
+
+Every derived dataset must keep:
+
+```yaml
+source_dataset_ids
+processing_pipeline
+parameters_json
+software_versions
+created_at
+created_by_job_id
+crs
+quality_flags
+```
+
+## V1 non-negotiables
+
+- Do not hardcode temporary demo assumptions into data models.
+- Every dataset must have metadata.
+- Every derived output must reference its source dataset and analysis run.
+- AI outputs must be stored as geospatial outputs, not only image overlays.