248 lines
10 KiB
Markdown
248 lines
10 KiB
Markdown
# Dataset Strategy
|
|
|
|
## Goal
|
|
Define exactly how GeoIntel obtains, stores, caches and uses source data. This prevents implementation drift during autonomous Codex passes.
|
|
|
|
## Source priority
|
|
| Priority | Source | Role | V1 status |
|
|
|---:|---|---|---|
|
|
| 1 | GRB / Basiskaart Vlaanderen | Professional reference vector data for Flemish features | Required foundation |
|
|
| 2 | OSM | Fallback and supplementary vector context | Required foundation |
|
|
| 3 | User-uploaded raster/vector data | Portfolio demo data and controlled test input | Required foundation |
|
|
| 4 | Sentinel-2 L2A | Remote-sensing indices | V2 |
|
|
| 5 | DHMV / DEM / DSM | Height analysis | V3 |
|
|
| 6 | LAS / LAZ LiDAR | Point-cloud workbench | V4 |
|
|
|
|
## GRB strategy
|
|
GRB is treated as the most important V1 external reference dataset.
|
|
|
|
### Why
|
|
- It is the Flemish large-scale reference map.
|
|
- It provides official topographic reference geometry for professional GIS workflows.
|
|
- It is ideal for QA/QC of AI-generated building detections and other extracted features.
|
|
|
|
### Access strategy
|
|
Preferred V1 strategy:
|
|
1. Implement `ReferenceLayerService` with a provider abstraction.
|
|
2. First provider: WFS/remote fetch if service endpoint configuration is available.
|
|
3. Second provider: local GeoPackage/GeoJSON/shapefile import for offline demos.
|
|
4. Cache fetched features in PostGIS with a dataset version record.
|
|
|
|
This dual approach avoids blocking the project on one external API while still aligning with the real GRB workflow.
|
|
|
|
### V1 GRB feature classes
|
|
Minimum required:
|
|
- buildings / building ground geometry
|
|
- roads or road-related reference geometry if easily available
|
|
- water features if easily available
|
|
|
|
V1 building QA only needs the building reference layer to be fully functional.
|
|
|
|
### Cache policy
|
|
- Cache by project area, layer name, provider and fetch date.
|
|
- Store original provider metadata.
|
|
- Store normalized geometry in PostGIS.
|
|
- Do not mutate cached reference features during analysis.
|
|
|
|
## OSM strategy
|
|
OSM is used for fast, broad, fallback context.
|
|
|
|
### Access options
|
|
- Overpass API for small selected areas.
|
|
- Local test fixtures for repeatable tests.
|
|
- Later: Geofabrik extracts if performance becomes an issue.
|
|
|
|
### V1 OSM layers
|
|
- buildings
|
|
- roads
|
|
- water
|
|
- green/landuse where available
|
|
|
|
### OSM caveat
|
|
OSM is community-maintained and may be incomplete. UI and reports must describe it as contextual/fallback data, not official ground truth.
|
|
|
|
## Official modern land-use strategy
|
|
|
|
The Departement Omgeving version 3 land-use maps are the authoritative modern
|
|
source series for categorical land use. GeoIntel retrieves only explicit
|
|
boundary subsets from MercatorNet WCS; no whole-Flanders raster is fetched at
|
|
startup or during an interactive map query.
|
|
|
|
- Source years: 2013, 2016, 2019, 2022 and 2025.
|
|
- Native representation: one integer category per 10 m cell in EPSG:31370.
|
|
- Preserved evidence: raw GeoTIFF, WCS request, catalogue URL, checksum, class
|
|
histogram and processing manifest.
|
|
- Operational representation: source cells polygonized in EPSG:31370, clipped
|
|
to the requested boundary and normalized to EPSG:4326 `vector_features`.
|
|
- Boundary behavior: retain every source cell touching the boundary, then
|
|
perform the exact metric geometry intersection; exclude and report NoData.
|
|
- Selection metric: exact polygon intersection area within the source's 10 m
|
|
representation, reported in hectares with a non-cadastral warning.
|
|
- Identity: raster polygons are not stable source objects; temporal object
|
|
lineage is unavailable.
|
|
|
|
Modern land-use series must remain separate from the 1778, 1873 and 1969
|
|
historical cartographic series. Comparisons are valid within each persisted
|
|
series and must retain the source methodology warning. Extension from Mol to
|
|
Kempen reuses the same operator with an explicit approved boundary, project,
|
|
area and scope key; an ambiguous regional label is never converted into an
|
|
invented boundary.
|
|
|
|
## Operational Kempen scope
|
|
|
|
The canonical regional workspace uses the official 28-municipality Vlaamse
|
|
`Vervoerregio Kempen` policy boundary. `scripts/geographic_scopes.py` is the
|
|
operator source of truth for its members and current NIS codes. The resulting
|
|
regional geometry is a union of current authoritative VRBG municipality
|
|
boundaries.
|
|
|
|
- Project: `Kempen Regional Workbench`.
|
|
- Regional Area: `Vervoerregio Kempen - officiële operationele grens`.
|
|
- Member Areas: one current official boundary for every municipality.
|
|
- Scope key: `kempen-transport-region`.
|
|
- Provenance: authority URL, VRBG request URL, member codes, fetch timestamp,
|
|
artifact checksums and the policy-boundary limitation.
|
|
- Scale policy: thematic providers are partitioned and validated separately;
|
|
no interactive request or application startup may download the entire
|
|
region or silently substitute missing data.
|
|
|
|
### Regional building partition policy
|
|
|
|
The current GRB building snapshot is fetched in one resumable partition per
|
|
registered municipality. Partitions are source/provenance artifacts, not 28
|
|
independent user-facing layers. A building intersecting more than one member
|
|
boundary is owned by the member with the largest intersection area; the lower
|
|
NIS code resolves exact ties. The retained geometry is clipped to the regional
|
|
union, preserving complete in-scope coverage while guaranteeing one source ID
|
|
per regional dataset.
|
|
|
|
After all partitions pass validation, one combined immutable GeoJSON artifact
|
|
is copied into managed storage and indexed in bounded batches through the
|
|
normal DatasetService/VectorFeatureService ownership boundary. The resulting
|
|
reference dataset uses `source_name=grb`, `reference_layer_name=buildings`,
|
|
`coverage_scope=kempen-transport-region` and an exact `feature_count`
|
|
selection aggregation. Existing viewport and bbox selection APIs remain the
|
|
only interactive delivery path.
|
|
|
|
### Regional GRB context partition policy
|
|
|
|
Roads, water and parcels reuse the building operator's 28 resumable source
|
|
partitions and one-dataset-per-theme persistence model. Polygon collections
|
|
use maximum intersection area for cross-boundary ownership; line collections
|
|
use maximum intersection length. Exact ties use the lower NIS code. Source
|
|
identities are collection-qualified because the water theme combines `WTZ`,
|
|
`WLAS` and `WGR`.
|
|
|
|
The user-facing datasets use `source_name=grb`, `dataset_role=reference`,
|
|
`coverage_scope=kempen-transport-region` and respectively
|
|
`reference_layer_name=roads`, `water` or `parcels`. They are current source
|
|
snapshots with immutable observation dates, not temporal series inferred from
|
|
successive operator runs. Feature counts retain source-object semantics and
|
|
must not be relabelled as road length, water area or legal parcel counts.
|
|
|
|
Only `scripts/provision_regional_grb_context.py` may fetch these bounded
|
|
artifacts. Application startup, map requests and the public provider registry
|
|
remain fetch-free. All combined artifacts enter PostGIS through the existing
|
|
DatasetService/VectorFeatureService boundary and existing bbox/Area selection
|
|
contracts.
|
|
|
|
## User-uploaded raster strategy
|
|
V1 must support controlled local datasets because public raster access and model compatibility can be difficult.
|
|
|
|
### Required upload types
|
|
- GeoTIFF / TIFF where raster georeferencing is available.
|
|
- JPG/PNG for non-georeferenced demo inference, with clear warning that outputs cannot be geospatially exported unless georeferencing is supplied.
|
|
|
|
### Raster metadata required
|
|
- CRS if available
|
|
- bounds if available
|
|
- transform if available
|
|
- resolution if available
|
|
- band count
|
|
- nodata value if available
|
|
- width/height
|
|
- dtype
|
|
|
|
## User-uploaded vector strategy
|
|
Required formats:
|
|
- GeoJSON
|
|
- zipped shapefile
|
|
- GeoPackage later if easy
|
|
|
|
Required normalization:
|
|
- validate geometry
|
|
- determine CRS
|
|
- reproject to canonical project CRS when necessary
|
|
- store original metadata
|
|
|
|
## Sentinel-2 strategy
|
|
Not required in first build, but architecture must prepare for it.
|
|
|
|
### Preferred discovery
|
|
Use STAC-style catalogue access for Sentinel-2 L2A scenes once implemented.
|
|
|
|
### V2 indices
|
|
- NDVI = `(NIR - Red) / (NIR + Red)`
|
|
- NDWI = `(Green - NIR) / (Green + NIR)`
|
|
- NDBI = `(SWIR - NIR) / (SWIR + NIR)`
|
|
|
|
### Cloud handling
|
|
V2 should include cloud percentage filtering and warn when cloud contamination may affect outputs.
|
|
|
|
## DHMV / height strategy
|
|
Not V1. Store architecture placeholders only.
|
|
|
|
Future uses:
|
|
- elevation statistics
|
|
- slope
|
|
- low-point detection
|
|
- water sensitivity proxy
|
|
- building height if DSM and building footprints are available
|
|
|
|
## Canonical CRS strategy
|
|
- Store project areas and vector outputs in PostGIS with SRID metadata.
|
|
- Use EPSG:4326 for API interchange when practical.
|
|
- Use a metric projected CRS for area/length calculations. For Flanders, prefer Belgian Lambert 72 / EPSG:31370 or another documented metric CRS.
|
|
- Never calculate area or distance on raw EPSG:4326 geometries.
|
|
|
|
## Dataset versioning
|
|
Every dataset must have a version record:
|
|
- source name
|
|
- provider
|
|
- fetch/import date
|
|
- original path or endpoint
|
|
- CRS
|
|
- bounds
|
|
- checksum where applicable
|
|
- processing status
|
|
|
|
## V1 acceptance
|
|
V1 dataset strategy is complete when:
|
|
- a user can upload one raster and one vector dataset;
|
|
- metadata is extracted and persisted;
|
|
- an area can request/cache a reference building layer;
|
|
- detection outputs can be compared with that reference layer;
|
|
- exports include source metadata.
|
|
|
|
## Temporal snapshots and evolution
|
|
|
|
- A historical observation is one persisted dataset snapshot. Existing
|
|
datasets are not overwritten and features are not hidden in job JSON.
|
|
- Related observations share a stable `temporal_series_key`; `observed_at`
|
|
records when the source describes reality, while `imported_at` records
|
|
ingestion time.
|
|
- The latest-state map uses the latest available observation but must not call
|
|
an old source edition current reality.
|
|
- Evolution metrics use the same selection geometry, aggregation and units for
|
|
both snapshots.
|
|
- Partial-sector population is an area-weighted estimate. GeoIntel must not
|
|
imply address-level distribution when only sector totals are available.
|
|
- Historical cartographic classes can change meaning between editions. Source
|
|
classes and processing notes remain provenance, and object changes require
|
|
explicit stable source identity.
|
|
- Canonical `vector_features.geometry` is 2D EPSG:4326. Valid source Z values
|
|
are removed only from the query index, while the original upload remains
|
|
unchanged and `z_dimension_feature_count` records that normalization.
|
|
- Dataset, version and vector-feature rows are committed atomically. Failed
|
|
geometry indexing rolls back all rows and removes the newly stored upload.
|