349 lines
12 KiB
Markdown
349 lines
12 KiB
Markdown
# GeoIntel Kempen — Storage Architecture v1.0
|
|
|
|
GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage.
|
|
|
|
## Principles
|
|
|
|
- Database stores metadata, relationships and vector geometries.
|
|
- Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts.
|
|
- Every stored file must have a dataset/export/model record in the database.
|
|
- Never store large raster binary data directly in regular application tables in V1.
|
|
|
|
## Root storage layout
|
|
|
|
```text
|
|
storage/
|
|
uploads/
|
|
{project_id}/
|
|
rasters/
|
|
vectors/
|
|
lidar/
|
|
rasters/
|
|
derived/
|
|
{project_id}/{dataset_id}/
|
|
tiles/
|
|
{project_id}/{dataset_id}/{tile_set_id}/
|
|
masks/
|
|
{project_id}/{analysis_run_id}/
|
|
previews/
|
|
{project_id}/{dataset_id}/
|
|
exports/
|
|
{project_id}/
|
|
geojson/
|
|
csv/
|
|
reports/
|
|
coco/
|
|
yolo/
|
|
models/
|
|
detection/
|
|
segmentation/
|
|
training-runs/
|
|
cache/
|
|
grb/
|
|
osm/
|
|
sentinel/
|
|
dhmv/
|
|
```
|
|
|
|
## Upload policy
|
|
|
|
When a file is uploaded:
|
|
|
|
1. Save original file unchanged.
|
|
2. Compute checksum.
|
|
3. Extract metadata.
|
|
4. Create dataset record.
|
|
5. Create preview if applicable.
|
|
|
|
Required file metadata:
|
|
|
|
```yaml
|
|
path
|
|
original_filename
|
|
mime_type
|
|
size_bytes
|
|
checksum_sha256
|
|
created_at
|
|
storage_backend
|
|
```
|
|
|
|
## Derived data policy
|
|
|
|
Derived files must record:
|
|
|
|
- source dataset id(s)
|
|
- analysis run id or processing job id
|
|
- processing parameters
|
|
- software component version
|
|
- created_at
|
|
|
|
## Segmentation mask artifacts
|
|
|
|
Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records.
|
|
|
|
Default mask path convention:
|
|
|
|
```text
|
|
storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png
|
|
```
|
|
|
|
Optional run manifest convention:
|
|
|
|
```text
|
|
storage/masks/{project_id}/{analysis_run_id}/manifest.json
|
|
```
|
|
|
|
Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted `segmentations.geometry` rather than mask files.
|
|
|
|
## Cleanup policy
|
|
|
|
Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.
|
|
|
|
## Official temporal source artifacts
|
|
|
|
Statbel population staging stores immutable official source ZIPs below the
|
|
selected operator output root:
|
|
|
|
```text
|
|
<population-output>/
|
|
raw/{year}/OPENDATA_SECTOREN_{year}[_NEW].zip
|
|
raw/{year}/sh_statbel_statistical_sectors_31370_{year}0101.geojson.zip
|
|
{scope}_statbel_population_{year}.geojson
|
|
{scope}_statbel_population_{year}.preflight.json
|
|
```
|
|
|
|
The preflight manifest records official URLs, archive/member size and SHA-256,
|
|
schema fingerprints, EPSG:31370 and situation-date evidence, bounded topology
|
|
repairs, national/scope join accounting, unlocated `ZZZZ` totals, baseline
|
|
change evidence and the final derived-snapshot checksum. The upload operator
|
|
revalidates all retained files against that manifest. Raw ZIPs and manifests
|
|
are provenance artifacts; queryable population geometry remains an ordinary
|
|
Dataset/DatasetVersion plus PostGIS `vector_features` through the canonical
|
|
persistence services. Temporary standalone preflight output does not create a
|
|
database record and may be removed explicitly after operator review.
|
|
|
|
Waterinfo raw station layers, timeseries responses and checksum manifests live
|
|
under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
|
|
evidence; queryable annual Point snapshots are normal Dataset/vector_feature
|
|
records. Bounded orthophotos are normal raster Dataset files. Their WMS URL,
|
|
product/layer, request/spatial hash, temporal validity and limitations are held
|
|
in source/provenance metadata. Browser PNG rendering is derived on request and
|
|
does not replace the stored GeoTIFF.
|
|
|
|
DHMV II DTM/DSM outputs are also normal raster Dataset files. The provider WCS
|
|
returns multipart coverage data; GeoIntel retains response and extracted
|
|
coverage SHA256 values in provenance, then stores one normalized, compressed,
|
|
Area-clipped GeoTIFF with its ordinary Dataset/DatasetVersion checksum. Source
|
|
metadata records the 1 m native product, 5 m default analysis grid, EPSG:31370,
|
|
`-9999` nodata, TAW and acquisition period 2013-2015. Colour-relief PNGs are
|
|
derived browser views and are never authoritative. No raster binary is stored
|
|
in PostgreSQL and no DHMV file is treated as water depth or volume.
|
|
|
|
`provision_regional_dhmv.py` creates no separate evidence hierarchy and no
|
|
monolithic regional raster. Each municipality/product result remains one
|
|
ordinary Area-linked raster Dataset plus DatasetVersion. The retained
|
|
provenance already includes every bounded WCS tile URL and aggregate response,
|
|
coverage and normalized-output checksum. A repeat operator run resolves the
|
|
same request identity and reuses the immutable file.
|
|
|
|
VMM flood-hazard scenario outputs follow the same raster Dataset policy. The
|
|
bounded acquisition service stores one normalized, compressed, Area-clipped
|
|
GeoTIFF per scenario/request identity. `provision_regional_flood_hazards.py`
|
|
does not create a parallel storage layout: every municipality/scenario result
|
|
is an ordinary Dataset and DatasetVersion with WCS request hashes, tile request
|
|
URLs, response/coverage/normalized checksums, EPSG:31370 bounds, scenario
|
|
metadata and explicit unsupported-volume flags. Repeat runs reuse matching
|
|
ready Datasets through the acquisition service cache.
|
|
|
|
The regional Map workspace does not create another authoritative raster or
|
|
copy pixels into PostgreSQL. Its partition-selection endpoints read only the
|
|
municipal GeoTIFF windows intersecting a bounded selection, mosaic those
|
|
windows in memory at the governed analysis resolution and return metrics plus
|
|
the complete contributing `dataset_ids`. The 12-million-cell limit applies to
|
|
the combined window. Persisted files, checksums and DatasetVersions remain the
|
|
only authoritative artifacts.
|
|
|
|
BWK/Natura 2000 evidence lives under
|
|
`storage/operator-evidence/bwk-natura2000-2025/mol/`. The `raw/` directory
|
|
contains immutable WFS pages; the adjacent manifest records their URLs,
|
|
checksums, boundary checksum, page/feature counts, clipping diagnostics and
|
|
official class totals. The final GeoJSON is the retained source artifact while
|
|
`datasets` and `vector_features` remain the canonical queryable PostGIS state.
|
|
|
|
Definitive agricultural-use parcel evidence lives under:
|
|
|
|
```text
|
|
storage/operator-evidence/agricultural-use-parcels/{scope}/{year}/
|
|
agpa_{year}_*_public.zip
|
|
agricultural_use_parcels_{year}_{scope}.geojson
|
|
agricultural_use_parcels_{year}_crop_codes.json
|
|
agricultural_use_parcels_{year}_{scope}.manifest.json
|
|
```
|
|
|
|
The official ZIP is immutable source evidence and is never removed by normal
|
|
cache cleanup. The extracted GeoPackage is temporary to avoid retaining a
|
|
second full source copy. The normalized GeoJSON is the canonical upload
|
|
artifact; Dataset and vector_feature rows remain the queryable PostGIS state.
|
|
The manifest binds source, crop-code list and upload artifact checksums. A
|
|
checksum conflict with an existing annual Dataset fails closed.
|
|
|
|
Buildings and Addresses Register snapshot evidence lives under:
|
|
|
|
```text
|
|
storage/operator-evidence/buildings-addresses-register/{area-key}/{observed-date}/
|
|
raw/
|
|
buildings_page_*.json
|
|
building_units_page_*.json
|
|
addresses_page_*.json
|
|
buildings_addresses_register.geojson
|
|
buildings_addresses_register.manifest.json
|
|
```
|
|
|
|
Raw pages contain the unmodified official response and are retained only as
|
|
checksummed operator evidence. They can contain address labels and must never
|
|
be served as a map/API artifact. The normalized GeoJSON deliberately contains
|
|
one polygon per register building with lifecycle state, aggregate relation
|
|
counts and classified GRB reconciliation only. It enters PostGIS exclusively
|
|
through DatasetService and ordinary `vector_features`; the operator never
|
|
writes database rows directly. The manifest binds all raw pages, the normalized
|
|
artifact, exact Area boundary and every GRB partition used for reconciliation.
|
|
The Area key is deterministic (`mol`, `geel`, and so on). Reuse also requires
|
|
the same Area label, observation date and boundary checksum, preventing one
|
|
municipality's evidence from being imported under another Area.
|
|
|
|
Offline demo export artifacts can be inspected and cleaned with:
|
|
|
|
```bash
|
|
python scripts/cleanup_demo_artifacts.py
|
|
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
|
|
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
|
|
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
|
|
```
|
|
|
|
The script is dry-run by default, targets only the explicit
|
|
`GeoIntel Demo - Building QA` project unless an exact `--project-name` is
|
|
provided, keeps the newest export artifacts per matching project and refuses to
|
|
delete files outside `STORAGE_ROOT`. It cleans `exports` records/files only; it
|
|
does not remove original uploads, vector features, QA/QC rows, projects, areas,
|
|
tiles, rasters or masks. `--max-delete` defaults to 25 and blocks oversized
|
|
apply runs until the operator increases the cap after reviewing dry-run output.
|
|
Repeat `--export-type` to restrict cleanup to selected artifact kinds.
|
|
|
|
Live runtime validation for this maintenance path is available as a dry-run
|
|
smoke:
|
|
|
|
```bash
|
|
bash scripts/verify_demo_cleanup_dry_run.sh
|
|
CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh
|
|
```
|
|
|
|
The smoke never passes `--apply`. It fails if the cleanup summary is not a
|
|
dry-run, if any export/file deletion is reported, or if the dry-run candidate
|
|
fields are missing.
|
|
|
|
## Governed cross-domain raster and soil evidence
|
|
|
|
MercatorNet thematic products use the ordinary raster Dataset and immutable
|
|
DatasetVersion paths under `STORAGE_ROOT`. `source_metadata` records product,
|
|
coverage, native resolution/unit, render range, reference year and request
|
|
bounds. `provenance_metadata` records every tiled WCS URL, transfer checksum,
|
|
normalized checksum, exact Area id and validation result. No new storage table
|
|
or provider-side path is introduced.
|
|
|
|
The DOV Mol soil operator retains evidence under:
|
|
|
|
```text
|
|
storage/operator-evidence/dov-soil-map/mol/
|
|
raw/dov_soil_map_page_*.json
|
|
dov_soil_map_mol.geojson
|
|
dov_soil_map_mol.manifest.json
|
|
```
|
|
|
|
Only the normalized, exactly clipped GeoJSON enters DatasetService and
|
|
`vector_features`. The manifest binds the persisted artifact, Mol boundary,
|
|
source page URLs/checksums, feature completeness, class-area summaries and
|
|
historical limitations. Raw source responses remain operator evidence.
|
|
|
|
The regional operator keeps the same source semantics but partitions its
|
|
evidence by approved municipality before assembling one snapshot:
|
|
|
|
```text
|
|
storage/operator-evidence/dov-soil-map/regional/
|
|
kempen-transport-region/partitions/<nis_code>/
|
|
raw/dov_soil_map_page_*.json.gz
|
|
dov_soil_map.geojson
|
|
manifest.json
|
|
snapshots/kempen-transport-region/
|
|
dov_soil_map.geojson
|
|
manifest.json
|
|
```
|
|
|
|
The snapshot manifest binds all 28 partition checksums, the official scope
|
|
artifact, feature completeness and area summaries by generalized legend,
|
|
texture and historical drainage class. Only the combined GeoJSON enters the
|
|
canonical DatasetService/vector-feature path.
|
|
|
|
## Model storage
|
|
|
|
Model artifacts live under:
|
|
|
|
```text
|
|
storage/models/
|
|
```
|
|
|
|
Database model registry records:
|
|
|
|
```yaml
|
|
model_id
|
|
name
|
|
task_type
|
|
framework
|
|
path
|
|
classes_json
|
|
version
|
|
created_at
|
|
metrics_json
|
|
```
|
|
|
|
## Exports
|
|
|
|
Every export is reproducible and linked to project/analysis.
|
|
|
|
Export record fields:
|
|
|
|
```yaml
|
|
id
|
|
project_id
|
|
analysis_run_id
|
|
export_type
|
|
path
|
|
format
|
|
created_at
|
|
parameters_json
|
|
```
|
|
|
|
## Local development default
|
|
|
|
Use local filesystem paths. Keep MinIO/object storage as future extension.
|
|
|
|
## Regional BWK/Natura 2000 evidence
|
|
|
|
The regional state-2025 operator stores immutable evidence under:
|
|
|
|
```text
|
|
storage/operator-evidence/bwk-natura2000-2025/regional/
|
|
kempen-transport-region/partitions/{nis_code}/
|
|
raw/*.json.gz
|
|
bwk_natura2000_2025.geojson
|
|
manifest.json
|
|
snapshots/kempen-transport-region/
|
|
bwk_natura2000_2025.geojson
|
|
manifest.json
|
|
```
|
|
|
|
Partition reuse requires the same source version, municipality identity,
|
|
boundary checksum, normalized output checksum and all raw evidence checksums.
|
|
Snapshot reuse additionally binds the ordered set of 28 partition output
|
|
checksums. Only the assembled snapshot enters DatasetService/PostGIS; raw WFS
|
|
responses and partition files remain operator evidence on persistent storage.
|