Initial public release

docs/STORAGE_ARCHITECTURE.md

@@ -0,0 +1,460 @@
+# GeoIntel Kempen — Storage Architecture v1.0
+
+GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage.
+
+## Principles
+
+- Database stores metadata, relationships and vector geometries.
+- Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts.
+- Every stored file must have a dataset/export/model record in the database.
+- Never store large raster binary data directly in regular application tables in V1.
+
+## Root storage layout
+
+```text
+storage/
+  uploads/
+    {project_id}/
+      rasters/
+      vectors/
+      lidar/
+  rasters/
+    derived/
+      {project_id}/{dataset_id}/
+  tiles/
+    {project_id}/{dataset_id}/{tile_set_id}/
+  masks/
+    {project_id}/{analysis_run_id}/
+  previews/
+    {project_id}/{dataset_id}/
+  exports/
+    {project_id}/
+      geojson/
+      csv/
+      reports/
+      coco/
+      yolo/
+  models/
+    detection/
+    segmentation/
+    training-runs/
+  cache/
+    grb/
+    osm/
+    sentinel/
+    dhmv/
+  operator-data/
+    geographic-scopes/
+      belgium-north-sea/
+        adminvector_4326.zip
+        adminvector/
+          adminvector_4326.gpkg
+        belgium_land_boundary.geojson
+        belgium_regions.geojson
+        belgium_provinces.geojson
+        belgium_municipalities.geojson
+        ngi_maritime_zone.geojson
+        marine_legal_scopes.geojson
+        marine_spatial_plan_2026.geojson
+        manifest.json
+```
+
+The Belgium/North Sea directory is written only by the explicit
+`provision_belgium_north_sea_scope.py` operator. The manifest binds every
+artifact to SHA256, source URL, edition/validity and feature count. Persistence
+still runs through the ordinary Dataset upload API; operator artifacts are
+evidence and replay inputs, not an alternate database.
+
+## Upload policy
+
+When a file is uploaded:
+
+1. Save original file unchanged.
+2. Compute checksum.
+3. Extract metadata.
+4. Create dataset record.
+5. Create preview if applicable.
+
+Required file metadata:
+
+```yaml
+path
+original_filename
+mime_type
+size_bytes
+checksum_sha256
+created_at
+storage_backend
+```
+
+## Derived data policy
+
+Derived files must record:
+
+- source dataset id(s)
+- analysis run id or processing job id
+- processing parameters
+- software component version
+- created_at
+
+## Segmentation mask artifacts
+
+Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records.
+
+Default mask path convention:
+
+```text
+storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png
+```
+
+Optional run manifest convention:
+
+```text
+storage/masks/{project_id}/{analysis_run_id}/manifest.json
+```
+
+Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted `segmentations.geometry` rather than mask files.
+
+## Cleanup policy
+
+Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.
+
+## Official temporal source artifacts
+
+Statbel population staging stores immutable official source ZIPs below the
+selected operator output root:
+
+```text
+<population-output>/
+  raw/{year}/OPENDATA_SECTOREN_{year}[_NEW].zip
+  raw/{year}/sh_statbel_statistical_sectors_31370_{year}0101.geojson.zip
+  {scope}_statbel_population_{year}.geojson
+  {scope}_statbel_population_{year}.preflight.json
+```
+
+The preflight manifest records official URLs, archive/member size and SHA-256,
+schema fingerprints, EPSG:31370 and situation-date evidence, bounded topology
+repairs, national/scope join accounting, unlocated `ZZZZ` totals, baseline
+change evidence and the final derived-snapshot checksum. The upload operator
+revalidates all retained files against that manifest. Raw ZIPs and manifests
+are provenance artifacts; queryable population geometry remains an ordinary
+Dataset/DatasetVersion plus PostGIS `vector_features` through the canonical
+persistence services. Temporary standalone preflight output does not create a
+database record and may be removed explicitly after operator review.
+
+SPW bathymetry follows the same immutable-source/derived-dataset separation:
+
+```text
+storage/operator-evidence/spw-bathymetry/2023-05-23/
+  raw/BATHY_50CM_ALTITUDE_DNG_GEOTIFF_3812.zip
+  derived/spw_bathymetry_{selection_hash}_3812.tif
+```
+
+The raw ZIP is pinned by official URL and SHA-256 and is never extracted as an
+8+ GiB working tree. Rasterio/GDAL reads the official GeoTIFF through
+`/vsizip/`; the operator writes a bounded compressed COG and uploads that COG
+through DatasetService. The queryable file therefore remains an ordinary
+versioned raster Dataset. Source URL, archive/member checksum, selection
+geometry/hash, survey period, EPSG:3812 and mDNG semantics are retained in
+source/provenance metadata. PNG map overlays are derived responses and are not
+authoritative storage.
+
+Governed release-decision evidence is separate from the source artifacts:
+
+```text
+storage/operator-evidence/statbel-population-refresh/{scope}/{year}/
+  staged-plan.json
+  review-evidence.json
+  applied-evidence.json
+```
+
+The staged plan binds the current official catalog identity to every retained
+source/snapshot/preflight hash and accounting summary. Review evidence binds a
+named approval to that exact plan. Applied evidence binds both approvals to
+the resulting Dataset id. These JSON files are operator audit artifacts, not
+database lifecycle entities. A plan/review path outside the configured
+evidence root or a retained archive outside the population output root is
+rejected. No evidence file authorizes deletion or in-place replacement.
+
+Waterinfo raw station layers, timeseries responses and checksum manifests live
+under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
+evidence; queryable annual Point snapshots are normal Dataset/vector_feature
+records. Bounded orthophotos are normal raster Dataset files. Their WMS URL,
+product/layer, request/spatial hash, temporal validity and limitations are held
+in source/provenance metadata. Browser PNG rendering is derived on request and
+does not replace the stored GeoTIFF.
+
+The orthophoto release preflight writes no source file, raster or database row.
+Its JSON stdout is not staging authorization. Governed release evidence is
+retained separately per scope, official edition and exact selection hash:
+
+```text
+storage/operator-evidence/orthophoto-release/{scope}/{YYYY.NN}/{selection-hash}/
+  official-wms-response.tif
+  orthophoto_{YYYY.NN}_{selection-hash}.tif
+  review-preview.png
+  staged-manifest.json
+  staged-plan.json
+  review-evidence.json
+  applied-evidence.json
+```
+
+The manifest binds the one bounded source response, normalized three-band
+EPSG:31370 GeoTIFF and PNG preview. The plan also binds WMS/WCS/catalog and
+flight-day preflight identities; review binds a named approval; applied
+evidence binds both to the immutable Dataset id and checksum. Paths outside
+this root, changed files and changed provider/local state fail closed. Only the
+normalized GeoTIFF enters ordinary Dataset storage through DatasetService.
+Existing release evidence and `most_recent_at_*` raster metadata are never
+rewritten. An exact apply retry validates and returns the original applied
+evidence rather than replacing it.
+
+DHMV II DTM/DSM outputs are also normal raster Dataset files. The provider WCS
+returns multipart coverage data; GeoIntel retains response and extracted
+coverage SHA256 values in provenance, then stores one normalized, compressed,
+Area-clipped GeoTIFF with its ordinary Dataset/DatasetVersion checksum. Source
+metadata records the 1 m native product, 5 m default analysis grid, EPSG:31370,
+`-9999` nodata, TAW and acquisition period 2013-2015. Colour-relief PNGs are
+derived browser views and are never authoritative. No raster binary is stored
+in PostgreSQL and no DHMV file is treated as water depth or volume.
+
+`provision_regional_dhmv.py` creates no separate evidence hierarchy and no
+monolithic regional raster. Each municipality/product result remains one
+ordinary Area-linked raster Dataset plus DatasetVersion. The retained
+provenance already includes every bounded WCS tile URL and aggregate response,
+coverage and normalized-output checksum. A repeat operator run resolves the
+same request identity and reuses the immutable file.
+
+VMM flood-hazard scenario outputs follow the same raster Dataset policy. The
+bounded acquisition service stores one normalized, compressed, Area-clipped
+GeoTIFF per scenario/request identity. `provision_regional_flood_hazards.py`
+does not create a parallel storage layout: every municipality/scenario result
+is an ordinary Dataset and DatasetVersion with WCS request hashes, tile request
+URLs, response/coverage/normalized checksums, EPSG:31370 bounds, scenario
+metadata and explicit unsupported-volume flags. Repeat runs reuse matching
+ready Datasets through the acquisition service cache.
+
+The regional Map workspace does not create another authoritative raster or
+copy pixels into PostgreSQL. Its partition-selection endpoints read only the
+municipal GeoTIFF windows intersecting a bounded selection, mosaic those
+windows in memory at the governed analysis resolution and return metrics plus
+the complete contributing `dataset_ids`. The 12-million-cell limit applies to
+the combined window. Persisted files, checksums and DatasetVersions remain the
+only authoritative artifacts.
+
+BWK/Natura 2000 evidence lives under
+`storage/operator-evidence/bwk-natura2000-2025/mol/`. The `raw/` directory
+contains immutable WFS pages; the adjacent manifest records their URLs,
+checksums, boundary checksum, page/feature counts, clipping diagnostics and
+official class totals. The final GeoJSON is the retained source artifact while
+`datasets` and `vector_features` remain the canonical queryable PostGIS state.
+
+Definitive agricultural-use parcel evidence lives under:
+
+```text
+storage/operator-evidence/agricultural-use-parcels/{scope}/{year}/
+  agpa_{year}_*_public.zip
+  agricultural_use_parcels_{year}_{scope}.geojson
+  agricultural_use_parcels_{year}_crop_codes.json
+  agricultural_use_parcels_{year}_{scope}.manifest.json
+```
+
+The official ZIP is immutable source evidence and is never removed by normal
+cache cleanup. The extracted GeoPackage is temporary to avoid retaining a
+second full source copy. The normalized GeoJSON is the canonical upload
+artifact; Dataset and vector_feature rows remain the queryable PostGIS state.
+The manifest binds source, crop-code list and upload artifact checksums. A
+checksum conflict with an existing annual Dataset fails closed.
+
+Future definitive-release decision evidence is stored separately from those
+source artifacts:
+
+```text
+storage/operator-evidence/alz-agriculture-refresh/{scope}/{year}/
+  staged-plan.json
+  review-evidence.json
+  applied-evidence.json
+```
+
+The staged plan binds the official publication-page SHA-256 and definitive
+archive identity to the retained archive, normalized GeoJSON, schema/CRS,
+crop-code list, scope accounting and previous-edition delta. Review evidence
+binds a named decision to the exact plan hash. Applied evidence binds both to
+the resulting immutable Dataset id. Evidence paths outside this root and
+source artifacts outside `agricultural-use-parcels` are rejected; these files
+authorize neither deletion nor in-place Dataset replacement.
+
+Buildings and Addresses Register snapshot evidence lives under:
+
+```text
+storage/operator-evidence/buildings-addresses-register/{area-key}/{observed-date}/
+  raw/
+    buildings_page_*.json
+    building_units_page_*.json
+    addresses_page_*.json
+  buildings_addresses_register.geojson
+  buildings_addresses_register.manifest.json
+```
+
+Raw pages contain the unmodified official response and are retained only as
+checksummed operator evidence. They can contain address labels and must never
+be served as a map/API artifact. The normalized GeoJSON deliberately contains
+one polygon per register building with lifecycle state, aggregate relation
+counts and classified GRB reconciliation only. It enters PostGIS exclusively
+through DatasetService and ordinary `vector_features`; the operator never
+writes database rows directly. The manifest binds all raw pages, the normalized
+artifact, exact Area boundary and every GRB partition used for reconciliation.
+The Area key is deterministic (`mol`, `geel`, and so on). Reuse also requires
+the same Area label, observation date and boundary checksum, preventing one
+municipality's evidence from being imported under another Area.
+
+Offline demo export artifacts can be inspected and cleaned with:
+
+```bash
+python scripts/cleanup_demo_artifacts.py
+python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
+python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
+docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
+```
+
+The script is dry-run by default, targets only the explicit
+`GeoIntel Demo - Building QA` project unless an exact `--project-name` is
+provided, keeps the newest export artifacts per matching project and refuses to
+delete files outside `STORAGE_ROOT`. It cleans `exports` records/files only; it
+does not remove original uploads, vector features, QA/QC rows, projects, areas,
+tiles, rasters or masks. `--max-delete` defaults to 25 and blocks oversized
+apply runs until the operator increases the cap after reviewing dry-run output.
+Repeat `--export-type` to restrict cleanup to selected artifact kinds.
+
+Live runtime validation for this maintenance path is available as a dry-run
+smoke:
+
+```bash
+bash scripts/verify_demo_cleanup_dry_run.sh
+CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh
+```
+
+The smoke never passes `--apply`. It fails if the cleanup summary is not a
+dry-run, if any export/file deletion is reported, or if the dry-run candidate
+fields are missing.
+
+## RC-10 data operations
+
+`docs/DATA_OPERATIONS_RUNBOOK.md` is the executable retention source of truth.
+`scripts/audit_data_operations.py` reports storage growth, disk pressure,
+persisted path integrity, old failed work and national/regional/maritime
+source families without mutation. `scripts/cleanup_storage_artifacts.py` can
+only remove old unreferenced artifacts from the explicit derived/cache/export
+allowlist.
+
+Every unknown category, upload, original, model, operator evidence and release
+evidence path is protected by default. Apply mode requires an exact
+confirmation token and a recent checksum-verified release backup containing a
+SHA-256 storage inventory. The backup root is mounted read-only at
+`/app/backups`; no cleanup is scheduled implicitly.
+
+## Governed cross-domain raster and soil evidence
+
+MercatorNet thematic products use the ordinary raster Dataset and immutable
+DatasetVersion paths under `STORAGE_ROOT`. `source_metadata` records product,
+coverage, native resolution/unit, render range, reference year and request
+bounds. `provenance_metadata` records every tiled WCS URL, transfer checksum,
+normalized checksum, exact Area id and validation result. No new storage table
+or provider-side path is introduced.
+
+The DOV Mol soil operator retains evidence under:
+
+```text
+storage/operator-evidence/dov-soil-map/mol/
+  raw/dov_soil_map_page_*.json
+  dov_soil_map_mol.geojson
+  dov_soil_map_mol.manifest.json
+```
+
+Only the normalized, exactly clipped GeoJSON enters DatasetService and
+`vector_features`. The manifest binds the persisted artifact, Mol boundary,
+source page URLs/checksums, feature completeness, class-area summaries and
+historical limitations. Raw source responses remain operator evidence.
+
+The regional operator keeps the same source semantics but partitions its
+evidence by approved municipality before assembling one snapshot:
+
+```text
+storage/operator-evidence/dov-soil-map/regional/
+  kempen-transport-region/partitions/<nis_code>/
+    raw/dov_soil_map_page_*.json.gz
+    dov_soil_map.geojson
+    manifest.json
+  snapshots/kempen-transport-region/
+    dov_soil_map.geojson
+    manifest.json
+```
+
+The snapshot manifest binds all 28 partition checksums, the official scope
+artifact, feature completeness and area summaries by generalized legend,
+texture and historical drainage class. Only the combined GeoJSON enters the
+canonical DatasetService/vector-feature path.
+
+## Model storage
+
+Model artifacts live under:
+
+```text
+storage/models/
+```
+
+Database model registry records:
+
+```yaml
+model_id
+name
+task_type
+framework
+path
+classes_json
+version
+created_at
+metrics_json
+```
+
+## Exports
+
+Every export is reproducible and linked to project/analysis.
+
+Export record fields:
+
+```yaml
+id
+project_id
+analysis_run_id
+export_type
+path
+format
+created_at
+parameters_json
+```
+
+## Local development default
+
+Use local filesystem paths. Keep MinIO/object storage as future extension.
+
+## Regional BWK/Natura 2000 evidence
+
+The regional state-2025 operator stores immutable evidence under:
+
+```text
+storage/operator-evidence/bwk-natura2000-2025/regional/
+  kempen-transport-region/partitions/{nis_code}/
+    raw/*.json.gz
+    bwk_natura2000_2025.geojson
+    manifest.json
+  snapshots/kempen-transport-region/
+    bwk_natura2000_2025.geojson
+    manifest.json
+```
+
+Partition reuse requires the same source version, municipality identity,
+boundary checksum, normalized output checksum and all raw evidence checksums.
+Snapshot reuse additionally binds the ordered set of 28 partition output
+checksums. Only the assembled snapshot enters DatasetService/PostGIS; raw WFS
+responses and partition files remain operator evidence on persistent storage.