geointel/docs/STORAGE_ARCHITECTURE.md

# GeoIntel Kempen — Storage Architecture v1.0

GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage.

## Principles

- Database stores metadata, relationships and vector geometries.
- Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts.
- Every stored file must have a dataset/export/model record in the database.
- Never store large raster binary data directly in regular application tables in V1.

## Root storage layout

```text
storage/
  uploads/
    {project_id}/
      rasters/
      vectors/
      lidar/
  rasters/
    derived/
      {project_id}/{dataset_id}/
  tiles/
    {project_id}/{dataset_id}/{tile_set_id}/
  masks/
    {project_id}/{analysis_run_id}/
  previews/
    {project_id}/{dataset_id}/
  exports/
    {project_id}/
      geojson/
      csv/
      reports/
      coco/
      yolo/
  models/
    detection/
    segmentation/
    training-runs/
  cache/
    grb/
    osm/
    sentinel/
    dhmv/
  operator-data/
    geographic-scopes/
      belgium-north-sea/
        adminvector_4326.zip
        adminvector/
          adminvector_4326.gpkg
        belgium_land_boundary.geojson
        belgium_regions.geojson
        belgium_provinces.geojson
        belgium_municipalities.geojson
        ngi_maritime_zone.geojson
        marine_legal_scopes.geojson
        marine_spatial_plan_2026.geojson
        manifest.json
```

The Belgium/North Sea directory is written only by the explicit
`provision_belgium_north_sea_scope.py` operator. The manifest binds every
artifact to SHA256, source URL, edition/validity and feature count. Persistence
still runs through the ordinary Dataset upload API; operator artifacts are
evidence and replay inputs, not an alternate database.

## Upload policy

When a file is uploaded:

1. Save original file unchanged.
2. Compute checksum.
3. Extract metadata.
4. Create dataset record.
5. Create preview if applicable.

Required file metadata:

```yaml
path
original_filename
mime_type
size_bytes
checksum_sha256
created_at
storage_backend
```

## Derived data policy

Derived files must record:

- source dataset id(s)
- analysis run id or processing job id
- processing parameters
- software component version
- created_at

## Segmentation mask artifacts

Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records.

Default mask path convention:

```text
storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png
```

Optional run manifest convention:

```text
storage/masks/{project_id}/{analysis_run_id}/manifest.json
```

Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted `segmentations.geometry` rather than mask files.

## Cleanup policy

Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.

## Official temporal source artifacts

Statbel population staging stores immutable official source ZIPs below the
selected operator output root:

```text
<population-output>/
  raw/{year}/OPENDATA_SECTOREN_{year}[_NEW].zip
  raw/{year}/sh_statbel_statistical_sectors_31370_{year}0101.geojson.zip
  {scope}_statbel_population_{year}.geojson
  {scope}_statbel_population_{year}.preflight.json
```

The preflight manifest records official URLs, archive/member size and SHA-256,
schema fingerprints, EPSG:31370 and situation-date evidence, bounded topology
repairs, national/scope join accounting, unlocated `ZZZZ` totals, baseline
change evidence and the final derived-snapshot checksum. The upload operator
revalidates all retained files against that manifest. Raw ZIPs and manifests
are provenance artifacts; queryable population geometry remains an ordinary
Dataset/DatasetVersion plus PostGIS `vector_features` through the canonical
persistence services. Temporary standalone preflight output does not create a
database record and may be removed explicitly after operator review.

SPW bathymetry follows the same immutable-source/derived-dataset separation:

```text
storage/operator-evidence/spw-bathymetry/2023-05-23/
  raw/BATHY_50CM_ALTITUDE_DNG_GEOTIFF_3812.zip
  derived/spw_bathymetry_{selection_hash}_3812.tif
```

The raw ZIP is pinned by official URL and SHA-256 and is never extracted as an
8+ GiB working tree. Rasterio/GDAL reads the official GeoTIFF through
`/vsizip/`; the operator writes a bounded compressed COG and uploads that COG
through DatasetService. The queryable file therefore remains an ordinary
versioned raster Dataset. Source URL, archive/member checksum, selection
geometry/hash, survey period, EPSG:3812 and mDNG semantics are retained in
source/provenance metadata. PNG map overlays are derived responses and are not
authoritative storage.

Governed release-decision evidence is separate from the source artifacts:

```text
storage/operator-evidence/statbel-population-refresh/{scope}/{year}/
  staged-plan.json
  review-evidence.json
  applied-evidence.json
```

The staged plan binds the current official catalog identity to every retained
source/snapshot/preflight hash and accounting summary. Review evidence binds a
named approval to that exact plan. Applied evidence binds both approvals to
the resulting Dataset id. These JSON files are operator audit artifacts, not
database lifecycle entities. A plan/review path outside the configured
evidence root or a retained archive outside the population output root is
rejected. No evidence file authorizes deletion or in-place replacement.

Waterinfo raw station layers, timeseries responses and checksum manifests live
under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
evidence; queryable annual Point snapshots are normal Dataset/vector_feature
records. Bounded orthophotos are normal raster Dataset files. Their WMS URL,
product/layer, request/spatial hash, temporal validity and limitations are held
in source/provenance metadata. Browser PNG rendering is derived on request and
does not replace the stored GeoTIFF.

The orthophoto release preflight writes no source file, raster or database row.
Its JSON stdout is not staging authorization. Governed release evidence is
retained separately per scope, official edition and exact selection hash:

```text
storage/operator-evidence/orthophoto-release/{scope}/{YYYY.NN}/{selection-hash}/
  official-wms-response.tif
  orthophoto_{YYYY.NN}_{selection-hash}.tif
  review-preview.png
  staged-manifest.json
  staged-plan.json
  review-evidence.json
  applied-evidence.json
```

The manifest binds the one bounded source response, normalized three-band
EPSG:31370 GeoTIFF and PNG preview. The plan also binds WMS/WCS/catalog and
flight-day preflight identities; review binds a named approval; applied
evidence binds both to the immutable Dataset id and checksum. Paths outside
this root, changed files and changed provider/local state fail closed. Only the
normalized GeoTIFF enters ordinary Dataset storage through DatasetService.
Existing release evidence and `most_recent_at_*` raster metadata are never
rewritten. An exact apply retry validates and returns the original applied
evidence rather than replacing it.

DHMV II DTM/DSM outputs are also normal raster Dataset files. The provider WCS
returns multipart coverage data; GeoIntel retains response and extracted
coverage SHA256 values in provenance, then stores one normalized, compressed,
Area-clipped GeoTIFF with its ordinary Dataset/DatasetVersion checksum. Source
metadata records the 1 m native product, 5 m default analysis grid, EPSG:31370,
`-9999` nodata, TAW and acquisition period 2013-2015. Colour-relief PNGs are
derived browser views and are never authoritative. No raster binary is stored
in PostgreSQL and no DHMV file is treated as water depth or volume.

`provision_regional_dhmv.py` creates no separate evidence hierarchy and no
monolithic regional raster. Each municipality/product result remains one
ordinary Area-linked raster Dataset plus DatasetVersion. The retained
provenance already includes every bounded WCS tile URL and aggregate response,
coverage and normalized-output checksum. A repeat operator run resolves the
same request identity and reuses the immutable file.

VMM flood-hazard scenario outputs follow the same raster Dataset policy. The
bounded acquisition service stores one normalized, compressed, Area-clipped
GeoTIFF per scenario/request identity. `provision_regional_flood_hazards.py`
does not create a parallel storage layout: every municipality/scenario result
is an ordinary Dataset and DatasetVersion with WCS request hashes, tile request
URLs, response/coverage/normalized checksums, EPSG:31370 bounds, scenario
metadata and explicit unsupported-volume flags. Repeat runs reuse matching
ready Datasets through the acquisition service cache.

The regional Map workspace does not create another authoritative raster or
copy pixels into PostgreSQL. Its partition-selection endpoints read only the
municipal GeoTIFF windows intersecting a bounded selection, mosaic those
windows in memory at the governed analysis resolution and return metrics plus
the complete contributing `dataset_ids`. The 12-million-cell limit applies to
the combined window. Persisted files, checksums and DatasetVersions remain the
only authoritative artifacts.

BWK/Natura 2000 evidence lives under
`storage/operator-evidence/bwk-natura2000-2025/mol/`. The `raw/` directory
contains immutable WFS pages; the adjacent manifest records their URLs,
checksums, boundary checksum, page/feature counts, clipping diagnostics and
official class totals. The final GeoJSON is the retained source artifact while
`datasets` and `vector_features` remain the canonical queryable PostGIS state.

Definitive agricultural-use parcel evidence lives under:

```text
storage/operator-evidence/agricultural-use-parcels/{scope}/{year}/
  agpa_{year}_*_public.zip
  agricultural_use_parcels_{year}_{scope}.geojson
  agricultural_use_parcels_{year}_crop_codes.json
  agricultural_use_parcels_{year}_{scope}.manifest.json
```

The official ZIP is immutable source evidence and is never removed by normal
cache cleanup. The extracted GeoPackage is temporary to avoid retaining a
second full source copy. The normalized GeoJSON is the canonical upload
artifact; Dataset and vector_feature rows remain the queryable PostGIS state.
The manifest binds source, crop-code list and upload artifact checksums. A
checksum conflict with an existing annual Dataset fails closed.

Future definitive-release decision evidence is stored separately from those
source artifacts:

```text
storage/operator-evidence/alz-agriculture-refresh/{scope}/{year}/
  staged-plan.json
  review-evidence.json
  applied-evidence.json
```

The staged plan binds the official publication-page SHA-256 and definitive
archive identity to the retained archive, normalized GeoJSON, schema/CRS,
crop-code list, scope accounting and previous-edition delta. Review evidence
binds a named decision to the exact plan hash. Applied evidence binds both to
the resulting immutable Dataset id. Evidence paths outside this root and
source artifacts outside `agricultural-use-parcels` are rejected; these files
authorize neither deletion nor in-place Dataset replacement.

Buildings and Addresses Register snapshot evidence lives under:

```text
storage/operator-evidence/buildings-addresses-register/{area-key}/{observed-date}/
  raw/
    buildings_page_*.json
    building_units_page_*.json
    addresses_page_*.json
  buildings_addresses_register.geojson
  buildings_addresses_register.manifest.json
```

Raw pages contain the unmodified official response and are retained only as
checksummed operator evidence. They can contain address labels and must never
be served as a map/API artifact. The normalized GeoJSON deliberately contains
one polygon per register building with lifecycle state, aggregate relation
counts and classified GRB reconciliation only. It enters PostGIS exclusively
through DatasetService and ordinary `vector_features`; the operator never
writes database rows directly. The manifest binds all raw pages, the normalized
artifact, exact Area boundary and every GRB partition used for reconciliation.
The Area key is deterministic (`mol`, `geel`, and so on). Reuse also requires
the same Area label, observation date and boundary checksum, preventing one
municipality's evidence from being imported under another Area.

Offline demo export artifacts can be inspected and cleaned with:

```bash
python scripts/cleanup_demo_artifacts.py
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
```

The script is dry-run by default, targets only the explicit
`GeoIntel Demo - Building QA` project unless an exact `--project-name` is
provided, keeps the newest export artifacts per matching project and refuses to
delete files outside `STORAGE_ROOT`. It cleans `exports` records/files only; it
does not remove original uploads, vector features, QA/QC rows, projects, areas,
tiles, rasters or masks. `--max-delete` defaults to 25 and blocks oversized
apply runs until the operator increases the cap after reviewing dry-run output.
Repeat `--export-type` to restrict cleanup to selected artifact kinds.

Live runtime validation for this maintenance path is available as a dry-run
smoke:

```bash
bash scripts/verify_demo_cleanup_dry_run.sh
CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh
```

The smoke never passes `--apply`. It fails if the cleanup summary is not a
dry-run, if any export/file deletion is reported, or if the dry-run candidate
fields are missing.

## RC-10 data operations

`docs/DATA_OPERATIONS_RUNBOOK.md` is the executable retention source of truth.
`scripts/audit_data_operations.py` reports storage growth, disk pressure,
persisted path integrity, old failed work and national/regional/maritime
source families without mutation. `scripts/cleanup_storage_artifacts.py` can
only remove old unreferenced artifacts from the explicit derived/cache/export
allowlist.

Every unknown category, upload, original, model, operator evidence and release
evidence path is protected by default. Apply mode requires an exact
confirmation token and a recent checksum-verified release backup containing a
SHA-256 storage inventory. The backup root is mounted read-only at
`/app/backups`; no cleanup is scheduled implicitly.

## Governed cross-domain raster and soil evidence

MercatorNet thematic products use the ordinary raster Dataset and immutable
DatasetVersion paths under `STORAGE_ROOT`. `source_metadata` records product,
coverage, native resolution/unit, render range, reference year and request
bounds. `provenance_metadata` records every tiled WCS URL, transfer checksum,
normalized checksum, exact Area id and validation result. No new storage table
or provider-side path is introduced.

The DOV Mol soil operator retains evidence under:

```text
storage/operator-evidence/dov-soil-map/mol/
  raw/dov_soil_map_page_*.json
  dov_soil_map_mol.geojson
  dov_soil_map_mol.manifest.json
```

Only the normalized, exactly clipped GeoJSON enters DatasetService and
`vector_features`. The manifest binds the persisted artifact, Mol boundary,
source page URLs/checksums, feature completeness, class-area summaries and
historical limitations. Raw source responses remain operator evidence.

The regional operator keeps the same source semantics but partitions its
evidence by approved municipality before assembling one snapshot:

```text
storage/operator-evidence/dov-soil-map/regional/
  kempen-transport-region/partitions/<nis_code>/
    raw/dov_soil_map_page_*.json.gz
    dov_soil_map.geojson
    manifest.json
  snapshots/kempen-transport-region/
    dov_soil_map.geojson
    manifest.json
```

The snapshot manifest binds all 28 partition checksums, the official scope
artifact, feature completeness and area summaries by generalized legend,
texture and historical drainage class. Only the combined GeoJSON enters the
canonical DatasetService/vector-feature path.

## Model storage

Model artifacts live under:

```text
storage/models/
```

Database model registry records:

```yaml
model_id
name
task_type
framework
path
classes_json
version
created_at
metrics_json
```

## Exports

Every export is reproducible and linked to project/analysis.

Export record fields:

```yaml
id
project_id
analysis_run_id
export_type
path
format
created_at
parameters_json
```

## Local development default

Use local filesystem paths. Keep MinIO/object storage as future extension.

## Regional BWK/Natura 2000 evidence

The regional state-2025 operator stores immutable evidence under:

```text
storage/operator-evidence/bwk-natura2000-2025/regional/
  kempen-transport-region/partitions/{nis_code}/
    raw/*.json.gz
    bwk_natura2000_2025.geojson
    manifest.json
  snapshots/kempen-transport-region/
    bwk_natura2000_2025.geojson
    manifest.json
```

Partition reuse requires the same source version, municipality identity,
boundary checksum, normalized output checksum and all raw evidence checksums.
Snapshot reuse additionally binds the ordered set of 28 partition output
checksums. Only the assembled snapshot enters DatasetService/PostGIS; raw WFS
responses and partition files remain operator evidence on persistent storage.