geointel/docs/STORAGE_ARCHITECTURE.md

# GeoIntel Kempen — Storage Architecture v1.0

GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage.

## Principles

- Database stores metadata, relationships and vector geometries.
- Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts.
- Every stored file must have a dataset/export/model record in the database.
- Never store large raster binary data directly in regular application tables in V1.

## Root storage layout

```text
storage/
  uploads/
    {project_id}/
      rasters/
      vectors/
      lidar/
  rasters/
    derived/
      {project_id}/{dataset_id}/
  tiles/
    {project_id}/{dataset_id}/{tile_set_id}/
  masks/
    {project_id}/{analysis_run_id}/
  previews/
    {project_id}/{dataset_id}/
  exports/
    {project_id}/
      geojson/
      csv/
      reports/
      coco/
      yolo/
  models/
    detection/
    segmentation/
    training-runs/
  cache/
    grb/
    osm/
    sentinel/
    dhmv/
```

## Upload policy

When a file is uploaded:

1. Save original file unchanged.
2. Compute checksum.
3. Extract metadata.
4. Create dataset record.
5. Create preview if applicable.

Required file metadata:

```yaml
path
original_filename
mime_type
size_bytes
checksum_sha256
created_at
storage_backend
```

## Derived data policy

Derived files must record:

- source dataset id(s)
- analysis run id or processing job id
- processing parameters
- software component version
- created_at

## Segmentation mask artifacts

Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records.

Default mask path convention:

```text
storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png
```

Optional run manifest convention:

```text
storage/masks/{project_id}/{analysis_run_id}/manifest.json
```

Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted `segmentations.geometry` rather than mask files.

## Cleanup policy

Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.

## Official temporal source artifacts

Waterinfo raw station layers, timeseries responses and checksum manifests live
under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
evidence; queryable annual Point snapshots are normal Dataset/vector_feature
records. Bounded orthophotos are normal raster Dataset files. Their WMS URL,
product/layer, request/spatial hash, temporal validity and limitations are held
in source/provenance metadata. Browser PNG rendering is derived on request and
does not replace the stored GeoTIFF.

BWK/Natura 2000 evidence lives under
`storage/operator-evidence/bwk-natura2000-2025/mol/`. The `raw/` directory
contains immutable WFS pages; the adjacent manifest records their URLs,
checksums, boundary checksum, page/feature counts, clipping diagnostics and
official class totals. The final GeoJSON is the retained source artifact while
`datasets` and `vector_features` remain the canonical queryable PostGIS state.

Definitive agricultural-use parcel evidence lives under:

```text
storage/operator-evidence/agricultural-use-parcels/{scope}/{year}/
  agpa_{year}_*_public.zip
  agricultural_use_parcels_{year}_{scope}.geojson
  agricultural_use_parcels_{year}_crop_codes.json
  agricultural_use_parcels_{year}_{scope}.manifest.json
```

The official ZIP is immutable source evidence and is never removed by normal
cache cleanup. The extracted GeoPackage is temporary to avoid retaining a
second full source copy. The normalized GeoJSON is the canonical upload
artifact; Dataset and vector_feature rows remain the queryable PostGIS state.
The manifest binds source, crop-code list and upload artifact checksums. A
checksum conflict with an existing annual Dataset fails closed.

Offline demo export artifacts can be inspected and cleaned with:

```bash
python scripts/cleanup_demo_artifacts.py
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
```

The script is dry-run by default, targets only the explicit
`GeoIntel Demo - Building QA` project unless an exact `--project-name` is
provided, keeps the newest export artifacts per matching project and refuses to
delete files outside `STORAGE_ROOT`. It cleans `exports` records/files only; it
does not remove original uploads, vector features, QA/QC rows, projects, areas,
tiles, rasters or masks. `--max-delete` defaults to 25 and blocks oversized
apply runs until the operator increases the cap after reviewing dry-run output.
Repeat `--export-type` to restrict cleanup to selected artifact kinds.

Live runtime validation for this maintenance path is available as a dry-run
smoke:

```bash
bash scripts/verify_demo_cleanup_dry_run.sh
CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh
```

The smoke never passes `--apply`. It fails if the cleanup summary is not a
dry-run, if any export/file deletion is reported, or if the dry-run candidate
fields are missing.

## Model storage

Model artifacts live under:

```text
storage/models/
```

Database model registry records:

```yaml
model_id
name
task_type
framework
path
classes_json
version
created_at
metrics_json
```

## Exports

Every export is reproducible and linked to project/analysis.

Export record fields:

```yaml
id
project_id
analysis_run_id
export_type
path
format
created_at
parameters_json
```

## Local development default

Use local filesystem paths. Keep MinIO/object storage as future extension.