geointel/docs/STORAGE_ARCHITECTURE.md

GeoIntel Kempen — Storage Architecture v1.0

GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage.

Principles

• Database stores metadata, relationships and vector geometries.
• Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts.
• Every stored file must have a dataset/export/model record in the database.
• Never store large raster binary data directly in regular application tables in V1.

Root storage layout

storage/
  uploads/
    {project_id}/
      rasters/
      vectors/
      lidar/
  rasters/
    derived/
      {project_id}/{dataset_id}/
  tiles/
    {project_id}/{dataset_id}/{tile_set_id}/
  masks/
    {project_id}/{analysis_run_id}/
  previews/
    {project_id}/{dataset_id}/
  exports/
    {project_id}/
      geojson/
      csv/
      reports/
      coco/
      yolo/
  models/
    detection/
    segmentation/
    training-runs/
  cache/
    grb/
    osm/
    sentinel/
    dhmv/

Upload policy

When a file is uploaded:

1. Save original file unchanged.
2. Compute checksum.
3. Extract metadata.
4. Create dataset record.
5. Create preview if applicable.

Required file metadata:

path
original_filename
mime_type
size_bytes
checksum_sha256
created_at
storage_backend

Derived data policy

Derived files must record:

• source dataset id(s)
• analysis run id or processing job id
• processing parameters
• software component version
• created_at

Segmentation mask artifacts

Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records.

Default mask path convention:

storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png

Optional run manifest convention:

storage/masks/{project_id}/{analysis_run_id}/manifest.json

Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted segmentations.geometry rather than mask files.

Cleanup policy

Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.

Model storage

Model artifacts live under:

storage/models/

Database model registry records:

model_id
name
task_type
framework
path
classes_json
version
created_at
metrics_json

Exports

Every export is reproducible and linked to project/analysis.

Export record fields:

id
project_id
analysis_run_id
export_type
path
format
created_at
parameters_json

Local development default

Use local filesystem paths. Keep MinIO/object storage as future extension.