# GeoIntel Kempen — Storage Architecture v1.0 GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage. ## Principles - Database stores metadata, relationships and vector geometries. - Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts. - Every stored file must have a dataset/export/model record in the database. - Never store large raster binary data directly in regular application tables in V1. ## Root storage layout ```text storage/ uploads/ {project_id}/ rasters/ vectors/ lidar/ rasters/ derived/ {project_id}/{dataset_id}/ tiles/ {project_id}/{dataset_id}/{tile_set_id}/ masks/ {project_id}/{analysis_run_id}/ previews/ {project_id}/{dataset_id}/ exports/ {project_id}/ geojson/ csv/ reports/ coco/ yolo/ models/ detection/ segmentation/ training-runs/ cache/ grb/ osm/ sentinel/ dhmv/ ``` ## Upload policy When a file is uploaded: 1. Save original file unchanged. 2. Compute checksum. 3. Extract metadata. 4. Create dataset record. 5. Create preview if applicable. Required file metadata: ```yaml path original_filename mime_type size_bytes checksum_sha256 created_at storage_backend ``` ## Derived data policy Derived files must record: - source dataset id(s) - analysis run id or processing job id - processing parameters - software component version - created_at ## Segmentation mask artifacts Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records. Default mask path convention: ```text storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png ``` Optional run manifest convention: ```text storage/masks/{project_id}/{analysis_run_id}/manifest.json ``` Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted `segmentations.geometry` rather than mask files. ## Cleanup policy Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management. ## Official temporal source artifacts Waterinfo raw station layers, timeseries responses and checksum manifests live under `storage/operator-data/waterinfo//`. These are immutable source evidence; queryable annual Point snapshots are normal Dataset/vector_feature records. Bounded orthophotos are normal raster Dataset files. Their WMS URL, product/layer, request/spatial hash, temporal validity and limitations are held in source/provenance metadata. Browser PNG rendering is derived on request and does not replace the stored GeoTIFF. BWK/Natura 2000 evidence lives under `storage/operator-evidence/bwk-natura2000-2025/mol/`. The `raw/` directory contains immutable WFS pages; the adjacent manifest records their URLs, checksums, boundary checksum, page/feature counts, clipping diagnostics and official class totals. The final GeoJSON is the retained source artifact while `datasets` and `vector_features` remain the canonical queryable PostGIS state. Definitive agricultural-use parcel evidence lives under: ```text storage/operator-evidence/agricultural-use-parcels/{scope}/{year}/ agpa_{year}_*_public.zip agricultural_use_parcels_{year}_{scope}.geojson agricultural_use_parcels_{year}_crop_codes.json agricultural_use_parcels_{year}_{scope}.manifest.json ``` The official ZIP is immutable source evidence and is never removed by normal cache cleanup. The extracted GeoPackage is temporary to avoid retaining a second full source copy. The normalized GeoJSON is the canonical upload artifact; Dataset and vector_feature rows remain the queryable PostGIS state. The manifest binds source, crop-code list and upload artifact checksums. A checksum conflict with an existing annual Dataset fails closed. Buildings and Addresses Register snapshot evidence lives under: ```text storage/operator-evidence/buildings-addresses-register/mol/{observed-date}/ raw/ buildings_page_*.json building_units_page_*.json addresses_page_*.json buildings_addresses_register.geojson buildings_addresses_register.manifest.json ``` Raw pages contain the unmodified official response and are retained only as checksummed operator evidence. They can contain address labels and must never be served as a map/API artifact. The normalized GeoJSON deliberately contains one polygon per register building with lifecycle state, aggregate relation counts and classified GRB reconciliation only. It enters PostGIS exclusively through DatasetService and ordinary `vector_features`; the operator never writes database rows directly. The manifest binds all raw pages, the normalized artifact, exact Area boundary and every GRB partition used for reconciliation. Offline demo export artifacts can be inspected and cleaned with: ```bash python scripts/cleanup_demo_artifacts.py python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply docker compose exec -T backend python scripts/cleanup_demo_artifacts.py ``` The script is dry-run by default, targets only the explicit `GeoIntel Demo - Building QA` project unless an exact `--project-name` is provided, keeps the newest export artifacts per matching project and refuses to delete files outside `STORAGE_ROOT`. It cleans `exports` records/files only; it does not remove original uploads, vector features, QA/QC rows, projects, areas, tiles, rasters or masks. `--max-delete` defaults to 25 and blocks oversized apply runs until the operator increases the cap after reviewing dry-run output. Repeat `--export-type` to restrict cleanup to selected artifact kinds. Live runtime validation for this maintenance path is available as a dry-run smoke: ```bash bash scripts/verify_demo_cleanup_dry_run.sh CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh ``` The smoke never passes `--apply`. It fails if the cleanup summary is not a dry-run, if any export/file deletion is reported, or if the dry-run candidate fields are missing. ## Model storage Model artifacts live under: ```text storage/models/ ``` Database model registry records: ```yaml model_id name task_type framework path classes_json version created_at metrics_json ``` ## Exports Every export is reproducible and linked to project/analysis. Export record fields: ```yaml id project_id analysis_run_id export_type path format created_at parameters_json ``` ## Local development default Use local filesystem paths. Keep MinIO/object storage as future extension.