Initial GeoIntel V1 foundation

2026-06-16 23:36:32 +02:00
commit 6ea3586a3e
605 changed files with 45284 additions and 0 deletions
@@ -0,0 +1,143 @@
+# GeoIntel Kempen — Storage Architecture v1.0
+
+GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage.
+
+## Principles
+
+- Database stores metadata, relationships and vector geometries.
+- Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts.
+- Every stored file must have a dataset/export/model record in the database.
+- Never store large raster binary data directly in regular application tables in V1.
+
+## Root storage layout
+
+```text
+storage/
+  uploads/
+    {project_id}/
+      rasters/
+      vectors/
+      lidar/
+  rasters/
+    derived/
+      {project_id}/{dataset_id}/
+  tiles/
+    {project_id}/{dataset_id}/{tile_set_id}/
+  masks/
+    {project_id}/{analysis_run_id}/
+  previews/
+    {project_id}/{dataset_id}/
+  exports/
+    {project_id}/
+      geojson/
+      csv/
+      reports/
+      coco/
+      yolo/
+  models/
+    detection/
+    segmentation/
+    training-runs/
+  cache/
+    grb/
+    osm/
+    sentinel/
+    dhmv/
+```
+
+## Upload policy
+
+When a file is uploaded:
+
+1. Save original file unchanged.
+2. Compute checksum.
+3. Extract metadata.
+4. Create dataset record.
+5. Create preview if applicable.
+
+Required file metadata:
+
+```yaml
+path
+original_filename
+mime_type
+size_bytes
+checksum_sha256
+created_at
+storage_backend
+```
+
+## Derived data policy
+
+Derived files must record:
+
+- source dataset id(s)
+- analysis run id or processing job id
+- processing parameters
+- software component version
+- created_at
+
+## Segmentation mask artifacts
+
+Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records.
+
+Default mask path convention:
+
+```text
+storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png
+```
+
+Optional run manifest convention:
+
+```text
+storage/masks/{project_id}/{analysis_run_id}/manifest.json
+```
+
+Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted `segmentations.geometry` rather than mask files.
+
+## Cleanup policy
+
+Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.
+
+## Model storage
+
+Model artifacts live under:
+
+```text
+storage/models/
+```
+
+Database model registry records:
+
+```yaml
+model_id
+name
+task_type
+framework
+path
+classes_json
+version
+created_at
+metrics_json
+```
+
+## Exports
+
+Every export is reproducible and linked to project/analysis.
+
+Export record fields:
+
+```yaml
+id
+project_id
+analysis_run_id
+export_type
+path
+format
+created_at
+parameters_json
+```
+
+## Local development default
+
+Use local filesystem paths. Keep MinIO/object storage as future extension.