Initial GeoIntel V1 foundation

docs/DATABASE_IMPLEMENTATION_PLAN.md

@@ -0,0 +1,231 @@
+# Database Implementation Plan
+
+Database: PostgreSQL + PostGIS.
+
+## Rules
+
+- Store geometries in PostGIS with explicit SRID.
+- Preserve original CRS metadata even when normalized geometry is stored as EPSG:4326 or a local projected CRS.
+- Prefer UUID primary keys.
+- Store large raster/mask/model files in filesystem or object storage; store metadata and paths in PostgreSQL.
+- Keep analysis outputs reproducible by storing parameters JSON.
+
+## Required extensions
+
+```sql
+CREATE EXTENSION IF NOT EXISTS postgis;
+CREATE EXTENSION IF NOT EXISTS postgis_topology;
+CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
+```
+
+## Core tables
+
+### projects
+
+- `id uuid primary key`
+- `name text not null`
+- `description text`
+- `region text default 'Kempen'`
+- `status text default 'active'`
+- `created_at timestamptz`
+- `updated_at timestamptz`
+
+### areas
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `name text not null`
+- `geometry geometry(MultiPolygon, 4326) not null`
+- `original_crs text`
+- `area_m2 double precision`
+- `bbox geometry(Polygon, 4326)`
+- `created_at timestamptz`
+
+Spatial index required on `geometry`.
+
+### datasets
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `area_id uuid nullable references areas(id)`
+- `name text not null`
+- `dataset_type text not null`
+- `source text not null`
+- `storage_path text`
+- `derived_from_dataset_id uuid nullable references datasets(id)`
+- `crs text`
+- `bounds_json jsonb`
+- `resolution_json jsonb`
+- `bands_json jsonb`
+- `metadata_json jsonb`
+- `status text default 'created'`
+- `created_at timestamptz`
+
+### vector_features
+
+Used for imported vector datasets and derived vector outputs when feature-level storage is needed. Original files remain source artifacts; this table is the queryable PostGIS state for vector features.
+
+- `id uuid primary key`
+- `dataset_id uuid references datasets(id) on delete cascade`
+- `feature_class text`
+- `source_feature_id text`
+- `properties_json jsonb`
+- `geometry geometry(Geometry, 4326) not null`
+- `created_at timestamptz`
+
+Required indexes:
+
+- `dataset_id`
+- GiST index on `geometry`
+
+### analysis_runs
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `area_id uuid references areas(id)`
+- `dataset_id uuid nullable references datasets(id)`
+- `job_id uuid nullable references jobs(id)`
+- `analysis_type text not null`
+- `status text not null`
+- `model_name text nullable`
+- `model_version text nullable`
+- `parameters_json jsonb not null`
+- `result_json jsonb nullable`
+- `created_at timestamptz`
+- `started_at timestamptz`
+- `finished_at timestamptz`
+- `error_message text`
+
+Analysis runs are domain lifecycle records. Jobs track execution state; analysis runs track reproducibility, model metadata, parameters and result summaries.
+
+### detections
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `dataset_id uuid nullable references datasets(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `job_id uuid nullable references jobs(id)`
+- `model_name text not null`
+- `model_version text nullable`
+- `class_name text not null`
+- `confidence double precision not null`
+- `geometry geometry(Geometry, 4326)`
+- `bbox_json jsonb`
+- `source_tile_path text nullable`
+- `properties_json jsonb`
+- `created_at timestamptz`
+
+Required indexes:
+
+- `project_id`
+- `dataset_id`
+- `analysis_run_id`
+- `class_name`
+- GiST index on `geometry`
+
+Sprint 8 persists detections as first-class PostGIS records. Detections are never stored only in `jobs.result_json`.
+
+### segmentations
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `dataset_id uuid nullable references datasets(id)`
+- `job_id uuid nullable references jobs(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `model_name text not null`
+- `model_version text nullable`
+- `class_name text not null`
+- `confidence double precision nullable`
+- `geometry geometry(MultiPolygon, 4326) not null`
+- `bbox_json jsonb`
+- `area_m2 double precision`
+- `mask_path text`
+- `source_tile_path text`
+- `tile_index integer`
+- `properties_json jsonb`
+- `provenance_json jsonb`
+- `created_at timestamptz`
+
+Required indexes:
+
+- `project_id`
+- `dataset_id`
+- `analysis_run_id`
+- `job_id`
+- `class_name`
+- GiST index on `geometry`
+
+Sprint 9 persists segmentation outputs as first-class PostGIS records. Mask paths are artifact/provenance references only; map display, QA and GeoJSON output use `segmentations.geometry`.
+
+### metrics
+
+- `id uuid primary key`
+- `quality_check_id uuid nullable references quality_checks(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `metric_key text not null`
+- `metric_value double precision`
+- `metric_unit text`
+- `label text`
+- `metadata_json jsonb`
+- `created_at timestamptz`
+
+Metrics may belong to a quality check, an analysis run, or both. Sprint 7A persists QA/QC metrics through `quality_check_id`.
+
+### quality_checks
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `job_id uuid nullable references jobs(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `candidate_dataset_id uuid nullable references datasets(id)`
+- `reference_dataset_id uuid references datasets(id)`
+- `check_type text not null`
+- `status text not null`
+- `score double precision`
+- `parameters_json jsonb`
+- `findings_json jsonb`
+- `created_at timestamptz`
+- `completed_at timestamptz nullable`
+
+Quality checks are domain records. Jobs track execution state; quality checks track the persisted QA/QC result; metrics track individual measurements.
+
+### exports
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `export_type text not null`
+- `storage_path text not null`
+- `metadata_json jsonb`
+- `created_at timestamptz`
+
+## Migration strategy
+
+- Use Alembic.
+- First migration creates extensions and core tables.
+- Second migration adds spatial indexes.
+- Seed script may create a sample project and sample area only if explicitly run.
+
+## Sprint 7B provider-to-dataset mapping
+
+Provider integration is a contract layer only in Sprint 7B. Providers do not write directly to `vector_features`; future provider output must flow through `DatasetService` and `VectorFeatureService` so dataset provenance, storage metadata and feature persistence remain consistent.
+
+- `grb`: maps to `dataset_role='reference'`, `source_name='grb'`.
+- `osm`: maps to `dataset_role='source'` by default, or `dataset_role='reference'` only when explicitly requested; `source_name='osm'`.
+- `manual`: maps to `dataset_role='reference'`, `source_name='manual'`.
+- `fixture`: maps to `dataset_role='reference'`, `source_name='fixture'`.
+
+GRB and OSM live imports are intentionally `not_configured` in Sprint 7B. Manual and fixture reference datasets use existing upload and fixture flows.
+
+## Geometry normalization
+
+- User-drawn polygons arrive as EPSG:4326.
+- Uploaded vector data may arrive in another CRS; preserve original CRS and reproject to EPSG:4326 for storage.
+- Area calculations should use a projected CRS suitable for Belgium, preferably EPSG:31370 or another documented Belgian projection.
+
+## Out of scope for V1
+
+- Raster-in-database storage.
+- Multi-tenant row-level security.
+- User accounts.
+- Full model registry tables.