# Database Implementation Plan Database: PostgreSQL + PostGIS. ## Rules - Store geometries in PostGIS with explicit SRID. - Preserve original CRS metadata even when normalized geometry is stored as EPSG:4326 or a local projected CRS. - Prefer UUID primary keys. - Store large raster/mask/model files in filesystem or object storage; store metadata and paths in PostgreSQL. - Keep analysis outputs reproducible by storing parameters JSON. ## Required extensions ```sql CREATE EXTENSION IF NOT EXISTS postgis; CREATE EXTENSION IF NOT EXISTS postgis_topology; CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; ``` ## Core tables ### projects - `id uuid primary key` - `name text not null` - `description text` - `region text default 'Kempen'` - `status text default 'active'` - `created_at timestamptz` - `updated_at timestamptz` ### areas - `id uuid primary key` - `project_id uuid references projects(id)` - `name text not null` - `geometry geometry(MultiPolygon, 4326) not null` - `original_crs text` - `area_m2 double precision` - `bbox geometry(Polygon, 4326)` - `created_at timestamptz` Spatial index required on `geometry`. ### datasets - `id uuid primary key` - `project_id uuid references projects(id)` - `area_id uuid nullable references areas(id)` - `name text not null` - `dataset_type text not null` - `source text not null` - `storage_path text` - `derived_from_dataset_id uuid nullable references datasets(id)` - `crs text` - `bounds_json jsonb` - `resolution_json jsonb` - `bands_json jsonb` - `metadata_json jsonb` - `status text default 'created'` - `created_at timestamptz` ### vector_features Used for imported vector datasets and derived vector outputs when feature-level storage is needed. Original files remain source artifacts; this table is the queryable PostGIS state for vector features. - `id uuid primary key` - `dataset_id uuid references datasets(id) on delete cascade` - `feature_class text` - `source_feature_id text` - `properties_json jsonb` - `geometry geometry(Geometry, 4326) not null` - `created_at timestamptz` Required indexes: - `dataset_id` - GiST index on `geometry` ### analysis_runs - `id uuid primary key` - `project_id uuid references projects(id)` - `area_id uuid references areas(id)` - `dataset_id uuid nullable references datasets(id)` - `job_id uuid nullable references jobs(id)` - `analysis_type text not null` - `status text not null` - `model_name text nullable` - `model_version text nullable` - `parameters_json jsonb not null` - `result_json jsonb nullable` - `created_at timestamptz` - `started_at timestamptz` - `finished_at timestamptz` - `error_message text` Analysis runs are domain lifecycle records. Jobs track execution state; analysis runs track reproducibility, model metadata, parameters and result summaries. ### detections - `id uuid primary key` - `project_id uuid references projects(id)` - `dataset_id uuid nullable references datasets(id)` - `analysis_run_id uuid nullable references analysis_runs(id)` - `job_id uuid nullable references jobs(id)` - `model_name text not null` - `model_version text nullable` - `class_name text not null` - `confidence double precision not null` - `geometry geometry(Geometry, 4326)` - `bbox_json jsonb` - `source_tile_path text nullable` - `properties_json jsonb` - `created_at timestamptz` Required indexes: - `project_id` - `dataset_id` - `analysis_run_id` - `class_name` - GiST index on `geometry` Sprint 8 persists detections as first-class PostGIS records. Detections are never stored only in `jobs.result_json`. ### segmentations - `id uuid primary key` - `project_id uuid references projects(id)` - `dataset_id uuid nullable references datasets(id)` - `job_id uuid nullable references jobs(id)` - `analysis_run_id uuid nullable references analysis_runs(id)` - `model_name text not null` - `model_version text nullable` - `class_name text not null` - `confidence double precision nullable` - `geometry geometry(MultiPolygon, 4326) not null` - `bbox_json jsonb` - `area_m2 double precision` - `mask_path text` - `source_tile_path text` - `tile_index integer` - `properties_json jsonb` - `provenance_json jsonb` - `created_at timestamptz` Required indexes: - `project_id` - `dataset_id` - `analysis_run_id` - `job_id` - `class_name` - GiST index on `geometry` Sprint 9 persists segmentation outputs as first-class PostGIS records. Mask paths are artifact/provenance references only; map display, QA and GeoJSON output use `segmentations.geometry`. ### metrics - `id uuid primary key` - `quality_check_id uuid nullable references quality_checks(id)` - `analysis_run_id uuid nullable references analysis_runs(id)` - `metric_key text not null` - `metric_value double precision` - `metric_unit text` - `label text` - `metadata_json jsonb` - `created_at timestamptz` Metrics may belong to a quality check, an analysis run, or both. Sprint 7A persists QA/QC metrics through `quality_check_id`. ### quality_checks - `id uuid primary key` - `project_id uuid references projects(id)` - `job_id uuid nullable references jobs(id)` - `analysis_run_id uuid nullable references analysis_runs(id)` - `candidate_dataset_id uuid nullable references datasets(id)` - `reference_dataset_id uuid references datasets(id)` - `check_type text not null` - `status text not null` - `score double precision` - `parameters_json jsonb` - `findings_json jsonb` - `created_at timestamptz` - `completed_at timestamptz nullable` Quality checks are domain records. Jobs track execution state; quality checks track the persisted QA/QC result; metrics track individual measurements. Detection QA coverage and box-to-footprint diagnostics require no schema change. The canonical metric rows remain precision, recall, F1, mean IoU and false-positive/negative counts. Tile coverage population counts and the diagnostic reference-envelope comparison are persisted in the existing `quality_checks.findings_json`; `parameters_json.coverage_policy` records the evaluation policy used for reproducibility. ### detection_reviews - `id uuid primary key` - `project_id uuid references projects(id) on delete cascade` - `quality_check_id uuid references quality_checks(id) on delete cascade` - `analysis_run_id uuid nullable references analysis_runs(id) on delete set null` - `evidence_role text not null` (`false_positive` or `false_negative`) - `evidence_feature_id text not null` - `detection_id uuid nullable references detections(id) on delete set null` - `reference_feature_id uuid nullable references vector_features(id) on delete set null` - `decision text not null default 'unreviewed'` - `notes text nullable` - `reviewed_by text not null default 'operator'` - `created_at timestamptz` - `updated_at timestamptz` The unique key is `(quality_check_id, evidence_role, evidence_feature_id)`. Indexes cover project, quality check, analysis run and decision. Reviews classify persisted QA evidence only; they do not replace or modify Detection, VectorFeature, QualityCheck or Metric records. Unreviewed, reference-gap, imagery-uncertain and QA-alignment cases are not training labels. ### exports - `id uuid primary key` - `project_id uuid references projects(id)` - `analysis_run_id uuid nullable references analysis_runs(id)` - `export_type text not null` - `storage_path text not null` - `metadata_json jsonb` - `created_at timestamptz` ## Migration strategy - Use Alembic. - First migration creates extensions and core tables. - Second migration adds spatial indexes. - Seed script may create a sample project and sample area only if explicitly run. ## Sprint 7B provider-to-dataset mapping Provider integration is a contract layer only in Sprint 7B. Providers do not write directly to `vector_features`; future provider output must flow through `DatasetService` and `VectorFeatureService` so dataset provenance, storage metadata and feature persistence remain consistent. - `grb`: maps to `dataset_role='reference'`, `source_name='grb'`. - `osm`: maps to `dataset_role='source'` by default, or `dataset_role='reference'` only when explicitly requested; `source_name='osm'`. - `manual`: maps to `dataset_role='reference'`, `source_name='manual'`. - `fixture`: maps to `dataset_role='reference'`, `source_name='fixture'`. GRB and OSM live imports are intentionally `not_configured` in Sprint 7B. Manual and fixture reference datasets use existing upload and fixture flows. Large explicit operator imports may invoke DatasetService directly inside the backend container when a single multipart upload would require loading the entire regional artifact into memory. The regional GRB building operator still creates a normal Dataset and DatasetVersion and delegates every queryable row to VectorFeatureService. It copies the immutable combined artifact in a stream, loads one retained municipality partition at a time, flushes bounded feature batches and commits only when the indexed count matches the manifest. It does not expose a direct SQL/provider write path and does not alter the public provider endpoint's `not_configured` status. ### Source freshness audit Source freshness is derived state and does not introduce a scheduler or status table. `Dataset.source_name`, `imported_at`, `observed_at`, `source_version`, checksum/storage metadata and immutable `DatasetVersion` rows remain the persistence source of truth. The project source-freshness service groups those records under explicit publication policies and performs a read-only local evidence audit. External catalogue checks and refresh jobs remain explicit operator actions; they may never overwrite a fixed edition or scenario in place. Regional GRB refreshes add no lifecycle table. A read-only plan compares the official dated edition with the latest Dataset in each governed temporal series. Staging is filesystem evidence only. Checksum-confirmed apply creates a new Dataset plus DatasetVersion and vector_features through the existing DatasetService/VectorFeatureService transaction; prior snapshots remain unchanged and queryable for temporal comparison. ## Geometry normalization - User-drawn polygons arrive as EPSG:4326. - Uploaded vector data may arrive in another CRS; preserve original CRS and reproject to EPSG:4326 for storage. - Area calculations should use a projected CRS suitable for Belgium, preferably EPSG:31370 or another documented Belgian projection. ## Out of scope for V1 - Raster-in-database storage. - Multi-tenant row-level security. - User accounts. - Full model registry tables. ## Governed flood-hazard rasters VMM flood-depth scenarios require no new table. Each acquired coverage is an ordinary `datasets` raster plus immutable `dataset_versions` provenance and a normal synchronous acquisition Job. The GeoTIFF remains filesystem/object storage; PostgreSQL keeps source identity, WCS checksums, EPSG:31370 bounds, source/normalized units, exact Area scope and scenario parameters. Scenario alternatives do not receive a fabricated `observed_at` value and are not grouped as a temporal series. Selection metrics are calculated on demand from the persisted raster. Bathymetry and permanent waterbody volume remain absent from persistence until a separately governed source/model exists. ## Temporal dataset foundation Historical observations remain normal `datasets` and `vector_features`; there is no parallel temporal feature store. Snapshots are grouped by `datasets.temporal_series_key` and carry `observed_at`, `valid_from`, `valid_to`, `temporal_granularity` and `source_version`. Observation time is kept separate from ingestion time (`imported_at`). `dataset_versions` records immutable storage provenance for every upload and derived output: dataset-local `version`, storage path, source version, observation/validity dates, checksum and source/provenance JSON. The `(dataset_id, version)` pair is unique. Temporal series lookup is indexed by `(project_id, temporal_series_key, observed_at)` and source-feature lookup by `(dataset_id, source_feature_id)`. Time-series comparison is read-only and aggregates persisted geometry inside a requested bbox. Object-level added/removed/modified evidence is only valid for sources that explicitly declare stable source feature identifiers. Regional 1778/1873/1969 historical land-use snapshots use the same tables. Their `provenance_metadata` records 28-partition identity, combined output checksum and retained raw-response status. Municipality-clipped source ids are partition-suffixed for uniqueness, while `identity_stable=false` prohibits object-lineage interpretation. No partition or temporal shadow table is added. ### Partitioned vector feature lookup index Regional operator Datasets retain their municipality partition identity in `vector_features.properties_json.municipality`. Migration `202607160001` adds the composite expression index `ix_vector_features_dataset_municipality(dataset_id, properties_json->>'municipality')`. It supports exact preclipped municipality selection without changing the canonical `vector_features` schema or introducing operator-specific tables. Free rectangle queries continue to use the geometry GiST index.