geointel/docs/DATABASE_IMPLEMENTATION_PLAN.md

Database Implementation Plan

Database: PostgreSQL + PostGIS.

Bathymetry regional completeness requires no new table. Every VHA municipality partition remains an ordinary datasets row plus vector_features. After a complete server-validated manifest, Dataset and DatasetVersion metadata retain partition_scope_key, partition counts, the manifest SHA-256, partitioned_source_audit=true and regional_partitions_complete=true. Empty source partitions are recorded in the manifest provenance and never represented by fabricated features.

Rules

• Store geometries in PostGIS with explicit SRID.
• Preserve original CRS metadata even when normalized geometry is stored as EPSG:4326 or a local projected CRS.
• Prefer UUID primary keys.
• Store large raster/mask/model files in filesystem or object storage; store metadata and paths in PostgreSQL.
• Keep analysis outputs reproducible by storing parameters JSON.

Required extensions

CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS postgis_topology;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

Core tables

projects

• id uuid primary key
• name text not null
• description text
• region text default 'Kempen'
• status text default 'active'
• created_at timestamptz
• updated_at timestamptz

areas

• id uuid primary key
• project_id uuid references projects(id)
• name text not null
• geometry geometry(MultiPolygon, 4326) not null
• original_crs text
• area_m2 double precision
• bbox geometry(Polygon, 4326)
• created_at timestamptz

Spatial index required on geometry.

datasets

• id uuid primary key
• project_id uuid references projects(id)
• area_id uuid nullable references areas(id)
• name text not null
• dataset_type text not null
• source text not null
• storage_path text
• derived_from_dataset_id uuid nullable references datasets(id)
• crs text
• bounds_json jsonb
• resolution_json jsonb
• bands_json jsonb
• metadata_json jsonb
• status text default 'created'
• created_at timestamptz

vector_features

Used for imported vector datasets and derived vector outputs when feature-level storage is needed. Original files remain source artifacts; this table is the queryable PostGIS state for vector features.

• id uuid primary key
• dataset_id uuid references datasets(id) on delete cascade
• feature_class text
• source_feature_id text
• properties_json jsonb
• geometry geometry(Geometry, 4326) not null
• created_at timestamptz

Required indexes:

• dataset_id
• GiST index on geometry

analysis_runs

• id uuid primary key
• project_id uuid references projects(id)
• area_id uuid references areas(id)
• dataset_id uuid nullable references datasets(id)
• job_id uuid nullable references jobs(id)
• analysis_type text not null
• status text not null
• model_name text nullable
• model_version text nullable
• parameters_json jsonb not null
• result_json jsonb nullable
• created_at timestamptz
• started_at timestamptz
• finished_at timestamptz
• error_message text

Analysis runs are domain lifecycle records. Jobs track execution state; analysis runs track reproducibility, model metadata, parameters and result summaries.

detections

• id uuid primary key
• project_id uuid references projects(id)
• dataset_id uuid nullable references datasets(id)
• analysis_run_id uuid nullable references analysis_runs(id)
• job_id uuid nullable references jobs(id)
• model_name text not null
• model_version text nullable
• class_name text not null
• confidence double precision not null
• geometry geometry(Geometry, 4326)
• bbox_json jsonb
• source_tile_path text nullable
• properties_json jsonb
• created_at timestamptz

Required indexes:

• project_id
• dataset_id
• analysis_run_id
• class_name
• GiST index on geometry

Sprint 8 persists detections as first-class PostGIS records. Detections are never stored only in jobs.result_json.

segmentations

• id uuid primary key
• project_id uuid references projects(id)
• dataset_id uuid nullable references datasets(id)
• job_id uuid nullable references jobs(id)
• analysis_run_id uuid nullable references analysis_runs(id)
• model_name text not null
• model_version text nullable
• class_name text not null
• confidence double precision nullable
• geometry geometry(MultiPolygon, 4326) not null
• bbox_json jsonb
• area_m2 double precision
• mask_path text
• source_tile_path text
• tile_index integer
• properties_json jsonb
• provenance_json jsonb
• created_at timestamptz

Required indexes:

• project_id
• dataset_id
• analysis_run_id
• job_id
• class_name
• GiST index on geometry

Sprint 9 persists segmentation outputs as first-class PostGIS records. Mask paths are artifact/provenance references only; map display, QA and GeoJSON output use segmentations.geometry.

metrics

• id uuid primary key
• quality_check_id uuid nullable references quality_checks(id)
• analysis_run_id uuid nullable references analysis_runs(id)
• metric_key text not null
• metric_value double precision
• metric_unit text
• label text
• metadata_json jsonb
• created_at timestamptz

Metrics may belong to a quality check, an analysis run, or both. Sprint 7A persists QA/QC metrics through quality_check_id.

quality_checks

• id uuid primary key
• project_id uuid references projects(id)
• job_id uuid nullable references jobs(id)
• analysis_run_id uuid nullable references analysis_runs(id)
• candidate_dataset_id uuid nullable references datasets(id)
• reference_dataset_id uuid references datasets(id)
• check_type text not null
• status text not null
• score double precision
• parameters_json jsonb
• findings_json jsonb
• created_at timestamptz
• completed_at timestamptz nullable

Quality checks are domain records. Jobs track execution state; quality checks track the persisted QA/QC result; metrics track individual measurements.

Detection QA coverage and box-to-footprint diagnostics require no schema change. The canonical metric rows remain precision, recall, F1, mean IoU and false-positive/negative counts. Tile coverage population counts and the diagnostic reference-envelope comparison are persisted in the existing quality_checks.findings_json; parameters_json.coverage_policy records the evaluation policy used for reproducibility.

detection_reviews

• id uuid primary key
• project_id uuid references projects(id) on delete cascade
• quality_check_id uuid references quality_checks(id) on delete cascade
• analysis_run_id uuid nullable references analysis_runs(id) on delete set null
• evidence_role text not null (false_positive or false_negative)
• evidence_feature_id text not null
• detection_id uuid nullable references detections(id) on delete set null
• reference_feature_id uuid nullable references vector_features(id) on delete set null
• decision text not null default 'unreviewed'
• notes text nullable
• reviewed_by text not null default 'operator'
• created_at timestamptz
• updated_at timestamptz

The unique key is (quality_check_id, evidence_role, evidence_feature_id). Indexes cover project, quality check, analysis run and decision. Reviews classify persisted QA evidence only; they do not replace or modify Detection, VectorFeature, QualityCheck or Metric records. Unreviewed, reference-gap, imagery-uncertain and QA-alignment cases are not training labels.

exports

• id uuid primary key
• project_id uuid references projects(id)
• analysis_run_id uuid nullable references analysis_runs(id)
• export_type text not null
• storage_path text not null
• metadata_json jsonb
• created_at timestamptz

Migration strategy

• Use Alembic.
• First migration creates extensions and core tables.
• Second migration adds spatial indexes.
• Seed script may create a sample project and sample area only if explicitly run.

Sprint 7B provider-to-dataset mapping

Provider integration is a contract layer only in Sprint 7B. Providers do not write directly to vector_features; future provider output must flow through DatasetService and VectorFeatureService so dataset provenance, storage metadata and feature persistence remain consistent.

• grb: maps to dataset_role='reference', source_name='grb'.
• osm: maps to dataset_role='source' by default, or dataset_role='reference' only when explicitly requested; source_name='osm'.
• manual: maps to dataset_role='reference', source_name='manual'.
• fixture: maps to dataset_role='reference', source_name='fixture'.

GRB and OSM live imports are intentionally not_configured in Sprint 7B. Manual and fixture reference datasets use existing upload and fixture flows.

Large explicit operator imports may invoke DatasetService directly inside the backend container when a single multipart upload would require loading the entire regional artifact into memory. The regional GRB building operator still creates a normal Dataset and DatasetVersion and delegates every queryable row to VectorFeatureService. It copies the immutable combined artifact in a stream, loads one retained municipality partition at a time, flushes bounded feature batches and commits only when the indexed count matches the manifest. It does not expose a direct SQL/provider write path and does not alter the public provider endpoint's not_configured status.

Source freshness audit

Source freshness is derived state and does not introduce a scheduler or status table. Dataset.source_name, imported_at, observed_at, source_version, checksum/storage metadata and immutable DatasetVersion rows remain the persistence source of truth. The project source-freshness service groups those records under explicit publication policies and performs a read-only local evidence audit. External catalogue checks and refresh jobs remain explicit operator actions; they may never overwrite a fixed edition or scenario in place.

Regional GRB refreshes add no lifecycle table. A read-only plan compares the official dated edition with the latest Dataset in each governed temporal series. Staging is filesystem evidence only. Checksum-confirmed apply creates a new Dataset plus DatasetVersion and vector_features through the existing DatasetService/VectorFeatureService transaction; prior snapshots remain unchanged and queryable for temporal comparison.

Statbel population release management likewise adds no lifecycle table. Planning reads the existing source-catalog response. Stage, named review and their SHA-256-bound evidence are filesystem-only operator artifacts. Apply revalidates those artifacts and invokes the existing Dataset upload service, which creates the ordinary annual datasets, dataset_versions and vector_features records in one established persistence flow. An existing year remains idempotent and previous annual snapshots are never updated or deleted.

Definitive ALZ release management uses the same persistence boundary and adds no migration or release table. Catalog planning, staged archive/GeoJSON, crop-code evidence and named review live on the filesystem. Only an approved apply invokes the existing Dataset upload service, producing the ordinary annual Dataset, DatasetVersion and vector_features records with source_version=<year>-definitive. Earlier annual snapshots are retained and provisional v1/v2 publications cannot create rows.

Current-orthophoto release management likewise adds no lifecycle table or migration. Preflight is read-only; stage and named review are filesystem-only. Approved apply uploads the exact checksummed EPSG:31370 GeoTIFF through the existing dataset endpoint and DatasetService transaction, creating one normal immutable raster Dataset and DatasetVersion with official YYYY.NN source_version, temporal flight-date evidence and release provenance. Official editions take precedence over legacy rolling markers in catalog comparison, but direct metadata backfill, update or deletion of those legacy rows remains prohibited.

Geometry normalization

• User-drawn polygons arrive as EPSG:4326.
• Uploaded vector data may arrive in another CRS; preserve original CRS and reproject to EPSG:4326 for storage.
• Area calculations should use a projected CRS suitable for Belgium, preferably EPSG:31370 or another documented Belgian projection.

Out of scope for V1

• Raster-in-database storage.
• Multi-tenant row-level security.
• User accounts.
• Full model registry tables.

Governed flood-hazard rasters

VMM flood-depth scenarios require no new table. Each acquired coverage is an ordinary datasets raster plus immutable dataset_versions provenance and a normal synchronous acquisition Job. The GeoTIFF remains filesystem/object storage; PostgreSQL keeps source identity, WCS checksums, EPSG:31370 bounds, source/normalized units, exact Area scope and scenario parameters.

Scenario alternatives do not receive a fabricated observed_at value and are not grouped as a temporal series. Selection metrics are calculated on demand from the persisted raster. Bathymetry and permanent waterbody volume remain absent from persistence until a separately governed source/model exists.

Temporal dataset foundation

Historical observations remain normal datasets and vector_features; there is no parallel temporal feature store. Snapshots are grouped by datasets.temporal_series_key and carry observed_at, valid_from, valid_to, temporal_granularity and source_version. Observation time is kept separate from ingestion time (imported_at).

dataset_versions records immutable storage provenance for every upload and derived output: dataset-local version, storage path, source version, observation/validity dates, checksum and source/provenance JSON. The (dataset_id, version) pair is unique. Temporal series lookup is indexed by (project_id, temporal_series_key, observed_at) and source-feature lookup by (dataset_id, source_feature_id).

Time-series comparison is read-only and aggregates persisted geometry inside a requested bbox. Object-level added/removed/modified evidence is only valid for sources that explicitly declare stable source feature identifiers.

Regional 1778/1873/1969 historical land-use snapshots use the same tables. Their provenance_metadata records 28-partition identity, combined output checksum and retained raw-response status. Municipality-clipped source ids are partition-suffixed for uniqueness, while identity_stable=false prohibits object-lineage interpretation. No partition or temporal shadow table is added.

Partitioned vector feature lookup index

Regional operator Datasets retain their municipality partition identity in vector_features.properties_json.municipality. Migration 202607160001 adds the composite expression index ix_vector_features_dataset_municipality(dataset_id, properties_json->>'municipality'). It supports exact preclipped municipality selection without changing the canonical vector_features schema or introducing operator-specific tables. Free rectangle queries continue to use the geometry GiST index.

Bathymetry profile persistence

VHA profile points require no new table or migration. The immutable GeoJSON artifact is stored as one normal reference datasets row and dataset_versions row; each exact point is a normal vector_features row with EPSG:4326 geometry and source properties. Existing dataset, source-feature and GiST indexes support selection.

Profile measurement dates remain feature properties because a bounded Dataset can contain many historical campaigns. No artificial Dataset observed_at or temporal series is assigned. A future bed raster remains file/object storage plus Dataset metadata, not raster-in-database storage. Any national/maritime extension keeps source vertical datum, survey epoch and Area partition explicit and does not create a provider-specific shadow schema.