geointel/docs/DATABASE_IMPLEMENTATION_PLAN.md

# Database Implementation Plan

Database: PostgreSQL + PostGIS.

Bathymetry regional completeness requires no new table. Every VHA municipality
partition remains an ordinary `datasets` row plus `vector_features`. After a
complete server-validated manifest, Dataset and DatasetVersion metadata retain
`partition_scope_key`, partition counts, the manifest SHA-256,
`partitioned_source_audit=true` and
`regional_partitions_complete=true`. Empty source partitions are recorded in
the manifest provenance and never represented by fabricated features.

## Rules

- Store geometries in PostGIS with explicit SRID.
- Preserve original CRS metadata even when normalized geometry is stored as EPSG:4326 or a local projected CRS.
- Prefer UUID primary keys.
- Store large raster/mask/model files in filesystem or object storage; store metadata and paths in PostgreSQL.
- Keep analysis outputs reproducible by storing parameters JSON.

## Required extensions

```sql
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS postgis_topology;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
```

## Core tables

### projects

- `id uuid primary key`
- `name text not null`
- `description text`
- `region text default 'Kempen'`
- `status text default 'active'`
- `created_at timestamptz`
- `updated_at timestamptz`

### areas

- `id uuid primary key`
- `project_id uuid references projects(id)`
- `name text not null`
- `geometry geometry(MultiPolygon, 4326) not null`
- `original_crs text`
- `area_m2 double precision`
- `bbox geometry(Polygon, 4326)`
- `created_at timestamptz`

Spatial index required on `geometry`.

### datasets

- `id uuid primary key`
- `project_id uuid references projects(id)`
- `area_id uuid nullable references areas(id)`
- `name text not null`
- `dataset_type text not null`
- `source text not null`
- `storage_path text`
- `derived_from_dataset_id uuid nullable references datasets(id)`
- `crs text`
- `bounds_json jsonb`
- `resolution_json jsonb`
- `bands_json jsonb`
- `metadata_json jsonb`
- `status text default 'created'`
- `created_at timestamptz`

### vector_features

Used for imported vector datasets and derived vector outputs when feature-level storage is needed. Original files remain source artifacts; this table is the queryable PostGIS state for vector features.

- `id uuid primary key`
- `dataset_id uuid references datasets(id) on delete cascade`
- `feature_class text`
- `source_feature_id text`
- `properties_json jsonb`
- `geometry geometry(Geometry, 4326) not null`
- `created_at timestamptz`

Required indexes:

- `dataset_id`
- GiST index on `geometry`

### analysis_runs

- `id uuid primary key`
- `project_id uuid references projects(id)`
- `area_id uuid references areas(id)`
- `dataset_id uuid nullable references datasets(id)`
- `job_id uuid nullable references jobs(id)`
- `analysis_type text not null`
- `status text not null`
- `model_name text nullable`
- `model_version text nullable`
- `parameters_json jsonb not null`
- `result_json jsonb nullable`
- `created_at timestamptz`
- `started_at timestamptz`
- `finished_at timestamptz`
- `error_message text`

Analysis runs are domain lifecycle records. Jobs track execution state; analysis runs track reproducibility, model metadata, parameters and result summaries.

### detections

- `id uuid primary key`
- `project_id uuid references projects(id)`
- `dataset_id uuid nullable references datasets(id)`
- `analysis_run_id uuid nullable references analysis_runs(id)`
- `job_id uuid nullable references jobs(id)`
- `model_name text not null`
- `model_version text nullable`
- `class_name text not null`
- `confidence double precision not null`
- `geometry geometry(Geometry, 4326)`
- `bbox_json jsonb`
- `source_tile_path text nullable`
- `properties_json jsonb`
- `created_at timestamptz`

Required indexes:

- `project_id`
- `dataset_id`
- `analysis_run_id`
- `class_name`
- GiST index on `geometry`

Sprint 8 persists detections as first-class PostGIS records. Detections are never stored only in `jobs.result_json`.

### segmentations

- `id uuid primary key`
- `project_id uuid references projects(id)`
- `dataset_id uuid nullable references datasets(id)`
- `job_id uuid nullable references jobs(id)`
- `analysis_run_id uuid nullable references analysis_runs(id)`
- `model_name text not null`
- `model_version text nullable`
- `class_name text not null`
- `confidence double precision nullable`
- `geometry geometry(MultiPolygon, 4326) not null`
- `bbox_json jsonb`
- `area_m2 double precision`
- `mask_path text`
- `source_tile_path text`
- `tile_index integer`
- `properties_json jsonb`
- `provenance_json jsonb`
- `created_at timestamptz`

Required indexes:

- `project_id`
- `dataset_id`
- `analysis_run_id`
- `job_id`
- `class_name`
- GiST index on `geometry`

Sprint 9 persists segmentation outputs as first-class PostGIS records. Mask paths are artifact/provenance references only; map display, QA and GeoJSON output use `segmentations.geometry`.

### metrics

- `id uuid primary key`
- `quality_check_id uuid nullable references quality_checks(id)`
- `analysis_run_id uuid nullable references analysis_runs(id)`
- `metric_key text not null`
- `metric_value double precision`
- `metric_unit text`
- `label text`
- `metadata_json jsonb`
- `created_at timestamptz`

Metrics may belong to a quality check, an analysis run, or both. Sprint 7A persists QA/QC metrics through `quality_check_id`.

### quality_checks

- `id uuid primary key`
- `project_id uuid references projects(id)`
- `job_id uuid nullable references jobs(id)`
- `analysis_run_id uuid nullable references analysis_runs(id)`
- `candidate_dataset_id uuid nullable references datasets(id)`
- `reference_dataset_id uuid references datasets(id)`
- `check_type text not null`
- `status text not null`
- `score double precision`
- `parameters_json jsonb`
- `findings_json jsonb`
- `created_at timestamptz`
- `completed_at timestamptz nullable`

Quality checks are domain records. Jobs track execution state; quality checks track the persisted QA/QC result; metrics track individual measurements.

Detection QA coverage and box-to-footprint diagnostics require no schema
change. The canonical metric rows remain precision, recall, F1, mean IoU and
false-positive/negative counts. Tile coverage population counts and the
diagnostic reference-envelope comparison are persisted in the existing
`quality_checks.findings_json`; `parameters_json.coverage_policy` records the
evaluation policy used for reproducibility.

### detection_reviews

- `id uuid primary key`
- `project_id uuid references projects(id) on delete cascade`
- `quality_check_id uuid references quality_checks(id) on delete cascade`
- `analysis_run_id uuid nullable references analysis_runs(id) on delete set null`
- `evidence_role text not null` (`false_positive` or `false_negative`)
- `evidence_feature_id text not null`
- `detection_id uuid nullable references detections(id) on delete set null`
- `reference_feature_id uuid nullable references vector_features(id) on delete set null`
- `decision text not null default 'unreviewed'`
- `notes text nullable`
- `reviewed_by text not null default 'operator'`
- `created_at timestamptz`
- `updated_at timestamptz`

The unique key is `(quality_check_id, evidence_role, evidence_feature_id)`.
Indexes cover project, quality check, analysis run and decision. Reviews
classify persisted QA evidence only; they do not replace or modify Detection,
VectorFeature, QualityCheck or Metric records. Unreviewed, reference-gap,
imagery-uncertain and QA-alignment cases are not training labels.

### exports

- `id uuid primary key`
- `project_id uuid references projects(id)`
- `analysis_run_id uuid nullable references analysis_runs(id)`
- `export_type text not null`
- `storage_path text not null`
- `metadata_json jsonb`
- `created_at timestamptz`

## Migration strategy

- Use Alembic.
- First migration creates extensions and core tables.
- Second migration adds spatial indexes.
- Seed script may create a sample project and sample area only if explicitly run.

## Sprint 7B provider-to-dataset mapping

Provider integration is a contract layer only in Sprint 7B. Providers do not write directly to `vector_features`; future provider output must flow through `DatasetService` and `VectorFeatureService` so dataset provenance, storage metadata and feature persistence remain consistent.

- `grb`: maps to `dataset_role='reference'`, `source_name='grb'`.
- `osm`: maps to `dataset_role='source'` by default, or `dataset_role='reference'` only when explicitly requested; `source_name='osm'`.
- `manual`: maps to `dataset_role='reference'`, `source_name='manual'`.
- `fixture`: maps to `dataset_role='reference'`, `source_name='fixture'`.

GRB and OSM live imports are intentionally `not_configured` in Sprint 7B. Manual and fixture reference datasets use existing upload and fixture flows.

Large explicit operator imports may invoke DatasetService directly inside the
backend container when a single multipart upload would require loading the
entire regional artifact into memory. The regional GRB building operator still
creates a normal Dataset and DatasetVersion and delegates every queryable row
to VectorFeatureService. It copies the immutable combined artifact in a
stream, loads one retained municipality partition at a time, flushes bounded
feature batches and commits only when the indexed count matches the manifest.
It does not expose a direct SQL/provider write path and does not alter the
public provider endpoint's `not_configured` status.

### Source freshness audit

Source freshness is derived state and does not introduce a scheduler or status
table. `Dataset.source_name`, `imported_at`, `observed_at`, `source_version`,
checksum/storage metadata and immutable `DatasetVersion` rows remain the
persistence source of truth. The project source-freshness service groups those
records under explicit publication policies and performs a read-only local
evidence audit. External catalogue checks and refresh jobs remain explicit
operator actions; they may never overwrite a fixed edition or scenario in
place.

Regional GRB refreshes add no lifecycle table. A read-only plan compares the
official dated edition with the latest Dataset in each governed temporal
series. Staging is filesystem evidence only. Checksum-confirmed apply creates a
new Dataset plus DatasetVersion and vector_features through the existing
DatasetService/VectorFeatureService transaction; prior snapshots remain
unchanged and queryable for temporal comparison.

Statbel population release management likewise adds no lifecycle table.
Planning reads the existing source-catalog response. Stage, named review and
their SHA-256-bound evidence are filesystem-only operator artifacts. Apply
revalidates those artifacts and invokes the existing Dataset upload service,
which creates the ordinary annual `datasets`, `dataset_versions` and
`vector_features` records in one established persistence flow. An existing
year remains idempotent and previous annual snapshots are never updated or
deleted.

SPW bathymetry also requires no migration. The immutable official ZIP and
bounded operator COG are filesystem artifacts. A successful API upload creates
an ordinary raster `Dataset` and `DatasetVersion`; raster values stay in the
versioned GeoTIFF rather than PostGIS. `source_metadata` retains EPSG:3812,
mDNG, survey period, source checksum and bounded extent, while
`provenance_metadata` retains the official URL, archive member, derived
checksum and selection geometry/hash. Analysis never writes metric rows and
never converts the vertical datum implicitly.

Definitive ALZ release management uses the same persistence boundary and adds
no migration or release table. Catalog planning, staged archive/GeoJSON,
crop-code evidence and named review live on the filesystem. Only an approved
apply invokes the existing Dataset upload service, producing the ordinary
annual Dataset, DatasetVersion and vector_features records with
`source_version=<year>-definitive`. Earlier annual snapshots are retained and
provisional v1/v2 publications cannot create rows.

Current-orthophoto release management likewise adds no lifecycle table or
migration. Preflight is read-only; stage and named review are filesystem-only.
Approved apply uploads the exact checksummed EPSG:31370 GeoTIFF through the
existing dataset endpoint and DatasetService transaction, creating one normal
immutable raster Dataset and DatasetVersion with official `YYYY.NN`
`source_version`, temporal flight-date evidence and release provenance.
Official editions take precedence over legacy rolling markers in catalog
comparison, but direct metadata backfill, update or deletion of those legacy
rows remains prohibited.

## Geometry normalization

- User-drawn polygons arrive as EPSG:4326.
- Uploaded vector data may arrive in another CRS; preserve original CRS and reproject to EPSG:4326 for storage.
- Area calculations should use a projected CRS suitable for Belgium, preferably EPSG:31370 or another documented Belgian projection.

## Out of scope for V1

- Raster-in-database storage.
- Multi-tenant row-level security.
- User accounts.
- Full model registry tables.

## Governed flood-hazard rasters

VMM flood-depth scenarios require no new table. Each acquired coverage is an
ordinary `datasets` raster plus immutable `dataset_versions` provenance and a
normal synchronous acquisition Job. The GeoTIFF remains filesystem/object
storage; PostgreSQL keeps source identity, WCS checksums, EPSG:31370 bounds,
source/normalized units, exact Area scope and scenario parameters.

Scenario alternatives do not receive a fabricated `observed_at` value and are
not grouped as a temporal series. Selection metrics are calculated on demand
from the persisted raster. Bathymetry and permanent waterbody volume remain
absent from persistence until a separately governed source/model exists.

## Temporal dataset foundation

Historical observations remain normal `datasets` and `vector_features`; there
is no parallel temporal feature store. Snapshots are grouped by
`datasets.temporal_series_key` and carry `observed_at`, `valid_from`,
`valid_to`, `temporal_granularity` and `source_version`. Observation time is
kept separate from ingestion time (`imported_at`).

`dataset_versions` records immutable storage provenance for every upload and
derived output: dataset-local `version`, storage path, source version,
observation/validity dates, checksum and source/provenance JSON. The
`(dataset_id, version)` pair is unique. Temporal series lookup is indexed by
`(project_id, temporal_series_key, observed_at)` and source-feature lookup by
`(dataset_id, source_feature_id)`.

Time-series comparison is read-only and aggregates persisted geometry inside a
requested bbox. Object-level added/removed/modified evidence is only valid for
sources that explicitly declare stable source feature identifiers.

Regional 1778/1873/1969 historical land-use snapshots use the same tables.
Their `provenance_metadata` records 28-partition identity, combined output
checksum and retained raw-response status. Municipality-clipped source ids are
partition-suffixed for uniqueness, while `identity_stable=false` prohibits
object-lineage interpretation. No partition or temporal shadow table is added.
### Partitioned vector feature lookup index

Regional operator Datasets retain their municipality partition identity in
`vector_features.properties_json.municipality`. Migration `202607160001` adds
the composite expression index
`ix_vector_features_dataset_municipality(dataset_id, properties_json->>'municipality')`.
It supports exact preclipped municipality selection without changing the
canonical `vector_features` schema or introducing operator-specific tables.
Free rectangle queries continue to use the geometry GiST index.

## Bathymetry profile persistence

VHA profile points require no new table or migration. The immutable GeoJSON
artifact is stored as one normal reference `datasets` row and
`dataset_versions` row; each exact point is a normal `vector_features` row with
EPSG:4326 geometry and source properties. Existing dataset, source-feature and
GiST indexes support selection.

Profile measurement dates remain feature properties because a bounded Dataset
can contain many historical campaigns. No artificial Dataset `observed_at` or
temporal series is assigned. A future bed raster remains file/object storage
plus Dataset metadata, not raster-in-database storage. Any national/maritime
extension keeps source vertical datum, survey epoch and Area partition
explicit and does not create a provider-specific shadow schema.