Initial public release

docs/DATABASE_IMPLEMENTATION_PLAN.md

@@ -0,0 +1,445 @@
+# Database Implementation Plan
+
+Database: PostgreSQL + PostGIS.
+
+Bathymetry regional completeness requires no new table. Every VHA municipality
+partition remains an ordinary `datasets` row plus `vector_features`. After a
+complete server-validated manifest, Dataset and DatasetVersion metadata retain
+`partition_scope_key`, partition counts, the manifest SHA-256,
+`partitioned_source_audit=true` and
+`regional_partitions_complete=true`. Empty source partitions are recorded in
+the manifest provenance and never represented by fabricated features.
+
+## Rules
+
+- Store geometries in PostGIS with explicit SRID.
+- Preserve original CRS metadata even when normalized geometry is stored as EPSG:4326 or a local projected CRS.
+- Prefer UUID primary keys.
+- Store large raster/mask/model files in filesystem or object storage; store metadata and paths in PostgreSQL.
+- Keep analysis outputs reproducible by storing parameters JSON.
+
+## Required extensions
+
+```sql
+CREATE EXTENSION IF NOT EXISTS postgis;
+CREATE EXTENSION IF NOT EXISTS postgis_topology;
+CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
+```
+
+## Core tables
+
+### projects
+
+- `id uuid primary key`
+- `name text not null`
+- `description text`
+- `region text default 'Kempen'`
+- `status text default 'active'`
+- `created_at timestamptz`
+- `updated_at timestamptz`
+
+### areas
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `name text not null`
+- `geometry geometry(MultiPolygon, 4326) not null`
+- `original_crs text`
+- `area_m2 double precision`
+- `bbox geometry(Polygon, 4326)`
+- `created_at timestamptz`
+
+Spatial index required on `geometry`.
+
+### datasets
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `area_id uuid nullable references areas(id)`
+- `name text not null`
+- `dataset_type text not null`
+- `source text not null`
+- `storage_path text`
+- `derived_from_dataset_id uuid nullable references datasets(id)`
+- `crs text`
+- `bounds_json jsonb`
+- `resolution_json jsonb`
+- `bands_json jsonb`
+- `metadata_json jsonb`
+- `status text default 'created'`
+- `created_at timestamptz`
+
+### Source registry, snapshots and quarantine (Accuracy Phase 2)
+
+Migration `202608010001_source_registry_provenance` adds a server-owned
+provenance boundary without deleting or inventing facts for existing rows.
+
+`source_registry` stores one governed source definition per `source_key`:
+
+- classification (`authoritative`, `corroborative`, `contextual`, `derived`,
+  `experimental`), authority/scope and provider adapter key;
+- licence, usage restrictions, expected CRS/units/resolution, temporal and
+  geographic coverage, expected geometry/attributes and known limitations;
+- usage policy, freshness and ingest status.
+
+`source_snapshots` stores immutable acquired editions keyed by
+`(source_registry_id, snapshot_key)`, including source version/snapshot/fetch
+time, lowercase SHA-256 checksum, CRS/units/resolution, coverage, observed
+schema, freshness, ingest status and raw snapshot metadata.
+
+`datasets` and `dataset_versions` receive `source_registry_id`,
+`source_snapshot_id`, data-contract key/version, validation report/status,
+provenance/lineage status and idempotent `ingest_key`. New governed records
+must have a passing report before they become `ready`. Legacy rows are retained
+with explicit incomplete/not-validated state; a migration never promotes them
+based on a historical `source_name` string.
+
+`dataset_lineage_edges` records parent/child Dataset(+Version), transform name
+and version, parameters and input/output hashes. `dataset_quarantines` retains
+the rejected dataset/version/snapshot, stage, reason, validation evidence and
+artifact location. Quarantine is a stateful preservation record, not a delete.
+
+The database constraints enforce accepted status vocabularies, nonblank ingest
+keys, immutable snapshot identity/checksum format, unique ingest idempotency
+keys and non-self lineage edges. Service-level contract validation remains
+responsible for CRS, topology, bbox, units and source-task semantics.
+
+### vector_features
+
+Used for imported vector datasets and derived vector outputs when feature-level storage is needed. Original files remain source artifacts; this table is the queryable PostGIS state for vector features.
+
+- `id uuid primary key`
+- `dataset_id uuid references datasets(id) on delete cascade`
+- `feature_class text`
+- `source_feature_id text`
+- `properties_json jsonb`
+- `geometry geometry(Geometry, 4326) not null`
+- `created_at timestamptz`
+
+Required indexes:
+
+- `dataset_id`
+- GiST index on `geometry`
+
+### analysis_runs
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `area_id uuid references areas(id)`
+- `dataset_id uuid nullable references datasets(id)`
+- `job_id uuid nullable references jobs(id)`
+- `analysis_type text not null`
+- `status text not null`
+- `model_name text nullable`
+- `model_version text nullable`
+- `parameters_json jsonb not null`
+- `result_json jsonb nullable`
+- `created_at timestamptz`
+- `started_at timestamptz`
+- `finished_at timestamptz`
+- `error_message text`
+
+Analysis runs are domain lifecycle records. Jobs track execution state; analysis runs track reproducibility, model metadata, parameters and result summaries.
+
+### detections
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `dataset_id uuid nullable references datasets(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `job_id uuid nullable references jobs(id)`
+- `model_name text not null`
+- `model_version text nullable`
+- `class_name text not null`
+- `confidence double precision not null`
+- `geometry geometry(Geometry, 4326)`
+- `bbox_json jsonb`
+- `source_tile_path text nullable`
+- `properties_json jsonb`
+- `created_at timestamptz`
+
+Required indexes:
+
+- `project_id`
+- `dataset_id`
+- `analysis_run_id`
+- `class_name`
+- GiST index on `geometry`
+
+Sprint 8 persists detections as first-class PostGIS records. Detections are never stored only in `jobs.result_json`.
+
+### segmentations
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `dataset_id uuid nullable references datasets(id)`
+- `job_id uuid nullable references jobs(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `model_name text not null`
+- `model_version text nullable`
+- `class_name text not null`
+- `confidence double precision nullable`
+- `geometry geometry(MultiPolygon, 4326) not null`
+- `bbox_json jsonb`
+- `area_m2 double precision`
+- `mask_path text`
+- `source_tile_path text`
+- `tile_index integer`
+- `properties_json jsonb`
+- `provenance_json jsonb`
+- `created_at timestamptz`
+
+Required indexes:
+
+- `project_id`
+- `dataset_id`
+- `analysis_run_id`
+- `job_id`
+- `class_name`
+- GiST index on `geometry`
+
+Sprint 9 persists segmentation outputs as first-class PostGIS records. Mask paths are artifact/provenance references only; map display, QA and GeoJSON output use `segmentations.geometry`.
+
+### metrics
+
+- `id uuid primary key`
+- `quality_check_id uuid nullable references quality_checks(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `metric_key text not null`
+- `metric_value double precision`
+- `metric_unit text`
+- `label text`
+- `metadata_json jsonb`
+- `created_at timestamptz`
+
+Metrics may belong to a quality check, an analysis run, or both. Sprint 7A persists QA/QC metrics through `quality_check_id`.
+
+### quality_checks
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `job_id uuid nullable references jobs(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `candidate_dataset_id uuid nullable references datasets(id)`
+- `reference_dataset_id uuid references datasets(id)`
+- `check_type text not null`
+- `status text not null`
+- `score double precision`
+- `parameters_json jsonb`
+- `findings_json jsonb`
+- `created_at timestamptz`
+- `completed_at timestamptz nullable`
+
+Quality checks are domain records. Jobs track execution state; quality checks track the persisted QA/QC result; metrics track individual measurements.
+
+Detection QA coverage and box-to-footprint diagnostics require no schema
+change. The canonical metric rows remain precision, recall, F1, mean IoU and
+false-positive/negative counts. Tile coverage population counts and the
+diagnostic reference-envelope comparison are persisted in the existing
+`quality_checks.findings_json`; `parameters_json.coverage_policy` records the
+evaluation policy used for reproducibility.
+
+### detection_reviews
+
+- `id uuid primary key`
+- `project_id uuid references projects(id) on delete cascade`
+- `quality_check_id uuid references quality_checks(id) on delete cascade`
+- `analysis_run_id uuid nullable references analysis_runs(id) on delete set null`
+- `evidence_role text not null` (`false_positive` or `false_negative`)
+- `evidence_feature_id text not null`
+- `detection_id uuid nullable references detections(id) on delete set null`
+- `reference_feature_id uuid nullable references vector_features(id) on delete set null`
+- `decision text not null default 'unreviewed'`
+- `notes text nullable`
+- `reviewed_by text not null default 'operator'`
+- `created_at timestamptz`
+- `updated_at timestamptz`
+
+The unique key is `(quality_check_id, evidence_role, evidence_feature_id)`.
+Indexes cover project, quality check, analysis run and decision. Reviews
+classify persisted QA evidence only; they do not replace or modify Detection,
+VectorFeature, QualityCheck or Metric records. Unreviewed, reference-gap,
+imagery-uncertain and QA-alignment cases are not training labels.
+
+### exports
+
+- `id uuid primary key`
+- `project_id uuid references projects(id)`
+- `analysis_run_id uuid nullable references analysis_runs(id)`
+- `export_type text not null`
+- `storage_path text not null`
+- `metadata_json jsonb`
+- `created_at timestamptz`
+
+## Migration strategy
+
+- Use Alembic.
+- First migration creates extensions and core tables.
+- Second migration adds spatial indexes.
+- Seed script may create a sample project and sample area only if explicitly run.
+
+## Sprint 7B provider-to-dataset mapping
+
+Provider integration is a contract layer only in Sprint 7B. Providers do not write directly to `vector_features`; future provider output must flow through `DatasetService` and `VectorFeatureService` so dataset provenance, storage metadata and feature persistence remain consistent.
+
+- `grb`: maps to `dataset_role='reference'`, `source_name='grb'`.
+- `osm`: maps to `dataset_role='source'` by default, or `dataset_role='reference'` only when explicitly requested; `source_name='osm'`.
+- `manual`: maps to `dataset_role='reference'`, `source_name='manual'`.
+- `fixture`: maps to `dataset_role='reference'`, `source_name='fixture'`.
+
+GRB and OSM live imports are intentionally `not_configured` in Sprint 7B. Manual and fixture reference datasets use existing upload and fixture flows.
+
+Large explicit operator imports may invoke DatasetService directly inside the
+backend container when a single multipart upload would require loading the
+entire regional artifact into memory. The regional GRB building operator still
+creates a normal Dataset and DatasetVersion and delegates every queryable row
+to VectorFeatureService. It copies the immutable combined artifact in a
+stream, loads one retained municipality partition at a time, flushes bounded
+feature batches and commits only when the indexed count matches the manifest.
+It does not expose a direct SQL/provider write path and does not alter the
+public provider endpoint's `not_configured` status.
+
+### Source freshness audit
+
+Source freshness is derived state and does not introduce a scheduler or status
+table. `Dataset.source_name`, `imported_at`, `observed_at`, `source_version`,
+checksum/storage metadata and immutable `DatasetVersion` rows remain the
+persistence source of truth. The project source-freshness service groups those
+records under explicit publication policies and performs a read-only local
+evidence audit. External catalogue checks and refresh jobs remain explicit
+operator actions; they may never overwrite a fixed edition or scenario in
+place.
+
+Regional GRB refreshes add no lifecycle table. A read-only plan compares the
+official dated edition with the latest Dataset in each governed temporal
+series. Staging is filesystem evidence only. Checksum-confirmed apply creates a
+new Dataset plus DatasetVersion and vector_features through the existing
+DatasetService/VectorFeatureService transaction; prior snapshots remain
+unchanged and queryable for temporal comparison.
+
+Statbel population release management likewise adds no lifecycle table.
+Planning reads the existing source-catalog response. Stage, named review and
+their SHA-256-bound evidence are filesystem-only operator artifacts. Apply
+revalidates those artifacts and invokes the existing Dataset upload service,
+which creates the ordinary annual `datasets`, `dataset_versions` and
+`vector_features` records in one established persistence flow. An existing
+year remains idempotent and previous annual snapshots are never updated or
+deleted.
+
+SPW bathymetry also requires no migration. The immutable official ZIP and
+bounded operator COG are filesystem artifacts. A successful API upload creates
+an ordinary raster `Dataset` and `DatasetVersion`; raster values stay in the
+versioned GeoTIFF rather than PostGIS. `source_metadata` retains EPSG:3812,
+mDNG, survey period, source checksum and bounded extent, while
+`provenance_metadata` retains the official URL, archive member, derived
+checksum and selection geometry/hash. Analysis never writes metric rows and
+never converts the vertical datum implicitly.
+
+Definitive ALZ release management uses the same persistence boundary and adds
+no migration or release table. Catalog planning, staged archive/GeoJSON,
+crop-code evidence and named review live on the filesystem. Only an approved
+apply invokes the existing Dataset upload service, producing the ordinary
+annual Dataset, DatasetVersion and vector_features records with
+`source_version=<year>-definitive`. Earlier annual snapshots are retained and
+provisional v1/v2 publications cannot create rows.
+
+Current-orthophoto release management likewise adds no lifecycle table or
+migration. Preflight is read-only; stage and named review are filesystem-only.
+Approved apply uploads the exact checksummed EPSG:31370 GeoTIFF through the
+existing dataset endpoint and DatasetService transaction, creating one normal
+immutable raster Dataset and DatasetVersion with official `YYYY.NN`
+`source_version`, temporal flight-date evidence and release provenance.
+Official editions take precedence over legacy rolling markers in catalog
+comparison, but direct metadata backfill, update or deletion of those legacy
+rows remains prohibited.
+
+## Resumable AOI operations
+
+Migration `202607260001` adds `aoi_operations` and
+`aoi_operation_partitions`. The parent stores the immutable EPSG:4326 AOI
+snapshot, original request and source-specific plan. Child rows store exact
+partition geometry, stable `(operation_id, partition_key)` identity, provider
+and product, attempt budget, checkpoint, result and child Job linkage.
+
+Workers atomically claim queued child rows. Interrupted running rows return to
+queued state while attempts remain and become failed when exhausted. Provider
+output still flows through the existing DatasetService/provider contracts;
+these tables orchestrate work and never become a parallel dataset store.
+The all-in-one runtime enables one database-polling worker. Multiple future
+workers remain safe because partition claim uses `FOR UPDATE SKIP LOCKED` and
+the partition key is unique inside its parent operation.
+
+## Geometry normalization
+
+- User-drawn polygons arrive as EPSG:4326.
+- Uploaded vector data may arrive in another CRS; preserve original CRS and reproject to EPSG:4326 for storage.
+- Area calculations should use a projected CRS suitable for Belgium, preferably EPSG:31370 or another documented Belgian projection.
+
+## Out of scope for V1
+
+- Raster-in-database storage.
+- Multi-tenant row-level security.
+- User accounts.
+- Full model registry tables.
+
+## Governed flood-hazard rasters
+
+VMM flood-depth scenarios require no new table. Each acquired coverage is an
+ordinary `datasets` raster plus immutable `dataset_versions` provenance and a
+normal synchronous acquisition Job. The GeoTIFF remains filesystem/object
+storage; PostgreSQL keeps source identity, WCS checksums, EPSG:31370 bounds,
+source/normalized units, exact Area scope and scenario parameters.
+
+Scenario alternatives do not receive a fabricated `observed_at` value and are
+not grouped as a temporal series. Selection metrics are calculated on demand
+from the persisted raster. Bathymetry and permanent waterbody volume remain
+absent from persistence until a separately governed source/model exists.
+
+## Temporal dataset foundation
+
+Historical observations remain normal `datasets` and `vector_features`; there
+is no parallel temporal feature store. Snapshots are grouped by
+`datasets.temporal_series_key` and carry `observed_at`, `valid_from`,
+`valid_to`, `temporal_granularity` and `source_version`. Observation time is
+kept separate from ingestion time (`imported_at`).
+
+`dataset_versions` records immutable storage provenance for every upload and
+derived output: dataset-local `version`, storage path, source version,
+observation/validity dates, checksum and source/provenance JSON. The
+`(dataset_id, version)` pair is unique. Temporal series lookup is indexed by
+`(project_id, temporal_series_key, observed_at)` and source-feature lookup by
+`(dataset_id, source_feature_id)`.
+
+Time-series comparison is read-only and aggregates persisted geometry inside a
+requested bbox. Object-level added/removed/modified evidence is only valid for
+sources that explicitly declare stable source feature identifiers.
+
+Regional 1778/1873/1969 historical land-use snapshots use the same tables.
+Their `provenance_metadata` records 28-partition identity, combined output
+checksum and retained raw-response status. Municipality-clipped source ids are
+partition-suffixed for uniqueness, while `identity_stable=false` prohibits
+object-lineage interpretation. No partition or temporal shadow table is added.
+### Partitioned vector feature lookup index
+
+Regional operator Datasets retain their municipality partition identity in
+`vector_features.properties_json.municipality`. Migration `202607160001` adds
+the composite expression index
+`ix_vector_features_dataset_municipality(dataset_id, properties_json->>'municipality')`.
+It supports exact preclipped municipality selection without changing the
+canonical `vector_features` schema or introducing operator-specific tables.
+Free rectangle queries continue to use the geometry GiST index.
+
+## Bathymetry profile persistence
+
+VHA profile points require no new table or migration. The immutable GeoJSON
+artifact is stored as one normal reference `datasets` row and
+`dataset_versions` row; each exact point is a normal `vector_features` row with
+EPSG:4326 geometry and source properties. Existing dataset, source-feature and
+GiST indexes support selection.
+
+Profile measurement dates remain feature properties because a bounded Dataset
+can contain many historical campaigns. No artificial Dataset `observed_at` or
+temporal series is assigned. A future bed raster remains file/object storage
+plus Dataset metadata, not raster-in-database storage. Any national/maritime
+extension keeps source vertical datum, survey epoch and Area partition
+explicit and does not create a provider-specific shadow schema.