feat(provenance): govern source snapshots and data inputs

2026-08-01 23:46:17 +02:00
parent cebeb5f3b4
commit 5b3c17b494
96 changed files with 20156 additions and 351 deletions
@@ -393,15 +393,48 @@ Fields:

 - `file`: dataset file.
 - `dataset_type`: `vector`, `geojson` (legacy), `raster`.
- `source`: free text, e.g. `user_upload`, `grb`, `osm`.
+- `source`: descriptive caller text, e.g. `user_upload`. It is retained as a
+  claim only and never establishes authority.
 - `dataset_role`: `source`, `derived`, or `reference` (default `source`).
- `source_name`: optional source identity, e.g. `manual`, `grb`, `osm`; reference uploads default to `manual` when omitted.
+- `source_name`: optional descriptive claim. Public uploads are always bound
+  to the server-owned `manual` registry entry, including when this field says
+  `grb`, `osm` or another official name. Reference uploads also default to
+  `manual`.
 - `reference_layer_name`: optional reference layer label, e.g. `buildings`; only retained for reference datasets.
 - `area_id`: optional.

-Response: `DatasetRead` with extracted metadata if supported.
+Response: `DatasetRead` with extracted metadata if supported, plus
+`source_registry_id`, `source_snapshot_id`, exact data-contract key/version,
+validation report/status, provenance/lineage status, quarantine status and an
+idempotent ingest key. A malformed or doubtful artifact is retained as
+`status=quarantined`; it is not silently discarded or made ready.

-Vector uploads remain stored as original files and are also persisted into `vector_features` as queryable PostGIS state.
+Vector uploads remain stored as original files and are also persisted into
+`vector_features` as queryable PostGIS state only after their contract passes.
+Non-EPSG:4326 vector coordinates are explicitly transformed before canonical
+feature persistence; relabelling Lambert coordinates as EPSG:4326 is rejected.
+
+### GET `/api/v1/source-registry`
+
+Lists server-owned source definitions. Optional query parameter
+`classification` is one of `authoritative`, `corroborative`, `contextual`,
+`derived`, or `experimental`. This endpoint reports policy and snapshot counts;
+it does not assert that historical datasets carrying a matching text field are
+trusted. It is operator-only because source snapshots can contain
+operator-acquisition provenance. A demo session uses its project-scoped
+dataset provenance endpoint instead.
+
+### GET `/api/v1/source-registry/{source_key}`
+
+Returns one source definition and its immutable snapshots, including authority
+scope, licence/restrictions, expected geometry/attributes, freshness and known
+limitations. Unknown source keys return `SOURCE_REGISTRY_ENTRY_NOT_FOUND`.
+
+### GET `/api/v1/projects/{project_id}/datasets/{dataset_id}/provenance`
+
+Returns the dataset's bound source/snapshot, contract report, lineage edges and
+quarantine records. A missing binding is visible as incomplete provenance; it
+is never backfilled from a display name or caller metadata.

 ### GET `/api/v1/projects/{project_id}/datasets/orthophoto/products`

@@ -69,6 +69,41 @@ Spatial index required on `geometry`.
 - `status text default 'created'`
 - `created_at timestamptz`

+### Source registry, snapshots and quarantine (Accuracy Phase 2)
+
+Migration `202608010001_source_registry_provenance` adds a server-owned
+provenance boundary without deleting or inventing facts for existing rows.
+
+`source_registry` stores one governed source definition per `source_key`:
+
+- classification (`authoritative`, `corroborative`, `contextual`, `derived`,
+  `experimental`), authority/scope and provider adapter key;
+- licence, usage restrictions, expected CRS/units/resolution, temporal and
+  geographic coverage, expected geometry/attributes and known limitations;
+- usage policy, freshness and ingest status.
+
+`source_snapshots` stores immutable acquired editions keyed by
+`(source_registry_id, snapshot_key)`, including source version/snapshot/fetch
+time, lowercase SHA-256 checksum, CRS/units/resolution, coverage, observed
+schema, freshness, ingest status and raw snapshot metadata.
+
+`datasets` and `dataset_versions` receive `source_registry_id`,
+`source_snapshot_id`, data-contract key/version, validation report/status,
+provenance/lineage status and idempotent `ingest_key`. New governed records
+must have a passing report before they become `ready`. Legacy rows are retained
+with explicit incomplete/not-validated state; a migration never promotes them
+based on a historical `source_name` string.
+
+`dataset_lineage_edges` records parent/child Dataset(+Version), transform name
+and version, parameters and input/output hashes. `dataset_quarantines` retains
+the rejected dataset/version/snapshot, stage, reason, validation evidence and
+artifact location. Quarantine is a stateful preservation record, not a delete.
+
+The database constraints enforce accepted status vocabularies, nonblank ingest
+keys, immutable snapshot identity/checksum format, unique ingest idempotency
+keys and non-self lineage edges. Service-level contract validation remains
+responsible for CRS, topology, bbox, units and source-task semantics.
+
 ### vector_features

 Used for imported vector datasets and derived vector outputs when feature-level storage is needed. Original files remain source artifacts; this table is the queryable PostGIS state for vector features.