Add governed source freshness audit
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-16 15:05:42 +02:00
parent 2723043e3d
commit 1e0e7b6499
22 changed files with 1148 additions and 0 deletions
+18
View File
@@ -394,6 +394,24 @@ the existing MapLibre image-overlay path.
List datasets.
### GET `/api/v1/projects/{project_id}/datasets/source-freshness`
Returns one read-only, canonical source-governance report for all persisted
project datasets. Datasets are grouped by `source_name` (falling back to
`source`) and classified as a rolling snapshot, annual release, fixed edition,
scenario, historical archive or local artifact.
Each source item reports dataset/version counts, latest import and observation
evidence, latest source version, next review date where meaningful, historical
series availability and local integrity counts for missing DatasetVersions,
checksum mismatches, missing storage files and size mismatches. Status is one
of `current`, `due`, `review_required` or `local`.
This endpoint never contacts an external provider, downloads data, mutates a
Dataset or silently refreshes a publication. Fixed editions and scenarios are
not marked stale merely because their source date is old. Unknown sources are
`review_required` until an explicit publication policy is defined.
### GET `/api/v1/projects/{project_id}/datasets/{dataset_id}`
Return metadata.
+30
View File
@@ -9272,3 +9272,33 @@ Next:
- Implement the governed Flemish thematic-raster registry Wave 1 for space
occupation, open space, population density, node value and total service
level, followed by the digital soil map vector operator.
## Sprint 221 - Governed source freshness audit (2026-07-16)
Implemented:
- Added a project-level canonical source-freshness endpoint derived exclusively
from persisted Dataset, DatasetVersion and local storage evidence.
- Defined explicit policies for GRB/VRBG/register snapshots, annual
population/land-use/agriculture/nature/water series, fixed DHMV/soil/thematic
editions, VMM scenarios, historical archives and local artifacts.
- Added fail-closed classification for unknown sources and local integrity
checks for missing versions, checksum mismatches, missing files and stored
size disagreement.
- Added a compact Status workspace panel and packaged
`audit_source_freshness.py` for explicit Unraid cron/reporting use.
Boundaries:
- No migration, provider call, background daemon, source download or automatic
Dataset refresh was introduced.
- A fixed historical edition/scenario remains current evidence of that edition;
old publication dates alone are not treated as corruption or staleness.
Validation:
- Focused service, policy, canonical envelope, CLI and UI contract tests added.
- Full release, live deployment and browser validation are recorded in the
delivery result after this implementation entry.
Next:
- Add machine-readable catalogue release probes only for providers with stable
official version endpoints, keeping every acquisition an explicit bounded
operator action.
+11
View File
@@ -256,6 +256,17 @@ feature batches and commits only when the indexed count matches the manifest.
It does not expose a direct SQL/provider write path and does not alter the
public provider endpoint's `not_configured` status.
### Source freshness audit
Source freshness is derived state and does not introduce a scheduler or status
table. `Dataset.source_name`, `imported_at`, `observed_at`, `source_version`,
checksum/storage metadata and immutable `DatasetVersion` rows remain the
persistence source of truth. The project source-freshness service groups those
records under explicit publication policies and performs a read-only local
evidence audit. External catalogue checks and refresh jobs remain explicit
operator actions; they may never overwrite a fixed edition or scenario in
place.
## Geometry normalization
- User-drawn polygons arrive as EPSG:4326.
+2
View File
@@ -38,6 +38,8 @@
- [x] Add a resumable regional DHMV DTM/DSM operator for all 28 approved Kempen municipality Areas.
- [x] Execute and audit the complete 336-product VMM and 56-product DHMV regional runtime matrices.
- [x] Present municipal DHMV/VMM partitions as logical regional layers and analyse cross-boundary rectangles without a municipality prerequisite.
- [x] Add a read-only source freshness/version audit with explicit publication policies, local integrity checks, operator CLI and compact Status workspace surface.
- [ ] Add external catalogue release probes only after each official provider exposes a stable machine-readable version contract; keep every refresh explicit and bounded.
## Governed source expansion backlog