Add governed source freshness audit
This commit is contained in:
@@ -394,6 +394,24 @@ the existing MapLibre image-overlay path.
|
||||
|
||||
List datasets.
|
||||
|
||||
### GET `/api/v1/projects/{project_id}/datasets/source-freshness`
|
||||
|
||||
Returns one read-only, canonical source-governance report for all persisted
|
||||
project datasets. Datasets are grouped by `source_name` (falling back to
|
||||
`source`) and classified as a rolling snapshot, annual release, fixed edition,
|
||||
scenario, historical archive or local artifact.
|
||||
|
||||
Each source item reports dataset/version counts, latest import and observation
|
||||
evidence, latest source version, next review date where meaningful, historical
|
||||
series availability and local integrity counts for missing DatasetVersions,
|
||||
checksum mismatches, missing storage files and size mismatches. Status is one
|
||||
of `current`, `due`, `review_required` or `local`.
|
||||
|
||||
This endpoint never contacts an external provider, downloads data, mutates a
|
||||
Dataset or silently refreshes a publication. Fixed editions and scenarios are
|
||||
not marked stale merely because their source date is old. Unknown sources are
|
||||
`review_required` until an explicit publication policy is defined.
|
||||
|
||||
### GET `/api/v1/projects/{project_id}/datasets/{dataset_id}`
|
||||
|
||||
Return metadata.
|
||||
|
||||
@@ -9272,3 +9272,33 @@ Next:
|
||||
- Implement the governed Flemish thematic-raster registry Wave 1 for space
|
||||
occupation, open space, population density, node value and total service
|
||||
level, followed by the digital soil map vector operator.
|
||||
|
||||
## Sprint 221 - Governed source freshness audit (2026-07-16)
|
||||
|
||||
Implemented:
|
||||
- Added a project-level canonical source-freshness endpoint derived exclusively
|
||||
from persisted Dataset, DatasetVersion and local storage evidence.
|
||||
- Defined explicit policies for GRB/VRBG/register snapshots, annual
|
||||
population/land-use/agriculture/nature/water series, fixed DHMV/soil/thematic
|
||||
editions, VMM scenarios, historical archives and local artifacts.
|
||||
- Added fail-closed classification for unknown sources and local integrity
|
||||
checks for missing versions, checksum mismatches, missing files and stored
|
||||
size disagreement.
|
||||
- Added a compact Status workspace panel and packaged
|
||||
`audit_source_freshness.py` for explicit Unraid cron/reporting use.
|
||||
|
||||
Boundaries:
|
||||
- No migration, provider call, background daemon, source download or automatic
|
||||
Dataset refresh was introduced.
|
||||
- A fixed historical edition/scenario remains current evidence of that edition;
|
||||
old publication dates alone are not treated as corruption or staleness.
|
||||
|
||||
Validation:
|
||||
- Focused service, policy, canonical envelope, CLI and UI contract tests added.
|
||||
- Full release, live deployment and browser validation are recorded in the
|
||||
delivery result after this implementation entry.
|
||||
|
||||
Next:
|
||||
- Add machine-readable catalogue release probes only for providers with stable
|
||||
official version endpoints, keeping every acquisition an explicit bounded
|
||||
operator action.
|
||||
|
||||
@@ -256,6 +256,17 @@ feature batches and commits only when the indexed count matches the manifest.
|
||||
It does not expose a direct SQL/provider write path and does not alter the
|
||||
public provider endpoint's `not_configured` status.
|
||||
|
||||
### Source freshness audit
|
||||
|
||||
Source freshness is derived state and does not introduce a scheduler or status
|
||||
table. `Dataset.source_name`, `imported_at`, `observed_at`, `source_version`,
|
||||
checksum/storage metadata and immutable `DatasetVersion` rows remain the
|
||||
persistence source of truth. The project source-freshness service groups those
|
||||
records under explicit publication policies and performs a read-only local
|
||||
evidence audit. External catalogue checks and refresh jobs remain explicit
|
||||
operator actions; they may never overwrite a fixed edition or scenario in
|
||||
place.
|
||||
|
||||
## Geometry normalization
|
||||
|
||||
- User-drawn polygons arrive as EPSG:4326.
|
||||
|
||||
@@ -38,6 +38,8 @@
|
||||
- [x] Add a resumable regional DHMV DTM/DSM operator for all 28 approved Kempen municipality Areas.
|
||||
- [x] Execute and audit the complete 336-product VMM and 56-product DHMV regional runtime matrices.
|
||||
- [x] Present municipal DHMV/VMM partitions as logical regional layers and analyse cross-boundary rectangles without a municipality prerequisite.
|
||||
- [x] Add a read-only source freshness/version audit with explicit publication policies, local integrity checks, operator CLI and compact Status workspace surface.
|
||||
- [ ] Add external catalogue release probes only after each official provider exposes a stable machine-readable version contract; keep every refresh explicit and bounded.
|
||||
|
||||
## Governed source expansion backlog
|
||||
|
||||
|
||||
Reference in New Issue
Block a user