Add governed Statbel release promotion
This commit is contained in:
@@ -450,6 +450,13 @@ reported only as transition evidence. `service_type=DCAT` uses
|
||||
A newer statistical-sector geometry edition is not interpreted as a newer
|
||||
population release.
|
||||
|
||||
Future Statbel execution remains outside the HTTP request cycle in
|
||||
`scripts/manage_statbel_population_release.py`. It reuses this read-only
|
||||
catalog response for `plan`, then separates filesystem-only `stage`, named
|
||||
human `review` and checksum-confirmed `apply`. No additional public endpoint
|
||||
is introduced. Apply delegates to the existing dataset upload contract and
|
||||
creates a new immutable annual snapshot only after all evidence is unchanged.
|
||||
|
||||
The endpoint accepts no arbitrary URL, feature query, area or layer. It does
|
||||
not fetch vector features, raster pixels or models, create jobs/datasets, write
|
||||
to PostGIS or trigger an import. The normal `source-freshness` endpoint remains
|
||||
|
||||
@@ -1,3 +1,55 @@
|
||||
## Sprint 228 - Governed Statbel population release promotion (2026-07-17)
|
||||
|
||||
Implemented:
|
||||
- Added `scripts/manage_statbel_population_release.py` with four separate
|
||||
operator actions for the approved Kempen scope. `plan` is read-only; `stage`
|
||||
requires exact year/layout confirmation and performs bounded fresh download
|
||||
plus preflight only; `review` requires a named explicit approval; `apply`
|
||||
requires the exact staged-plan and review-evidence SHA-256 values.
|
||||
- Derived the only accepted population and matching sector-geometry URLs from
|
||||
the allowlisted catalog year/layout contract. The coordinator accepts no
|
||||
arbitrary source URL or process, and current, older, unavailable, ambiguous
|
||||
or catalog-drifted releases are not stageable.
|
||||
- Bound catalog identity, source archives, preflight manifest, derived
|
||||
snapshot, scope/national accounting, ZZZZ totals, baseline trend and geometry
|
||||
repairs into `staged-plan.json`. Bound the named human decision to that plan
|
||||
in `review-evidence.json`; successful apply records the immutable Dataset in
|
||||
`applied-evidence.json` without deleting prior snapshots.
|
||||
- Extended `provision_mol_population_history.py` with a complete all-or-none
|
||||
release config for one future year. Existing 2021-2025 arguments remain
|
||||
compatible. Source downloads now enforce response and streaming byte bounds,
|
||||
validate final official URLs and discover the latest retained baseline from
|
||||
actual snapshot files rather than a hardcoded year list.
|
||||
- Replaced the population operator's 200-row workspace lookup with complete,
|
||||
total-consistent pagination. This keeps repeated apply idempotent in the live
|
||||
project with 2,429 Datasets.
|
||||
- Added Docker/readiness packaging and updated source, API, persistence,
|
||||
storage and operator documentation. No API route, migration, scheduler,
|
||||
automatic fetch or frontend behavior changed.
|
||||
|
||||
Validation:
|
||||
- 15 focused Sprint 228 tests and 38 combined Sprint 194/227/228 tests passed.
|
||||
Coverage includes release ordering, future URL/layout validation, dynamic
|
||||
baseline discovery, bounded downloads, >200-row pagination, stage/apply
|
||||
command separation, plan/review hashes, source/review tampering, evidence-root
|
||||
confinement, catalog drift, named review, apply evidence and current-edition
|
||||
refusal.
|
||||
- Complete readiness passed with 830 backend tests, 110 documented routes,
|
||||
one Alembic head `202607160001`, frontend typecheck and production build.
|
||||
Static Alembic SQL, shell syntax, target Ruff and diff checks passed.
|
||||
- A compatibility run against the live Tower API used the temporary candidate
|
||||
scripts before deployment. The official Statbel catalog reported
|
||||
`NodeID6475`, remote/local edition `2025`, layout `new`, catalog hash
|
||||
`64b17ce059a9c2f936d4b5741b20aed4b178409d8b5be81be86e6fdc9fe7c9d9` and
|
||||
decision `current`; `plan` wrote no evidence. A deliberate `stage` attempt
|
||||
for that current edition exited 1 with `not safely stageable: current` and
|
||||
left the database Dataset count unchanged at 2,429.
|
||||
|
||||
Boundary:
|
||||
- No newer population edition is currently advertised, so no real stage,
|
||||
review or apply was executed. Their complete state machine is fixture-tested;
|
||||
the first future release must still pass all four explicit operator phases.
|
||||
|
||||
## Sprint 227 - Statbel population import compatibility preflight (2026-07-16)
|
||||
|
||||
Implemented:
|
||||
|
||||
@@ -274,6 +274,15 @@ new Dataset plus DatasetVersion and vector_features through the existing
|
||||
DatasetService/VectorFeatureService transaction; prior snapshots remain
|
||||
unchanged and queryable for temporal comparison.
|
||||
|
||||
Statbel population release management likewise adds no lifecycle table.
|
||||
Planning reads the existing source-catalog response. Stage, named review and
|
||||
their SHA-256-bound evidence are filesystem-only operator artifacts. Apply
|
||||
revalidates those artifacts and invokes the existing Dataset upload service,
|
||||
which creates the ordinary annual `datasets`, `dataset_versions` and
|
||||
`vector_features` records in one established persistence flow. An existing
|
||||
year remains idempotent and previous annual snapshots are never updated or
|
||||
deleted.
|
||||
|
||||
## Geometry normalization
|
||||
|
||||
- User-drawn polygons arrive as EPSG:4326.
|
||||
|
||||
@@ -222,6 +222,22 @@ bronarchieven en de afgeleide GeoJSON. Zij geeft alleen technische
|
||||
importgeschiktheid aan; zij vervangt of importeert nooit automatisch een
|
||||
bestaande Dataset.
|
||||
|
||||
`scripts/manage_statbel_population_release.py` beheert een toekomstige editie
|
||||
in vier afzonderlijke operatorstappen. `plan` vergelijkt de nieuwste
|
||||
allowlisted DCAT-editie read-only met de lokale bronversie. `stage` vereist
|
||||
expliciete bevestiging van jaar en REDEGEO-layout, haalt de exact uit het
|
||||
cataloguscontract afgeleide Statbel-URL's begrensd op en voert alleen de
|
||||
preflight/artefactstaging uit. `review` schrijft pas na een benoemde menselijke
|
||||
goedkeuring SHA-256-gebonden reviewevidence. `apply` vereist zowel plan- als
|
||||
reviewhash, controleert catalogusidentiteit en alle bestanden opnieuw en maakt
|
||||
via de bestaande uploadservice hoogstens een nieuwe immutable Dataset.
|
||||
|
||||
De coordinator accepteert geen vrije provider-URL, start geen achtergrondtaak
|
||||
en kan een actuele of oudere remote editie niet stagen. Een ontbrekende
|
||||
bijbehorende sectorgeometrie, gewijzigde catalogus, gewijzigde bronbyte,
|
||||
onverklaarde totalensprong of onvolledige review blokkeert de flow. Bestaande
|
||||
snapshots blijven beschikbaar voor historische vergelijking.
|
||||
|
||||
### Mol population history
|
||||
|
||||
`scripts/provision_mol_population_history.py` imports official Statbel
|
||||
|
||||
@@ -122,6 +122,23 @@ Dataset/DatasetVersion plus PostGIS `vector_features` through the canonical
|
||||
persistence services. Temporary standalone preflight output does not create a
|
||||
database record and may be removed explicitly after operator review.
|
||||
|
||||
Governed release-decision evidence is separate from the source artifacts:
|
||||
|
||||
```text
|
||||
storage/operator-evidence/statbel-population-refresh/{scope}/{year}/
|
||||
staged-plan.json
|
||||
review-evidence.json
|
||||
applied-evidence.json
|
||||
```
|
||||
|
||||
The staged plan binds the current official catalog identity to every retained
|
||||
source/snapshot/preflight hash and accounting summary. Review evidence binds a
|
||||
named approval to that exact plan. Applied evidence binds both approvals to
|
||||
the resulting Dataset id. These JSON files are operator audit artifacts, not
|
||||
database lifecycle entities. A plan/review path outside the configured
|
||||
evidence root or a retained archive outside the population output root is
|
||||
rejected. No evidence file authorizes deletion or in-place replacement.
|
||||
|
||||
Waterinfo raw station layers, timeseries responses and checksum manifests live
|
||||
under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
|
||||
evidence; queryable annual Point snapshots are normal Dataset/vector_feature
|
||||
|
||||
@@ -44,6 +44,7 @@
|
||||
- [x] Add a fail-closed ALZ publication probe that distinguishes provisional v1/v2 snapshots from the definitive v3 historical edition and never downloads an archive.
|
||||
- [x] Add a fail-closed Statbel DCAT publication probe that distinguishes population year, sector-geometry year and the 2025 REDEGEO transition without downloading distributions.
|
||||
- [x] Add a fail-closed Statbel population import preflight with archive/schema/CRS/join/total/baseline checks, retained checksums and explicit ZZZZ accounting before any new Dataset import.
|
||||
- [x] Add an explicit Statbel population plan -> stage -> named review -> checksum-confirmed apply workflow that preserves every prior annual snapshot.
|
||||
- [ ] Extend catalogue probes only to additional sources that publish a stable official edition contract; do not add background polling or infer releases from HTTP dates alone.
|
||||
|
||||
## Governed source expansion backlog
|
||||
|
||||
Reference in New Issue
Block a user