Add governed Statbel release promotion
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-17 00:19:47 +02:00
parent 6187a9d41d
commit d0a8a3ad15
14 changed files with 1401 additions and 28 deletions
+7
View File
@@ -450,6 +450,13 @@ reported only as transition evidence. `service_type=DCAT` uses
A newer statistical-sector geometry edition is not interpreted as a newer
population release.
Future Statbel execution remains outside the HTTP request cycle in
`scripts/manage_statbel_population_release.py`. It reuses this read-only
catalog response for `plan`, then separates filesystem-only `stage`, named
human `review` and checksum-confirmed `apply`. No additional public endpoint
is introduced. Apply delegates to the existing dataset upload contract and
creates a new immutable annual snapshot only after all evidence is unchanged.
The endpoint accepts no arbitrary URL, feature query, area or layer. It does
not fetch vector features, raster pixels or models, create jobs/datasets, write
to PostGIS or trigger an import. The normal `source-freshness` endpoint remains
+52
View File
@@ -1,3 +1,55 @@
## Sprint 228 - Governed Statbel population release promotion (2026-07-17)
Implemented:
- Added `scripts/manage_statbel_population_release.py` with four separate
operator actions for the approved Kempen scope. `plan` is read-only; `stage`
requires exact year/layout confirmation and performs bounded fresh download
plus preflight only; `review` requires a named explicit approval; `apply`
requires the exact staged-plan and review-evidence SHA-256 values.
- Derived the only accepted population and matching sector-geometry URLs from
the allowlisted catalog year/layout contract. The coordinator accepts no
arbitrary source URL or process, and current, older, unavailable, ambiguous
or catalog-drifted releases are not stageable.
- Bound catalog identity, source archives, preflight manifest, derived
snapshot, scope/national accounting, ZZZZ totals, baseline trend and geometry
repairs into `staged-plan.json`. Bound the named human decision to that plan
in `review-evidence.json`; successful apply records the immutable Dataset in
`applied-evidence.json` without deleting prior snapshots.
- Extended `provision_mol_population_history.py` with a complete all-or-none
release config for one future year. Existing 2021-2025 arguments remain
compatible. Source downloads now enforce response and streaming byte bounds,
validate final official URLs and discover the latest retained baseline from
actual snapshot files rather than a hardcoded year list.
- Replaced the population operator's 200-row workspace lookup with complete,
total-consistent pagination. This keeps repeated apply idempotent in the live
project with 2,429 Datasets.
- Added Docker/readiness packaging and updated source, API, persistence,
storage and operator documentation. No API route, migration, scheduler,
automatic fetch or frontend behavior changed.
Validation:
- 15 focused Sprint 228 tests and 38 combined Sprint 194/227/228 tests passed.
Coverage includes release ordering, future URL/layout validation, dynamic
baseline discovery, bounded downloads, >200-row pagination, stage/apply
command separation, plan/review hashes, source/review tampering, evidence-root
confinement, catalog drift, named review, apply evidence and current-edition
refusal.
- Complete readiness passed with 830 backend tests, 110 documented routes,
one Alembic head `202607160001`, frontend typecheck and production build.
Static Alembic SQL, shell syntax, target Ruff and diff checks passed.
- A compatibility run against the live Tower API used the temporary candidate
scripts before deployment. The official Statbel catalog reported
`NodeID6475`, remote/local edition `2025`, layout `new`, catalog hash
`64b17ce059a9c2f936d4b5741b20aed4b178409d8b5be81be86e6fdc9fe7c9d9` and
decision `current`; `plan` wrote no evidence. A deliberate `stage` attempt
for that current edition exited 1 with `not safely stageable: current` and
left the database Dataset count unchanged at 2,429.
Boundary:
- No newer population edition is currently advertised, so no real stage,
review or apply was executed. Their complete state machine is fixture-tested;
the first future release must still pass all four explicit operator phases.
## Sprint 227 - Statbel population import compatibility preflight (2026-07-16)
Implemented:
+9
View File
@@ -274,6 +274,15 @@ new Dataset plus DatasetVersion and vector_features through the existing
DatasetService/VectorFeatureService transaction; prior snapshots remain
unchanged and queryable for temporal comparison.
Statbel population release management likewise adds no lifecycle table.
Planning reads the existing source-catalog response. Stage, named review and
their SHA-256-bound evidence are filesystem-only operator artifacts. Apply
revalidates those artifacts and invokes the existing Dataset upload service,
which creates the ordinary annual `datasets`, `dataset_versions` and
`vector_features` records in one established persistence flow. An existing
year remains idempotent and previous annual snapshots are never updated or
deleted.
## Geometry normalization
- User-drawn polygons arrive as EPSG:4326.
+16
View File
@@ -222,6 +222,22 @@ bronarchieven en de afgeleide GeoJSON. Zij geeft alleen technische
importgeschiktheid aan; zij vervangt of importeert nooit automatisch een
bestaande Dataset.
`scripts/manage_statbel_population_release.py` beheert een toekomstige editie
in vier afzonderlijke operatorstappen. `plan` vergelijkt de nieuwste
allowlisted DCAT-editie read-only met de lokale bronversie. `stage` vereist
expliciete bevestiging van jaar en REDEGEO-layout, haalt de exact uit het
cataloguscontract afgeleide Statbel-URL's begrensd op en voert alleen de
preflight/artefactstaging uit. `review` schrijft pas na een benoemde menselijke
goedkeuring SHA-256-gebonden reviewevidence. `apply` vereist zowel plan- als
reviewhash, controleert catalogusidentiteit en alle bestanden opnieuw en maakt
via de bestaande uploadservice hoogstens een nieuwe immutable Dataset.
De coordinator accepteert geen vrije provider-URL, start geen achtergrondtaak
en kan een actuele of oudere remote editie niet stagen. Een ontbrekende
bijbehorende sectorgeometrie, gewijzigde catalogus, gewijzigde bronbyte,
onverklaarde totalensprong of onvolledige review blokkeert de flow. Bestaande
snapshots blijven beschikbaar voor historische vergelijking.
### Mol population history
`scripts/provision_mol_population_history.py` imports official Statbel
+17
View File
@@ -122,6 +122,23 @@ Dataset/DatasetVersion plus PostGIS `vector_features` through the canonical
persistence services. Temporary standalone preflight output does not create a
database record and may be removed explicitly after operator review.
Governed release-decision evidence is separate from the source artifacts:
```text
storage/operator-evidence/statbel-population-refresh/{scope}/{year}/
staged-plan.json
review-evidence.json
applied-evidence.json
```
The staged plan binds the current official catalog identity to every retained
source/snapshot/preflight hash and accounting summary. Review evidence binds a
named approval to that exact plan. Applied evidence binds both approvals to
the resulting Dataset id. These JSON files are operator audit artifacts, not
database lifecycle entities. A plan/review path outside the configured
evidence root or a retained archive outside the population output root is
rejected. No evidence file authorizes deletion or in-place replacement.
Waterinfo raw station layers, timeseries responses and checksum manifests live
under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
evidence; queryable annual Point snapshots are normal Dataset/vector_feature
+1
View File
@@ -44,6 +44,7 @@
- [x] Add a fail-closed ALZ publication probe that distinguishes provisional v1/v2 snapshots from the definitive v3 historical edition and never downloads an archive.
- [x] Add a fail-closed Statbel DCAT publication probe that distinguishes population year, sector-geometry year and the 2025 REDEGEO transition without downloading distributions.
- [x] Add a fail-closed Statbel population import preflight with archive/schema/CRS/join/total/baseline checks, retained checksums and explicit ZZZZ accounting before any new Dataset import.
- [x] Add an explicit Statbel population plan -> stage -> named review -> checksum-confirmed apply workflow that preserves every prior annual snapshot.
- [ ] Extend catalogue probes only to additional sources that publish a stable official edition contract; do not add background polling or infer releases from HTTP dates alone.
## Governed source expansion backlog