Harden Statbel population import compatibility
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-16 22:54:21 +02:00
parent 5942712c7b
commit 4f4d467d11
11 changed files with 1533 additions and 18 deletions
+61
View File
@@ -1,3 +1,64 @@
## Sprint 227 - Statbel population import compatibility preflight (2026-07-16)
Implemented:
- Added `scripts/statbel_population_preflight.py` as a local, fail-closed gate
between staged official Statbel archives and the existing population
provisioner. It performs exact source/edition allowlisting, bounded ZIP
inspection, population and geometry schema checks, EPSG:31370 and situation
date validation, geometry validity/area checks, sector joins, explicit
municipality reconciliation, national/scope total reconciliation and an
annualized baseline-change guard.
- Removed the invalid historical assumption that `CD_SECTOR[:5]` always equals
the current `CD_REFNIS`. The 2025 REDEGEO layout can retain old sector codes
after a municipal merger; population and geometry must instead agree on
their explicit current municipality fields.
- Kept official `ZZZZ` population rows in national and scope accounting while
excluding them honestly from map geometry. Bounded `make_valid` repair is
allowed only when the result remains polygonal, valid and area-preserving;
repaired sector codes are recorded in the manifest.
- Hardened `provision_mol_population_history.py` so a new fetch is preflighted
before source retention or derivation, source ZIPs and the derived GeoJSON
are written atomically, and all three SHA-256 values are rechecked before
upload. Existing immutable Datasets remain idempotent; a legacy cached file
cannot create a new Dataset without `--force` restaging evidence.
- Packaged the script in the all-in-one image, added it to readiness compile
checks and documented command, storage layout, evidence and limitations.
No API route, migration, scheduler, background fetch or existing Dataset was
changed.
Validation:
- Focused preflight/regional-time-series suite: 23 passed. It covers source and
member identity, missing schema, duplicate sectors, invalid totals, archive
traversal, CRS/date mismatch, explicit municipality mismatch, unexpected
non-spatial rows, bounded topology repair, excessive trend change, atomic
staging, source/snapshot tampering and legacy parser strictness.
- Complete readiness passed with 815 backend tests, 110 documented routes,
one Alembic head `202607160001`, frontend typecheck and production build.
Static Alembic SQL, shell syntax, target Ruff and diff checks passed. A
repository-wide Ruff audit still reports 17 pre-existing warnings outside
this change; no unrelated refactor was performed.
- Live compatibility-only execution inside the healthy Tower container used
locally staged official archives and made no API/database call:
- 2024 standard layout: 646 spatial Kempen sectors, spatial population
503,405, 28 `ZZZZ` rows / 276 unlocated inhabitants and accounted total
503,681 against the retained 2023 baseline.
- 2025 new REDEGEO layout: 733 spatial Kempen sectors, spatial population
506,473, 26 `ZZZZ` rows / 294 unlocated inhabitants and accounted total
506,767 against the retained 2024 baseline; annualized change 0.6094%.
- National 2025 accounting reconciled 21,183 population rows and 20,781
geometries to 11,825,551 inhabitants, including 402 `ZZZZ` rows / 7,654
unlocated inhabitants. Four official self-intersection cases were repaired
without area change and recorded by sector code.
- The complete real 2025 operator staging path retained both source archives,
generated a 733-feature scoped GeoJSON and authorized source/snapshot hashes
in an isolated `/tmp/statbel-stage` directory. It did not import or replace a
Dataset.
Boundary:
- A passed preflight proves technical compatibility only. Adding a future
edition to `POPULATION_URLS` and replacing or importing any immutable Dataset
remains an explicit reviewed operator action.
## Sprint 226 - Governed Statbel population edition probe (2026-07-16)
Implemented: