Harden Statbel population import compatibility
This commit is contained in:
@@ -1,3 +1,64 @@
|
||||
## Sprint 227 - Statbel population import compatibility preflight (2026-07-16)
|
||||
|
||||
Implemented:
|
||||
- Added `scripts/statbel_population_preflight.py` as a local, fail-closed gate
|
||||
between staged official Statbel archives and the existing population
|
||||
provisioner. It performs exact source/edition allowlisting, bounded ZIP
|
||||
inspection, population and geometry schema checks, EPSG:31370 and situation
|
||||
date validation, geometry validity/area checks, sector joins, explicit
|
||||
municipality reconciliation, national/scope total reconciliation and an
|
||||
annualized baseline-change guard.
|
||||
- Removed the invalid historical assumption that `CD_SECTOR[:5]` always equals
|
||||
the current `CD_REFNIS`. The 2025 REDEGEO layout can retain old sector codes
|
||||
after a municipal merger; population and geometry must instead agree on
|
||||
their explicit current municipality fields.
|
||||
- Kept official `ZZZZ` population rows in national and scope accounting while
|
||||
excluding them honestly from map geometry. Bounded `make_valid` repair is
|
||||
allowed only when the result remains polygonal, valid and area-preserving;
|
||||
repaired sector codes are recorded in the manifest.
|
||||
- Hardened `provision_mol_population_history.py` so a new fetch is preflighted
|
||||
before source retention or derivation, source ZIPs and the derived GeoJSON
|
||||
are written atomically, and all three SHA-256 values are rechecked before
|
||||
upload. Existing immutable Datasets remain idempotent; a legacy cached file
|
||||
cannot create a new Dataset without `--force` restaging evidence.
|
||||
- Packaged the script in the all-in-one image, added it to readiness compile
|
||||
checks and documented command, storage layout, evidence and limitations.
|
||||
No API route, migration, scheduler, background fetch or existing Dataset was
|
||||
changed.
|
||||
|
||||
Validation:
|
||||
- Focused preflight/regional-time-series suite: 23 passed. It covers source and
|
||||
member identity, missing schema, duplicate sectors, invalid totals, archive
|
||||
traversal, CRS/date mismatch, explicit municipality mismatch, unexpected
|
||||
non-spatial rows, bounded topology repair, excessive trend change, atomic
|
||||
staging, source/snapshot tampering and legacy parser strictness.
|
||||
- Complete readiness passed with 815 backend tests, 110 documented routes,
|
||||
one Alembic head `202607160001`, frontend typecheck and production build.
|
||||
Static Alembic SQL, shell syntax, target Ruff and diff checks passed. A
|
||||
repository-wide Ruff audit still reports 17 pre-existing warnings outside
|
||||
this change; no unrelated refactor was performed.
|
||||
- Live compatibility-only execution inside the healthy Tower container used
|
||||
locally staged official archives and made no API/database call:
|
||||
- 2024 standard layout: 646 spatial Kempen sectors, spatial population
|
||||
503,405, 28 `ZZZZ` rows / 276 unlocated inhabitants and accounted total
|
||||
503,681 against the retained 2023 baseline.
|
||||
- 2025 new REDEGEO layout: 733 spatial Kempen sectors, spatial population
|
||||
506,473, 26 `ZZZZ` rows / 294 unlocated inhabitants and accounted total
|
||||
506,767 against the retained 2024 baseline; annualized change 0.6094%.
|
||||
- National 2025 accounting reconciled 21,183 population rows and 20,781
|
||||
geometries to 11,825,551 inhabitants, including 402 `ZZZZ` rows / 7,654
|
||||
unlocated inhabitants. Four official self-intersection cases were repaired
|
||||
without area change and recorded by sector code.
|
||||
- The complete real 2025 operator staging path retained both source archives,
|
||||
generated a 733-feature scoped GeoJSON and authorized source/snapshot hashes
|
||||
in an isolated `/tmp/statbel-stage` directory. It did not import or replace a
|
||||
Dataset.
|
||||
|
||||
Boundary:
|
||||
- A passed preflight proves technical compatibility only. Adding a future
|
||||
edition to `POPULATION_URLS` and replacing or importing any immutable Dataset
|
||||
remains an explicit reviewed operator action.
|
||||
|
||||
## Sprint 226 - Governed Statbel population edition probe (2026-07-16)
|
||||
|
||||
Implemented:
|
||||
|
||||
Reference in New Issue
Block a user