Harden Statbel population import compatibility
This commit is contained in:
Codex
@@ -1,3 +1,64 @@
|
||||
## Sprint 227 - Statbel population import compatibility preflight (2026-07-16)
|
||||
|
||||
Implemented:
|
||||
- Added `scripts/statbel_population_preflight.py` as a local, fail-closed gate
|
||||
between staged official Statbel archives and the existing population
|
||||
provisioner. It performs exact source/edition allowlisting, bounded ZIP
|
||||
inspection, population and geometry schema checks, EPSG:31370 and situation
|
||||
date validation, geometry validity/area checks, sector joins, explicit
|
||||
municipality reconciliation, national/scope total reconciliation and an
|
||||
annualized baseline-change guard.
|
||||
- Removed the invalid historical assumption that `CD_SECTOR[:5]` always equals
|
||||
the current `CD_REFNIS`. The 2025 REDEGEO layout can retain old sector codes
|
||||
after a municipal merger; population and geometry must instead agree on
|
||||
their explicit current municipality fields.
|
||||
- Kept official `ZZZZ` population rows in national and scope accounting while
|
||||
excluding them honestly from map geometry. Bounded `make_valid` repair is
|
||||
allowed only when the result remains polygonal, valid and area-preserving;
|
||||
repaired sector codes are recorded in the manifest.
|
||||
- Hardened `provision_mol_population_history.py` so a new fetch is preflighted
|
||||
before source retention or derivation, source ZIPs and the derived GeoJSON
|
||||
are written atomically, and all three SHA-256 values are rechecked before
|
||||
upload. Existing immutable Datasets remain idempotent; a legacy cached file
|
||||
cannot create a new Dataset without `--force` restaging evidence.
|
||||
- Packaged the script in the all-in-one image, added it to readiness compile
|
||||
checks and documented command, storage layout, evidence and limitations.
|
||||
No API route, migration, scheduler, background fetch or existing Dataset was
|
||||
changed.
|
||||
|
||||
Validation:
|
||||
- Focused preflight/regional-time-series suite: 23 passed. It covers source and
|
||||
member identity, missing schema, duplicate sectors, invalid totals, archive
|
||||
traversal, CRS/date mismatch, explicit municipality mismatch, unexpected
|
||||
non-spatial rows, bounded topology repair, excessive trend change, atomic
|
||||
staging, source/snapshot tampering and legacy parser strictness.
|
||||
- Complete readiness passed with 815 backend tests, 110 documented routes,
|
||||
one Alembic head `202607160001`, frontend typecheck and production build.
|
||||
Static Alembic SQL, shell syntax, target Ruff and diff checks passed. A
|
||||
repository-wide Ruff audit still reports 17 pre-existing warnings outside
|
||||
this change; no unrelated refactor was performed.
|
||||
- Live compatibility-only execution inside the healthy Tower container used
|
||||
locally staged official archives and made no API/database call:
|
||||
- 2024 standard layout: 646 spatial Kempen sectors, spatial population
|
||||
503,405, 28 `ZZZZ` rows / 276 unlocated inhabitants and accounted total
|
||||
503,681 against the retained 2023 baseline.
|
||||
- 2025 new REDEGEO layout: 733 spatial Kempen sectors, spatial population
|
||||
506,473, 26 `ZZZZ` rows / 294 unlocated inhabitants and accounted total
|
||||
506,767 against the retained 2024 baseline; annualized change 0.6094%.
|
||||
- National 2025 accounting reconciled 21,183 population rows and 20,781
|
||||
geometries to 11,825,551 inhabitants, including 402 `ZZZZ` rows / 7,654
|
||||
unlocated inhabitants. Four official self-intersection cases were repaired
|
||||
without area change and recorded by sector code.
|
||||
- The complete real 2025 operator staging path retained both source archives,
|
||||
generated a 733-feature scoped GeoJSON and authorized source/snapshot hashes
|
||||
in an isolated `/tmp/statbel-stage` directory. It did not import or replace a
|
||||
Dataset.
|
||||
|
||||
Boundary:
|
||||
- A passed preflight proves technical compatibility only. Adding a future
|
||||
edition to `POPULATION_URLS` and replacing or importing any immutable Dataset
|
||||
remains an explicit reviewed operator action.
|
||||
|
||||
## Sprint 226 - Governed Statbel population edition probe (2026-07-16)
|
||||
|
||||
Implemented:
|
||||
|
||||
@@ -198,6 +198,30 @@ naar TXT/ZIP en XLSX worden op host, pad, jaar en variant gevalideerd maar niet
|
||||
opgehaald. De afzonderlijke statistische-sectorgeometrie 2026 betekent niet
|
||||
dat er al bevolkingscijfers per sector voor 2026 zijn gepubliceerd.
|
||||
|
||||
Een afzonderlijke import-preflight in
|
||||
`scripts/statbel_population_preflight.py` controleert lokaal gestagede
|
||||
officiele ZIP-archieven voordat de bevolkingsoperator een nieuwe afgeleide
|
||||
snapshot mag aanbieden aan DatasetService. De controle is fail-closed voor
|
||||
bron-URL en editie, ZIP-veiligheid, verplichte kolommen, REDEGEO-layout,
|
||||
EPSG:31370, situatiedatum, geometrie/sectorjoin, gemeentetoewijzing, nationale
|
||||
en scopespecifieke totalen en een begrensde vergelijking met de vorige
|
||||
snapshot. Herstelbare bron-topologiefouten worden uitsluitend met
|
||||
`make_valid` genormaliseerd wanneer geometriesoort en oppervlakte behouden
|
||||
blijven; elk herstel staat in het manifest.
|
||||
|
||||
Vanaf de editie 2025 mag de eerste vijf tekens van `CD_SECTOR` niet als actuele
|
||||
gemeentecode worden gebruikt. Gemeentefusies kunnen een historische
|
||||
sectorcode onder een nieuwe `CD_REFNIS` plaatsen. GeoIntel valideert daarom de
|
||||
expliciete gemeentevelden uit de bevolkings- en geometriebron tegen elkaar.
|
||||
Bevolkingsregels met suffix `ZZZZ` zijn officiele niet-lokaliseerbare totalen:
|
||||
ze tellen mee in de reconciliatie en het manifest, maar niet in kaartselecties
|
||||
of ruimtelijke schattingen.
|
||||
|
||||
Een geslaagde preflight maakt een controlespoor met SHA-256 voor beide
|
||||
bronarchieven en de afgeleide GeoJSON. Zij geeft alleen technische
|
||||
importgeschiktheid aan; zij vervangt of importeert nooit automatisch een
|
||||
bestaande Dataset.
|
||||
|
||||
### Mol population history
|
||||
|
||||
`scripts/provision_mol_population_history.py` imports official Statbel
|
||||
@@ -207,6 +231,12 @@ population-by-statistical-sector tables and matching sector geometries for
|
||||
each year as a separate dataset in
|
||||
`statbel:population-statistical-sector:mol`.
|
||||
|
||||
Nieuwe of met `--force` herbouwde edities doorlopen verplicht de
|
||||
import-preflight. Bestaande historische caches zonder preflightmanifest mogen
|
||||
alleen worden hergebruikt wanneer de overeenkomende Dataset al bestaat; een
|
||||
nieuwe import uit zo'n cache wordt geweigerd totdat de officiele bron opnieuw
|
||||
wordt gestaged en gevalideerd.
|
||||
|
||||
Complete sectors use their published population total. A rectangle that cuts
|
||||
through a sector uses an explicitly labelled area-weighted estimate; the
|
||||
source does not justify a more precise intra-sector distribution.
|
||||
|
||||
@@ -101,6 +101,27 @@ Do not delete originals automatically. Derived outputs may be cleaned through ex
|
||||
|
||||
## Official temporal source artifacts
|
||||
|
||||
Statbel population staging stores immutable official source ZIPs below the
|
||||
selected operator output root:
|
||||
|
||||
```text
|
||||
<population-output>/
|
||||
raw/{year}/OPENDATA_SECTOREN_{year}[_NEW].zip
|
||||
raw/{year}/sh_statbel_statistical_sectors_31370_{year}0101.geojson.zip
|
||||
{scope}_statbel_population_{year}.geojson
|
||||
{scope}_statbel_population_{year}.preflight.json
|
||||
```
|
||||
|
||||
The preflight manifest records official URLs, archive/member size and SHA-256,
|
||||
schema fingerprints, EPSG:31370 and situation-date evidence, bounded topology
|
||||
repairs, national/scope join accounting, unlocated `ZZZZ` totals, baseline
|
||||
change evidence and the final derived-snapshot checksum. The upload operator
|
||||
revalidates all retained files against that manifest. Raw ZIPs and manifests
|
||||
are provenance artifacts; queryable population geometry remains an ordinary
|
||||
Dataset/DatasetVersion plus PostGIS `vector_features` through the canonical
|
||||
persistence services. Temporary standalone preflight output does not create a
|
||||
database record and may be removed explicitly after operator review.
|
||||
|
||||
Waterinfo raw station layers, timeseries responses and checksum manifests live
|
||||
under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
|
||||
evidence; queryable annual Point snapshots are normal Dataset/vector_feature
|
||||
|
||||
@@ -43,6 +43,7 @@
|
||||
- [x] Add a governed regional GRB plan -> stage -> checksum-confirmed apply workflow that preserves every previous snapshot.
|
||||
- [x] Add a fail-closed ALZ publication probe that distinguishes provisional v1/v2 snapshots from the definitive v3 historical edition and never downloads an archive.
|
||||
- [x] Add a fail-closed Statbel DCAT publication probe that distinguishes population year, sector-geometry year and the 2025 REDEGEO transition without downloading distributions.
|
||||
- [x] Add a fail-closed Statbel population import preflight with archive/schema/CRS/join/total/baseline checks, retained checksums and explicit ZZZZ accounting before any new Dataset import.
|
||||
- [ ] Extend catalogue probes only to additional sources that publish a stable official edition contract; do not add background polling or infer releases from HTTP dates alone.
|
||||
|
||||
## Governed source expansion backlog
|
||||
|
||||
Reference in New Issue
Block a user