Add governed ALZ release promotion
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-17 00:53:34 +02:00
parent 904f8bd9d2
commit ad2c3481c4
15 changed files with 1357 additions and 19 deletions
+9
View File
@@ -457,6 +457,15 @@ human `review` and checksum-confirmed `apply`. No additional public endpoint
is introduced. Apply delegates to the existing dataset upload contract and
creates a new immutable annual snapshot only after all evidence is unchanged.
Future definitive ALZ execution likewise remains outside the HTTP request
cycle in `scripts/manage_alz_agriculture_release.py`. It derives the exact
`agpa_<campaign>_<publication-date>_public.zip` identity from the existing
catalog response and accepts only `YYYY-v3`. `stage` is filesystem-only,
`review` is a named approval and `apply` requires exact plan/review hashes,
revalidates the current catalog and delegates to the existing Dataset upload
contract. No ALZ release endpoint, background task or provider URL parameter
is added; v1/v2 campaign snapshots remain non-importable.
The endpoint accepts no arbitrary URL, feature query, area or layer. It does
not fetch vector features, raster pixels or models, create jobs/datasets, write
to PostGIS or trigger an import. The normal `source-freshness` endpoint remains
+43
View File
@@ -1,3 +1,46 @@
## Sprint 229 - Governed ALZ definitive release promotion (2026-07-17)
Implemented:
- Added `scripts/manage_alz_agriculture_release.py` with separate read-only
`plan`, filesystem-only `stage`, named `review` and exact-hash `apply`
actions for the approved Kempen regional scope.
- Derived the only accepted future archive from the allowlisted definitive
`YYYY-v3` campaign and publication date exposed by the existing source-
catalog probe. Provisional v1/v2 snapshots remain visible but non-importable;
arbitrary URLs, current/older releases and catalog drift fail closed.
- Extended the existing agricultural provisioner to accept exactly one
explicitly governed future edition per run. Final response URLs and streamed
archive size are bounded, retained editions cannot be overridden and all
workspace API collections are read with stable-total pagination.
- Bound archive, normalized GeoJSON, GeoPackage schema/CRS, complete crop-code
list, scope counts/area and previous-definitive-edition manifest/deltas into
the staged plan. ZIP member/extracted size is bounded. Review and apply
require exact SHA-256 values; apply still uses DatasetService and preserves
all historical snapshots.
- Packaged the manager in the Unraid image/readiness gate and updated source,
data, API, persistence, storage and operator documentation. No endpoint,
migration, scheduler, browser fetch or frontend behavior changed.
Validation so far:
- 43 tests passed across the original ALZ importer, official catalog probe and
new release manager; 58 passed together with the Statbel release suite.
- Target Python compilation and Ruff pass. Coverage includes future release
identity, v1/v2 exclusion, pagination drift, download bounds, release
ordering, stage/apply separation, source/schema/codelist/baseline evidence,
path confinement, byte tampering, named review, catalog drift, full mocked
apply, current-edition refusal and runtime packaging.
- Complete readiness passed with 850 backend tests, 110 documented routes,
one Alembic head `202607160001`, frontend typecheck and production build.
Static full-chain Alembic SQL and shell syntax checks also passed. Local
Docker is not installed in the Codex Windows environment; Compose and live
PostGIS validation are therefore deferred to the Tower deployment gate.
Boundary:
- The official page currently advertises definitive `2025-v3` plus
provisional `2026-v1`. No newer definitive edition exists, so live stage,
review and apply must remain blocked; only a read-only plan and deliberate
current-edition refusal may be exercised after deployment.
## Sprint 228 - Governed Statbel population release promotion (2026-07-17)
Implemented:
+8
View File
@@ -283,6 +283,14 @@ which creates the ordinary annual `datasets`, `dataset_versions` and
year remains idempotent and previous annual snapshots are never updated or
deleted.
Definitive ALZ release management uses the same persistence boundary and adds
no migration or release table. Catalog planning, staged archive/GeoJSON,
crop-code evidence and named review live on the filesystem. Only an approved
apply invokes the existing Dataset upload service, producing the ordinary
annual Dataset, DatasetVersion and vector_features records with
`source_version=<year>-definitive`. Earlier annual snapshots are retained and
provisional v1/v2 publications cannot create rows.
## Geometry normalization
- User-drawn polygons arrive as EPSG:4326.
+23 -1
View File
@@ -173,7 +173,7 @@ gebeurd.
| --- | --- | --- |
| GRB gebouwen/wegen/water/percelen | operationele, expliciete plan-stage-apply refresh met onveranderlijke snapshots | alleen een nieuw officieel gedateerd cataloguseditie na operatorbevestiging ophalen |
| Statbel bevolking | jaarlijkse, expliciete edities in één tijdreeks; officiële DCAT-releaseprobe | een nieuwe publicatie alleen na schema-, sectorgeometrie- en totalencontrole toevoegen |
| ALZ landbouwgebruikspercelen | definitieve jaarlijkse edities 2008-2025; expliciete officiële publicatieprobe; metricvergelijking zonder objectlineage | een nieuwere definitieve v3-editie eerst handmatig beoordelen en daarna via de bestaande begrensde operatorflow toevoegen |
| ALZ landbouwgebruikspercelen | definitieve jaarlijkse edities 2008-2025; expliciete publicatieprobe en plan-stage-review-apply promotie; metricvergelijking zonder objectlineage | alleen een nieuwere definitieve v3-editie na gestagede schema-/codelijst-/scopecontrole en benoemde review toevoegen |
| orthofoto | vaste lokale opname per expliciete analysezone; catalogusprobe is alleen een signaal | vluchtjaar, productvariant en dekking vergelijken voordat nieuwe pixels worden opgehaald |
| landgebruik, thematische rasters, DHMV en VMM-scenario's | vaste product-/scenario-edities, geen rolling snapshot | alleen een nieuwe gedocumenteerde producteditie als afzonderlijke Dataset verwerven |
| bodemkaart en historische kaarten | historische referentie-editie | niet als verouderde actuele bron labelen; alleen vervangen bij een officiële inhoudelijke heruitgave |
@@ -190,6 +190,22 @@ actuele `2026-v1` zichtbaar maar niet updategerechtigd blijft. De probe leest
alleen de allowlisted HTML-pagina en linkidentiteiten; ZIP-archieven worden pas
door de afzonderlijke operator opgehaald na menselijke editiebevestiging.
`scripts/manage_alz_agriculture_release.py` beheert zo'n toekomstige
definitieve editie in vier afzonderlijke operatorstappen. `plan` vergelijkt
read-only de nieuwste v3 met de lokale `*-definitive` Dataset. `stage` vereist
de exacte `YYYY-v3`, leidt de archief-URL af uit de gecontroleerde campagne en
publicatiedatum en voert de bestaande provisioner uitsluitend met
`--force --fetch-only` uit. Het staged plan bindt bronarchief, GeoPackage-
schema/CRS, genormaliseerde GeoJSON, gewascodelijst, scope-aantallen en de
verschillen met de vorige definitieve editie aan SHA-256.
`review` vereist een benoemde menselijke goedkeuring van exact dat plan.
`apply` vereist de plan- en reviewhash, controleert catalogus en alle bestanden
opnieuw en maakt via de bestaande uploadservice hoogstens een nieuwe immutable
Dataset. Een v1/v2-snapshot, huidige of oudere editie, cataloguswijziging,
gewijzigde bronbyte of evidence buiten de beheerde opslagroot blokkeert de
flow. Er is geen scheduler, browserdownload of automatische vervanging.
De Statbel-probe leest uitsluitend de officiële DCAT Turtle-catalogus en
selecteert de nieuwste unieke Nederlandstalige publicatie `Bevolking per
statistische sector`. Voor 2025 vereist GeoIntel de nieuwe REDEGEO-indeling;
@@ -409,6 +425,12 @@ annual field set, read through the GIS optional dependencies and deleted after
the normalized GeoJSON has been built. Geometry is clipped exactly against the
persisted scope Area in Lambert 72 and imported only through DatasetService.
Future definitive editions are accepted only as one explicit release per
operator run with an exact official
`agpa_<campaign>_<publication-date>_public.zip` identity. Reaching that URL is
not sufficient for persistence: the separate release manager must first stage
and bind all evidence, receive named approval and revalidate it during apply.
The regional default creates the series
`alz:agricultural-use-parcels:kempen-transport-region`; `--scope mol` creates an
independent Mol series. Selection and evolution expose exact intersected
+6
View File
@@ -307,6 +307,12 @@ official ZIP archive containing a Belgian Lambert 72 GeoPackage. Queryable
geometry is clipped against the persisted Area in EPSG:31370 and normalized to
EPSG:4326 before canonical `vector_features` persistence.
A future campaign becomes historical input only when the official publication
contract labels it `v3`. Early `v1` and `v2` snapshots are provisional and may
not create annual GeoIntel Datasets. The governed release plan binds the exact
archive publication date, schema, CRS, crop-code list, scope totals and deltas
against the previous definitive edition before named review and apply.
Stable source fields include `agpakey`, parcel number, declared source area,
reference id, spring crop, main crop, official main-crop group, production
method and source municipality. All annual source properties remain available,
+18
View File
@@ -204,6 +204,24 @@ artifact; Dataset and vector_feature rows remain the queryable PostGIS state.
The manifest binds source, crop-code list and upload artifact checksums. A
checksum conflict with an existing annual Dataset fails closed.
Future definitive-release decision evidence is stored separately from those
source artifacts:
```text
storage/operator-evidence/alz-agriculture-refresh/{scope}/{year}/
staged-plan.json
review-evidence.json
applied-evidence.json
```
The staged plan binds the official publication-page SHA-256 and definitive
archive identity to the retained archive, normalized GeoJSON, schema/CRS,
crop-code list, scope accounting and previous-edition delta. Review evidence
binds a named decision to the exact plan hash. Applied evidence binds both to
the resulting immutable Dataset id. Evidence paths outside this root and
source artifacts outside `agricultural-use-parcels` are rejected; these files
authorize neither deletion nor in-place Dataset replacement.
Buildings and Addresses Register snapshot evidence lives under:
```text
+1
View File
@@ -45,6 +45,7 @@
- [x] Add a fail-closed Statbel DCAT publication probe that distinguishes population year, sector-geometry year and the 2025 REDEGEO transition without downloading distributions.
- [x] Add a fail-closed Statbel population import preflight with archive/schema/CRS/join/total/baseline checks, retained checksums and explicit ZZZZ accounting before any new Dataset import.
- [x] Add an explicit Statbel population plan -> stage -> named review -> checksum-confirmed apply workflow that preserves every prior annual snapshot.
- [x] Add an explicit definitive ALZ plan -> stage -> named review -> checksum-confirmed apply workflow while keeping v1/v2 campaign snapshots non-importable.
- [ ] Extend catalogue probes only to additional sources that publish a stable official edition contract; do not add background polling or infer releases from HTTP dates alone.
## Governed source expansion backlog