Add governed ALZ release promotion
This commit is contained in:
@@ -457,6 +457,15 @@ human `review` and checksum-confirmed `apply`. No additional public endpoint
|
||||
is introduced. Apply delegates to the existing dataset upload contract and
|
||||
creates a new immutable annual snapshot only after all evidence is unchanged.
|
||||
|
||||
Future definitive ALZ execution likewise remains outside the HTTP request
|
||||
cycle in `scripts/manage_alz_agriculture_release.py`. It derives the exact
|
||||
`agpa_<campaign>_<publication-date>_public.zip` identity from the existing
|
||||
catalog response and accepts only `YYYY-v3`. `stage` is filesystem-only,
|
||||
`review` is a named approval and `apply` requires exact plan/review hashes,
|
||||
revalidates the current catalog and delegates to the existing Dataset upload
|
||||
contract. No ALZ release endpoint, background task or provider URL parameter
|
||||
is added; v1/v2 campaign snapshots remain non-importable.
|
||||
|
||||
The endpoint accepts no arbitrary URL, feature query, area or layer. It does
|
||||
not fetch vector features, raster pixels or models, create jobs/datasets, write
|
||||
to PostGIS or trigger an import. The normal `source-freshness` endpoint remains
|
||||
|
||||
@@ -1,3 +1,46 @@
|
||||
## Sprint 229 - Governed ALZ definitive release promotion (2026-07-17)
|
||||
|
||||
Implemented:
|
||||
- Added `scripts/manage_alz_agriculture_release.py` with separate read-only
|
||||
`plan`, filesystem-only `stage`, named `review` and exact-hash `apply`
|
||||
actions for the approved Kempen regional scope.
|
||||
- Derived the only accepted future archive from the allowlisted definitive
|
||||
`YYYY-v3` campaign and publication date exposed by the existing source-
|
||||
catalog probe. Provisional v1/v2 snapshots remain visible but non-importable;
|
||||
arbitrary URLs, current/older releases and catalog drift fail closed.
|
||||
- Extended the existing agricultural provisioner to accept exactly one
|
||||
explicitly governed future edition per run. Final response URLs and streamed
|
||||
archive size are bounded, retained editions cannot be overridden and all
|
||||
workspace API collections are read with stable-total pagination.
|
||||
- Bound archive, normalized GeoJSON, GeoPackage schema/CRS, complete crop-code
|
||||
list, scope counts/area and previous-definitive-edition manifest/deltas into
|
||||
the staged plan. ZIP member/extracted size is bounded. Review and apply
|
||||
require exact SHA-256 values; apply still uses DatasetService and preserves
|
||||
all historical snapshots.
|
||||
- Packaged the manager in the Unraid image/readiness gate and updated source,
|
||||
data, API, persistence, storage and operator documentation. No endpoint,
|
||||
migration, scheduler, browser fetch or frontend behavior changed.
|
||||
|
||||
Validation so far:
|
||||
- 43 tests passed across the original ALZ importer, official catalog probe and
|
||||
new release manager; 58 passed together with the Statbel release suite.
|
||||
- Target Python compilation and Ruff pass. Coverage includes future release
|
||||
identity, v1/v2 exclusion, pagination drift, download bounds, release
|
||||
ordering, stage/apply separation, source/schema/codelist/baseline evidence,
|
||||
path confinement, byte tampering, named review, catalog drift, full mocked
|
||||
apply, current-edition refusal and runtime packaging.
|
||||
- Complete readiness passed with 850 backend tests, 110 documented routes,
|
||||
one Alembic head `202607160001`, frontend typecheck and production build.
|
||||
Static full-chain Alembic SQL and shell syntax checks also passed. Local
|
||||
Docker is not installed in the Codex Windows environment; Compose and live
|
||||
PostGIS validation are therefore deferred to the Tower deployment gate.
|
||||
|
||||
Boundary:
|
||||
- The official page currently advertises definitive `2025-v3` plus
|
||||
provisional `2026-v1`. No newer definitive edition exists, so live stage,
|
||||
review and apply must remain blocked; only a read-only plan and deliberate
|
||||
current-edition refusal may be exercised after deployment.
|
||||
|
||||
## Sprint 228 - Governed Statbel population release promotion (2026-07-17)
|
||||
|
||||
Implemented:
|
||||
|
||||
@@ -283,6 +283,14 @@ which creates the ordinary annual `datasets`, `dataset_versions` and
|
||||
year remains idempotent and previous annual snapshots are never updated or
|
||||
deleted.
|
||||
|
||||
Definitive ALZ release management uses the same persistence boundary and adds
|
||||
no migration or release table. Catalog planning, staged archive/GeoJSON,
|
||||
crop-code evidence and named review live on the filesystem. Only an approved
|
||||
apply invokes the existing Dataset upload service, producing the ordinary
|
||||
annual Dataset, DatasetVersion and vector_features records with
|
||||
`source_version=<year>-definitive`. Earlier annual snapshots are retained and
|
||||
provisional v1/v2 publications cannot create rows.
|
||||
|
||||
## Geometry normalization
|
||||
|
||||
- User-drawn polygons arrive as EPSG:4326.
|
||||
|
||||
+23
-1
@@ -173,7 +173,7 @@ gebeurd.
|
||||
| --- | --- | --- |
|
||||
| GRB gebouwen/wegen/water/percelen | operationele, expliciete plan-stage-apply refresh met onveranderlijke snapshots | alleen een nieuw officieel gedateerd cataloguseditie na operatorbevestiging ophalen |
|
||||
| Statbel bevolking | jaarlijkse, expliciete edities in één tijdreeks; officiële DCAT-releaseprobe | een nieuwe publicatie alleen na schema-, sectorgeometrie- en totalencontrole toevoegen |
|
||||
| ALZ landbouwgebruikspercelen | definitieve jaarlijkse edities 2008-2025; expliciete officiële publicatieprobe; metricvergelijking zonder objectlineage | een nieuwere definitieve v3-editie eerst handmatig beoordelen en daarna via de bestaande begrensde operatorflow toevoegen |
|
||||
| ALZ landbouwgebruikspercelen | definitieve jaarlijkse edities 2008-2025; expliciete publicatieprobe en plan-stage-review-apply promotie; metricvergelijking zonder objectlineage | alleen een nieuwere definitieve v3-editie na gestagede schema-/codelijst-/scopecontrole en benoemde review toevoegen |
|
||||
| orthofoto | vaste lokale opname per expliciete analysezone; catalogusprobe is alleen een signaal | vluchtjaar, productvariant en dekking vergelijken voordat nieuwe pixels worden opgehaald |
|
||||
| landgebruik, thematische rasters, DHMV en VMM-scenario's | vaste product-/scenario-edities, geen rolling snapshot | alleen een nieuwe gedocumenteerde producteditie als afzonderlijke Dataset verwerven |
|
||||
| bodemkaart en historische kaarten | historische referentie-editie | niet als verouderde actuele bron labelen; alleen vervangen bij een officiële inhoudelijke heruitgave |
|
||||
@@ -190,6 +190,22 @@ actuele `2026-v1` zichtbaar maar niet updategerechtigd blijft. De probe leest
|
||||
alleen de allowlisted HTML-pagina en linkidentiteiten; ZIP-archieven worden pas
|
||||
door de afzonderlijke operator opgehaald na menselijke editiebevestiging.
|
||||
|
||||
`scripts/manage_alz_agriculture_release.py` beheert zo'n toekomstige
|
||||
definitieve editie in vier afzonderlijke operatorstappen. `plan` vergelijkt
|
||||
read-only de nieuwste v3 met de lokale `*-definitive` Dataset. `stage` vereist
|
||||
de exacte `YYYY-v3`, leidt de archief-URL af uit de gecontroleerde campagne en
|
||||
publicatiedatum en voert de bestaande provisioner uitsluitend met
|
||||
`--force --fetch-only` uit. Het staged plan bindt bronarchief, GeoPackage-
|
||||
schema/CRS, genormaliseerde GeoJSON, gewascodelijst, scope-aantallen en de
|
||||
verschillen met de vorige definitieve editie aan SHA-256.
|
||||
|
||||
`review` vereist een benoemde menselijke goedkeuring van exact dat plan.
|
||||
`apply` vereist de plan- en reviewhash, controleert catalogus en alle bestanden
|
||||
opnieuw en maakt via de bestaande uploadservice hoogstens een nieuwe immutable
|
||||
Dataset. Een v1/v2-snapshot, huidige of oudere editie, cataloguswijziging,
|
||||
gewijzigde bronbyte of evidence buiten de beheerde opslagroot blokkeert de
|
||||
flow. Er is geen scheduler, browserdownload of automatische vervanging.
|
||||
|
||||
De Statbel-probe leest uitsluitend de officiële DCAT Turtle-catalogus en
|
||||
selecteert de nieuwste unieke Nederlandstalige publicatie `Bevolking per
|
||||
statistische sector`. Voor 2025 vereist GeoIntel de nieuwe REDEGEO-indeling;
|
||||
@@ -409,6 +425,12 @@ annual field set, read through the GIS optional dependencies and deleted after
|
||||
the normalized GeoJSON has been built. Geometry is clipped exactly against the
|
||||
persisted scope Area in Lambert 72 and imported only through DatasetService.
|
||||
|
||||
Future definitive editions are accepted only as one explicit release per
|
||||
operator run with an exact official
|
||||
`agpa_<campaign>_<publication-date>_public.zip` identity. Reaching that URL is
|
||||
not sufficient for persistence: the separate release manager must first stage
|
||||
and bind all evidence, receive named approval and revalidate it during apply.
|
||||
|
||||
The regional default creates the series
|
||||
`alz:agricultural-use-parcels:kempen-transport-region`; `--scope mol` creates an
|
||||
independent Mol series. Selection and evolution expose exact intersected
|
||||
|
||||
@@ -307,6 +307,12 @@ official ZIP archive containing a Belgian Lambert 72 GeoPackage. Queryable
|
||||
geometry is clipped against the persisted Area in EPSG:31370 and normalized to
|
||||
EPSG:4326 before canonical `vector_features` persistence.
|
||||
|
||||
A future campaign becomes historical input only when the official publication
|
||||
contract labels it `v3`. Early `v1` and `v2` snapshots are provisional and may
|
||||
not create annual GeoIntel Datasets. The governed release plan binds the exact
|
||||
archive publication date, schema, CRS, crop-code list, scope totals and deltas
|
||||
against the previous definitive edition before named review and apply.
|
||||
|
||||
Stable source fields include `agpakey`, parcel number, declared source area,
|
||||
reference id, spring crop, main crop, official main-crop group, production
|
||||
method and source municipality. All annual source properties remain available,
|
||||
|
||||
@@ -204,6 +204,24 @@ artifact; Dataset and vector_feature rows remain the queryable PostGIS state.
|
||||
The manifest binds source, crop-code list and upload artifact checksums. A
|
||||
checksum conflict with an existing annual Dataset fails closed.
|
||||
|
||||
Future definitive-release decision evidence is stored separately from those
|
||||
source artifacts:
|
||||
|
||||
```text
|
||||
storage/operator-evidence/alz-agriculture-refresh/{scope}/{year}/
|
||||
staged-plan.json
|
||||
review-evidence.json
|
||||
applied-evidence.json
|
||||
```
|
||||
|
||||
The staged plan binds the official publication-page SHA-256 and definitive
|
||||
archive identity to the retained archive, normalized GeoJSON, schema/CRS,
|
||||
crop-code list, scope accounting and previous-edition delta. Review evidence
|
||||
binds a named decision to the exact plan hash. Applied evidence binds both to
|
||||
the resulting immutable Dataset id. Evidence paths outside this root and
|
||||
source artifacts outside `agricultural-use-parcels` are rejected; these files
|
||||
authorize neither deletion nor in-place Dataset replacement.
|
||||
|
||||
Buildings and Addresses Register snapshot evidence lives under:
|
||||
|
||||
```text
|
||||
|
||||
@@ -45,6 +45,7 @@
|
||||
- [x] Add a fail-closed Statbel DCAT publication probe that distinguishes population year, sector-geometry year and the 2025 REDEGEO transition without downloading distributions.
|
||||
- [x] Add a fail-closed Statbel population import preflight with archive/schema/CRS/join/total/baseline checks, retained checksums and explicit ZZZZ accounting before any new Dataset import.
|
||||
- [x] Add an explicit Statbel population plan -> stage -> named review -> checksum-confirmed apply workflow that preserves every prior annual snapshot.
|
||||
- [x] Add an explicit definitive ALZ plan -> stage -> named review -> checksum-confirmed apply workflow while keeping v1/v2 campaign snapshots non-importable.
|
||||
- [ ] Extend catalogue probes only to additional sources that publish a stable official edition contract; do not add background polling or infer releases from HTTP dates alone.
|
||||
|
||||
## Governed source expansion backlog
|
||||
|
||||
Reference in New Issue
Block a user