Add governed ALZ edition probe
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-16 20:27:07 +02:00
parent 5e897206b2
commit 0a23e3484c
18 changed files with 451 additions and 38 deletions
+16 -5
View File
@@ -414,11 +414,15 @@ not marked stale merely because their source date is old. Unknown sources are
### GET `/api/v1/projects/{project_id}/datasets/source-catalog-probes`
Performs an explicit, read-only release probe for the allowlisted GRB WFS and
most-recent orthophoto WMS. The endpoint reads bounded `GetCapabilities`
responses, confirms the expected layers and follows only HTTPS ISO 19139
`GetRecordById` links on `metadata.vlaanderen.be`. Query parameter
`refresh=true` bypasses the short in-memory response cache.
Performs an explicit, read-only release probe for the allowlisted GRB WFS,
most-recent orthophoto WMS and ALZ agricultural-use parcel publication page.
For OGC services, the endpoint reads bounded `GetCapabilities` responses,
confirms the expected layers and follows only HTTPS ISO 19139 `GetRecordById`
links on `metadata.vlaanderen.be`. For ALZ, it reads only the bounded official
HTML release page and accepts only exact
`www.landbouwvlaanderen.be/bestanden/gis/agpa_<year>_<date>_public.zip` link
identities. It does not request those archives. Query parameter `refresh=true`
bypasses the short in-memory response cache.
Each provider item returns service reachability, matched/missing layers, the
official metadata identifier, title, edition, publication/metadata dates,
@@ -427,6 +431,13 @@ local `source_version` and one comparison status: `same`, `different`,
`available`, `degraded`, `unavailable` or `disabled`. A difference means only
that an operator should review provenance; it is not an update instruction.
ALZ comparisons use only the latest definitive third snapshot, normalized as
`<campaign>-v3`. A newer first or second snapshot is reported in the item title
and message as provisional but cannot mark a local definitive historical
edition as outdated. `service_type=HTML` uses `definitive_archive` and
`current_snapshot` as evidence markers in the existing expected/matched/missing
arrays; no parallel response shape is introduced.
The endpoint accepts no arbitrary URL, feature query, area or layer. It does
not fetch vector features, raster pixels or models, create jobs/datasets, write
to PostGIS or trigger an import. The normal `source-freshness` endpoint remains
+8 -1
View File
@@ -173,7 +173,7 @@ gebeurd.
| --- | --- | --- |
| GRB gebouwen/wegen/water/percelen | operationele, expliciete plan-stage-apply refresh met onveranderlijke snapshots | alleen een nieuw officieel gedateerd cataloguseditie na operatorbevestiging ophalen |
| Statbel bevolking | jaarlijkse, expliciete edities in één tijdreeks | een nieuwe publicatie alleen na schema-, sectorgeometrie- en totalencontrole toevoegen |
| ALZ landbouwgebruikspercelen | definitieve jaarlijkse edities 2008-2025; metricvergelijking zonder objectlineage | eerst een stabiele machineleesbare editiebron verifiëren, daarna dezelfde begrensde operatorflow toepassen |
| ALZ landbouwgebruikspercelen | definitieve jaarlijkse edities 2008-2025; expliciete officiële publicatieprobe; metricvergelijking zonder objectlineage | een nieuwere definitieve v3-editie eerst handmatig beoordelen en daarna via de bestaande begrensde operatorflow toevoegen |
| orthofoto | vaste lokale opname per expliciete analysezone; catalogusprobe is alleen een signaal | vluchtjaar, productvariant en dekking vergelijken voordat nieuwe pixels worden opgehaald |
| landgebruik, thematische rasters, DHMV en VMM-scenario's | vaste product-/scenario-edities, geen rolling snapshot | alleen een nieuwe gedocumenteerde producteditie als afzonderlijke Dataset verwerven |
| bodemkaart en historische kaarten | historische referentie-editie | niet als verouderde actuele bron labelen; alleen vervangen bij een officiële inhoudelijke heruitgave |
@@ -183,6 +183,13 @@ Geen van deze regels activeert een browserfetch, scheduler of automatische
vervanging. Een bron wordt pas `refreshable` wanneer versie, dekking, schema,
provenance en een begrensde acquisitieroute afzonderlijk verifieerbaar zijn.
De ALZ-publicatiepagina levert drie campagnesnapshots: v1 en v2 zijn
voorlopig, v3 is de definitieve historische editie. GeoIntel vergelijkt daarom
de lokale `2025-definitive` uitsluitend met de officiële `2025-v3`, terwijl de
actuele `2026-v1` zichtbaar maar niet updategerechtigd blijft. De probe leest
alleen de allowlisted HTML-pagina en linkidentiteiten; ZIP-archieven worden pas
door de afzonderlijke operator opgehaald na menselijke editiebevestiging.
### Mol population history
`scripts/provision_mol_population_history.py` imports official Statbel
+1
View File
@@ -29,6 +29,7 @@ GRB_WFS_URL=
# Explicit read-only source-edition probes (no provider import).
SOURCE_CATALOG_PROBE_ENABLED=true
SOURCE_CATALOG_GRB_WFS_URL=https://geo.api.vlaanderen.be/GRB/wfs
SOURCE_CATALOG_ALZ_RELEASE_URL=https://landbouwcijfers.vlaanderen.be/open-geodata-landbouwgebruikspercelen
SOURCE_CATALOG_PROBE_TIMEOUT_SECONDS=10
SOURCE_CATALOG_PROBE_MAX_RESPONSE_MB=2
SOURCE_CATALOG_PROBE_CACHE_TTL_SECONDS=900
+4 -3
View File
@@ -41,7 +41,8 @@
- [x] Add a read-only source freshness/version audit with explicit publication policies, local integrity checks, operator CLI and compact Status workspace surface.
- [x] Add explicit bounded GRB/orthophoto catalogue release probes using official capabilities and ISO metadata; keep every acquisition separate and operator-controlled.
- [x] Add a governed regional GRB plan -> stage -> checksum-confirmed apply workflow that preserves every previous snapshot.
- [ ] Extend catalogue probes only to additional sources that publish a stable machine-readable edition contract; do not add background polling or infer releases from HTTP dates alone.
- [x] Add a fail-closed ALZ publication probe that distinguishes provisional v1/v2 snapshots from the definitive v3 historical edition and never downloads an archive.
- [ ] Extend catalogue probes only to additional sources that publish a stable official edition contract; do not add background polling or infer releases from HTTP dates alone.
## Governed source expansion backlog
@@ -683,7 +684,7 @@ This file now starts with the current implementation status. Older preparation/b
- [x] Show precise daily edition ranges and explain registration-date nuance in
the Evolution workspace.
- [x] Document refresh readiness across the complete official-source portfolio.
- [ ] Add the next governed catalog probe only after an official stable version
contract is verified; ALZ annual releases are the first candidate.
- [x] Verify and add the governed ALZ catalog probe against the official
campaign-snapshot and definitive-archive publication contract.
- [ ] Keep orthophoto refresh manual until product variant, flight year and
complete selected-area coverage can be compared deterministically.