Add official source edition probes
GeoIntel CI / docs-smoke (push) Canceled after 0s
GeoIntel CI / contract-smoke (push) Canceled after 0s

This commit is contained in:
Codex
2026-07-16 17:33:40 +02:00
parent fd05c46e2a
commit 595967f892
27 changed files with 1513 additions and 18 deletions
+20
View File
@@ -412,6 +412,26 @@ Dataset or silently refreshes a publication. Fixed editions and scenarios are
not marked stale merely because their source date is old. Unknown sources are
`review_required` until an explicit publication policy is defined.
### GET `/api/v1/projects/{project_id}/datasets/source-catalog-probes`
Performs an explicit, read-only release probe for the allowlisted GRB WFS and
most-recent orthophoto WMS. The endpoint reads bounded `GetCapabilities`
responses, confirms the expected layers and follows only HTTPS ISO 19139
`GetRecordById` links on `metadata.vlaanderen.be`. Query parameter
`refresh=true` bypasses the short in-memory response cache.
Each provider item returns service reachability, matched/missing layers, the
official metadata identifier, title, edition, publication/metadata dates,
local `source_version` and one comparison status: `same`, `different`,
`not_comparable`, `no_local_data` or `unavailable`. Provider status is
`available`, `degraded`, `unavailable` or `disabled`. A difference means only
that an operator should review provenance; it is not an update instruction.
The endpoint accepts no arbitrary URL, feature query, area or layer. It does
not fetch vector features, raster pixels or models, create jobs/datasets, write
to PostGIS or trigger an import. The normal `source-freshness` endpoint remains
local-only and never invokes this probe implicitly.
### GET `/api/v1/projects/{project_id}/datasets/{dataset_id}`
Return metadata.
+37
View File
@@ -9302,3 +9302,40 @@ Next:
- Add machine-readable catalogue release probes only for providers with stable
official version endpoints, keeping every acquisition an explicit bounded
operator action.
## Sprint 222 - Official source edition probes (2026-07-16)
Implemented:
- Verified the public machine-readable contracts published by Digitaal
Vlaanderen. GRB WFS and OMWRGBMRVL WMS capabilities expose stable ISO 19139
CSW records containing real source editions and metadata dates.
- Added a canonical project endpoint that reads only allowlisted capabilities
and linked `metadata.vlaanderen.be` `GetRecordById` responses. It confirms
GRB `GBG/WBN/WGO/ADP` and orthophoto `Ortho/Vliegdagcontour` availability,
then reports the official edition beside local `source_version` evidence.
- Added strict HTTP(S)/metadata allowlists, timeout and byte limits,
post-redirect validation, SHA-256 capabilities evidence, provider-isolated
degraded/unavailable states and a short in-memory cache.
- Added an explicit Status action and operator CLI flags. Neither path runs on
page load, starts a job, downloads source data or changes a Dataset.
- Added Compose, all-in-one Unraid and DockerMan controls for the bounded probe.
Validation:
- Deterministic tests cover official XML parsing, edition comparison,
provider failure isolation, metadata URL rejection, size limits, caching,
disabled mode and canonical envelopes.
- Frontend typecheck/build and targeted backend tests pass before the full
release and Tower validation recorded in the delivery result.
Boundaries:
- The general Datavindplaats API requires authentication and is not scraped or
silently bypassed. GeoIntel uses only public metadata records advertised by
the official OGC services.
- A version difference remains a manual provenance-review signal. No automatic
import, refresh scheduler, GRB feature fetch or orthophoto pixel request was
added.
Next:
- Extend the same probe pattern only where another official source publishes a
stable machine-readable edition. Do not infer release versions from service
protocol versions, ETags or HTTP modification dates.
+7
View File
@@ -25,6 +25,13 @@ YOLO_MAX_DETECTIONS=1000
YOLO_DUPLICATE_IOU_THRESHOLD=0.5
ENABLE_GRB_WFS=false
GRB_WFS_URL=
# Explicit read-only source-edition probes (no provider import).
SOURCE_CATALOG_PROBE_ENABLED=true
SOURCE_CATALOG_GRB_WFS_URL=https://geo.api.vlaanderen.be/GRB/wfs
SOURCE_CATALOG_PROBE_TIMEOUT_SECONDS=10
SOURCE_CATALOG_PROBE_MAX_RESPONSE_MB=2
SOURCE_CATALOG_PROBE_CACHE_TTL_SECONDS=900
OSM_OVERPASS_URL=https://overpass-api.de/api/interpreter
```
+2 -1
View File
@@ -39,7 +39,8 @@
- [x] Execute and audit the complete 336-product VMM and 56-product DHMV regional runtime matrices.
- [x] Present municipal DHMV/VMM partitions as logical regional layers and analyse cross-boundary rectangles without a municipality prerequisite.
- [x] Add a read-only source freshness/version audit with explicit publication policies, local integrity checks, operator CLI and compact Status workspace surface.
- [ ] Add external catalogue release probes only after each official provider exposes a stable machine-readable version contract; keep every refresh explicit and bounded.
- [x] Add explicit bounded GRB/orthophoto catalogue release probes using official capabilities and ISO metadata; keep every acquisition separate and operator-controlled.
- [ ] Extend catalogue probes only to additional sources that publish a stable machine-readable edition contract; do not add background polling or infer releases from HTTP dates alone.
## Governed source expansion backlog