Add official source edition probes
This commit is contained in:
@@ -412,6 +412,26 @@ Dataset or silently refreshes a publication. Fixed editions and scenarios are
|
||||
not marked stale merely because their source date is old. Unknown sources are
|
||||
`review_required` until an explicit publication policy is defined.
|
||||
|
||||
### GET `/api/v1/projects/{project_id}/datasets/source-catalog-probes`
|
||||
|
||||
Performs an explicit, read-only release probe for the allowlisted GRB WFS and
|
||||
most-recent orthophoto WMS. The endpoint reads bounded `GetCapabilities`
|
||||
responses, confirms the expected layers and follows only HTTPS ISO 19139
|
||||
`GetRecordById` links on `metadata.vlaanderen.be`. Query parameter
|
||||
`refresh=true` bypasses the short in-memory response cache.
|
||||
|
||||
Each provider item returns service reachability, matched/missing layers, the
|
||||
official metadata identifier, title, edition, publication/metadata dates,
|
||||
local `source_version` and one comparison status: `same`, `different`,
|
||||
`not_comparable`, `no_local_data` or `unavailable`. Provider status is
|
||||
`available`, `degraded`, `unavailable` or `disabled`. A difference means only
|
||||
that an operator should review provenance; it is not an update instruction.
|
||||
|
||||
The endpoint accepts no arbitrary URL, feature query, area or layer. It does
|
||||
not fetch vector features, raster pixels or models, create jobs/datasets, write
|
||||
to PostGIS or trigger an import. The normal `source-freshness` endpoint remains
|
||||
local-only and never invokes this probe implicitly.
|
||||
|
||||
### GET `/api/v1/projects/{project_id}/datasets/{dataset_id}`
|
||||
|
||||
Return metadata.
|
||||
|
||||
@@ -9302,3 +9302,40 @@ Next:
|
||||
- Add machine-readable catalogue release probes only for providers with stable
|
||||
official version endpoints, keeping every acquisition an explicit bounded
|
||||
operator action.
|
||||
|
||||
## Sprint 222 - Official source edition probes (2026-07-16)
|
||||
|
||||
Implemented:
|
||||
- Verified the public machine-readable contracts published by Digitaal
|
||||
Vlaanderen. GRB WFS and OMWRGBMRVL WMS capabilities expose stable ISO 19139
|
||||
CSW records containing real source editions and metadata dates.
|
||||
- Added a canonical project endpoint that reads only allowlisted capabilities
|
||||
and linked `metadata.vlaanderen.be` `GetRecordById` responses. It confirms
|
||||
GRB `GBG/WBN/WGO/ADP` and orthophoto `Ortho/Vliegdagcontour` availability,
|
||||
then reports the official edition beside local `source_version` evidence.
|
||||
- Added strict HTTP(S)/metadata allowlists, timeout and byte limits,
|
||||
post-redirect validation, SHA-256 capabilities evidence, provider-isolated
|
||||
degraded/unavailable states and a short in-memory cache.
|
||||
- Added an explicit Status action and operator CLI flags. Neither path runs on
|
||||
page load, starts a job, downloads source data or changes a Dataset.
|
||||
- Added Compose, all-in-one Unraid and DockerMan controls for the bounded probe.
|
||||
|
||||
Validation:
|
||||
- Deterministic tests cover official XML parsing, edition comparison,
|
||||
provider failure isolation, metadata URL rejection, size limits, caching,
|
||||
disabled mode and canonical envelopes.
|
||||
- Frontend typecheck/build and targeted backend tests pass before the full
|
||||
release and Tower validation recorded in the delivery result.
|
||||
|
||||
Boundaries:
|
||||
- The general Datavindplaats API requires authentication and is not scraped or
|
||||
silently bypassed. GeoIntel uses only public metadata records advertised by
|
||||
the official OGC services.
|
||||
- A version difference remains a manual provenance-review signal. No automatic
|
||||
import, refresh scheduler, GRB feature fetch or orthophoto pixel request was
|
||||
added.
|
||||
|
||||
Next:
|
||||
- Extend the same probe pattern only where another official source publishes a
|
||||
stable machine-readable edition. Do not infer release versions from service
|
||||
protocol versions, ETags or HTTP modification dates.
|
||||
|
||||
@@ -25,6 +25,13 @@ YOLO_MAX_DETECTIONS=1000
|
||||
YOLO_DUPLICATE_IOU_THRESHOLD=0.5
|
||||
ENABLE_GRB_WFS=false
|
||||
GRB_WFS_URL=
|
||||
|
||||
# Explicit read-only source-edition probes (no provider import).
|
||||
SOURCE_CATALOG_PROBE_ENABLED=true
|
||||
SOURCE_CATALOG_GRB_WFS_URL=https://geo.api.vlaanderen.be/GRB/wfs
|
||||
SOURCE_CATALOG_PROBE_TIMEOUT_SECONDS=10
|
||||
SOURCE_CATALOG_PROBE_MAX_RESPONSE_MB=2
|
||||
SOURCE_CATALOG_PROBE_CACHE_TTL_SECONDS=900
|
||||
OSM_OVERPASS_URL=https://overpass-api.de/api/interpreter
|
||||
```
|
||||
|
||||
|
||||
+2
-1
@@ -39,7 +39,8 @@
|
||||
- [x] Execute and audit the complete 336-product VMM and 56-product DHMV regional runtime matrices.
|
||||
- [x] Present municipal DHMV/VMM partitions as logical regional layers and analyse cross-boundary rectangles without a municipality prerequisite.
|
||||
- [x] Add a read-only source freshness/version audit with explicit publication policies, local integrity checks, operator CLI and compact Status workspace surface.
|
||||
- [ ] Add external catalogue release probes only after each official provider exposes a stable machine-readable version contract; keep every refresh explicit and bounded.
|
||||
- [x] Add explicit bounded GRB/orthophoto catalogue release probes using official capabilities and ISO metadata; keep every acquisition separate and operator-controlled.
|
||||
- [ ] Extend catalogue probes only to additional sources that publish a stable machine-readable edition contract; do not add background polling or infer releases from HTTP dates alone.
|
||||
|
||||
## Governed source expansion backlog
|
||||
|
||||
|
||||
Reference in New Issue
Block a user