geointel/backend/README.md

# GeoIntel Backend (Sprint 3 foundation layer)

FastAPI backend for the GeoIntel Belgium and Belgian North Sea workbench.

Runtime probes:

- `GET /health/live`: process liveness, always independent from PostgreSQL.
- `GET /health/ready`: fail-closed database/PostGIS/migration/storage
  readiness used by Docker.
- `GET /health`: compatibility alias for readiness.
- `GET /api/v1/system/capabilities`: runtime-derived PostGIS, GIS dependency,
  configured YOLO and provider state.

The all-in-one production runtime enables interrupted job/analysis-run
reconciliation at startup. Local tests and development leave it disabled
unless `GEOINTEL_RECONCILE_INTERRUPTED_RUNS_ON_STARTUP=true`.

The map-first explorer uses the existing persisted vector selection endpoint. Its bounded GeoJSON preview reports `feature_count`, while `total_feature_count` reports the exact PostGIS intersection count before the 1,000-feature response cap. When an active `area_id` is supplied, vector, temporal, derived-dataset and export paths all use `bbox ∩ Area`. A full-work-area bbox resolves to the exact persisted Area geometry; a boundary-crossing rectangle is clipped to the official boundary.

## Scope implemented
- Project CRUD
- Area CRUD with PostGIS geometry
- Vector and raster dataset upload/registration
- Deterministic local storage metadata capture
- PostGIS migration and database foundation
- Job foundation for async-ready GIS operations

## Sprint 2 additions
- Dataset typing and lifecycle support:
  - `uploaded`
  - `validating`
  - `ready`
  - `failed`
- Vector metadata extraction:
  - feature count
  - geometry type summary
  - bounds
  - approximate area
  - CRS and CRS assumption
- Raster metadata endpoint:
  - returns raster profile when `rasterio` is available
  - returns clear `RASTER_PROCESSING_UNAVAILABLE` error when dependency is missing
- Deterministic storage metadata capture:
  - original filename
  - stored filename
  - MIME/content type
  - size bytes
  - checksum SHA-256

## Sprint 3 additions
- Lightweight job architecture:
  - `jobs` table and migrations
  - job create/list/read/status API
  - synchronous execution behind job abstraction
- Vector operations foundation:
  - inspect
  - bbox
  - stats
  - clip by area
  - buffer
  - intersect
  - invalid geometry rejection with typed errors
- Raster operation foundation:
  - inspect
  - metadata
  - preview readiness
  - clip by area (dependency-aware with unavailable fallback)
  - tile generation with manifest output
  - real preview image generation when dependencies are installed

## Sprint 4 additions
- Raster foundation is now implemented with real extraction and deterministic artifact outputs:
  - metadata returns width, height, band count, CRS, bounds, resolution, dtype, nodata, transform
  - preview endpoint generates and reuses PNG previews with width/height
  - clip operation persists a derived raster dataset with:
    - `source_dataset_id`
    - `operation`
    - `operation_parameters`
  - tile operation writes deterministic raster tiles under `tiles/{project_id}/{source_dataset_id}/{tile_set_id}`
  - tile manifest includes tile path, pixel window, bounds, transform, and count
- Dependency behavior:
  - when `rasterio` is missing, raster processing returns `RASTER_PROCESSING_UNAVAILABLE`
  - preview endpoint additionally requires numpy/pillow and returns `RASTER_PROCESSING_UNAVAILABLE` when missing

## Sprint 5 additions
- Raster analytics hardening:
  - raster band statistics now include:
    - min, max, mean, std
    - nodata count and ratio
    - valid pixel count
    - dtype
    - optional histogram bins (default 16 bins)
  - raster reproject operation implemented (CRS transform + rasterio reprojection) using dependency-aware raster processing checks.
  - reproject failures are explicit (`INVALID_PARAMETERS`, `INVALID_DATASET_CRS`, `RASTER_PROCESSING_UNAVAILABLE`).
- Raster clip and tile hardening:
  - clip validates area presence and CRS alignment constraints.
  - tile manifest records `tile_set_id`, `tile_size`, `overlap`, `source_dataset_id`, `source_raster_id`, bounds, parameters, count, tile paths, `ai_inference`, and `tile_server`.
- Job result persistence for raster ops:
  - raster clip/reproject/tile job payloads persist derived dataset references when outputs are produced.

## Sprint 6 additions
- Added local spectral index operations:
  - NDVI endpoint: `POST /raster/indices/ndvi`
  - NDWI endpoint: `POST /raster/indices/ndwi`
  - NDBI endpoint: `POST /raster/indices/ndbi`
- Spectral index input validation:
  - band parameters must be positive integers
  - band parameters must exist in source raster band count
- Dependency-aware execution:
  - returns `RASTER_PROCESSING_UNAVAILABLE` when rasterio or numpy are unavailable
- Real index output handling:
  - local windowed float32 GeoTIFF generation
  - `NaN` strategy for invalid pixels / division by zero
- Provenance capture for derived index datasets:
  - `source_dataset_id`, `operation`, `band_mapping`, `formula`
  - `output_dtype`, `nodata_strategy`, `value_range_note`
  - `output_dataset_id`, `created_at`, `path`

## Sprint 7B additions
- Added provider registry skeleton for `grb`, `osm`, `manual` and `fixture`.
- Added provider capability endpoints:
  - `GET /api/v1/external/providers`
  - `GET /api/v1/external/providers/{provider_name}`
  - `GET /api/v1/external/providers/{provider_name}/layers`
  - `GET /api/v1/external/providers/{provider_name}/status`
  - `POST /api/v1/external/providers/{provider_name}/import`
- GRB and OSM imports return explicit `not_configured` responses; no live WFS or Overpass calls are made.
- Manual and fixture providers describe existing upload/fixture flows only.
- Added live PostGIS migration smoke script for environments with a real database:

```bash
bash scripts/live_migration_smoke.sh
```

## Sprint 8 additions
- Added Detection Lab foundation:
  - `detections` ORM model and Alembic migration with PostGIS geometry storage.
  - hardened `analysis_runs` for dataset/job/model/result metadata.
  - model registry capability service for `yolo-placeholder` and `manual-fixture-detector`.
  - detection service boundary for creating jobs, analysis runs and dependency-aware unavailable responses.
- Added detection endpoints:
  - `GET /api/v1/detection/models`
  - `GET /api/v1/detection/model-assets`
  - `POST /api/v1/detection/run`
  - `POST /api/v1/detection/run-async` (production browser path)
  - `GET /api/v1/detection/runs/{analysis_run_id}`
  - `GET /api/v1/detection/runs/{analysis_run_id}/detections`
- YOLO/PyTorch real inference is not enabled in Sprint 8.
- Fixture detector mode is test/demo-only and requires explicit `fixture_mode=true`.

## Sprint 8B additions
- Added optional configured YOLO integration foundation:
  - `yolo-configured` model registry capability.
  - import-safe adapter for local Ultralytics model files.
  - raster tile manifest validation and tile limit enforcement.
  - pixel bbox to EPSG:4326 detection polygon conversion.
  - persisted detections through the existing detection/job/analysis-run path.
- YOLO dependencies are optional extras and are not required for backend startup.
- GeoIntel does not download YOLO model weights automatically.

## Sprint 8C additions
- Added detection visualization/review API support:
  - list detection runs
  - list detections by run or dataset with class/confidence filters
  - get detection detail
  - return persisted detections as GeoJSON FeatureCollections
- Added detection QA against reference vector datasets:
  - compares persisted detection geometries against persisted `vector_features`
  - persists `quality_checks` and `metrics`
  - returns precision, recall, F1, mean IoU and false positive/negative counts
  - configured-YOLO runs clip both QA populations to persisted tile-manifest
    coverage before matching and fail closed on missing/mismatched coverage
    provenance
  - persists a diagnostic-only candidate-box versus reference-envelope pass so
    box-to-footprint matching artifacts are visible without altering canonical
    footprint-IoU metrics
- Segmentation, LiDAR, AI Copilot, Training Studio and Reports remain out of scope.

## Sprint 9 additions
- Added Segmentation Lab foundation:
  - `segmentations` ORM model and Alembic migration with PostGIS MultiPolygon geometry storage.
  - segmentation model registry capabilities for `segmentation-placeholder`, `fixture-segmenter`, `yolo-seg-configured` and `sam-configured`.
  - segmentation service boundary for creating jobs, analysis runs and unavailable model responses.
  - explicit fixture segmenter mode for tests/demo fixtures only.
- Added segmentation endpoints:
  - `GET /api/v1/segmentation/models`
  - `POST /api/v1/segmentation/run`
  - `POST /api/v1/segmentation/run-async` (production browser path)
  - `GET /api/v1/segmentation/runs`
  - `GET /api/v1/segmentation/runs/{analysis_run_id}`
  - `GET /api/v1/segmentation/runs/{analysis_run_id}/segmentations`
  - `GET /api/v1/segmentation/runs/{analysis_run_id}/geojson`
  - `POST /api/v1/segmentation/runs/{analysis_run_id}/qa/reference`
- Real SAM and YOLO-seg inference are not enabled in Sprint 9.
- Mask paths are provenance/debug artifacts; persisted PostGIS geometry is authoritative for QA, map display and GeoJSON.
- Current configured detection and segmentation run through the async analysis
  worker (`GEOINTEL_ANALYSIS_WORKER_ENABLED`) and are followed through
  `GET /api/v1/projects/{project_id}/jobs/{job_id}`. The Unraid profile sets
  `YOLO_REQUIRE_CUDA=true`, so both pipelines fail closed instead of silently
  falling back from NVIDIA CUDA to CPU.

## Sprint 17 additions
- Added export foundation backed by the existing `exports` table.
- GeoJSON exports now persist export records and write JSON artifacts for:
  - vector datasets
  - detection analysis runs
  - segmentation analysis runs
- Added project metadata JSON export for project, dataset and QA/QC summary state.
- Added export read/list/content endpoints:
  - `POST /api/v1/exports/geojson`
  - `POST /api/v1/exports/map-result`
  - `POST /api/v1/exports/metadata`
  - `GET /api/v1/exports/projects/{project_id}/exports`
  - `GET /api/v1/exports/{export_id}`
  - `GET /api/v1/exports/{export_id}/content`
- Exported detection and segmentation GeoJSON is generated from persisted first-class geometry rows.
- No new migrations, product lines, live providers or AI dependencies are introduced by this export pass.
- Map-result exports recompute current governed vector/raster selections or
  historical comparisons on the backend before persisting the artifact. Client
  metrics are never accepted as authoritative export content.
- Old offline demo export artifacts can be inspected with `python scripts/cleanup_demo_artifacts.py`
  and removed only with an explicit `--apply`. The script keeps the newest exports
  per demo project, refuses to delete files outside `STORAGE_ROOT`, and blocks
  apply runs above `--max-delete` until the cap is raised after a dry-run review.
  Use repeated `--export-type` values to target only specific artifact kinds.
  In Docker, use `docker compose exec -T backend python scripts/cleanup_demo_artifacts.py`.
- Live cleanup validation is available with `bash scripts/verify_demo_cleanup_dry_run.sh`.
  It runs the same maintenance path without `--apply` and fails if the summary
  reports anything other than a dry-run with zero deleted exports/files.

## Run locally

### Prerequisites

- Python 3.11+
- PostgreSQL with PostGIS

### Install dependencies

```bash
cd backend
python -m pip install -e .[dev]
```

Optional AI dependencies for configured local YOLO inference:

```bash
cd backend
python -m pip install -e .[ai]
```

Docker and Unraid builds keep AI dependencies disabled by default. To build an
image with local PyTorch/Ultralytics support, set:

```bash
GEOINTEL_INSTALL_AI=true
```

The default remains `false` so normal GIS deployments do not install the large AI
runtime. GeoIntel still requires an explicit local model path and never downloads
weights automatically.

AI-enabled Docker images include the native OpenCV runtime libraries required by
Ultralytics. Dependency availability is checked with real `torch` and
`ultralytics` imports, so missing shared libraries are reported as
`dependency_unavailable` instead of being treated as configured.
Docker/Unraid runtimes set `YOLO_CONFIG_DIR` to a writable storage path so
Ultralytics does not attempt to write settings under the root user config
directory.

Configured YOLO requires:

```bash
YOLO_ENABLED=true
YOLO_MODELS_DIR=/absolute/path/to/models
YOLO_MODEL_PATH=/absolute/path/to/local-model.pt
```

Optional local model compatibility smoke:

```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --check-model-load --json
```

In Docker, run the same smoke through the backend container:

```bash
docker compose exec -T backend python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --check-model-load --json
```

In the all-in-one Unraid runtime, place model files under the configured models
directory, mounted as `/app/models` by default:

```bash
GEOINTEL_MODELS_PATH=/mnt/user/appdata/geointel/models
YOLO_ENABLED=true
YOLO_MODELS_DIR=/app/models
YOLO_MODEL_PATH=/app/models/local-model.pt
```

The root helper can write those values safely after a local model is placed:

```bash
python scripts/configure_yolo_model.py \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env \
  --apply
```

When a promotion report recommends an exact model/tile/threshold candidate,
prefer the guarded activation helper. It validates the report, checks the local
model asset and writes `.env` only when `--apply` is supplied:

```bash
python scripts/activate_promoted_yolo_candidate.py \
  --promotion-report /mnt/user/appdata/geointel/artifacts/detection-model-promotion/split-aware/aoi1024bg512r3e50-high-threshold-split-20260710T222934Z/detection_model_promotion_report.json \
  --candidate-key 'geointel-building-yolov8s-aoi1024bg512r3e50-pt|512|64|0.35' \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env \
  --json
```

Add `--apply` only after reviewing the emitted env updates. The smoke and
activation helpers load no model by default, run no inference and do not
download weights. Restart or rebuild the runtime after applying because the
active model is read from `YOLO_MODEL_PATH`.

Operator-only local training preparation is available when real public model
candidates are too weak for the target imagery. It is not a browser feature and
does not change API contracts:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-dataset \
  --val-samples turnhout \
  --force
```

In an AI-enabled runtime with an existing local base model:

```bash
docker exec \
  -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-dataset \
  -e YOLO_BASE_MODEL_PATH=/app/models/yolov8n.pt \
  -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-detector.pt \
  -e TRAIN_EPOCHS=8 \
  -e TRAIN_IMGSZ=512 \
  -e TRAIN_BATCH=2 \
  -e TRAIN_WORKERS=0 \
  -e TRAIN_DEVICE=cpu \
  -e PYTHON_BIN=python3 \
  geointel bash /app/scripts/train_operator_yolo_detector.sh
```

The exporter creates a YOLO `dataset.yaml` plus image/label folders from the
explicit operator sample manifest. The training wrapper writes
`training_summary.json` and a local `.pt` artifact, which still must be
validated through model preflight and the real-data QA matrix before use.

For a larger tile-level training set, use overlapping windows instead of one
image per AOI:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-tile-dataset \
  --tile-size 192 \
  --stride 96 \
  --negative-keep-ratio 0.5 \
  --val-samples turnhout \
  --force
```

Then point `OPERATOR_YOLO_DATASET_DIR` at
`/app/storage/operator-data/yolo-building-tile-dataset` and keep the same
training wrapper. Tile-level output remains operator tooling outside the V1
browser product.

Use `--samples` (or `OPERATOR_YOLO_SAMPLES`) when an experiment needs a
deliberate manifest subset. The generated summary records the source manifest
count plus selected and excluded sample slugs. Unknown samples and any selected
manifest holdout that is omitted from `--val-samples` fail before files are
written.

The backend also exposes a read-only model asset catalog for the mounted model
directory:

```bash
curl http://localhost:1202/api/v1/detection/model-assets
```

The catalog lists local `.pt`, `.onnx` and `.engine` files with size, SHA-256
and active-model status. Detection runs may submit `model_asset_id` with
`model_id="yolo-configured"` to use a cataloged local model for that run. The
backend resolves the ID to a file inside `YOLO_MODELS_DIR`; browser clients do
not send arbitrary model paths.

Configured YOLO inference uses raster tile artifacts from the existing tile
manifest flow. Single-band or otherwise non-RGB tile images are converted to a
temporary RGB prediction image before inference; georeferencing still comes
from the persisted tile manifest transform/bounds metadata.

Optional tuning:

```bash
YOLO_MODEL_ID=yolo-configured
YOLO_MODEL_DISPLAY_NAME="Configured YOLO detector"
YOLO_MODEL_VERSION=local-v1
YOLO_MODELS_DIR=/app/models
YOLO_CONFIG_DIR=/app/storage/ultralytics
YOLO_DEVICE=cpu
YOLO_IMAGE_SIZE=640
YOLO_MAX_TILES=100
YOLO_MAX_DETECTIONS=1000
YOLO_DUPLICATE_IOU_THRESHOLD=0.5
YOLO_BATCH_SIZE=1
```

### YOLO local preflight

Sprint 13 adds a local-only preflight for configured YOLO paths:

```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json
```

Machine-readable output:

```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --json
```

To validate only local model/manifest paths on a machine without optional AI dependencies:

```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --assume-dependencies --json
```

The preflight checks configuration, dependency availability, local model file existence, tile manifest validity, tile count and referenced tile paths. JSON output also includes runtime diagnostics for the model directory, `YOLO_CONFIG_DIR`, installed `torch`/`ultralytics` versions and CUDA availability when dependency checks pass. It does not load a YOLO model, run inference or download weights.

`YOLO_MAX_DETECTIONS` is forwarded to Ultralytics as `max_det`. The default is
`1000` because dense building AOIs can exceed the upstream default cap of 300
detections before QA/QC can measure recall honestly.

`YOLO_DUPLICATE_IOU_THRESHOLD` controls GeoIntel-side cross-tile duplicate
suppression after YOLO pixel boxes are converted to EPSG:4326 polygons and
before `Detection` rows are persisted. Candidates are sorted by confidence per
class; lower-confidence same-class candidates with geometry IoU greater than or
equal to the threshold are suppressed. The default is `0.5`; set `0` to disable
this post-processing for debugging.

The same read-only status is available through the API and Detection Lab UI:

```bash
curl http://localhost:1202/api/v1/detection/yolo/preflight
```

To validate the full configured-YOLO runtime path against Docker/Tower after a
model is mounted and selected, run:

```bash
bash scripts/verify_model_asset_detection_workflow.sh http://192.0.2.10:1202
```

The smoke uses the existing demo raster to generate a tile manifest, selects a
cataloged local model asset, verifies read-only preflight, submits the existing
detection run endpoint and checks persisted AnalysisRun, Detection list and
Detection GeoJSON output. It does not download weights or inject detector
fixtures. A zero detection result is still a valid runtime smoke outcome on the
synthetic demo raster.

To validate the configured building model on operator-provided GIS data, mount
or copy a real georeferenced raster and a real reference-building GeoJSON onto
the runtime host, then run:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/data/orthophoto.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/data/reference-buildings.geojson \
bash scripts/verify_real_data_detection_qa_workflow.sh http://192.0.2.10:1202
```

This smoke refuses missing/unsupported inputs, uploads the raster and reference
dataset through the normal dataset service, generates raster tiles, selects a
local model asset, runs configured YOLO detection, compares persisted
detections against persisted `vector_features`, persists QA/QC rows and exports
the detection GeoJSON. It never seeds demo detections, enables fixture mode,
fetches live providers or downloads model weights. Configured-YOLO model class
labels are normalized to lowercase for filtering and persisted detections while
the original model label is retained in detection provenance. Raster tile
manifests generated for AI handoff include source CRS metadata so pixel-space
model outputs can be transformed to WGS84 GeoJSON coordinates. Current V1 upload
support is limited to GeoTIFF-style rasters and GeoJSON/JSON reference vectors.

When `REAL_AREA_BBOX=minx,miny,maxx,maxy` is supplied, the same workflow also
persists an EPSG:4326 project Area before uploading data. `REAL_AREA_NAME` and
`REAL_PROJECT_REGION` retain operator context. The multi-sample runner fills
these values from manifest `wgs84_bbox` and municipality metadata, so generated
projects are immediately usable in the map without an alternate persistence
path or API contract.

To prepare the documented operator sample corpus inside the all-in-one runtime
container, run:

```bash
docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py
```

The helper writes GeoTIFF orthophotos, GRB GBG building GeoJSON files and
`operator_samples_manifest.json` under `/app/storage/operator-data`. In
addition to the established positive and background AOIs, the registry contains
Beerse, Rijkevorsel, Hoogstraten and Vorselaar as focused small-building
training AOIs. Vosselaar and Grobbendonk are independent validation AOIs and
must not be exported into the training split. Background candidates can persist
empty GRB FeatureCollections for negative-tile training; normal reference AOIs
still fail when GRB returns no buildings. These are runtime artifacts only and
are not committed to Git.

Mol additionally has operational holdouts for Achterbos, Gompel, Donk and
Postel, with Mol center as the historical baseline and Postel-bos as a separate
background control. Prepare and execute that pack with the documented
`prepare_operator_real_data_samples.py` and
`run_mol_operational_validation.sh` commands in `scripts/README.md`. The runner
produces a coverage-aware operational decision report: canonical footprint-IoU
metrics remain authoritative, reference-envelope matches remain diagnostic,
and no report can activate or mutate a model asset.

For municipality-wide navigation, run
`/app/scripts/provision_mol_municipality_workspace.py` inside the all-in-one
container. It verifies the official Mol boundary (NIS `13025`), pages and clips
all GRB GBG buildings, records checksums/provenance under persistent operator
storage and imports both datasets through the existing HTTP service boundary.
The command is explicit and idempotent; it is never executed during backend
startup. See `scripts/README.md` for exact usage and refresh controls.

The current recommended local building model is
`geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt` with tile size
`512`, overlap `64` and confidence threshold `0.15`. Its SHA256 is
`a9088b8491dfae36694b53e9e9406cb4e3511d334a5712fa34f75078a47759c1`.
The promotion evidence covers seven positive AOIs at QA match IoU `0.25` and
three pure-empty background AOIs. The model improves recall and persisted
false-negative counts, but has lower precision than the previous balanced
model; operators must review and persist QA/QC rather than treating detections
as ground truth.

The latest coverage-aligned rerun of this exact profile measured mean precision
`0.6141`, recall `0.6062` and F1 `0.6069` over Mol Achterbos, Donk, Gompel and
Postel plus Retie, Turnhout and Westerlo. The three pure-empty controls remained
at zero detections. A reviewed six-AOI fine-tuning challenger reached mean F1
`0.6248` but remained inactive because it produced two false detections in the
Postel-bos empty control.

For model-quality calibration, run the confidence sweep wrapper:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
CALIBRATION_THRESHOLDS="0.50 0.35 0.25 0.15" \
bash scripts/run_detection_calibration_sweep.sh http://192.0.2.10:1202
```

The sweep creates one real persisted workflow run per threshold, fetches the
persisted `QualityCheck`/`Metric` rows and writes a `calibration_summary.json`
with persisted detection count, raw candidate count, suppressed duplicate count,
duplicate IoU threshold, score, precision, recall, F1, mean IoU and false
positive/negative counts. It is intended to tune confidence/IoU/model choices,
not to add new inference behavior.

To compare local model assets and tile settings as well as thresholds, run the
quality matrix wrapper:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_detection_quality_matrix.sh http://192.0.2.10:1202
```

The matrix creates one real persisted workflow run per combination and writes
`quality_matrix_summary.json` with the selected model asset, tile size, tile
overlap, threshold, detection count, QA score, precision, recall, F1, mean IoU
and false-positive/false-negative counts. It ranks `best_by_score`,
`best_by_recall` and `best_by_precision`. It does not download weights, create
fake detections, fetch live providers or change backend API behavior.

To aggregate the same matrix over every prepared operator sample, run:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.0.2.10:1202
```

The combined `multi_sample_quality_summary.json` reports per-sample and overall
best configurations. It is an operator benchmarking command, not a backend API
or provider import path.

Before promoting any local model as a default, also run the hard-negative
matrix against the documented background candidates:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
QUALITY_TILE_SIZES="640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
bash scripts/run_operator_hard_negative_detection_matrix.sh http://192.0.2.10:1202
```

This path uploads only background rasters, runs configured-YOLO detection and
counts detections as false-positive pressure. It does not upload reference
vectors or run QA/QC, so it cannot produce fake precision/recall metrics for
empty background AOIs.

To inspect the evidence behind a calibration run, export the persisted QA
evidence bundle:

```bash
CALIBRATION_SUMMARY_PATH=/mnt/user/appdata/geointel/artifacts/detection-calibration/20260707T002103Z/calibration_summary.json \
bash scripts/export_detection_calibration_evidence.sh http://192.0.2.10:1202
```

The bundle writes combined QA evidence GeoJSON plus a standalone HTML/SVG review
artifact that separates matched detections, matched references, false positives
and false negatives by role. It reads existing persisted `QualityCheck` evidence
only and does not rerun inference.

For source-image review of false negatives, run
`scripts/render_detection_false_negative_review_contact_sheets.py` against a
fixed-threshold evidence portfolio. It uses the selected run's persisted tile
manifest, overlays candidate/reference context and explicitly exports reference
features outside tile coverage. The command is read-only and never changes
`QualityCheck`, `Metric`, `Detection` or model state.

### Run backend

```bash
cd backend
python -m uvicorn app.main:app --reload
```

### Run backend tests

```bash
cd backend
python -m pytest
```

For warning-sensitive release checks, the backend is expected to pass with Python deprecation warnings promoted to errors for the timestamp-heavy service paths:

```bash
cd backend
python -m pytest -W error::DeprecationWarning tests/test_geojson_dataset_service.py tests/test_qa_service.py tests/test_sprint7a_persistence_foundation.py tests/test_sprint8c_detection_visualization_qa.py tests/test_sprint9_segmentation_foundation.py tests/test_vector_operations_service.py
```

The repository readiness gate now applies the same warning policy to the full backend suite:

```bash
bash scripts/run_readiness_check.sh
```

That readiness gate also runs the API contract smoke check before backend/frontend compilation and tests.

The RC contract gate loads the generated FastAPI OpenAPI document and requires
every successful JSON operation to expose a concrete Pydantic response schema
inside the canonical `{"data": ...}` envelope. Run it directly with:

```bash
python scripts/audit_api_contracts.py
```

The only tracked non-envelope operations are the three health probes, the four
persisted raster PNG responses and the streamed export download. A newly added
free-form JSON response or undocumented exception fails both the focused RC-7
test and the repository readiness gate.

### Golden QA/QC benchmark

Sprint 12 includes a deterministic QA/QC regression benchmark using explicit fixture data:

```bash
python scripts/run_golden_qa_benchmark.py
```

Machine-readable output:

```bash
python scripts/run_golden_qa_benchmark.py --json
```

Shell wrapper used by release-readiness checks:

```bash
bash scripts/verify_golden_qa_benchmark.sh
```

The benchmark compares `fixtures/golden/predicted_buildings.geojson` against `fixtures/golden/reference_buildings.geojson` and fails on metric drift. Expected baseline:

- precision: `0.5`
- recall: `0.5`
- F1: `0.5`
- mean IoU: `0.8339768339761133`
- false positives: `1`
- false negatives: `1`

The command uses existing QA/QC service logic and verifies `QualityCheck`/`Metric` persistence through an in-memory test session. It does not require live providers, AI models, Docker or PostGIS.

`scripts/run_readiness_check.sh` runs this benchmark automatically, so any
change that alters the golden QA/QC metric baseline must update the fixture and
expected metrics deliberately.

### Demo workflow seed

Sprint 15 adds an explicit offline demo workflow seed. It creates or returns a
demo project, AOI, fixture reference buildings, fixture candidate buildings and
a persisted QA/QC result. It does not fetch live GRB/OSM data and does not run
AI inference.

API:

```bash
curl -X POST http://localhost:1202/api/v1/demo/workflow
```

CLI:

```bash
python scripts/seed_demo_workflow.py --json
```

In Docker Compose on a LAN host:

```bash
curl -X POST http://192.0.2.10:1202/api/v1/demo/workflow
```

### QA/QC result listing

Persisted project quality checks and metric rows can be listed with:

```bash
curl http://localhost:1202/api/v1/projects/{project_id}/quality-checks
```

The frontend QA/QC Results panel uses this endpoint after loading the demo
workflow or running QA.

Detection QA evidence can be reviewed without changing its persisted metrics:

```bash
curl "http://localhost:1202/api/v1/projects/{project_id}/quality-checks/{quality_check_id}/reviews?reviewed=false&limit=50"

curl -X POST "http://localhost:1202/api/v1/projects/{project_id}/quality-checks/{quality_check_id}/reviews" \
  -H "Content-Type: application/json" \
  -d '{"evidence_role":"false_positive","evidence_feature_id":"DETECTION_UUID","decision":"qa_alignment_mismatch","notes":"Box and footprint represent the same building."}'
```

The list is derived from persisted quality-check evidence and paginates at a
maximum of 200 rows. The upsert verifies project ownership, quality-check type,
role-specific decisions and persisted Detection/VectorFeature ownership.
`detection_reviews` never mutates model output, reference geometry or canonical
Metric rows. Evidence GeoJSON queries only stored evidence ids instead of a
complete regional GRB dataset.

### Export foundation

Persisted exports can be created from the existing workbench state:

```bash
curl -X POST http://localhost:1202/api/v1/exports/metadata \
  -H "Content-Type: application/json" \
  -d '{"project_id":"PROJECT_UUID"}'
```

Vector dataset GeoJSON export:

```bash
curl -X POST http://localhost:1202/api/v1/exports/geojson \
  -H "Content-Type: application/json" \
  -d '{"export_kind":"dataset","dataset_id":"DATASET_UUID"}'
```

Detection or segmentation run GeoJSON export:

```bash
curl -X POST http://localhost:1202/api/v1/exports/geojson \
  -H "Content-Type: application/json" \
  -d '{"export_kind":"detection_run","analysis_run_id":"ANALYSIS_RUN_UUID"}'
```

List and inspect exports:

```bash
curl http://localhost:1202/api/v1/exports/projects/PROJECT_UUID/exports
curl http://localhost:1202/api/v1/exports/EXPORT_UUID/content
```

Download an artifact as a browser/file response:

```bash
curl -OJ http://localhost:1202/api/v1/exports/EXPORT_UUID/download
```

Create a lightweight HTML project report artifact:

```bash
curl -X POST http://localhost:1202/api/v1/exports/report \
  -H "Content-Type: application/json" \
  -d '{"project_id":"PROJECT_UUID"}'
```

The report contains project, dataset, QA/QC summary and export history state
only. It is not a PDF designer and does not add a separate reporting module.

After rebuilding a Docker/LAN deployment, verify the end-to-end demo and export
flow through the browser-facing frontend proxy:

```bash
bash scripts/verify_demo_export_workflow.sh http://192.0.2.10:1202
```

The script seeds the explicit demo workflow, verifies persisted QA/QC results,
creates metadata/report/vector GeoJSON exports, lists exports and downloads the
JSON/GeoJSON/HTML artifacts.

### Backend import smoke

```bash
cd backend
python -c "from app.main import app; print(app.title)"
```

### Dockerized backend

```bash
docker compose up --build backend db
```

The Docker Compose stack does not require a root `.env` file for the default local runtime. The database service exposes a container-internal Postgres healthcheck, and the backend also runs `docker_start.sh`, which retries an actual SQL `SELECT 1` connection before running `python -m alembic upgrade head` and starting Uvicorn.

PostGIS is not published on the host `5432` port by default. This avoids conflicts with existing Postgres/PostGIS services on NAS or server hosts. The backend connects over Docker networking with `db:5432`.

Backend and frontend Docker build contexts exclude dependency folders, build outputs and Python bytecode caches via `.dockerignore`.

The Docker Compose frontend is published at `http://localhost:1202`.

Compose healthchecks are enabled for all runtime services:

- `db` uses `pg_isready`.
- `backend` checks `http://127.0.0.1:8000/health` inside the container.
- `frontend` checks `http://127.0.0.1/health` through nginx, which also verifies the frontend-to-backend proxy path.

The frontend waits for a healthy backend before starting. Check runtime state:

```bash
docker compose ps
docker compose logs --tail=80 backend
docker compose logs --tail=80 frontend
```

The backend Docker image installs the approved GIS runtime extra (`.[gis]`) so
browser-facing Docker deployments can report raster/vector processing
capabilities accurately:

- `rasterio`
- `numpy`
- `pillow`
- `geopandas`
- `pyogrio`
- GDAL/GEOS/PROJ system libraries

After rebuilding the backend image, verify the LAN/browser runtime from the
repository root:

```bash
bash scripts/verify_gis_runtime.sh http://localhost:1202
```

On a NAS or server host, use the published LAN URL:

```bash
bash scripts/verify_gis_runtime.sh http://192.0.2.10:1202
```

The script calls `/api/v1/system/capabilities` through the frontend proxy and
fails if `postgis`, `rasterio` or `geopandas` are not reported as available.

The backend Docker build also runs:

```bash
python scripts/gis_import_smoke.py
```

Inside the backend Docker build context this resolves to
`backend/scripts/gis_import_smoke.py`. The root `scripts/gis_import_smoke.py`
wrapper calls the same smoke locally. The smoke imports `rasterio`, `geopandas`
and `pyogrio`; if one of those imports fails, the backend image build fails
before deployment.

### Live Docker/PostGIS migration smoke

Sprint 11 validates the real PostGIS runtime path with the existing database service. From the repository root:

```bash
docker compose config
docker compose up -d db
DATABASE_URL=postgresql+psycopg://geointel:geointel@localhost:5432/geointel bash scripts/live_migration_smoke.sh
```

The smoke script:

- opens a backend SQLAlchemy connection and runs `SELECT 1`
- runs `alembic upgrade head`
- checks `PostGIS_Version()` after migrations have created the extension
- verifies one Alembic head
- verifies required migrated tables and GiST indexes exist

Expected local environment:

```bash
DATABASE_URL=postgresql+psycopg://geointel:geointel@localhost:5432/geointel
```

If the database is not reachable, confirm Docker Desktop is running and that port `5432` is not already occupied. To clean up the local database container without deleting the named volume:

```bash
docker compose stop db
```

To remove the local PostGIS volume as well, use only when you explicitly want a fresh database:

```bash
docker compose down -v
```

## Key docs
- `docs/API_CONTRACTS.md`
- `docs/DATABASE_IMPLEMENTATION_PLAN.md`
- `docs/DEFINITION_OF_DONE.md`
- `docs/40-build-launch/SPRINT_1_SCOPE_FREEZE.md`

## Raster dependency note

Raster metadata and raster operations depend on local GDAL/rasterio availability.

To enable raster processing locally:

```bash
python -m pip install rasterio
```

If `rasterio` is unavailable:
- raster metadata responses return `503` with `RASTER_PROCESSING_UNAVAILABLE`
- raster clip/tile endpoints return explicit unavailable responses

## Export report artifact

`POST /api/v1/exports/report` creates the existing lightweight
`project_report_html` artifact. The report is a self-contained HTML handoff
view rendered from persisted project, dataset, QA/QC and export-history state.
It includes readiness scorecards, dataset inventory, QA/QC evidence, artifact
history, known limitations and print-friendly CSS.

This remains a simple HTML export. It does not add a PDF designer, report
builder, live provider fetching or new analysis behavior.

## Vector area selection

`POST /api/v1/projects/{project_id}/datasets/{dataset_id}/vector/select` runs a
read-only EPSG:4326 bbox query against persisted PostGIS `vector_features` and
returns a canonical-envelope GeoJSON FeatureCollection. It is intended for the
Map workspace area-extract flow and does not create derived datasets or export
records by itself.

The same bounded endpoint is the canonical large-layer map delivery path. The
frontend requests at most 1,000 features for the current viewport and surfaces
the response `truncated` flag; the backend does not provide or imply an
unbounded municipality-wide map response.

Selection summaries expose a primary metric plus an additive `metrics` list.
Known persisted themes are aggregated in `EPSG:31370`: building footprints,
forest, water surfaces and parcels return hectares; roads and linear
watercourses return kilometres; population keeps its configured inhabitant
aggregation. Intersecting feature counts remain available as supporting
evidence. Water volume is deliberately unavailable because the current GRB
source has no reliable depth/bathymetry dimension; GeoIntel does not manufacture
volume from 2D polygons.

`POST /api/v1/projects/{project_id}/datasets/{dataset_id}/vector/select/derive`
uses the same persisted `vector_features` selection but writes the result as a
new derived vector dataset. The created dataset uses
`source="operation:selection"`, `source_name="map_selection"` and
`derived_from_dataset_id` for source provenance, stores a GeoJSON artifact and
indexes its features back into `vector_features` for later QA/QC and analysis.

`POST /api/v1/exports/geojson` with `export_kind="vector_selection"` persists
the same bbox-selected FeatureCollection as a normal export record with
`export_type="vector_selection_geojson"`. This creates a handoff artifact only;
it does not create a derived dataset.

## Geographic scope provisioning

The release-candidate national foundation is provisioned explicitly:

```bash
docker exec geointel python /app/scripts/provision_belgium_north_sea_scope.py
```

Use `--fetch-only` to validate official NGI AdminVector, RBINS marine
reporting units and the Belgian Marine Spatial Plan 2026-2034 without changing
application persistence. The normal command creates or reuses
`Belgium and North Sea Workbench`, persists Belgium, all three regions,
territorial sea, EEZ and continental shelf as Areas, and uploads six
checksum-bound reference Datasets through `DatasetService`.

The operator has a fixed URL/layer allowlist, verified TLS, archive and
response-size limits, safe ZIP extraction, complete WFS pagination and
immutable artifact checksums. It does not run at startup and does not write
directly to `vector_features`.

`GET /api/v1/external/coverage/catalog` exposes audited national source
contracts. `POST /api/v1/external/coverage/resolve` intersects a drawn bbox
with persisted legal/administrative Areas and reports a split zone/theme
matrix. `operational` requires a matching `ready` Dataset; integration without
materialized data is only `partial`.

The explicit operator command below provisions the official 28-municipality
Vlaamse vervoerregio Kempen boundary foundation:

```bash
docker exec geointel python /app/scripts/provision_geographic_scope.py \
  --scope kempen-transport-region
```

It reads current `VRBG/Refgem` boundaries, validates every registered name and
NIS code, unions the regional geometry and creates one project, one regional
Area, 28 municipality Areas and two source datasets through the public API.
It never writes directly to PostGIS and does not run on startup. The persisted
scope limitation explicitly distinguishes the transport-policy region from a
cultural or landscape definition of Kempen.

Use `--fetch-only` for a source/geometry/checksum audit. The scope pass does
not fetch thematic GRB, population or land-use data; those remain separate,
bounded operator jobs.

Provision the regional GRB building theme after the scope pass:

```bash
docker exec geointel python /app/scripts/provision_regional_grb_buildings.py \
  --scope kempen-transport-region
```

The operator retains 28 checksummed municipality partitions but exposes one
normal regional reference dataset. `StorageService` copies the combined
artifact without materializing it as upload bytes; `DatasetService` creates
the Dataset and immutable DatasetVersion; `VectorFeatureService` validates and
flushes partition features in bounded batches. The transaction must index the
exact manifest feature count or it rolls back and removes the managed copy.
No public API contract or provider readiness claim is changed by this
operator-only path.

Provision the regional current road, water and parcel context through the
same persistence boundary:

```bash
docker exec geointel python /app/scripts/provision_regional_grb_context.py \
  --scope kempen-transport-region --layers roads water parcels
```

The operator keeps one resumable municipality partition set per theme and
creates one regional reference Dataset per theme. Polygon ownership uses
maximum overlap area; line ownership uses maximum overlap length. It preserves
source geometry dimensions and collection-qualified source IDs, copies the
combined artifact through StorageService and indexes bounded batches through
DatasetService/VectorFeatureService. It does not add API routes, direct SQL or
interactive provider downloads.

## Temporal Mol data and evolution

Dataset uploads accept `temporal_series_key`, `observed_at`, `valid_from`,
`valid_to`, `temporal_granularity` and `source_version`. Every new source or
derived dataset also writes dataset version 1 in the same transaction.

After the Mol municipality workspace is available, import the official source
snapshots explicitly:

```bash
docker exec geointel python /app/scripts/provision_mol_population_history.py
docker exec geointel python /app/scripts/provision_mol_historical_landuse.py
docker exec geointel python /app/scripts/provision_official_landuse_timeseries.py
```

The first command imports Statbel sector population for 2021-2025. The second
imports Digitaal Vlaanderen historical land use for 1778, 1873 and 1969. The
third imports the Departement Omgeving 10 m forest class for 2013, 2016, 2019,
2022 and 2025. All commands are idempotent, use the normal
API/DatasetService flow and retain fetched artifacts in persistent operator
storage. They never run on app startup.

Every newly fetched or `--force` rebuilt Statbel population edition now passes
`statbel_population_preflight.py` before a derived GeoJSON can reach the
upload API. The operator retains both official ZIPs, writes an atomic
preflight manifest and verifies the source and derived SHA-256 values again at
upload time. A passed preflight does not replace an existing Dataset.

The preflight can also be run without downloads or database mutation against
already staged official artifacts:

```bash
docker exec geointel python /app/scripts/statbel_population_preflight.py \
  --year 2025 \
  --layout new \
  --scope kempen-transport-region \
  --population-archive /tmp/OPENDATA_SECTOREN_2025_NEW.zip \
  --population-url https://statbel.fgov.be/sites/default/files/files/opendata/bevolking/sectoren/OPENDATA_SECTOREN_2025_NEW.zip \
  --geometry-archive /tmp/sh_statbel_statistical_sectors_31370_20250101.geojson.zip \
  --geometry-url https://statbel.fgov.be/sites/default/files/files/opendata/Statistische%20sectoren/sh_statbel_statistical_sectors_31370_20250101.geojson.zip \
  --baseline-snapshot /app/storage/operator-data/regional-timeseries/kempen-transport-region/population/kempen_transport_region_statbel_population_2024.geojson \
  --output /app/storage/operator-evidence/statbel-population/2025-kempen.preflight.json
```

The command exits non-zero and emits a stable `error_code` when source
identity, archive safety, schema, CRS, geometry, join, total reconciliation,
scope coverage or the default 5% annualized population-change review limit
fails. `ZZZZ` rows are reconciled as official unlocated population but remain
excluded from spatial metrics. The 2025 REDEGEO contract deliberately compares
explicit municipality fields; it does not assume that `CD_SECTOR` still starts
with the current `CD_REFNIS` after municipal mergers.

Future official editions use the separate four-phase release coordinator. The
project id must belong to `Kempen Regional Workbench`:

```bash
docker exec geointel python /app/scripts/manage_statbel_population_release.py plan \
  --project-id <KEMPEN_PROJECT_ID> \
  --api-url http://127.0.0.1:8000/api/v1 \
  --refresh-catalog

docker exec geointel python /app/scripts/manage_statbel_population_release.py stage \
  --project-id <KEMPEN_PROJECT_ID> \
  --api-url http://127.0.0.1:8000/api/v1 \
  --confirm-edition <YEAR_FROM_PLAN> \
  --confirm-layout <LAYOUT_FROM_PLAN>

docker exec geointel python /app/scripts/manage_statbel_population_release.py review \
  --project-id <KEMPEN_PROJECT_ID> \
  --api-url http://127.0.0.1:8000/api/v1 \
  --confirm-edition <YEAR_FROM_PLAN> \
  --confirm-layout <LAYOUT_FROM_PLAN> \
  --confirm-plan-sha256 <SHA256_FROM_STAGE> \
  --approve --reviewer "<OPERATOR_NAME>" \
  --review-note "Schema, totalen, ZZZZ en geometrieherstel nagekeken"

docker exec geointel python /app/scripts/manage_statbel_population_release.py apply \
  --project-id <KEMPEN_PROJECT_ID> \
  --api-url http://127.0.0.1:8000/api/v1 \
  --confirm-edition <YEAR_FROM_PLAN> \
  --confirm-layout <LAYOUT_FROM_PLAN> \
  --confirm-plan-sha256 <SHA256_FROM_STAGE> \
  --confirm-review-sha256 <SHA256_FROM_REVIEW>
```

`plan` is read-only and creates no file. `stage` always uses bounded fresh
downloads and `--fetch-only`; `review` imports nothing; `apply` revalidates
the current catalog, plan, review, source archives, preflight manifest and
derived snapshot before using the existing upload API. An already-current
release cannot be staged. A repeated successful apply resolves the existing
Dataset through complete paginated lookup rather than creating a duplicate.

Historical land-use work can be bounded explicitly:

```bash
docker exec geointel python /app/scripts/provision_mol_historical_landuse.py --years 1778,1969 --themes forest,water
```

`GET /api/v1/projects/{project_id}/temporal/series` discovers the series and
`POST /api/v1/projects/{project_id}/temporal/compare` compares two snapshots
inside one EPSG:4326 bbox. Partial statistical sectors are estimates; old map
editions without stable identities do not produce invented object changes.
Modern raster-derived forest polygons have the same identity limitation. Their
area is measured in EPSG:31370 and is exact within the 10 m source
representation, not a cadastral forest survey.

The same source-governed operators can synchronize the approved regional
scope in one explicit pass:

```bash
docker exec geointel python /app/scripts/provision_regional_timeseries.py
```

This resolves the retained official boundary and imports five Statbel
population snapshots plus five modern forest, water, built-function and
transport-infrastructure snapshots, followed by the 1778/1873/1969 historical
building, water and road snapshots, into
`Kempen Regional Workbench`. Mol and regional series keys remain separate and
existing immutable datasets are reused. Complete statistical sectors use exact
published totals; a rectangle cutting a sector remains an area-weighted
estimate. Forest area is measured within the official 10 m representation.
Use `--fetch-only` to validate source artifacts without database mutation.
The regional forest path partitions WCS requests by official municipality to
stay within upstream response limits, then builds one retained 10 m mosaic and
one normal regional vector Dataset. A failed source request leaves completed
partition artifacts reusable and never lowers source resolution silently.
Historical WFS retrieval is likewise partitioned by all 28 municipality
boundaries because broad WFS counts stop at 10,000. Exact source responses are
retained as checksummed gzip artifacts before clipping and regional assembly.
Run that stage independently when needed:

```bash
docker exec geointel python /app/scripts/provision_regional_historical_landuse.py
```

Use `--fetch-only` for source/artifact validation without persistence. The
historical building class represents mapped built land-use surfaces, not
individual building footprints; water remains surface area, not depth or
volume; historical roads are mapped road surfaces, not present-day centerline
length.

Official operator datasets record that their geometries were clipped to the
persisted Area. When that exact Area is selected, vector totals and aggregate
metrics use the already clipped geometries directly rather than intersecting
every row with the same detailed boundary again. This optimization is allowed
only for matching Dataset/Area ids with explicit clipping metadata or a known
clipping operator; drawn rectangles and ordinary uploads keep the normal exact
PostGIS intersection path.

## Local Ollama GIS assistant

The optional assistant is a read-only backend integration. It lists locally
installed Ollama models, calculates the active Area/bbox metrics from persisted
PostGIS features and sends only that compact JSON context to Ollama. It never
downloads models, sends geometries or treats model prose as source data.

Configuration:

```text
OLLAMA_ENABLED=true
OLLAMA_BASE_URL=http://host.docker.internal:11434
OLLAMA_DEFAULT_MODEL=qwen3.5:9b
OLLAMA_TIMEOUT_SECONDS=120
OLLAMA_MAX_OUTPUT_TOKENS=1200
OLLAMA_CONTEXT_TOKENS=16384
```

The Unraid deployment adds `host.docker.internal:host-gateway` automatically.
Verify the connection with `GET /api/v1/assistant/status`, inspect installed
models with `GET /api/v1/assistant/models` and ask a grounded question through
`POST /api/v1/projects/{project_id}/assistant/query`. A requested model must be
present in Ollama `/api/tags`. Missing water depth/bathymetry remains explicit;
the assistant cannot turn 2D water geometry into volume. GeoIntel rejects an
answer when Ollama reports `done_reason=length`, so a visibly truncated sentence
is never presented as a complete result. The 1,200-token default leaves enough
room for a compact cross-domain profile while the system prompt requires every
explicitly requested theme and excludes unrelated themes.

## Agricultural-use parcel history

Prepare all definitive 2008-2025 regional editions without database writes:

```bash
docker exec geointel python /app/scripts/provision_agricultural_parcel_history.py --fetch-only
```

Import the checked artifacts through the canonical Dataset upload route:

```bash
docker exec geointel python /app/scripts/provision_agricultural_parcel_history.py
```

Use `--scope mol`, `--years 2008,2019,2025` or `--force` only as explicit
operator choices. The default scope is the persisted 28-municipality Kempen
transport region. Every annual source ZIP and crop code list remains under the
storage volume. PostGIS computes exact hectares for drawn rectangles and
persisted Areas; parcel identities are deliberately unavailable for lineage.

Future definitive editions use the separate four-phase release coordinator:

```bash
docker exec geointel python /app/scripts/manage_alz_agriculture_release.py plan \
  --project-id <KEMPEN_PROJECT_ID> --refresh-catalog

docker exec geointel python /app/scripts/manage_alz_agriculture_release.py stage \
  --project-id <KEMPEN_PROJECT_ID> \
  --confirm-edition <YYYY-v3_FROM_PLAN>

docker exec geointel python /app/scripts/manage_alz_agriculture_release.py review \
  --project-id <KEMPEN_PROJECT_ID> \
  --confirm-edition <YYYY-v3_FROM_PLAN> \
  --confirm-plan-sha256 <SHA256_FROM_STAGE> \
  --approve --reviewer "<OPERATOR_NAME>"

docker exec geointel python /app/scripts/manage_alz_agriculture_release.py apply \
  --project-id <KEMPEN_PROJECT_ID> \
  --confirm-edition <YYYY-v3_FROM_PLAN> \
  --confirm-plan-sha256 <SHA256_FROM_STAGE> \
  --confirm-review-sha256 <SHA256_FROM_REVIEW>
```

The manager accepts only one catalog-confirmed definitive v3 release. `plan`
writes nothing; `stage` downloads and normalizes without PostGIS mutation;
`review` binds a named approval; `apply` revalidates catalog, hashes, schema,
crop codes, scope accounting and previous-edition deltas before delegating to
DatasetService. Provisional v1/v2 snapshots never enter the historical series.

## Buildings and Addresses Register snapshot

After the Mol Area and regional GRB buildings have been provisioned, prepare
the official register evidence with:

```bash
docker exec geointel python /app/scripts/provision_buildings_addresses_register.py --fetch-only
```

Review the generated manifest and then persist through DatasetService:

```bash
docker exec geointel python /app/scripts/provision_buildings_addresses_register.py
```

The resulting `building_registry` Dataset uses ordinary EPSG:4326
`vector_features`; no register-specific table or direct operator database write
exists. Exact PostGIS selection exposes footprint hectares, lifecycle counts,
aggregate unit/address counts and GRB reconciliation counts. Raw address pages
are checksummed storage evidence only. Address labels and house/box numbers are
not copied into queryable properties.

## Helpful repository scripts

- `bash scripts/backend_install.sh`
- `bash scripts/backend_test.sh`
- `bash scripts/backend_dev.sh`
- `bash scripts/smoke_backend_import.sh`

## Bounded official orthophoto acquisition

`GET /api/v1/projects/{project_id}/datasets/orthophoto/products` lists the
governed product allowlist. `POST .../datasets/orthophoto/acquire` accepts an
explicit EPSG:4326 map rectangle plus `product_key` and stores the official
regional WMS response as a canonical EPSG:31370 raster Dataset. Digitaal
Vlaanderen, SPW (`wallonia_latest`) and Paradigm UrbIS (`brussels_latest`) are
allowlisted. The two regional products are bound to persisted Wallonia and
Brussels-Capital Region Areas. The
default safety envelope is 128-1,024 m per side, 1 m/pixel, 32 MiB and a
24-hour exact-request cache. It runs synchronously behind the existing Job
abstraction and never during startup.

Available products cover the most recent winter image, annual winter mosaics
for 2012-2025, three older winter periods, RGB 1979-1990 and panchromatic 1971.
Historical products persist validity metadata and are deliberately excluded
from configured-YOLO/current-GRB QA. `GET .../datasets/{dataset_id}/raster/image`
is the constrained binary PNG endpoint used by the MapLibre image overlay.

Settings: `ORTHOPHOTO_ENABLED`, `ORTHOPHOTO_WMS_URL`,
`SPW_ORTHOPHOTO_WMS_URL`, `BRUSSELS_ORTHOPHOTO_WMS_URL`,
`ORTHOPHOTO_WMS_LAYER`, `ORTHOPHOTO_RESOLUTION_M`,
`ORTHOPHOTO_MIN_SIDE_M`, `ORTHOPHOTO_MAX_SIDE_M`,
`ORTHOPHOTO_TIMEOUT_SECONDS`, `ORTHOPHOTO_MAX_RESPONSE_MB` and
`ORTHOPHOTO_CACHE_TTL_HOURS`. Keep the official HTTPS URL and 1 m profile
unless a separately verified deployment/model profile requires a change.
An explicit bounded request may provide `resolution_m` down to the governed
product's native resolution. This is intended for reviewed training corpora;
the service rejects source oversampling and records rolling-latest observation
time as unknown per pixel rather than equating it with download time.

Before a future `most_recent` source release is allowed into a governed pixel
stage, run the metadata-only preflight for the exact intended rectangle:

```bash
docker exec geointel python /app/scripts/orthophoto_release_preflight.py \
  --project-id 82a85913-c522-45d7-84a1-02b393d89e55 \
  --api-url http://127.0.0.1:8000/api/v1 \
  --bbox 5.110 51.180 5.117 51.185 \
  --refresh-catalog
```

The command reads canonical API envelopes, exact official WMS capabilities,
WCS `DescribeCoverage` and at most 64 queryable flight-day points. It never
requests raster pixels or mutates application/storage state. `current`,
remote-older and mixed/incorrect flight years remain non-stageable. The
report's point grid is flight-date evidence; complete selected-area coverage
comes from containment inside the official 15 cm WCS raster domain.

Official release promotion is a separate four-action operator workflow. Run it
inside the all-in-one container so stage/apply can use only the loopback API:

```bash
# Read-only decision; copy the reported edition and current local marker.
docker exec geointel python /app/scripts/manage_orthophoto_release.py plan \
  --project-id 82a85913-c522-45d7-84a1-02b393d89e55 \
  --bbox 5.110 51.180 5.117 51.185 --refresh-catalog

# First official baseline only: both values must match the fresh preflight.
docker exec geointel python /app/scripts/manage_orthophoto_release.py stage \
  --project-id 82a85913-c522-45d7-84a1-02b393d89e55 \
  --bbox 5.110 51.180 5.117 51.185 \
  --confirm-edition 2025.04 \
  --establish-official-baseline \
  --confirm-local-version most_recent_at_2026-07-15

# Inspect review-preview.png, then use the exact plan SHA printed by stage.
docker exec geointel python /app/scripts/manage_orthophoto_release.py review \
  --project-id 82a85913-c522-45d7-84a1-02b393d89e55 \
  --bbox 5.110 51.180 5.117 51.185 \
  --confirm-edition 2025.04 --confirm-plan-sha256 <plan-sha256> \
  --approve --reviewer "<operator name>" --review-note "<bounded review>"

# Apply only the exact approved bytes and hashes.
docker exec geointel python /app/scripts/manage_orthophoto_release.py apply \
  --project-id 82a85913-c522-45d7-84a1-02b393d89e55 \
  --bbox 5.110 51.180 5.117 51.185 \
  --confirm-edition 2025.04 --confirm-plan-sha256 <plan-sha256> \
  --confirm-review-sha256 <review-sha256>
```

For a later comparable `YYYY.NN` update, omit the two first-baseline flags.
Stage performs one bounded pixel request but no database mutation. Apply is
idempotent for the exact plan/raster checksum, creates a new immutable raster
Dataset and DatasetVersion with the official edition, and retains every older
snapshot. No command is scheduled or invoked by startup or browser actions.

## Governed bounded GRB acquisition

`GET /api/v1/projects/{project_id}/datasets/grb/products` exposes four fixed
official vector products: building footprints, road segments, water
surfaces/lines and administrative parcels. `POST .../datasets/grb/acquire`
accepts an EPSG:4326 rectangle, optional project Area, one product key and an
explicit refresh flag.

The service queries only the allowlisted GRB OGC API collection paths, follows
complete same-host pagination and clips every geometry to `bbox ∩ Area`.
It explicitly requests OGC CRS84 GeoJSON for bbox and output; the native
EPSG:31370 storage CRS remains provenance rather than being guessed from raw
coordinates.
Requests fail closed above 20 km per side, 200 pages, 150,000 retained
features, 20 MiB per page or 256 MiB total. No partial Dataset is persisted
when a limit is exceeded. Official ids, request URLs, page checksums and the
final artifact checksum are retained as provenance.

Persistence uses the existing synchronous Job plus
`DatasetService.import_vector_bytes`, so Dataset, DatasetVersion and
VectorFeature rows remain one canonical flow. The browser never contacts the
provider directly. Exact request identities are reused for 24 hours. Buildings
return footprint area in hectares, roads return line length in kilometres,
water returns surface area plus supporting water-line length, and parcels
return mapped area. GRB cannot provide water volume, legal parcel boundaries
or traffic information.

Settings: `GRB_ENABLED`, `GRB_OGC_API_URL`, `GRB_MIN_SIDE_M`,
`GRB_MAX_SIDE_M`, `GRB_PAGE_SIZE`, `GRB_MAX_PAGES`, `GRB_MAX_FEATURES`,
`GRB_TIMEOUT_SECONDS`, `GRB_MAX_RESPONSE_MB`,
`GRB_MAX_TOTAL_RESPONSE_MB` and `GRB_CACHE_TTL_HOURS`.

## Governed DHMV terrain acquisition

`GET /api/v1/projects/{project_id}/datasets/dhmv/products` exposes the fixed
official DTM/DSM registry. `POST .../datasets/dhmv/acquire` requests only
`DHMVII_DTM_1m` or `DHMVII_DSM_1m` from the production Digitaal Vlaanderen WCS.
The default 5 m analysis copy keeps complete-Mol processing bounded while
retaining native 1 m resolution, EPSG:31370, TAW, `-9999` nodata and the
2013-2015 acquisition period in provenance.

Municipality-sized requests are split into sequential WCS tiles of at most
10 km per side. The client sends the explicit media accept header required by
the production service, waits between requests, retries transient provider
statuses once and mosaics only tiles that validate against EPSG:31370, one
band and the requested resolution. Every tile URL and aggregate transfer
checksum remains in provenance.

Run the complete Mol operator after the regional workspace and Mol Area exist:

```bash
docker exec geointel python /app/scripts/provision_mol_dhmv.py
```

The operator acquires DTM and DSM, clips each raster to the exact persisted
Area, validates checksums and calls the terrain selection endpoint as a smoke.
Use `--products dtm_1m`, `--resolution-m 5` or `--force` when explicitly
needed. `POST .../raster/terrain/select` returns height in m TAW, relief in
metres and slope in degrees. `GET .../raster/terrain/image` returns the
constrained MapLibre PNG. Water depth, volume and drainage remain unavailable.

Provision the same governed DTM/DSM pair for every persisted municipality in
the approved Kempen scope:

```bash
docker exec geointel python /app/scripts/provision_regional_dhmv.py \
  --scope kempen-transport-region --dry-run
docker exec geointel python /app/scripts/provision_regional_dhmv.py \
  --scope kempen-transport-region
```

This plans 56 municipality/product acquisitions. It supports bounded
`--members` and `--products` subsets, backend cache reuse, per-item progress
and a complete failure summary. Persistence remains inside the canonical
DHMV acquisition service and Dataset/DatasetVersion/Job flow; the operator
does not fetch WCS bytes or write raster metadata directly.

The complete live matrix contains 56 ready Datasets and 56 DatasetVersions
across 28 Areas. On the complete Kempen Area the Map workspace presents those
partitions as one logical DTM/DSM layer. `POST .../datasets/raster/terrain/select`
opens only partitions intersecting the drawn rectangle and computes exact
global cell statistics. It does not create a hidden regional mosaic.

Settings: `DHMV_ENABLED`, `DHMV_WCS_URL`, `DHMV_RESOLUTION_M`,
`DHMV_MIN_SIDE_M`, `DHMV_MAX_SIDE_M`, `DHMV_MAX_PIXELS`,
`DHMV_TIMEOUT_SECONDS` and `DHMV_MAX_RESPONSE_MB`.

## Governed VMM flood-hazard depth scenarios

`GET /api/v1/projects/{project_id}/datasets/flood-hazard/products` exposes the
twelve allowlisted VMM OGRK coverages. `POST .../flood-hazard/acquire` performs
bounded WCS 1.1 requests, exact Area clipping, checksum validation and ordinary
Dataset/DatasetVersion/Job persistence. The source's positive centimetre
values are normalized to metres; null/zero cells are transparent nodata.

Run all scenarios for Mol after the regional workspace and Mol Area exist:

```bash
docker exec geointel python /app/scripts/provision_mol_flood_hazards.py
```

Provision the same official VMM scenario set for every persisted municipality
Area in the approved Kempen regional workspace:

```bash
docker exec geointel python /app/scripts/provision_regional_flood_hazards.py \
  --scope kempen-transport-region
```

Inspect the planned municipality/scenario matrix without writing data:

```bash
docker exec geointel python /app/scripts/provision_regional_flood_hazards.py \
  --scope kempen-transport-region --dry-run
```

Useful bounded runs:

```bash
docker exec geointel python /app/scripts/provision_regional_flood_hazards.py \
  --members Mol,Geel --products pluviaal_current_t100
docker exec geointel python /app/scripts/provision_regional_flood_hazards.py \
  --members 13025 --products pluviaal_current_t10,pluviaal_current_t100
```

The regional operator uses the canonical API only. It requires the geographic
scope Areas to exist first, persists one ordinary raster Dataset per
municipality/scenario and reuses existing Datasets unless `--force` is supplied.
The full Kempen scope with all products means 28 municipalities times 12
scenario rasters. This is intentionally explicit operator work, not startup
work and not a browser-side provider fetch.

The complete live matrix contains 336 ready Datasets and 336 DatasetVersions.
The regional Map workspace deduplicates them into twelve scenario choices,
renders all municipality image partitions for the selected scenario and uses
`POST .../datasets/raster/flood-hazard/select` for exact bounded cross-boundary
analysis. The same 12-million-cell guard prevents unsafe full-region reads.

Use `--products pluviaal_current_t100`, `--resolution-m 5` or `--force` for an
explicit subset/refresh. `POST .../raster/flood-hazard/select` returns mapped
inundated hectares, selection share and local modeled maximum-depth statistics.
The `modelled_max_depth_area_integral_m3` metric is an area integral of local
maxima and must not be called actual, permanent or concurrent water volume.
`GET .../raster/flood-hazard/image` serves the constrained transparent PNG.

Settings: `FLOOD_HAZARD_ENABLED`, `FLOOD_HAZARD_WCS_URL`,
`FLOOD_HAZARD_RESOLUTION_M`, `FLOOD_HAZARD_MIN_SIDE_M`,
`FLOOD_HAZARD_MAX_SIDE_M`, `FLOOD_HAZARD_MAX_PIXELS`,
`FLOOD_HAZARD_TIMEOUT_SECONDS` and `FLOOD_HAZARD_MAX_RESPONSE_MB`.

## Cross-domain thematic rasters and DOV soil

The governed thematic registry exposes five fixed MercatorNet products through
`GET .../datasets/thematic-raster/products`. Acquisition uses
`POST .../datasets/thematic-raster/acquire`; selection and PNG rendering use
`POST .../raster/thematic/select` and `GET .../raster/thematic/image`.

Provision every product for the exact persisted Mol Area:

```bash
docker exec geointel python /app/scripts/provision_thematic_rasters.py
```

Inspect the complete 28-municipality matrix without writes, then run it after
the Mol source/runtime gate passes:

```bash
docker exec geointel python /app/scripts/provision_thematic_rasters.py \
  --project-name "Kempen Regional Workbench" --all-municipalities --dry-run
```

Settings: `THEMATIC_RASTER_ENABLED`, `THEMATIC_RASTER_WCS_URL`,
`THEMATIC_RASTER_MIN_SIDE_M`, `THEMATIC_RASTER_MAX_SIDE_M`,
`THEMATIC_RASTER_MAX_PIXELS`, `THEMATIC_RASTER_TIMEOUT_SECONDS` and
`THEMATIC_RASTER_MAX_RESPONSE_MB`.
The default thematic ceiling is 60 km and 30 million cells so the exact
Kempen work area fits. External WCS transfers remain split into fixed 10 km
tiles, product identifiers remain server-allowlisted and other raster
pipelines retain their smaller independent limits.

The Flanders browser workflow uses these same endpoints on demand. It never
performs a startup import or direct browser WCS request: an explicit
municipality or drawn rectangle starts five bounded acquisitions, followed by
the existing persisted-raster analyses. Exact request hashes reuse ready
Datasets. A full-Flanders raster request remains blocked by the same 60 km and
30 million cell limits. Dataset metadata labels only Areas named
`Gemeente ...` as `coverage_scope=municipality`; regional Area clipping is
stored as `bounded_selection`.

## Walloon WALOUS land cover and flood hazard

The Wallonia map flow uses bounded PICC vector products, the queryable legal
SPW flood-hazard polygon layer and provisioned official WALOUS land-cover
rasters. Provision the 2018, 2020 and 2023 source editions once in the persistent
storage mount:

```bash
docker exec geointel python /app/scripts/provision_walous_sources.py \
  --years 2018 2020 2023 \
  --destination /app/storage/source-cache/walous
```

The provisioner verifies advertised archive sizes, safe ZIP structure,
EPSG:3812, one band, 1 m cells, the official non-contiguous class codes
`1,2,3,4,5,6,7,8,9,80,90` and SHA-256 checksums. It does not run at
application startup. `GET .../datasets/walous/products` therefore reports
`source_not_provisioned` for each edition whose source file is absent.

For a bounded Walloon selection the browser persists the latest edition and
all other configured comparable editions. `POST .../raster/walous/select`
returns cell-area hectares; the temporal API compares the same semantic metric
keys for 2018, 2020 and 2023. The 2018 stacked classes use the official visible-
class crosswalk and retain the earlier-method limitation. WALOUS is land cover,
not legal land use, ownership,
tree count, timber volume or water volume.

The class semantics follow the official raster codes, not display-list
positions: 1 artificial ground, 2 above-ground construction, 3 railway, 4 bare
soil, 5 surface water, 6 rotating herbaceous cover, 7 continuous herbaceous
cover, 8/9 trees above 3 m and 80/90 woody cover up to 3 m. Observation ranges
are retained from the SPW metadata rather than replaced by arbitrary year-end
dates.

Settings: `WALOUS_ENABLED`, `WALOUS_SOURCE_DIR`,
`WALOUS_ANALYSIS_RESOLUTION_M`, `WALOUS_MAX_SIDE_M` and
`WALOUS_MAX_PIXELS`. The SPW flood polygon adapter uses
`SPW_FLOOD_HAZARD_ENABLED` and `SPW_FLOOD_HAZARD_MAPSERVER_URL`.

The official Walloon 2021-2022 1 m MNT is an explicit operator asset. Provision
it once with `scripts/provision_spw_terrain_source.py`; the runtime then reads
only bounded windows and persists 5 m analysis derivatives. The full 0.5 m
artifact remains intentionally excluded because it adds no V1 metric and is
about 213 GB. Settings: `SPW_TERRAIN_ENABLED`, `SPW_TERRAIN_SOURCE_DIR`,
`SPW_TERRAIN_ANALYSIS_RESOLUTION_M`, `SPW_TERRAIN_MAX_SIDE_M` and
`SPW_TERRAIN_MAX_PIXELS`.

Provision the official DOV soil polygons for Mol through the existing vector
upload path:

```bash
docker exec geointel python /app/scripts/provision_mol_soil_map.py
```

Provision all approved Kempen municipalities and the complete work area after
the official geographic-scope artifacts exist:

```bash
docker exec geointel python /app/scripts/provision_regional_soil_map.py
```

The regional operator retains one checksummed WFS evidence chain per
municipality, gives boundary-split source features a NIS suffix and assembles
one Dataset linked to the complete Kempen Area. EPSG:31370 clipping followed by
EPSG:4326 persistence can create submeter coordinate-rounding slivers at the
stored boundary, so regional and municipal selection metrics deliberately run
the exact PostGIS intersection instead of using a preclipped fast path.

Use `--fetch-only` to retain and validate source evidence without importing.
The operator never writes directly to PostGIS. Soil drainage and related map
classes represent the 1949-1971 survey and are not current observations.

## Waterinfo station histories

Run the explicit operator after the regional workspace and Mol Area exist:

```bash
docker exec geointel python /app/scripts/provision_waterinfo_station_history.py \
  --project-name "Kempen Regional Workbench" \
  --area-name "Gemeente Mol" \
  --from-year 2013 --to-year 2025
```

The command retains raw KiWIS JSON/checksums and imports only real annual
observations through the canonical dataset upload API. Every station has its
own temporal-series key. Water levels and discharges remain Point measurements;
they are never averaged across stations or presented as municipal water volume.
Use `--fetch-only` to prepare and audit artifacts without persistence.

## BWK/Natura 2000 state 2025

Run the governed Mol operator after the regional workspace and Mol Area exist:

```bash
docker exec geointel python /app/scripts/provision_mol_bwk_natura2000.py
```

The command fetches the official INBO WFS, retains raw checksummed pages,
clips in EPSG:31370 and imports through DatasetService. `--fetch-only` builds
evidence without persistence. A conflicting checksum for an already persisted
state-2025 Mol Dataset fails closed instead of creating a silent replacement.
PostGIS selection summaries keep BWK value classes separate and label
PHAB-derived habitat hectares as estimates.

For the complete approved Kempen transport region, run the partitioned
operator after `provision_geographic_scope.py --scope kempen-transport-region`:

```bash
docker exec geointel python /app/scripts/provision_regional_bwk_natura2000.py
```

Use `--fetch-only` to build and validate all 28 municipality partitions without
database persistence. A normal rerun validates and reuses the immutable source
evidence and existing Dataset. `--force` explicitly refetches the WFS but still
fails closed if a different state-2025 checksum is already persisted. The
regional output uses the same selection-summary API as Mol; no new endpoint or
direct PostGIS write is introduced.

## Source freshness and version audit

`GET /api/v1/projects/{project_id}/datasets/source-freshness` derives a
read-only source status from persisted Dataset, DatasetVersion and storage
evidence. It distinguishes rolling snapshots and annual publications from
fixed editions, scenarios, historical archives and local artifacts. Fixed
source editions are never called stale solely because they are old.

The endpoint checks missing versions, checksum disagreement, missing local
files and stored-size disagreement. It performs no provider request and no
database write. The packaged operator command is suitable for an explicit
Unraid cron entry:

```bash
docker exec geointel python /app/scripts/audit_source_freshness.py \
  --project-id 82a85913-c522-45d7-84a1-02b393d89e55 \
  --api-url http://127.0.0.1/api/v1 \
  --fail-on integrity \
  --output /app/storage/operator-evidence/source-freshness/latest.json
```

Use `--fail-on due` to make a planned review date fail automation, or
`--fail-on never` for reporting only. The command never starts a refresh.

An operator can explicitly add the official GRB, orthophoto, Statbel and ALZ
edition check:

```bash
docker exec geointel python /app/scripts/audit_source_freshness.py \
  --project-id 82a85913-c522-45d7-84a1-02b393d89e55 \
  --api-url http://127.0.0.1/api/v1 \
  --probe-catalogs \
  --output /app/storage/operator-evidence/source-freshness/with-catalogs.json
```

Use `--refresh-catalogs` to bypass the 15-minute in-memory cache and
`--fail-on-catalog` only when temporary official-provider unavailability must
fail an operator job. This path reads bounded WFS/WMS capabilities and their
fixed ISO 19139 metadata records. It confirms GRB `GBG`, `WBN`, `WGO`, `ADP`
and orthophoto `Ortho`, `Vliegdagcontour`. It parses the exact official Statbel
DCAT Turtle catalog to identify the latest population-by-statistical-sector
year, landing page, license and allowed distribution identities. It never
follows those ZIP/XLSX links. It also reads the exact official ALZ
publication page and validates only allowlisted archive-link identities. It
never requests feature, raster or ALZ ZIP content. ALZ v1/v2 campaign snapshots
remain provisional; only a v3 publication is compared with a local definitive
historical edition.

Runtime controls are `SOURCE_CATALOG_PROBE_ENABLED`,
`SOURCE_CATALOG_GRB_WFS_URL`, `SOURCE_CATALOG_STATBEL_DCAT_URL`,
`SOURCE_CATALOG_STATBEL_MAX_RESPONSE_MB`, `SOURCE_CATALOG_ALZ_RELEASE_URL`,
`SOURCE_CATALOG_PROBE_TIMEOUT_SECONDS`, `SOURCE_CATALOG_PROBE_MAX_RESPONSE_MB` and
`SOURCE_CATALOG_PROBE_CACHE_TTL_SECONDS`. Metadata links remain restricted to
the official HTTPS CSW path on `metadata.vlaanderen.be` even when an operator
overrides the capabilities endpoint. The ALZ release URL is fail-closed to the
exact HTTPS host/path and cannot be redirected to another page or download
host.

The Statbel catalog has a separate 5 MiB default response bound. Population
year, sector-geometry year and REDEGEO layout remain separate concepts: the
2025 population release uses the new layout, while the concurrently published
old layout is transition evidence only. The existence of 2026 sector geometry
does not imply a 2026 population-by-sector release.

## Governed regional GRB refresh

`GET /api/v1/projects/{project_id}/datasets/grb-refresh-plan` combines the
explicit official GRB edition probe with the four existing regional snapshot
series. It is read-only: it reports local feature/storage impact and whether
buildings, roads, water and parcels are current, updateable or require review.

Regional refreshes use a two-phase operator flow inside the all-in-one
container. First stage and validate every source partition without touching
PostGIS:

```bash
docker exec geointel python /app/scripts/manage_grb_refresh.py stage \
  --project-id 82a85913-c522-45d7-84a1-02b393d89e55 \
  --api-url http://127.0.0.1:8000/api/v1 \
  --confirm-edition 2026-07-15 \
  --layers buildings roads water parcels
```

The JSON result gives `plan_path`, exact feature deltas, artifact sizes and
`plan_sha256`. Review that evidence, then apply those exact staged bytes:

```bash
docker exec geointel python /app/scripts/manage_grb_refresh.py apply \
  --project-id 82a85913-c522-45d7-84a1-02b393d89e55 \
  --api-url http://127.0.0.1:8000/api/v1 \
  --confirm-edition 2026-07-15 \
  --confirm-plan-sha256 SHA256_FROM_STAGE
```

Both commands fail when project/scope, official edition, manifests, partition
count or any checksum differs. Interrupted staging is safely resumable because
the existing municipal manifests are reused. Apply imports through
DatasetService/VectorFeatureService, creates new temporal Datasets and retains
all previous snapshots. Do not add `--force` to this coordinator; a source
refetch remains a separate deliberate recovery action in the lower-level
operators.

## Safe project lifecycle cleanup

Operational validation, calibration and benchmark runs can create technical
projects. The default project API now returns active workspaces only, while
archived workspaces remain queryable with `GET /api/v1/projects?status=archived`.
Use the packaged cleanup command to archive only the strict technical-name
allowlist:

```bash
# Dry-run: inspect the number of matches without changing the database.
python scripts/archive_technical_projects.py

# Apply the exact allowlisted plan.
python scripts/archive_technical_projects.py --apply
```

Inside the all-in-one Unraid container:

```bash
docker exec geointel python /app/scripts/archive_technical_projects.py
docker exec geointel python /app/scripts/archive_technical_projects.py --apply
```

The command never deletes projects or related datasets, jobs, analyses,
quality checks and exports. It always preserves `Kempen Regional Workbench`
and `Mol Municipality Workbench`, defaults to dry-run and can print every
matched name with `--show-names`.

## Governed VHA bathymetry profiles

`POST /api/v1/projects/{project_id}/datasets/bathymetry/profiles/acquire`
performs a bounded official VHA ArcGIS query, exact persisted-Area clipping,
watercourse-name normalization and ordinary Dataset/VectorFeature persistence.
`GET /api/v1/projects/{project_id}/datasets/bathymetry/sources` reports VHA as
operational, MDK as probe-only and the pinned SPW raster operator as
operational.

Runtime controls are `BATHYMETRY_PROFILES_ENABLED`,
`BATHYMETRY_PROFILES_LAYER_URL`, `BATHYMETRY_WATERCOURSE_LAYER_URL`,
`BATHYMETRY_PROFILES_PAGE_SIZE`, `BATHYMETRY_PROFILES_MAX_FEATURES`,
`BATHYMETRY_PROFILES_TIMEOUT_SECONDS` and
`BATHYMETRY_PROFILES_MAX_RESPONSE_MB`. The feature limit intentionally forces
large Flemish scopes into exact Area partitions.

The Dataset exposes profile count and nullable structured depth/width metrics.
It does not claim a continuous bed model, current depth or volume. Use
`scripts/provision_mol_bathymetry_profiles.py` for the canonical Mol operator
flow.

Provision the complete current Flemish land scope and then run the resumable
VHA municipality coordinator:

```bash
docker exec geointel python /app/scripts/provision_flanders_geographic_scope.py
docker exec geointel python /app/scripts/provision_flanders_bathymetry_profiles.py
```

The first command discovers all current VRBG RefGem municipalities, validates
a 270..300 safety range and persists their exact Areas. The second writes its
manifest after every partition. Repeating it reuses completed source
identities; `--force` deliberately refreshes them. A partial `--members` or
`--max-partitions` run never marks regional coverage complete.

Regional map analysis uses
`POST /api/v1/projects/{project_id}/datasets/bathymetry/profiles/partitions/select`.
It selects the latest complete manifest, prefilters overlapping municipality
partitions and performs one PostGIS query over their persisted
`vector_features`. Municipality Areas use only their exact partition. The
response and server-side map export retain every contributing Dataset id and
never substitute a single municipality Dataset for all of Flanders.

Inspect the MDK North Sea WCS without downloading coverage:

```bash
docker exec geointel python /app/scripts/probe_mdk_bathymetry.py
```

Exit code `0` means verified capabilities; `2` means a truthful blocked
readiness state such as TLS or endpoint failure. Runtime controls are
`MDK_BATHYMETRY_PROBE_ENABLED`, `MDK_BATHYMETRY_WCS_URL`,
`MDK_BATHYMETRY_PROBE_TIMEOUT_SECONDS` and
`MDK_BATHYMETRY_PROBE_MAX_RESPONSE_MB`. TLS verification cannot be disabled.

### SPW waterbed raster

The official 2023-05-23 SPW bathymetry ZIP is integrated only through the
bounded operator. Stage the immutable ZIP under persistent storage and run:

```bash
docker exec geointel python /app/scripts/import_spw_bathymetry.py \
  --base-url http://127.0.0.1:8000 \
  --project-name "Belgium and North Sea Workbench" \
  --area "RC Golden - Wallonia urban-rural" \
  --bbox 4.85,50.45,4.87,50.47 \
  --raw-zip /app/storage/operator-evidence/spw-bathymetry/2023-05-23/raw/BATHY_50CM_ALTITUDE_DNG_GEOTIFF_3812.zip \
  --output-dir /app/storage/operator-evidence/spw-bathymetry/2023-05-23/derived
```

The script validates the pinned official checksum, safe archive members,
EPSG:3812, one Float32 band, approximately 0.5 m cells and nodata `-9999`.
It then creates a bounded COG and uploads it through `/datasets/upload`.
`POST .../raster/bathymetry/select` returns waterbed elevation in mDNG,
surveyed surface and coverage. Current depth, volume and datum conversion stay
unavailable without a compatible water-surface source. Selection analysis is
bounded by `BATHYMETRY_RASTER_MAX_PIXELS` (30 million by default).

## Governed regional official-vector acquisition

The thematic raster registry includes forest and agricultural land-use masks
derived from Landgebruik Vlaanderen 2025 classes 12 and 13/14. They use the
existing thematic acquisition and selection routes.

Eight fixed products are exposed through
`/datasets/official-vector/products` and
`/datasets/official-vector/acquire`: INBO BWK/Natura 2000 2025 and DOV soil
for Flanders; PICC buildings, roads, hydrographic axes and surfaces for
Wallonia; and UrbIS buildings and cadastral parcels for Brussels. All require
an EPSG:4326 rectangle, clip in a provider-appropriate metric CRS and persist
through `DatasetService.import_vector_bytes`. SPW/PICC and UrbIS additionally
require a persisted exact regional coverage Area and never write directly to
`vector_features`.

Runtime controls are `OFFICIAL_VECTOR_ENABLED`, `BWK_WFS_URL`,
`DOV_SOIL_WFS_URL`, `SPW_PICC_ENABLED`, `SPW_PICC_MAPSERVER_URL`,
`URBIS_ENABLED`, `URBIS_WFS_URL`, `OFFICIAL_VECTOR_MIN_SIDE_M`,
`OFFICIAL_VECTOR_MAX_SIDE_M`, `OFFICIAL_VECTOR_PAGE_SIZE`,
`OFFICIAL_VECTOR_MAX_PAGES`, `OFFICIAL_VECTOR_MAX_FEATURES`,
`OFFICIAL_VECTOR_TIMEOUT_SECONDS`, `OFFICIAL_VECTOR_MAX_RESPONSE_MB`,
`OFFICIAL_VECTOR_MAX_TOTAL_RESPONSE_MB` and
`OFFICIAL_VECTOR_CACHE_TTL_HOURS`.

## Locked CI dependencies

The release image installs the hashed Linux/Python 3.11 base/GIS graph from
`requirements-runtime.lock`; CI adds test tools through
`requirements-ci.lock`. Both deliberately exclude the optional `ai` extra.
Regenerate and validate them from the repository root with:

```bash
bash scripts/generate_python_lock.sh
python scripts/verify_python_lock.py
```

The complete gate and vulnerability/SBOM policy are documented in
`docs/CI_SUPPLY_CHAIN.md`.

## Release golden areas

The RC browser suite uses seven deterministic, bounded regression areas across
Belgium and the Belgian North Sea. Preview the required changes without
mutating the runtime:

```bash
python scripts/provision_release_golden_areas.py \
  --base-url http://127.0.0.1:8000 \
  --output artifacts/rc8-golden-areas.json
```

Create only missing Areas through the canonical project/area APIs:

```bash
python scripts/provision_release_golden_areas.py \
  --base-url http://127.0.0.1:8000 \
  --output artifacts/rc8-golden-areas.json \
  --apply
```

The operator copies the governed Mol and Kempen geometries into the national
workbench with their source project/Area identifiers and provisions bounded
Wallonia, Brussels, language-boundary, coast and offshore multi-zone Areas.
Every geometry receives a deterministic SHA-256 fingerprint in the evidence
file. It never imports provider data or writes directly to database tables.

Explicit demo seeding also reactivates its own archived technical project.
This keeps the opt-in fixture workflow selectable without changing the normal
active-project lifecycle.

## Data operations and retention

The runtime packages `audit_data_operations.py`,
`cleanup_storage_artifacts.py` and the shared release-backup guard. The audit
is read-only and combines disk pressure, storage lifecycle, persisted path
integrity, failed-work counts and national/regional/maritime source-family
inventory. Cleanup is limited to old unreferenced derived/cache/export files.

Unknown paths, official source material, uploads, models and release/operator
evidence are protected by default. Apply mode requires an exact confirmation,
an explicit candidate ceiling and a recent checksum-verified database plus
SHA-256 storage backup mounted read-only under `/app/backups`. See
`docs/DATA_OPERATIONS_RUNBOOK.md`.

## Release candidate operations

The semantic release version is stored in the repository `VERSION` file and
is exposed by health responses plus the OCI image version label. Fresh
install, checksum-verified backup, isolated restore/upgrade, rollback,
Belgium/North Sea browser journeys, SBOM, vulnerability evidence, SSH-signed
release manifest and final verification commands are defined in
`docs/RELEASE_RUNBOOK.md`.