geointel/scripts/README.md

# Scripts

Setup-, import-, demo- en maintenance-scripts voor GeoIntel.

## Runtime verification

Audit the active backend route surface against `docs/API_CONTRACTS.md`:

```bash
python scripts/audit_api_contracts.py
```

The audit imports the FastAPI app, compares implemented `GET`/`POST`/`PATCH`/
`DELETE` routes with active API contract headings and tracks the explicit
non-envelope exceptions (`/health` and export downloads). It fails when a route
exists without docs or when docs claim an endpoint that is not implemented.

Verify the browser-facing Docker/LAN runtime:

```bash
bash scripts/verify_browser_runtime.sh http://192.168.10.150:1202
bash scripts/verify_gis_runtime.sh http://192.168.10.150:1202
```

Verify the explicit demo workflow plus export artifact path:

```bash
bash scripts/verify_demo_export_workflow.sh http://192.168.10.150:1202
```

The demo/export smoke is intentionally mutating and idempotent: it seeds the
offline fixture demo if needed, verifies the project area GeoJSON, fixture
datasets, vector FeatureCollection content, vector feature summary, persisted
QA/QC metrics, creates metadata/report/vector GeoJSON exports, lists exports
and downloads the JSON/GeoJSON/HTML artifacts through the frontend proxy. The
persisted QA/QC result is compared against `fixtures/golden/expected_qa_metrics.json`
so runtime demo precision, recall, F1, mean IoU and false-positive/negative
counts cannot drift silently.

Verify the explicit demo raster workflow:

```bash
bash scripts/verify_demo_raster_workflow.sh http://192.168.10.150:1202
```

The raster smoke is intentionally mutating and idempotent enough for local
runtime checks: it seeds the offline demo workflow, validates the
`demo_context_raster.tif` fixture dataset, then exercises raster inspect,
preview, stats and one small tile/manifest generation through canonical
`data` envelopes. It does not run AI inference or fetch external imagery.

Verify that the browser-facing workbench can populate the default demo start
state through the frontend proxy:

```bash
bash scripts/verify_workbench_default_state.sh http://192.168.10.150:1202
```

This smoke is dependency-light and intentionally idempotent: it seeds the
offline demo workflow, then verifies that `GeoIntel Demo - Building QA` exposes
the `Demo AOI - Geel buildings` map geometry, `3/3 ready` demo datasets
(candidate vector, reference vector and raster fixture) and a persisted QA/QC
result through canonical `data.items` envelopes. Pair it with a Codex/browser
screenshot pass when checking visual layout or overflow.

Verify the backing state for the core workbench interactions:

```bash
bash scripts/verify_workbench_interactions.sh http://192.168.10.150:1202
```

This smoke validates the state behind project switching, AOI/map selection,
dataset selection, QA refresh and export refresh through the same frontend
proxy used by the browser. The frontend also exposes stable `data-testid`
anchors for Codex/browser click checks on those controls.

Verify the browser click handoff from raster tiling into Detection and
Segmentation Lab:

```bash
bash scripts/verify_ai_handoff_interactions.sh http://192.168.10.150:1202
```

The AI handoff smoke seeds the explicit offline demo workflow, generates a
small raster tile manifest, opens the workbench in Chromium, clicks the raster
inspector `Use in Detection Lab` and `Use in Segmentation Lab` actions, and
verifies that the selected raster dataset plus manifest path are populated in
the AI workspace. Playwright/Chromium must be available in the runner
environment; GeoIntel does not add Playwright as a frontend dependency by
default. The main readiness gate checks this script's syntax only.

Capture visual regression handoff screenshots for the workbench:

```bash
bash scripts/capture_workbench_screenshots.sh http://192.168.10.150:1202
CAPTURE_MOBILE=0 bash scripts/capture_workbench_screenshots.sh http://192.168.10.150:1202 /tmp/geointel-screens
```

The capture script seeds the explicit offline demo workflow, opens each main
workspace, captures viewport desktop screenshots and, by default, viewport
mobile screenshots.
It writes PNG files plus `manifest.json` under `artifacts/screenshots/...` or a
caller-provided output directory. Playwright/Chromium must be available in the
runner environment; GeoIntel does not add Playwright as a frontend dependency
by default. The main readiness gate checks script syntax only.

Verify the deterministic QA/QC golden benchmark:

```bash
bash scripts/verify_golden_qa_benchmark.sh
python scripts/run_golden_qa_benchmark.py --json
```

The benchmark uses only explicit local fixtures under `fixtures/golden`,
executes the existing QA/QC matching logic, verifies the expected precision,
recall, F1, mean IoU and false-positive/false-negative counts, and checks that
`QualityCheck` plus `Metric` rows would be persisted. Scenarios are listed in
`fixtures/golden/golden_qa_benchmarks.json` and currently cover partial match,
perfect match, no-overlap and MultiPolygon building comparisons. The main
readiness gate runs this benchmark so QA metric drift fails before a release.

Verify a configured local YOLO model without running inference:

```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/model.pt --tile-manifest-path /absolute/path/to/manifest.json --check-model-load --json
```

Against the Docker runtime:

```bash
docker compose exec -T backend python scripts/yolo_preflight.py --model-path /absolute/path/to/model.pt --tile-manifest-path /absolute/path/to/manifest.json --check-model-load --json
```

The model-load smoke is opt-in, requires real optional AI dependencies, refuses
`--assume-dependencies`, loads only the supplied local file and does not download
weights or run prediction.

Verify the full configured-YOLO model asset workflow against a running runtime:

```bash
bash scripts/verify_model_asset_detection_workflow.sh http://192.168.10.150:1202
```

This smoke is intentionally mutating and requires a real AI-enabled runtime with
at least one mounted local model asset. It seeds the explicit offline demo
workflow, generates a small raster tile manifest, selects the active local model
asset from `GET /api/v1/detection/model-assets`, validates read-only YOLO
preflight, runs `POST /api/v1/detection/run`, and verifies the persisted
AnalysisRun, Detection list and Detection GeoJSON endpoints. A zero detection
count is allowed because the demo raster is a synthetic runtime fixture; the
script validates the operational path and provenance, not production model
quality. The main readiness gate checks this script's syntax only.

Verify the full operator-provided raster/reference detection and QA path:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/data/orthophoto.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/data/reference-buildings.geojson \
bash scripts/verify_real_data_detection_qa_workflow.sh http://192.168.10.150:1202
```

The current Tower operator sample is available at:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
bash scripts/verify_real_data_detection_qa_workflow.sh http://192.168.10.150:1202
```

Reuse the definitive Mol municipality project for a bounded Mol-Centrum
analysis zone while keeping the municipality-wide reference layer distinct:

```bash
REAL_PROJECT_ID=d74c1f87-29c0-4c67-adfc-560764f2b80e \
REAL_PROJECT_NAME='Mol Municipality Workbench' \
REAL_PROJECT_REGION='Mol, Kempen' \
REAL_AREA_NAME='Mol Centrum - AI analysezone 500m' \
REAL_AREA_BBOX='5.113116,51.189653,5.120284,51.194147' \
REAL_DATASET_NAME_PREFIX='mol_center_ai_500m' \
REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/mol_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/mol_grb_gbg_buildings.geojson \
bash scripts/verify_real_data_detection_qa_workflow.sh http://192.168.10.150:1202
```

`REAL_PROJECT_ID` is validated through the canonical project endpoint; it does
not create a shadow validation project. When `REAL_AREA_BBOX` is present, both
the raster and matching reference upload persist that new Area id. The optional
safe filename prefix prevents the bounded sample from sharing a display name
with the complete Mol GRB dataset. The workflow still performs no data fetch or
model download: the files and configured local model asset must already exist.

Those files are runtime artifacts generated from Digitaal Vlaanderen's
OMWRGBMRVL WMS `Ortho` layer and GRB OGC API Features `GBG` building collection
for a small Geel AOI. They are intentionally not repository fixtures.

To prepare the documented operator samples reproducibly inside the all-in-one
runtime container, run:

```bash
docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py
```

This writes GeoTIFF/GeoJSON pairs and `operator_samples_manifest.json` under
`/app/storage/operator-data` inside the container, which maps to
`storage/operator-data` in the Tower appdata checkout. The default corpus
contains reference AOIs for Geel, Mol, Turnhout, Herentals, Balen, Retie and
Westerlo plus background candidates for Postel-bos, Lommel-heide,
Kasterlee-bos, Dessel-heide, Ravels-bos, Meerhout-bos, Geel-Bel,
Arendonk-heide and Herenthout-bos. Normal reference AOIs still fail when GRB
returns no buildings; background candidates are explicitly marked with
`sample_role` and may write an empty reference FeatureCollection for
negative-tile training. Generated manifests also classify background samples as
`pure_empty_negative` when GRB returns zero reference buildings or
`sparse_building_context` when GRB returns one or more contextual buildings.
The helper fetches only the explicit documented AOIs, records Digitaal
Vlaanderen attribution and reuses existing files by default.
Use `--force` only when the local runtime artifacts should be regenerated.
GRB building references are fetched through the provider's OGC API
`rel=next` pagination links, so dense AOIs are not silently limited to the
first 1000 features. The default page size is `1000`; override it with
`--reference-page-limit` or `OPERATOR_GRB_PAGE_LIMIT`. The safety cap defaults
to `100000` features per sample and can be adjusted with
`--reference-max-features` or `OPERATOR_GRB_MAX_FEATURES`. Generated reference
GeoJSON files record `reference_pages_fetched`, `reference_truncated`,
`reference_page_limit`, `reference_max_features` and every fetched
`source_urls` page for auditability.

Mol has a dedicated operational pack with five positive contexts: center,
Achterbos residential, Gompel mixed settlement, Donk canal/industrial and
Postel rural village. The four new contexts are validation holdouts and are not
silently added to training. Postel-bos remains a separate background control.
Prepare the 1 km / 1024 px pack explicitly:

```bash
docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py \
  --output-dir /app/storage/operator-data/mol-operational-1024 \
  --samples mol,mol_achterbos,mol_gompel,mol_donk,mol_postel,postel_bos \
  --width 1024 \
  --height 1024 \
  --half-size-scale 2 \
  --force
```

Then run the existing persisted positive QA and background-control paths as one
operator command:

```bash
docker exec -it \
  -e OPERATOR_SAMPLE_MANIFEST_PATH=/app/storage/operator-data/mol-operational-1024/operator_samples_manifest.json \
  -e MOL_VALIDATION_OUTPUT_DIR=/app/storage/operator-evidence/mol-operational-validation/current \
  geointel bash /app/scripts/run_mol_operational_validation.sh http://127.0.0.1
```

Inside the all-in-one image that persistent storage path is also the automatic
default. Evidence therefore survives container replacement. Local repository
runs keep using `artifacts/mol-operational-validation/<timestamp>` unless the
output variable is set explicitly.

The runner defaults to the active local model at tile `512`, overlap `64`,
confidence `0.15` and QA IoU `0.25`. Every positive run persists Project, Area,
Dataset, Job, AnalysisRun, Detection, QualityCheck, Metric and Export records.
The background run persists its project, AOI, raster, job, analysis and
detections but intentionally does not invent QA metrics for an empty or sparse
reference context. Each matrix row now preserves the exact persisted inference
coverage counts and the diagnostic-only reference-envelope comparison beside
the canonical footprint-IoU metrics.

The runner also writes `mol_operational_benchmark_report.json` and `.md`. The
default operational gates require four positive holdouts, one background
control, coverage provenance for every positive run, at least 90% reference
coverage in every zone, mean F1 at least `0.25`, per-zone F1 at least `0.10`
and zero detections in each pure-empty control. Override the numeric gates only
through the documented `MOL_MIN_MEAN_F1`, `MOL_MIN_ZONE_F1`,
`MOL_MIN_REFERENCE_COVERAGE` and `MOL_MAX_BACKGROUND_DETECTIONS` variables.
Source bbox queries retain complete edge features and can intentionally
over-cover the exact projected raster polygon, so the coverage gate detects a
gross source/raster mismatch rather than scoring the model. An `accepted`
report records bounded operational evidence; it does not mutate
the active model. A `review_required` report is still a successful benchmark
execution but explicitly blocks a promotion recommendation.

For model-training candidates, prepare a larger operator-only sample manifest so
tile overlap can create meaningful context instead of one tile per source
raster:

```bash
docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py \
  --output-dir /app/storage/operator-data/operator-samples-1024 \
  --manifest-name operator_samples_manifest.json \
  --width 1024 \
  --height 1024 \
  --half-size-scale 2 \
  --reference-page-limit 1000 \
  --reference-max-features 100000 \
  --force
```

This keeps the same documented AOI centers but requests larger WMS rasters and a
larger GRB reference bbox. Use the generated manifest path for the next YOLO
tile export. The default remains 512x512 for quick smoke runs.

The real-data smoke is intentionally mutating and refuses to run without
operator-supplied files. Current V1 upload support expects a georeferenced
`.tif`, `.tiff` or `.geotiff` raster and a `.geojson` or `.json` reference
building vector. The script creates a project, uploads the raster as a source
dataset, uploads the vector as a `reference` dataset, validates raster/vector
metadata, tiles the raster, selects a mounted local model asset, verifies
read-only YOLO preflight, runs configured YOLO detection, runs detection QA
against persisted `vector_features`, and exports the detection run as GeoJSON.
It does not seed demo data, enable fixture detections, fetch external data or
download model weights. Configured-YOLO model class labels are normalized to
lowercase for filtering and persisted detections, while the original model label
is retained in detection provenance. Raster tile manifests generated by the
workflow include source CRS metadata so persisted detection GeoJSON coordinates
can be transformed to WGS84. A zero detection count is accepted
operationally only when the selected model genuinely returns no usable
detections after class filtering; it must be interpreted as model/data quality
evidence rather than as a successful building extraction result.

Run a confidence-threshold calibration sweep against the same real-data path:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
CALIBRATION_THRESHOLDS="0.50 0.35 0.25 0.15" \
bash scripts/run_detection_calibration_sweep.sh http://192.168.10.150:1202
```

The sweep reuses `verify_real_data_detection_qa_workflow.sh` once per
threshold, so every row is backed by persisted Project, Dataset, AnalysisRun,
Detection, QualityCheck, Metric and export records. It writes per-threshold
logs plus `calibration_summary.json` under
`artifacts/detection-calibration/<timestamp>` unless
`CALIBRATION_OUTPUT_DIR` is set. This is a calibration/benchmarking tool only:
it does not seed demo data, enable fixture detections, fetch external data or
download model weights.
Per-threshold summaries include persisted detection count, raw candidate count
before GeoIntel duplicate suppression, suppressed duplicate count and the
configured duplicate IoU threshold.

Run a broader model/tile/threshold quality matrix when multiple local model
assets or tile settings need to be compared:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_detection_quality_matrix.sh http://192.168.10.150:1202
```

The quality matrix repeats the same real-data upload, tiling, configured-YOLO,
QA/QC and export workflow for every model/tile/threshold row. It writes per-run
logs plus `quality_matrix_summary.json` under
`artifacts/detection-quality-matrix/<timestamp>` unless `QUALITY_OUTPUT_DIR` is
set. The summary ranks `best_by_score`, `best_by_recall` and
`best_by_precision` so the next model decision is based on persisted
`QualityCheck`/`Metric` evidence rather than visual guesses. It does not create
provider data, use fixtures or download model weights. Coverage-aware rows also
record raw/evaluated/excluded/clipped candidate and reference counts, tile
coverage provenance and the separately labelled box-to-footprint diagnostic
gap.

Run the same matrix across every prepared operator sample:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202
```

The multi-sample wrapper writes one per-sample `quality_matrix_summary.json`
plus a combined `multi_sample_quality_summary.json` with
`best_overall_by_score`, `best_overall_by_recall`,
`best_overall_by_precision` and `best_by_sample` rankings. It resolves
container-style `/app/storage/...` manifest paths to repo-relative
`storage/...` paths when run from the Tower host checkout.
Manifest-backed runs also persist the declared EPSG:4326 AOI and municipality
region, and retain municipality/operational-zone metadata in the combined
summary.

Export the same operator samples to a local YOLO detection dataset when the
public model candidates are not strong enough for the target imagery:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-dataset \
  --val-samples turnhout \
  --force
```

The exporter writes `dataset.yaml`, `images/train`, `labels/train`,
`images/val`, `labels/val` and `yolo_dataset_summary.json`. It uses only the
explicit operator sample manifest and GRB building references where
`source_name=grb` and `reference_layer_name=buildings`. It does not call
GeoIntel APIs, create provider data, run inference or train a model.

Run a small local training smoke only in an AI-enabled runtime with an existing
local base model file:

```bash
docker exec \
  -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-dataset \
  -e YOLO_BASE_MODEL_PATH=/app/models/yolov8n.pt \
  -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-detector.pt \
  -e TRAIN_EPOCHS=8 \
  -e TRAIN_IMGSZ=512 \
  -e TRAIN_BATCH=2 \
  -e TRAIN_WORKERS=0 \
  -e TRAIN_DEVICE=cpu \
  geointel bash /app/scripts/train_operator_yolo_detector.sh
```

The training wrapper is intentionally outside the product UI. It runs
Ultralytics from the existing runtime, copies the best trained artifact to
`TRAIN_MODEL_OUTPUT_PATH` and writes `training_summary.json`. The summary records
SHA256 provenance for `dataset.yaml`, the available YOLO dataset summary, the
local base model and the copied trained model. Afterward, treat
the resulting `.pt` file like any other local model asset: verify preflight,
run the real-data matrix and compare persisted QA/QC metrics before activating
it as a useful default.
Inside the all-in-one image the wrapper prefers
`/opt/geointel/venv/bin/python` when that AI runtime exists. Set `PYTHON_BIN`
only when intentionally overriding the interpreter.

When whole-image training does not improve QA/QC, export a tile-level dataset
with overlapping raster windows:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-aoi1024-visible025 \
  --tile-size 512 \
  --stride 256 \
  --negative-keep-ratio 1.0 \
  --min-label-visible-ratio 0.25 \
  --val-samples turnhout,retie,westerlo,arendonk_heide \
  --force
```

The tile exporter clips GRB building bounding boxes into each tile, writes
YOLO labels beside each tile image, keeps a deterministic ratio of empty
negative tiles, and records `yolo_tile_dataset_summary.json` with
`positive_tile_count`, `negative_tile_count` and skipped negative tile counts.
It separately records configured, retained and empty validation sample slugs;
this keeps a holdout that lost every tile to quality filtering visible without
pretending it contributed evaluation data.
`--min-label-visible-ratio` drops labels where only a small clipped fragment of
the original building bbox is visible inside the tile; this reduces noisy
tile-edge labels in overlapping-tile datasets. Use `0` for legacy behavior.
Use `--drop-low-variance-negatives` to skip negative tiles whose rendered image
has a max-min pixel range at or below `--blank-range-threshold`. This gate is
intended for blank/no-data pure-empty negatives only; positive/labeled tiles are
not removed by this filter. The summary records
`skipped_low_variance_negative_tile_count` and skipped tile records with
`skip_reason=low_visual_variance_negative`.
For legacy operator manifests that predate explicit `background_category`, the
exporter derives the same categories as the split-background evaluator:
background samples with `reference_feature_count == 0` become
`pure_empty_negative`, and background samples with one or more reference
features become `sparse_building_context`.
It remains operator tooling only: no provider fetch, no API mutation and no
automatic model training.

For the current AOI1024 baseline, prefer the stricter clean-label profile before
spending another training run:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035 \
  --tile-size 512 \
  --stride 256 \
  --negative-keep-ratio 1.0 \
  --min-label-px 12 \
  --min-label-visible-ratio 0.35 \
  --drop-low-variance-negatives \
  --blank-range-threshold 3 \
  --val-samples turnhout,retie,westerlo,arendonk_heide \
  --force
```

This refreshed cleanpx dataset is the minimum pre-training baseline after the
visual contact-sheet pass found six blank-looking `arendonk_heide` validation
negatives in the older export.

The persisted false-negative audit subsequently showed that the cleanpx12
candidate still misses about 79-92% of the comparable reference population and
misses every reference building below 25 m2 in the seven-AOI review. Do not
train another candidate from the same four positive training AOIs. Refresh the
existing AOI1024 sample directory after pulling Sprint 171; existing files are
reused and only the four new explicit positive AOIs need to be fetched:

```bash
docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py \
  --output-dir /app/storage/operator-data/operator-samples-1024 \
  --width 1024 \
  --height 1024 \
  --half-size-scale 2
```

The expansion adds Olen, Lille, Oud-Turnhout and Kasterlee center as training
samples. Turnhout, Retie, Westerlo and Arendonk-heide remain explicit validation
holdouts in generated manifest provenance. The tile exporter defaults to those
four holdouts and rejects a manifest-aware split that leaks one into training.
Use the lower `min-label-px=4` profile first to measure small-building retention;
it remains subject to dataset audit and visual contact-sheet review before any
training:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-aoi1024-expanded-minpx4vis035 \
  --tile-size 512 \
  --stride 256 \
  --negative-keep-ratio 1.0 \
  --min-label-px 4 \
  --min-label-visible-ratio 0.35 \
  --drop-low-variance-negatives \
  --blank-range-threshold 3 \
  --force
```

Then audit with stricter small-box gates:

```bash
docker exec -it geointel python3 /app/scripts/audit_operator_yolo_dataset_quality.py \
  --summary-path /app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035/yolo_tile_dataset_summary.json \
  --output-dir /app/artifacts/operator-yolo-dataset-audit/aoi1024-cleanpx12vis035 \
  --max-small-box-share 0.25 \
  --min-median-box-area 0.001
```

Audit the generated tile dataset before spending another long training run:

```bash
python scripts/audit_operator_yolo_dataset_quality.py \
  --summary-path /mnt/user/appdata/geointel/storage/operator-data/yolo-building-tile-hardneg160r8/yolo_tile_dataset_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/operator-yolo-dataset-audit/hardneg160r8
```

The audit reads the tile summary and YOLO label files, then writes
`operator_yolo_dataset_quality_audit.json` and
`operator_yolo_dataset_quality_audit.md`. It reports positive/background sample
coverage, train/validation split coverage, repeated hard-negative pressure,
minimum visible label ratio, missing or invalid label rows, normalized
box-area signals and per-sample label diagnostics such as parsed label count,
median box area, small-box share and sample-specific quality warnings. Treat
`needs_attention` as a dataset-design warning, not as a runtime failure: the
next action is usually more positive AOIs, better validation coverage or more
unique hard negatives rather than simply extending epochs.

Render visual label QA contact sheets before spending CPU on another training
run:

```bash
docker exec -it geointel python3 /app/scripts/render_operator_yolo_label_qa_contact_sheets.py \
  --summary-path /app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035/yolo_tile_dataset_summary.json \
  --output-dir /app/artifacts/operator-yolo-label-qa/aoi1024-cleanpx12vis035 \
  --max-tiles 32 \
  --columns 4 \
  --thumb-size 256
```

The renderer writes `operator_yolo_label_qa_summary.json`,
`operator_yolo_label_qa_contact_sheet.md` and `contact_sheet_001.png`. It draws
existing YOLO labels on existing tile images only; it does not run inference,
train a model, fetch providers or create fake detections. Missing image files,
missing label files, invalid YOLO rows and low-variance/blank-looking rendered
tiles are reported in the JSON/Markdown artifacts. Selection is balanced by
`sample_slug` before taking additional high-density tiles, so one dense urban
AOI cannot hide the other source samples from visual review. The report records
`selected_sample_count` and `selected_sample_slugs` for coverage evidence.

Current Tower audit status:

- `yolo-building-tile-expanded160`: clean baseline; no missing/invalid labels.
- `yolo-building-tile-hardneg160r4` and `yolo-building-tile-hardneg160r8`:
  repeat-heavy hard-negative variants; useful evidence, but add more unique
  background AOIs before training another hard-negative-balanced candidate.
- Regenerate `operator_samples_manifest.json` after pulling Sprint 147+ so the
  expanded unique background AOI set is available for the next tile export.
- `yolo-building-tile-uniquehardneg160`: clean expanded-background baseline;
  576 tiles, 346 positive, 230 negative, 11,757 labels, 0 invalid labels and
  0 repeated background negatives in the first Tower audit.
- `yolo-building-aoi1024-visible025`: larger AOI candidate baseline regenerated
  after paged GRB references; 144 tiles, 117 positive tiles, 27 negative tiles,
  29,170 labels, 0 missing label files, 0 invalid labels and
  `min_label_visible_ratio=0.25`. Its audit remains `needs_attention` because
  the median normalized box area is still small (`0.000762939453125`) and the
  small-box share is high (`0.34744600617072335`). The previously trained
  `geointel-building-yolov8s-aoi1024visible025e50-pt` asset is inactive; do not
  retrain or activate from this dataset without addressing label quality or
  explicitly accepting the small-box warning.
- `yolo-building-aoi1024-cleanpx12vis035`: stricter clean-label AOI1024 export;
  144 tiles, 114 positive tiles, 30 negative tiles, 14,632 labels,
  `min_label_px=12`, `min_label_visible_ratio=0.35`, 0 invalid labels,
  median normalized box area `0.001373291015625` and small-box share `0.0`.
  The trained `geointel-building-yolov8s-aoi1024cleanpx12vis035e50-pt` asset is
  available but inactive. The split-aware promotion report rejected all tested
  thresholds: `0.25` and `0.35` passed the pure-empty background gate but had
  mean F1 below `0.25`; `0.15` also failed the pure-empty false-positive gate.
- `yolo-building-aoi1024-expanded-minpx4vis035`: expanded small-building
  recovery dataset with 20 source AOIs, 171 retained tiles, 45,892 labels,
  144 train tiles, 27 validation tiles and 9 low-variance negatives removed.
  Its configured audit passed with no warnings, median normalized box area
  `0.000694274766`, small-box share `0.3832694151486098`, no invalid labels and
  no missing label files. The balanced visual pass rendered 40 tiles across all
  19 source samples that retained at least one tile, with no invalid labels,
  missing images or low-variance selections. Its promoted model remains the
  higher-precision legacy `0.15` operator profile.
- `yolo-building-aoi1024-smallbld-minpx3vis035`: focused small-building corpus
  exported from an explicit 23-sample subset. Beerse, Rijkevorsel, Hoogstraten
  and Vorselaar extend training; Vosselaar and Grobbendonk are validation-only;
  Turnhout, Retie and Westerlo remain external operation-level holdouts. The
  Tower export retained 198 tiles and 58,820 labels. Its small-object-aware
  audit passed with no invalid/missing labels, and the 48-tile balanced visual
  review contained no missing, invalid or low-variance selections. The trained
  `geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt` candidate passed
  seven positive-AOI and three pure-empty background gates at tile `512`,
  overlap `64`, threshold `0.15` and QA match IoU `0.25`. Original promotion
  evidence measured mean F1 `0.5825`; the later coverage-aligned rerun measured
  `0.6069`. All pure-empty samples remain at zero detections. The original
  persisted comparison found 1,571 fewer false negatives than the previous
  balanced model, with a lower mean precision and therefore a higher operator
  review load.
- `yolo-building-aoi1024-reviewedexp6-minpx3vis035`: leakage-free accuracy
  expansion of the focused corpus with new training-only AOIs in Arendonk,
  Dessel, Meerhout, Laakdal, Nijlen and Hulshout. The paged GRB preparation
  added 9,964 reference features. The export retained 252 tiles (234 positive,
  18 negative) and 79,192 labels; its configured audit and 64-tile visual
  review found no invalid, missing or low-variance selections. The inactive
  `geointel-building-yolov8s-reviewedexp6-minpx3-img640-ft20-pt` model improved
  coverage-aligned seven-zone mean F1 from `0.6069` to `0.6248`, but its two
  detections in empty Postel-bos failed the strict background gate. The active
  focused model remains unchanged.

Use `--samples` or `OPERATOR_YOLO_SAMPLES` to make an experimental corpus
membership explicit. Dataset summaries preserve the complete manifest count,
selected sample slugs and excluded sample slugs. Split validation still applies
after filtering, so a manifest-backed holdout cannot be selected as training by
omitting it from `--val-samples`.

After rebuilding the all-in-one image, the operator scripts are available inside
the container at `/app/scripts/...`. Before rebuilding, use the host checkout or
temporarily copy scripts into the running container for one-off data prep.

For hard-negative-balanced experiments, repeat only train-split negative tiles
from samples marked `sample_role=background_candidate`:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-tile-hardneg160r8 \
  --tile-size 160 \
  --stride 80 \
  --negative-keep-ratio 1.0 \
  --background-negative-repeat 8 \
  --val-samples turnhout,retie,kasterlee_bos \
  --force
```

The repeat option can also be set with
`OPERATOR_YOLO_BACKGROUND_NEGATIVE_REPEAT`. It does not duplicate validation
tiles, positive tiles or normal reference-sample negatives. Repeated background
tiles receive deterministic `_hnXX` filenames and tile metadata records
`sample_role`, `repeat_index` and `is_repeated_background_negative`.

Train against the tile dataset by pointing the existing wrapper at the tile
output directory:

```bash
docker exec \
  -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-tile-expanded160 \
  -e YOLO_BASE_MODEL_PATH=/app/models/yolov8n.pt \
  -e TRAIN_OUTPUT_DIR=/app/storage/training/operator-yolo \
  -e TRAIN_RUN_NAME=geointel-building-yolov8n-expanded160e50 \
  -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-yolov8n-expanded160e50.pt \
  -e TRAIN_EPOCHS=50 \
  -e TRAIN_IMGSZ=256 \
  -e TRAIN_BATCH=8 \
  -e TRAIN_WORKERS=0 \
  -e TRAIN_DEVICE=cpu \
  geointel bash /app/scripts/train_operator_yolo_detector.sh
```

Benchmark any trained candidate through the same persisted QA/QC matrix before
using it operationally:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_SAMPLE_SLUGS="geel mol turnhout retie kasterlee_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
QUALITY_TILE_SIZES="640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
MULTI_SAMPLE_OUTPUT_DIR=artifacts/detection-quality-matrix/multi-sample/expanded160e50-live \
bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202
```

The expanded 50-epoch candidate improved dense Geel/Mol/Turnhout/Retie scores,
but the sparse Kasterlee-bos run still showed too many false positives. Treat it
as the best current experimental dense-AOI candidate, not as a V1 default.

Run a dedicated hard-negative matrix against documented background candidates
before changing model defaults:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos dessel_heide ravels_bos meerhout_bos geel_bel arendonk_heide herenthout_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8s-aoi1024bg512r3e50-pt" \
QUALITY_TILE_SIZES="512" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.35 0.15" \
BACKGROUND_SPLIT_OUTPUT_DIR=artifacts/detection-hard-negatives/aoi1024bg512r3e50-split \
bash scripts/run_background_corpus_split_matrix.sh http://192.168.10.150:1202
```

The split runner executes the strict `pure_empty_negative` matrix and the
review-only `sparse_building_context` matrix as separate runs, then writes
`background_corpus_split_summary.json` and
`background_corpus_split_summary.md`. Use the pure-empty block for the
default-promotion false-positive gate; use sparse-context results as review
evidence only.

The lower-level hard-negative matrix can still be run directly:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_BACKGROUND_CATEGORIES="pure_empty_negative" \
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos dessel_heide ravels_bos meerhout_bos geel_bel arendonk_heide herenthout_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
QUALITY_TILE_SIZES="640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
HARD_NEGATIVE_OUTPUT_DIR=artifacts/detection-hard-negatives/expanded160e50-live \
bash scripts/run_operator_hard_negative_detection_matrix.sh http://192.168.10.150:1202
```

The hard-negative matrix uploads only the background raster, generates tiles,
runs configured-YOLO detection and counts persisted detections as
`false_positive_pressure`. It does not upload a reference vector and does not
run QA/QC, because empty or sparse background AOIs do not have a meaningful
precision/recall target. Use `OPERATOR_BACKGROUND_CATEGORIES="pure_empty_negative"`
for the strict default-promotion false-positive gate. Run
`OPERATOR_BACKGROUND_CATEGORIES="sparse_building_context"` separately for
contextual review; sparse-context detections should be inspected, not counted
as fake precision/recall metrics. In the first live run,
`geointel-building-yolov8n-expanded160e50-pt`
was clean on Postel-bos and Lommel-heide at thresholds `0.25` and `0.15`, but
produced 38 detections on Kasterlee-bos even at `0.25`. That blocks it from
becoming a V1 default until a hard-negative-balanced candidate improves.

The hard-negative-balanced `geointel-building-yolov8n-hardneg160r8e40-pt`
candidate reduced Kasterlee-bos false-positive pressure to 5/9/25 detections
at thresholds `0.25`/`0.15`/`0.05` and stayed at 0 detections on Postel-bos and
Lommel-heide across all tested thresholds. It also regressed dense-AOI F1
against `geointel-building-yolov8n-expanded160e50-pt`, so it is useful model
quality evidence but not a V1 default.

An external remote-sensing YOLOv8l candidate was also benchmarked as an
operator-provided local model asset:

```bash
mkdir -p models
curl -L --fail \
  -o models/yolo-remote-sensing-photovoltaic-v8l-detect-1000.pt \
  https://huggingface.co/agademer/yolo-remote-sensing-photovoltaic/resolve/main/yolo-remote-sensing-photovoltaic-v8l-solar-farms-and-cities-v20260331-detect-1000_epochs.pt
```

GeoIntel exposed the file as
`yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt` with SHA256
`242ff4ab889569278f0eb9fcd22eb2c4bf2a52e48d05d89cc7cfa7941165d203`, and
YOLO preflight loaded it without downloads. On the live dense matrix it missed
most buildings and scored far below `geointel-building-yolov8n-expanded160e50-pt`
on Geel, Mol, Turnhout and Retie. On Kasterlee-bos it was clean and precise,
but that sparse-AOI behavior is not enough for V1 extraction. Keep it as
benchmark evidence only, not as a default model.

Export calibration QA evidence for visual review:

```bash
CALIBRATION_SUMMARY_PATH=/mnt/user/appdata/geointel/artifacts/detection-calibration/20260707T002103Z/calibration_summary.json \
bash scripts/export_detection_calibration_evidence.sh http://192.168.10.150:1202
```

Browser Detection Lab calibration summary exports are supported too:

```bash
bash scripts/export_detection_calibration_evidence.sh http://192.168.10.150:1202 ./detection-calibration-summary.json
```

Run the local browser-summary evidence bundle smoke without touching live
application data:

```bash
bash scripts/smoke_detection_calibration_evidence_bundle.sh
```

The smoke creates a temporary Detection Lab-style calibration summary, mocks
the canonical persisted QA evidence endpoint responses, runs the real evidence
exporter and verifies that `calibration_evidence.geojson`,
`calibration_evidence_summary.json` and `calibration_evidence_review.html` are
written correctly.

Assemble multiple AOI evidence bundles into one model-review portfolio:

```bash
bash scripts/assemble_detection_calibration_evidence_portfolio.sh \
  http://192.168.10.150:1202 \
  ./calibration-evidence-portfolio-manifest.json
```

Example `calibration-evidence-portfolio-manifest.json`:

```json
{
  "portfolio_name": "Kempen building model calibration",
  "model_asset_id": "geointel-building-yolov8s-hardneg160r4e50-pt",
  "model_sha256": "optional-model-checksum",
  "notes": "Operator comparison notes.",
  "samples": [
    {
      "sample_slug": "geel",
      "aoi_label": "Geel center",
      "summary_path": "/path/to/detection-calibration-summary.json",
      "operator_notes": "Dense urban validation sample."
    }
  ]
}
```

The portfolio assembler copies each summary into a deterministic sample folder,
runs the existing evidence exporter per AOI and writes
`calibration_evidence_portfolio.json` plus
`calibration_evidence_portfolio.md`. It is evidence packaging only: it does not
run inference, create QA checks or mutate application data.

The evidence export reads each persisted `quality_check_id`, calls the existing
QA evidence GeoJSON endpoint, writes `calibration_evidence.geojson`,
`calibration_evidence_summary.json` and a standalone
`calibration_evidence_review.html` with an SVG overview of matched detections,
matched references, false positives and false negatives. Set
`CALIBRATION_EVIDENCE_MODE=best` to export only the `best_by_score` run.

Build fixed-threshold portfolio inputs when two model runs must be compared at
the same confidence threshold across every AOI:

```bash
python scripts/build_fixed_threshold_evidence_portfolio_inputs.py \
  --multi-sample-summary artifacts/detection-quality-matrix/multi-sample/<run>/multi_sample_quality_summary.json \
  --threshold 0.35 \
  --model-asset-id geointel-building-yolov8s-aoi1024bg512r3e50-pt \
  --model-sha256 e0980572aac90e7efc514608eb16d7de5bfbf27a4bbec04e7bc1bc8c02f9601f \
  --tile-size 512 \
  --tile-overlap 64 \
  --output-dir artifacts/detection-false-negative-review/active-inputs
```

The builder selects exactly one persisted QA run per AOI and refuses ambiguous
model/tile/threshold matches. Pass its emitted manifest to
`assemble_detection_calibration_evidence_portfolio.sh` with
`CALIBRATION_EVIDENCE_MODE=all`; each filtered summary contains one run.

Compare two or more downloaded evidence portfolios with geodetic WGS84 areas:

```bash
python scripts/audit_detection_false_negative_evidence.py \
  --portfolio active=artifacts/detection-false-negative-review/active/calibration_evidence_portfolio.json \
  --portfolio candidate=artifacts/detection-false-negative-review/candidate/calibration_evidence_portfolio.json \
  --output-dir artifacts/detection-false-negative-review/audit
```

The audit reports false-negative rates and area buckets per AOI/model, plus
reference buildings missed by every compared portfolio. It writes the combined
`persistent_false_negatives.geojson`, records geodetic persistent-miss area and
adds persistent area buckets so operators can inspect the shared misses on a
map instead of relying only on counts. Stable
`source_feature_id` values are preferred; a normalized geometry fingerprint is
used only when source IDs are absent. Invalid or missing geometry fails the
audit instead of being silently skipped. The tools do not run inference,
create QA records, mutate model defaults or download data/models.

Audit the false-positive review load of one persisted evidence portfolio before
turning detections into hard-negative training input:

```bash
python scripts/audit_detection_false_positive_evidence.py \
  --portfolio artifacts/model-review/small-building-candidate/evidence-portfolio/calibration_evidence_portfolio.json \
  --output-dir artifacts/model-review/small-building-candidate/false-positive-audit
```

The command validates the portfolio role counts against each persisted evidence
GeoJSON, rejects invalid/non-polygon geometry, computes WGS84 geodesic area and
size buckets, and reports false-positive pressure per AOI, class and
AOI-qualified source tile. It writes `detection_false_positive_audit.json`, a
Markdown handoff and combined `false_positives.geojson` for map review. Original
evidence properties and geometry are preserved. Confidence statistics are only
computed when confidence is actually present in persisted evidence; missing
coverage is reported explicitly and never inferred from the run threshold.

Render a deterministic, stratified visual review over persisted false-positive
evidence. Static portfolios created before detection provenance was added must
first be re-exported from the current backend; existing `QualityCheck` and
`Detection` rows do not need to be rerun:

```bash
docker exec \
  -e CALIBRATION_EVIDENCE_MODE=all \
  -e CALIBRATION_PORTFOLIO_OUTPUT_DIR=/app/storage/operator-data/model-review/small-building-candidate/evidence-portfolio-enriched \
  geointel bash /app/scripts/assemble_detection_calibration_evidence_portfolio.sh \
    http://127.0.0.1 \
    /app/storage/operator-data/model-review/small-building-candidate/evidence-inputs/calibration-evidence-portfolio-manifest.json

docker exec geointel /opt/geointel/venv/bin/python \
  /app/scripts/render_detection_false_positive_review_contact_sheets.py \
  --portfolio /app/storage/operator-data/model-review/small-building-candidate/evidence-portfolio-enriched/calibration_evidence_portfolio.json \
  --storage-root /app/storage \
  --output-dir /app/storage/operator-data/model-review/small-building-candidate/false-positive-visual-review \
  --sample-slugs turnhout,herentals,geel \
  --max-features 48 \
  --columns 4 \
  --cards-per-sheet 16 \
  --thumb-size 256
```

The renderer validates source paths against `--storage-root`, checks persisted
confidence/bbox/tile provenance, samples across AOI, WGS84 area bucket and
confidence band, and overlays persisted matched/missed reference polygons. It
writes PNG sheets, a JSON/Markdown summary and
`false_positive_review_decisions.csv` with every row set to `unreviewed`.

After manual inspection, validate the edited CSV:

```bash
docker exec geointel /opt/geointel/venv/bin/python \
  /app/scripts/validate_detection_false_positive_review_decisions.py \
  --review-summary /app/storage/operator-data/model-review/small-building-candidate/false-positive-visual-review/detection_false_positive_review_summary.json \
  --decisions-csv /app/storage/operator-data/model-review/small-building-candidate/false-positive-visual-review/false_positive_review_decisions.csv \
  --output-dir /app/storage/operator-data/model-review/small-building-candidate/false-positive-visual-review/validated \
  --require-complete
```

`--require-complete` exits with code `2` while any record is still `unreviewed`.
Only explicit `confirmed_model_false_positive` decisions are written to
`confirmed_model_false_positives.geojson`; the tool never promotes generic QA
false-positives into training labels.

Render persisted false negatives against the exact tile manifest recorded by
the selected analysis run:

```bash
docker exec geointel /opt/geointel/venv/bin/python \
  /app/scripts/render_detection_false_negative_review_contact_sheets.py \
  --portfolio /app/storage/operator-evidence/model-review/portfolio/calibration_evidence_portfolio.json \
  --storage-root /app/storage \
  --output-dir /app/storage/operator-evidence/model-review/false-negative-visual-review \
  --sample-slugs mol_donk,mol_postel \
  --max-features 48 \
  --columns 4 \
  --cards-per-sheet 12 \
  --thumb-size 256
```

The read-only renderer resolves the one persisted `manifest_path` from each
fixed-threshold sample summary, confines manifests and source tiles to
`--storage-root`, and projects WGS84 missed-reference polygons onto the real
source tiles. Red is the missed reference, blue is persisted candidate
geometry and green is a matched reference. Selection is deterministic and
stratified by AOI and geodetic area bucket. Every CSV decision starts as
`unreviewed`; no positive-training example is inferred.

Validate the completed false-negative decisions symmetrically with the
false-positive workflow:

```bash
docker exec geointel /opt/geointel/venv/bin/python \
  /app/scripts/validate_detection_false_negative_review_decisions.py \
  --review-summary /app/storage/operator-data/model-review/small-building-candidate/false-negative-visual-review/detection_false_negative_review_summary.json \
  --decisions-csv /app/storage/operator-data/model-review/small-building-candidate/false-negative-visual-review/false_negative_review_decisions.csv \
  --output-dir /app/storage/operator-data/model-review/small-building-candidate/false-negative-visual-review/validated \
  --require-complete
```

The validator exits with code `2` while any row remains `unreviewed`. It emits
only explicit `confirmed_model_false_negative` geometries; alignment,
reference-gap and uncertain imagery decisions never become positive training
labels.

Reference features that do not intersect any persisted inference tile are not
silently counted as reviewable model misses. They are reported separately in
`false_negatives_outside_tile_coverage.geojson` with
`review_exclusion_reason=outside_tile_coverage`. Fix the QA evaluation
population before using those records in recall or training decisions.

Docker images install only the GIS runtime by default. To build a local/Tower
image with PyTorch/Ultralytics available for the configured-YOLO preflight and
runtime path, set:

```bash
GEOINTEL_INSTALL_AI=true
```

For Unraid/all-in-one deployments, place model files under
`GEOINTEL_MODELS_PATH` so they appear in the container under `/app/models`, then
set `YOLO_ENABLED=true`, `YOLO_MODELS_DIR=/app/models` and
`YOLO_MODEL_PATH=/app/models/<model>.pt`.

Configure the Unraid/Tower env file from an existing local model without
downloading weights or running inference:

```bash
python scripts/configure_yolo_model.py \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env
```

If exactly one supported model file (`.pt`, `.onnx` or `.engine`) is present,
apply the env update explicitly:

```bash
python scripts/configure_yolo_model.py \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env \
  --apply
```

The configurator refuses to proceed when no model exists or when multiple model
files are present without `--model-file`. It writes only
`GEOINTEL_INSTALL_AI=true`, `YOLO_ENABLED=true`, `YOLO_MODELS_DIR=/app/models`
and the mounted `YOLO_MODEL_PATH`.

When a split-background promotion report recommends a specific candidate, use
the guarded activation helper instead of choosing a model path manually. The
helper validates the exact `candidate_key`, promotion status and local model
asset before writing anything, and it mutates `.env` only with `--apply`:

```bash
python scripts/activate_promoted_yolo_candidate.py \
  --promotion-report /mnt/user/appdata/geointel/artifacts/detection-model-promotion/split-aware/aoi1024bg512r3e50-high-threshold-split-20260710T222934Z/detection_model_promotion_report.json \
  --candidate-key 'geointel-building-yolov8s-aoi1024bg512r3e50-pt|512|64|0.35' \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env \
  --json
```

Add `--apply` only after reviewing the emitted updates. The helper never
downloads weights, loads the model or runs inference; restart or rebuild the
container after applying because `YOLO_MODEL_PATH` is read from the environment.

Tower-local model evaluation status:

- `geointel-building-yolov8s-hardneg160r4e50.pt` is available as an evaluated
  local runtime artifact after the hard-negative YOLOv8s training pass.
- The live catalog id is `geointel-building-yolov8s-hardneg160r4e50-pt`.
- The model SHA256 is
  `9bf71ad4742048ac77f07060b677bacd9757b8d310497fcada334d543e320d19`.
- The current safest observed threshold is `0.25`, but the model remains an
  evaluation candidate because one hard-negative forest sample still produced
  false detections at that threshold.
- Do not silently activate this model as a default. Apply it only as an explicit
  operator choice until the model catalog/threshold workflow is hardened.

Build a model promotion decision report from an existing positive-AOI evidence
portfolio and one or more hard-negative/background summaries:

```bash
python scripts/build_detection_model_promotion_report.py \
  --positive-portfolio /mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/positive-aoi-expanded-20260708/output/calibration_evidence_portfolio.json \
  --hard-negative-summary /mnt/user/appdata/geointel/artifacts/detection-hard-negatives/hardneg160r8e40-live/hard_negative_matrix_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/detection-model-promotion/expanded-positive-vs-hard-negative-20260708
```

The report writes `detection_model_promotion_report.json` and
`detection_model_promotion_report.md`. It groups candidates by
`model_asset_id`, tile size, tile overlap and confidence threshold, then applies
explicit gates for positive-AOI sample count, background sample count, mean F1
and maximum background detections per sample. It is evidence/report tooling
only: it does not run inference, mutate application data, download models or
change the active YOLO configuration.

When the background corpus has been split with
`run_background_corpus_split_matrix.sh`, pass the combined split summary instead
of manually wiring both category summaries:

```bash
python scripts/build_detection_model_promotion_report.py \
  --positive-portfolio /mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
  --background-split-summary /mnt/user/appdata/geointel/artifacts/detection-hard-negatives/aoi1024bg512r3e50-split/background_corpus_split_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/detection-model-promotion/aoi1024bg512r3e50-split-aware \
  --min-positive-samples 7 \
  --min-background-samples 2 \
  --min-mean-f1 0.25 \
  --max-background-detections-per-sample 0
```

The promotion report resolves the split summary's `pure_empty_negative` source
summary as the strict default-promotion false-positive gate. The
`sparse_building_context` source remains visible in the JSON/Markdown report as
review evidence only and is not counted as a default-promotion gate.

To run both steps after one redeploy, use the workflow wrapper:

```bash
PROMOTION_POSITIVE_PORTFOLIO_PATH=/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8s-aoi1024bg512r3e50-pt" \
QUALITY_TILE_SIZES="512" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.35 0.15" \
BACKGROUND_SPLIT_OUTPUT_DIR=artifacts/detection-hard-negatives/aoi1024bg512r3e50-split \
PROMOTION_OUTPUT_DIR=artifacts/detection-model-promotion/aoi1024bg512r3e50-split-aware \
bash scripts/run_split_background_promotion_workflow.sh http://192.168.10.150:1202
```

The wrapper first calls `run_background_corpus_split_matrix.sh`, then feeds the
generated `background_corpus_split_summary.json` into the split-aware promotion
report. It still uses only existing upload, detection and report paths; it does
not fetch providers, fetch model weights or activate a default.

Run the same command with `--preflight-only` first when checking a fresh
redeploy. Preflight validates the positive portfolio path, operator manifest
presence, required `pure_empty_negative` and `sparse_building_context`
background categories, Python/curl availability and the frontend API proxy
envelope without starting inference:

For older operator manifests that predate explicit `background_category`,
preflight uses the same fallback as the matrix runner: background samples with
`reference_feature_count == 0` are treated as `pure_empty_negative`, and
background samples with references are treated as `sparse_building_context`.

```bash
PROMOTION_POSITIVE_PORTFOLIO_PATH=/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
bash scripts/run_split_background_promotion_workflow.sh --preflight-only http://192.168.10.150:1202
```

If a legacy positive evidence portfolio records `model_asset_id` at portfolio
level but does not include per-run tile size/overlap, pass explicit tile
defaults instead of letting the report guess:

```bash
python scripts/build_detection_model_promotion_report.py \
  --positive-portfolio /mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/uniquehardneg160e50-positive/calibration_evidence_portfolio.json \
  --hard-negative-summary /mnt/user/appdata/geointel/artifacts/detection-hard-negatives/uniquehardneg160e50/hard_negative_matrix_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/detection-model-promotion/uniquehardneg160e50-positive-vs-hard-negative \
  --min-positive-samples 7 \
  --min-background-samples 9 \
  --min-mean-f1 0.25 \
  --max-background-detections-per-sample 0 \
  --default-positive-tile-size 640 \
  --default-positive-tile-overlap 64
```

Clean old offline demo export artifacts without touching uploaded source data:

```bash
python scripts/cleanup_demo_artifacts.py
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
```

Against the Docker runtime, run the backend-container entrypoint:

```bash
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
```

The cleanup script is dry-run by default. It only targets the explicit
`GeoIntel Demo - Building QA` project unless `--project-name` is provided, keeps
the newest exports per matching project, deletes only `exports` rows/files when
`--apply` is set, and refuses to remove files outside the configured
`STORAGE_ROOT`. `--max-delete` defaults to 25 and blocks large cleanup runs until
the operator raises it after reviewing dry-run output. Repeat `--export-type` to
limit cleanup to specific artifact kinds such as `project_report_html` or
`project_metadata_json`.

Verify the cleanup path against a running backend without deleting anything:

```bash
bash scripts/verify_demo_cleanup_dry_run.sh
CLEANUP_MODE=compose bash scripts/verify_demo_cleanup_dry_run.sh
CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh
```

The smoke runs the cleanup command without `--apply`, expects `dry_run=true`,
expects `deleted_export_count=0`, verifies candidate fields are present and
prints the matched/type-filtered/selected counts. Use `KEEP_LATEST`,
`MAX_DELETE` and `EXPORT_TYPE` environment variables to adjust the dry-run
thresholds without changing the script. The main readiness gate checks this
script's syntax; run it explicitly against Docker/PostGIS when validating a
live deployment.

## Complete Mol municipality workspace

Provision the official municipality boundary and every GRB GBG building that
intersects it through the existing project, area, dataset and vector-feature
persistence paths:

```bash
docker exec -it geointel python3 \
  /app/scripts/provision_mol_municipality_workspace.py
```

The command queries `VRBG/Refgem` for municipality `Mol`, verifies NIS code
`13025`, follows every `GRB/GBG` pagination link and clips the resulting
buildings to the official boundary. It writes source artefacts and a checksum
manifest below `/app/storage/operator-data/mol-municipality`, then creates or
reuses `Mol Municipality Workbench` and imports both layers through the public
API. It never writes directly to PostGIS and never runs implicitly at startup.

Completed artefacts and ready datasets are reused on a repeat run. Use
`--force` only when an operator deliberately wants to refetch and replace the
local source artefacts. Use `--fetch-only` to prepare and inspect the manifest
without changing application persistence. The internal API default is
`http://127.0.0.1:8000`, avoiding proxy timeouts during the large vector
import; override it with `--base-url` when running outside the all-in-one
container.

After the municipality boundary/building workspace exists, provision the
official GRB road, water and parcel context layers:

```bash
docker exec -it geointel python3 \
  /app/scripts/provision_mol_context_layers.py
```

The command reads `Wegsegment` for roads, `WTZ`/`WLAS`/`WGR` for water and
`ADP` for parcels from the Digitaal Vlaanderen OGC API, clips every geometry
to the persisted official Mol boundary and uploads each artifact through the
normal dataset API. Artifacts and manifests are retained below
`/app/storage/operator-data/mol-context`. Repeat runs reuse both artifacts and
datasets; use `--force` only for an explicit source refresh. Use
`--layers roads,water` or `--fetch-only` for a bounded operator run.

The context provisioner itself does not add population or forest values. Use
the dedicated official-source operators below; zero is never substituted for
missing source data.

## Official Mol temporal sources

After the municipality workspace exists, provision official Statbel population
and both independently modelled land-use series:

```bash
docker exec -it geointel python3 /app/scripts/provision_mol_population_history.py
docker exec -it geointel python3 /app/scripts/provision_mol_historical_landuse.py
docker exec -it geointel python3 /app/scripts/provision_official_landuse_timeseries.py
```

The modern land-use command checks the MercatorNet WCS capabilities, downloads
only the Mol bounding subset of each 10 m `EPSG:31370` raster, validates the
categorical integer grid, clips against the official boundary and polygonizes
class `12` (`Bos`). Raw rasters, vector artifacts and checksum manifests are
stored under `/app/storage/operator-data/official-landuse/mol`. The resulting
2013, 2016, 2019, 2022 and 2025 vectors are uploaded through the canonical API
as `department-omgeving:land-use:forest:mol`.

Prepare and inspect artifacts without changing the database:

```bash
docker exec -it geointel python3 \
  /app/scripts/provision_official_landuse_timeseries.py --fetch-only
```

Use `--force` only to refetch and rebuild local artifacts. Existing persisted
snapshots remain immutable and are reused by year/series. To use the operator
for another approved region, pass all scope inputs explicitly, for example
`--boundary-path`, `--project-name`, `--area-name`, `--municipality-name`,
`--nis-code`, `--scope-key` and `--output-dir`. GeoIntel does not infer what
"Kempen" means administratively.

The 2013-2025 series is methodologically separate from the historical
1778/1873/1969 series. The map-first Evolution view exposes a series selector
when both exist; it never calculates one continuous trend across those source
families.

## Official Kempen operational scope

GeoIntel defines its regional `Kempen` workspace as the official Vlaamse
`Vervoerregio Kempen`: 28 explicitly registered municipalities. This is a
reproducible policy boundary, not a claim about the wider cultural,
landscape or historical Kempen.

Prepare and inspect the current VRBG union and all member boundaries without
changing persistence:

```bash
docker exec -it geointel python3 /app/scripts/provision_geographic_scope.py \
  --scope kempen-transport-region --fetch-only
```

Persist the complete scope foundation:

```bash
docker exec -it geointel python3 /app/scripts/provision_geographic_scope.py \
  --scope kempen-transport-region
```

The command creates or reuses `Kempen Regional Workbench`, the regional Area,
28 municipality Areas and two VRBG source datasets through the canonical API.
Artifacts and checksums remain below
`/app/storage/operator-data/geographic-scopes/kempen-transport-region`.
Repeat runs are idempotent; `--force` refreshes today's source snapshot.

This command provisions boundaries only. Regional buildings, population,
land use, roads, water and parcels must be added by bounded source operators;
missing themes remain unavailable and are never filled with synthetic values.

### Regional Kempen GRB buildings

After the scope foundation exists, fetch and persist the complete current GRB
`GBG` building layer for the approved region:

```bash
docker exec -it geointel python3 /app/scripts/provision_regional_grb_buildings.py \
  --scope kempen-transport-region
```

Use `--fetch-only` to build and inspect artifacts without touching the
database. The operator uses one bounded request/pagination sequence per
official municipality and writes resumable partitions below
`/app/storage/operator-data/regional-themes/kempen-transport-region/buildings/<date>`.
Every source feature is assigned to exactly one partition using its largest
intersection with the member boundaries; exact ties use the lowest NIS code.
This avoids duplicate building counts at shared borders while retaining the
feature clipped only to the complete regional scope.

Once all 28 partitions are complete, the operator streams one combined
GeoJSON artifact and invokes `DatasetService`/`VectorFeatureService` inside
the container. Features are indexed in batches as one normal regional
reference dataset, so the existing viewport and rectangle-selection paths do
not need a parallel API. The service import is intentionally local-only and
refuses a remote backend URL. Repeat runs reuse checksummed artifacts and the
persisted dataset; use a new `--observed-date` for a newer immutable snapshot.

Safety limits can be adjusted explicitly with
`--max-features-per-member`, `--max-total-features`, `--page-limit` and
`--batch-size`. Exceeding a limit fails the run instead of producing a
truncated dataset. The command never runs during startup or an interactive
map query.

The verified Tower snapshot for `2026-07-14` contains 466,078 unique
buildings from 879 GRB source pages. Its 28 retained partitions produce one
478,143,249-byte managed artifact and Dataset
`e236fa03-7fce-4b37-bc7d-8bddd4d50346`; an immediate repeat run reused the
same checksums and Dataset instead of fetching or inserting duplicates.

### Regional Kempen GRB roads, water and parcels

Provision the remaining current GRB context snapshots after the scope
foundation is present:

```bash
docker exec -it geointel python3 /app/scripts/provision_regional_grb_context.py \
  --scope kempen-transport-region \
  --layers roads water parcels
```

Each theme has an independent manifest and can be resumed or rerun alone:

```bash
docker exec -it geointel python3 /app/scripts/provision_regional_grb_context.py \
  --scope kempen-transport-region --layers roads
```

Use `--fetch-only` to prepare checksummed artifacts without database writes.
Artifacts are retained below
`/app/storage/operator-data/regional-themes/kempen-transport-region/<theme>/<date>`.
The operator follows all OGC API pagination links, fails rather than hiding a
safety-limit truncation and persists one normal PostGIS dataset per theme via
DatasetService. Repeat runs reuse valid partitions, combined artifacts and
datasets; `--force` refreshes the selected observation date.

The semantic limits are deliberate: `Wegsegment` is road-map context rather
than traffic or routing data; `WTZ`/`WLAS`/`WGR` are heterogeneous water
objects rather than a volume metric; `ADP` is the graphical presumed cadastral
parcel location rather than a legal survey. The public GRB provider remains
`not_configured`, and no fetch runs from the browser or during startup.

The verified Tower snapshot for `2026-07-14` contains:

- roads: 84,504 unique `Wegsegment` objects in Dataset
  `715b24f1-148a-4d50-9c40-d5468f670ffb`;
- water: 88,332 unique objects (25,648 `WTZ`, 4,605 `WLAS` and
  58,079 `WGR`) in Dataset `689bb65d-0b03-4c5c-8f42-b5d2d899d8ca`;
- parcels: 415,288 unique `ADP` objects in Dataset
  `3cf5e8da-9ab0-49ef-96e4-f74b7557e0a4`.

All three manifests report `reference_truncated=false`. PostGIS row counts
and distinct source IDs match the manifests exactly, all geometries are
non-empty and valid in EPSG:4326, and immediate repeat runs reuse the same
artifact and Dataset.

## Regional official time series

Synchronize official population and modern forest snapshots after the
geographic scope workspace exists:

```bash
docker exec geointel python /app/scripts/provision_regional_timeseries.py
```

The explicit operator resolves the checksummed official scope boundary and
coordinates Statbel 2021-2025 with Departement Omgeving
2013/2016/2019/2022/2025. It is idempotent and persists only through the
canonical dataset upload API. It never runs at application startup. Prepare
artifacts without persistence using `--fetch-only`; bound a run with
`--skip-population`, `--skip-landuse`, `--population-years` or
`--landuse-years`. Regional WCS downloads use the 28 official municipality
boundaries as resumable request partitions, preserve the native 10 m
resolution and merge locally before exact clipping to the regional union.

## Waterinfo station histories

Provision real annual station observations for the persisted Mol Area:

```bash
docker exec geointel python /app/scripts/provision_waterinfo_station_history.py \
  --project-name "Kempen Regional Workbench" \
  --area-name "Gemeente Mol" \
  --parameters water_level,discharge \
  --from-year 2013 --to-year 2025
```

Use `--fetch-only` before first persistence or `--force` to refresh retained
source JSON. The operator is idempotent for existing station/year Datasets,
retains source checksums and refuses station sets above `--max-stations`. A
missing discharge series is reported without synthesizing values. Different
stations remain separate temporal series and may not be treated as area-wide
water level or volume.

## Governed BWK/Natura 2000 state 2025 for Mol

Build and validate raw WFS evidence without importing:

```bash
docker exec geointel python /app/scripts/provision_mol_bwk_natura2000.py --fetch-only
```

Import after reviewing the manifest:

```bash
docker exec geointel python /app/scripts/provision_mol_bwk_natura2000.py
```

The operator uses the official `BWK:Bwkhab` layer, follows complete WFS
pagination, retains raw JSON/checksums, clips polygon geometry in EPSG:31370
and writes only through the Dataset upload flow. `--force` refetches source
evidence but never silently replaces a conflicting persisted 2025 artifact.

## Definitive agricultural-use parcel history

Prepare a bounded subset of years for source review:

```bash
docker exec geointel python /app/scripts/provision_agricultural_parcel_history.py \
  --years 2008,2019,2025 --fetch-only
```

Import all definitive annual editions for the regional workspace:

```bash
docker exec geointel python /app/scripts/provision_agricultural_parcel_history.py
```

The operator requires the optional GIS runtime already included in the Unraid
image. It paginates GeoIntel API collections within the canonical 200-item
limit, downloads only the fixed official archive allowlist, enforces a 250 MiB
per-archive ceiling, validates one EPSG:31370 polygon GeoPackage and clips
exactly to the persisted scope. Repeat runs reuse matching manifests and
Datasets. `--force` refreshes retained evidence but cannot silently replace a
conflicting persisted annual checksum. Use `--scope mol` for an independent
municipal series.

## Buildings and Addresses Register snapshot

Prepare and audit the current official Mol snapshot without persistence:

```bash
docker exec geointel python /app/scripts/provision_buildings_addresses_register.py --fetch-only
```

Import the audited artifact through the canonical Dataset upload route:

```bash
docker exec geointel python /app/scripts/provision_buildings_addresses_register.py
```

The operator requires a persisted Mol Area and the complete regional GRB
buildings Dataset with valid manifest/partition checksums. It reads only the
official `Gebouw`, `Gebouweenheid` and `Adres` OGC collections, clips in
EPSG:31370 and retains every raw response page under the storage volume.
Advertised `next` links are followed; a full page without `next` is continued
with `startIndex` until a short final page proves completeness.
`--force` refetches evidence; it cannot overwrite a conflicting snapshot for
the same Area/date. Safety limits are configurable with `--page-limit`,
`--max-buildings`, `--max-units` and `--max-addresses`.
The 150,000-building default is a bbox acquisition ceiling: exact Mol clipping
retains far fewer rows, but the rectangular source request also intersects
neighbouring municipalities.

Only aggregate unit/address counts enter the queryable building layer. Review
`address_relations`, `grb_reconciliation`, checksums and limitations in the
manifest before accepting a broader import. Raw address response pages are
operator evidence and must not be published.

## Mol DHMV terrain rasters

Acquire and validate the official DHMV II DTM and DSM for the exact persisted
Mol Area:

```bash
docker exec geointel python /app/scripts/provision_mol_dhmv.py
```

The operator resolves project and Area through the API, derives the bounded
EPSG:4326 request rectangle and calls the canonical DHMV endpoints. The backend
requests the fixed official WCS coverages, extracts multipart GeoTIFF, clips to
the exact Area, validates EPSG:31370/resolution/nodata/valid cells and stores
through DatasetService. It then runs a full-Area terrain metric smoke.

Useful safe overrides:

```bash
docker exec geointel python /app/scripts/provision_mol_dhmv.py --products dtm_1m
docker exec geointel python /app/scripts/provision_mol_dhmv.py --resolution-m 5 --force
```

Do not use DHMV output as water depth or water volume. The command fails when
the API no longer reports those metrics as explicitly unsupported.

## Tower deployment

Push the local branch to Gitea, then rebuild the Unraid/Tower Docker runtime:

```bash
bash scripts/deploy_tower.sh
```

From the Codex Windows workspace, use the PowerShell wrapper:

```powershell
.\scripts\deploy_tower.ps1
```

For the first deployment into an existing non-Git appdata folder, bootstrap the
checkout explicitly:

```bash
DEPLOY_BOOTSTRAP=1 bash scripts/deploy_tower.sh
```

```powershell
.\scripts\deploy_tower.ps1 -Bootstrap
```

Useful overrides:

```bash
REMOTE_HOST=root@192.168.10.150
REMOTE_PATH=/mnt/user/appdata/geointel
REMOTE_REPO=gitea-widefrog:NuklearRabbit/geointel.git
FRONTEND_URL=http://192.168.10.150:1202
```