geointel/scripts/README.md

# Scripts

Setup-, import-, demo- en maintenance-scripts voor GeoIntel.

## Runtime verification

Audit the active backend route surface against `docs/API_CONTRACTS.md`:

```bash
python scripts/audit_api_contracts.py
```

The audit imports the FastAPI app, compares implemented `GET`/`POST`/`PATCH`/
`DELETE` routes with active API contract headings and tracks the explicit
non-envelope exceptions (`/health` and export downloads). It fails when a route
exists without docs or when docs claim an endpoint that is not implemented.

Verify the browser-facing Docker/LAN runtime:

```bash
bash scripts/verify_browser_runtime.sh http://192.168.10.150:1202
bash scripts/verify_gis_runtime.sh http://192.168.10.150:1202
```

Verify the explicit demo workflow plus export artifact path:

```bash
bash scripts/verify_demo_export_workflow.sh http://192.168.10.150:1202
```

The demo/export smoke is intentionally mutating and idempotent: it seeds the
offline fixture demo if needed, verifies the project area GeoJSON, fixture
datasets, vector FeatureCollection content, vector feature summary, persisted
QA/QC metrics, creates metadata/report/vector GeoJSON exports, lists exports
and downloads the JSON/GeoJSON/HTML artifacts through the frontend proxy. The
persisted QA/QC result is compared against `fixtures/golden/expected_qa_metrics.json`
so runtime demo precision, recall, F1, mean IoU and false-positive/negative
counts cannot drift silently.

Verify the explicit demo raster workflow:

```bash
bash scripts/verify_demo_raster_workflow.sh http://192.168.10.150:1202
```

The raster smoke is intentionally mutating and idempotent enough for local
runtime checks: it seeds the offline demo workflow, validates the
`demo_context_raster.tif` fixture dataset, then exercises raster inspect,
preview, stats and one small tile/manifest generation through canonical
`data` envelopes. It does not run AI inference or fetch external imagery.

Verify that the browser-facing workbench can populate the default demo start
state through the frontend proxy:

```bash
bash scripts/verify_workbench_default_state.sh http://192.168.10.150:1202
```

This smoke is dependency-light and intentionally idempotent: it seeds the
offline demo workflow, then verifies that `GeoIntel Demo - Building QA` exposes
the `Demo AOI - Geel buildings` map geometry, `3/3 ready` demo datasets
(candidate vector, reference vector and raster fixture) and a persisted QA/QC
result through canonical `data.items` envelopes. Pair it with a Codex/browser
screenshot pass when checking visual layout or overflow.

Verify the backing state for the core workbench interactions:

```bash
bash scripts/verify_workbench_interactions.sh http://192.168.10.150:1202
```

This smoke validates the state behind project switching, AOI/map selection,
dataset selection, QA refresh and export refresh through the same frontend
proxy used by the browser. The frontend also exposes stable `data-testid`
anchors for Codex/browser click checks on those controls.

Verify the browser click handoff from raster tiling into Detection and
Segmentation Lab:

```bash
bash scripts/verify_ai_handoff_interactions.sh http://192.168.10.150:1202
```

The AI handoff smoke seeds the explicit offline demo workflow, generates a
small raster tile manifest, opens the workbench in Chromium, clicks the raster
inspector `Use in Detection Lab` and `Use in Segmentation Lab` actions, and
verifies that the selected raster dataset plus manifest path are populated in
the AI workspace. Playwright/Chromium must be available in the runner
environment; GeoIntel does not add Playwright as a frontend dependency by
default. The main readiness gate checks this script's syntax only.

Capture visual regression handoff screenshots for the workbench:

```bash
bash scripts/capture_workbench_screenshots.sh http://192.168.10.150:1202
CAPTURE_MOBILE=0 bash scripts/capture_workbench_screenshots.sh http://192.168.10.150:1202 /tmp/geointel-screens
```

The capture script seeds the explicit offline demo workflow, opens each main
workspace, captures viewport desktop screenshots and, by default, viewport
mobile screenshots.
It writes PNG files plus `manifest.json` under `artifacts/screenshots/...` or a
caller-provided output directory. Playwright/Chromium must be available in the
runner environment; GeoIntel does not add Playwright as a frontend dependency
by default. The main readiness gate checks script syntax only.

Verify the deterministic QA/QC golden benchmark:

```bash
bash scripts/verify_golden_qa_benchmark.sh
python scripts/run_golden_qa_benchmark.py --json
```

The benchmark uses only explicit local fixtures under `fixtures/golden`,
executes the existing QA/QC matching logic, verifies the expected precision,
recall, F1, mean IoU and false-positive/false-negative counts, and checks that
`QualityCheck` plus `Metric` rows would be persisted. Scenarios are listed in
`fixtures/golden/golden_qa_benchmarks.json` and currently cover partial match,
perfect match, no-overlap and MultiPolygon building comparisons. The main
readiness gate runs this benchmark so QA metric drift fails before a release.

Verify a configured local YOLO model without running inference:

```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/model.pt --tile-manifest-path /absolute/path/to/manifest.json --check-model-load --json
```

Against the Docker runtime:

```bash
docker compose exec -T backend python scripts/yolo_preflight.py --model-path /absolute/path/to/model.pt --tile-manifest-path /absolute/path/to/manifest.json --check-model-load --json
```

The model-load smoke is opt-in, requires real optional AI dependencies, refuses
`--assume-dependencies`, loads only the supplied local file and does not download
weights or run prediction.

Verify the full configured-YOLO model asset workflow against a running runtime:

```bash
bash scripts/verify_model_asset_detection_workflow.sh http://192.168.10.150:1202
```

This smoke is intentionally mutating and requires a real AI-enabled runtime with
at least one mounted local model asset. It seeds the explicit offline demo
workflow, generates a small raster tile manifest, selects the active local model
asset from `GET /api/v1/detection/model-assets`, validates read-only YOLO
preflight, runs `POST /api/v1/detection/run`, and verifies the persisted
AnalysisRun, Detection list and Detection GeoJSON endpoints. A zero detection
count is allowed because the demo raster is a synthetic runtime fixture; the
script validates the operational path and provenance, not production model
quality. The main readiness gate checks this script's syntax only.

Verify the full operator-provided raster/reference detection and QA path:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/data/orthophoto.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/data/reference-buildings.geojson \
bash scripts/verify_real_data_detection_qa_workflow.sh http://192.168.10.150:1202
```

The current Tower operator sample is available at:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
bash scripts/verify_real_data_detection_qa_workflow.sh http://192.168.10.150:1202
```

Those files are runtime artifacts generated from Digitaal Vlaanderen's
OMWRGBMRVL WMS `Ortho` layer and GRB OGC API Features `GBG` building collection
for a small Geel AOI. They are intentionally not repository fixtures.

To prepare the documented operator samples reproducibly inside the all-in-one
runtime container, run:

```bash
docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py
```

This writes GeoTIFF/GeoJSON pairs and `operator_samples_manifest.json` under
`/app/storage/operator-data` inside the container, which maps to
`storage/operator-data` in the Tower appdata checkout. The default corpus
contains reference AOIs for Geel, Mol, Turnhout, Herentals, Balen, Retie and
Westerlo plus background candidates for Postel-bos, Lommel-heide,
Kasterlee-bos, Dessel-heide, Ravels-bos, Meerhout-bos, Geel-Bel,
Arendonk-heide and Herenthout-bos. Normal reference AOIs still fail when GRB
returns no buildings; background candidates are explicitly marked with
`sample_role` and may write an empty reference FeatureCollection for
negative-tile training. Generated manifests also classify background samples as
`pure_empty_negative` when GRB returns zero reference buildings or
`sparse_building_context` when GRB returns one or more contextual buildings.
The helper fetches only the explicit documented AOIs, records Digitaal
Vlaanderen attribution and reuses existing files by default.
Use `--force` only when the local runtime artifacts should be regenerated.
GRB building references are fetched through the provider's OGC API
`rel=next` pagination links, so dense AOIs are not silently limited to the
first 1000 features. The default page size is `1000`; override it with
`--reference-page-limit` or `OPERATOR_GRB_PAGE_LIMIT`. The safety cap defaults
to `100000` features per sample and can be adjusted with
`--reference-max-features` or `OPERATOR_GRB_MAX_FEATURES`. Generated reference
GeoJSON files record `reference_pages_fetched`, `reference_truncated`,
`reference_page_limit`, `reference_max_features` and every fetched
`source_urls` page for auditability.

For model-training candidates, prepare a larger operator-only sample manifest so
tile overlap can create meaningful context instead of one tile per source
raster:

```bash
docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py \
  --output-dir /app/storage/operator-data/operator-samples-1024 \
  --manifest-name operator_samples_manifest.json \
  --width 1024 \
  --height 1024 \
  --half-size-scale 2 \
  --reference-page-limit 1000 \
  --reference-max-features 100000 \
  --force
```

This keeps the same documented AOI centers but requests larger WMS rasters and a
larger GRB reference bbox. Use the generated manifest path for the next YOLO
tile export. The default remains 512x512 for quick smoke runs.

The real-data smoke is intentionally mutating and refuses to run without
operator-supplied files. Current V1 upload support expects a georeferenced
`.tif`, `.tiff` or `.geotiff` raster and a `.geojson` or `.json` reference
building vector. The script creates a project, uploads the raster as a source
dataset, uploads the vector as a `reference` dataset, validates raster/vector
metadata, tiles the raster, selects a mounted local model asset, verifies
read-only YOLO preflight, runs configured YOLO detection, runs detection QA
against persisted `vector_features`, and exports the detection run as GeoJSON.
It does not seed demo data, enable fixture detections, fetch external data or
download model weights. Configured-YOLO model class labels are normalized to
lowercase for filtering and persisted detections, while the original model label
is retained in detection provenance. Raster tile manifests generated by the
workflow include source CRS metadata so persisted detection GeoJSON coordinates
can be transformed to WGS84. A zero detection count is accepted
operationally only when the selected model genuinely returns no usable
detections after class filtering; it must be interpreted as model/data quality
evidence rather than as a successful building extraction result.

Run a confidence-threshold calibration sweep against the same real-data path:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
CALIBRATION_THRESHOLDS="0.50 0.35 0.25 0.15" \
bash scripts/run_detection_calibration_sweep.sh http://192.168.10.150:1202
```

The sweep reuses `verify_real_data_detection_qa_workflow.sh` once per
threshold, so every row is backed by persisted Project, Dataset, AnalysisRun,
Detection, QualityCheck, Metric and export records. It writes per-threshold
logs plus `calibration_summary.json` under
`artifacts/detection-calibration/<timestamp>` unless
`CALIBRATION_OUTPUT_DIR` is set. This is a calibration/benchmarking tool only:
it does not seed demo data, enable fixture detections, fetch external data or
download model weights.
Per-threshold summaries include persisted detection count, raw candidate count
before GeoIntel duplicate suppression, suppressed duplicate count and the
configured duplicate IoU threshold.

Run a broader model/tile/threshold quality matrix when multiple local model
assets or tile settings need to be compared:

```bash
REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_detection_quality_matrix.sh http://192.168.10.150:1202
```

The quality matrix repeats the same real-data upload, tiling, configured-YOLO,
QA/QC and export workflow for every model/tile/threshold row. It writes per-run
logs plus `quality_matrix_summary.json` under
`artifacts/detection-quality-matrix/<timestamp>` unless `QUALITY_OUTPUT_DIR` is
set. The summary ranks `best_by_score`, `best_by_recall` and
`best_by_precision` so the next model decision is based on persisted
`QualityCheck`/`Metric` evidence rather than visual guesses. It does not create
provider data, use fixtures or download model weights.

Run the same matrix across every prepared operator sample:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202
```

The multi-sample wrapper writes one per-sample `quality_matrix_summary.json`
plus a combined `multi_sample_quality_summary.json` with
`best_overall_by_score`, `best_overall_by_recall`,
`best_overall_by_precision` and `best_by_sample` rankings. It resolves
container-style `/app/storage/...` manifest paths to repo-relative
`storage/...` paths when run from the Tower host checkout.

Export the same operator samples to a local YOLO detection dataset when the
public model candidates are not strong enough for the target imagery:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-dataset \
  --val-samples turnhout \
  --force
```

The exporter writes `dataset.yaml`, `images/train`, `labels/train`,
`images/val`, `labels/val` and `yolo_dataset_summary.json`. It uses only the
explicit operator sample manifest and GRB building references where
`source_name=grb` and `reference_layer_name=buildings`. It does not call
GeoIntel APIs, create provider data, run inference or train a model.

Run a small local training smoke only in an AI-enabled runtime with an existing
local base model file:

```bash
docker exec \
  -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-dataset \
  -e YOLO_BASE_MODEL_PATH=/app/models/yolov8n.pt \
  -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-detector.pt \
  -e TRAIN_EPOCHS=8 \
  -e TRAIN_IMGSZ=512 \
  -e TRAIN_BATCH=2 \
  -e TRAIN_WORKERS=0 \
  -e TRAIN_DEVICE=cpu \
  geointel bash /app/scripts/train_operator_yolo_detector.sh
```

The training wrapper is intentionally outside the product UI. It runs
Ultralytics from the existing runtime, copies the best trained artifact to
`TRAIN_MODEL_OUTPUT_PATH` and writes `training_summary.json`. The summary records
SHA256 provenance for `dataset.yaml`, the available YOLO dataset summary, the
local base model and the copied trained model. Afterward, treat
the resulting `.pt` file like any other local model asset: verify preflight,
run the real-data matrix and compare persisted QA/QC metrics before activating
it as a useful default.
Inside the all-in-one image the wrapper prefers
`/opt/geointel/venv/bin/python` when that AI runtime exists. Set `PYTHON_BIN`
only when intentionally overriding the interpreter.

When whole-image training does not improve QA/QC, export a tile-level dataset
with overlapping raster windows:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-aoi1024-visible025 \
  --tile-size 512 \
  --stride 256 \
  --negative-keep-ratio 1.0 \
  --min-label-visible-ratio 0.25 \
  --val-samples turnhout,retie,westerlo,arendonk_heide \
  --force
```

The tile exporter clips GRB building bounding boxes into each tile, writes
YOLO labels beside each tile image, keeps a deterministic ratio of empty
negative tiles, and records `yolo_tile_dataset_summary.json` with
`positive_tile_count`, `negative_tile_count` and skipped negative tile counts.
It separately records configured, retained and empty validation sample slugs;
this keeps a holdout that lost every tile to quality filtering visible without
pretending it contributed evaluation data.
`--min-label-visible-ratio` drops labels where only a small clipped fragment of
the original building bbox is visible inside the tile; this reduces noisy
tile-edge labels in overlapping-tile datasets. Use `0` for legacy behavior.
Use `--drop-low-variance-negatives` to skip negative tiles whose rendered image
has a max-min pixel range at or below `--blank-range-threshold`. This gate is
intended for blank/no-data pure-empty negatives only; positive/labeled tiles are
not removed by this filter. The summary records
`skipped_low_variance_negative_tile_count` and skipped tile records with
`skip_reason=low_visual_variance_negative`.
For legacy operator manifests that predate explicit `background_category`, the
exporter derives the same categories as the split-background evaluator:
background samples with `reference_feature_count == 0` become
`pure_empty_negative`, and background samples with one or more reference
features become `sparse_building_context`.
It remains operator tooling only: no provider fetch, no API mutation and no
automatic model training.

For the current AOI1024 baseline, prefer the stricter clean-label profile before
spending another training run:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035 \
  --tile-size 512 \
  --stride 256 \
  --negative-keep-ratio 1.0 \
  --min-label-px 12 \
  --min-label-visible-ratio 0.35 \
  --drop-low-variance-negatives \
  --blank-range-threshold 3 \
  --val-samples turnhout,retie,westerlo,arendonk_heide \
  --force
```

This refreshed cleanpx dataset is the minimum pre-training baseline after the
visual contact-sheet pass found six blank-looking `arendonk_heide` validation
negatives in the older export.

The persisted false-negative audit subsequently showed that the cleanpx12
candidate still misses about 79-92% of the comparable reference population and
misses every reference building below 25 m2 in the seven-AOI review. Do not
train another candidate from the same four positive training AOIs. Refresh the
existing AOI1024 sample directory after pulling Sprint 171; existing files are
reused and only the four new explicit positive AOIs need to be fetched:

```bash
docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py \
  --output-dir /app/storage/operator-data/operator-samples-1024 \
  --width 1024 \
  --height 1024 \
  --half-size-scale 2
```

The expansion adds Olen, Lille, Oud-Turnhout and Kasterlee center as training
samples. Turnhout, Retie, Westerlo and Arendonk-heide remain explicit validation
holdouts in generated manifest provenance. The tile exporter defaults to those
four holdouts and rejects a manifest-aware split that leaks one into training.
Use the lower `min-label-px=4` profile first to measure small-building retention;
it remains subject to dataset audit and visual contact-sheet review before any
training:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-aoi1024-expanded-minpx4vis035 \
  --tile-size 512 \
  --stride 256 \
  --negative-keep-ratio 1.0 \
  --min-label-px 4 \
  --min-label-visible-ratio 0.35 \
  --drop-low-variance-negatives \
  --blank-range-threshold 3 \
  --force
```

Then audit with stricter small-box gates:

```bash
docker exec -it geointel python3 /app/scripts/audit_operator_yolo_dataset_quality.py \
  --summary-path /app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035/yolo_tile_dataset_summary.json \
  --output-dir /app/artifacts/operator-yolo-dataset-audit/aoi1024-cleanpx12vis035 \
  --max-small-box-share 0.25 \
  --min-median-box-area 0.001
```

Audit the generated tile dataset before spending another long training run:

```bash
python scripts/audit_operator_yolo_dataset_quality.py \
  --summary-path /mnt/user/appdata/geointel/storage/operator-data/yolo-building-tile-hardneg160r8/yolo_tile_dataset_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/operator-yolo-dataset-audit/hardneg160r8
```

The audit reads the tile summary and YOLO label files, then writes
`operator_yolo_dataset_quality_audit.json` and
`operator_yolo_dataset_quality_audit.md`. It reports positive/background sample
coverage, train/validation split coverage, repeated hard-negative pressure,
minimum visible label ratio, missing or invalid label rows, normalized
box-area signals and per-sample label diagnostics such as parsed label count,
median box area, small-box share and sample-specific quality warnings. Treat
`needs_attention` as a dataset-design warning, not as a runtime failure: the
next action is usually more positive AOIs, better validation coverage or more
unique hard negatives rather than simply extending epochs.

Render visual label QA contact sheets before spending CPU on another training
run:

```bash
docker exec -it geointel python3 /app/scripts/render_operator_yolo_label_qa_contact_sheets.py \
  --summary-path /app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035/yolo_tile_dataset_summary.json \
  --output-dir /app/artifacts/operator-yolo-label-qa/aoi1024-cleanpx12vis035 \
  --max-tiles 32 \
  --columns 4 \
  --thumb-size 256
```

The renderer writes `operator_yolo_label_qa_summary.json`,
`operator_yolo_label_qa_contact_sheet.md` and `contact_sheet_001.png`. It draws
existing YOLO labels on existing tile images only; it does not run inference,
train a model, fetch providers or create fake detections. Missing image files,
missing label files, invalid YOLO rows and low-variance/blank-looking rendered
tiles are reported in the JSON/Markdown artifacts. Selection is balanced by
`sample_slug` before taking additional high-density tiles, so one dense urban
AOI cannot hide the other source samples from visual review. The report records
`selected_sample_count` and `selected_sample_slugs` for coverage evidence.

Current Tower audit status:

- `yolo-building-tile-expanded160`: clean baseline; no missing/invalid labels.
- `yolo-building-tile-hardneg160r4` and `yolo-building-tile-hardneg160r8`:
  repeat-heavy hard-negative variants; useful evidence, but add more unique
  background AOIs before training another hard-negative-balanced candidate.
- Regenerate `operator_samples_manifest.json` after pulling Sprint 147+ so the
  expanded unique background AOI set is available for the next tile export.
- `yolo-building-tile-uniquehardneg160`: clean expanded-background baseline;
  576 tiles, 346 positive, 230 negative, 11,757 labels, 0 invalid labels and
  0 repeated background negatives in the first Tower audit.
- `yolo-building-aoi1024-visible025`: larger AOI candidate baseline regenerated
  after paged GRB references; 144 tiles, 117 positive tiles, 27 negative tiles,
  29,170 labels, 0 missing label files, 0 invalid labels and
  `min_label_visible_ratio=0.25`. Its audit remains `needs_attention` because
  the median normalized box area is still small (`0.000762939453125`) and the
  small-box share is high (`0.34744600617072335`). The previously trained
  `geointel-building-yolov8s-aoi1024visible025e50-pt` asset is inactive; do not
  retrain or activate from this dataset without addressing label quality or
  explicitly accepting the small-box warning.
- `yolo-building-aoi1024-cleanpx12vis035`: stricter clean-label AOI1024 export;
  144 tiles, 114 positive tiles, 30 negative tiles, 14,632 labels,
  `min_label_px=12`, `min_label_visible_ratio=0.35`, 0 invalid labels,
  median normalized box area `0.001373291015625` and small-box share `0.0`.
  The trained `geointel-building-yolov8s-aoi1024cleanpx12vis035e50-pt` asset is
  available but inactive. The split-aware promotion report rejected all tested
  thresholds: `0.25` and `0.35` passed the pure-empty background gate but had
  mean F1 below `0.25`; `0.15` also failed the pure-empty false-positive gate.
- `yolo-building-aoi1024-expanded-minpx4vis035`: expanded small-building
  recovery dataset with 20 source AOIs, 171 retained tiles, 45,892 labels,
  144 train tiles, 27 validation tiles and 9 low-variance negatives removed.
  Its configured audit passed with no warnings, median normalized box area
  `0.000694274766`, small-box share `0.3832694151486098`, no invalid labels and
  no missing label files. The balanced visual pass rendered 40 tiles across all
  19 source samples that retained at least one tile, with no invalid labels,
  missing images or low-variance selections. Its promoted model remains the
  higher-precision legacy `0.15` operator profile.
- `yolo-building-aoi1024-smallbld-minpx3vis035`: focused small-building corpus
  exported from an explicit 23-sample subset. Beerse, Rijkevorsel, Hoogstraten
  and Vorselaar extend training; Vosselaar and Grobbendonk are validation-only;
  Turnhout, Retie and Westerlo remain external operation-level holdouts. The
  Tower export retained 198 tiles and 58,820 labels. Its small-object-aware
  audit passed with no invalid/missing labels, and the 48-tile balanced visual
  review contained no missing, invalid or low-variance selections. The trained
  `geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt` candidate passed
  seven positive-AOI and three pure-empty background gates at tile `512`,
  overlap `64`, threshold `0.15` and QA match IoU `0.25`. Mean F1 is `0.5825`
  and all pure-empty samples remain at zero detections. Persisted comparison
  found 1,571 fewer false negatives than the previous balanced model, with a
  lower mean precision and therefore a higher operator review load.

Use `--samples` or `OPERATOR_YOLO_SAMPLES` to make an experimental corpus
membership explicit. Dataset summaries preserve the complete manifest count,
selected sample slugs and excluded sample slugs. Split validation still applies
after filtering, so a manifest-backed holdout cannot be selected as training by
omitting it from `--val-samples`.

After rebuilding the all-in-one image, the operator scripts are available inside
the container at `/app/scripts/...`. Before rebuilding, use the host checkout or
temporarily copy scripts into the running container for one-off data prep.

For hard-negative-balanced experiments, repeat only train-split negative tiles
from samples marked `sample_role=background_candidate`:

```bash
docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-tile-hardneg160r8 \
  --tile-size 160 \
  --stride 80 \
  --negative-keep-ratio 1.0 \
  --background-negative-repeat 8 \
  --val-samples turnhout,retie,kasterlee_bos \
  --force
```

The repeat option can also be set with
`OPERATOR_YOLO_BACKGROUND_NEGATIVE_REPEAT`. It does not duplicate validation
tiles, positive tiles or normal reference-sample negatives. Repeated background
tiles receive deterministic `_hnXX` filenames and tile metadata records
`sample_role`, `repeat_index` and `is_repeated_background_negative`.

Train against the tile dataset by pointing the existing wrapper at the tile
output directory:

```bash
docker exec \
  -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-tile-expanded160 \
  -e YOLO_BASE_MODEL_PATH=/app/models/yolov8n.pt \
  -e TRAIN_OUTPUT_DIR=/app/storage/training/operator-yolo \
  -e TRAIN_RUN_NAME=geointel-building-yolov8n-expanded160e50 \
  -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-yolov8n-expanded160e50.pt \
  -e TRAIN_EPOCHS=50 \
  -e TRAIN_IMGSZ=256 \
  -e TRAIN_BATCH=8 \
  -e TRAIN_WORKERS=0 \
  -e TRAIN_DEVICE=cpu \
  geointel bash /app/scripts/train_operator_yolo_detector.sh
```

Benchmark any trained candidate through the same persisted QA/QC matrix before
using it operationally:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_SAMPLE_SLUGS="geel mol turnhout retie kasterlee_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
QUALITY_TILE_SIZES="640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
MULTI_SAMPLE_OUTPUT_DIR=artifacts/detection-quality-matrix/multi-sample/expanded160e50-live \
bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202
```

The expanded 50-epoch candidate improved dense Geel/Mol/Turnhout/Retie scores,
but the sparse Kasterlee-bos run still showed too many false positives. Treat it
as the best current experimental dense-AOI candidate, not as a V1 default.

Run a dedicated hard-negative matrix against documented background candidates
before changing model defaults:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos dessel_heide ravels_bos meerhout_bos geel_bel arendonk_heide herenthout_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8s-aoi1024bg512r3e50-pt" \
QUALITY_TILE_SIZES="512" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.35 0.15" \
BACKGROUND_SPLIT_OUTPUT_DIR=artifacts/detection-hard-negatives/aoi1024bg512r3e50-split \
bash scripts/run_background_corpus_split_matrix.sh http://192.168.10.150:1202
```

The split runner executes the strict `pure_empty_negative` matrix and the
review-only `sparse_building_context` matrix as separate runs, then writes
`background_corpus_split_summary.json` and
`background_corpus_split_summary.md`. Use the pure-empty block for the
default-promotion false-positive gate; use sparse-context results as review
evidence only.

The lower-level hard-negative matrix can still be run directly:

```bash
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_BACKGROUND_CATEGORIES="pure_empty_negative" \
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos dessel_heide ravels_bos meerhout_bos geel_bel arendonk_heide herenthout_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
QUALITY_TILE_SIZES="640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
HARD_NEGATIVE_OUTPUT_DIR=artifacts/detection-hard-negatives/expanded160e50-live \
bash scripts/run_operator_hard_negative_detection_matrix.sh http://192.168.10.150:1202
```

The hard-negative matrix uploads only the background raster, generates tiles,
runs configured-YOLO detection and counts persisted detections as
`false_positive_pressure`. It does not upload a reference vector and does not
run QA/QC, because empty or sparse background AOIs do not have a meaningful
precision/recall target. Use `OPERATOR_BACKGROUND_CATEGORIES="pure_empty_negative"`
for the strict default-promotion false-positive gate. Run
`OPERATOR_BACKGROUND_CATEGORIES="sparse_building_context"` separately for
contextual review; sparse-context detections should be inspected, not counted
as fake precision/recall metrics. In the first live run,
`geointel-building-yolov8n-expanded160e50-pt`
was clean on Postel-bos and Lommel-heide at thresholds `0.25` and `0.15`, but
produced 38 detections on Kasterlee-bos even at `0.25`. That blocks it from
becoming a V1 default until a hard-negative-balanced candidate improves.

The hard-negative-balanced `geointel-building-yolov8n-hardneg160r8e40-pt`
candidate reduced Kasterlee-bos false-positive pressure to 5/9/25 detections
at thresholds `0.25`/`0.15`/`0.05` and stayed at 0 detections on Postel-bos and
Lommel-heide across all tested thresholds. It also regressed dense-AOI F1
against `geointel-building-yolov8n-expanded160e50-pt`, so it is useful model
quality evidence but not a V1 default.

An external remote-sensing YOLOv8l candidate was also benchmarked as an
operator-provided local model asset:

```bash
mkdir -p models
curl -L --fail \
  -o models/yolo-remote-sensing-photovoltaic-v8l-detect-1000.pt \
  https://huggingface.co/agademer/yolo-remote-sensing-photovoltaic/resolve/main/yolo-remote-sensing-photovoltaic-v8l-solar-farms-and-cities-v20260331-detect-1000_epochs.pt
```

GeoIntel exposed the file as
`yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt` with SHA256
`242ff4ab889569278f0eb9fcd22eb2c4bf2a52e48d05d89cc7cfa7941165d203`, and
YOLO preflight loaded it without downloads. On the live dense matrix it missed
most buildings and scored far below `geointel-building-yolov8n-expanded160e50-pt`
on Geel, Mol, Turnhout and Retie. On Kasterlee-bos it was clean and precise,
but that sparse-AOI behavior is not enough for V1 extraction. Keep it as
benchmark evidence only, not as a default model.

Export calibration QA evidence for visual review:

```bash
CALIBRATION_SUMMARY_PATH=/mnt/user/appdata/geointel/artifacts/detection-calibration/20260707T002103Z/calibration_summary.json \
bash scripts/export_detection_calibration_evidence.sh http://192.168.10.150:1202
```

Browser Detection Lab calibration summary exports are supported too:

```bash
bash scripts/export_detection_calibration_evidence.sh http://192.168.10.150:1202 ./detection-calibration-summary.json
```

Run the local browser-summary evidence bundle smoke without touching live
application data:

```bash
bash scripts/smoke_detection_calibration_evidence_bundle.sh
```

The smoke creates a temporary Detection Lab-style calibration summary, mocks
the canonical persisted QA evidence endpoint responses, runs the real evidence
exporter and verifies that `calibration_evidence.geojson`,
`calibration_evidence_summary.json` and `calibration_evidence_review.html` are
written correctly.

Assemble multiple AOI evidence bundles into one model-review portfolio:

```bash
bash scripts/assemble_detection_calibration_evidence_portfolio.sh \
  http://192.168.10.150:1202 \
  ./calibration-evidence-portfolio-manifest.json
```

Example `calibration-evidence-portfolio-manifest.json`:

```json
{
  "portfolio_name": "Kempen building model calibration",
  "model_asset_id": "geointel-building-yolov8s-hardneg160r4e50-pt",
  "model_sha256": "optional-model-checksum",
  "notes": "Operator comparison notes.",
  "samples": [
    {
      "sample_slug": "geel",
      "aoi_label": "Geel center",
      "summary_path": "/path/to/detection-calibration-summary.json",
      "operator_notes": "Dense urban validation sample."
    }
  ]
}
```

The portfolio assembler copies each summary into a deterministic sample folder,
runs the existing evidence exporter per AOI and writes
`calibration_evidence_portfolio.json` plus
`calibration_evidence_portfolio.md`. It is evidence packaging only: it does not
run inference, create QA checks or mutate application data.

The evidence export reads each persisted `quality_check_id`, calls the existing
QA evidence GeoJSON endpoint, writes `calibration_evidence.geojson`,
`calibration_evidence_summary.json` and a standalone
`calibration_evidence_review.html` with an SVG overview of matched detections,
matched references, false positives and false negatives. Set
`CALIBRATION_EVIDENCE_MODE=best` to export only the `best_by_score` run.

Build fixed-threshold portfolio inputs when two model runs must be compared at
the same confidence threshold across every AOI:

```bash
python scripts/build_fixed_threshold_evidence_portfolio_inputs.py \
  --multi-sample-summary artifacts/detection-quality-matrix/multi-sample/<run>/multi_sample_quality_summary.json \
  --threshold 0.35 \
  --model-asset-id geointel-building-yolov8s-aoi1024bg512r3e50-pt \
  --model-sha256 e0980572aac90e7efc514608eb16d7de5bfbf27a4bbec04e7bc1bc8c02f9601f \
  --tile-size 512 \
  --tile-overlap 64 \
  --output-dir artifacts/detection-false-negative-review/active-inputs
```

The builder selects exactly one persisted QA run per AOI and refuses ambiguous
model/tile/threshold matches. Pass its emitted manifest to
`assemble_detection_calibration_evidence_portfolio.sh` with
`CALIBRATION_EVIDENCE_MODE=all`; each filtered summary contains one run.

Compare two or more downloaded evidence portfolios with geodetic WGS84 areas:

```bash
python scripts/audit_detection_false_negative_evidence.py \
  --portfolio active=artifacts/detection-false-negative-review/active/calibration_evidence_portfolio.json \
  --portfolio candidate=artifacts/detection-false-negative-review/candidate/calibration_evidence_portfolio.json \
  --output-dir artifacts/detection-false-negative-review/audit
```

The audit reports false-negative rates and area buckets per AOI/model, plus
reference buildings missed by every compared portfolio. It writes the combined
`persistent_false_negatives.geojson`, records geodetic persistent-miss area and
adds persistent area buckets so operators can inspect the shared misses on a
map instead of relying only on counts. Stable
`source_feature_id` values are preferred; a normalized geometry fingerprint is
used only when source IDs are absent. Invalid or missing geometry fails the
audit instead of being silently skipped. The tools do not run inference,
create QA records, mutate model defaults or download data/models.

Docker images install only the GIS runtime by default. To build a local/Tower
image with PyTorch/Ultralytics available for the configured-YOLO preflight and
runtime path, set:

```bash
GEOINTEL_INSTALL_AI=true
```

For Unraid/all-in-one deployments, place model files under
`GEOINTEL_MODELS_PATH` so they appear in the container under `/app/models`, then
set `YOLO_ENABLED=true`, `YOLO_MODELS_DIR=/app/models` and
`YOLO_MODEL_PATH=/app/models/<model>.pt`.

Configure the Unraid/Tower env file from an existing local model without
downloading weights or running inference:

```bash
python scripts/configure_yolo_model.py \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env
```

If exactly one supported model file (`.pt`, `.onnx` or `.engine`) is present,
apply the env update explicitly:

```bash
python scripts/configure_yolo_model.py \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env \
  --apply
```

The configurator refuses to proceed when no model exists or when multiple model
files are present without `--model-file`. It writes only
`GEOINTEL_INSTALL_AI=true`, `YOLO_ENABLED=true`, `YOLO_MODELS_DIR=/app/models`
and the mounted `YOLO_MODEL_PATH`.

When a split-background promotion report recommends a specific candidate, use
the guarded activation helper instead of choosing a model path manually. The
helper validates the exact `candidate_key`, promotion status and local model
asset before writing anything, and it mutates `.env` only with `--apply`:

```bash
python scripts/activate_promoted_yolo_candidate.py \
  --promotion-report /mnt/user/appdata/geointel/artifacts/detection-model-promotion/split-aware/aoi1024bg512r3e50-high-threshold-split-20260710T222934Z/detection_model_promotion_report.json \
  --candidate-key 'geointel-building-yolov8s-aoi1024bg512r3e50-pt|512|64|0.35' \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env \
  --json
```

Add `--apply` only after reviewing the emitted updates. The helper never
downloads weights, loads the model or runs inference; restart or rebuild the
container after applying because `YOLO_MODEL_PATH` is read from the environment.

Tower-local model evaluation status:

- `geointel-building-yolov8s-hardneg160r4e50.pt` is available as an evaluated
  local runtime artifact after the hard-negative YOLOv8s training pass.
- The live catalog id is `geointel-building-yolov8s-hardneg160r4e50-pt`.
- The model SHA256 is
  `9bf71ad4742048ac77f07060b677bacd9757b8d310497fcada334d543e320d19`.
- The current safest observed threshold is `0.25`, but the model remains an
  evaluation candidate because one hard-negative forest sample still produced
  false detections at that threshold.
- Do not silently activate this model as a default. Apply it only as an explicit
  operator choice until the model catalog/threshold workflow is hardened.

Build a model promotion decision report from an existing positive-AOI evidence
portfolio and one or more hard-negative/background summaries:

```bash
python scripts/build_detection_model_promotion_report.py \
  --positive-portfolio /mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/positive-aoi-expanded-20260708/output/calibration_evidence_portfolio.json \
  --hard-negative-summary /mnt/user/appdata/geointel/artifacts/detection-hard-negatives/hardneg160r8e40-live/hard_negative_matrix_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/detection-model-promotion/expanded-positive-vs-hard-negative-20260708
```

The report writes `detection_model_promotion_report.json` and
`detection_model_promotion_report.md`. It groups candidates by
`model_asset_id`, tile size, tile overlap and confidence threshold, then applies
explicit gates for positive-AOI sample count, background sample count, mean F1
and maximum background detections per sample. It is evidence/report tooling
only: it does not run inference, mutate application data, download models or
change the active YOLO configuration.

When the background corpus has been split with
`run_background_corpus_split_matrix.sh`, pass the combined split summary instead
of manually wiring both category summaries:

```bash
python scripts/build_detection_model_promotion_report.py \
  --positive-portfolio /mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
  --background-split-summary /mnt/user/appdata/geointel/artifacts/detection-hard-negatives/aoi1024bg512r3e50-split/background_corpus_split_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/detection-model-promotion/aoi1024bg512r3e50-split-aware \
  --min-positive-samples 7 \
  --min-background-samples 2 \
  --min-mean-f1 0.25 \
  --max-background-detections-per-sample 0
```

The promotion report resolves the split summary's `pure_empty_negative` source
summary as the strict default-promotion false-positive gate. The
`sparse_building_context` source remains visible in the JSON/Markdown report as
review evidence only and is not counted as a default-promotion gate.

To run both steps after one redeploy, use the workflow wrapper:

```bash
PROMOTION_POSITIVE_PORTFOLIO_PATH=/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8s-aoi1024bg512r3e50-pt" \
QUALITY_TILE_SIZES="512" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.35 0.15" \
BACKGROUND_SPLIT_OUTPUT_DIR=artifacts/detection-hard-negatives/aoi1024bg512r3e50-split \
PROMOTION_OUTPUT_DIR=artifacts/detection-model-promotion/aoi1024bg512r3e50-split-aware \
bash scripts/run_split_background_promotion_workflow.sh http://192.168.10.150:1202
```

The wrapper first calls `run_background_corpus_split_matrix.sh`, then feeds the
generated `background_corpus_split_summary.json` into the split-aware promotion
report. It still uses only existing upload, detection and report paths; it does
not fetch providers, fetch model weights or activate a default.

Run the same command with `--preflight-only` first when checking a fresh
redeploy. Preflight validates the positive portfolio path, operator manifest
presence, required `pure_empty_negative` and `sparse_building_context`
background categories, Python/curl availability and the frontend API proxy
envelope without starting inference:

For older operator manifests that predate explicit `background_category`,
preflight uses the same fallback as the matrix runner: background samples with
`reference_feature_count == 0` are treated as `pure_empty_negative`, and
background samples with references are treated as `sparse_building_context`.

```bash
PROMOTION_POSITIVE_PORTFOLIO_PATH=/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
bash scripts/run_split_background_promotion_workflow.sh --preflight-only http://192.168.10.150:1202
```

If a legacy positive evidence portfolio records `model_asset_id` at portfolio
level but does not include per-run tile size/overlap, pass explicit tile
defaults instead of letting the report guess:

```bash
python scripts/build_detection_model_promotion_report.py \
  --positive-portfolio /mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/uniquehardneg160e50-positive/calibration_evidence_portfolio.json \
  --hard-negative-summary /mnt/user/appdata/geointel/artifacts/detection-hard-negatives/uniquehardneg160e50/hard_negative_matrix_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/detection-model-promotion/uniquehardneg160e50-positive-vs-hard-negative \
  --min-positive-samples 7 \
  --min-background-samples 9 \
  --min-mean-f1 0.25 \
  --max-background-detections-per-sample 0 \
  --default-positive-tile-size 640 \
  --default-positive-tile-overlap 64
```

Clean old offline demo export artifacts without touching uploaded source data:

```bash
python scripts/cleanup_demo_artifacts.py
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
```

Against the Docker runtime, run the backend-container entrypoint:

```bash
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
```

The cleanup script is dry-run by default. It only targets the explicit
`GeoIntel Demo - Building QA` project unless `--project-name` is provided, keeps
the newest exports per matching project, deletes only `exports` rows/files when
`--apply` is set, and refuses to remove files outside the configured
`STORAGE_ROOT`. `--max-delete` defaults to 25 and blocks large cleanup runs until
the operator raises it after reviewing dry-run output. Repeat `--export-type` to
limit cleanup to specific artifact kinds such as `project_report_html` or
`project_metadata_json`.

Verify the cleanup path against a running backend without deleting anything:

```bash
bash scripts/verify_demo_cleanup_dry_run.sh
CLEANUP_MODE=compose bash scripts/verify_demo_cleanup_dry_run.sh
CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh
```

The smoke runs the cleanup command without `--apply`, expects `dry_run=true`,
expects `deleted_export_count=0`, verifies candidate fields are present and
prints the matched/type-filtered/selected counts. Use `KEEP_LATEST`,
`MAX_DELETE` and `EXPORT_TYPE` environment variables to adjust the dry-run
thresholds without changing the script. The main readiness gate checks this
script's syntax; run it explicitly against Docker/PostGIS when validating a
live deployment.

## Tower deployment

Push the local branch to Gitea, then rebuild the Unraid/Tower Docker runtime:

```bash
bash scripts/deploy_tower.sh
```

From the Codex Windows workspace, use the PowerShell wrapper:

```powershell
.\scripts\deploy_tower.ps1
```

For the first deployment into an existing non-Git appdata folder, bootstrap the
checkout explicitly:

```bash
DEPLOY_BOOTSTRAP=1 bash scripts/deploy_tower.sh
```

```powershell
.\scripts\deploy_tower.ps1 -Bootstrap
```

Useful overrides:

```bash
REMOTE_HOST=root@192.168.10.150
REMOTE_PATH=/mnt/user/appdata/geointel
REMOTE_REPO=gitea-widefrog:NuklearRabbit/geointel.git
FRONTEND_URL=http://192.168.10.150:1202
```