The audit imports the FastAPI app, compares implemented GET/POST/PATCH/ DELETE routes with active API contract headings and tracks the explicit non-envelope exceptions (/health and export downloads). It fails when a route exists without docs or when docs claim an endpoint that is not implemented.

Verify the browser-facing Docker/LAN runtime:

bash scripts/verify_browser_runtime.sh http://192.168.10.150:1202
bash scripts/verify_gis_runtime.sh http://192.168.10.150:1202

Verify the explicit demo workflow plus export artifact path:

bash scripts/verify_demo_export_workflow.sh http://192.168.10.150:1202

The demo/export smoke is intentionally mutating and idempotent: it seeds the offline fixture demo if needed, verifies the project area GeoJSON, fixture datasets, vector FeatureCollection content, vector feature summary, persisted QA/QC metrics, creates metadata/report/vector GeoJSON exports, lists exports and downloads the JSON/GeoJSON/HTML artifacts through the frontend proxy. The persisted QA/QC result is compared against fixtures/golden/expected_qa_metrics.json so runtime demo precision, recall, F1, mean IoU and false-positive/negative counts cannot drift silently.

Verify the explicit demo raster workflow:

bash scripts/verify_demo_raster_workflow.sh http://192.168.10.150:1202

The raster smoke is intentionally mutating and idempotent enough for local runtime checks: it seeds the offline demo workflow, validates the demo_context_raster.tif fixture dataset, then exercises raster inspect, preview, stats and one small tile/manifest generation through canonical data envelopes. It does not run AI inference or fetch external imagery.

Verify that the browser-facing workbench can populate the default demo start state through the frontend proxy:

bash scripts/verify_workbench_default_state.sh http://192.168.10.150:1202

This smoke is dependency-light and intentionally idempotent: it seeds the offline demo workflow, then verifies that GeoIntel Demo - Building QA exposes the Demo AOI - Geel buildings map geometry, 3/3 ready demo datasets (candidate vector, reference vector and raster fixture) and a persisted QA/QC result through canonical data.items envelopes. Pair it with a Codex/browser screenshot pass when checking visual layout or overflow.

Verify the backing state for the core workbench interactions:

bash scripts/verify_workbench_interactions.sh http://192.168.10.150:1202

This smoke validates the state behind project switching, AOI/map selection, dataset selection, QA refresh and export refresh through the same frontend proxy used by the browser. The frontend also exposes stable data-testid anchors for Codex/browser click checks on those controls.

Verify the browser click handoff from raster tiling into Detection and Segmentation Lab:

bash scripts/verify_ai_handoff_interactions.sh http://192.168.10.150:1202

The AI handoff smoke seeds the explicit offline demo workflow, generates a small raster tile manifest, opens the workbench in Chromium, clicks the raster inspector Use in Detection Lab and Use in Segmentation Lab actions, and verifies that the selected raster dataset plus manifest path are populated in the AI workspace. Playwright/Chromium must be available in the runner environment; GeoIntel does not add Playwright as a frontend dependency by default. The main readiness gate checks this script's syntax only.

Capture visual regression handoff screenshots for the workbench:

bash scripts/capture_workbench_screenshots.sh http://192.168.10.150:1202
CAPTURE_MOBILE=0 bash scripts/capture_workbench_screenshots.sh http://192.168.10.150:1202 /tmp/geointel-screens

The capture script seeds the explicit offline demo workflow, opens each main workspace, captures viewport desktop screenshots and, by default, viewport mobile screenshots. It writes PNG files plus manifest.json under artifacts/screenshots/... or a caller-provided output directory. Playwright/Chromium must be available in the runner environment; GeoIntel does not add Playwright as a frontend dependency by default. The main readiness gate checks script syntax only.

Verify the deterministic QA/QC golden benchmark:

bash scripts/verify_golden_qa_benchmark.sh
python scripts/run_golden_qa_benchmark.py --json

The benchmark uses only explicit local fixtures under fixtures/golden, executes the existing QA/QC matching logic, verifies the expected precision, recall, F1, mean IoU and false-positive/false-negative counts, and checks that QualityCheck plus Metric rows would be persisted. Scenarios are listed in fixtures/golden/golden_qa_benchmarks.json and currently cover partial match, perfect match, no-overlap and MultiPolygon building comparisons. The main readiness gate runs this benchmark so QA metric drift fails before a release.

Verify a configured local YOLO model without running inference:

python scripts/yolo_preflight.py --model-path /absolute/path/to/model.pt --tile-manifest-path /absolute/path/to/manifest.json --check-model-load --json

Against the Docker runtime:

docker compose exec -T backend python scripts/yolo_preflight.py --model-path /absolute/path/to/model.pt --tile-manifest-path /absolute/path/to/manifest.json --check-model-load --json

The model-load smoke is opt-in, requires real optional AI dependencies, refuses --assume-dependencies, loads only the supplied local file and does not download weights or run prediction.

Verify the full configured-YOLO model asset workflow against a running runtime:

bash scripts/verify_model_asset_detection_workflow.sh http://192.168.10.150:1202

This smoke is intentionally mutating and requires a real AI-enabled runtime with at least one mounted local model asset. It seeds the explicit offline demo workflow, generates a small raster tile manifest, selects the active local model asset from GET /api/v1/detection/model-assets, validates read-only YOLO preflight, runs POST /api/v1/detection/run, and verifies the persisted AnalysisRun, Detection list and Detection GeoJSON endpoints. A zero detection count is allowed because the demo raster is a synthetic runtime fixture; the script validates the operational path and provenance, not production model quality. The main readiness gate checks this script's syntax only.

Verify the full operator-provided raster/reference detection and QA path:

REAL_RASTER_PATH=/mnt/user/appdata/geointel/data/orthophoto.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/data/reference-buildings.geojson \
bash scripts/verify_real_data_detection_qa_workflow.sh http://192.168.10.150:1202

The current Tower operator sample is available at:

REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
bash scripts/verify_real_data_detection_qa_workflow.sh http://192.168.10.150:1202

Those files are runtime artifacts generated from Digitaal Vlaanderen's OMWRGBMRVL WMS Ortho layer and GRB OGC API Features GBG building collection for a small Geel AOI. They are intentionally not repository fixtures.

To prepare the documented operator samples reproducibly inside the all-in-one runtime container, run:

docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py

This writes GeoTIFF/GeoJSON pairs and operator_samples_manifest.json under /app/storage/operator-data inside the container, which maps to storage/operator-data in the Tower appdata checkout. The default corpus contains reference AOIs for Geel, Mol, Turnhout, Herentals, Balen, Retie and Westerlo plus background candidates for Postel-bos, Lommel-heide, Kasterlee-bos, Dessel-heide, Ravels-bos, Meerhout-bos, Geel-Bel, Arendonk-heide and Herenthout-bos. Normal reference AOIs still fail when GRB returns no buildings; background candidates are explicitly marked with sample_role and may write an empty reference FeatureCollection for negative-tile training. Generated manifests also classify background samples as pure_empty_negative when GRB returns zero reference buildings or sparse_building_context when GRB returns one or more contextual buildings. The helper fetches only the explicit documented AOIs, records Digitaal Vlaanderen attribution and reuses existing files by default. Use --force only when the local runtime artifacts should be regenerated. GRB building references are fetched through the provider's OGC API rel=next pagination links, so dense AOIs are not silently limited to the first 1000 features. The default page size is 1000; override it with --reference-page-limit or OPERATOR_GRB_PAGE_LIMIT. The safety cap defaults to 100000 features per sample and can be adjusted with --reference-max-features or OPERATOR_GRB_MAX_FEATURES. Generated reference GeoJSON files record reference_pages_fetched, reference_truncated, reference_page_limit, reference_max_features and every fetched source_urls page for auditability.

For model-training candidates, prepare a larger operator-only sample manifest so tile overlap can create meaningful context instead of one tile per source raster:

docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py \
  --output-dir /app/storage/operator-data/operator-samples-1024 \
  --manifest-name operator_samples_manifest.json \
  --width 1024 \
  --height 1024 \
  --half-size-scale 2 \
  --reference-page-limit 1000 \
  --reference-max-features 100000 \
  --force

This keeps the same documented AOI centers but requests larger WMS rasters and a larger GRB reference bbox. Use the generated manifest path for the next YOLO tile export. The default remains 512x512 for quick smoke runs.

The real-data smoke is intentionally mutating and refuses to run without operator-supplied files. Current V1 upload support expects a georeferenced .tif, .tiff or .geotiff raster and a .geojson or .json reference building vector. The script creates a project, uploads the raster as a source dataset, uploads the vector as a reference dataset, validates raster/vector metadata, tiles the raster, selects a mounted local model asset, verifies read-only YOLO preflight, runs configured YOLO detection, runs detection QA against persisted vector_features, and exports the detection run as GeoJSON. It does not seed demo data, enable fixture detections, fetch external data or download model weights. Configured-YOLO model class labels are normalized to lowercase for filtering and persisted detections, while the original model label is retained in detection provenance. Raster tile manifests generated by the workflow include source CRS metadata so persisted detection GeoJSON coordinates can be transformed to WGS84. A zero detection count is accepted operationally only when the selected model genuinely returns no usable detections after class filtering; it must be interpreted as model/data quality evidence rather than as a successful building extraction result.

Run a confidence-threshold calibration sweep against the same real-data path:

REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
CALIBRATION_THRESHOLDS="0.50 0.35 0.25 0.15" \
bash scripts/run_detection_calibration_sweep.sh http://192.168.10.150:1202

The sweep reuses verify_real_data_detection_qa_workflow.sh once per threshold, so every row is backed by persisted Project, Dataset, AnalysisRun, Detection, QualityCheck, Metric and export records. It writes per-threshold logs plus calibration_summary.json under artifacts/detection-calibration/<timestamp> unless CALIBRATION_OUTPUT_DIR is set. This is a calibration/benchmarking tool only: it does not seed demo data, enable fixture detections, fetch external data or download model weights. Per-threshold summaries include persisted detection count, raw candidate count before GeoIntel duplicate suppression, suppressed duplicate count and the configured duplicate IoU threshold.

Run a broader model/tile/threshold quality matrix when multiple local model assets or tile settings need to be compared:

REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_detection_quality_matrix.sh http://192.168.10.150:1202

The quality matrix repeats the same real-data upload, tiling, configured-YOLO, QA/QC and export workflow for every model/tile/threshold row. It writes per-run logs plus quality_matrix_summary.json under artifacts/detection-quality-matrix/<timestamp> unless QUALITY_OUTPUT_DIR is set. The summary ranks best_by_score, best_by_recall and best_by_precision so the next model decision is based on persisted QualityCheck/Metric evidence rather than visual guesses. It does not create provider data, use fixtures or download model weights.

Run the same matrix across every prepared operator sample:

OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202

The multi-sample wrapper writes one per-sample quality_matrix_summary.json plus a combined multi_sample_quality_summary.json with best_overall_by_score, best_overall_by_recall, best_overall_by_precision and best_by_sample rankings. It resolves container-style /app/storage/... manifest paths to repo-relative storage/... paths when run from the Tower host checkout.

Export the same operator samples to a local YOLO detection dataset when the public model candidates are not strong enough for the target imagery:

docker exec -it geointel python3 /app/scripts/export_operator_yolo_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-dataset \
  --val-samples turnhout \
  --force

The exporter writes dataset.yaml, images/train, labels/train, images/val, labels/val and yolo_dataset_summary.json. It uses only the explicit operator sample manifest and GRB building references where source_name=grb and reference_layer_name=buildings. It does not call GeoIntel APIs, create provider data, run inference or train a model.

Run a small local training smoke only in an AI-enabled runtime with an existing local base model file:

docker exec \
  -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-dataset \
  -e YOLO_BASE_MODEL_PATH=/app/models/yolov8n.pt \
  -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-detector.pt \
  -e TRAIN_EPOCHS=8 \
  -e TRAIN_IMGSZ=512 \
  -e TRAIN_BATCH=2 \
  -e TRAIN_WORKERS=0 \
  -e TRAIN_DEVICE=cpu \
  geointel bash /app/scripts/train_operator_yolo_detector.sh

The training wrapper is intentionally outside the product UI. It runs Ultralytics from the existing runtime, copies the best trained artifact to TRAIN_MODEL_OUTPUT_PATH and writes training_summary.json. The summary records SHA256 provenance for dataset.yaml, the available YOLO dataset summary, the local base model and the copied trained model. Afterward, treat the resulting .pt file like any other local model asset: verify preflight, run the real-data matrix and compare persisted QA/QC metrics before activating it as a useful default. Inside the all-in-one image the wrapper prefers /opt/geointel/venv/bin/python when that AI runtime exists. Set PYTHON_BIN only when intentionally overriding the interpreter.

When whole-image training does not improve QA/QC, export a tile-level dataset with overlapping raster windows:

docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-aoi1024-visible025 \
  --tile-size 512 \
  --stride 256 \
  --negative-keep-ratio 1.0 \
  --min-label-visible-ratio 0.25 \
  --val-samples turnhout,retie,westerlo,arendonk_heide \
  --force

The tile exporter clips GRB building bounding boxes into each tile, writes YOLO labels beside each tile image, keeps a deterministic ratio of empty negative tiles, and records yolo_tile_dataset_summary.json with positive_tile_count, negative_tile_count and skipped negative tile counts. --min-label-visible-ratio drops labels where only a small clipped fragment of the original building bbox is visible inside the tile; this reduces noisy tile-edge labels in overlapping-tile datasets. Use 0 for legacy behavior. Use --drop-low-variance-negatives to skip negative tiles whose rendered image has a max-min pixel range at or below --blank-range-threshold. This gate is intended for blank/no-data pure-empty negatives only; positive/labeled tiles are not removed by this filter. The summary records skipped_low_variance_negative_tile_count and skipped tile records with skip_reason=low_visual_variance_negative. For legacy operator manifests that predate explicit background_category, the exporter derives the same categories as the split-background evaluator: background samples with reference_feature_count == 0 become pure_empty_negative, and background samples with one or more reference features become sparse_building_context. It remains operator tooling only: no provider fetch, no API mutation and no automatic model training.

For the current AOI1024 baseline, prefer the stricter clean-label profile before spending another training run:

docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035 \
  --tile-size 512 \
  --stride 256 \
  --negative-keep-ratio 1.0 \
  --min-label-px 12 \
  --min-label-visible-ratio 0.35 \
  --drop-low-variance-negatives \
  --blank-range-threshold 3 \
  --val-samples turnhout,retie,westerlo,arendonk_heide \
  --force

This refreshed cleanpx dataset is the minimum pre-training baseline after the visual contact-sheet pass found six blank-looking arendonk_heide validation negatives in the older export.

The persisted false-negative audit subsequently showed that the cleanpx12 candidate still misses about 79-92% of the comparable reference population and misses every reference building below 25 m2 in the seven-AOI review. Do not train another candidate from the same four positive training AOIs. Refresh the existing AOI1024 sample directory after pulling Sprint 171; existing files are reused and only the four new explicit positive AOIs need to be fetched:

docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py \
  --output-dir /app/storage/operator-data/operator-samples-1024 \
  --width 1024 \
  --height 1024 \
  --half-size-scale 2

The expansion adds Olen, Lille, Oud-Turnhout and Kasterlee center as training samples. Turnhout, Retie, Westerlo and Arendonk-heide remain explicit validation holdouts in generated manifest provenance. The tile exporter defaults to those four holdouts and rejects a manifest-aware split that leaks one into training. Use the lower min-label-px=4 profile first to measure small-building retention; it remains subject to dataset audit and visual contact-sheet review before any training:

docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator-samples-1024/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-aoi1024-expanded-minpx4vis035 \
  --tile-size 512 \
  --stride 256 \
  --negative-keep-ratio 1.0 \
  --min-label-px 4 \
  --min-label-visible-ratio 0.35 \
  --drop-low-variance-negatives \
  --blank-range-threshold 3 \
  --force

Then audit with stricter small-box gates:

docker exec -it geointel python3 /app/scripts/audit_operator_yolo_dataset_quality.py \
  --summary-path /app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035/yolo_tile_dataset_summary.json \
  --output-dir /app/artifacts/operator-yolo-dataset-audit/aoi1024-cleanpx12vis035 \
  --max-small-box-share 0.25 \
  --min-median-box-area 0.001

Audit the generated tile dataset before spending another long training run:

python scripts/audit_operator_yolo_dataset_quality.py \
  --summary-path /mnt/user/appdata/geointel/storage/operator-data/yolo-building-tile-hardneg160r8/yolo_tile_dataset_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/operator-yolo-dataset-audit/hardneg160r8

The audit reads the tile summary and YOLO label files, then writes operator_yolo_dataset_quality_audit.json and operator_yolo_dataset_quality_audit.md. It reports positive/background sample coverage, train/validation split coverage, repeated hard-negative pressure, minimum visible label ratio, missing or invalid label rows, normalized box-area signals and per-sample label diagnostics such as parsed label count, median box area, small-box share and sample-specific quality warnings. Treat needs_attention as a dataset-design warning, not as a runtime failure: the next action is usually more positive AOIs, better validation coverage or more unique hard negatives rather than simply extending epochs.

Render visual label QA contact sheets before spending CPU on another training run:

docker exec -it geointel python3 /app/scripts/render_operator_yolo_label_qa_contact_sheets.py \
  --summary-path /app/storage/operator-data/yolo-building-aoi1024-cleanpx12vis035/yolo_tile_dataset_summary.json \
  --output-dir /app/artifacts/operator-yolo-label-qa/aoi1024-cleanpx12vis035 \
  --max-tiles 32 \
  --columns 4 \
  --thumb-size 256

The renderer writes operator_yolo_label_qa_summary.json, operator_yolo_label_qa_contact_sheet.md and contact_sheet_001.png. It draws existing YOLO labels on existing tile images only; it does not run inference, train a model, fetch providers or create fake detections. Missing image files, missing label files, invalid YOLO rows and low-variance/blank-looking rendered tiles are reported in the JSON/Markdown artifacts. Selection is balanced by sample_slug before taking additional high-density tiles, so one dense urban AOI cannot hide the other source samples from visual review. The report records selected_sample_count and selected_sample_slugs for coverage evidence.

Current Tower audit status:

yolo-building-tile-expanded160: clean baseline; no missing/invalid labels.
yolo-building-tile-hardneg160r4 and yolo-building-tile-hardneg160r8: repeat-heavy hard-negative variants; useful evidence, but add more unique background AOIs before training another hard-negative-balanced candidate.
Regenerate operator_samples_manifest.json after pulling Sprint 147+ so the expanded unique background AOI set is available for the next tile export.
yolo-building-tile-uniquehardneg160: clean expanded-background baseline; 576 tiles, 346 positive, 230 negative, 11,757 labels, 0 invalid labels and 0 repeated background negatives in the first Tower audit.
yolo-building-aoi1024-visible025: larger AOI candidate baseline regenerated after paged GRB references; 144 tiles, 117 positive tiles, 27 negative tiles, 29,170 labels, 0 missing label files, 0 invalid labels and min_label_visible_ratio=0.25. Its audit remains needs_attention because the median normalized box area is still small (0.000762939453125) and the small-box share is high (0.34744600617072335). The previously trained geointel-building-yolov8s-aoi1024visible025e50-pt asset is inactive; do not retrain or activate from this dataset without addressing label quality or explicitly accepting the small-box warning.
yolo-building-aoi1024-cleanpx12vis035: stricter clean-label AOI1024 export; 144 tiles, 114 positive tiles, 30 negative tiles, 14,632 labels, min_label_px=12, min_label_visible_ratio=0.35, 0 invalid labels, median normalized box area 0.001373291015625 and small-box share 0.0. The trained geointel-building-yolov8s-aoi1024cleanpx12vis035e50-pt asset is available but inactive. The split-aware promotion report rejected all tested thresholds: 0.25 and 0.35 passed the pure-empty background gate but had mean F1 below 0.25; 0.15 also failed the pure-empty false-positive gate.
yolo-building-aoi1024-expanded-minpx4vis035: expanded small-building recovery dataset with 20 source AOIs, 171 retained tiles, 45,892 labels, 144 train tiles, 27 validation tiles and 9 low-variance negatives removed. Its configured audit passed with no warnings, median normalized box area 0.000694274766, small-box share 0.3832694151486098, no invalid labels and no missing label files. The balanced visual pass rendered 40 tiles across all 19 source samples that retained at least one tile, with no invalid labels, missing images or low-variance selections. A new candidate may be trained, but remains inactive until positive and split-background promotion gates pass.

After rebuilding the all-in-one image, the operator scripts are available inside the container at /app/scripts/.... Before rebuilding, use the host checkout or temporarily copy scripts into the running container for one-off data prep.

For hard-negative-balanced experiments, repeat only train-split negative tiles from samples marked sample_role=background_candidate:

docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-tile-hardneg160r8 \
  --tile-size 160 \
  --stride 80 \
  --negative-keep-ratio 1.0 \
  --background-negative-repeat 8 \
  --val-samples turnhout,retie,kasterlee_bos \
  --force

The repeat option can also be set with OPERATOR_YOLO_BACKGROUND_NEGATIVE_REPEAT. It does not duplicate validation tiles, positive tiles or normal reference-sample negatives. Repeated background tiles receive deterministic _hnXX filenames and tile metadata records sample_role, repeat_index and is_repeated_background_negative.

Train against the tile dataset by pointing the existing wrapper at the tile output directory:

docker exec \
  -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-tile-expanded160 \
  -e YOLO_BASE_MODEL_PATH=/app/models/yolov8n.pt \
  -e TRAIN_OUTPUT_DIR=/app/storage/training/operator-yolo \
  -e TRAIN_RUN_NAME=geointel-building-yolov8n-expanded160e50 \
  -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-yolov8n-expanded160e50.pt \
  -e TRAIN_EPOCHS=50 \
  -e TRAIN_IMGSZ=256 \
  -e TRAIN_BATCH=8 \
  -e TRAIN_WORKERS=0 \
  -e TRAIN_DEVICE=cpu \
  geointel bash /app/scripts/train_operator_yolo_detector.sh

Benchmark any trained candidate through the same persisted QA/QC matrix before using it operationally:

OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_SAMPLE_SLUGS="geel mol turnhout retie kasterlee_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
QUALITY_TILE_SIZES="640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
MULTI_SAMPLE_OUTPUT_DIR=artifacts/detection-quality-matrix/multi-sample/expanded160e50-live \
bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202

The expanded 50-epoch candidate improved dense Geel/Mol/Turnhout/Retie scores, but the sparse Kasterlee-bos run still showed too many false positives. Treat it as the best current experimental dense-AOI candidate, not as a V1 default.

Run a dedicated hard-negative matrix against documented background candidates before changing model defaults:

OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos dessel_heide ravels_bos meerhout_bos geel_bel arendonk_heide herenthout_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8s-aoi1024bg512r3e50-pt" \
QUALITY_TILE_SIZES="512" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.35 0.15" \
BACKGROUND_SPLIT_OUTPUT_DIR=artifacts/detection-hard-negatives/aoi1024bg512r3e50-split \
bash scripts/run_background_corpus_split_matrix.sh http://192.168.10.150:1202

The split runner executes the strict pure_empty_negative matrix and the review-only sparse_building_context matrix as separate runs, then writes background_corpus_split_summary.json and background_corpus_split_summary.md. Use the pure-empty block for the default-promotion false-positive gate; use sparse-context results as review evidence only.

The lower-level hard-negative matrix can still be run directly:

OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_BACKGROUND_CATEGORIES="pure_empty_negative" \
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos dessel_heide ravels_bos meerhout_bos geel_bel arendonk_heide herenthout_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
QUALITY_TILE_SIZES="640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
HARD_NEGATIVE_OUTPUT_DIR=artifacts/detection-hard-negatives/expanded160e50-live \
bash scripts/run_operator_hard_negative_detection_matrix.sh http://192.168.10.150:1202

The hard-negative matrix uploads only the background raster, generates tiles, runs configured-YOLO detection and counts persisted detections as false_positive_pressure. It does not upload a reference vector and does not run QA/QC, because empty or sparse background AOIs do not have a meaningful precision/recall target. Use OPERATOR_BACKGROUND_CATEGORIES="pure_empty_negative" for the strict default-promotion false-positive gate. Run OPERATOR_BACKGROUND_CATEGORIES="sparse_building_context" separately for contextual review; sparse-context detections should be inspected, not counted as fake precision/recall metrics. In the first live run, geointel-building-yolov8n-expanded160e50-pt was clean on Postel-bos and Lommel-heide at thresholds 0.25 and 0.15, but produced 38 detections on Kasterlee-bos even at 0.25. That blocks it from becoming a V1 default until a hard-negative-balanced candidate improves.

The hard-negative-balanced geointel-building-yolov8n-hardneg160r8e40-pt candidate reduced Kasterlee-bos false-positive pressure to 5/9/25 detections at thresholds 0.25/0.15/0.05 and stayed at 0 detections on Postel-bos and Lommel-heide across all tested thresholds. It also regressed dense-AOI F1 against geointel-building-yolov8n-expanded160e50-pt, so it is useful model quality evidence but not a V1 default.

An external remote-sensing YOLOv8l candidate was also benchmarked as an operator-provided local model asset:

mkdir -p models
curl -L --fail \
  -o models/yolo-remote-sensing-photovoltaic-v8l-detect-1000.pt \
  https://huggingface.co/agademer/yolo-remote-sensing-photovoltaic/resolve/main/yolo-remote-sensing-photovoltaic-v8l-solar-farms-and-cities-v20260331-detect-1000_epochs.pt

GeoIntel exposed the file as yolo-remote-sensing-photovoltaic-v8l-detect-1000-pt with SHA256 242ff4ab889569278f0eb9fcd22eb2c4bf2a52e48d05d89cc7cfa7941165d203, and YOLO preflight loaded it without downloads. On the live dense matrix it missed most buildings and scored far below geointel-building-yolov8n-expanded160e50-pt on Geel, Mol, Turnhout and Retie. On Kasterlee-bos it was clean and precise, but that sparse-AOI behavior is not enough for V1 extraction. Keep it as benchmark evidence only, not as a default model.

Export calibration QA evidence for visual review:

CALIBRATION_SUMMARY_PATH=/mnt/user/appdata/geointel/artifacts/detection-calibration/20260707T002103Z/calibration_summary.json \
bash scripts/export_detection_calibration_evidence.sh http://192.168.10.150:1202

Browser Detection Lab calibration summary exports are supported too:

bash scripts/export_detection_calibration_evidence.sh http://192.168.10.150:1202 ./detection-calibration-summary.json

Run the local browser-summary evidence bundle smoke without touching live application data:

bash scripts/smoke_detection_calibration_evidence_bundle.sh

The smoke creates a temporary Detection Lab-style calibration summary, mocks the canonical persisted QA evidence endpoint responses, runs the real evidence exporter and verifies that calibration_evidence.geojson, calibration_evidence_summary.json and calibration_evidence_review.html are written correctly.

Assemble multiple AOI evidence bundles into one model-review portfolio:

bash scripts/assemble_detection_calibration_evidence_portfolio.sh \
  http://192.168.10.150:1202 \
  ./calibration-evidence-portfolio-manifest.json

Example calibration-evidence-portfolio-manifest.json:

{
  "portfolio_name": "Kempen building model calibration",
  "model_asset_id": "geointel-building-yolov8s-hardneg160r4e50-pt",
  "model_sha256": "optional-model-checksum",
  "notes": "Operator comparison notes.",
  "samples": [
    {
      "sample_slug": "geel",
      "aoi_label": "Geel center",
      "summary_path": "/path/to/detection-calibration-summary.json",
      "operator_notes": "Dense urban validation sample."
    }
  ]
}

The portfolio assembler copies each summary into a deterministic sample folder, runs the existing evidence exporter per AOI and writes calibration_evidence_portfolio.json plus calibration_evidence_portfolio.md. It is evidence packaging only: it does not run inference, create QA checks or mutate application data.

The evidence export reads each persisted quality_check_id, calls the existing QA evidence GeoJSON endpoint, writes calibration_evidence.geojson, calibration_evidence_summary.json and a standalone calibration_evidence_review.html with an SVG overview of matched detections, matched references, false positives and false negatives. Set CALIBRATION_EVIDENCE_MODE=best to export only the best_by_score run.

Build fixed-threshold portfolio inputs when two model runs must be compared at the same confidence threshold across every AOI:

python scripts/build_fixed_threshold_evidence_portfolio_inputs.py \
  --multi-sample-summary artifacts/detection-quality-matrix/multi-sample/<run>/multi_sample_quality_summary.json \
  --threshold 0.35 \
  --model-asset-id geointel-building-yolov8s-aoi1024bg512r3e50-pt \
  --model-sha256 e0980572aac90e7efc514608eb16d7de5bfbf27a4bbec04e7bc1bc8c02f9601f \
  --tile-size 512 \
  --tile-overlap 64 \
  --output-dir artifacts/detection-false-negative-review/active-inputs

The builder selects exactly one persisted QA run per AOI and refuses ambiguous model/tile/threshold matches. Pass its emitted manifest to assemble_detection_calibration_evidence_portfolio.sh with CALIBRATION_EVIDENCE_MODE=all; each filtered summary contains one run.

Compare two or more downloaded evidence portfolios with geodetic WGS84 areas:

python scripts/audit_detection_false_negative_evidence.py \
  --portfolio active=artifacts/detection-false-negative-review/active/calibration_evidence_portfolio.json \
  --portfolio candidate=artifacts/detection-false-negative-review/candidate/calibration_evidence_portfolio.json \
  --output-dir artifacts/detection-false-negative-review/audit

The audit reports false-negative rates and area buckets per AOI/model, plus reference buildings missed by every compared portfolio. Stable source_feature_id values are preferred; a normalized geometry fingerprint is used only when source IDs are absent. Invalid or missing geometry fails the audit instead of being silently skipped. The tools do not run inference, create QA records, mutate model defaults or download data/models.

Docker images install only the GIS runtime by default. To build a local/Tower image with PyTorch/Ultralytics available for the configured-YOLO preflight and runtime path, set:

GEOINTEL_INSTALL_AI=true

For Unraid/all-in-one deployments, place model files under GEOINTEL_MODELS_PATH so they appear in the container under /app/models, then set YOLO_ENABLED=true, YOLO_MODELS_DIR=/app/models and YOLO_MODEL_PATH=/app/models/<model>.pt.

Configure the Unraid/Tower env file from an existing local model without downloading weights or running inference:

python scripts/configure_yolo_model.py \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env

If exactly one supported model file (.pt, .onnx or .engine) is present, apply the env update explicitly:

python scripts/configure_yolo_model.py \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env \
  --apply

The configurator refuses to proceed when no model exists or when multiple model files are present without --model-file. It writes only GEOINTEL_INSTALL_AI=true, YOLO_ENABLED=true, YOLO_MODELS_DIR=/app/models and the mounted YOLO_MODEL_PATH.

When a split-background promotion report recommends a specific candidate, use the guarded activation helper instead of choosing a model path manually. The helper validates the exact candidate_key, promotion status and local model asset before writing anything, and it mutates .env only with --apply:

python scripts/activate_promoted_yolo_candidate.py \
  --promotion-report /mnt/user/appdata/geointel/artifacts/detection-model-promotion/split-aware/aoi1024bg512r3e50-high-threshold-split-20260710T222934Z/detection_model_promotion_report.json \
  --candidate-key 'geointel-building-yolov8s-aoi1024bg512r3e50-pt|512|64|0.35' \
  --models-dir /mnt/user/appdata/geointel/models \
  --env-file /mnt/user/appdata/geointel/.env \
  --json

Add --apply only after reviewing the emitted updates. The helper never downloads weights, loads the model or runs inference; restart or rebuild the container after applying because YOLO_MODEL_PATH is read from the environment.

Tower-local model evaluation status:

geointel-building-yolov8s-hardneg160r4e50.pt is available as an evaluated local runtime artifact after the hard-negative YOLOv8s training pass.
The live catalog id is geointel-building-yolov8s-hardneg160r4e50-pt.
The model SHA256 is 9bf71ad4742048ac77f07060b677bacd9757b8d310497fcada334d543e320d19.
The current safest observed threshold is 0.25, but the model remains an evaluation candidate because one hard-negative forest sample still produced false detections at that threshold.
Do not silently activate this model as a default. Apply it only as an explicit operator choice until the model catalog/threshold workflow is hardened.

Build a model promotion decision report from an existing positive-AOI evidence portfolio and one or more hard-negative/background summaries:

python scripts/build_detection_model_promotion_report.py \
  --positive-portfolio /mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/positive-aoi-expanded-20260708/output/calibration_evidence_portfolio.json \
  --hard-negative-summary /mnt/user/appdata/geointel/artifacts/detection-hard-negatives/hardneg160r8e40-live/hard_negative_matrix_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/detection-model-promotion/expanded-positive-vs-hard-negative-20260708

The report writes detection_model_promotion_report.json and detection_model_promotion_report.md. It groups candidates by model_asset_id, tile size, tile overlap and confidence threshold, then applies explicit gates for positive-AOI sample count, background sample count, mean F1 and maximum background detections per sample. It is evidence/report tooling only: it does not run inference, mutate application data, download models or change the active YOLO configuration.

When the background corpus has been split with run_background_corpus_split_matrix.sh, pass the combined split summary instead of manually wiring both category summaries:

python scripts/build_detection_model_promotion_report.py \
  --positive-portfolio /mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
  --background-split-summary /mnt/user/appdata/geointel/artifacts/detection-hard-negatives/aoi1024bg512r3e50-split/background_corpus_split_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/detection-model-promotion/aoi1024bg512r3e50-split-aware \
  --min-positive-samples 7 \
  --min-background-samples 2 \
  --min-mean-f1 0.25 \
  --max-background-detections-per-sample 0

The promotion report resolves the split summary's pure_empty_negative source summary as the strict default-promotion false-positive gate. The sparse_building_context source remains visible in the JSON/Markdown report as review evidence only and is not counted as a default-promotion gate.

To run both steps after one redeploy, use the workflow wrapper:

PROMOTION_POSITIVE_PORTFOLIO_PATH=/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8s-aoi1024bg512r3e50-pt" \
QUALITY_TILE_SIZES="512" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.35 0.15" \
BACKGROUND_SPLIT_OUTPUT_DIR=artifacts/detection-hard-negatives/aoi1024bg512r3e50-split \
PROMOTION_OUTPUT_DIR=artifacts/detection-model-promotion/aoi1024bg512r3e50-split-aware \
bash scripts/run_split_background_promotion_workflow.sh http://192.168.10.150:1202

The wrapper first calls run_background_corpus_split_matrix.sh, then feeds the generated background_corpus_split_summary.json into the split-aware promotion report. It still uses only existing upload, detection and report paths; it does not fetch providers, fetch model weights or activate a default.

Run the same command with --preflight-only first when checking a fresh redeploy. Preflight validates the positive portfolio path, operator manifest presence, required pure_empty_negative and sparse_building_context background categories, Python/curl availability and the frontend API proxy envelope without starting inference:

For older operator manifests that predate explicit background_category, preflight uses the same fallback as the matrix runner: background samples with reference_feature_count == 0 are treated as pure_empty_negative, and background samples with references are treated as sparse_building_context.

PROMOTION_POSITIVE_PORTFOLIO_PATH=/mnt/user/appdata/geointel/artifacts/detection-quality-matrix/multi-sample/aoi1024bg512r3e50-positive/multi_sample_quality_summary.json \
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
bash scripts/run_split_background_promotion_workflow.sh --preflight-only http://192.168.10.150:1202

If a legacy positive evidence portfolio records model_asset_id at portfolio level but does not include per-run tile size/overlap, pass explicit tile defaults instead of letting the report guess:

python scripts/build_detection_model_promotion_report.py \
  --positive-portfolio /mnt/user/appdata/geointel/artifacts/detection-calibration-portfolio/uniquehardneg160e50-positive/calibration_evidence_portfolio.json \
  --hard-negative-summary /mnt/user/appdata/geointel/artifacts/detection-hard-negatives/uniquehardneg160e50/hard_negative_matrix_summary.json \
  --output-dir /mnt/user/appdata/geointel/artifacts/detection-model-promotion/uniquehardneg160e50-positive-vs-hard-negative \
  --min-positive-samples 7 \
  --min-background-samples 9 \
  --min-mean-f1 0.25 \
  --max-background-detections-per-sample 0 \
  --default-positive-tile-size 640 \
  --default-positive-tile-overlap 64

Clean old offline demo export artifacts without touching uploaded source data:

python scripts/cleanup_demo_artifacts.py
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply

Against the Docker runtime, run the backend-container entrypoint:

docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply

The cleanup script is dry-run by default. It only targets the explicit GeoIntel Demo - Building QA project unless --project-name is provided, keeps the newest exports per matching project, deletes only exports rows/files when --apply is set, and refuses to remove files outside the configured STORAGE_ROOT. --max-delete defaults to 25 and blocks large cleanup runs until the operator raises it after reviewing dry-run output. Repeat --export-type to limit cleanup to specific artifact kinds such as project_report_html or project_metadata_json.

Verify the cleanup path against a running backend without deleting anything:

bash scripts/verify_demo_cleanup_dry_run.sh
CLEANUP_MODE=compose bash scripts/verify_demo_cleanup_dry_run.sh
CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh

The smoke runs the cleanup command without --apply, expects dry_run=true, expects deleted_export_count=0, verifies candidate fields are present and prints the matched/type-filtered/selected counts. Use KEEP_LATEST, MAX_DELETE and EXPORT_TYPE environment variables to adjust the dry-run thresholds without changing the script. The main readiness gate checks this script's syntax; run it explicitly against Docker/PostGIS when validating a live deployment.

Tower deployment

Push the local branch to Gitea, then rebuild the Unraid/Tower Docker runtime:

bash scripts/deploy_tower.sh

From the Codex Windows workspace, use the PowerShell wrapper:

.\scripts\deploy_tower.ps1

For the first deployment into an existing non-Git appdata folder, bootstrap the checkout explicitly:

DEPLOY_BOOTSTRAP=1 bash scripts/deploy_tower.sh

.\scripts\deploy_tower.ps1 -Bootstrap

Useful overrides:

REMOTE_HOST=root@192.168.10.150
REMOTE_PATH=/mnt/user/appdata/geointel
REMOTE_REPO=gitea-widefrog:NuklearRabbit/geointel.git
FRONTEND_URL=http://192.168.10.150:1202