geointel/docs/AI_PIPELINES.md

AI Pipelines

1. Object Detection Pipeline

Raster dataset
↓
Clip to analysis area
↓
Tile raster
↓
Normalize tiles
↓
Run YOLO/PyTorch inference
↓
Filter by confidence
↓
Convert pixel boxes to geospatial polygons
↓
Merge overlapping detections
↓
Store in PostGIS
↓
Expose as GeoJSON layer
↓
Run QA/QC if reference data exists

Sprint 8 foundation status

Sprint 8 implements the detection persistence and execution boundary only:

• detections are first-class PostGIS records linked to project, dataset, job and analysis run.
• analysis_runs remain separate from jobs and store model metadata, parameters, result summaries and lifecycle status.
• yolo-placeholder reports not_configured; no YOLO/PyTorch model is downloaded or executed.
• manual-fixture-detector is test/demo-only and persists detections only when fixture_mode=true and fixture detections are explicitly supplied.
• Normal application behavior must not create fake detections.

Sprint 8B configured YOLO status

Sprint 8B adds an import-safe real YOLO adapter path:

• ultralytics and torch are optional backend extras, not default runtime dependencies.
• yolo-configured reports not_configured until YOLO_ENABLED=true, YOLO_MODEL_PATH points to an existing local model file and optional AI dependencies are installed.
• GeoIntel never downloads model weights automatically.
• Real YOLO inference uses an existing raster tile manifest generated by the raster tile operation.
• YOLO raster tiles are normalized to RGB for inference when the tile artifact is not already a 3-band RGB image; the persisted georeferencing still comes from the tile manifest.
• YOLO pixel boxes are converted to EPSG:4326 detection polygons from tile transform or tile bounds metadata.
• YOLO class labels are normalized to lowercase for persisted detection records and filtering, while the original model label remains available in detection provenance.
• Detection runs remain synchronous behind the existing job and analysis-run persistence boundary for Sprint 8B.

Sprint 13 YOLO operational preflight

Sprint 13 adds a local preflight command for configured YOLO operation:

python scripts/yolo_preflight.py --model-path /absolute/path/to/model.pt --tile-manifest-path /absolute/path/to/manifest.json

For machines without optional AI dependencies, path and manifest checks can be exercised without pretending inference is available:

python scripts/yolo_preflight.py --model-path /absolute/path/to/model.pt --tile-manifest-path /absolute/path/to/manifest.json --assume-dependencies --json

The preflight checks:

• YOLO_ENABLED / explicit enabled state;
• optional dependency availability unless --assume-dependencies is used;
• local model file existence;
• tile manifest JSON validity;
• tile count against YOLO_MAX_TILES;
• referenced tile file existence.

JSON output also reports runtime diagnostics: whether dependencies were assumed, the configured model directory, YOLO_CONFIG_DIR, installed torch and ultralytics versions, and CUDA availability when dependency checks pass.

The preflight does not load the model, does not import Ultralytics unless dependency discovery requires package metadata, does not run inference and never downloads model weights.

Sprint 25 adds an explicit local model compatibility smoke:

python scripts/yolo_preflight.py --model-path /absolute/path/to/model.pt --tile-manifest-path /absolute/path/to/manifest.json --check-model-load --json

--check-model-load requires real optional AI dependencies and an existing local model file. It loads that local file through the configured adapter to verify Ultralytics/PyTorch compatibility, but it still does not run tile prediction and does not download weights. It cannot be combined with --assume-dependencies because that would turn the smoke into a false positive.

Docker and Unraid runtime support remains opt-in. Set GEOINTEL_INSTALL_AI=true at build time to install the backend .[gis,ai] extra into the container. Leave it unset or false for the default GIS-only image. Runtime model files should be mounted into the container, for example /app/models/local-model.pt, and enabled with YOLO_ENABLED=true plus YOLO_MODEL_PATH=/app/models/local-model.pt. GeoIntel never downloads weights automatically.

Environment variables:

• GEOINTEL_INSTALL_AI
• YOLO_ENABLED
• YOLO_MODELS_DIR
• YOLO_MODEL_PATH
• YOLO_MODEL_ID
• YOLO_MODEL_DISPLAY_NAME
• YOLO_MODEL_VERSION
• YOLO_DEVICE
• YOLO_IMAGE_SIZE
• YOLO_MAX_TILES
• YOLO_BATCH_SIZE

Local model asset catalog

GeoIntel can list local runtime model files mounted into the backend model directory through GET /api/v1/detection/model-assets. The catalog is filesystem-backed and read-only: it reports existing .pt, .onnx and .engine files, size, checksum and whether the file matches YOLO_MODEL_PATH.

Detection runs still use model_id="yolo-configured" for the configured YOLO execution path. A selected model_asset_id can be supplied to use one specific cataloged file for that run. The backend resolves the ID to a local path and persists the selected asset metadata in Job/AnalysisRun parameters. GeoIntel does not download weights or accept arbitrary model paths from the browser.

Operational runtime validation can be run against Docker/Tower with:

bash scripts/verify_model_asset_detection_workflow.sh http://192.168.10.150:1202

The smoke seeds the explicit offline demo raster, creates a tile manifest, selects a local model asset, checks read-only preflight, runs the existing configured-YOLO detection endpoint and verifies persisted AnalysisRun, Detection list and Detection GeoJSON outputs. It intentionally does not inject detector fixtures or download weights. A zero detection count is acceptable on the synthetic demo raster; production usefulness still requires validation on real georeferenced orthophotos and reference vectors.

Real-data detection and QA validation

The real operational validation path uses operator-provided files rather than demo fixtures:

REAL_RASTER_PATH=/mnt/user/appdata/geointel/data/orthophoto.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/data/reference-buildings.geojson \
bash scripts/verify_real_data_detection_qa_workflow.sh http://192.168.10.150:1202

The script verifies the full persisted chain:

• source raster upload with CRS and bounds metadata;
• reference building vector upload as dataset_role=reference;
• raster inspect and tile manifest generation;
• local model asset selection and read-only YOLO preflight;
• configured-YOLO detection run through Job, AnalysisRun and Detection rows;
• detection GeoJSON generated from persisted geometry;
• detection QA against persisted reference vector_features with persisted QualityCheck and Metric rows;
• detection run GeoJSON export.

It refuses to run without a real GeoTIFF-style raster and GeoJSON/JSON reference vector. It does not seed demo data, use fixture_mode, fetch live providers or download model weights. A zero detection count is valid as runtime evidence only when the selected model genuinely returns no usable detections after canonical class filtering; it does not prove the model is useful for the target imagery.

Documented operator samples can be prepared inside the all-in-one runtime container:

docker exec -it geointel python3 /app/scripts/prepare_operator_real_data_samples.py

The helper fetches explicit Digitaal Vlaanderen orthophoto/GRB GBG sample pairs for the documented AOIs only and writes operator_samples_manifest.json. The default corpus includes dense reference AOIs for Geel, Mol, Turnhout, Herentals, Balen, Retie and Westerlo plus explicitly marked background candidates for Postel-bos, Lommel-heide and Kasterlee-bos. Background candidates may persist empty GRB FeatureCollections for negative-tile training; normal reference AOIs still fail on empty GRB responses. The application itself still does not perform live provider fetching.

For confidence-threshold calibration, use the sweep wrapper:

REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
CALIBRATION_THRESHOLDS="0.50 0.35 0.25 0.15" \
bash scripts/run_detection_calibration_sweep.sh http://192.168.10.150:1202

The sweep runs the real-data workflow once per threshold and then reads the persisted project quality-check list to build calibration_summary.json. Results are honest QA/QC evidence from persisted detections and persisted reference vector_features; no demo detections, live provider fetches or model downloads are introduced by the calibration tool.

For model/tile/threshold selection, use the quality matrix wrapper:

REAL_RASTER_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_orthophoto_wms_512.tif \
REAL_REFERENCE_VECTOR_PATH=/mnt/user/appdata/geointel/storage/operator-data/geel_grb_gbg_buildings.geojson \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_detection_quality_matrix.sh http://192.168.10.150:1202

The matrix repeats the same persisted real-data workflow for every combination and writes quality_matrix_summary.json with detection count, QA score, precision, recall, F1, mean IoU and false-positive/false-negative counts. The rankings best_by_score, best_by_recall and best_by_precision are operator decision aids only; GeoIntel still does not download models, seed fixture detections or treat AI detections as ground truth without QA/QC. The same candidate should also pass the background false-positive matrix before it is considered as a default:

OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos" \
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
QUALITY_TILE_SIZES="640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
bash scripts/run_operator_hard_negative_detection_matrix.sh http://192.168.10.150:1202

The hard-negative matrix uploads only background rasters and counts detections as false-positive pressure. It does not run QA/QC or invent reference metrics for empty/sparse background AOIs. The first expanded local model improved dense AOI F1, but Kasterlee-bos false positives block default promotion.

To compare the same model/tile/threshold grid across all prepared operator samples, use:

OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
QUALITY_MODEL_ASSET_IDS="yolov8n-building-segmentation-pt yolov8n-pt" \
QUALITY_TILE_SIZES="512 640" \
QUALITY_TILE_OVERLAPS="64" \
QUALITY_THRESHOLDS="0.50 0.15" \
bash scripts/run_multi_sample_detection_quality_matrix.sh http://192.168.10.150:1202

The multi-sample summary exposes best_overall_by_score, best_overall_by_recall, best_overall_by_precision and best_by_sample so model-quality decisions are based on repeated persisted QA/QC evidence rather than one AOI.

When repeated public model benchmarks remain too weak, the operator can convert the prepared real-data samples into a local YOLO training dataset:

docker exec -it geointel python3 /app/scripts/export_operator_yolo_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-dataset \
  --val-samples turnhout \
  --force

The exporter creates a standard YOLO detection layout with dataset.yaml, images/train, labels/train, images/val and labels/val. It converts GRB building reference geometries to pixel-space bounding boxes for the matching orthophoto sample and records yolo_dataset_summary.json.

A minimal local training smoke can then be run explicitly in an AI-enabled runtime:

docker exec \
  -e OPERATOR_YOLO_DATASET_DIR=/app/storage/operator-data/yolo-building-dataset \
  -e YOLO_BASE_MODEL_PATH=/app/models/yolov8n.pt \
  -e TRAIN_MODEL_OUTPUT_PATH=/app/models/geointel-building-detector.pt \
  -e TRAIN_EPOCHS=8 \
  -e TRAIN_IMGSZ=512 \
  -e TRAIN_BATCH=2 \
  -e TRAIN_WORKERS=0 \
  -e TRAIN_DEVICE=cpu \
  -e PYTHON_BIN=python3 \
  geointel bash /app/scripts/train_operator_yolo_detector.sh

This remains operator tooling only. GeoIntel does not expose Training Studio in V1, does not generate labels from predictions and does not treat the trained artifact as useful until it passes the same real-data Detection + QA matrix.

If the whole-image dataset underfits or produces unusable detections, export overlapping tile-level samples:

docker exec -it geointel python3 /app/scripts/export_operator_yolo_tile_dataset.py \
  --manifest-path /app/storage/operator-data/operator_samples_manifest.json \
  --output-dir /app/storage/operator-data/yolo-building-tile-dataset \
  --tile-size 192 \
  --stride 96 \
  --negative-keep-ratio 0.5 \
  --val-samples turnhout \
  --force

The tile exporter clips reference building boxes into tile-local YOLO labels and records the positive/negative tile counts. This gives the training smoke more image samples while preserving the same explicit operator-data and QA/QC validation boundary.

For visual error inspection, export the persisted QA evidence from a calibration summary:

CALIBRATION_SUMMARY_PATH=/mnt/user/appdata/geointel/artifacts/detection-calibration/20260707T002103Z/calibration_summary.json \
bash scripts/export_detection_calibration_evidence.sh http://192.168.10.150:1202

The evidence bundle calls /api/v1/projects/{project_id}/quality-checks/{quality_check_id}/evidence/geojson for the persisted quality checks and writes combined GeoJSON plus an HTML/SVG review artifact. This is an inspection aid only; it does not rerun inference or alter stored detections.

Sprint 8C detection visualization and QA status

Sprint 8C makes persisted detections reviewable:

• Detection runs can be listed and selected.
• Persisted detections can be listed and filtered by run, dataset, class and minimum confidence.
• Persisted detection geometries can be returned as GeoJSON FeatureCollections for MapLibre display.
• Detection QA compares candidate detection geometries against persisted reference vector_features.
• QA results reuse quality_checks and metrics; no parallel QA persistence system is introduced.
• Segmentation remains out of scope for Sprint 8C.

2. Tile Metadata

Elke tile moet opslaan:

• tile path
• parent raster id
• pixel window
• geospatial bounds
• transform
• CRS, and the manifest must also carry source CRS metadata
• tile size
• overlap

Zonder tile metadata kunnen modeloutputs niet correct teruggeprojecteerd worden.

3. Detection Output Contract

Elke detectie bevat:

• class_name
• confidence
• bbox pixel coords
• source tile
• geospatial polygon
• model id/version
• analysis run id

4. Segmentation Pipeline

Raster dataset
↓
Clip/tile
↓
Run segmentation model
↓
Generate mask
↓
Georeference mask
↓
Polygonize mask
↓
Simplify/clean geometries
↓
Store polygons + mask path
↓
Expose as map layer

Sprint 9 segmentation foundation status

Sprint 9 implements the segmentation persistence and review boundary only:

• segmentations are first-class PostGIS records linked to project, dataset, job and analysis run.
• PostGIS MultiPolygon geometry in EPSG:4326 is authoritative for map display, QA and GeoJSON output.
• Mask paths are persisted as artifact/provenance references, not authoritative feature state.
• segmentation-placeholder, yolo-seg-configured and sam-configured report not_configured.
• fixture-segmenter is test/demo-only and persists segmentations only when fixture_mode=true and fixture segmentations are explicitly supplied.
• Segmentation QA compares persisted segmentation geometries against persisted reference vector_features.
• QA results reuse quality_checks and metrics; no parallel QA system is introduced.
• GeoIntel does not install SAM, run YOLO-seg, download model weights or fake production segmentations in Sprint 9.

5. Change Detection Pipeline

Fase 1: vector/detection based.

Run A detections
+
Run B detections
↓
Spatial matching
↓
added / removed / changed
↓
Change polygons
↓
Metrics

Fase 2: raster index based.

Raster A index
+
Raster B index
↓
Difference raster
↓
Threshold
↓
Polygonize changed zones

Fase 3: segmentation based.

Mask A
+
Mask B
↓
Class difference
↓
Change polygons

6. Model Strategy

V1:

• gebruik een bestaande YOLO-integratie met configureerbaar modelpad
• demo-model mag lokaal worden geplaatst in models/
• code moet ook zonder model kunnen starten, maar detection job moet dan duidelijke fout geven

V2:

• SAM/YOLO segmentation

V3:

• annotation export
• finetuning

7. Reproduceerbaarheid

Elke analysis run moet bewaren:

• model id
• model version
• parameters
• confidence threshold
• tile size
• overlap
• input dataset id
• code path/version indien mogelijk