geointel/backend/README.md

# GeoIntel Backend (Sprint 3 foundation layer)

FastAPI backend for GeoIntel Kempen Foundation Sprints.

## Scope implemented
- Project CRUD
- Area CRUD with PostGIS geometry
- Vector and raster dataset upload/registration
- Deterministic local storage metadata capture
- PostGIS migration and database foundation
- Job foundation for async-ready GIS operations

## Sprint 2 additions
- Dataset typing and lifecycle support:
  - `uploaded`
  - `validating`
  - `ready`
  - `failed`
- Vector metadata extraction:
  - feature count
  - geometry type summary
  - bounds
  - approximate area
  - CRS and CRS assumption
- Raster metadata endpoint:
  - returns raster profile when `rasterio` is available
  - returns clear `RASTER_PROCESSING_UNAVAILABLE` error when dependency is missing
- Deterministic storage metadata capture:
  - original filename
  - stored filename
  - MIME/content type
  - size bytes
  - checksum SHA-256

## Sprint 3 additions
- Lightweight job architecture:
  - `jobs` table and migrations
  - job create/list/read/status API
  - synchronous execution behind job abstraction
- Vector operations foundation:
  - inspect
  - bbox
  - stats
  - clip by area
  - buffer
  - intersect
  - invalid geometry rejection with typed errors
- Raster operation foundation:
  - inspect
  - metadata
  - preview readiness
  - clip by area (dependency-aware with unavailable fallback)
  - tile generation with manifest output
  - real preview image generation when dependencies are installed

## Sprint 4 additions
- Raster foundation is now implemented with real extraction and deterministic artifact outputs:
  - metadata returns width, height, band count, CRS, bounds, resolution, dtype, nodata, transform
  - preview endpoint generates and reuses PNG previews with width/height
  - clip operation persists a derived raster dataset with:
    - `source_dataset_id`
    - `operation`
    - `operation_parameters`
  - tile operation writes deterministic raster tiles under `tiles/{project_id}/{source_dataset_id}/{tile_set_id}`
  - tile manifest includes tile path, pixel window, bounds, transform, and count
- Dependency behavior:
  - when `rasterio` is missing, raster processing returns `RASTER_PROCESSING_UNAVAILABLE`
  - preview endpoint additionally requires numpy/pillow and returns `RASTER_PROCESSING_UNAVAILABLE` when missing

## Sprint 5 additions
- Raster analytics hardening:
  - raster band statistics now include:
    - min, max, mean, std
    - nodata count and ratio
    - valid pixel count
    - dtype
    - optional histogram bins (default 16 bins)
  - raster reproject operation implemented (CRS transform + rasterio reprojection) using dependency-aware raster processing checks.
  - reproject failures are explicit (`INVALID_PARAMETERS`, `INVALID_DATASET_CRS`, `RASTER_PROCESSING_UNAVAILABLE`).
- Raster clip and tile hardening:
  - clip validates area presence and CRS alignment constraints.
  - tile manifest records `tile_set_id`, `tile_size`, `overlap`, `source_dataset_id`, `source_raster_id`, bounds, parameters, count, tile paths, `ai_inference`, and `tile_server`.
- Job result persistence for raster ops:
  - raster clip/reproject/tile job payloads persist derived dataset references when outputs are produced.

## Sprint 6 additions
- Added local spectral index operations:
  - NDVI endpoint: `POST /raster/indices/ndvi`
  - NDWI endpoint: `POST /raster/indices/ndwi`
  - NDBI endpoint: `POST /raster/indices/ndbi`
- Spectral index input validation:
  - band parameters must be positive integers
  - band parameters must exist in source raster band count
- Dependency-aware execution:
  - returns `RASTER_PROCESSING_UNAVAILABLE` when rasterio or numpy are unavailable
- Real index output handling:
  - local windowed float32 GeoTIFF generation
  - `NaN` strategy for invalid pixels / division by zero
- Provenance capture for derived index datasets:
  - `source_dataset_id`, `operation`, `band_mapping`, `formula`
  - `output_dtype`, `nodata_strategy`, `value_range_note`
  - `output_dataset_id`, `created_at`, `path`

## Sprint 7B additions
- Added provider registry skeleton for `grb`, `osm`, `manual` and `fixture`.
- Added provider capability endpoints:
  - `GET /api/v1/external/providers`
  - `GET /api/v1/external/providers/{provider_name}`
  - `GET /api/v1/external/providers/{provider_name}/layers`
  - `GET /api/v1/external/providers/{provider_name}/status`
  - `POST /api/v1/external/providers/{provider_name}/import`
- GRB and OSM imports return explicit `not_configured` responses; no live WFS or Overpass calls are made.
- Manual and fixture providers describe existing upload/fixture flows only.
- Added live PostGIS migration smoke script for environments with a real database:

```bash
bash scripts/live_migration_smoke.sh
```

## Sprint 8 additions
- Added Detection Lab foundation:
  - `detections` ORM model and Alembic migration with PostGIS geometry storage.
  - hardened `analysis_runs` for dataset/job/model/result metadata.
  - model registry capability service for `yolo-placeholder` and `manual-fixture-detector`.
  - detection service boundary for creating jobs, analysis runs and dependency-aware unavailable responses.
- Added detection endpoints:
  - `GET /api/v1/detection/models`
  - `POST /api/v1/detection/run`
  - `GET /api/v1/detection/runs/{analysis_run_id}`
  - `GET /api/v1/detection/runs/{analysis_run_id}/detections`
- YOLO/PyTorch real inference is not enabled in Sprint 8.
- Fixture detector mode is test/demo-only and requires explicit `fixture_mode=true`.

## Sprint 8B additions
- Added optional configured YOLO integration foundation:
  - `yolo-configured` model registry capability.
  - import-safe adapter for local Ultralytics model files.
  - raster tile manifest validation and tile limit enforcement.
  - pixel bbox to EPSG:4326 detection polygon conversion.
  - persisted detections through the existing detection/job/analysis-run path.
- YOLO dependencies are optional extras and are not required for backend startup.
- GeoIntel does not download YOLO model weights automatically.

## Sprint 8C additions
- Added detection visualization/review API support:
  - list detection runs
  - list detections by run or dataset with class/confidence filters
  - get detection detail
  - return persisted detections as GeoJSON FeatureCollections
- Added detection QA against reference vector datasets:
  - compares persisted detection geometries against persisted `vector_features`
  - persists `quality_checks` and `metrics`
  - returns precision, recall, F1, mean IoU and false positive/negative counts
- Segmentation, LiDAR, AI Copilot, Training Studio and Reports remain out of scope.

## Sprint 9 additions
- Added Segmentation Lab foundation:
  - `segmentations` ORM model and Alembic migration with PostGIS MultiPolygon geometry storage.
  - segmentation model registry capabilities for `segmentation-placeholder`, `fixture-segmenter`, `yolo-seg-configured` and `sam-configured`.
  - segmentation service boundary for creating jobs, analysis runs and unavailable model responses.
  - explicit fixture segmenter mode for tests/demo fixtures only.
- Added segmentation endpoints:
  - `GET /api/v1/segmentation/models`
  - `POST /api/v1/segmentation/run`
  - `GET /api/v1/segmentation/runs`
  - `GET /api/v1/segmentation/runs/{analysis_run_id}`
  - `GET /api/v1/segmentation/runs/{analysis_run_id}/segmentations`
  - `GET /api/v1/segmentation/runs/{analysis_run_id}/geojson`
  - `POST /api/v1/segmentation/runs/{analysis_run_id}/qa/reference`
- Real SAM and YOLO-seg inference are not enabled in Sprint 9.
- Mask paths are provenance/debug artifacts; persisted PostGIS geometry is authoritative for QA, map display and GeoJSON.

## Sprint 17 additions
- Added export foundation backed by the existing `exports` table.
- GeoJSON exports now persist export records and write JSON artifacts for:
  - vector datasets
  - detection analysis runs
  - segmentation analysis runs
- Added project metadata JSON export for project, dataset and QA/QC summary state.
- Added export read/list/content endpoints:
  - `POST /api/v1/exports/geojson`
  - `POST /api/v1/exports/metadata`
  - `GET /api/v1/exports/projects/{project_id}/exports`
  - `GET /api/v1/exports/{export_id}`
  - `GET /api/v1/exports/{export_id}/content`
- Exported detection and segmentation GeoJSON is generated from persisted first-class geometry rows.
- No new migrations, product lines, live providers or AI dependencies are introduced by this export pass.

## Run locally

### Prerequisites

- Python 3.11+
- PostgreSQL with PostGIS

### Install dependencies

```bash
cd backend
python -m pip install -e .[dev]
```

Optional AI dependencies for configured local YOLO inference:

```bash
cd backend
python -m pip install -e .[ai]
```

Configured YOLO requires:

```bash
YOLO_ENABLED=true
YOLO_MODEL_PATH=/absolute/path/to/local-model.pt
```

Optional tuning:

```bash
YOLO_MODEL_ID=yolo-configured
YOLO_MODEL_DISPLAY_NAME="Configured YOLO detector"
YOLO_MODEL_VERSION=local-v1
YOLO_DEVICE=cpu
YOLO_IMAGE_SIZE=640
YOLO_MAX_TILES=100
YOLO_BATCH_SIZE=1
```

### YOLO local preflight

Sprint 13 adds a local-only preflight for configured YOLO paths:

```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json
```

Machine-readable output:

```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --json
```

To validate only local model/manifest paths on a machine without optional AI dependencies:

```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --assume-dependencies --json
```

The preflight checks configuration, dependency availability, local model file existence, tile manifest validity, tile count and referenced tile paths. It does not load a YOLO model, run inference or download weights.

### Run backend

```bash
cd backend
python -m uvicorn app.main:app --reload
```

### Run backend tests

```bash
cd backend
python -m pytest
```

For warning-sensitive release checks, the backend is expected to pass with Python deprecation warnings promoted to errors for the timestamp-heavy service paths:

```bash
cd backend
python -m pytest -W error::DeprecationWarning tests/test_geojson_dataset_service.py tests/test_qa_service.py tests/test_sprint7a_persistence_foundation.py tests/test_sprint8c_detection_visualization_qa.py tests/test_sprint9_segmentation_foundation.py tests/test_vector_operations_service.py
```

The repository readiness gate now applies the same warning policy to the full backend suite:

```bash
bash scripts/run_readiness_check.sh
```

That readiness gate also runs the API contract smoke check before backend/frontend compilation and tests.

### Golden QA/QC benchmark

Sprint 12 includes a deterministic QA/QC regression benchmark using explicit fixture data:

```bash
python scripts/run_golden_qa_benchmark.py
```

Machine-readable output:

```bash
python scripts/run_golden_qa_benchmark.py --json
```

The benchmark compares `fixtures/golden/predicted_buildings.geojson` against `fixtures/golden/reference_buildings.geojson` and fails on metric drift. Expected baseline:

- precision: `0.5`
- recall: `0.5`
- F1: `0.5`
- mean IoU: `0.8339768339761133`
- false positives: `1`
- false negatives: `1`

The command uses existing QA/QC service logic and verifies `QualityCheck`/`Metric` persistence through an in-memory test session. It does not require live providers, AI models, Docker or PostGIS.

### Demo workflow seed

Sprint 15 adds an explicit offline demo workflow seed. It creates or returns a
demo project, AOI, fixture reference buildings, fixture candidate buildings and
a persisted QA/QC result. It does not fetch live GRB/OSM data and does not run
AI inference.

API:

```bash
curl -X POST http://localhost:1202/api/v1/demo/workflow
```

CLI:

```bash
python scripts/seed_demo_workflow.py --json
```

In Docker Compose on a LAN host:

```bash
curl -X POST http://192.168.10.150:1202/api/v1/demo/workflow
```

### QA/QC result listing

Persisted project quality checks and metric rows can be listed with:

```bash
curl http://localhost:1202/api/v1/projects/{project_id}/quality-checks
```

The frontend QA/QC Results panel uses this endpoint after loading the demo
workflow or running QA.

### Export foundation

Persisted exports can be created from the existing workbench state:

```bash
curl -X POST http://localhost:1202/api/v1/exports/metadata \
  -H "Content-Type: application/json" \
  -d '{"project_id":"PROJECT_UUID"}'
```

Vector dataset GeoJSON export:

```bash
curl -X POST http://localhost:1202/api/v1/exports/geojson \
  -H "Content-Type: application/json" \
  -d '{"export_kind":"dataset","dataset_id":"DATASET_UUID"}'
```

Detection or segmentation run GeoJSON export:

```bash
curl -X POST http://localhost:1202/api/v1/exports/geojson \
  -H "Content-Type: application/json" \
  -d '{"export_kind":"detection_run","analysis_run_id":"ANALYSIS_RUN_UUID"}'
```

List and inspect exports:

```bash
curl http://localhost:1202/api/v1/exports/projects/PROJECT_UUID/exports
curl http://localhost:1202/api/v1/exports/EXPORT_UUID/content
```

Download an artifact as a browser/file response:

```bash
curl -OJ http://localhost:1202/api/v1/exports/EXPORT_UUID/download
```

Create a lightweight HTML project report artifact:

```bash
curl -X POST http://localhost:1202/api/v1/exports/report \
  -H "Content-Type: application/json" \
  -d '{"project_id":"PROJECT_UUID"}'
```

The report contains project, dataset, QA/QC summary and export history state
only. It is not a PDF designer and does not add a separate reporting module.

After rebuilding a Docker/LAN deployment, verify the end-to-end demo and export
flow through the browser-facing frontend proxy:

```bash
bash scripts/verify_demo_export_workflow.sh http://192.168.10.150:1202
```

The script seeds the explicit demo workflow, verifies persisted QA/QC results,
creates metadata/report/vector GeoJSON exports, lists exports and downloads the
JSON/GeoJSON/HTML artifacts.

### Backend import smoke

```bash
cd backend
python -c "from app.main import app; print(app.title)"
```

### Dockerized backend

```bash
docker compose up --build backend db
```

The Docker Compose stack does not require a root `.env` file for the default local runtime. The database service exposes a container-internal Postgres healthcheck, and the backend also runs `docker_start.sh`, which retries an actual SQL `SELECT 1` connection before running `python -m alembic upgrade head` and starting Uvicorn.

PostGIS is not published on the host `5432` port by default. This avoids conflicts with existing Postgres/PostGIS services on NAS or server hosts. The backend connects over Docker networking with `db:5432`.

Backend and frontend Docker build contexts exclude dependency folders, build outputs and Python bytecode caches via `.dockerignore`.

The Docker Compose frontend is published at `http://localhost:1202`.

Compose healthchecks are enabled for all runtime services:

- `db` uses `pg_isready`.
- `backend` checks `http://127.0.0.1:8000/health` inside the container.
- `frontend` checks `http://127.0.0.1/health` through nginx, which also verifies the frontend-to-backend proxy path.

The frontend waits for a healthy backend before starting. Check runtime state:

```bash
docker compose ps
docker compose logs --tail=80 backend
docker compose logs --tail=80 frontend
```

The backend Docker image installs the approved GIS runtime extra (`.[gis]`) so
browser-facing Docker deployments can report raster/vector processing
capabilities accurately:

- `rasterio`
- `numpy`
- `pillow`
- `geopandas`
- `pyogrio`
- GDAL/GEOS/PROJ system libraries

After rebuilding the backend image, verify the LAN/browser runtime from the
repository root:

```bash
bash scripts/verify_gis_runtime.sh http://localhost:1202
```

On a NAS or server host, use the published LAN URL:

```bash
bash scripts/verify_gis_runtime.sh http://192.168.10.150:1202
```

The script calls `/api/v1/system/capabilities` through the frontend proxy and
fails if `postgis`, `rasterio` or `geopandas` are not reported as available.

The backend Docker build also runs:

```bash
python scripts/gis_import_smoke.py
```

Inside the backend Docker build context this resolves to
`backend/scripts/gis_import_smoke.py`. The root `scripts/gis_import_smoke.py`
wrapper calls the same smoke locally. The smoke imports `rasterio`, `geopandas`
and `pyogrio`; if one of those imports fails, the backend image build fails
before deployment.

### Live Docker/PostGIS migration smoke

Sprint 11 validates the real PostGIS runtime path with the existing database service. From the repository root:

```bash
docker compose config
docker compose up -d db
DATABASE_URL=postgresql+psycopg://geointel:geointel@localhost:5432/geointel bash scripts/live_migration_smoke.sh
```

The smoke script:

- opens a backend SQLAlchemy connection and runs `SELECT 1`
- runs `alembic upgrade head`
- checks `PostGIS_Version()` after migrations have created the extension
- verifies one Alembic head
- verifies required migrated tables and GiST indexes exist

Expected local environment:

```bash
DATABASE_URL=postgresql+psycopg://geointel:geointel@localhost:5432/geointel
```

If the database is not reachable, confirm Docker Desktop is running and that port `5432` is not already occupied. To clean up the local database container without deleting the named volume:

```bash
docker compose stop db
```

To remove the local PostGIS volume as well, use only when you explicitly want a fresh database:

```bash
docker compose down -v
```

## Key docs
- `docs/API_CONTRACTS.md`
- `docs/DATABASE_IMPLEMENTATION_PLAN.md`
- `docs/DEFINITION_OF_DONE.md`
- `docs/40-build-launch/SPRINT_1_SCOPE_FREEZE.md`

## Raster dependency note

Raster metadata and raster operations depend on local GDAL/rasterio availability.

To enable raster processing locally:

```bash
python -m pip install rasterio
```

If `rasterio` is unavailable:
- raster metadata responses return `503` with `RASTER_PROCESSING_UNAVAILABLE`
- raster clip/tile endpoints return explicit unavailable responses

## Helpful repository scripts

- `bash scripts/backend_install.sh`
- `bash scripts/backend_test.sh`
- `bash scripts/backend_dev.sh`
- `bash scripts/smoke_backend_import.sh`