Initial GeoIntel V1 foundation

2026-06-16 23:36:32 +02:00
commit 6ea3586a3e
605 changed files with 45284 additions and 0 deletions
@@ -0,0 +1,537 @@
+# GeoIntel Backend (Sprint 3 foundation layer)
+
+FastAPI backend for GeoIntel Kempen Foundation Sprints.
+
+## Scope implemented
+- Project CRUD
+- Area CRUD with PostGIS geometry
+- Vector and raster dataset upload/registration
+- Deterministic local storage metadata capture
+- PostGIS migration and database foundation
+- Job foundation for async-ready GIS operations
+
+## Sprint 2 additions
+- Dataset typing and lifecycle support:
+  - `uploaded`
+  - `validating`
+  - `ready`
+  - `failed`
+- Vector metadata extraction:
+  - feature count
+  - geometry type summary
+  - bounds
+  - approximate area
+  - CRS and CRS assumption
+- Raster metadata endpoint:
+  - returns raster profile when `rasterio` is available
+  - returns clear `RASTER_PROCESSING_UNAVAILABLE` error when dependency is missing
+- Deterministic storage metadata capture:
+  - original filename
+  - stored filename
+  - MIME/content type
+  - size bytes
+  - checksum SHA-256
+
+## Sprint 3 additions
+- Lightweight job architecture:
+  - `jobs` table and migrations
+  - job create/list/read/status API
+  - synchronous execution behind job abstraction
+- Vector operations foundation:
+  - inspect
+  - bbox
+  - stats
+  - clip by area
+  - buffer
+  - intersect
+  - invalid geometry rejection with typed errors
+- Raster operation foundation:
+  - inspect
+  - metadata
+  - preview readiness
+  - clip by area (dependency-aware with unavailable fallback)
+  - tile generation with manifest output
+  - real preview image generation when dependencies are installed
+
+## Sprint 4 additions
+- Raster foundation is now implemented with real extraction and deterministic artifact outputs:
+  - metadata returns width, height, band count, CRS, bounds, resolution, dtype, nodata, transform
+  - preview endpoint generates and reuses PNG previews with width/height
+  - clip operation persists a derived raster dataset with:
+    - `source_dataset_id`
+    - `operation`
+    - `operation_parameters`
+  - tile operation writes deterministic raster tiles under `tiles/{project_id}/{source_dataset_id}/{tile_set_id}`
+  - tile manifest includes tile path, pixel window, bounds, transform, and count
+- Dependency behavior:
+  - when `rasterio` is missing, raster processing returns `RASTER_PROCESSING_UNAVAILABLE`
+  - preview endpoint additionally requires numpy/pillow and returns `RASTER_PROCESSING_UNAVAILABLE` when missing
+
+## Sprint 5 additions
+- Raster analytics hardening:
+  - raster band statistics now include:
+    - min, max, mean, std
+    - nodata count and ratio
+    - valid pixel count
+    - dtype
+    - optional histogram bins (default 16 bins)
+  - raster reproject operation implemented (CRS transform + rasterio reprojection) using dependency-aware raster processing checks.
+  - reproject failures are explicit (`INVALID_PARAMETERS`, `INVALID_DATASET_CRS`, `RASTER_PROCESSING_UNAVAILABLE`).
+- Raster clip and tile hardening:
+  - clip validates area presence and CRS alignment constraints.
+  - tile manifest records `tile_set_id`, `tile_size`, `overlap`, `source_dataset_id`, `source_raster_id`, bounds, parameters, count, tile paths, `ai_inference`, and `tile_server`.
+- Job result persistence for raster ops:
+  - raster clip/reproject/tile job payloads persist derived dataset references when outputs are produced.
+
+## Sprint 6 additions
+- Added local spectral index operations:
+  - NDVI endpoint: `POST /raster/indices/ndvi`
+  - NDWI endpoint: `POST /raster/indices/ndwi`
+  - NDBI endpoint: `POST /raster/indices/ndbi`
+- Spectral index input validation:
+  - band parameters must be positive integers
+  - band parameters must exist in source raster band count
+- Dependency-aware execution:
+  - returns `RASTER_PROCESSING_UNAVAILABLE` when rasterio or numpy are unavailable
+- Real index output handling:
+  - local windowed float32 GeoTIFF generation
+  - `NaN` strategy for invalid pixels / division by zero
+- Provenance capture for derived index datasets:
+  - `source_dataset_id`, `operation`, `band_mapping`, `formula`
+  - `output_dtype`, `nodata_strategy`, `value_range_note`
+  - `output_dataset_id`, `created_at`, `path`
+
+## Sprint 7B additions
+- Added provider registry skeleton for `grb`, `osm`, `manual` and `fixture`.
+- Added provider capability endpoints:
+  - `GET /api/v1/external/providers`
+  - `GET /api/v1/external/providers/{provider_name}`
+  - `GET /api/v1/external/providers/{provider_name}/layers`
+  - `GET /api/v1/external/providers/{provider_name}/status`
+  - `POST /api/v1/external/providers/{provider_name}/import`
+- GRB and OSM imports return explicit `not_configured` responses; no live WFS or Overpass calls are made.
+- Manual and fixture providers describe existing upload/fixture flows only.
+- Added live PostGIS migration smoke script for environments with a real database:
+
+```bash
+bash scripts/live_migration_smoke.sh
+```
+
+## Sprint 8 additions
+- Added Detection Lab foundation:
+  - `detections` ORM model and Alembic migration with PostGIS geometry storage.
+  - hardened `analysis_runs` for dataset/job/model/result metadata.
+  - model registry capability service for `yolo-placeholder` and `manual-fixture-detector`.
+  - detection service boundary for creating jobs, analysis runs and dependency-aware unavailable responses.
+- Added detection endpoints:
+  - `GET /api/v1/detection/models`
+  - `POST /api/v1/detection/run`
+  - `GET /api/v1/detection/runs/{analysis_run_id}`
+  - `GET /api/v1/detection/runs/{analysis_run_id}/detections`
+- YOLO/PyTorch real inference is not enabled in Sprint 8.
+- Fixture detector mode is test/demo-only and requires explicit `fixture_mode=true`.
+
+## Sprint 8B additions
+- Added optional configured YOLO integration foundation:
+  - `yolo-configured` model registry capability.
+  - import-safe adapter for local Ultralytics model files.
+  - raster tile manifest validation and tile limit enforcement.
+  - pixel bbox to EPSG:4326 detection polygon conversion.
+  - persisted detections through the existing detection/job/analysis-run path.
+- YOLO dependencies are optional extras and are not required for backend startup.
+- GeoIntel does not download YOLO model weights automatically.
+
+## Sprint 8C additions
+- Added detection visualization/review API support:
+  - list detection runs
+  - list detections by run or dataset with class/confidence filters
+  - get detection detail
+  - return persisted detections as GeoJSON FeatureCollections
+- Added detection QA against reference vector datasets:
+  - compares persisted detection geometries against persisted `vector_features`
+  - persists `quality_checks` and `metrics`
+  - returns precision, recall, F1, mean IoU and false positive/negative counts
+- Segmentation, LiDAR, AI Copilot, Training Studio and Reports remain out of scope.
+
+## Sprint 9 additions
+- Added Segmentation Lab foundation:
+  - `segmentations` ORM model and Alembic migration with PostGIS MultiPolygon geometry storage.
+  - segmentation model registry capabilities for `segmentation-placeholder`, `fixture-segmenter`, `yolo-seg-configured` and `sam-configured`.
+  - segmentation service boundary for creating jobs, analysis runs and unavailable model responses.
+  - explicit fixture segmenter mode for tests/demo fixtures only.
+- Added segmentation endpoints:
+  - `GET /api/v1/segmentation/models`
+  - `POST /api/v1/segmentation/run`
+  - `GET /api/v1/segmentation/runs`
+  - `GET /api/v1/segmentation/runs/{analysis_run_id}`
+  - `GET /api/v1/segmentation/runs/{analysis_run_id}/segmentations`
+  - `GET /api/v1/segmentation/runs/{analysis_run_id}/geojson`
+  - `POST /api/v1/segmentation/runs/{analysis_run_id}/qa/reference`
+- Real SAM and YOLO-seg inference are not enabled in Sprint 9.
+- Mask paths are provenance/debug artifacts; persisted PostGIS geometry is authoritative for QA, map display and GeoJSON.
+
+## Sprint 17 additions
+- Added export foundation backed by the existing `exports` table.
+- GeoJSON exports now persist export records and write JSON artifacts for:
+  - vector datasets
+  - detection analysis runs
+  - segmentation analysis runs
+- Added project metadata JSON export for project, dataset and QA/QC summary state.
+- Added export read/list/content endpoints:
+  - `POST /api/v1/exports/geojson`
+  - `POST /api/v1/exports/metadata`
+  - `GET /api/v1/exports/projects/{project_id}/exports`
+  - `GET /api/v1/exports/{export_id}`
+  - `GET /api/v1/exports/{export_id}/content`
+- Exported detection and segmentation GeoJSON is generated from persisted first-class geometry rows.
+- No new migrations, product lines, live providers or AI dependencies are introduced by this export pass.
+
+## Run locally
+
+### Prerequisites
+
+- Python 3.11+
+- PostgreSQL with PostGIS
+
+### Install dependencies
+
+```bash
+cd backend
+python -m pip install -e .[dev]
+```
+
+Optional AI dependencies for configured local YOLO inference:
+
+```bash
+cd backend
+python -m pip install -e .[ai]
+```
+
+Configured YOLO requires:
+
+```bash
+YOLO_ENABLED=true
+YOLO_MODEL_PATH=/absolute/path/to/local-model.pt
+```
+
+Optional tuning:
+
+```bash
+YOLO_MODEL_ID=yolo-configured
+YOLO_MODEL_DISPLAY_NAME="Configured YOLO detector"
+YOLO_MODEL_VERSION=local-v1
+YOLO_DEVICE=cpu
+YOLO_IMAGE_SIZE=640
+YOLO_MAX_TILES=100
+YOLO_BATCH_SIZE=1
+```
+
+### YOLO local preflight
+
+Sprint 13 adds a local-only preflight for configured YOLO paths:
+
+```bash
+python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json
+```
+
+Machine-readable output:
+
+```bash
+python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --json
+```
+
+To validate only local model/manifest paths on a machine without optional AI dependencies:
+
+```bash
+python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --assume-dependencies --json
+```
+
+The preflight checks configuration, dependency availability, local model file existence, tile manifest validity, tile count and referenced tile paths. It does not load a YOLO model, run inference or download weights.
+
+### Run backend
+
+```bash
+cd backend
+python -m uvicorn app.main:app --reload
+```
+
+### Run backend tests
+
+```bash
+cd backend
+python -m pytest
+```
+
+For warning-sensitive release checks, the backend is expected to pass with Python deprecation warnings promoted to errors for the timestamp-heavy service paths:
+
+```bash
+cd backend
+python -m pytest -W error::DeprecationWarning tests/test_geojson_dataset_service.py tests/test_qa_service.py tests/test_sprint7a_persistence_foundation.py tests/test_sprint8c_detection_visualization_qa.py tests/test_sprint9_segmentation_foundation.py tests/test_vector_operations_service.py
+```
+
+The repository readiness gate now applies the same warning policy to the full backend suite:
+
+```bash
+bash scripts/run_readiness_check.sh
+```
+
+That readiness gate also runs the API contract smoke check before backend/frontend compilation and tests.
+
+### Golden QA/QC benchmark
+
+Sprint 12 includes a deterministic QA/QC regression benchmark using explicit fixture data:
+
+```bash
+python scripts/run_golden_qa_benchmark.py
+```
+
+Machine-readable output:
+
+```bash
+python scripts/run_golden_qa_benchmark.py --json
+```
+
+The benchmark compares `fixtures/golden/predicted_buildings.geojson` against `fixtures/golden/reference_buildings.geojson` and fails on metric drift. Expected baseline:
+
+- precision: `0.5`
+- recall: `0.5`
+- F1: `0.5`
+- mean IoU: `0.8339768339761133`
+- false positives: `1`
+- false negatives: `1`
+
+The command uses existing QA/QC service logic and verifies `QualityCheck`/`Metric` persistence through an in-memory test session. It does not require live providers, AI models, Docker or PostGIS.
+
+### Demo workflow seed
+
+Sprint 15 adds an explicit offline demo workflow seed. It creates or returns a
+demo project, AOI, fixture reference buildings, fixture candidate buildings and
+a persisted QA/QC result. It does not fetch live GRB/OSM data and does not run
+AI inference.
+
+API:
+
+```bash
+curl -X POST http://localhost:1202/api/v1/demo/workflow
+```
+
+CLI:
+
+```bash
+python scripts/seed_demo_workflow.py --json
+```
+
+In Docker Compose on a LAN host:
+
+```bash
+curl -X POST http://192.168.10.150:1202/api/v1/demo/workflow
+```
+
+### QA/QC result listing
+
+Persisted project quality checks and metric rows can be listed with:
+
+```bash
+curl http://localhost:1202/api/v1/projects/{project_id}/quality-checks
+```
+
+The frontend QA/QC Results panel uses this endpoint after loading the demo
+workflow or running QA.
+
+### Export foundation
+
+Persisted exports can be created from the existing workbench state:
+
+```bash
+curl -X POST http://localhost:1202/api/v1/exports/metadata \
+  -H "Content-Type: application/json" \
+  -d '{"project_id":"PROJECT_UUID"}'
+```
+
+Vector dataset GeoJSON export:
+
+```bash
+curl -X POST http://localhost:1202/api/v1/exports/geojson \
+  -H "Content-Type: application/json" \
+  -d '{"export_kind":"dataset","dataset_id":"DATASET_UUID"}'
+```
+
+Detection or segmentation run GeoJSON export:
+
+```bash
+curl -X POST http://localhost:1202/api/v1/exports/geojson \
+  -H "Content-Type: application/json" \
+  -d '{"export_kind":"detection_run","analysis_run_id":"ANALYSIS_RUN_UUID"}'
+```
+
+List and inspect exports:
+
+```bash
+curl http://localhost:1202/api/v1/exports/projects/PROJECT_UUID/exports
+curl http://localhost:1202/api/v1/exports/EXPORT_UUID/content
+```
+
+Download an artifact as a browser/file response:
+
+```bash
+curl -OJ http://localhost:1202/api/v1/exports/EXPORT_UUID/download
+```
+
+Create a lightweight HTML project report artifact:
+
+```bash
+curl -X POST http://localhost:1202/api/v1/exports/report \
+  -H "Content-Type: application/json" \
+  -d '{"project_id":"PROJECT_UUID"}'
+```
+
+The report contains project, dataset, QA/QC summary and export history state
+only. It is not a PDF designer and does not add a separate reporting module.
+
+After rebuilding a Docker/LAN deployment, verify the end-to-end demo and export
+flow through the browser-facing frontend proxy:
+
+```bash
+bash scripts/verify_demo_export_workflow.sh http://192.168.10.150:1202
+```
+
+The script seeds the explicit demo workflow, verifies persisted QA/QC results,
+creates metadata/report/vector GeoJSON exports, lists exports and downloads the
+JSON/GeoJSON/HTML artifacts.
+
+### Backend import smoke
+
+```bash
+cd backend
+python -c "from app.main import app; print(app.title)"
+```
+
+### Dockerized backend
+
+```bash
+docker compose up --build backend db
+```
+
+The Docker Compose stack does not require a root `.env` file for the default local runtime. The database service exposes a container-internal Postgres healthcheck, and the backend also runs `docker_start.sh`, which retries an actual SQL `SELECT 1` connection before running `python -m alembic upgrade head` and starting Uvicorn.
+
+PostGIS is not published on the host `5432` port by default. This avoids conflicts with existing Postgres/PostGIS services on NAS or server hosts. The backend connects over Docker networking with `db:5432`.
+
+Backend and frontend Docker build contexts exclude dependency folders, build outputs and Python bytecode caches via `.dockerignore`.
+
+The Docker Compose frontend is published at `http://localhost:1202`.
+
+Compose healthchecks are enabled for all runtime services:
+
+- `db` uses `pg_isready`.
+- `backend` checks `http://127.0.0.1:8000/health` inside the container.
+- `frontend` checks `http://127.0.0.1/health` through nginx, which also verifies the frontend-to-backend proxy path.
+
+The frontend waits for a healthy backend before starting. Check runtime state:
+
+```bash
+docker compose ps
+docker compose logs --tail=80 backend
+docker compose logs --tail=80 frontend
+```
+
+The backend Docker image installs the approved GIS runtime extra (`.[gis]`) so
+browser-facing Docker deployments can report raster/vector processing
+capabilities accurately:
+
+- `rasterio`
+- `numpy`
+- `pillow`
+- `geopandas`
+- `pyogrio`
+- GDAL/GEOS/PROJ system libraries
+
+After rebuilding the backend image, verify the LAN/browser runtime from the
+repository root:
+
+```bash
+bash scripts/verify_gis_runtime.sh http://localhost:1202
+```
+
+On a NAS or server host, use the published LAN URL:
+
+```bash
+bash scripts/verify_gis_runtime.sh http://192.168.10.150:1202
+```
+
+The script calls `/api/v1/system/capabilities` through the frontend proxy and
+fails if `postgis`, `rasterio` or `geopandas` are not reported as available.
+
+The backend Docker build also runs:
+
+```bash
+python scripts/gis_import_smoke.py
+```
+
+Inside the backend Docker build context this resolves to
+`backend/scripts/gis_import_smoke.py`. The root `scripts/gis_import_smoke.py`
+wrapper calls the same smoke locally. The smoke imports `rasterio`, `geopandas`
+and `pyogrio`; if one of those imports fails, the backend image build fails
+before deployment.
+
+### Live Docker/PostGIS migration smoke
+
+Sprint 11 validates the real PostGIS runtime path with the existing database service. From the repository root:
+
+```bash
+docker compose config
+docker compose up -d db
+DATABASE_URL=postgresql+psycopg://geointel:geointel@localhost:5432/geointel bash scripts/live_migration_smoke.sh
+```
+
+The smoke script:
+
+- opens a backend SQLAlchemy connection and runs `SELECT 1`
+- runs `alembic upgrade head`
+- checks `PostGIS_Version()` after migrations have created the extension
+- verifies one Alembic head
+- verifies required migrated tables and GiST indexes exist
+
+Expected local environment:
+
+```bash
+DATABASE_URL=postgresql+psycopg://geointel:geointel@localhost:5432/geointel
+```
+
+If the database is not reachable, confirm Docker Desktop is running and that port `5432` is not already occupied. To clean up the local database container without deleting the named volume:
+
+```bash
+docker compose stop db
+```
+
+To remove the local PostGIS volume as well, use only when you explicitly want a fresh database:
+
+```bash
+docker compose down -v
+```
+
+## Key docs
+- `docs/API_CONTRACTS.md`
+- `docs/DATABASE_IMPLEMENTATION_PLAN.md`
+- `docs/DEFINITION_OF_DONE.md`
+- `docs/40-build-launch/SPRINT_1_SCOPE_FREEZE.md`
+
+## Raster dependency note
+
+Raster metadata and raster operations depend on local GDAL/rasterio availability.
+
+To enable raster processing locally:
+
+```bash
+python -m pip install rasterio
+```
+
+If `rasterio` is unavailable:
+- raster metadata responses return `503` with `RASTER_PROCESSING_UNAVAILABLE`
+- raster clip/tile endpoints return explicit unavailable responses
+
+## Helpful repository scripts
+
+- `bash scripts/backend_install.sh`
+- `bash scripts/backend_test.sh`
+- `bash scripts/backend_dev.sh`
+- `bash scripts/smoke_backend_import.sh`