Initial GeoIntel V1 foundation
GeoIntel CI / docs-smoke (push) Has been cancelled
GeoIntel CI / contract-smoke (push) Has been cancelled

This commit is contained in:
Codex
2026-06-16 23:36:32 +02:00
commit 6ea3586a3e
605 changed files with 45284 additions and 0 deletions
+10
View File
@@ -0,0 +1,10 @@
__pycache__
*.pyc
.pytest_cache
.mypy_cache
.ruff_cache
geointel_backend.egg-info
storage
dist
node_modules
.env
View File
+25
View File
@@ -0,0 +1,25 @@
FROM python:3.12-slim
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
gdal-bin \
libgdal-dev \
libgeos-dev \
libproj-dev \
libpq-dev \
proj-bin \
&& rm -rf /var/lib/apt/lists/*
COPY pyproject.toml README.md /app/
COPY app /app/app
RUN pip install --no-cache-dir --upgrade pip setuptools
RUN pip install --no-cache-dir ".[gis]"
COPY . /app
RUN python scripts/gis_import_smoke.py
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
+537
View File
@@ -0,0 +1,537 @@
# GeoIntel Backend (Sprint 3 foundation layer)
FastAPI backend for GeoIntel Kempen Foundation Sprints.
## Scope implemented
- Project CRUD
- Area CRUD with PostGIS geometry
- Vector and raster dataset upload/registration
- Deterministic local storage metadata capture
- PostGIS migration and database foundation
- Job foundation for async-ready GIS operations
## Sprint 2 additions
- Dataset typing and lifecycle support:
- `uploaded`
- `validating`
- `ready`
- `failed`
- Vector metadata extraction:
- feature count
- geometry type summary
- bounds
- approximate area
- CRS and CRS assumption
- Raster metadata endpoint:
- returns raster profile when `rasterio` is available
- returns clear `RASTER_PROCESSING_UNAVAILABLE` error when dependency is missing
- Deterministic storage metadata capture:
- original filename
- stored filename
- MIME/content type
- size bytes
- checksum SHA-256
## Sprint 3 additions
- Lightweight job architecture:
- `jobs` table and migrations
- job create/list/read/status API
- synchronous execution behind job abstraction
- Vector operations foundation:
- inspect
- bbox
- stats
- clip by area
- buffer
- intersect
- invalid geometry rejection with typed errors
- Raster operation foundation:
- inspect
- metadata
- preview readiness
- clip by area (dependency-aware with unavailable fallback)
- tile generation with manifest output
- real preview image generation when dependencies are installed
## Sprint 4 additions
- Raster foundation is now implemented with real extraction and deterministic artifact outputs:
- metadata returns width, height, band count, CRS, bounds, resolution, dtype, nodata, transform
- preview endpoint generates and reuses PNG previews with width/height
- clip operation persists a derived raster dataset with:
- `source_dataset_id`
- `operation`
- `operation_parameters`
- tile operation writes deterministic raster tiles under `tiles/{project_id}/{source_dataset_id}/{tile_set_id}`
- tile manifest includes tile path, pixel window, bounds, transform, and count
- Dependency behavior:
- when `rasterio` is missing, raster processing returns `RASTER_PROCESSING_UNAVAILABLE`
- preview endpoint additionally requires numpy/pillow and returns `RASTER_PROCESSING_UNAVAILABLE` when missing
## Sprint 5 additions
- Raster analytics hardening:
- raster band statistics now include:
- min, max, mean, std
- nodata count and ratio
- valid pixel count
- dtype
- optional histogram bins (default 16 bins)
- raster reproject operation implemented (CRS transform + rasterio reprojection) using dependency-aware raster processing checks.
- reproject failures are explicit (`INVALID_PARAMETERS`, `INVALID_DATASET_CRS`, `RASTER_PROCESSING_UNAVAILABLE`).
- Raster clip and tile hardening:
- clip validates area presence and CRS alignment constraints.
- tile manifest records `tile_set_id`, `tile_size`, `overlap`, `source_dataset_id`, `source_raster_id`, bounds, parameters, count, tile paths, `ai_inference`, and `tile_server`.
- Job result persistence for raster ops:
- raster clip/reproject/tile job payloads persist derived dataset references when outputs are produced.
## Sprint 6 additions
- Added local spectral index operations:
- NDVI endpoint: `POST /raster/indices/ndvi`
- NDWI endpoint: `POST /raster/indices/ndwi`
- NDBI endpoint: `POST /raster/indices/ndbi`
- Spectral index input validation:
- band parameters must be positive integers
- band parameters must exist in source raster band count
- Dependency-aware execution:
- returns `RASTER_PROCESSING_UNAVAILABLE` when rasterio or numpy are unavailable
- Real index output handling:
- local windowed float32 GeoTIFF generation
- `NaN` strategy for invalid pixels / division by zero
- Provenance capture for derived index datasets:
- `source_dataset_id`, `operation`, `band_mapping`, `formula`
- `output_dtype`, `nodata_strategy`, `value_range_note`
- `output_dataset_id`, `created_at`, `path`
## Sprint 7B additions
- Added provider registry skeleton for `grb`, `osm`, `manual` and `fixture`.
- Added provider capability endpoints:
- `GET /api/v1/external/providers`
- `GET /api/v1/external/providers/{provider_name}`
- `GET /api/v1/external/providers/{provider_name}/layers`
- `GET /api/v1/external/providers/{provider_name}/status`
- `POST /api/v1/external/providers/{provider_name}/import`
- GRB and OSM imports return explicit `not_configured` responses; no live WFS or Overpass calls are made.
- Manual and fixture providers describe existing upload/fixture flows only.
- Added live PostGIS migration smoke script for environments with a real database:
```bash
bash scripts/live_migration_smoke.sh
```
## Sprint 8 additions
- Added Detection Lab foundation:
- `detections` ORM model and Alembic migration with PostGIS geometry storage.
- hardened `analysis_runs` for dataset/job/model/result metadata.
- model registry capability service for `yolo-placeholder` and `manual-fixture-detector`.
- detection service boundary for creating jobs, analysis runs and dependency-aware unavailable responses.
- Added detection endpoints:
- `GET /api/v1/detection/models`
- `POST /api/v1/detection/run`
- `GET /api/v1/detection/runs/{analysis_run_id}`
- `GET /api/v1/detection/runs/{analysis_run_id}/detections`
- YOLO/PyTorch real inference is not enabled in Sprint 8.
- Fixture detector mode is test/demo-only and requires explicit `fixture_mode=true`.
## Sprint 8B additions
- Added optional configured YOLO integration foundation:
- `yolo-configured` model registry capability.
- import-safe adapter for local Ultralytics model files.
- raster tile manifest validation and tile limit enforcement.
- pixel bbox to EPSG:4326 detection polygon conversion.
- persisted detections through the existing detection/job/analysis-run path.
- YOLO dependencies are optional extras and are not required for backend startup.
- GeoIntel does not download YOLO model weights automatically.
## Sprint 8C additions
- Added detection visualization/review API support:
- list detection runs
- list detections by run or dataset with class/confidence filters
- get detection detail
- return persisted detections as GeoJSON FeatureCollections
- Added detection QA against reference vector datasets:
- compares persisted detection geometries against persisted `vector_features`
- persists `quality_checks` and `metrics`
- returns precision, recall, F1, mean IoU and false positive/negative counts
- Segmentation, LiDAR, AI Copilot, Training Studio and Reports remain out of scope.
## Sprint 9 additions
- Added Segmentation Lab foundation:
- `segmentations` ORM model and Alembic migration with PostGIS MultiPolygon geometry storage.
- segmentation model registry capabilities for `segmentation-placeholder`, `fixture-segmenter`, `yolo-seg-configured` and `sam-configured`.
- segmentation service boundary for creating jobs, analysis runs and unavailable model responses.
- explicit fixture segmenter mode for tests/demo fixtures only.
- Added segmentation endpoints:
- `GET /api/v1/segmentation/models`
- `POST /api/v1/segmentation/run`
- `GET /api/v1/segmentation/runs`
- `GET /api/v1/segmentation/runs/{analysis_run_id}`
- `GET /api/v1/segmentation/runs/{analysis_run_id}/segmentations`
- `GET /api/v1/segmentation/runs/{analysis_run_id}/geojson`
- `POST /api/v1/segmentation/runs/{analysis_run_id}/qa/reference`
- Real SAM and YOLO-seg inference are not enabled in Sprint 9.
- Mask paths are provenance/debug artifacts; persisted PostGIS geometry is authoritative for QA, map display and GeoJSON.
## Sprint 17 additions
- Added export foundation backed by the existing `exports` table.
- GeoJSON exports now persist export records and write JSON artifacts for:
- vector datasets
- detection analysis runs
- segmentation analysis runs
- Added project metadata JSON export for project, dataset and QA/QC summary state.
- Added export read/list/content endpoints:
- `POST /api/v1/exports/geojson`
- `POST /api/v1/exports/metadata`
- `GET /api/v1/exports/projects/{project_id}/exports`
- `GET /api/v1/exports/{export_id}`
- `GET /api/v1/exports/{export_id}/content`
- Exported detection and segmentation GeoJSON is generated from persisted first-class geometry rows.
- No new migrations, product lines, live providers or AI dependencies are introduced by this export pass.
## Run locally
### Prerequisites
- Python 3.11+
- PostgreSQL with PostGIS
### Install dependencies
```bash
cd backend
python -m pip install -e .[dev]
```
Optional AI dependencies for configured local YOLO inference:
```bash
cd backend
python -m pip install -e .[ai]
```
Configured YOLO requires:
```bash
YOLO_ENABLED=true
YOLO_MODEL_PATH=/absolute/path/to/local-model.pt
```
Optional tuning:
```bash
YOLO_MODEL_ID=yolo-configured
YOLO_MODEL_DISPLAY_NAME="Configured YOLO detector"
YOLO_MODEL_VERSION=local-v1
YOLO_DEVICE=cpu
YOLO_IMAGE_SIZE=640
YOLO_MAX_TILES=100
YOLO_BATCH_SIZE=1
```
### YOLO local preflight
Sprint 13 adds a local-only preflight for configured YOLO paths:
```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json
```
Machine-readable output:
```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --json
```
To validate only local model/manifest paths on a machine without optional AI dependencies:
```bash
python scripts/yolo_preflight.py --model-path /absolute/path/to/local-model.pt --tile-manifest-path /absolute/path/to/manifest.json --assume-dependencies --json
```
The preflight checks configuration, dependency availability, local model file existence, tile manifest validity, tile count and referenced tile paths. It does not load a YOLO model, run inference or download weights.
### Run backend
```bash
cd backend
python -m uvicorn app.main:app --reload
```
### Run backend tests
```bash
cd backend
python -m pytest
```
For warning-sensitive release checks, the backend is expected to pass with Python deprecation warnings promoted to errors for the timestamp-heavy service paths:
```bash
cd backend
python -m pytest -W error::DeprecationWarning tests/test_geojson_dataset_service.py tests/test_qa_service.py tests/test_sprint7a_persistence_foundation.py tests/test_sprint8c_detection_visualization_qa.py tests/test_sprint9_segmentation_foundation.py tests/test_vector_operations_service.py
```
The repository readiness gate now applies the same warning policy to the full backend suite:
```bash
bash scripts/run_readiness_check.sh
```
That readiness gate also runs the API contract smoke check before backend/frontend compilation and tests.
### Golden QA/QC benchmark
Sprint 12 includes a deterministic QA/QC regression benchmark using explicit fixture data:
```bash
python scripts/run_golden_qa_benchmark.py
```
Machine-readable output:
```bash
python scripts/run_golden_qa_benchmark.py --json
```
The benchmark compares `fixtures/golden/predicted_buildings.geojson` against `fixtures/golden/reference_buildings.geojson` and fails on metric drift. Expected baseline:
- precision: `0.5`
- recall: `0.5`
- F1: `0.5`
- mean IoU: `0.8339768339761133`
- false positives: `1`
- false negatives: `1`
The command uses existing QA/QC service logic and verifies `QualityCheck`/`Metric` persistence through an in-memory test session. It does not require live providers, AI models, Docker or PostGIS.
### Demo workflow seed
Sprint 15 adds an explicit offline demo workflow seed. It creates or returns a
demo project, AOI, fixture reference buildings, fixture candidate buildings and
a persisted QA/QC result. It does not fetch live GRB/OSM data and does not run
AI inference.
API:
```bash
curl -X POST http://localhost:1202/api/v1/demo/workflow
```
CLI:
```bash
python scripts/seed_demo_workflow.py --json
```
In Docker Compose on a LAN host:
```bash
curl -X POST http://192.168.10.150:1202/api/v1/demo/workflow
```
### QA/QC result listing
Persisted project quality checks and metric rows can be listed with:
```bash
curl http://localhost:1202/api/v1/projects/{project_id}/quality-checks
```
The frontend QA/QC Results panel uses this endpoint after loading the demo
workflow or running QA.
### Export foundation
Persisted exports can be created from the existing workbench state:
```bash
curl -X POST http://localhost:1202/api/v1/exports/metadata \
-H "Content-Type: application/json" \
-d '{"project_id":"PROJECT_UUID"}'
```
Vector dataset GeoJSON export:
```bash
curl -X POST http://localhost:1202/api/v1/exports/geojson \
-H "Content-Type: application/json" \
-d '{"export_kind":"dataset","dataset_id":"DATASET_UUID"}'
```
Detection or segmentation run GeoJSON export:
```bash
curl -X POST http://localhost:1202/api/v1/exports/geojson \
-H "Content-Type: application/json" \
-d '{"export_kind":"detection_run","analysis_run_id":"ANALYSIS_RUN_UUID"}'
```
List and inspect exports:
```bash
curl http://localhost:1202/api/v1/exports/projects/PROJECT_UUID/exports
curl http://localhost:1202/api/v1/exports/EXPORT_UUID/content
```
Download an artifact as a browser/file response:
```bash
curl -OJ http://localhost:1202/api/v1/exports/EXPORT_UUID/download
```
Create a lightweight HTML project report artifact:
```bash
curl -X POST http://localhost:1202/api/v1/exports/report \
-H "Content-Type: application/json" \
-d '{"project_id":"PROJECT_UUID"}'
```
The report contains project, dataset, QA/QC summary and export history state
only. It is not a PDF designer and does not add a separate reporting module.
After rebuilding a Docker/LAN deployment, verify the end-to-end demo and export
flow through the browser-facing frontend proxy:
```bash
bash scripts/verify_demo_export_workflow.sh http://192.168.10.150:1202
```
The script seeds the explicit demo workflow, verifies persisted QA/QC results,
creates metadata/report/vector GeoJSON exports, lists exports and downloads the
JSON/GeoJSON/HTML artifacts.
### Backend import smoke
```bash
cd backend
python -c "from app.main import app; print(app.title)"
```
### Dockerized backend
```bash
docker compose up --build backend db
```
The Docker Compose stack does not require a root `.env` file for the default local runtime. The database service exposes a container-internal Postgres healthcheck, and the backend also runs `docker_start.sh`, which retries an actual SQL `SELECT 1` connection before running `python -m alembic upgrade head` and starting Uvicorn.
PostGIS is not published on the host `5432` port by default. This avoids conflicts with existing Postgres/PostGIS services on NAS or server hosts. The backend connects over Docker networking with `db:5432`.
Backend and frontend Docker build contexts exclude dependency folders, build outputs and Python bytecode caches via `.dockerignore`.
The Docker Compose frontend is published at `http://localhost:1202`.
Compose healthchecks are enabled for all runtime services:
- `db` uses `pg_isready`.
- `backend` checks `http://127.0.0.1:8000/health` inside the container.
- `frontend` checks `http://127.0.0.1/health` through nginx, which also verifies the frontend-to-backend proxy path.
The frontend waits for a healthy backend before starting. Check runtime state:
```bash
docker compose ps
docker compose logs --tail=80 backend
docker compose logs --tail=80 frontend
```
The backend Docker image installs the approved GIS runtime extra (`.[gis]`) so
browser-facing Docker deployments can report raster/vector processing
capabilities accurately:
- `rasterio`
- `numpy`
- `pillow`
- `geopandas`
- `pyogrio`
- GDAL/GEOS/PROJ system libraries
After rebuilding the backend image, verify the LAN/browser runtime from the
repository root:
```bash
bash scripts/verify_gis_runtime.sh http://localhost:1202
```
On a NAS or server host, use the published LAN URL:
```bash
bash scripts/verify_gis_runtime.sh http://192.168.10.150:1202
```
The script calls `/api/v1/system/capabilities` through the frontend proxy and
fails if `postgis`, `rasterio` or `geopandas` are not reported as available.
The backend Docker build also runs:
```bash
python scripts/gis_import_smoke.py
```
Inside the backend Docker build context this resolves to
`backend/scripts/gis_import_smoke.py`. The root `scripts/gis_import_smoke.py`
wrapper calls the same smoke locally. The smoke imports `rasterio`, `geopandas`
and `pyogrio`; if one of those imports fails, the backend image build fails
before deployment.
### Live Docker/PostGIS migration smoke
Sprint 11 validates the real PostGIS runtime path with the existing database service. From the repository root:
```bash
docker compose config
docker compose up -d db
DATABASE_URL=postgresql+psycopg://geointel:geointel@localhost:5432/geointel bash scripts/live_migration_smoke.sh
```
The smoke script:
- opens a backend SQLAlchemy connection and runs `SELECT 1`
- runs `alembic upgrade head`
- checks `PostGIS_Version()` after migrations have created the extension
- verifies one Alembic head
- verifies required migrated tables and GiST indexes exist
Expected local environment:
```bash
DATABASE_URL=postgresql+psycopg://geointel:geointel@localhost:5432/geointel
```
If the database is not reachable, confirm Docker Desktop is running and that port `5432` is not already occupied. To clean up the local database container without deleting the named volume:
```bash
docker compose stop db
```
To remove the local PostGIS volume as well, use only when you explicitly want a fresh database:
```bash
docker compose down -v
```
## Key docs
- `docs/API_CONTRACTS.md`
- `docs/DATABASE_IMPLEMENTATION_PLAN.md`
- `docs/DEFINITION_OF_DONE.md`
- `docs/40-build-launch/SPRINT_1_SCOPE_FREEZE.md`
## Raster dependency note
Raster metadata and raster operations depend on local GDAL/rasterio availability.
To enable raster processing locally:
```bash
python -m pip install rasterio
```
If `rasterio` is unavailable:
- raster metadata responses return `503` with `RASTER_PROCESSING_UNAVAILABLE`
- raster clip/tile endpoints return explicit unavailable responses
## Helpful repository scripts
- `bash scripts/backend_install.sh`
- `bash scripts/backend_test.sh`
- `bash scripts/backend_dev.sh`
- `bash scripts/smoke_backend_import.sh`
+38
View File
@@ -0,0 +1,38 @@
[alembic]
script_location = alembic
prepend_sys_path = .
sqlalchemy.url = postgresql+psycopg://geointel:geointel@localhost:5432/geointel
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARNING
handlers = console
qualname =
[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
class_ = logging.Formatter
+48
View File
@@ -0,0 +1,48 @@
from __future__ import annotations
import os
import sys
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
sys.path.append(os.path.realpath(os.path.join(os.path.dirname(__file__), '..')))
from app.core.config import get_settings
from app.db.base import Base
import app.models.entities # noqa: F401
settings = get_settings()
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
config.set_main_option("sqlalchemy.url", settings.database_url)
target_metadata = Base.metadata
def run_migrations_offline() -> None:
url = config.get_main_option("sqlalchemy.url")
context.configure(url=url, target_metadata=target_metadata, literal_binds=True)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
+20
View File
@@ -0,0 +1,20 @@
"""
${message}
"""
from alembic import op
import sqlalchemy as sa
${imports}
revision = ${repr(revision)}
down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
def upgrade():
${upgrades if upgrades else "pass"}
def downgrade():
${downgrades if downgrades else "pass"}
@@ -0,0 +1,108 @@
"""Initial PostGIS schema for Sprint 1 foundation."""
from alembic import op
import sqlalchemy as sa
from geoalchemy2 import Geometry
revision = "202601110001"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
op.execute("CREATE EXTENSION IF NOT EXISTS postgis")
op.execute("CREATE EXTENSION IF NOT EXISTS postgis_topology")
op.execute('CREATE EXTENSION IF NOT EXISTS "uuid-ossp"')
op.create_table(
"projects",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("name", sa.Text(), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("region", sa.Text(), nullable=False, server_default="Kempen"),
sa.Column("status", sa.Text(), nullable=False, server_default="active"),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
)
op.create_table(
"areas",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("project_id", sa.UUID(as_uuid=True), sa.ForeignKey("projects.id", ondelete="CASCADE"), nullable=False),
sa.Column("name", sa.Text(), nullable=False),
sa.Column("geometry", Geometry("MULTIPOLYGON", srid=4326), nullable=False),
sa.Column("original_crs", sa.Text(), nullable=True),
sa.Column("area_m2", sa.Float(), nullable=True),
sa.Column("bbox", Geometry("POLYGON", srid=4326), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
)
op.create_table(
"datasets",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("project_id", sa.UUID(as_uuid=True), sa.ForeignKey("projects.id", ondelete="CASCADE"), nullable=False),
sa.Column("area_id", sa.UUID(as_uuid=True), sa.ForeignKey("areas.id", ondelete="SET NULL"), nullable=True),
sa.Column("name", sa.Text(), nullable=False),
sa.Column("dataset_type", sa.Text(), nullable=False),
sa.Column("source", sa.Text(), nullable=False),
sa.Column("storage_path", sa.Text(), nullable=True),
sa.Column("derived_from_dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True),
sa.Column("crs", sa.Text(), nullable=True),
sa.Column("bounds_json", sa.JSON(), nullable=True),
sa.Column("resolution_json", sa.JSON(), nullable=True),
sa.Column("bands_json", sa.JSON(), nullable=True),
sa.Column("metadata_json", sa.JSON(), nullable=True),
sa.Column("status", sa.Text(), nullable=False, server_default="created"),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
)
op.create_table(
"dataset_versions",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="CASCADE"), nullable=False),
sa.Column("version", sa.Integer(), nullable=False, server_default="1"),
sa.Column("storage_path", sa.Text(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
)
op.create_table(
"analysis_runs",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("project_id", sa.UUID(as_uuid=True), sa.ForeignKey("projects.id", ondelete="CASCADE"), nullable=False),
sa.Column("area_id", sa.UUID(as_uuid=True), sa.ForeignKey("areas.id", ondelete="SET NULL"), nullable=True),
sa.Column("analysis_type", sa.Text(), nullable=False),
sa.Column("status", sa.Text(), nullable=False),
sa.Column("parameters_json", sa.JSON(), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("error_message", sa.Text(), nullable=True),
)
op.create_table(
"exports",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("project_id", sa.UUID(as_uuid=True), sa.ForeignKey("projects.id", ondelete="CASCADE"), nullable=False),
sa.Column("analysis_run_id", sa.UUID(as_uuid=True), sa.ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True),
sa.Column("export_type", sa.Text(), nullable=False),
sa.Column("storage_path", sa.Text(), nullable=False),
sa.Column("metadata_json", sa.JSON(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
)
op.create_index("ix_areas_geometry", "areas", ["geometry"], postgresql_using="gist")
op.create_index("ix_areas_project_id", "areas", ["project_id"])
op.create_index("ix_datasets_project_id", "datasets", ["project_id"])
def downgrade() -> None:
op.drop_index("ix_datasets_project_id", table_name="datasets")
op.drop_index("ix_areas_project_id", table_name="areas")
op.drop_index("ix_areas_geometry", table_name="areas", postgresql_using="gist")
op.drop_table("exports")
op.drop_table("analysis_runs")
op.drop_table("dataset_versions")
op.drop_table("datasets")
op.drop_table("areas")
op.drop_table("projects")
@@ -0,0 +1,27 @@
"""Add dataset storage metadata columns."""
from alembic import op
import sqlalchemy as sa
revision = "202601120001"
down_revision = "202601110001"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("datasets", sa.Column("original_filename", sa.Text(), nullable=True))
op.add_column("datasets", sa.Column("stored_filename", sa.Text(), nullable=True))
op.add_column("datasets", sa.Column("content_type", sa.Text(), nullable=True))
op.add_column("datasets", sa.Column("size_bytes", sa.Integer(), nullable=True))
op.add_column("datasets", sa.Column("checksum_sha256", sa.Text(), nullable=True))
op.alter_column("datasets", "status", server_default="uploaded")
def downgrade() -> None:
op.drop_column("datasets", "checksum_sha256")
op.drop_column("datasets", "size_bytes")
op.drop_column("datasets", "content_type")
op.drop_column("datasets", "stored_filename")
op.drop_column("datasets", "original_filename")
@@ -0,0 +1,38 @@
"""Add lightweight job table for sprint-3 async architecture foundation."""
from alembic import op
import sqlalchemy as sa
revision = "20260611212435"
down_revision = "202601120001"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"jobs",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("job_type", sa.Text(), nullable=False),
sa.Column("status", sa.Text(), nullable=False, server_default="queued"),
sa.Column("project_id", sa.UUID(as_uuid=True), sa.ForeignKey("projects.id", ondelete="CASCADE"), nullable=False),
sa.Column("dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True),
sa.Column("input_dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True),
sa.Column("output_dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True),
sa.Column("parameters_json", sa.JSON(), nullable=False),
sa.Column("result_json", sa.JSON(), nullable=True),
sa.Column("error_message", sa.Text(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_jobs_project_id", "jobs", ["project_id"])
op.create_index("ix_jobs_status", "jobs", ["status"])
def downgrade() -> None:
op.drop_index("ix_jobs_status", table_name="jobs")
op.drop_index("ix_jobs_project_id", table_name="jobs")
op.drop_table("jobs")
@@ -0,0 +1,28 @@
"""Add dataset reference and provenance metadata columns."""
from alembic import op
import sqlalchemy as sa
revision = "202606120001"
down_revision = "20260611212435"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("datasets", sa.Column("dataset_role", sa.Text(), nullable=False, server_default="source"))
op.add_column("datasets", sa.Column("source_name", sa.Text(), nullable=True))
op.add_column("datasets", sa.Column("reference_layer_name", sa.Text(), nullable=True))
op.add_column("datasets", sa.Column("source_metadata", sa.JSON(), nullable=True))
op.add_column("datasets", sa.Column("provenance_metadata", sa.JSON(), nullable=True))
op.add_column("datasets", sa.Column("imported_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()"), nullable=False))
def downgrade() -> None:
op.drop_column("datasets", "imported_at")
op.drop_column("datasets", "provenance_metadata")
op.drop_column("datasets", "source_metadata")
op.drop_column("datasets", "reference_layer_name")
op.drop_column("datasets", "source_name")
op.drop_column("datasets", "dataset_role")
@@ -0,0 +1,76 @@
"""Add Sprint 7A vector feature and QA persistence foundation."""
from alembic import op
import sqlalchemy as sa
from geoalchemy2 import Geometry
revision = "202606120700"
down_revision = "202606120001"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"vector_features",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="CASCADE"), nullable=False),
sa.Column("feature_class", sa.Text(), nullable=True),
sa.Column("source_feature_id", sa.Text(), nullable=True),
sa.Column("properties_json", sa.JSON(), nullable=True),
sa.Column("geometry", Geometry("GEOMETRY", srid=4326, spatial_index=False), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
)
op.create_index("ix_vector_features_dataset_id", "vector_features", ["dataset_id"])
op.create_index("ix_vector_features_geometry", "vector_features", ["geometry"], postgresql_using="gist")
op.create_table(
"quality_checks",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("project_id", sa.UUID(as_uuid=True), sa.ForeignKey("projects.id", ondelete="CASCADE"), nullable=False),
sa.Column("job_id", sa.UUID(as_uuid=True), sa.ForeignKey("jobs.id", ondelete="SET NULL"), nullable=True),
sa.Column("analysis_run_id", sa.UUID(as_uuid=True), sa.ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True),
sa.Column("candidate_dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True),
sa.Column("reference_dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="CASCADE"), nullable=False),
sa.Column("check_type", sa.Text(), nullable=False),
sa.Column("status", sa.Text(), nullable=False),
sa.Column("score", sa.Float(), nullable=True),
sa.Column("parameters_json", sa.JSON(), nullable=True),
sa.Column("findings_json", sa.JSON(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_quality_checks_project_id", "quality_checks", ["project_id"])
op.create_index("ix_quality_checks_reference_dataset_id", "quality_checks", ["reference_dataset_id"])
op.create_index("ix_quality_checks_candidate_dataset_id", "quality_checks", ["candidate_dataset_id"])
op.create_index("ix_quality_checks_analysis_run_id", "quality_checks", ["analysis_run_id"])
op.create_table(
"metrics",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("quality_check_id", sa.UUID(as_uuid=True), sa.ForeignKey("quality_checks.id", ondelete="CASCADE"), nullable=True),
sa.Column("analysis_run_id", sa.UUID(as_uuid=True), sa.ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True),
sa.Column("metric_key", sa.Text(), nullable=False),
sa.Column("metric_value", sa.Float(), nullable=True),
sa.Column("metric_unit", sa.Text(), nullable=True),
sa.Column("label", sa.Text(), nullable=True),
sa.Column("metadata_json", sa.JSON(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()")),
)
op.create_index("ix_metrics_quality_check_id", "metrics", ["quality_check_id"])
op.create_index("ix_metrics_analysis_run_id", "metrics", ["analysis_run_id"])
def downgrade() -> None:
op.drop_index("ix_metrics_analysis_run_id", table_name="metrics")
op.drop_index("ix_metrics_quality_check_id", table_name="metrics")
op.drop_table("metrics")
op.drop_index("ix_quality_checks_analysis_run_id", table_name="quality_checks")
op.drop_index("ix_quality_checks_candidate_dataset_id", table_name="quality_checks")
op.drop_index("ix_quality_checks_reference_dataset_id", table_name="quality_checks")
op.drop_index("ix_quality_checks_project_id", table_name="quality_checks")
op.drop_table("quality_checks")
op.drop_index("ix_vector_features_geometry", table_name="vector_features", postgresql_using="gist")
op.drop_index("ix_vector_features_dataset_id", table_name="vector_features")
op.drop_table("vector_features")
@@ -0,0 +1,59 @@
"""Add Sprint 8 detection foundation."""
from alembic import op
import sqlalchemy as sa
from geoalchemy2 import Geometry
revision = "202606120800"
down_revision = "202606120700"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("analysis_runs", sa.Column("dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True))
op.add_column("analysis_runs", sa.Column("job_id", sa.UUID(as_uuid=True), sa.ForeignKey("jobs.id", ondelete="SET NULL"), nullable=True))
op.add_column("analysis_runs", sa.Column("model_name", sa.String(length=255), nullable=True))
op.add_column("analysis_runs", sa.Column("model_version", sa.String(length=120), nullable=True))
op.add_column("analysis_runs", sa.Column("result_json", sa.JSON(), nullable=True))
op.add_column("analysis_runs", sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()"), nullable=False))
op.create_table(
"detections",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("project_id", sa.UUID(as_uuid=True), sa.ForeignKey("projects.id", ondelete="CASCADE"), nullable=False),
sa.Column("dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True),
sa.Column("analysis_run_id", sa.UUID(as_uuid=True), sa.ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True),
sa.Column("job_id", sa.UUID(as_uuid=True), sa.ForeignKey("jobs.id", ondelete="SET NULL"), nullable=True),
sa.Column("model_name", sa.String(length=255), nullable=False),
sa.Column("model_version", sa.String(length=120), nullable=True),
sa.Column("class_name", sa.String(length=120), nullable=False),
sa.Column("confidence", sa.Float(), nullable=False),
sa.Column("geometry", Geometry("GEOMETRY", srid=4326, spatial_index=False), nullable=False),
sa.Column("bbox_json", sa.JSON(), nullable=True),
sa.Column("source_tile_path", sa.String(length=500), nullable=True),
sa.Column("properties_json", sa.JSON(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()"), nullable=False),
)
op.create_index("ix_detections_project_id", "detections", ["project_id"])
op.create_index("ix_detections_dataset_id", "detections", ["dataset_id"])
op.create_index("ix_detections_analysis_run_id", "detections", ["analysis_run_id"])
op.create_index("ix_detections_class_name", "detections", ["class_name"])
op.create_index("ix_detections_geometry", "detections", ["geometry"], postgresql_using="gist")
def downgrade() -> None:
op.drop_index("ix_detections_geometry", table_name="detections", postgresql_using="gist")
op.drop_index("ix_detections_class_name", table_name="detections")
op.drop_index("ix_detections_analysis_run_id", table_name="detections")
op.drop_index("ix_detections_dataset_id", table_name="detections")
op.drop_index("ix_detections_project_id", table_name="detections")
op.drop_table("detections")
op.drop_column("analysis_runs", "created_at")
op.drop_column("analysis_runs", "result_json")
op.drop_column("analysis_runs", "model_version")
op.drop_column("analysis_runs", "model_name")
op.drop_column("analysis_runs", "job_id")
op.drop_column("analysis_runs", "dataset_id")
@@ -0,0 +1,51 @@
"""Add Sprint 9 segmentation foundation."""
from alembic import op
import sqlalchemy as sa
from geoalchemy2 import Geometry
revision = "202606120900"
down_revision = "202606120800"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"segmentations",
sa.Column("id", sa.UUID(as_uuid=True), primary_key=True),
sa.Column("project_id", sa.UUID(as_uuid=True), sa.ForeignKey("projects.id", ondelete="CASCADE"), nullable=False),
sa.Column("dataset_id", sa.UUID(as_uuid=True), sa.ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True),
sa.Column("job_id", sa.UUID(as_uuid=True), sa.ForeignKey("jobs.id", ondelete="SET NULL"), nullable=True),
sa.Column("analysis_run_id", sa.UUID(as_uuid=True), sa.ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True),
sa.Column("model_name", sa.String(length=255), nullable=False),
sa.Column("model_version", sa.String(length=120), nullable=True),
sa.Column("class_name", sa.String(length=120), nullable=False),
sa.Column("confidence", sa.Float(), nullable=True),
sa.Column("geometry", Geometry("MultiPolygon", srid=4326, spatial_index=False), nullable=False),
sa.Column("bbox_json", sa.JSON(), nullable=True),
sa.Column("area_m2", sa.Float(), nullable=True),
sa.Column("mask_path", sa.Text(), nullable=True),
sa.Column("source_tile_path", sa.String(length=500), nullable=True),
sa.Column("tile_index", sa.Integer(), nullable=True),
sa.Column("properties_json", sa.JSON(), nullable=True),
sa.Column("provenance_json", sa.JSON(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("NOW()"), nullable=False),
)
op.create_index("ix_segmentations_project_id", "segmentations", ["project_id"])
op.create_index("ix_segmentations_dataset_id", "segmentations", ["dataset_id"])
op.create_index("ix_segmentations_analysis_run_id", "segmentations", ["analysis_run_id"])
op.create_index("ix_segmentations_job_id", "segmentations", ["job_id"])
op.create_index("ix_segmentations_class_name", "segmentations", ["class_name"])
op.create_index("ix_segmentations_geometry", "segmentations", ["geometry"], postgresql_using="gist")
def downgrade() -> None:
op.drop_index("ix_segmentations_geometry", table_name="segmentations", postgresql_using="gist")
op.drop_index("ix_segmentations_class_name", table_name="segmentations")
op.drop_index("ix_segmentations_job_id", table_name="segmentations")
op.drop_index("ix_segmentations_analysis_run_id", table_name="segmentations")
op.drop_index("ix_segmentations_dataset_id", table_name="segmentations")
op.drop_index("ix_segmentations_project_id", table_name="segmentations")
op.drop_table("segmentations")
View File
+3
View File
@@ -0,0 +1,3 @@
from app.models.entities import AnalysisRun, Area, Dataset, Export, Project
__all__ = ["AnalysisRun", "Area", "Dataset", "Export", "Project"]
View File
View File
View File
View File
+1
View File
@@ -0,0 +1 @@
__all__ = ["areas", "datasets", "health", "projects", "exports", "jobs", "external", "qa"]
+58
View File
@@ -0,0 +1,58 @@
from __future__ import annotations
from uuid import UUID
from fastapi import APIRouter, Depends, Query
from fastapi import HTTPException
from sqlalchemy.orm import Session
from app.db.session import get_db
from app.models import Area
from app.schemas.area import AreaCreate, AreaRead, AreaUpdate
from app.services.area_service import AreaService
from app.utils.response import envelope
router = APIRouter(prefix="/projects/{project_id}/areas", tags=["areas"])
@router.get("", response_model=dict)
def list_areas(
project_id: UUID,
limit: int = Query(default=50, ge=1, le=200),
offset: int = Query(default=0, ge=0),
db: Session = Depends(get_db),
):
areas, total = AreaService.list_areas(db, project_id=project_id, limit=limit, offset=offset)
return envelope({"items": [AreaRead.model_validate(area).model_dump() for area in areas], "total": total, "limit": limit, "offset": offset})
@router.post("", status_code=201, response_model=dict)
def create_area(project_id: UUID, payload: AreaCreate, db: Session = Depends(get_db)):
area = AreaService.create_area(db, project_id, payload)
return envelope(AreaRead.model_validate(area).model_dump())
@router.get("/{area_id}", response_model=dict)
def get_area(
project_id: UUID,
area_id: UUID,
db: Session = Depends(get_db),
):
area = AreaService.get_area(db, area_id)
if area.project_id != project_id:
raise HTTPException(status_code=404, detail="Area not found")
return envelope(AreaRead.model_validate(area).model_dump())
@router.patch("/{area_id}", response_model=dict)
def update_area(
project_id: UUID,
area_id: UUID,
payload: AreaUpdate,
db: Session = Depends(get_db),
):
existing = db.get(Area, area_id)
if not existing or existing.project_id != project_id:
raise HTTPException(status_code=404, detail="Area not found")
area = AreaService.update_area(db, area_id, payload)
return envelope(AreaRead.model_validate(area).model_dump())
+489
View File
@@ -0,0 +1,489 @@
from __future__ import annotations
import json
from typing import Any
from uuid import UUID
from uuid import UUID as _UUID
from fastapi import APIRouter, Depends, File, Form, HTTPException, Query
from fastapi import UploadFile
from sqlalchemy.orm import Session
from app.models import Area
from app.core.errors import AppError
from app.db.session import get_db
from app.schemas import (
RasterClipRequest,
RasterStatsResponse,
RasterReprojectRequest,
RasterTileRequest,
RasterNdviRequest,
RasterNdwiRequest,
RasterNdbiRequest,
VectorBBoxResponse,
VectorBufferRequest,
VectorClipRequest,
VectorIntersectRequest,
)
from app.schemas.job import JobCreate
from app.schemas.dataset import DatasetCreateResponse
from app.schemas.operations import VectorOperationResult
from app.services.job_service import JobService
from app.services.raster_operations_service import RasterOperationsService
from app.services.vector_operations_service import VectorOperationsService
from app.services.dataset_service import DatasetService
from app.utils.response import envelope
router = APIRouter(prefix="/projects/{project_id}", tags=["datasets"])
def _parse_metadata_json(raw: str | None, field_name: str) -> dict | None:
if raw is None:
return None
raw = raw.strip()
if not raw:
return None
try:
value = json.loads(raw)
except json.JSONDecodeError as exc:
raise AppError(code="INVALID_JSON", message=f"Invalid JSON for {field_name}", details={"field": field_name}, status_code=400) from exc
if not isinstance(value, dict):
raise AppError(code="INVALID_JSON", message=f"{field_name} must be a JSON object", details={"field": field_name}, status_code=400)
return value
def _run_job_sync(
db: Session,
project_id: UUID,
input_dataset_id: UUID,
job_type: str,
parameters: dict[str, Any],
operation,
) -> dict[str, Any]:
return JobService.run_sync_job(
db=db,
project_id=project_id,
job_type=job_type,
parameters=parameters,
operation=operation,
input_dataset_id=input_dataset_id,
)
@router.post("/datasets/upload", status_code=201, response_model=dict)
async def upload_dataset(
project_id: UUID,
file: UploadFile = File(...),
dataset_type: str = Form(...),
source: str = Form("user_upload"),
area_id: UUID | None = Form(None),
dataset_role: str = Form("source"),
source_name: str | None = Form(None),
reference_layer_name: str | None = Form(None),
source_metadata_json: str | None = Form(None),
provenance_metadata_json: str | None = Form(None),
db: Session = Depends(get_db),
):
if area_id is not None:
area = db.get(Area, area_id)
if not area:
raise AppError(code="AREA_NOT_FOUND", message="Area not found", status_code=404)
if area.project_id != project_id:
raise AppError(code="INVALID_DATASET_SCOPE", message="Area does not belong to this project", status_code=400)
created = await DatasetService.upload_dataset(
db,
project_id=project_id,
file=file,
dataset_type=dataset_type,
source=source,
dataset_role=dataset_role,
source_name=source_name,
reference_layer_name=reference_layer_name,
source_metadata=_parse_metadata_json(source_metadata_json, "source_metadata_json"),
provenance_metadata=_parse_metadata_json(provenance_metadata_json, "provenance_metadata_json"),
area_id=area_id,
)
return envelope(created.model_dump())
@router.get("/datasets", response_model=dict)
def list_datasets(
project_id: UUID,
limit: int = Query(default=50, ge=1, le=200),
offset: int = Query(default=0, ge=0),
db: Session = Depends(get_db),
):
datasets, total = DatasetService.list_datasets(db, project_id, limit=limit, offset=offset)
return envelope({"items": [item.model_dump() for item in datasets], "total": total, "limit": limit, "offset": offset})
@router.get("/datasets/{dataset_id}", response_model=dict)
def get_dataset(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
return envelope(DatasetCreateResponse.model_validate(dataset).model_dump())
@router.post("/datasets/{dataset_id}/metadata/refresh", response_model=dict)
def refresh_dataset_metadata(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
refreshed = DatasetService.refresh_metadata(db, dataset_id)
return envelope(refreshed.model_dump())
@router.get("/datasets/{dataset_id}/vector/inspect", response_model=dict)
def inspect_vector_dataset(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
return envelope(VectorOperationsService.inspect(db, dataset_id).model_dump())
@router.get("/datasets/{dataset_id}/vector/bbox", response_model=dict)
def vector_bbox(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
payload = VectorOperationsService.bbox(db, dataset_id)
return envelope(VectorBBoxResponse(**payload).model_dump())
@router.get("/datasets/{dataset_id}/vector/stats", response_model=dict)
def vector_stats(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
return envelope(VectorOperationsService.stats(db, dataset_id))
@router.post("/datasets/{dataset_id}/vector/clip", status_code=201, response_model=dict)
def clip_vector_dataset(
project_id: UUID,
dataset_id: UUID,
payload: VectorClipRequest,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
job = _run_job_sync(
db=db,
project_id=project_id,
input_dataset_id=dataset_id,
job_type="vector.clip",
parameters=payload.model_dump(),
operation=lambda: VectorOperationsService.clip_by_area(
db,
dataset_id=dataset_id,
area_id=payload.area_id,
output_name=payload.output_name,
),
)
return envelope(job)
@router.post("/datasets/{dataset_id}/vector/buffer", status_code=201, response_model=dict)
def buffer_vector_dataset(
project_id: UUID,
dataset_id: UUID,
payload: VectorBufferRequest,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
job = _run_job_sync(
db=db,
project_id=project_id,
input_dataset_id=dataset_id,
job_type="vector.buffer",
parameters=payload.model_dump(),
operation=lambda: VectorOperationsService.buffer(
db,
dataset_id=dataset_id,
distance_m=payload.distance_m,
dissolve=payload.dissolve,
output_name=payload.output_name,
),
)
return envelope(job)
@router.post("/datasets/{dataset_id}/vector/intersect", status_code=201, response_model=dict)
def intersect_vector_dataset(
project_id: UUID,
dataset_id: UUID,
payload: VectorIntersectRequest,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
job = _run_job_sync(
db=db,
project_id=project_id,
input_dataset_id=dataset_id,
job_type="vector.intersect",
parameters=payload.model_dump(),
operation=lambda: VectorOperationsService.intersect(
db,
source_dataset_id=dataset_id,
target_dataset_id=UUID(payload.other_dataset_id),
output_name=payload.output_name,
),
)
return envelope(job)
@router.get("/datasets/{dataset_id}/vector/summary", response_model=dict)
def vector_dataset_summary(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
return envelope(DatasetService.vector_summary(db, dataset_id))
@router.get("/datasets/{dataset_id}/raster/inspect", response_model=dict)
def raster_dataset_inspect(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
payload = RasterOperationsService.inspect(db, dataset_id)
return envelope(payload)
@router.get("/datasets/{dataset_id}/raster/preview", response_model=dict)
def raster_preview_readiness(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
return envelope(RasterOperationsService.preview(db, dataset_id))
@router.get("/datasets/{dataset_id}/raster/stats", response_model=dict)
def raster_stats(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
payload = RasterOperationsService.stats(db, dataset_id)
return envelope(RasterStatsResponse(**payload).model_dump())
@router.post("/datasets/{dataset_id}/raster/reproject", status_code=201, response_model=dict)
def raster_reproject_dataset(
project_id: UUID,
dataset_id: UUID,
payload: RasterReprojectRequest,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
job = _run_job_sync(
db=db,
project_id=project_id,
input_dataset_id=dataset_id,
job_type="raster.reproject",
parameters=payload.model_dump(),
operation=lambda: RasterOperationsService.reproject(
db,
dataset_id,
target_crs=payload.target_crs,
output_name=payload.output_name,
resampling=payload.resampling,
),
)
return envelope(job)
@router.post("/datasets/{dataset_id}/raster/clip", status_code=201, response_model=dict)
def raster_clip_dataset(
project_id: UUID,
dataset_id: UUID,
payload: RasterClipRequest,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
job = _run_job_sync(
db=db,
project_id=project_id,
input_dataset_id=dataset_id,
job_type="raster.clip",
parameters=payload.model_dump(),
operation=lambda: RasterOperationsService.clip(db, dataset_id, UUID(payload.area_id), payload.output_name),
)
return envelope(job)
@router.post("/datasets/{dataset_id}/raster/tile", status_code=201, response_model=dict)
def raster_tile_dataset(
project_id: UUID,
dataset_id: UUID,
payload: RasterTileRequest,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
job = _run_job_sync(
db=db,
project_id=project_id,
input_dataset_id=dataset_id,
job_type="raster.tile",
parameters=payload.model_dump(),
operation=lambda: RasterOperationsService.tile(
db,
dataset_id,
tile_size=payload.tile_size,
overlap=payload.overlap,
output_name=payload.output_name,
),
)
return envelope(job)
@router.post("/datasets/{dataset_id}/raster/indices/ndvi", status_code=201, response_model=dict)
def raster_ndvi_dataset(
project_id: UUID,
dataset_id: UUID,
payload: RasterNdviRequest,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
job = _run_job_sync(
db=db,
project_id=project_id,
input_dataset_id=dataset_id,
job_type="raster.ndvi",
parameters=payload.model_dump(),
operation=lambda: RasterOperationsService.ndvi(
db,
dataset_id=dataset_id,
nir_band=payload.nir_band,
red_band=payload.red_band,
output_name=payload.output_name,
),
)
return envelope(job)
@router.post("/datasets/{dataset_id}/raster/indices/ndwi", status_code=201, response_model=dict)
def raster_ndwi_dataset(
project_id: UUID,
dataset_id: UUID,
payload: RasterNdwiRequest,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
job = _run_job_sync(
db=db,
project_id=project_id,
input_dataset_id=dataset_id,
job_type="raster.ndwi",
parameters=payload.model_dump(),
operation=lambda: RasterOperationsService.ndwi(
db,
dataset_id=dataset_id,
green_band=payload.green_band,
nir_band=payload.nir_band,
output_name=payload.output_name,
),
)
return envelope(job)
@router.post("/datasets/{dataset_id}/raster/indices/ndbi", status_code=201, response_model=dict)
def raster_ndbi_dataset(
project_id: UUID,
dataset_id: UUID,
payload: RasterNdbiRequest,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
job = _run_job_sync(
db=db,
project_id=project_id,
input_dataset_id=dataset_id,
job_type="raster.ndbi",
parameters=payload.model_dump(),
operation=lambda: RasterOperationsService.ndbi(
db,
dataset_id=dataset_id,
swir_band=payload.swir_band,
nir_band=payload.nir_band,
output_name=payload.output_name,
),
)
return envelope(job)
@router.get("/datasets/{dataset_id}/raster/metadata", response_model=dict)
def raster_dataset_metadata(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
return envelope(RasterOperationsService.metadata(db, dataset_id))
@router.get("/datasets/{dataset_id}/content", response_model=dict)
def dataset_content(
project_id: UUID,
dataset_id: UUID,
db: Session = Depends(get_db),
):
dataset = DatasetService.get_dataset(db, dataset_id)
if dataset.project_id != project_id:
raise HTTPException(status_code=404, detail="Dataset not found")
return envelope(DatasetService.get_dataset_geojson(db, dataset_id))
+17
View File
@@ -0,0 +1,17 @@
from __future__ import annotations
from fastapi import APIRouter, Depends, status
from sqlalchemy.orm import Session
from app.db.session import get_db
from app.schemas.demo import DemoWorkflowResponse
from app.services.demo_workflow_service import DemoWorkflowService
from app.utils.response import envelope
router = APIRouter(prefix="/demo", tags=["demo"])
@router.post("/workflow", status_code=status.HTTP_201_CREATED, response_model=dict)
def seed_demo_workflow(db: Session = Depends(get_db)) -> dict:
result: DemoWorkflowResponse = DemoWorkflowService.seed(db)
return envelope(result.model_dump())
+145
View File
@@ -0,0 +1,145 @@
from __future__ import annotations
from uuid import UUID
from fastapi import APIRouter, Depends
from sqlalchemy.orm import Session
from app.db.session import get_db
from app.schemas import DetectionQaRequest, DetectionRunRequest
from app.services.detection_service import DetectionService
from app.services.model_registry_service import ModelRegistryService
from app.utils.response import envelope
router = APIRouter(prefix="/detection", tags=["detection"])
@router.get("/models", response_model=dict)
def list_detection_models() -> dict:
return envelope({"models": [model.model_dump() for model in ModelRegistryService.list_model_capabilities()]})
@router.post("/run", response_model=dict)
def run_detection(payload: DetectionRunRequest, db: Session = Depends(get_db)) -> dict:
result = DetectionService.run_detection(
db=db,
project_id=payload.project_id,
dataset_id=payload.dataset_id,
model_id=payload.model_id,
confidence_threshold=payload.confidence_threshold,
class_filter=payload.class_filter,
tile_manifest_path=payload.tile_manifest_path,
parameters_json=payload.parameters_json,
)
return envelope(result.model_dump())
@router.get("/runs", response_model=dict)
def list_detection_runs(
project_id: UUID | None = None,
dataset_id: UUID | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(DetectionService.list_runs(db, project_id=project_id, dataset_id=dataset_id).model_dump())
@router.get("/runs/{analysis_run_id}", response_model=dict)
def get_detection_run(analysis_run_id: UUID, db: Session = Depends(get_db)) -> dict:
return envelope(DetectionService.get_run(db, analysis_run_id).model_dump())
@router.get("/runs/{analysis_run_id}/detections", response_model=dict)
def list_detection_run_detections(
analysis_run_id: UUID,
dataset_id: UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(
DetectionService.list_detections(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
).model_dump()
)
@router.get("/datasets/{dataset_id}/detections", response_model=dict)
def list_dataset_detections(
dataset_id: UUID,
analysis_run_id: UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(
DetectionService.list_detections(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
).model_dump()
)
@router.get("/detections/{detection_id}", response_model=dict)
def get_detection(detection_id: UUID, db: Session = Depends(get_db)) -> dict:
return envelope(DetectionService.get_detection(db, detection_id).model_dump())
@router.get("/runs/{analysis_run_id}/geojson", response_model=dict)
def get_detection_run_geojson(
analysis_run_id: UUID,
class_name: str | None = None,
min_confidence: float | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(
DetectionService.detections_to_geojson(
db,
analysis_run_id=analysis_run_id,
class_name=class_name,
min_confidence=min_confidence,
)
)
@router.get("/datasets/{dataset_id}/geojson", response_model=dict)
def get_dataset_detection_geojson(
dataset_id: UUID,
analysis_run_id: UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(
DetectionService.detections_to_geojson(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
)
)
@router.post("/runs/{analysis_run_id}/qa/reference", response_model=dict)
def compare_detection_run_with_reference(
analysis_run_id: UUID,
payload: DetectionQaRequest,
db: Session = Depends(get_db),
) -> dict:
return envelope(
DetectionService.compare_detections_with_reference(
db=db,
analysis_run_id=analysis_run_id,
reference_dataset_id=payload.reference_dataset_id,
iou_threshold=payload.iou_threshold,
class_name=payload.class_name,
min_confidence=payload.min_confidence,
)
)
+66
View File
@@ -0,0 +1,66 @@
from __future__ import annotations
from uuid import UUID
from fastapi import APIRouter, Depends, Query
from fastapi.responses import FileResponse
from sqlalchemy.orm import Session
from app.db.session import get_db
from app.schemas.export import GeoJsonExportRequest, MetadataExportRequest, ReportExportRequest
from app.services.export_service import ExportService
from app.utils.response import envelope
router = APIRouter(prefix="/exports", tags=["exports"])
@router.post("/geojson", response_model=dict)
def export_geojson(payload: GeoJsonExportRequest, db: Session = Depends(get_db)):
if payload.export_kind == "detection_run" and payload.analysis_run_id is not None:
return envelope(
ExportService.export_detection_run_geojson(db, payload.analysis_run_id, payload.name).model_dump(mode="json")
)
if payload.export_kind == "segmentation_run" and payload.analysis_run_id is not None:
return envelope(
ExportService.export_segmentation_run_geojson(db, payload.analysis_run_id, payload.name).model_dump(mode="json")
)
if payload.dataset_id is not None:
return envelope(ExportService.export_dataset_geojson(db, payload.dataset_id, payload.name).model_dump(mode="json"))
return envelope({})
@router.post("/metadata", response_model=dict)
def export_project_metadata(payload: MetadataExportRequest, db: Session = Depends(get_db)):
return envelope(ExportService.export_project_metadata(db, payload.project_id, payload.name).model_dump(mode="json"))
@router.post("/report", response_model=dict)
def export_project_report(payload: ReportExportRequest, db: Session = Depends(get_db)):
return envelope(ExportService.export_project_report(db, payload.project_id, payload.name).model_dump(mode="json"))
@router.get("/projects/{project_id}/exports", response_model=dict)
def list_project_exports(
project_id: UUID,
limit: int = Query(default=50, ge=1, le=100),
offset: int = Query(default=0, ge=0),
db: Session = Depends(get_db),
):
return envelope(ExportService.list_project_exports(db, project_id, limit=limit, offset=offset).model_dump(mode="json"))
@router.get("/{export_id}", response_model=dict)
def get_export(export_id: UUID, db: Session = Depends(get_db)):
return envelope(ExportService.get_export(db, export_id).model_dump(mode="json"))
@router.get("/{export_id}/download")
def download_export(export_id: UUID, db: Session = Depends(get_db)):
path = ExportService.get_export_download_path(db, export_id)
media_type = "text/html" if path.suffix.lower() in {".html", ".htm"} else "application/json"
return FileResponse(path, filename=path.name, media_type=media_type)
@router.get("/{export_id}/content", response_model=dict)
def get_export_content(export_id: UUID, db: Session = Depends(get_db)):
return envelope(ExportService.get_export_content(db, export_id).model_dump(mode="json"))
+122
View File
@@ -0,0 +1,122 @@
from __future__ import annotations
from fastapi import APIRouter, Depends
from sqlalchemy.orm import Session
from app.core.errors import AppError
from app.db.session import get_db
from app.models import Area, Project
from app.providers.registry import fetch_provider_data, get_provider, import_provider_dataset, list_provider_capabilities
from app.schemas import ExternalFetchRequest, ExternalFetchResponse, ProviderImportRequest
from app.utils.response import envelope
router = APIRouter(prefix="/external", tags=["external"])
def _validate_area_in_project(db: Session, project_id, area_id: str | None) -> None:
if area_id is None:
return
area = db.get(Area, area_id)
if not area:
raise AppError(code="AREA_NOT_FOUND", message="Area not found", status_code=404)
if area.project_id != project_id:
raise AppError(code="INVALID_DATASET_SCOPE", message="Area does not belong to this project", status_code=400)
def _assert_project_exists(db: Session, project_id):
project = db.get(Project, project_id)
if not project:
raise AppError(code="PROJECT_NOT_FOUND", message="Project not found", status_code=404)
def _normalize_layer_input(layers: list[str] | None) -> list[str]:
return [layer.strip() for layer in (layers or []) if isinstance(layer, str) and layer.strip()]
def _provider_payload(provider_name: str) -> dict:
return get_provider(provider_name).capability.to_dict()
@router.get("/providers")
def list_external_providers() -> dict:
return envelope({
"providers": [provider.to_dict() for provider in list_provider_capabilities()],
})
@router.get("/providers/capabilities")
def get_external_provider_capabilities() -> dict:
return envelope({
"providers": [provider.to_dict() for provider in list_provider_capabilities()],
})
@router.get("/providers/{provider_name}")
def get_external_provider(provider_name: str) -> dict:
return envelope(_provider_payload(provider_name))
@router.get("/providers/{provider_name}/layers")
def get_external_provider_layers(provider_name: str) -> dict:
provider = get_provider(provider_name)
return envelope({
"provider_name": provider.provider_name,
"layers": provider.supported_layers,
})
@router.get("/providers/{provider_name}/status")
def get_external_provider_status(provider_name: str) -> dict:
provider = get_provider(provider_name)
return envelope({
"provider_name": provider.provider_name,
"configured": provider.is_configured,
"status": provider.capability.status,
"limitation_message": provider.limitation_message,
})
@router.post("/providers/{provider_name}/import")
def import_external_provider_dataset(provider_name: str, payload: ProviderImportRequest) -> dict:
result = import_provider_dataset(
provider_name=provider_name,
project_id=payload.project_id,
area_id=payload.area_id,
layers=_normalize_layer_input(payload.layers),
requested_dataset_role=payload.dataset_role,
)
return envelope(result.model_dump())
def _run_fetch(payload: ExternalFetchRequest, provider_name: str) -> ExternalFetchResponse:
area_id_str = str(payload.area_id) if payload.area_id else None
response = fetch_provider_data(
provider_name=provider_name,
project_id=str(payload.project_id),
area_id=area_id_str,
layers=_normalize_layer_input(payload.layers),
)
return ExternalFetchResponse(
provider=provider_name,
status=response.get("status", "not_configured"),
message=response.get("message", "Provider fetch executed."),
requested_layers=_normalize_layer_input(payload.layers),
project_id=payload.project_id,
area_id=payload.area_id,
)
@router.post("/osm/fetch")
def fetch_osm(payload: ExternalFetchRequest, db: Session = Depends(get_db)) -> dict:
_assert_project_exists(db, payload.project_id)
_validate_area_in_project(db, payload.project_id, payload.area_id)
return envelope(_run_fetch(payload, "osm").model_dump())
@router.post("/grb/fetch")
def fetch_grb(payload: ExternalFetchRequest, db: Session = Depends(get_db)) -> dict:
_assert_project_exists(db, payload.project_id)
_validate_area_in_project(db, payload.project_id, payload.area_id)
return envelope(_run_fetch(payload, "grb").model_dump())
+45
View File
@@ -0,0 +1,45 @@
from __future__ import annotations
from importlib import import_module
from sqlalchemy import text
from fastapi import APIRouter
from app.schemas.health import HealthResponse, SystemCapabilities
from app.providers.registry import list_provider_capabilities
from app.db.session import get_engine
router = APIRouter()
def _dependency_enabled(module: str) -> bool:
try:
import_module(module)
return True
except Exception:
return False
@router.get("/health")
def readiness() -> HealthResponse:
db_status = "ok"
try:
with get_engine().connect() as connection:
connection.execute(text("SELECT 1"))
except Exception:
db_status = "degraded"
return HealthResponse(status="ok", service="geointel-backend", version="0.1.0", database=db_status)
@router.get("/api/v1/system/capabilities")
def capabilities() -> dict:
providers = [item.to_dict() for item in list_provider_capabilities()]
return {"data": SystemCapabilities(
postgis=True,
rasterio=_dependency_enabled("rasterio"),
geopandas=_dependency_enabled("geopandas"),
yolo=False,
sam=False,
grb="planned",
sentinel="planned",
providers=providers,
).model_dump()}
+67
View File
@@ -0,0 +1,67 @@
from __future__ import annotations
from uuid import UUID
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy.orm import Session
from app.db.session import get_db
from app.schemas import JobCreate, JobList, JobRead, JobStatus
from app.services.job_service import JobService
from app.utils.response import envelope
router = APIRouter(prefix="/projects/{project_id}", tags=["jobs"])
@router.post("/jobs", status_code=201, response_model=dict)
def create_job(
project_id: UUID,
payload: JobCreate,
db: Session = Depends(get_db),
):
if payload.project_id != project_id:
raise HTTPException(status_code=400, detail="project_id mismatch")
return envelope(JobService.create_job(db, payload).model_dump())
@router.get("/jobs", response_model=dict)
def list_jobs(
project_id: UUID,
dataset_id: UUID | None = Query(default=None),
limit: int = Query(default=50, ge=1, le=200),
offset: int = Query(default=0, ge=0),
db: Session = Depends(get_db),
):
items, total = JobService.list_jobs(
db,
project_id=project_id,
dataset_id=dataset_id,
limit=limit,
offset=offset,
)
return envelope(JobList(items=items, total=total, limit=limit, offset=offset).model_dump())
@router.get("/jobs/{job_id}", response_model=dict)
def read_job(
project_id: UUID,
job_id: UUID,
db: Session = Depends(get_db),
):
job = JobService.get_job(db, job_id)
if job.project_id != project_id:
raise HTTPException(status_code=404, detail="Job not found")
return envelope(job.model_dump())
@router.get("/jobs/{job_id}/status", response_model=dict)
def read_job_status(
project_id: UUID,
job_id: UUID,
db: Session = Depends(get_db),
):
status_row = JobService.get_job_status(db, job_id)
if status_row["project_id"] != str(project_id):
raise HTTPException(status_code=404, detail="Job not found")
return envelope(JobStatus(**status_row).model_dump())
+50
View File
@@ -0,0 +1,50 @@
from __future__ import annotations
from uuid import UUID
from fastapi import APIRouter, Depends, HTTPException, Query, status
from sqlalchemy.orm import Session
from app.db.session import get_db
from app.schemas.project import ProjectCreate, ProjectRead, ProjectUpdate
from app.services.project_service import ProjectService
from app.utils.response import envelope
router = APIRouter(prefix="/projects", tags=["projects"])
@router.get("", response_model=dict)
def list_projects(
limit: int = Query(default=50, ge=1, le=200),
offset: int = Query(default=0, ge=0),
db: Session = Depends(get_db),
):
projects, total = ProjectService.list_projects(db, limit=limit, offset=offset)
return envelope({"items": [ProjectRead.model_validate(item).model_dump() for item in projects], "total": total, "limit": limit, "offset": offset})
@router.post("", status_code=status.HTTP_201_CREATED, response_model=dict)
def create_project(payload: ProjectCreate, db: Session = Depends(get_db)):
project = ProjectService.create_project(db, payload)
return envelope(ProjectRead.model_validate(project).model_dump())
@router.get("/{project_id}", response_model=dict)
def get_project(project_id: UUID, db: Session = Depends(get_db)):
project = ProjectService.get_project(db, project_id)
if not project:
raise HTTPException(status_code=404, detail="Project not found")
return envelope(ProjectRead.model_validate(project).model_dump())
@router.patch("/{project_id}", response_model=dict)
def update_project(project_id: UUID, payload: ProjectUpdate, db: Session = Depends(get_db)):
project = ProjectService.update_project(db, project_id, payload)
return envelope(ProjectRead.model_validate(project).model_dump())
@router.delete("/{project_id}", status_code=status.HTTP_200_OK, response_model=dict)
def delete_project(project_id: UUID, db: Session = Depends(get_db)):
if not ProjectService.delete_project(db, project_id):
raise HTTPException(status_code=404, detail="Project not found")
return envelope({"deleted": True})
+80
View File
@@ -0,0 +1,80 @@
from __future__ import annotations
import uuid
from fastapi import APIRouter, Depends
from sqlalchemy.orm import Session
from app.db.session import get_db
from app.core.errors import AppError
from app.models import Dataset, Job
from app.schemas import QaProviderComparisonRequest
from app.services.qa_service import QaService
from app.services.job_service import JobService
from app.services.quality_service import QualityService
from app.utils.response import envelope
router = APIRouter(prefix="/qa", tags=["qa"])
@router.post("/detections-vs-reference")
def compare_candidate_with_reference(
payload: QaProviderComparisonRequest,
db: Session = Depends(get_db),
) -> dict:
candidate_dataset = db.get(Dataset, payload.candidate_dataset_id)
if not candidate_dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Candidate dataset not found", status_code=404)
job = JobService.run_sync_job(
db=db,
project_id=candidate_dataset.project_id,
job_type="qa.compare-candidate-with-reference",
parameters=payload.model_dump(mode="json"),
input_dataset_id=candidate_dataset.id,
operation=lambda: QaService.compare_candidate_with_reference(
db=db,
project_id=candidate_dataset.project_id,
candidate_dataset_id=payload.candidate_dataset_id,
reference_dataset_id=payload.reference_dataset_id,
iou_threshold=payload.iou_threshold,
area_id=payload.area_id,
).model_dump(mode="json"),
)
result_json = job.get("result_json") if isinstance(job, dict) else None
if isinstance(result_json, dict) and job.get("status") == "success":
quality_check = QualityService.persist_quality_check(
db=db,
project_id=candidate_dataset.project_id,
job_id=uuid.UUID(str(job["id"])),
candidate_dataset_id=payload.candidate_dataset_id,
reference_dataset_id=payload.reference_dataset_id,
check_type="candidate_vs_reference",
status=str(result_json.get("status", "ok")),
score=result_json.get("f1_score"),
parameters=payload.model_dump(mode="json"),
findings={
"matches": result_json.get("matches"),
"false_positives": result_json.get("false_positives"),
"false_negatives": result_json.get("false_negatives"),
"warnings": result_json.get("warnings", []),
"unsupported_geometry": result_json.get("unsupported_geometry", False),
"unsupported_geometries": result_json.get("unsupported_geometries", []),
},
metrics={
"precision": result_json.get("precision"),
"recall": result_json.get("recall"),
"f1": result_json.get("f1_score"),
"mean_iou": result_json.get("mean_iou"),
"false_positive_count": result_json.get("false_positives"),
"false_negative_count": result_json.get("false_negatives"),
},
)
result_json["quality_check_id"] = str(quality_check.id)
job_record = db.get(Job, uuid.UUID(str(job["id"])))
if job_record:
job_record.result_json = result_json
db.add(job_record)
db.commit()
return envelope(job)
+29
View File
@@ -0,0 +1,29 @@
from __future__ import annotations
from uuid import UUID
from fastapi import APIRouter, Depends, Query
from sqlalchemy.orm import Session
from app.db.session import get_db
from app.schemas.qa import QualityCheckList
from app.services.quality_check_service import QualityCheckService
from app.utils.response import envelope
router = APIRouter(prefix="/projects/{project_id}", tags=["quality-checks"])
@router.get("/quality-checks", response_model=dict)
def list_quality_checks(
project_id: UUID,
limit: int = Query(default=50, ge=1, le=200),
offset: int = Query(default=0, ge=0),
db: Session = Depends(get_db),
) -> dict:
items, total = QualityCheckService.list_quality_checks(
db,
project_id=project_id,
limit=limit,
offset=offset,
)
return envelope(QualityCheckList(items=items, total=total, limit=limit, offset=offset).model_dump())
+145
View File
@@ -0,0 +1,145 @@
from __future__ import annotations
from uuid import UUID
from fastapi import APIRouter, Depends
from sqlalchemy.orm import Session
from app.db.session import get_db
from app.schemas import SegmentationQaRequest, SegmentationRunRequest
from app.services.model_registry_service import ModelRegistryService
from app.services.segmentation_service import SegmentationService
from app.utils.response import envelope
router = APIRouter(prefix="/segmentation", tags=["segmentation"])
@router.get("/models", response_model=dict)
def list_segmentation_models() -> dict:
return envelope({"models": [model.model_dump() for model in ModelRegistryService.list_model_capabilities(task_type="segmentation")]})
@router.post("/run", response_model=dict)
def run_segmentation(payload: SegmentationRunRequest, db: Session = Depends(get_db)) -> dict:
result = SegmentationService.run_segmentation(
db=db,
project_id=payload.project_id,
dataset_id=payload.dataset_id,
model_id=payload.model_id,
confidence_threshold=payload.confidence_threshold,
class_filter=payload.class_filter,
tile_manifest_path=payload.tile_manifest_path,
parameters_json=payload.parameters_json,
)
return envelope(result.model_dump())
@router.get("/runs", response_model=dict)
def list_segmentation_runs(
project_id: UUID | None = None,
dataset_id: UUID | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(SegmentationService.list_runs(db, project_id=project_id, dataset_id=dataset_id).model_dump())
@router.get("/runs/{analysis_run_id}", response_model=dict)
def get_segmentation_run(analysis_run_id: UUID, db: Session = Depends(get_db)) -> dict:
return envelope(SegmentationService.get_run(db, analysis_run_id).model_dump())
@router.get("/runs/{analysis_run_id}/segmentations", response_model=dict)
def list_segmentation_run_outputs(
analysis_run_id: UUID,
dataset_id: UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(
SegmentationService.list_segmentations(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
).model_dump()
)
@router.get("/datasets/{dataset_id}/segmentations", response_model=dict)
def list_dataset_segmentations(
dataset_id: UUID,
analysis_run_id: UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(
SegmentationService.list_segmentations(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
).model_dump()
)
@router.get("/segmentations/{segmentation_id}", response_model=dict)
def get_segmentation(segmentation_id: UUID, db: Session = Depends(get_db)) -> dict:
return envelope(SegmentationService.get_segmentation(db, segmentation_id).model_dump())
@router.get("/runs/{analysis_run_id}/geojson", response_model=dict)
def get_segmentation_run_geojson(
analysis_run_id: UUID,
class_name: str | None = None,
min_confidence: float | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(
SegmentationService.segmentations_to_geojson(
db,
analysis_run_id=analysis_run_id,
class_name=class_name,
min_confidence=min_confidence,
)
)
@router.get("/datasets/{dataset_id}/geojson", response_model=dict)
def get_dataset_segmentation_geojson(
dataset_id: UUID,
analysis_run_id: UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
db: Session = Depends(get_db),
) -> dict:
return envelope(
SegmentationService.segmentations_to_geojson(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
)
)
@router.post("/runs/{analysis_run_id}/qa/reference", response_model=dict)
def compare_segmentation_run_with_reference(
analysis_run_id: UUID,
payload: SegmentationQaRequest,
db: Session = Depends(get_db),
) -> dict:
return envelope(
SegmentationService.compare_segmentations_with_reference(
db=db,
analysis_run_id=analysis_run_id,
reference_dataset_id=payload.reference_dataset_id,
iou_threshold=payload.iou_threshold,
class_name=payload.class_name,
min_confidence=payload.min_confidence,
)
)
View File
+52
View File
@@ -0,0 +1,52 @@
from pydantic import Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
populate_by_name=True,
)
app_env: str = Field(default="development", validation_alias="GEOINTEL_ENV")
app_version: str = Field(default="0.1.0")
api_prefix: str = Field(default="/api/v1", validation_alias="GEOINTEL_API_PREFIX")
database_url: str = Field(
default="postgresql+psycopg://geointel:geointel@localhost:5432/geointel?connect_timeout=1",
validation_alias="DATABASE_URL",
)
storage_root: str = Field(default="./storage", validation_alias="STORAGE_ROOT")
max_upload_mb: int = Field(default=500, validation_alias="MAX_UPLOAD_MB")
redis_url: str | None = Field(default=None, validation_alias="REDIS_URL")
log_level: str = Field(default="INFO", validation_alias="GEOINTEL_LOG_LEVEL")
database_statement_timeout_ms: int = Field(default=5_000, validation_alias="DATABASE_STATEMENT_TIMEOUT_MS")
yolo_enabled: bool = Field(default=False, validation_alias="YOLO_ENABLED")
yolo_model_path: str | None = Field(default=None, validation_alias="YOLO_MODEL_PATH")
yolo_model_id: str = Field(default="yolo-configured", validation_alias="YOLO_MODEL_ID")
yolo_model_display_name: str = Field(default="Configured YOLO detector", validation_alias="YOLO_MODEL_DISPLAY_NAME")
yolo_model_version: str | None = Field(default=None, validation_alias="YOLO_MODEL_VERSION")
yolo_device: str = Field(default="cpu", validation_alias="YOLO_DEVICE")
yolo_image_size: int = Field(default=640, validation_alias="YOLO_IMAGE_SIZE")
yolo_max_tiles: int = Field(default=100, validation_alias="YOLO_MAX_TILES")
yolo_batch_size: int = Field(default=1, validation_alias="YOLO_BATCH_SIZE")
cors_origins: list[str] | str = Field(
default=["http://localhost:5173", "http://127.0.0.1:5173"],
validation_alias="CORS_ORIGINS",
)
@field_validator("cors_origins", mode="before")
@classmethod
def parse_cors_origins(cls, value: object) -> list[str]:
if isinstance(value, str):
return [item.strip() for item in value.split(",") if item.strip()]
if isinstance(value, list):
return value
if value is None:
return ["http://localhost:5173", "http://127.0.0.1:5173"]
return [str(value)]
def get_settings() -> Settings:
return Settings()
+15
View File
@@ -0,0 +1,15 @@
class AppError(Exception):
"""Domain error used by services to return canonical API errors."""
def __init__(
self,
code: str,
message: str,
details: dict | list | None = None,
status_code: int = 400,
) -> None:
super().__init__(message)
self.code = code
self.message = message
self.details = details or {}
self.status_code = status_code
+12
View File
@@ -0,0 +1,12 @@
import logging
import sys
def configure_logging(level: str = "INFO") -> None:
logging.basicConfig(
level=level,
format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
stream=sys.stdout,
)
for name in ["uvicorn", "uvicorn.error", "uvicorn.access", "sqlalchemy.engine"]:
logging.getLogger(name).setLevel(level)
View File
+4
View File
@@ -0,0 +1,4 @@
from .base import Base
from .session import get_db, get_engine
__all__ = ["Base", "get_db", "get_engine"]
+5
View File
@@ -0,0 +1,5 @@
from sqlalchemy.orm import DeclarativeBase
class Base(DeclarativeBase):
pass
+20
View File
@@ -0,0 +1,20 @@
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, Session
from app.core.config import get_settings
engine = create_engine(get_settings().database_url, pool_pre_ping=True, future=True)
SessionLocal = sessionmaker(bind=engine, autocommit=False, autoflush=False, future=True)
def get_db():
db: Session = SessionLocal()
try:
yield db
finally:
db.close()
def get_engine():
return engine
View File
+99
View File
@@ -0,0 +1,99 @@
from __future__ import annotations
from fastapi import FastAPI, HTTPException, Request
from fastapi.exceptions import RequestValidationError
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from app.api.routes import areas, datasets, demo, detection, exports, external, health, jobs, projects, qa, quality_checks, segmentation
from app.core.config import get_settings
from app.core.errors import AppError
from app.core.logging import configure_logging
def _to_error_payload(code: str, message: str, details: dict | list | None = None) -> dict:
return {
"error": {
"code": code,
"message": message,
"details": details or {},
},
}
def create_app() -> FastAPI:
settings = get_settings()
configure_logging(settings.log_level)
app = FastAPI(
title="GeoIntel Kempen",
version=settings.app_version,
docs_url="/docs",
redoc_url="/redoc",
)
app.add_middleware(
CORSMiddleware,
allow_origins=settings.cors_origins,
allow_methods=["*"],
allow_headers=["*"],
allow_credentials=True,
)
app.include_router(health.router)
app.include_router(projects.router, prefix=settings.api_prefix)
app.include_router(areas.router, prefix=settings.api_prefix)
app.include_router(datasets.router, prefix=settings.api_prefix)
app.include_router(jobs.router, prefix=settings.api_prefix)
app.include_router(quality_checks.router, prefix=settings.api_prefix)
app.include_router(exports.router, prefix=settings.api_prefix)
app.include_router(external.router, prefix=settings.api_prefix)
app.include_router(demo.router, prefix=settings.api_prefix)
app.include_router(qa.router, prefix=settings.api_prefix)
app.include_router(detection.router, prefix=settings.api_prefix)
app.include_router(segmentation.router, prefix=settings.api_prefix)
@app.exception_handler(AppError)
async def app_error(request: Request, exc: AppError): # noqa: ARG001
return JSONResponse(
status_code=exc.status_code,
content=_to_error_payload(exc.code, exc.message, exc.details),
)
@app.exception_handler(HTTPException)
async def http_error(request: Request, exc: HTTPException): # noqa: ARG001
return JSONResponse(
status_code=exc.status_code,
content=_to_error_payload("HTTP_ERROR", str(exc.detail), {}),
)
@app.exception_handler(RequestValidationError)
async def validation_error(request: Request, exc: RequestValidationError): # noqa: ARG001
return JSONResponse(
status_code=422,
content=_to_error_payload("VALIDATION_ERROR", "Validation failed", exc.errors()),
)
@app.exception_handler(Exception)
async def unexpected_error(request: Request, exc: Exception): # noqa: ARG001
return JSONResponse(
status_code=500,
content=_to_error_payload("INTERNAL_ERROR", "Unexpected server error", {"type": exc.__class__.__name__}),
)
return app
app = create_app()
def main() -> None:
import uvicorn
settings = get_settings()
uvicorn.run(
"app.main:app",
host="0.0.0.0",
port=8000,
reload=settings.app_env == "development",
)
+1
View File
@@ -0,0 +1 @@
from app.models import *
View File
+16
View File
@@ -0,0 +1,16 @@
from .entities import AnalysisRun, Area, Dataset, DatasetVersion, Detection, Export, Job, Metric, Project, QualityCheck, Segmentation, VectorFeature
__all__ = [
"AnalysisRun",
"Area",
"Dataset",
"DatasetVersion",
"Detection",
"Export",
"Job",
"Metric",
"Project",
"QualityCheck",
"Segmentation",
"VectorFeature",
]
+269
View File
@@ -0,0 +1,269 @@
from __future__ import annotations
import uuid
from datetime import datetime
from geoalchemy2 import Geometry
from sqlalchemy import DateTime, ForeignKey, Float, Index, JSON, String, Text, func
from sqlalchemy.sql.sqltypes import Integer
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db.base import Base
class Project(Base):
__tablename__ = "projects"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
name: Mapped[str] = mapped_column(String(255), nullable=False)
description: Mapped[str | None] = mapped_column(Text, nullable=True)
region: Mapped[str] = mapped_column(String(120), default="Kempen")
status: Mapped[str] = mapped_column(String(32), default="active")
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
areas: Mapped[list["Area"]] = relationship("Area", back_populates="project", cascade="all, delete-orphan")
datasets: Mapped[list["Dataset"]] = relationship("Dataset", back_populates="project", cascade="all, delete-orphan")
class Area(Base):
__tablename__ = "areas"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
project_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("projects.id", ondelete="CASCADE"), nullable=False)
name: Mapped[str] = mapped_column(String(255), nullable=False)
geometry: Mapped[str] = mapped_column(Geometry("MultiPolygon", srid=4326), nullable=False)
original_crs: Mapped[str | None] = mapped_column(String(64), nullable=True)
area_m2: Mapped[float | None] = mapped_column(Float, nullable=True)
bbox: Mapped[str | None] = mapped_column(Geometry("Polygon", srid=4326), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
project: Mapped[Project] = relationship("Project", back_populates="areas")
class Dataset(Base):
__tablename__ = "datasets"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
project_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("projects.id", ondelete="CASCADE"), nullable=False)
area_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("areas.id", ondelete="SET NULL"), nullable=True)
name: Mapped[str] = mapped_column(String(255), nullable=False)
dataset_type: Mapped[str] = mapped_column(String(64), nullable=False)
source: Mapped[str] = mapped_column(String(120), nullable=False)
storage_path: Mapped[str | None] = mapped_column(String(500), nullable=True)
original_filename: Mapped[str | None] = mapped_column(String(255), nullable=True)
stored_filename: Mapped[str | None] = mapped_column(String(255), nullable=True)
content_type: Mapped[str | None] = mapped_column(String(120), nullable=True)
size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
checksum_sha256: Mapped[str | None] = mapped_column(String(64), nullable=True)
derived_from_dataset_id: Mapped[uuid.UUID | None] = mapped_column(
UUID(as_uuid=True),
ForeignKey("datasets.id", ondelete="SET NULL"),
nullable=True,
)
crs: Mapped[str | None] = mapped_column(String(64), nullable=True)
bounds_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
resolution_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
bands_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
metadata_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
dataset_role: Mapped[str] = mapped_column(String(32), nullable=False, default="source", server_default="source")
source_name: Mapped[str | None] = mapped_column(String(120), nullable=True)
reference_layer_name: Mapped[str | None] = mapped_column(String(120), nullable=True)
source_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
provenance_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
imported_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
status: Mapped[str] = mapped_column(String(32), default="uploaded")
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
project: Mapped[Project] = relationship("Project", back_populates="datasets")
versions: Mapped[list["DatasetVersion"]] = relationship(
"DatasetVersion",
back_populates="dataset",
cascade="all, delete-orphan",
)
vector_features: Mapped[list["VectorFeature"]] = relationship(
"VectorFeature",
back_populates="dataset",
cascade="all, delete-orphan",
)
class DatasetVersion(Base):
__tablename__ = "dataset_versions"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
dataset_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="CASCADE"), nullable=False)
version: Mapped[int] = mapped_column(Integer, default=1)
storage_path: Mapped[str | None] = mapped_column(String(500), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
dataset: Mapped[Dataset] = relationship("Dataset", back_populates="versions")
class VectorFeature(Base):
__tablename__ = "vector_features"
__table_args__ = (
Index("ix_vector_features_dataset_id", "dataset_id"),
Index("ix_vector_features_geometry", "geometry", postgresql_using="gist"),
)
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
dataset_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="CASCADE"), nullable=False)
feature_class: Mapped[str | None] = mapped_column(String(120), nullable=True)
source_feature_id: Mapped[str | None] = mapped_column(String(255), nullable=True)
properties_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
geometry: Mapped[str] = mapped_column(Geometry("Geometry", srid=4326, spatial_index=False), nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
dataset: Mapped[Dataset] = relationship("Dataset", back_populates="vector_features")
class AnalysisRun(Base):
__tablename__ = "analysis_runs"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
project_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("projects.id", ondelete="CASCADE"), nullable=False)
area_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("areas.id", ondelete="SET NULL"), nullable=True)
dataset_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True)
job_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("jobs.id", ondelete="SET NULL"), nullable=True)
analysis_type: Mapped[str] = mapped_column(String(64), nullable=False)
status: Mapped[str] = mapped_column(String(32), nullable=False)
model_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
model_version: Mapped[str | None] = mapped_column(String(120), nullable=True)
parameters_json: Mapped[dict] = mapped_column(JSON, nullable=False)
result_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
error_message: Mapped[str | None] = mapped_column(Text, nullable=True)
class Detection(Base):
__tablename__ = "detections"
__table_args__ = (
Index("ix_detections_project_id", "project_id"),
Index("ix_detections_dataset_id", "dataset_id"),
Index("ix_detections_analysis_run_id", "analysis_run_id"),
Index("ix_detections_class_name", "class_name"),
Index("ix_detections_geometry", "geometry", postgresql_using="gist"),
)
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
project_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("projects.id", ondelete="CASCADE"), nullable=False)
dataset_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True)
analysis_run_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True)
job_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("jobs.id", ondelete="SET NULL"), nullable=True)
model_name: Mapped[str] = mapped_column(String(255), nullable=False)
model_version: Mapped[str | None] = mapped_column(String(120), nullable=True)
class_name: Mapped[str] = mapped_column(String(120), nullable=False)
confidence: Mapped[float] = mapped_column(Float, nullable=False)
geometry: Mapped[str] = mapped_column(Geometry("Geometry", srid=4326, spatial_index=False), nullable=False)
bbox_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
source_tile_path: Mapped[str | None] = mapped_column(String(500), nullable=True)
properties_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
class Segmentation(Base):
__tablename__ = "segmentations"
__table_args__ = (
Index("ix_segmentations_project_id", "project_id"),
Index("ix_segmentations_dataset_id", "dataset_id"),
Index("ix_segmentations_analysis_run_id", "analysis_run_id"),
Index("ix_segmentations_job_id", "job_id"),
Index("ix_segmentations_class_name", "class_name"),
Index("ix_segmentations_geometry", "geometry", postgresql_using="gist"),
)
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
project_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("projects.id", ondelete="CASCADE"), nullable=False)
dataset_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True)
job_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("jobs.id", ondelete="SET NULL"), nullable=True)
analysis_run_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True)
model_name: Mapped[str] = mapped_column(String(255), nullable=False)
model_version: Mapped[str | None] = mapped_column(String(120), nullable=True)
class_name: Mapped[str] = mapped_column(String(120), nullable=False)
confidence: Mapped[float | None] = mapped_column(Float, nullable=True)
geometry: Mapped[str] = mapped_column(Geometry("MultiPolygon", srid=4326, spatial_index=False), nullable=False)
bbox_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
area_m2: Mapped[float | None] = mapped_column(Float, nullable=True)
mask_path: Mapped[str | None] = mapped_column(Text, nullable=True)
source_tile_path: Mapped[str | None] = mapped_column(String(500), nullable=True)
tile_index: Mapped[int | None] = mapped_column(Integer, nullable=True)
properties_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
provenance_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
class QualityCheck(Base):
__tablename__ = "quality_checks"
__table_args__ = (
Index("ix_quality_checks_project_id", "project_id"),
Index("ix_quality_checks_reference_dataset_id", "reference_dataset_id"),
Index("ix_quality_checks_candidate_dataset_id", "candidate_dataset_id"),
Index("ix_quality_checks_analysis_run_id", "analysis_run_id"),
)
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
project_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("projects.id", ondelete="CASCADE"), nullable=False)
job_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("jobs.id", ondelete="SET NULL"), nullable=True)
analysis_run_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True)
candidate_dataset_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True)
reference_dataset_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="CASCADE"), nullable=False)
check_type: Mapped[str] = mapped_column(String(120), nullable=False)
status: Mapped[str] = mapped_column(String(32), nullable=False)
score: Mapped[float | None] = mapped_column(Float, nullable=True)
parameters_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
findings_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
class Metric(Base):
__tablename__ = "metrics"
__table_args__ = (
Index("ix_metrics_quality_check_id", "quality_check_id"),
Index("ix_metrics_analysis_run_id", "analysis_run_id"),
)
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
quality_check_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("quality_checks.id", ondelete="CASCADE"), nullable=True)
analysis_run_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True)
metric_key: Mapped[str] = mapped_column(String(120), nullable=False)
metric_value: Mapped[float | None] = mapped_column(Float, nullable=True)
metric_unit: Mapped[str | None] = mapped_column(String(64), nullable=True)
label: Mapped[str | None] = mapped_column(String(120), nullable=True)
metadata_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
class Export(Base):
__tablename__ = "exports"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
project_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("projects.id", ondelete="CASCADE"), nullable=False)
analysis_run_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("analysis_runs.id", ondelete="SET NULL"), nullable=True)
export_type: Mapped[str] = mapped_column(String(64), nullable=False)
storage_path: Mapped[str] = mapped_column(String(500), nullable=False)
metadata_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
class Job(Base):
__tablename__ = "jobs"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
job_type: Mapped[str] = mapped_column(String(128), nullable=False)
status: Mapped[str] = mapped_column(String(32), nullable=False, default="queued")
project_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("projects.id", ondelete="CASCADE"), nullable=False)
dataset_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True)
input_dataset_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True)
output_dataset_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("datasets.id", ondelete="SET NULL"), nullable=True)
parameters_json: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
result_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
error_message: Mapped[str | None] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
View File
+5
View File
@@ -0,0 +1,5 @@
from __future__ import annotations
from app.providers import base, fixture, grb, manual, osm, registry
__all__ = ["base", "fixture", "grb", "manual", "osm", "registry"]
+96
View File
@@ -0,0 +1,96 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
@dataclass(frozen=True)
class ProviderCapability:
provider_name: str
display_name: str
authority_level: str
supported_layers: list[str]
supported_geometry_types: list[str]
supported_query_modes: list[str]
fetch_signature: str
configured: bool
status: str
limitation_message: str
attribution: str
license_note: str
not_configured_reason: str | None = None
def to_dict(self) -> dict[str, Any]:
return {
"provider_name": self.provider_name,
"display_name": self.display_name,
"authority_level": self.authority_level,
"supported_layers": self.supported_layers,
"supported_geometry_types": self.supported_geometry_types,
"supported_query_modes": self.supported_query_modes,
"fetch_signature": self.fetch_signature,
"configured": self.configured,
"status": self.status,
"limitation_message": self.limitation_message,
"attribution": self.attribution,
"license_note": self.license_note,
"not_configured_reason": self.not_configured_reason,
}
class BaseReferenceProvider:
def __init__(
self,
provider_name: str,
display_name: str,
authority_level: str,
supported_layers: list[str],
supported_geometry_types: list[str],
supported_query_modes: list[str],
fetch_signature: str,
limitation_message: str,
attribution: str,
license_note: str,
configured: bool = False,
) -> None:
self.provider_name = provider_name
self.display_name = display_name
self.authority_level = authority_level
self.supported_layers = supported_layers
self.supported_geometry_types = supported_geometry_types
self.supported_query_modes = supported_query_modes
self.fetch_signature = fetch_signature
self.limitation_message = limitation_message
self.attribution = attribution
self.license_note = license_note
self._configured = configured
@property
def capability(self) -> ProviderCapability:
return ProviderCapability(
provider_name=self.provider_name,
display_name=self.display_name,
authority_level=self.authority_level,
supported_layers=self.supported_layers,
supported_geometry_types=self.supported_geometry_types,
supported_query_modes=self.supported_query_modes,
fetch_signature=self.fetch_signature,
configured=self.is_configured,
status="configured" if self.is_configured else "not_configured",
limitation_message=self.limitation_message,
attribution=self.attribution,
license_note=self.license_note,
not_configured_reason=None if self.is_configured else "Provider integration is not configured yet",
)
@property
def is_configured(self) -> bool:
return self._configured
def fetch(self, project_id: str, area_id: str | None, layers: list[str]) -> dict[str, Any]:
del project_id, area_id, layers
return {
"provider": self.provider_name,
"status": "not_configured",
"message": "Provider integration is not configured yet",
}
+20
View File
@@ -0,0 +1,20 @@
from __future__ import annotations
from app.providers.base import BaseReferenceProvider
class FixtureProvider(BaseReferenceProvider):
def __init__(self) -> None:
super().__init__(
provider_name="fixture",
display_name="Fixture data",
authority_level="fixture",
supported_layers=["buildings", "roads", "water", "landuse", "custom"],
supported_geometry_types=["Point", "MultiPoint", "LineString", "MultiLineString", "Polygon", "MultiPolygon"],
supported_query_modes=["fixture"],
fetch_signature="tests/fixtures and demo fixture upload flow",
limitation_message="Fixture provider represents local test/demo fixtures only.",
attribution="GeoIntel local fixtures",
license_note="Fixtures are for local development and tests; do not present them as official data.",
configured=True,
)
+20
View File
@@ -0,0 +1,20 @@
from __future__ import annotations
from app.providers.base import BaseReferenceProvider
class GRBProvider(BaseReferenceProvider):
def __init__(self) -> None:
super().__init__(
provider_name="grb",
display_name="GRB",
authority_level="authoritative",
supported_layers=["buildings", "roads", "parcels"],
supported_geometry_types=["Polygon", "MultiPolygon", "LineString", "MultiLineString"],
supported_query_modes=["area"],
fetch_signature="POST /api/v1/external/grb/fetch",
limitation_message="GRB live WFS/download integration is not configured in Sprint 7B.",
attribution="Digitaal Vlaanderen - Basiskaart Vlaanderen (GRB)",
license_note="Use must follow Digitaal Vlaanderen open data and attribution terms.",
configured=False,
)
+20
View File
@@ -0,0 +1,20 @@
from __future__ import annotations
from app.providers.base import BaseReferenceProvider
class ManualProvider(BaseReferenceProvider):
def __init__(self) -> None:
super().__init__(
provider_name="manual",
display_name="Manual upload",
authority_level="manual",
supported_layers=["buildings", "roads", "water", "landuse", "custom"],
supported_geometry_types=["Point", "MultiPoint", "LineString", "MultiLineString", "Polygon", "MultiPolygon"],
supported_query_modes=["upload"],
fetch_signature="POST /api/v1/projects/{project_id}/datasets/upload",
limitation_message="Manual provider data is supplied through the existing dataset upload flow.",
attribution="User supplied",
license_note="License and attribution must be supplied by the uploader in source metadata.",
configured=True,
)
+20
View File
@@ -0,0 +1,20 @@
from __future__ import annotations
from app.providers.base import BaseReferenceProvider
class OSMProvider(BaseReferenceProvider):
def __init__(self) -> None:
super().__init__(
provider_name="osm",
display_name="OpenStreetMap",
authority_level="contextual",
supported_layers=["buildings", "roads", "water", "landuse"],
supported_geometry_types=["Polygon", "MultiPolygon", "LineString", "MultiLineString"],
supported_query_modes=["area"],
fetch_signature="POST /api/v1/external/osm/fetch",
limitation_message="OSM live Overpass/download integration is not configured in Sprint 7B.",
attribution="OpenStreetMap contributors",
license_note="OpenStreetMap data is available under ODbL; attribution is required.",
configured=False,
)
+141
View File
@@ -0,0 +1,141 @@
from __future__ import annotations
from pydantic import BaseModel
from app.core.errors import AppError
from app.providers.base import ProviderCapability
from app.providers.fixture import FixtureProvider
from app.providers.grb import GRBProvider
from app.providers.manual import ManualProvider
from app.providers.osm import OSMProvider
class ProviderDatasetMapping(BaseModel):
provider_name: str
dataset_role: str
source_name: str
reference_required: bool
write_path: str = "DatasetService"
class ProviderImportResult(BaseModel):
provider_name: str
status: str
message: str
requested_layers: list[str]
dataset_id: str | None = None
dataset_role: str | None = None
source_name: str | None = None
class ExternalProviderRegistry:
def __init__(self) -> None:
self.providers = {
"grb": GRBProvider(),
"osm": OSMProvider(),
"manual": ManualProvider(),
"fixture": FixtureProvider(),
}
def list_capabilities(self) -> list[ProviderCapability]:
return [provider.capability for provider in self.providers.values()]
def get(self, provider_name: str):
normalized = provider_name.strip().lower()
if normalized not in self.providers:
raise AppError(code="PROVIDER_NOT_FOUND", message="Provider not found", status_code=404)
return self.providers[normalized]
def fetch(self, provider_name: str, project_id: str, area_id: str | None, layers: list[str]) -> dict:
provider = self.get(provider_name)
return provider.fetch(project_id=project_id, area_id=area_id, layers=layers)
def dataset_mapping(self, provider_name: str, requested_dataset_role: str | None = None) -> ProviderDatasetMapping:
provider = self.get(provider_name)
if provider.provider_name == "osm":
dataset_role = "reference" if requested_dataset_role == "reference" else "source"
return ProviderDatasetMapping(
provider_name="osm",
dataset_role=dataset_role,
source_name="osm",
reference_required=requested_dataset_role == "reference",
)
return ProviderDatasetMapping(
provider_name=provider.provider_name,
dataset_role="reference",
source_name=provider.provider_name,
reference_required=True,
)
def import_contract(
self,
provider_name: str,
project_id: str,
area_id: str | None,
layers: list[str],
requested_dataset_role: str | None = None,
) -> ProviderImportResult:
del project_id, area_id
provider = self.get(provider_name)
mapping = self.dataset_mapping(provider.provider_name, requested_dataset_role=requested_dataset_role)
if provider.provider_name in {"grb", "osm"}:
return ProviderImportResult(
provider_name=provider.provider_name,
status="not_configured",
message=f"No live {provider.display_name} import is configured in Sprint 7B.",
requested_layers=layers,
dataset_role=mapping.dataset_role,
source_name=mapping.source_name,
)
if provider.provider_name == "manual":
return ProviderImportResult(
provider_name="manual",
status="upload_flow_required",
message="Manual provider data must use the existing dataset upload/reference flow.",
requested_layers=layers,
dataset_role=mapping.dataset_role,
source_name=mapping.source_name,
)
return ProviderImportResult(
provider_name="fixture",
status="fixture_flow_required",
message="Fixture provider data must use checked-in demo/test fixture flows.",
requested_layers=layers,
dataset_role=mapping.dataset_role,
source_name=mapping.source_name,
)
_registry = ExternalProviderRegistry()
def list_provider_capabilities() -> list[ProviderCapability]:
return _registry.list_capabilities()
def get_provider(provider_name: str):
return _registry.get(provider_name)
def fetch_provider_data(provider_name: str, project_id: str, area_id: str | None, layers: list[str]) -> dict:
return _registry.fetch(provider_name, project_id, area_id, layers)
def get_provider_dataset_mapping(provider_name: str, requested_dataset_role: str | None = None) -> ProviderDatasetMapping:
return _registry.dataset_mapping(provider_name, requested_dataset_role=requested_dataset_role)
def import_provider_dataset(
provider_name: str,
project_id: str,
area_id: str | None,
layers: list[str],
requested_dataset_role: str | None = None,
) -> ProviderImportResult:
return _registry.import_contract(
provider_name=provider_name,
project_id=project_id,
area_id=area_id,
layers=layers,
requested_dataset_role=requested_dataset_role,
)
View File
View File
+156
View File
@@ -0,0 +1,156 @@
from __future__ import annotations
from .common import ApiErrorEnvelope, ApiErrorItem, Envelope, PaginationEnvelope
from .project import ProjectCreate, ProjectList, ProjectRead, ProjectUpdate
from .area import AreaCreate, AreaList, AreaRead, AreaUpdate
from .dataset import DatasetCreateResponse, DatasetList
from .detection import (
DetectionListResponse,
DetectionModelCapability,
DetectionModelsResponse,
DetectionQaRequest,
DetectionRead,
DetectionRunListResponse,
DetectionRunRead,
DetectionRunRequest,
DetectionRunResponse,
)
from .segmentation import (
SegmentationListResponse,
SegmentationModelCapability,
SegmentationModelsResponse,
SegmentationQaRequest,
SegmentationRead,
SegmentationRunListResponse,
SegmentationRunRead,
SegmentationRunRequest,
SegmentationRunResponse,
)
from .health import HealthResponse, SystemCapabilities
from .job import JobCreate, JobList, JobRead, JobStatus
from .external import (
ExternalFetchRequest,
ExternalFetchResponse,
ProviderCapabilitiesResponse,
ProviderCapabilityResponse,
ProviderImportRequest,
ProviderImportResponse,
ProviderLayersResponse,
ProviderStatusResponse,
)
from .export import (
ExportContentResponse,
ExportCreateResponse,
ExportListResponse,
ExportRead,
GeoJsonExportRequest,
MetadataExportRequest,
ReportExportRequest,
)
from .qa import QaProviderComparisonRequest, QaProviderComparisonResult
from .operations import (
RasterClipRequest,
RasterIndexBaseRequest,
RasterMetadataResponse,
RasterNdviRequest,
RasterNdwiRequest,
RasterNdbiRequest,
RasterOperationResult,
RasterPreviewResponse,
RasterReprojectRequest,
RasterReprojectResponse,
RasterStatsResponse,
RasterTileManifest,
RasterTileManifestTile,
RasterTileRequest,
RasterTileResponse,
VectorBBoxResponse,
VectorBufferRequest,
VectorClipRequest,
VectorIntersectRequest,
VectorOperationRequest,
VectorOperationResult,
VectorStatsRequest,
VectorStatsResponse,
)
__all__ = [
"Envelope",
"ApiErrorEnvelope",
"ApiErrorItem",
"PaginationEnvelope",
"ProjectCreate",
"ProjectRead",
"ProjectUpdate",
"ProjectList",
"AreaCreate",
"AreaRead",
"AreaUpdate",
"AreaList",
"DatasetCreateResponse",
"DatasetList",
"DetectionListResponse",
"DetectionModelCapability",
"DetectionModelsResponse",
"DetectionQaRequest",
"DetectionRead",
"DetectionRunListResponse",
"DetectionRunRead",
"DetectionRunRequest",
"DetectionRunResponse",
"SegmentationListResponse",
"SegmentationModelCapability",
"SegmentationModelsResponse",
"SegmentationQaRequest",
"SegmentationRead",
"SegmentationRunListResponse",
"SegmentationRunRead",
"SegmentationRunRequest",
"SegmentationRunResponse",
"HealthResponse",
"SystemCapabilities",
"JobCreate",
"JobList",
"JobRead",
"JobStatus",
"VectorBBoxResponse",
"VectorClipRequest",
"VectorBufferRequest",
"VectorIntersectRequest",
"VectorOperationRequest",
"VectorOperationResult",
"RasterClipRequest",
"RasterStatsResponse",
"RasterReprojectRequest",
"RasterReprojectResponse",
"RasterTileRequest",
"RasterMetadataResponse",
"RasterOperationResult",
"RasterPreviewResponse",
"RasterTileManifestTile",
"RasterTileManifest",
"RasterTileResponse",
"RasterIndexBaseRequest",
"RasterNdviRequest",
"RasterNdwiRequest",
"RasterNdbiRequest",
"VectorStatsRequest",
"VectorStatsResponse",
"ExternalFetchRequest",
"ExternalFetchResponse",
"ProviderCapabilitiesResponse",
"ProviderCapabilityResponse",
"ProviderImportRequest",
"ProviderImportResponse",
"ProviderLayersResponse",
"ProviderStatusResponse",
"GeoJsonExportRequest",
"MetadataExportRequest",
"ReportExportRequest",
"ExportRead",
"ExportCreateResponse",
"ExportListResponse",
"ExportContentResponse",
"QaProviderComparisonRequest",
"QaProviderComparisonResult",
]
+40
View File
@@ -0,0 +1,40 @@
from __future__ import annotations
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel
class AreaCreate(BaseModel):
name: str
geometry: dict
crs: str | None = "EPSG:4326"
class AreaUpdate(BaseModel):
name: str | None = None
crs: str | None = None
class AreaRead(BaseModel):
id: UUID
project_id: UUID
name: str
original_crs: str | None
area_m2: float | None
created_at: datetime | None = None
geometry_type: str | None = None
model_config = {"from_attributes": True}
class AreaListItem(AreaRead):
pass
class AreaList(BaseModel):
items: list[AreaRead]
total: int
limit: int
offset: int
+31
View File
@@ -0,0 +1,31 @@
from __future__ import annotations
from pydantic import BaseModel, Field
class Envelope(BaseModel):
data: object
class PaginatedEnvelope(BaseModel):
items: list
total: int
limit: int
offset: int
class PaginationEnvelope(BaseModel):
items: list
total: int
limit: int = Field(default=50)
offset: int = Field(default=0)
class ApiErrorItem(BaseModel):
code: str
message: str
details: dict = Field(default_factory=dict)
class ApiErrorEnvelope(BaseModel):
error: ApiErrorItem
+81
View File
@@ -0,0 +1,81 @@
from __future__ import annotations
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel
class DatasetStorageResponse(BaseModel):
original_filename: str | None = None
stored_filename: str | None = None
content_type: str | None = None
size_bytes: int | None = None
checksum_sha256: str | None = None
class DatasetVectorSummary(BaseModel):
feature_count: int | None = None
geometry_types: list[str] | None = None
bounds_json: dict | None = None
approximate_area_m2: float | None = None
crs: str | None = None
feature_geometry_count: int | None = None
invalid_features: int | None = None
crs_assumed: bool | None = None
class DatasetCreateResponse(BaseModel):
id: UUID
name: str
dataset_type: str
source: str
dataset_role: str = "source"
source_name: str | None = None
reference_layer_name: str | None = None
source_metadata: dict | None = None
provenance_metadata: dict | None = None
imported_at: datetime | None = None
project_id: UUID
area_id: UUID | None = None
storage_path: str | None = None
original_filename: str | None = None
stored_filename: str | None = None
content_type: str | None = None
size_bytes: int | None = None
checksum_sha256: str | None = None
crs: str | None = None
bounds_json: dict | None = None
metadata_json: dict | None = None
vector_summary: DatasetVectorSummary | None = None
status: str
derived_from_dataset_id: UUID | None = None
created_at: datetime | None = None
feature_count: int | None = None
model_config = {"from_attributes": True}
class DatasetList(BaseModel):
items: list[DatasetCreateResponse]
total: int
limit: int
offset: int
class DatasetMetadataRefresh(BaseModel):
feature_count: int | None = None
geometry_types: list[str] | None = None
bounds_json: dict | None = None
crs: str | None = None
class ExportRequest(BaseModel):
dataset_id: UUID
name: str | None = None
class ExportRead(BaseModel):
export_id: UUID
path: str
status: str
+17
View File
@@ -0,0 +1,17 @@
from __future__ import annotations
from uuid import UUID
from pydantic import BaseModel
class DemoWorkflowResponse(BaseModel):
project_id: UUID
area_id: UUID
reference_dataset_id: UUID
candidate_dataset_id: UUID
quality_check_id: UUID
metric_count: int
status: str
message: str
created: bool
+98
View File
@@ -0,0 +1,98 @@
from __future__ import annotations
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel, Field
class DetectionModelCapability(BaseModel):
model_id: str
display_name: str
framework: str
task_type: str
supported_classes: list[str]
configured: bool
status: str
limitation_message: str
version: str | None = None
class DetectionModelsResponse(BaseModel):
models: list[DetectionModelCapability]
class DetectionRunRequest(BaseModel):
project_id: UUID
dataset_id: UUID
model_id: str
confidence_threshold: float = Field(default=0.5, ge=0.0, le=1.0)
class_filter: list[str] | None = None
tile_manifest_path: str | None = None
parameters_json: dict = Field(default_factory=dict)
class DetectionQaRequest(BaseModel):
reference_dataset_id: UUID
iou_threshold: float = Field(default=0.5, ge=0.0, le=1.0)
class_name: str | None = None
min_confidence: float | None = Field(default=None, ge=0.0, le=1.0)
class DetectionRunResponse(BaseModel):
analysis_run_id: UUID
job_id: UUID
project_id: UUID
dataset_id: UUID
model_id: str
status: str
detection_count: int
error_code: str | None = None
message: str
class DetectionRunRead(BaseModel):
id: UUID
project_id: UUID
dataset_id: UUID | None = None
job_id: UUID | None = None
analysis_type: str
status: str
model_name: str | None = None
model_version: str | None = None
parameters_json: dict
result_json: dict | None = None
error_message: str | None = None
created_at: datetime | None = None
started_at: datetime | None = None
finished_at: datetime | None = None
model_config = {"from_attributes": True}
class DetectionRunListResponse(BaseModel):
items: list[DetectionRunRead]
total: int
class DetectionRead(BaseModel):
id: UUID
project_id: UUID
dataset_id: UUID | None = None
analysis_run_id: UUID | None = None
job_id: UUID | None = None
model_name: str
model_version: str | None = None
class_name: str
confidence: float
bbox_json: dict | None = None
source_tile_path: str | None = None
properties_json: dict | None = None
created_at: datetime | None = None
model_config = {"from_attributes": True}
class DetectionListResponse(BaseModel):
items: list[DetectionRead]
total: int
+69
View File
@@ -0,0 +1,69 @@
from __future__ import annotations
from datetime import datetime
from typing import Literal
from uuid import UUID
from pydantic import BaseModel, model_validator
ExportKind = Literal["dataset", "detection_run", "segmentation_run"]
class GeoJsonExportRequest(BaseModel):
dataset_id: UUID | None = None
analysis_run_id: UUID | None = None
export_kind: ExportKind = "dataset"
name: str | None = None
@model_validator(mode="after")
def validate_target(self) -> "GeoJsonExportRequest":
if self.export_kind == "dataset" and self.dataset_id is None:
raise ValueError("dataset_id is required for dataset GeoJSON exports")
if self.export_kind in {"detection_run", "segmentation_run"} and self.analysis_run_id is None:
raise ValueError("analysis_run_id is required for run GeoJSON exports")
return self
class MetadataExportRequest(BaseModel):
project_id: UUID
name: str | None = None
class ReportExportRequest(BaseModel):
project_id: UUID
name: str | None = None
class ExportRead(BaseModel):
id: UUID
project_id: UUID
analysis_run_id: UUID | None = None
export_type: str
storage_path: str
metadata_json: dict | None = None
created_at: datetime | None = None
status: str = "ready"
model_config = {"from_attributes": True}
class ExportCreateResponse(BaseModel):
export_id: UUID
path: str
status: str
export_type: str
metadata_json: dict | None = None
class ExportListResponse(BaseModel):
items: list[ExportRead]
total: int
limit: int
offset: int
class ExportContentResponse(BaseModel):
export_id: UUID
export_type: str
content: dict
+68
View File
@@ -0,0 +1,68 @@
from __future__ import annotations
from uuid import UUID
from pydantic import BaseModel
class ProviderCapabilityResponse(BaseModel):
provider_name: str
display_name: str
authority_level: str
supported_layers: list[str]
supported_geometry_types: list[str]
supported_query_modes: list[str]
fetch_signature: str
configured: bool
status: str
limitation_message: str
attribution: str
license_note: str
not_configured_reason: str | None = None
class ProviderCapabilitiesResponse(BaseModel):
providers: list[ProviderCapabilityResponse]
class ProviderLayersResponse(BaseModel):
provider_name: str
layers: list[str]
class ProviderStatusResponse(BaseModel):
provider_name: str
configured: bool
status: str
limitation_message: str
class ExternalFetchRequest(BaseModel):
project_id: UUID
area_id: UUID | None = None
layers: list[str] = []
class ExternalFetchResponse(BaseModel):
provider: str
status: str
message: str
requested_layers: list[str]
project_id: UUID
area_id: UUID | None = None
class ProviderImportRequest(BaseModel):
project_id: str
area_id: str | None = None
layers: list[str] = []
dataset_role: str | None = None
class ProviderImportResponse(BaseModel):
provider_name: str
status: str
message: str
requested_layers: list[str]
dataset_id: str | None = None
dataset_role: str | None = None
source_name: str | None = None
+37
View File
@@ -0,0 +1,37 @@
from __future__ import annotations
from pydantic import BaseModel, Field
class ProviderCapability(BaseModel):
provider_name: str
display_name: str
authority_level: str
supported_layers: list[str]
supported_geometry_types: list[str]
supported_query_modes: list[str]
fetch_signature: str
configured: bool
status: str
limitation_message: str
attribution: str
license_note: str
not_configured_reason: str | None = None
class HealthResponse(BaseModel):
status: str
service: str
version: str
database: str | None = None
class SystemCapabilities(BaseModel):
postgis: bool
rasterio: bool
geopandas: bool
yolo: bool | str
sam: bool | str
grb: str
sentinel: str
providers: list[ProviderCapability] = Field(default_factory=list)
+51
View File
@@ -0,0 +1,51 @@
from __future__ import annotations
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel, Field
class JobCreate(BaseModel):
job_type: str
project_id: UUID
dataset_id: UUID | None = None
input_dataset_id: UUID | None = None
output_dataset_id: UUID | None = None
parameters_json: dict = Field(default_factory=dict)
class JobRead(BaseModel):
id: UUID
job_type: str
status: str
project_id: UUID
dataset_id: UUID | None = None
input_dataset_id: UUID | None = None
output_dataset_id: UUID | None = None
parameters_json: dict
result_json: dict | None = None
error_message: str | None = None
created_at: datetime | None = None
started_at: datetime | None = None
finished_at: datetime | None = None
model_config = {"from_attributes": True}
class JobStatus(BaseModel):
id: UUID
status: str
error_message: str | None = None
started_at: datetime | None = None
finished_at: datetime | None = None
result_json: dict | None = None
model_config = {"from_attributes": True}
class JobList(BaseModel):
items: list[JobRead]
total: int
limit: int
offset: int
+193
View File
@@ -0,0 +1,193 @@
from __future__ import annotations
from pydantic import BaseModel
class VectorOperationResult(BaseModel):
feature_count: int
geometry_type_summary: dict[str, int]
bounds_json: dict | None = None
crs: str | None = None
source_dataset_id: str
class VectorOperationRequest(BaseModel):
output_name: str | None = None
class VectorClipRequest(VectorOperationRequest):
area_id: str
class VectorBufferRequest(VectorOperationRequest):
distance_m: float
dissolve: bool = False
class VectorIntersectRequest(VectorOperationRequest):
other_dataset_id: str
class VectorStatsRequest(BaseModel):
pass
class RasterReadyResponse(BaseModel):
dataset_id: str
ready: bool
message: str | None = None
class RasterOperationResult(BaseModel):
dataset_id: str
ready: bool
metadata: dict | None = None
output_dataset_id: str | None = None
operation: str | None = None
class RasterMetadataResponse(BaseModel):
dataset_id: str
driver: str | None = None
width: int | None = None
height: int | None = None
band_count: int | None = None
crs: str | None = None
bounds: list[float] | None = None
resolution: list[float] | None = None
dtype: list[str] | None = None
nodata: list[float] | float | None = None
transform: list[float] | None = None
size_bytes: int | None = None
checksum_sha256: str | None = None
path: str | None = None
class RasterPreviewResponse(BaseModel):
dataset_id: str
ready: bool
preview: dict
metadata: dict | None = None
class RasterBandStats(BaseModel):
band_index: int
dtype: str | None = None
min: float | None = None
max: float | None = None
mean: float | None = None
std: float | None = None
nodata_count: int
nodata_ratio: float
valid_pixel_count: int
histogram: list[int] | None = None
histogram_bins: list[float] | None = None
class RasterStatsResponse(BaseModel):
dataset_id: str
source_dataset_id: str | None = None
bands: list[RasterBandStats]
generated_at: str | None = None
metadata: dict | None = None
class RasterReprojectRequest(BaseModel):
target_crs: str | None = "EPSG:31370"
resampling: str = "nearest"
output_name: str | None = None
class RasterClipRequest(BaseModel):
area_id: str
output_name: str | None = None
class RasterTileRequest(BaseModel):
tile_size: int = 512
overlap: int = 64
output_name: str | None = None
class RasterIndexBaseRequest(BaseModel):
output_name: str | None = None
class RasterNdviRequest(RasterIndexBaseRequest):
nir_band: int
red_band: int
class RasterNdwiRequest(RasterIndexBaseRequest):
green_band: int
nir_band: int
class RasterNdbiRequest(RasterIndexBaseRequest):
swir_band: int
nir_band: int
class RasterTileManifestTile(BaseModel):
path: str
pixel_window: list[int]
bounds: list[float]
transform: list[float]
index: int
class RasterTileManifest(BaseModel):
tile_set_id: str
source_dataset_id: str
source_raster_id: str
bounds: list[float]
tile_size: int
overlap: int
parameters: dict[str, str | int | float | bool | None]
created_at: str
tile_paths: list[str]
count: int
tiles: list[RasterTileManifestTile]
ai_inference: bool = False
tile_server: str | None = None
class RasterTileResponse(BaseModel):
dataset_id: str
ready: bool
operation: str
tile_set_id: str
tile_size: int
overlap: int
manifest_path: str
count: int
manifest: RasterTileManifest
class RasterReprojectResponse(BaseModel):
dataset_id: str
ready: bool
operation: str
output_dataset_id: str
source_dataset_id: str
target_dataset_id: str | None = None
class RasterOperationUnavailable(BaseModel):
code: str
message: str
class VectorBBoxResponse(BaseModel):
dataset_id: str
bounds_json: dict | None
feature_count: int
crs: str | None = None
class VectorStatsResponse(BaseModel):
dataset_id: str
feature_count: int
geometry_type_summary: dict[str, int]
bounds_json: dict | None
crs: str | None = None
+41
View File
@@ -0,0 +1,41 @@
from __future__ import annotations
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel
class ProjectCreate(BaseModel):
name: str
description: str | None = None
region: str | None = "Kempen"
class ProjectUpdate(BaseModel):
name: str | None = None
description: str | None = None
region: str | None = None
class ProjectRead(BaseModel):
id: UUID
name: str
description: str | None = None
region: str
status: str
created_at: datetime | None = None
updated_at: datetime | None = None
model_config = {"from_attributes": True}
class ProjectListItem(ProjectRead):
pass
class ProjectList(BaseModel):
items: list[ProjectRead]
total: int
limit: int
offset: int
+71
View File
@@ -0,0 +1,71 @@
from __future__ import annotations
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel, Field
class QaProviderComparisonRequest(BaseModel):
candidate_dataset_id: UUID
reference_dataset_id: UUID
iou_threshold: float = Field(default=0.5, ge=0.0, le=1.0)
area_id: UUID | None = None
class QaProviderComparisonResult(BaseModel):
status: str
warnings: list[str] = Field(default_factory=list)
candidate_feature_count: int
reference_feature_count: int
matches: int
false_positives: int
false_negatives: int
precision: float | None
recall: float | None
f1_score: float | None
mean_iou: float | None
iou_threshold: float
unsupported_geometry: bool = False
unsupported_geometries: list[str] = Field(default_factory=list)
generated_at: datetime
class MetricRead(BaseModel):
id: UUID
quality_check_id: UUID | None = None
analysis_run_id: UUID | None = None
metric_key: str
metric_value: float | None = None
metric_unit: str | None = None
label: str | None = None
metadata_json: dict | None = None
created_at: datetime | None = None
model_config = {"from_attributes": True}
class QualityCheckRead(BaseModel):
id: UUID
project_id: UUID
job_id: UUID | None = None
analysis_run_id: UUID | None = None
candidate_dataset_id: UUID | None = None
reference_dataset_id: UUID
check_type: str
status: str
score: float | None = None
parameters_json: dict | None = None
findings_json: dict | None = None
created_at: datetime | None = None
completed_at: datetime | None = None
metrics: list[MetricRead] = Field(default_factory=list)
model_config = {"from_attributes": True}
class QualityCheckList(BaseModel):
items: list[QualityCheckRead]
total: int
limit: int
offset: int
+95
View File
@@ -0,0 +1,95 @@
from __future__ import annotations
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel, Field
from app.schemas.detection import DetectionModelCapability
SegmentationModelCapability = DetectionModelCapability
class SegmentationModelsResponse(BaseModel):
models: list[SegmentationModelCapability]
class SegmentationRunRequest(BaseModel):
project_id: UUID
dataset_id: UUID
model_id: str
confidence_threshold: float = Field(default=0.5, ge=0.0, le=1.0)
class_filter: list[str] | None = None
tile_manifest_path: str | None = None
parameters_json: dict = Field(default_factory=dict)
class SegmentationQaRequest(BaseModel):
reference_dataset_id: UUID
iou_threshold: float = Field(default=0.5, ge=0.0, le=1.0)
class_name: str | None = None
min_confidence: float | None = Field(default=None, ge=0.0, le=1.0)
class SegmentationRunResponse(BaseModel):
analysis_run_id: UUID
job_id: UUID
project_id: UUID
dataset_id: UUID
model_id: str
status: str
segmentation_count: int
error_code: str | None = None
message: str
class SegmentationRunRead(BaseModel):
id: UUID
project_id: UUID
dataset_id: UUID | None = None
job_id: UUID | None = None
analysis_type: str
status: str
model_name: str | None = None
model_version: str | None = None
parameters_json: dict
result_json: dict | None = None
error_message: str | None = None
created_at: datetime | None = None
started_at: datetime | None = None
finished_at: datetime | None = None
model_config = {"from_attributes": True}
class SegmentationRunListResponse(BaseModel):
items: list[SegmentationRunRead]
total: int
class SegmentationRead(BaseModel):
id: UUID
project_id: UUID
dataset_id: UUID | None = None
analysis_run_id: UUID | None = None
job_id: UUID | None = None
model_name: str
model_version: str | None = None
class_name: str
confidence: float | None = None
bbox_json: dict | None = None
area_m2: float | None = None
mask_path: str | None = None
source_tile_path: str | None = None
tile_index: int | None = None
properties_json: dict | None = None
provenance_json: dict | None = None
created_at: datetime | None = None
model_config = {"from_attributes": True}
class SegmentationListResponse(BaseModel):
items: list[SegmentationRead]
total: int
View File
+77
View File
@@ -0,0 +1,77 @@
from __future__ import annotations
import uuid
from sqlalchemy.orm import Session
from geoalchemy2.shape import from_shape
from app.core.errors import AppError
from app.models import Area, Project
from app.schemas.area import AreaCreate, AreaRead, AreaUpdate
from app.utils.geometry import area_m2, geometry_bbox_polygon, normalize_to_multipolygon
class AreaService:
@staticmethod
def list_areas(db: Session, project_id: uuid.UUID, limit: int = 50, offset: int = 0) -> tuple[list[AreaRead], int]:
total = db.query(Area).filter(Area.project_id == project_id).count()
areas = (
db.query(Area)
.filter(Area.project_id == project_id)
.order_by(Area.created_at.desc())
.offset(offset)
.limit(limit)
.all()
)
return [AreaRead.model_validate(area) for area in areas], total
@staticmethod
def create_area(db: Session, project_id: uuid.UUID, payload: AreaCreate) -> AreaRead:
if not db.get(Project, project_id):
raise AppError(code="PROJECT_NOT_FOUND", message="Project not found", status_code=404)
try:
multipolygon = normalize_to_multipolygon(payload.geometry)
except ValueError as exc:
raise AppError(code="INVALID_GEOMETRY", message=str(exc), status_code=400) from exc
area = Area(
project_id=project_id,
name=payload.name.strip() or "Unnamed area",
geometry=from_shape(multipolygon, srid=4326),
original_crs=payload.crs or "EPSG:4326",
area_m2=area_m2(multipolygon),
bbox=from_shape(geometry_bbox_polygon(multipolygon), srid=4326),
)
db.add(area)
db.commit()
db.refresh(area)
return AreaRead.model_validate(area)
@staticmethod
def get_area(db: Session, area_id: uuid.UUID) -> AreaRead:
area = db.get(Area, area_id)
if not area:
raise AppError(code="AREA_NOT_FOUND", message="Area not found", status_code=404)
return AreaRead.model_validate(area)
@staticmethod
def update_area(db: Session, area_id: uuid.UUID, payload: AreaUpdate) -> AreaRead:
area = db.get(Area, area_id)
if not area:
raise AppError(code="AREA_NOT_FOUND", message="Area not found", status_code=404)
changed = False
if payload.name:
area.name = payload.name.strip() or area.name
changed = True
if payload.crs:
area.original_crs = payload.crs
changed = True
if not changed:
raise AppError(code="NO_CHANGES", message="No updatable fields provided", status_code=422)
db.add(area)
db.commit()
db.refresh(area)
return AreaRead.model_validate(area)
+452
View File
@@ -0,0 +1,452 @@
from __future__ import annotations
import json
import pathlib
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from uuid import UUID
import uuid
from fastapi import UploadFile
from sqlalchemy.orm import Session
from app.core.errors import AppError
from app.models import Dataset, Project
from app.schemas.dataset import DatasetCreateResponse, DatasetStorageResponse, DatasetVectorSummary
from app.services.geojson_service import parse_geojson_payload, load_dataset_text
from app.services.raster_service import extract_raster_metadata
from app.services.storage_service import StorageService
from app.services.vector_feature_service import VectorFeatureService
class DatasetService:
VECTOR_EXTENSIONS = {".geojson", ".json"}
RASTER_EXTENSIONS = {".tif", ".tiff", ".geotiff"}
VECTOR_TYPES = {"vector", "geojson"}
RASTER_TYPES = {"raster", "tif", "tiff", "geotiff"}
VALID_DATASET_ROLES = {"source", "derived", "reference"}
@staticmethod
def _canonical_dataset_type(dataset_type: str) -> str:
normalized = (dataset_type or "").strip().lower()
if normalized in DatasetService.VECTOR_TYPES:
return "vector"
if normalized in DatasetService.RASTER_TYPES:
return "raster"
raise AppError(
code="INVALID_DATASET_TYPE",
message="dataset_type must be 'vector' or 'raster' (or legacy 'geojson')",
status_code=400,
)
@staticmethod
def _normalize_stored_dataset_type(dataset_type: str) -> str:
normalized = (dataset_type or "").strip().lower()
if normalized in DatasetService.VECTOR_TYPES:
return "vector"
if normalized in DatasetService.RASTER_TYPES:
return "raster"
return normalized
@staticmethod
def _is_vector_type(dataset_type: str) -> bool:
return DatasetService._normalize_stored_dataset_type(dataset_type) == "vector"
@staticmethod
def _is_raster_type(dataset_type: str) -> bool:
return DatasetService._normalize_stored_dataset_type(dataset_type) == "raster"
@staticmethod
def _normalize_dataset_role(dataset_role: str | None) -> str:
normalized = (dataset_role or "").strip().lower() or "source"
if normalized not in DatasetService.VALID_DATASET_ROLES:
raise AppError(
code="INVALID_DATASET_ROLE",
message="dataset_role must be one of: source, derived, reference",
status_code=400,
)
return normalized
@staticmethod
def _extension_for_path(filename: str) -> str:
return Path(filename).suffix.lower()
@staticmethod
def _validate_upload_filename(filename: str | None) -> str:
if not filename:
raise AppError(code="INVALID_UPLOAD", message="Missing file name", status_code=400)
return filename
@staticmethod
def list_datasets(db: Session, project_id: UUID, limit: int = 50, offset: int = 0) -> tuple[list[DatasetCreateResponse], int]:
total = db.query(Dataset).filter(Dataset.project_id == project_id).count()
rows = (
db.query(Dataset)
.filter(Dataset.project_id == project_id)
.order_by(Dataset.created_at.desc())
.offset(offset)
.limit(limit)
.all()
)
response_items = []
for row in rows:
feature_count = None
metadata_json = row.metadata_json or {}
vector_summary = DatasetService._extract_vector_summary(row.dataset_type, metadata_json)
if isinstance(metadata_json, dict):
feature_count = metadata_json.get("feature_count")
response_items.append(
DatasetCreateResponse(
id=row.id,
name=row.name,
dataset_type=row.dataset_type,
source=row.source,
dataset_role=row.dataset_role,
source_name=row.source_name,
reference_layer_name=row.reference_layer_name,
source_metadata=row.source_metadata,
provenance_metadata=row.provenance_metadata,
imported_at=row.imported_at,
project_id=row.project_id,
area_id=row.area_id,
storage_path=row.storage_path,
original_filename=row.original_filename,
stored_filename=row.stored_filename,
content_type=row.content_type,
size_bytes=row.size_bytes,
checksum_sha256=row.checksum_sha256,
crs=row.crs,
bounds_json=row.bounds_json,
metadata_json=row.metadata_json,
vector_summary=vector_summary,
status=row.status,
derived_from_dataset_id=row.derived_from_dataset_id,
created_at=row.created_at,
feature_count=feature_count,
)
)
return response_items, total
@staticmethod
def _extract_vector_summary(dataset_type: str, metadata_json: dict) -> DatasetVectorSummary | None:
if not DatasetService._is_vector_type(dataset_type):
return None
if not isinstance(metadata_json, dict):
return None
return DatasetVectorSummary(
feature_count=metadata_json.get("feature_count"),
geometry_types=metadata_json.get("geometry_types"),
bounds_json=metadata_json.get("bounds_json"),
approximate_area_m2=metadata_json.get("approximate_area_m2"),
crs=metadata_json.get("crs"),
feature_geometry_count=metadata_json.get("feature_geometry_count"),
invalid_features=metadata_json.get("invalid_features"),
crs_assumed=metadata_json.get("crs_assumed"),
)
@staticmethod
async def upload_dataset(
db: Session,
project_id: UUID,
file: UploadFile,
dataset_type: str,
source: str,
dataset_role: str = "source",
source_name: str | None = None,
reference_layer_name: str | None = None,
source_metadata: dict | None = None,
provenance_metadata: dict | None = None,
area_id: UUID | None = None,
) -> DatasetCreateResponse:
if not db.get(Project, project_id):
raise AppError(code="PROJECT_NOT_FOUND", message="Project not found", status_code=404)
filename = DatasetService._validate_upload_filename(file.filename)
canonical_type = DatasetService._canonical_dataset_type(dataset_type)
normalized_role = DatasetService._normalize_dataset_role(dataset_role)
normalized_source_name = source_name
if normalized_role == "reference" and not normalized_source_name:
normalized_source_name = "manual"
if normalized_role == "reference" and canonical_type == "raster":
raise AppError(code="INVALID_DATASET_ROLE", message="reference role requires vector dataset type", status_code=400)
extension = DatasetService._extension_for_path(filename)
if canonical_type == "vector" and extension not in DatasetService.VECTOR_EXTENSIONS:
raise AppError(code="INVALID_UPLOAD", message="Vector uploads require .geojson or .json files", status_code=415)
if canonical_type == "raster" and extension not in DatasetService.RASTER_EXTENSIONS:
raise AppError(
code="INVALID_UPLOAD",
message="Raster uploads require .tif, .tiff or .geotiff files",
status_code=415,
)
raw = await file.read()
storage_info = StorageService.persist_dataset_file(
project_id=str(project_id),
dataset_id=str(dataset_id := uuid.uuid4()),
dataset_type=canonical_type,
original_filename=filename,
content=raw,
content_type=file.content_type,
)
metadata: dict[str, Any] = {}
vector_payload: dict[str, Any] | None = None
status = "uploaded"
try:
status = "validating"
if canonical_type == "vector":
try:
text = raw.decode("utf-8")
except UnicodeDecodeError as exc:
raise AppError(code="INVALID_UPLOAD", message="Upload must be UTF-8 encoded", status_code=400) from exc
metadata = parse_geojson_payload(text)
vector_payload = json.loads(text)
status = "ready"
else:
metadata = extract_raster_metadata(storage_info["storage_path"])
status = "ready"
except ValueError as exc:
status = "failed"
StorageService.remove_dataset_file(storage_info["storage_path"])
raise AppError(code="INVALID_GEOJSON", message=str(exc), status_code=400) from exc
except AppError as exc:
if canonical_type == "raster" and exc.code == "RASTER_PROCESSING_UNAVAILABLE":
status = "failed"
metadata = {
"processing_error": exc.message,
"processing_code": exc.code,
}
else:
StorageService.remove_dataset_file(storage_info["storage_path"])
raise
dataset = Dataset(
id=dataset_id,
project_id=project_id,
area_id=area_id,
name=filename,
dataset_type=canonical_type,
source=source,
dataset_role=normalized_role,
source_name=normalized_source_name,
reference_layer_name=reference_layer_name if normalized_role == "reference" else None,
source_metadata=source_metadata,
provenance_metadata=provenance_metadata,
imported_at=datetime.now(timezone.utc),
storage_path=storage_info["storage_path"],
original_filename=storage_info["original_filename"],
stored_filename=storage_info["stored_filename"],
content_type=storage_info["content_type"],
size_bytes=storage_info["size_bytes"],
checksum_sha256=storage_info["checksum_sha256"],
crs=metadata.get("crs") if isinstance(metadata, dict) else None,
bounds_json=metadata.get("bounds_json") if isinstance(metadata, dict) else None,
resolution_json=metadata.get("resolution_json") if isinstance(metadata, dict) else None,
bands_json=metadata.get("bands_json") if isinstance(metadata, dict) else None,
metadata_json=metadata,
status=status,
)
db.add(dataset)
db.commit()
db.refresh(dataset)
if canonical_type == "vector" and vector_payload is not None and status == "ready":
feature_class = reference_layer_name if normalized_role == "reference" else None
VectorFeatureService.persist_geojson_features(
db=db,
dataset_id=dataset.id,
payload=vector_payload,
feature_class=feature_class,
)
return DatasetCreateResponse(
id=dataset.id,
name=dataset.name,
dataset_type=dataset.dataset_type,
source=dataset.source,
dataset_role=dataset.dataset_role,
source_name=dataset.source_name,
reference_layer_name=dataset.reference_layer_name,
source_metadata=dataset.source_metadata,
provenance_metadata=dataset.provenance_metadata,
imported_at=dataset.imported_at,
project_id=dataset.project_id,
area_id=dataset.area_id,
storage_path=dataset.storage_path,
original_filename=dataset.original_filename,
stored_filename=dataset.stored_filename,
content_type=dataset.content_type,
size_bytes=dataset.size_bytes,
checksum_sha256=dataset.checksum_sha256,
crs=dataset.crs,
derived_from_dataset_id=dataset.derived_from_dataset_id,
bounds_json=dataset.bounds_json,
metadata_json=dataset.metadata_json,
vector_summary=DatasetService._extract_vector_summary(dataset.dataset_type, dataset.metadata_json or {}),
status=dataset.status,
created_at=dataset.created_at,
feature_count=metadata.get("feature_count") if isinstance(metadata, dict) else None,
)
@staticmethod
def refresh_metadata(db: Session, dataset_id: UUID) -> DatasetCreateResponse:
dataset = DatasetService._get_dataset(db, dataset_id)
if not dataset.storage_path:
raise AppError(code="DATASET_FILE_MISSING", message="Stored file missing", status_code=404)
if not Path(dataset.storage_path).exists():
raise AppError(code="DATASET_FILE_MISSING", message="Stored file missing", status_code=404)
try:
if DatasetService._is_vector_type(dataset.dataset_type):
metadata = parse_geojson_payload(load_dataset_text(dataset.storage_path))
elif DatasetService._is_raster_type(dataset.dataset_type):
metadata = extract_raster_metadata(dataset.storage_path)
else:
raise AppError(code="INVALID_DATASET_TYPE", message="Cannot refresh metadata for this dataset type", status_code=400)
dataset.status = "ready"
except ValueError as exc:
dataset.status = "failed"
raise AppError(code="INVALID_GEOJSON", message=str(exc), status_code=400) from exc
except AppError as exc:
if DatasetService._is_raster_type(dataset.dataset_type) and exc.code == "RASTER_PROCESSING_UNAVAILABLE":
dataset.status = "failed"
metadata = {"processing_error": exc.message, "processing_code": exc.code}
else:
dataset.status = "failed"
raise
dataset.crs = metadata.get("crs") if isinstance(metadata, dict) else dataset.crs
dataset.bounds_json = metadata.get("bounds_json") if isinstance(metadata, dict) else dataset.bounds_json
dataset.metadata_json = metadata
dataset.resolution_json = metadata.get("resolution_json") if isinstance(metadata, dict) else dataset.resolution_json
dataset.bands_json = metadata.get("bands_json") if isinstance(metadata, dict) else dataset.bands_json
db.add(dataset)
db.commit()
db.refresh(dataset)
return DatasetCreateResponse(
id=dataset.id,
name=dataset.name,
dataset_type=dataset.dataset_type,
source=dataset.source,
dataset_role=dataset.dataset_role,
source_name=dataset.source_name,
reference_layer_name=dataset.reference_layer_name,
source_metadata=dataset.source_metadata,
provenance_metadata=dataset.provenance_metadata,
imported_at=dataset.imported_at,
project_id=dataset.project_id,
area_id=dataset.area_id,
storage_path=dataset.storage_path,
original_filename=dataset.original_filename,
stored_filename=dataset.stored_filename,
content_type=dataset.content_type,
size_bytes=dataset.size_bytes,
checksum_sha256=dataset.checksum_sha256,
crs=dataset.crs,
derived_from_dataset_id=dataset.derived_from_dataset_id,
bounds_json=dataset.bounds_json,
metadata_json=dataset.metadata_json,
vector_summary=DatasetService._extract_vector_summary(dataset.dataset_type, dataset.metadata_json or {}),
status=dataset.status,
created_at=dataset.created_at,
feature_count=metadata.get("feature_count") if isinstance(metadata, dict) else None,
)
@staticmethod
def get_dataset(db: Session, dataset_id: UUID) -> Dataset:
dataset = db.get(Dataset, dataset_id)
if not dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Dataset not found", status_code=404)
return dataset
@staticmethod
def _get_dataset(db: Session, dataset_id: UUID) -> Dataset:
return DatasetService.get_dataset(db, dataset_id)
@staticmethod
def get_dataset_geojson(db: Session, dataset_id: UUID) -> dict:
dataset = DatasetService._get_dataset(db, dataset_id)
if not DatasetService._is_vector_type(dataset.dataset_type):
raise AppError(code="INVALID_DATASET_TYPE", message="Dataset is not a vector dataset", status_code=400)
if not dataset.storage_path:
raise AppError(code="DATASET_FILE_MISSING", message="Stored file missing", status_code=404)
if not pathlib.Path(dataset.storage_path).exists():
raise AppError(code="DATASET_FILE_MISSING", message="Stored file missing", status_code=404)
raw = load_dataset_text(dataset.storage_path)
try:
return json.loads(raw)
except Exception as exc:
raise AppError(code="INVALID_GEOJSON", message="Stored dataset is not valid JSON", status_code=500) from exc
@staticmethod
def inspect_vector_dataset(db: Session, dataset_id: UUID) -> dict[str, Any]:
dataset = DatasetService._get_dataset(db, dataset_id)
if not DatasetService._is_vector_type(dataset.dataset_type):
raise AppError(code="INVALID_DATASET_TYPE", message="Dataset is not a vector dataset", status_code=400)
if not dataset.storage_path or not Path(dataset.storage_path).exists():
raise AppError(code="DATASET_FILE_MISSING", message="Stored file missing", status_code=404)
metadata = dataset.metadata_json or {}
if not isinstance(metadata, dict):
metadata = {}
summary = DatasetService._extract_vector_summary(dataset.dataset_type, metadata)
return {
"dataset": {
"id": str(dataset.id),
"name": dataset.name,
"dataset_type": dataset.dataset_type,
"status": dataset.status,
"source": dataset.source,
"storage": DatasetStorageResponse(
original_filename=dataset.original_filename,
stored_filename=dataset.stored_filename,
content_type=dataset.content_type,
size_bytes=dataset.size_bytes,
checksum_sha256=dataset.checksum_sha256,
).model_dump(),
"feature_count": metadata.get("feature_count"),
"crs": metadata.get("crs"),
},
"summary": summary.model_dump() if summary else None,
"metadata": metadata,
}
@staticmethod
def vector_summary(db: Session, dataset_id: UUID) -> dict[str, Any]:
dataset = DatasetService._get_dataset(db, dataset_id)
if not DatasetService._is_vector_type(dataset.dataset_type):
raise AppError(code="INVALID_DATASET_TYPE", message="Dataset is not a vector dataset", status_code=400)
metadata = dataset.metadata_json or {}
if not isinstance(metadata, dict):
metadata = {}
summary = DatasetService._extract_vector_summary(dataset.dataset_type, metadata)
if not summary:
raise AppError(code="INVALID_GEOJSON", message="Vector summary unavailable", status_code=422)
return summary.model_dump()
@staticmethod
def raster_metadata(db: Session, dataset_id: UUID) -> dict[str, Any]:
dataset = DatasetService._get_dataset(db, dataset_id)
if not DatasetService._is_raster_type(dataset.dataset_type):
raise AppError(code="INVALID_DATASET_TYPE", message="Dataset is not a raster dataset", status_code=400)
if not dataset.storage_path:
raise AppError(code="DATASET_FILE_MISSING", message="Stored file missing", status_code=404)
if not Path(dataset.storage_path).exists():
raise AppError(code="DATASET_FILE_MISSING", message="Stored file missing", status_code=404)
if isinstance(dataset.metadata_json, dict) and dataset.metadata_json.get("driver"):
return dataset.metadata_json
metadata = extract_raster_metadata(dataset.storage_path)
dataset.metadata_json = dict(dataset.metadata_json or {})
dataset.metadata_json.update(metadata)
dataset.status = "ready"
db.add(dataset)
db.commit()
db.refresh(dataset)
return metadata
@@ -0,0 +1,288 @@
from __future__ import annotations
import json
from datetime import datetime, timezone
from pathlib import Path
from uuid import UUID, uuid4
from geoalchemy2.shape import from_shape
from sqlalchemy.orm import Session
from app.models import Area, Dataset, Metric, Project, QualityCheck
from app.schemas.demo import DemoWorkflowResponse
from app.services.geojson_service import parse_geojson_payload
from app.services.qa_service import QaService
from app.services.quality_service import QualityService
from app.services.storage_service import StorageService
from app.services.vector_feature_service import VectorFeatureService
from app.utils.geometry import area_m2, geometry_bbox_polygon, normalize_to_multipolygon
class DemoWorkflowService:
PROJECT_NAME = "GeoIntel Demo - Building QA"
AREA_NAME = "Demo AOI - Geel buildings"
REFERENCE_FILENAME = "demo_reference_buildings.geojson"
CANDIDATE_FILENAME = "demo_predicted_buildings.geojson"
@staticmethod
def _repo_root() -> Path:
return Path(__file__).resolve().parents[3]
@staticmethod
def _fixture_path(filename: str) -> Path:
return DemoWorkflowService._repo_root() / "fixtures" / "golden" / filename
@staticmethod
def _load_fixture(filename: str) -> tuple[dict, bytes]:
path = DemoWorkflowService._fixture_path(filename)
raw = path.read_bytes()
return json.loads(raw.decode("utf-8")), raw
@staticmethod
def _find_existing_project(db: Session) -> Project | None:
return (
db.query(Project)
.filter(Project.name == DemoWorkflowService.PROJECT_NAME)
.filter(Project.status != "deleted")
.first()
)
@staticmethod
def _create_area(db: Session, project_id: UUID) -> Area:
geometry = {
"type": "MultiPolygon",
"coordinates": [
[
[
[4.30, 51.18],
[4.45, 51.18],
[4.45, 51.33],
[4.30, 51.33],
[4.30, 51.18],
]
]
],
}
multipolygon = normalize_to_multipolygon(geometry)
area = Area(
id=uuid4(),
project_id=project_id,
name=DemoWorkflowService.AREA_NAME,
geometry=from_shape(multipolygon, srid=4326),
original_crs="EPSG:4326",
area_m2=area_m2(multipolygon),
bbox=from_shape(geometry_bbox_polygon(multipolygon), srid=4326),
)
db.add(area)
db.commit()
db.refresh(area)
return area
@staticmethod
def _create_dataset(
db: Session,
*,
project_id: UUID,
area_id: UUID,
filename: str,
payload: dict,
raw: bytes,
role: str,
source_name: str,
reference_layer_name: str | None,
) -> Dataset:
dataset_id = uuid4()
storage_info = StorageService.persist_dataset_file(
project_id=str(project_id),
dataset_id=str(dataset_id),
dataset_type="vector",
original_filename=filename,
content=raw,
content_type="application/geo+json",
)
metadata = parse_geojson_payload(payload)
dataset = Dataset(
id=dataset_id,
project_id=project_id,
area_id=area_id,
name=filename,
dataset_type="vector",
source="fixture",
dataset_role=role,
source_name=source_name,
reference_layer_name=reference_layer_name,
source_metadata={
"fixture": True,
"fixture_name": filename,
"usage": "offline demo workflow only",
},
provenance_metadata={
"created_by": "demo_workflow",
"source_path": str(DemoWorkflowService._fixture_path(filename)),
},
imported_at=datetime.now(timezone.utc),
storage_path=storage_info["storage_path"],
original_filename=storage_info["original_filename"],
stored_filename=storage_info["stored_filename"],
content_type=storage_info["content_type"],
size_bytes=storage_info["size_bytes"],
checksum_sha256=storage_info["checksum_sha256"],
crs=metadata.get("crs"),
bounds_json=metadata.get("bounds_json"),
metadata_json=metadata,
status="ready",
)
db.add(dataset)
db.commit()
db.refresh(dataset)
VectorFeatureService.persist_geojson_features(
db=db,
dataset_id=dataset.id,
payload=payload,
feature_class=reference_layer_name or "building",
)
return dataset
@staticmethod
def _persist_qa(
db: Session,
*,
project_id: UUID,
candidate_dataset_id: UUID,
reference_dataset_id: UUID,
area_id: UUID,
) -> QualityCheck:
result = QaService.compare_candidate_with_reference(
db=db,
project_id=project_id,
candidate_dataset_id=candidate_dataset_id,
reference_dataset_id=reference_dataset_id,
iou_threshold=0.5,
area_id=area_id,
)
return QualityService.persist_quality_check(
db=db,
project_id=project_id,
candidate_dataset_id=candidate_dataset_id,
reference_dataset_id=reference_dataset_id,
check_type="demo_candidate_vs_reference",
status=result.status,
score=result.f1_score,
parameters={
"iou_threshold": result.iou_threshold,
"area_id": str(area_id),
"fixture_workflow": True,
},
findings={
"matches": result.matches,
"false_positives": result.false_positives,
"false_negatives": result.false_negatives,
"warnings": result.warnings,
"unsupported_geometry": result.unsupported_geometry,
"unsupported_geometries": result.unsupported_geometries,
},
metrics={
"precision": result.precision,
"recall": result.recall,
"f1": result.f1_score,
"mean_iou": result.mean_iou,
"false_positive_count": result.false_positives,
"false_negative_count": result.false_negatives,
},
)
@staticmethod
def seed(db: Session) -> DemoWorkflowResponse:
existing = DemoWorkflowService._find_existing_project(db)
if existing:
area = db.query(Area).filter(Area.project_id == existing.id).order_by(Area.created_at.asc()).first()
reference = (
db.query(Dataset)
.filter(Dataset.project_id == existing.id)
.filter(Dataset.dataset_role == "reference")
.filter(Dataset.source_name == "fixture")
.first()
)
candidate = (
db.query(Dataset)
.filter(Dataset.project_id == existing.id)
.filter(Dataset.dataset_role == "source")
.filter(Dataset.source_name == "fixture")
.first()
)
quality_check = (
db.query(QualityCheck)
.filter(QualityCheck.project_id == existing.id)
.filter(QualityCheck.check_type == "demo_candidate_vs_reference")
.order_by(QualityCheck.created_at.desc())
.first()
)
if area and reference and candidate and quality_check:
return DemoWorkflowResponse(
project_id=existing.id,
area_id=area.id,
reference_dataset_id=reference.id,
candidate_dataset_id=candidate.id,
quality_check_id=quality_check.id,
metric_count=db.query(Metric).filter(Metric.quality_check_id == quality_check.id).count(),
status="ready",
message="Demo workflow already exists.",
created=False,
)
project = Project(
id=uuid4(),
name=DemoWorkflowService.PROJECT_NAME,
description="Offline fixture workflow: reference buildings, predicted buildings and persisted QA metrics.",
region="Kempen",
status="active",
)
db.add(project)
db.commit()
db.refresh(project)
area = DemoWorkflowService._create_area(db, project.id)
reference_payload, reference_raw = DemoWorkflowService._load_fixture("reference_buildings.geojson")
candidate_payload, candidate_raw = DemoWorkflowService._load_fixture("predicted_buildings.geojson")
reference = DemoWorkflowService._create_dataset(
db=db,
project_id=project.id,
area_id=area.id,
filename=DemoWorkflowService.REFERENCE_FILENAME,
payload=reference_payload,
raw=reference_raw,
role="reference",
source_name="fixture",
reference_layer_name="buildings",
)
candidate = DemoWorkflowService._create_dataset(
db=db,
project_id=project.id,
area_id=area.id,
filename=DemoWorkflowService.CANDIDATE_FILENAME,
payload=candidate_payload,
raw=candidate_raw,
role="source",
source_name="fixture",
reference_layer_name=None,
)
quality_check = DemoWorkflowService._persist_qa(
db=db,
project_id=project.id,
candidate_dataset_id=candidate.id,
reference_dataset_id=reference.id,
area_id=area.id,
)
return DemoWorkflowResponse(
project_id=project.id,
area_id=area.id,
reference_dataset_id=reference.id,
candidate_dataset_id=candidate.id,
quality_check_id=quality_check.id,
metric_count=6,
status="ready",
message="Demo workflow seeded from explicit local fixtures.",
created=True,
)
@@ -0,0 +1,73 @@
from __future__ import annotations
from typing import Any
from pyproj import Transformer
from shapely.geometry import Polygon
from app.core.errors import AppError
def pixel_bbox_to_epsg4326_polygon(bbox: list[float], tile: dict[str, Any], crs: str | None = None) -> Polygon:
if len(bbox) != 4:
raise AppError(code="DETECTION_INVALID_BBOX", message="YOLO detection bbox must contain four pixel coordinates", status_code=422)
x_min, y_min, x_max, y_max = [float(value) for value in bbox]
if x_max <= x_min or y_max <= y_min:
raise AppError(code="DETECTION_INVALID_BBOX", message="YOLO detection bbox must have positive width and height", status_code=422)
transform = tile.get("transform")
if isinstance(transform, list) and len(transform) >= 6:
corners = [
_apply_gdal_transform(transform, x_min, y_min),
_apply_gdal_transform(transform, x_max, y_min),
_apply_gdal_transform(transform, x_max, y_max),
_apply_gdal_transform(transform, x_min, y_max),
_apply_gdal_transform(transform, x_min, y_min),
]
else:
corners = _corners_from_bounds(bbox=[x_min, y_min, x_max, y_max], tile=tile)
source_crs = crs or tile.get("crs") or tile.get("source_crs") or "EPSG:4326"
if str(source_crs).upper() not in {"EPSG:4326", "4326"}:
transformer = Transformer.from_crs(source_crs, "EPSG:4326", always_xy=True)
corners = [transformer.transform(x, y) for x, y in corners]
polygon = Polygon(corners)
if polygon.is_empty or not polygon.is_valid:
raise AppError(code="DETECTION_INVALID_GEOMETRY", message="Georeferenced detection geometry is invalid", status_code=422)
return polygon
def _apply_gdal_transform(transform: list[float], x: float, y: float) -> tuple[float, float]:
c, a, b, f, d, e = [float(value) for value in transform[:6]]
return (a * x + b * y + c, d * x + e * y + f)
def _corners_from_bounds(bbox: list[float], tile: dict[str, Any]) -> list[tuple[float, float]]:
bounds = tile.get("bounds")
pixel_window = tile.get("pixel_window")
if not (isinstance(bounds, list) and len(bounds) == 4 and isinstance(pixel_window, list) and len(pixel_window) == 4):
raise AppError(
code="DETECTION_TILE_MANIFEST_INVALID",
message="Tile manifest entries require transform or bounds plus pixel_window for georeferencing",
status_code=422,
)
x_min, y_min, x_max, y_max = bbox
left, bottom, right, top = [float(value) for value in bounds]
_, _, width, height = [float(value) for value in pixel_window]
if width <= 0 or height <= 0:
raise AppError(code="DETECTION_TILE_MANIFEST_INVALID", message="Tile pixel_window must have positive size", status_code=422)
def project(px: float, py: float) -> tuple[float, float]:
x = left + (px / width) * (right - left)
y = top - (py / height) * (top - bottom)
return (x, y)
return [
project(x_min, y_min),
project(x_max, y_min),
project(x_max, y_max),
project(x_min, y_max),
project(x_min, y_min),
]
+618
View File
@@ -0,0 +1,618 @@
from __future__ import annotations
import uuid
import json
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from typing import Type
from geoalchemy2.shape import from_shape, to_shape
from shapely.geometry import mapping, shape
from app.core.config import Settings, get_settings
from app.core.errors import AppError
from app.models import AnalysisRun, Dataset, Detection, Job, Project, VectorFeature
from app.schemas.detection import DetectionListResponse, DetectionRead, DetectionRunListResponse, DetectionRunRead, DetectionRunResponse
from app.services.detection_georeferencing import pixel_bbox_to_epsg4326_polygon
from app.services.model_registry_service import ModelRegistryService
from app.services.qa_service import QaService
from app.services.quality_service import QualityService
from app.services.yolo_adapter import YoloDetectionAdapter
class DetectionService:
@staticmethod
def _now() -> datetime:
return datetime.now(UTC)
@staticmethod
def run_detection(
db,
project_id: uuid.UUID,
dataset_id: uuid.UUID,
model_id: str,
confidence_threshold: float,
class_filter: list[str] | None = None,
tile_manifest_path: str | None = None,
parameters_json: dict[str, Any] | None = None,
settings: Settings | None = None,
yolo_adapter_class: Type[YoloDetectionAdapter] = YoloDetectionAdapter,
) -> DetectionRunResponse:
parameters = dict(parameters_json or {})
resolved_settings = settings or get_settings()
project = db.get(Project, project_id)
if not project:
raise AppError(code="PROJECT_NOT_FOUND", message="Project not found", status_code=404)
dataset = db.get(Dataset, dataset_id)
if not dataset or dataset.project_id != project_id:
raise AppError(code="DATASET_NOT_FOUND", message="Dataset not found", status_code=404)
if dataset.dataset_type != "raster":
raise AppError(
code="INVALID_DATASET_TYPE",
message="Detection requires a raster dataset",
details={"dataset_type": dataset.dataset_type},
status_code=400,
)
model = ModelRegistryService.get_model_capability(
model_id,
settings=resolved_settings,
yolo_adapter_class=yolo_adapter_class,
)
if model is None:
raise AppError(code="DETECTION_MODEL_NOT_FOUND", message="Detection model not found", status_code=404)
if model.model_id == "manual-fixture-detector" and parameters.get("fixture_mode") is not True:
raise AppError(
code="FIXTURE_MODE_REQUIRED",
message="Fixture detector requires explicit fixture_mode=true",
status_code=400,
)
if model.model_id == resolved_settings.yolo_model_id and not tile_manifest_path:
raise AppError(
code="DETECTION_TILE_MANIFEST_REQUIRED",
message="Configured YOLO inference requires an existing raster tile manifest path",
status_code=400,
)
run_parameters = {
"model_id": model.model_id,
"confidence_threshold": confidence_threshold,
"class_filter": class_filter or [],
"tile_manifest_path": tile_manifest_path,
"parameters_json": parameters,
}
job = DetectionService._create_job(db, project_id, dataset_id, run_parameters)
analysis_run = DetectionService._create_analysis_run(db, project_id, dataset_id, job.id, model, run_parameters)
if not model.configured:
message = model.limitation_message
code = "DETECTION_DEPENDENCY_UNAVAILABLE" if model.status == "dependency_unavailable" else "DETECTION_MODEL_UNAVAILABLE"
DetectionService._mark_failed(db, analysis_run, job, code=code, message=message)
return DetectionRunResponse(
analysis_run_id=analysis_run.id,
job_id=job.id,
project_id=project_id,
dataset_id=dataset_id,
model_id=model.model_id,
status="failed",
detection_count=0,
error_code=code,
message=message,
)
if model.model_id == "manual-fixture-detector":
detections = DetectionService._persist_fixture_detections(
db=db,
project_id=project_id,
dataset_id=dataset_id,
analysis_run=analysis_run,
job=job,
model_name=model.model_id,
model_version=model.version,
raw_detections=parameters.get("fixture_detections"),
confidence_threshold=confidence_threshold,
class_filter=class_filter or [],
)
DetectionService._mark_success(db, analysis_run, job, detection_count=len(detections))
return DetectionRunResponse(
analysis_run_id=analysis_run.id,
job_id=job.id,
project_id=project_id,
dataset_id=dataset_id,
model_id=model.model_id,
status="success",
detection_count=len(detections),
message="Fixture detections persisted.",
)
if model.model_id == resolved_settings.yolo_model_id:
try:
detections = DetectionService._run_configured_yolo(
db=db,
project_id=project_id,
dataset_id=dataset_id,
analysis_run=analysis_run,
job=job,
model_name=model.model_id,
model_version=model.version,
tile_manifest_path=tile_manifest_path,
confidence_threshold=confidence_threshold,
class_filter=class_filter or [],
settings=resolved_settings,
yolo_adapter_class=yolo_adapter_class,
)
except AppError as exc:
DetectionService._mark_failed(db, analysis_run, job, code=exc.code, message=exc.message)
return DetectionRunResponse(
analysis_run_id=analysis_run.id,
job_id=job.id,
project_id=project_id,
dataset_id=dataset_id,
model_id=model.model_id,
status="failed",
detection_count=0,
error_code=exc.code,
message=exc.message,
)
DetectionService._mark_success(db, analysis_run, job, detection_count=len(detections))
return DetectionRunResponse(
analysis_run_id=analysis_run.id,
job_id=job.id,
project_id=project_id,
dataset_id=dataset_id,
model_id=model.model_id,
status="success",
detection_count=len(detections),
message="YOLO detections persisted.",
)
raise AppError(code="DETECTION_MODEL_UNAVAILABLE", message="Detection model is unavailable", status_code=503)
@staticmethod
def get_run(db, analysis_run_id: uuid.UUID) -> DetectionRunRead:
run = db.get(AnalysisRun, analysis_run_id)
if not run or run.analysis_type != "detection":
raise AppError(code="DETECTION_RUN_NOT_FOUND", message="Detection run not found", status_code=404)
return DetectionRunRead.model_validate(run)
@staticmethod
def list_runs(
db,
*,
project_id: uuid.UUID | None = None,
dataset_id: uuid.UUID | None = None,
) -> DetectionRunListResponse:
query = db.query(AnalysisRun).filter(AnalysisRun.analysis_type == "detection")
if project_id is not None:
query = query.filter(AnalysisRun.project_id == project_id)
if dataset_id is not None:
query = query.filter(AnalysisRun.dataset_id == dataset_id)
rows = query.order_by(AnalysisRun.created_at.desc()).all()
return DetectionRunListResponse(items=[DetectionRunRead.model_validate(row) for row in rows], total=len(rows))
@staticmethod
def list_detections(
db,
analysis_run_id: uuid.UUID | None = None,
*,
dataset_id: uuid.UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
) -> DetectionListResponse:
if analysis_run_id is not None:
run = db.get(AnalysisRun, analysis_run_id)
if not run or run.analysis_type != "detection":
raise AppError(code="DETECTION_RUN_NOT_FOUND", message="Detection run not found", status_code=404)
rows = DetectionService._query_detection_rows(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
)
items = [DetectionRead.model_validate(row) for row in rows]
return DetectionListResponse(items=items, total=len(items))
@staticmethod
def get_detection(db, detection_id: uuid.UUID) -> DetectionRead:
detection = db.get(Detection, detection_id)
if not detection:
raise AppError(code="DETECTION_NOT_FOUND", message="Detection not found", status_code=404)
return DetectionRead.model_validate(detection)
@staticmethod
def detections_to_geojson(
db,
*,
analysis_run_id: uuid.UUID | None = None,
dataset_id: uuid.UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
) -> dict[str, Any]:
detections = DetectionService._query_detection_rows(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
)
return {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"id": str(detection.id),
"properties": DetectionService._detection_properties(detection),
"geometry": mapping(to_shape(detection.geometry)),
}
for detection in detections
],
}
@staticmethod
def compare_detections_with_reference(
db,
analysis_run_id: uuid.UUID,
reference_dataset_id: uuid.UUID,
iou_threshold: float = 0.5,
class_name: str | None = None,
min_confidence: float | None = None,
) -> dict[str, Any]:
run = db.get(AnalysisRun, analysis_run_id)
if not run or run.analysis_type != "detection":
raise AppError(code="DETECTION_RUN_NOT_FOUND", message="Detection run not found", status_code=404)
reference_dataset = db.get(Dataset, reference_dataset_id)
if not reference_dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Reference dataset not found", status_code=404)
if reference_dataset.project_id != run.project_id:
raise AppError(code="INVALID_DATASET_SCOPE", message="Reference dataset does not belong to detection project", status_code=400)
if reference_dataset.dataset_type not in {"vector", "geojson"}:
raise AppError(code="INVALID_DATASET_TYPE", message="Reference dataset must be vector data", status_code=400)
detections = DetectionService._query_detection_rows(
db,
analysis_run_id=analysis_run_id,
dataset_id=run.dataset_id,
class_name=class_name,
min_confidence=min_confidence,
)
references = db.query(VectorFeature).filter(VectorFeature.dataset_id == reference_dataset_id).all()
if not references:
raise AppError(
code="REFERENCE_FEATURES_NOT_FOUND",
message="Reference dataset has no persisted vector features for QA",
status_code=422,
)
candidate_geometries = [({"id": str(row.id), "class_name": row.class_name}, to_shape(row.geometry)) for row in detections]
reference_geometries = [({"id": str(row.id), "feature_class": row.feature_class}, to_shape(row.geometry)) for row in references]
matches, false_positives, false_negatives, match_iou_values, warnings, unsupported = QaService._match_io_u_metrics(
candidate_geometries,
reference_geometries,
iou_threshold,
)
mean_iou = None if not match_iou_values else sum(match_iou_values) / len(match_iou_values)
precision = matches / (matches + false_positives) if matches + false_positives > 0 else None
recall = matches / (matches + false_negatives) if matches + false_negatives > 0 else None
f1_score = None
if precision is not None and recall is not None:
f1_score = (2 * precision * recall) / (precision + recall) if precision + recall > 0 else 0.0
status = "unsupported" if unsupported else "ok"
quality_check = QualityService.persist_quality_check(
db=db,
project_id=run.project_id,
analysis_run_id=analysis_run_id,
candidate_dataset_id=run.dataset_id,
reference_dataset_id=reference_dataset_id,
check_type="detections_vs_reference",
status=status,
score=f1_score,
parameters={
"analysis_run_id": str(analysis_run_id),
"reference_dataset_id": str(reference_dataset_id),
"iou_threshold": iou_threshold,
"class_name": class_name,
"min_confidence": min_confidence,
},
findings={
"matches": matches,
"false_positives": false_positives,
"false_negatives": false_negatives,
"warnings": warnings,
"unsupported_geometry": unsupported,
},
metrics={
"precision": precision,
"recall": recall,
"f1": f1_score,
"mean_iou": mean_iou,
"false_positive_count": false_positives,
"false_negative_count": false_negatives,
},
)
return {
"status": status,
"quality_check_id": str(quality_check.id),
"analysis_run_id": str(analysis_run_id),
"reference_dataset_id": str(reference_dataset_id),
"candidate_feature_count": len(candidate_geometries),
"reference_feature_count": len(reference_geometries),
"matches": matches,
"false_positives": false_positives,
"false_negatives": false_negatives,
"precision": precision,
"recall": recall,
"f1_score": f1_score,
"mean_iou": mean_iou,
"iou_threshold": iou_threshold,
"warnings": warnings,
}
@staticmethod
def _create_job(db, project_id: uuid.UUID, dataset_id: uuid.UUID, parameters: dict[str, Any]) -> Job:
job = Job(
id=uuid.uuid4(),
job_type="detection.run",
status="running",
project_id=project_id,
dataset_id=dataset_id,
input_dataset_id=dataset_id,
parameters_json=parameters,
started_at=DetectionService._now(),
)
db.add(job)
db.commit()
db.refresh(job)
return job
@staticmethod
def _query_detection_rows(
db,
*,
analysis_run_id: uuid.UUID | None = None,
dataset_id: uuid.UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
) -> list[Detection]:
query = db.query(Detection)
if analysis_run_id is not None:
query = query.filter(Detection.analysis_run_id == analysis_run_id)
if dataset_id is not None:
query = query.filter(Detection.dataset_id == dataset_id)
if class_name:
query = query.filter(Detection.class_name == class_name)
if min_confidence is not None:
query = query.filter(Detection.confidence >= min_confidence)
return query.order_by(Detection.created_at.desc()).all()
@staticmethod
def _detection_properties(detection: Detection) -> dict[str, Any]:
return {
"detection_id": str(detection.id),
"class_name": detection.class_name,
"confidence": detection.confidence,
"model_name": detection.model_name,
"model_version": detection.model_version,
"analysis_run_id": str(detection.analysis_run_id) if detection.analysis_run_id else None,
"dataset_id": str(detection.dataset_id) if detection.dataset_id else None,
"job_id": str(detection.job_id) if detection.job_id else None,
"source_tile_path": detection.source_tile_path,
"bbox_json": detection.bbox_json,
}
@staticmethod
def _create_analysis_run(db, project_id, dataset_id, job_id, model, parameters: dict[str, Any]) -> AnalysisRun:
analysis_run = AnalysisRun(
id=uuid.uuid4(),
project_id=project_id,
dataset_id=dataset_id,
job_id=job_id,
analysis_type="detection",
status="running",
model_name=model.model_id,
model_version=model.version,
parameters_json=parameters,
started_at=DetectionService._now(),
)
db.add(analysis_run)
db.commit()
db.refresh(analysis_run)
return analysis_run
@staticmethod
def _mark_failed(db, analysis_run: AnalysisRun, job: Job, code: str, message: str) -> None:
result = {"error_code": code, "message": message, "detection_count": 0}
analysis_run.status = "failed"
analysis_run.finished_at = DetectionService._now()
analysis_run.error_message = message
analysis_run.result_json = result
job.status = "failed"
job.finished_at = analysis_run.finished_at
job.error_message = message
job.result_json = result
db.add(analysis_run)
db.add(job)
db.commit()
db.refresh(analysis_run)
db.refresh(job)
@staticmethod
def _mark_success(db, analysis_run: AnalysisRun, job: Job, detection_count: int) -> None:
result = {"detection_count": detection_count}
analysis_run.status = "success"
analysis_run.finished_at = DetectionService._now()
analysis_run.result_json = result
job.status = "success"
job.finished_at = analysis_run.finished_at
job.result_json = result
db.add(analysis_run)
db.add(job)
db.commit()
db.refresh(analysis_run)
db.refresh(job)
@staticmethod
def _persist_fixture_detections(
db,
project_id: uuid.UUID,
dataset_id: uuid.UUID,
analysis_run: AnalysisRun,
job: Job,
model_name: str,
model_version: str | None,
raw_detections: Any,
confidence_threshold: float,
class_filter: list[str],
) -> list[Detection]:
if not isinstance(raw_detections, list):
raise AppError(code="INVALID_FIXTURE_DETECTIONS", message="fixture_detections must be a list", status_code=400)
persisted: list[Detection] = []
allowed_classes = set(class_filter)
for raw in raw_detections:
if not isinstance(raw, dict):
raise AppError(code="INVALID_FIXTURE_DETECTION", message="Each fixture detection must be an object", status_code=400)
class_name = str(raw.get("class_name") or "")
confidence = float(raw.get("confidence", 0.0))
if allowed_classes and class_name not in allowed_classes:
continue
if confidence < confidence_threshold:
continue
geometry_payload = raw.get("geometry")
if not isinstance(geometry_payload, dict):
raise AppError(code="INVALID_FIXTURE_DETECTION", message="Fixture detection geometry is required", status_code=400)
geometry = shape(geometry_payload)
if geometry.is_empty or not geometry.is_valid:
raise AppError(code="INVALID_FIXTURE_GEOMETRY", message="Fixture detection geometry must be valid", status_code=400)
detection = Detection(
id=uuid.uuid4(),
project_id=project_id,
dataset_id=dataset_id,
analysis_run_id=analysis_run.id,
job_id=job.id,
model_name=model_name,
model_version=model_version,
class_name=class_name,
confidence=confidence,
geometry=from_shape(geometry, srid=4326),
bbox_json=raw.get("bbox_json"),
source_tile_path=raw.get("source_tile_path"),
properties_json=raw.get("properties_json"),
)
db.add(detection)
persisted.append(detection)
db.commit()
for detection in persisted:
db.refresh(detection)
return persisted
@staticmethod
def _run_configured_yolo(
db,
project_id: uuid.UUID,
dataset_id: uuid.UUID,
analysis_run: AnalysisRun,
job: Job,
model_name: str,
model_version: str | None,
tile_manifest_path: str | None,
confidence_threshold: float,
class_filter: list[str],
settings: Settings,
yolo_adapter_class: Type[YoloDetectionAdapter],
) -> list[Detection]:
manifest = DetectionService._load_tile_manifest(tile_manifest_path, settings.yolo_max_tiles)
model_path = Path(settings.yolo_model_path or "").expanduser()
adapter = yolo_adapter_class(settings)
model = adapter.load_model(model_path)
allowed_classes = set(class_filter)
persisted: list[Detection] = []
manifest_crs = manifest.get("crs") or manifest.get("source_crs") or manifest.get("dataset_crs") or "EPSG:4326"
for tile in manifest["tiles"]:
tile_path = DetectionService._resolve_tile_path(tile, Path(tile_manifest_path or "").expanduser())
for raw in adapter.predict_tile(model, tile_path, confidence_threshold):
class_name = str(raw.get("class_name") or "")
confidence = float(raw.get("confidence", 0.0))
if allowed_classes and class_name not in allowed_classes:
continue
if confidence < confidence_threshold:
continue
bbox = raw.get("bbox")
if not isinstance(bbox, list):
raise AppError(code="DETECTION_INVALID_BBOX", message="YOLO adapter returned a detection without bbox", status_code=422)
geometry = pixel_bbox_to_epsg4326_polygon(bbox=bbox, tile=tile, crs=tile.get("crs") or manifest_crs)
detection = Detection(
id=uuid.uuid4(),
project_id=project_id,
dataset_id=dataset_id,
analysis_run_id=analysis_run.id,
job_id=job.id,
model_name=model_name,
model_version=model_version,
class_name=class_name,
confidence=confidence,
geometry=from_shape(geometry, srid=4326),
bbox_json={
"x_min": float(bbox[0]),
"y_min": float(bbox[1]),
"x_max": float(bbox[2]),
"y_max": float(bbox[3]),
},
source_tile_path=str(tile_path),
properties_json={**dict(raw.get("properties") or {}), "tile_index": tile.get("index")},
)
db.add(detection)
persisted.append(detection)
db.commit()
for detection in persisted:
db.refresh(detection)
return persisted
@staticmethod
def _load_tile_manifest(tile_manifest_path: str | None, max_tiles: int) -> dict[str, Any]:
if not tile_manifest_path:
raise AppError(
code="DETECTION_TILE_MANIFEST_REQUIRED",
message="Configured YOLO inference requires an existing raster tile manifest path",
status_code=400,
)
manifest_path = Path(tile_manifest_path).expanduser()
if not manifest_path.exists() or not manifest_path.is_file():
raise AppError(
code="DETECTION_TILE_MANIFEST_NOT_FOUND",
message="Raster tile manifest path does not exist",
details={"tile_manifest_path": str(manifest_path)},
status_code=422,
)
try:
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise AppError(code="DETECTION_TILE_MANIFEST_INVALID", message="Raster tile manifest must be valid JSON", status_code=422) from exc
tiles = manifest.get("tiles")
if not isinstance(tiles, list) or not tiles:
raise AppError(code="DETECTION_TILE_MANIFEST_INVALID", message="Raster tile manifest must contain tiles", status_code=422)
if len(tiles) > max_tiles:
raise AppError(
code="DETECTION_TILE_LIMIT_EXCEEDED",
message="Raster tile manifest exceeds configured YOLO tile limit",
details={"tile_count": len(tiles), "max_tiles": max_tiles},
status_code=422,
)
return manifest
@staticmethod
def _resolve_tile_path(tile: dict[str, Any], manifest_path: Path) -> Path:
raw_path = tile.get("path")
if not isinstance(raw_path, str) or not raw_path:
raise AppError(code="DETECTION_TILE_MANIFEST_INVALID", message="Tile manifest entries require a path", status_code=422)
tile_path = Path(raw_path).expanduser()
if not tile_path.is_absolute():
tile_path = manifest_path.parent / tile_path
if not tile_path.exists() or not tile_path.is_file():
raise AppError(
code="DETECTION_TILE_NOT_FOUND",
message="Tile referenced by manifest does not exist",
details={"tile_path": str(tile_path)},
status_code=422,
)
return tile_path
+431
View File
@@ -0,0 +1,431 @@
from __future__ import annotations
import json
import re
import uuid
from html import escape
from pathlib import Path
from typing import Any
from sqlalchemy.orm import Session
from app.core.errors import AppError
from app.models import AnalysisRun, Dataset, Export, Project, QualityCheck
from app.schemas.export import ExportContentResponse, ExportCreateResponse, ExportListResponse, ExportRead
from app.services.dataset_service import DatasetService
from app.services.detection_service import DetectionService
from app.services.segmentation_service import SegmentationService
from app.services.storage_service import StorageService
class ExportService:
@staticmethod
def export_dataset_geojson(db: Session, dataset_id: uuid.UUID, name: str | None = None) -> ExportCreateResponse:
dataset = db.get(Dataset, dataset_id)
if not dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Dataset not found", status_code=404)
if dataset.dataset_type not in DatasetService.VECTOR_TYPES:
raise AppError(
code="INVALID_DATASET_TYPE",
message="GeoJSON dataset export requires a vector dataset",
details={"dataset_type": dataset.dataset_type},
status_code=400,
)
feature_collection = DatasetService.get_dataset_geojson(db, dataset_id)
filename = ExportService._filename(name, f"{dataset.id}.geojson", ".geojson")
export_path = StorageService.dataset_export_path(str(dataset.project_id), str(dataset.id), filename)
metadata = {
"source": "dataset",
"dataset_id": str(dataset.id),
"project_id": str(dataset.project_id),
"dataset_type": dataset.dataset_type,
"feature_count": len(feature_collection.get("features", [])),
}
export = ExportService._write_json_export(
db,
project_id=dataset.project_id,
analysis_run_id=None,
export_type="dataset_geojson",
storage_path=export_path,
content=feature_collection,
metadata=metadata,
)
return ExportService._create_response(export)
@staticmethod
def export_detection_run_geojson(db: Session, analysis_run_id: uuid.UUID, name: str | None = None) -> ExportCreateResponse:
run = db.get(AnalysisRun, analysis_run_id)
if not run or run.analysis_type != "detection":
raise AppError(code="DETECTION_RUN_NOT_FOUND", message="Detection run not found", status_code=404)
feature_collection = DetectionService.detections_to_geojson(db, analysis_run_id=analysis_run_id)
filename = ExportService._filename(name, f"{run.id}-detections.geojson", ".geojson")
export_path = StorageService.dataset_export_path(str(run.project_id), str(run.dataset_id or run.id), filename)
metadata = {
"source": "detection_run",
"analysis_run_id": str(run.id),
"project_id": str(run.project_id),
"dataset_id": str(run.dataset_id) if run.dataset_id else None,
"feature_count": len(feature_collection.get("features", [])),
}
export = ExportService._write_json_export(
db,
project_id=run.project_id,
analysis_run_id=run.id,
export_type="detection_geojson",
storage_path=export_path,
content=feature_collection,
metadata=metadata,
)
return ExportService._create_response(export)
@staticmethod
def export_segmentation_run_geojson(db: Session, analysis_run_id: uuid.UUID, name: str | None = None) -> ExportCreateResponse:
run = db.get(AnalysisRun, analysis_run_id)
if not run or run.analysis_type != "segmentation":
raise AppError(code="SEGMENTATION_RUN_NOT_FOUND", message="Segmentation run not found", status_code=404)
feature_collection = SegmentationService.segmentations_to_geojson(db, analysis_run_id=analysis_run_id)
filename = ExportService._filename(name, f"{run.id}-segmentations.geojson", ".geojson")
export_path = StorageService.dataset_export_path(str(run.project_id), str(run.dataset_id or run.id), filename)
metadata = {
"source": "segmentation_run",
"analysis_run_id": str(run.id),
"project_id": str(run.project_id),
"dataset_id": str(run.dataset_id) if run.dataset_id else None,
"feature_count": len(feature_collection.get("features", [])),
}
export = ExportService._write_json_export(
db,
project_id=run.project_id,
analysis_run_id=run.id,
export_type="segmentation_geojson",
storage_path=export_path,
content=feature_collection,
metadata=metadata,
)
return ExportService._create_response(export)
@staticmethod
def export_project_metadata(db: Session, project_id: uuid.UUID, name: str | None = None) -> ExportCreateResponse:
project = db.get(Project, project_id)
if not project:
raise AppError(code="PROJECT_NOT_FOUND", message="Project not found", status_code=404)
content = ExportService._project_summary(db, project)
filename = ExportService._filename(name, f"{project.id}-metadata.json", ".json")
export_path = StorageService.dataset_export_path(str(project.id), "project", filename)
metadata = {
"source": "project_metadata",
"project_id": str(project.id),
"dataset_count": len(content["datasets"]),
"quality_check_count": len(content["quality_checks"]),
"export_count": len(content["exports"]),
}
export = ExportService._write_json_export(
db,
project_id=project.id,
analysis_run_id=None,
export_type="project_metadata_json",
storage_path=export_path,
content=content,
metadata=metadata,
)
return ExportService._create_response(export)
@staticmethod
def export_project_report(db: Session, project_id: uuid.UUID, name: str | None = None) -> ExportCreateResponse:
project = db.get(Project, project_id)
if not project:
raise AppError(code="PROJECT_NOT_FOUND", message="Project not found", status_code=404)
summary = ExportService._project_summary(db, project)
html = ExportService._render_project_report_html(summary)
filename = ExportService._filename(name, f"{project.id}-report.html", ".html")
export_path = StorageService.dataset_export_path(str(project.id), "project", filename)
metadata = {
"source": "project_report",
"project_id": str(project.id),
"dataset_count": len(summary["datasets"]),
"quality_check_count": len(summary["quality_checks"]),
"export_count": len(summary["exports"]),
"format": "html",
}
export = ExportService._write_text_export(
db,
project_id=project.id,
analysis_run_id=None,
export_type="project_report_html",
storage_path=export_path,
content=html,
metadata=metadata,
)
return ExportService._create_response(export)
@staticmethod
def list_project_exports(db: Session, project_id: uuid.UUID, limit: int = 50, offset: int = 0) -> ExportListResponse:
if not db.get(Project, project_id):
raise AppError(code="PROJECT_NOT_FOUND", message="Project not found", status_code=404)
query = db.query(Export).filter(Export.project_id == project_id).order_by(Export.created_at.desc())
rows = query.offset(offset).limit(limit).all()
total = query.count()
return ExportListResponse(
items=[ExportRead.model_validate(row) for row in rows],
total=total,
limit=limit,
offset=offset,
)
@staticmethod
def get_export(db: Session, export_id: uuid.UUID) -> ExportRead:
export = db.get(Export, export_id)
if not export:
raise AppError(code="EXPORT_NOT_FOUND", message="Export not found", status_code=404)
return ExportRead.model_validate(export)
@staticmethod
def get_export_content(db: Session, export_id: uuid.UUID) -> ExportContentResponse:
export = db.get(Export, export_id)
if not export:
raise AppError(code="EXPORT_NOT_FOUND", message="Export not found", status_code=404)
path = ExportService.get_export_download_path(db, export_id)
try:
content = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise AppError(code="EXPORT_CONTENT_INVALID", message="Export artifact is not valid JSON", status_code=422) from exc
return ExportContentResponse(export_id=export.id, export_type=export.export_type, content=content)
@staticmethod
def get_export_download_path(db: Session, export_id: uuid.UUID) -> Path:
export = db.get(Export, export_id)
if not export:
raise AppError(code="EXPORT_NOT_FOUND", message="Export not found", status_code=404)
path = Path(export.storage_path)
if not path.exists() or not path.is_file():
raise AppError(
code="EXPORT_CONTENT_NOT_FOUND",
message="Export artifact is missing from storage",
details={"storage_path": export.storage_path},
status_code=404,
)
return path
@staticmethod
def _write_json_export(
db: Session,
*,
project_id: uuid.UUID,
analysis_run_id: uuid.UUID | None,
export_type: str,
storage_path: str,
content: dict[str, Any],
metadata: dict[str, Any],
) -> Export:
path = Path(storage_path)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(content, ensure_ascii=False, indent=2), encoding="utf-8")
return ExportService._persist_export(
db,
project_id=project_id,
analysis_run_id=analysis_run_id,
export_type=export_type,
storage_path=str(path),
metadata=metadata,
)
@staticmethod
def _write_text_export(
db: Session,
*,
project_id: uuid.UUID,
analysis_run_id: uuid.UUID | None,
export_type: str,
storage_path: str,
content: str,
metadata: dict[str, Any],
) -> Export:
path = Path(storage_path)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
return ExportService._persist_export(
db,
project_id=project_id,
analysis_run_id=analysis_run_id,
export_type=export_type,
storage_path=str(path),
metadata=metadata,
)
@staticmethod
def _persist_export(
db: Session,
*,
project_id: uuid.UUID,
analysis_run_id: uuid.UUID | None,
export_type: str,
storage_path: str,
metadata: dict[str, Any],
) -> Export:
export = Export(
id=uuid.uuid4(),
project_id=project_id,
analysis_run_id=analysis_run_id,
export_type=export_type,
storage_path=storage_path,
metadata_json=metadata,
)
db.add(export)
db.commit()
db.refresh(export)
return export
@staticmethod
def _project_summary(db: Session, project: Project) -> dict[str, Any]:
datasets = db.query(Dataset).filter(Dataset.project_id == project.id).order_by(Dataset.created_at.desc()).all()
quality_checks = (
db.query(QualityCheck)
.filter(QualityCheck.project_id == project.id)
.order_by(QualityCheck.created_at.desc())
.all()
)
exports = db.query(Export).filter(Export.project_id == project.id).order_by(Export.created_at.desc()).all()
return {
"project": {
"id": str(project.id),
"name": project.name,
"description": project.description,
"region": project.region,
"status": project.status,
},
"datasets": [
{
"id": str(dataset.id),
"name": dataset.name,
"dataset_type": dataset.dataset_type,
"dataset_role": dataset.dataset_role,
"source_name": dataset.source_name,
"reference_layer_name": dataset.reference_layer_name,
"status": dataset.status,
"crs": dataset.crs,
"bounds_json": dataset.bounds_json,
"feature_count": (dataset.metadata_json or {}).get("feature_count"),
}
for dataset in datasets
],
"quality_checks": [
{
"id": str(check.id),
"analysis_run_id": str(check.analysis_run_id) if check.analysis_run_id else None,
"candidate_dataset_id": str(check.candidate_dataset_id) if check.candidate_dataset_id else None,
"reference_dataset_id": str(check.reference_dataset_id),
"check_type": check.check_type,
"status": check.status,
"score": check.score,
}
for check in quality_checks
],
"exports": [
{
"id": str(export.id),
"analysis_run_id": str(export.analysis_run_id) if export.analysis_run_id else None,
"export_type": export.export_type,
"storage_path": export.storage_path,
"metadata_json": export.metadata_json,
"created_at": export.created_at.isoformat() if export.created_at else None,
}
for export in exports
],
}
@staticmethod
def _render_project_report_html(summary: dict[str, Any]) -> str:
project = summary["project"]
datasets = summary["datasets"]
quality_checks = summary["quality_checks"]
exports = summary["exports"]
dataset_rows = "\n".join(
"<tr>"
f"<td>{escape(str(item['name']))}</td>"
f"<td>{escape(str(item['dataset_type']))}</td>"
f"<td>{escape(str(item['dataset_role']))}</td>"
f"<td>{escape(str(item['status']))}</td>"
f"<td>{escape(str(item['feature_count'] if item['feature_count'] is not None else 'n/a'))}</td>"
"</tr>"
for item in datasets
)
quality_rows = "\n".join(
"<tr>"
f"<td>{escape(str(item['check_type']))}</td>"
f"<td>{escape(str(item['status']))}</td>"
f"<td>{escape(str(item['score'] if item['score'] is not None else 'n/a'))}</td>"
f"<td>{escape(str(item['reference_dataset_id']))}</td>"
"</tr>"
for item in quality_checks
)
export_rows = "\n".join(
"<tr>"
f"<td>{escape(str(item['export_type']))}</td>"
f"<td>{escape(str(item['storage_path']))}</td>"
f"<td>{escape(str(item['created_at'] or 'n/a'))}</td>"
"</tr>"
for item in exports
)
return f"""<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>GeoIntel Project Report - {escape(str(project["name"]))}</title>
<style>
body {{ font-family: Arial, sans-serif; color: #0f172a; margin: 2rem; }}
h1, h2 {{ margin-bottom: 0.4rem; }}
table {{ width: 100%; border-collapse: collapse; margin: 1rem 0 2rem; }}
th, td {{ border: 1px solid #cbd5e1; padding: 0.5rem; text-align: left; }}
th {{ background: #e2e8f0; }}
.muted {{ color: #475569; }}
</style>
</head>
<body>
<h1>{escape(str(project["name"]))}</h1>
<p class="muted">GeoIntel project report artifact</p>
<p>Region: {escape(str(project["region"]))}</p>
<p>Status: {escape(str(project["status"]))}</p>
<p>Description: {escape(str(project["description"] or "n/a"))}</p>
<h2>Datasets ({len(datasets)})</h2>
<table>
<thead><tr><th>Name</th><th>Type</th><th>Role</th><th>Status</th><th>Features</th></tr></thead>
<tbody>{dataset_rows or '<tr><td colspan="5">No datasets</td></tr>'}</tbody>
</table>
<h2>QA/QC Results ({len(quality_checks)})</h2>
<table>
<thead><tr><th>Check</th><th>Status</th><th>Score</th><th>Reference dataset</th></tr></thead>
<tbody>{quality_rows or '<tr><td colspan="4">No QA/QC results</td></tr>'}</tbody>
</table>
<h2>Export History ({len(exports)})</h2>
<table>
<thead><tr><th>Type</th><th>Storage path</th><th>Created</th></tr></thead>
<tbody>{export_rows or '<tr><td colspan="3">No exports</td></tr>'}</tbody>
</table>
</body>
</html>
"""
@staticmethod
def _create_response(export: Export) -> ExportCreateResponse:
return ExportCreateResponse(
export_id=export.id,
path=export.storage_path,
status="ready",
export_type=export.export_type,
metadata_json=export.metadata_json,
)
@staticmethod
def _filename(name: str | None, fallback: str, suffix: str) -> str:
raw_name = name or fallback
cleaned = re.sub(r"[^A-Za-z0-9_.-]+", "_", raw_name).strip("._")
if not cleaned:
cleaned = fallback
if not cleaned.lower().endswith(suffix):
cleaned = f"{cleaned}{suffix}"
return cleaned
+134
View File
@@ -0,0 +1,134 @@
from __future__ import annotations
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import json
from pyproj import Transformer, CRS
from shapely.geometry import shape
from shapely.geometry.base import BaseGeometry
from shapely.ops import unary_union
from shapely.ops import transform as _transform_geometry
from shapely.validation import make_valid
def parse_geojson_payload(raw_text: str | dict[str, Any]) -> dict[str, Any]:
if isinstance(raw_text, dict):
payload = raw_text
else:
try:
payload = json.loads(raw_text)
except Exception as exc:
raise ValueError("Uploaded dataset is not valid JSON") from exc
if not isinstance(payload, dict) or payload.get("type") != "FeatureCollection":
raise ValueError("Upload must be a GeoJSON FeatureCollection")
features = payload.get("features") or []
if not isinstance(features, list):
raise ValueError("FeatureCollection features is invalid")
geometry_types: set[str] = set()
geometries = []
invalid_features = 0
polygon_area_m2: float | None = None
crs_assumed = None
for feature in features:
if not isinstance(feature, dict):
continue
geometry = feature.get("geometry")
if not geometry:
continue
try:
geom = shape(geometry)
except Exception as exc:
raise ValueError("Invalid feature geometry") from exc
if not geom.is_valid:
geom = make_valid(geom)
if not geom.is_valid:
invalid_features += 1
raise ValueError("Invalid geometry remains after repair")
geometry_types.add(str(geom.geom_type))
geometries.append(geom)
if geometries:
unioned = unary_union(geometries)
bounds = unioned.bounds
bounds_json = {
"min_x": float(bounds[0]),
"min_y": float(bounds[1]),
"max_x": float(bounds[2]),
"max_y": float(bounds[3]),
}
else:
bounds_json = None
crs = None
crs_assumed = False
raw_crs = payload.get("crs")
if isinstance(raw_crs, dict):
raw_name = raw_crs.get("properties", {}).get("name")
if isinstance(raw_name, str):
crs = raw_name
elif isinstance(raw_crs, str):
crs = raw_crs
if not crs:
crs = "EPSG:4326"
crs_assumed = True
polygon_area_m2 = _approximate_polygon_area_m2(geometries, crs)
return {
"feature_count": len(features),
"geometry_types": sorted(geometry_types),
"bounds_json": bounds_json,
"approximate_area_m2": polygon_area_m2,
"invalid_features": invalid_features,
"crs": crs,
"crs_assumed": crs_assumed,
"extracted_at": datetime.now(timezone.utc).isoformat(),
"feature_geometry_count": len(geometries),
}
def load_dataset_text(file_path: str) -> str:
return Path(file_path).read_text(encoding="utf-8")
def _approximate_polygon_area_m2(geometries: list[BaseGeometry], crs: str | None) -> float | None:
if not geometries:
return 0.0
try:
polygons = [geometry for geometry in geometries if geometry.geom_type.lower() in {"polygon", "multipolygon"}]
if not polygons:
return None
target_crs = CRS.from_epsg(31370)
source_crs = _crs_to_epsg(crs)
transformer = Transformer.from_crs(source_crs, target_crs, always_xy=True)
projected = [_transform_polygon_for_area(geometry, transformer) for geometry in polygons]
area = sum(item.area for item in projected)
if area < 0:
area = 0.0
return float(area)
except Exception:
return None
def _crs_to_epsg(value: str | None) -> str:
if not value:
return "EPSG:4326"
normalized = value.upper().strip().replace(" ", "")
if normalized.startswith("EPSG:"):
return normalized
if normalized.replace("-", "").isdigit():
return f"EPSG:{normalized}"
return "EPSG:4326"
def _transform_polygon_for_area(geometry: BaseGeometry, transformer: Transformer):
if geometry.is_empty:
return geometry
if geometry.geom_type.lower() in {"polygon", "multipolygon"}:
return _transform_geometry(transformer.transform, geometry)
return geometry
+190
View File
@@ -0,0 +1,190 @@
from __future__ import annotations
import uuid
from datetime import datetime, timezone
from collections.abc import Callable
from typing import Any
from app.core.errors import AppError
from app.models import Job
from app.schemas.job import JobCreate, JobRead
class JobService:
VALID_STATUSES = {"queued", "running", "success", "failed"}
@staticmethod
def run_sync_job(
db,
project_id: uuid.UUID,
job_type: str,
parameters: dict[str, Any] | None,
operation: Callable[[], Any],
input_dataset_id: uuid.UUID | None = None,
) -> dict[str, Any]:
created = JobService.create_job(
db,
JobCreate(
job_type=job_type,
project_id=project_id,
input_dataset_id=input_dataset_id,
parameters_json=JobService._coerce_payload(parameters),
),
)
try:
JobService.mark_running(db, created.id)
result = operation()
output_dataset_id = None
if isinstance(result, uuid.UUID):
output_dataset_id = result
result = {"output_dataset_id": str(result)}
if isinstance(result, dict):
candidate_output_dataset_id = result.get("output_dataset_id")
if isinstance(candidate_output_dataset_id, str):
try:
output_dataset_id = uuid.UUID(candidate_output_dataset_id)
except ValueError:
output_dataset_id = None
elif isinstance(candidate_output_dataset_id, uuid.UUID):
output_dataset_id = candidate_output_dataset_id
if isinstance(result, dict):
job = JobService.mark_success(db, created.id, result=result, output_dataset_id=output_dataset_id)
else:
job = JobService.mark_success(db, created.id, result={"result": result}, output_dataset_id=output_dataset_id)
job_payload = job.model_dump()
if isinstance(job_payload.get("output_dataset_id"), uuid.UUID):
job_payload["output_dataset_id"] = str(job_payload["output_dataset_id"])
result_json = job_payload.get("result_json")
if isinstance(result_json, dict):
if isinstance(result_json.get("output_dataset_id"), uuid.UUID):
result_json["output_dataset_id"] = str(result_json["output_dataset_id"])
job_payload["result_json"] = result_json
return job_payload
except AppError as exc:
failed = JobService.mark_failed(
db,
created.id,
error_message=exc.message,
details={"code": exc.code, "details": exc.details},
)
payload = failed.model_dump()
if isinstance(payload.get("output_dataset_id"), uuid.UUID):
payload["output_dataset_id"] = str(payload["output_dataset_id"])
result_json = payload.get("result_json")
if isinstance(result_json, dict):
if isinstance(result_json.get("output_dataset_id"), uuid.UUID):
result_json["output_dataset_id"] = str(result_json["output_dataset_id"])
payload["result_json"] = result_json
raise
@staticmethod
def _coerce_payload(payload: dict[str, Any] | None) -> dict[str, Any]:
return dict(payload or {})
@staticmethod
def create_job(db, payload: JobCreate) -> JobRead:
job = Job(
id=uuid.uuid4(),
job_type=payload.job_type,
status="queued",
project_id=payload.project_id,
dataset_id=payload.dataset_id,
input_dataset_id=payload.input_dataset_id,
output_dataset_id=payload.output_dataset_id,
parameters_json=JobService._coerce_payload(payload.parameters_json),
result_json=None,
error_message=None,
)
db.add(job)
db.commit()
db.refresh(job)
return JobRead.model_validate(job)
@staticmethod
def mark_running(db, job_id: uuid.UUID) -> JobRead:
job = JobService._get_job(db, job_id)
job.status = "running"
job.started_at = datetime.now(timezone.utc)
job.error_message = None
db.add(job)
db.commit()
db.refresh(job)
return JobRead.model_validate(job)
@staticmethod
def mark_success(
db,
job_id: uuid.UUID,
result: dict[str, Any] | None = None,
output_dataset_id: uuid.UUID | None = None,
) -> JobRead:
job = JobService._get_job(db, job_id)
job.status = "success"
job.finished_at = datetime.now(timezone.utc)
if output_dataset_id is not None:
job.output_dataset_id = output_dataset_id
job.result_json = result
job.error_message = None
db.add(job)
db.commit()
db.refresh(job)
return JobRead.model_validate(job)
@staticmethod
def mark_failed(db, job_id: uuid.UUID, error_message: str, details: dict[str, Any] | None = None) -> JobRead:
job = JobService._get_job(db, job_id)
job.status = "failed"
job.finished_at = datetime.now(timezone.utc)
if details:
job.result_json = details
job.error_message = error_message
db.add(job)
db.commit()
db.refresh(job)
return JobRead.model_validate(job)
@staticmethod
def get_job(db, job_id: uuid.UUID) -> JobRead:
return JobRead.model_validate(JobService._get_job(db, job_id))
@staticmethod
def get_job_status(db, job_id: uuid.UUID) -> dict:
job = JobService._get_job(db, job_id)
return {
"id": job.id,
"project_id": str(job.project_id),
"status": job.status,
"error_message": job.error_message,
"started_at": job.started_at,
"finished_at": job.finished_at,
"result_json": job.result_json,
}
@staticmethod
def list_jobs(
db,
project_id: uuid.UUID | None = None,
dataset_id: uuid.UUID | None = None,
limit: int = 50,
offset: int = 0,
) -> tuple[list[JobRead], int]:
query = db.query(Job)
if project_id is not None:
query = query.filter(Job.project_id == project_id)
if dataset_id is not None:
query = query.filter((Job.dataset_id == dataset_id) | (Job.input_dataset_id == dataset_id) | (Job.output_dataset_id == dataset_id))
total = query.count()
rows = query.order_by(Job.created_at.desc()).offset(offset).limit(limit).all()
return [JobRead.model_validate(row) for row in rows], total
@staticmethod
def _get_job(db, job_id: uuid.UUID) -> Job:
job = db.get(Job, job_id)
if not job:
raise AppError(code="JOB_NOT_FOUND", message="Job not found", status_code=404)
return job
@staticmethod
def validate_status(status: str) -> None:
if status not in JobService.VALID_STATUSES:
raise AppError(code="INVALID_JOB_STATUS", message="Invalid job status", status_code=400)
@@ -0,0 +1,144 @@
from __future__ import annotations
from pathlib import Path
from typing import Type
from app.core.config import Settings, get_settings
from app.schemas.detection import DetectionModelCapability
from app.services.yolo_adapter import YoloDetectionAdapter
class ModelRegistryService:
@staticmethod
def list_model_capabilities(
settings: Settings | None = None,
yolo_adapter_class: Type[YoloDetectionAdapter] = YoloDetectionAdapter,
task_type: str = "object_detection",
) -> list[DetectionModelCapability]:
resolved_settings = settings or get_settings()
if task_type == "segmentation":
return ModelRegistryService.list_segmentation_model_capabilities()
if task_type != "object_detection":
return []
return [
DetectionModelCapability(
model_id="yolo-placeholder",
display_name="YOLO detector placeholder",
framework="ultralytics/pytorch",
task_type="object_detection",
supported_classes=["building", "road", "water", "landuse"],
configured=False,
status="not_configured",
limitation_message="YOLO/PyTorch inference is not configured in Sprint 8; no model is downloaded or executed.",
version=None,
),
ModelRegistryService._configured_yolo_capability(resolved_settings, yolo_adapter_class),
DetectionModelCapability(
model_id="manual-fixture-detector",
display_name="Manual fixture detector",
framework="fixture",
task_type="object_detection",
supported_classes=["building"],
configured=True,
status="configured",
limitation_message="Fixture detector is for explicit tests/demo fixtures only and is not production inference.",
version="fixture-v1",
),
]
@staticmethod
def get_model_capability(
model_id: str,
settings: Settings | None = None,
yolo_adapter_class: Type[YoloDetectionAdapter] = YoloDetectionAdapter,
task_type: str = "object_detection",
) -> DetectionModelCapability | None:
normalized = model_id.strip()
for model in ModelRegistryService.list_model_capabilities(settings=settings, yolo_adapter_class=yolo_adapter_class, task_type=task_type):
if model.model_id == normalized:
return model
return None
@staticmethod
def list_segmentation_model_capabilities() -> list[DetectionModelCapability]:
return [
DetectionModelCapability(
model_id="segmentation-placeholder",
display_name="Segmentation placeholder",
framework="placeholder",
task_type="segmentation",
supported_classes=["building", "vegetation", "water", "landuse"],
configured=False,
status="not_configured",
limitation_message="Segmentation inference is not configured in Sprint 9; no SAM/YOLO-seg model is downloaded or executed.",
version=None,
),
DetectionModelCapability(
model_id="fixture-segmenter",
display_name="Fixture segmenter",
framework="fixture",
task_type="segmentation",
supported_classes=["building", "vegetation", "water", "landuse"],
configured=True,
status="configured",
limitation_message="Fixture segmenter is for explicit tests/demo fixtures only and is not production inference.",
version="fixture-v1",
),
DetectionModelCapability(
model_id="yolo-seg-configured",
display_name="Configured YOLO segmentation",
framework="ultralytics/pytorch",
task_type="segmentation",
supported_classes=["building", "vegetation", "water", "landuse"],
configured=False,
status="not_configured",
limitation_message="YOLO-seg is not configured in Sprint 9. GeoIntel will not download segmentation model weights automatically.",
version=None,
),
DetectionModelCapability(
model_id="sam-configured",
display_name="Configured SAM segmentation",
framework="sam",
task_type="segmentation",
supported_classes=["building", "vegetation", "water", "landuse"],
configured=False,
status="not_configured",
limitation_message="SAM is not configured in Sprint 9 and is not installed as a backend dependency.",
version=None,
),
]
@staticmethod
def _configured_yolo_capability(
settings: Settings,
yolo_adapter_class: Type[YoloDetectionAdapter],
) -> DetectionModelCapability:
configured = False
status = "not_configured"
limitation = "YOLO is disabled. Set YOLO_ENABLED=true and YOLO_MODEL_PATH to a local model file to enable inference."
model_path = Path(settings.yolo_model_path).expanduser() if settings.yolo_model_path else None
if settings.yolo_enabled:
if not yolo_adapter_class.dependencies_available():
status = "dependency_unavailable"
limitation = "YOLO dependencies are not installed. Install backend optional extras with geointel-backend[ai]."
elif model_path is None:
limitation = "YOLO_MODEL_PATH is not set. GeoIntel will not download model weights automatically."
elif not model_path.exists() or not model_path.is_file():
limitation = "YOLO_MODEL_PATH does not point to an existing local model file. GeoIntel will not download model weights automatically."
else:
configured = True
status = "configured"
limitation = "Configured for local YOLO inference over an existing raster tile manifest."
return DetectionModelCapability(
model_id=settings.yolo_model_id,
display_name=settings.yolo_model_display_name,
framework="ultralytics/pytorch",
task_type="object_detection",
supported_classes=["building", "road", "water", "landuse"],
configured=configured,
status=status,
limitation_message=limitation,
version=settings.yolo_model_version,
)
+64
View File
@@ -0,0 +1,64 @@
from __future__ import annotations
import uuid
from sqlalchemy.orm import Session
from app.core.errors import AppError
from app.models import Project
from app.schemas.project import ProjectCreate, ProjectRead, ProjectUpdate
class ProjectService:
@staticmethod
def list_projects(db: Session, limit: int = 50, offset: int = 0) -> tuple[list[ProjectRead], int]:
query = db.query(Project).filter(Project.status != "deleted").order_by(Project.created_at.desc())
total = query.count()
items = query.offset(offset).limit(limit).all()
return [ProjectRead.model_validate(item) for item in items], total
@staticmethod
def create_project(db: Session, payload: ProjectCreate) -> ProjectRead:
project = Project(name=payload.name.strip(), description=(payload.description or "").strip() or None, region=payload.region or "Kempen")
db.add(project)
db.commit()
db.refresh(project)
return ProjectRead.model_validate(project)
@staticmethod
def get_project(db: Session, project_id: uuid.UUID) -> ProjectRead | None:
project = db.get(Project, project_id)
if not project or project.status == "deleted":
return None
return ProjectRead.model_validate(project)
@staticmethod
def update_project(db: Session, project_id: uuid.UUID, payload: ProjectUpdate) -> ProjectRead | None:
project = db.get(Project, project_id)
if not project or project.status == "deleted":
return None
payload_data = payload.model_dump(exclude_unset=True)
changed = False
for key, value in payload_data.items():
if value is None:
continue
setattr(project, key, value)
changed = True
if not changed:
raise AppError(code="NO_CHANGES", message="No updatable fields provided", status_code=422)
db.add(project)
db.commit()
db.refresh(project)
return ProjectRead.model_validate(project)
@staticmethod
def delete_project(db: Session, project_id: uuid.UUID) -> bool:
project = db.get(Project, project_id)
if not project or project.status == "deleted":
return False
project.status = "deleted"
db.add(project)
db.commit()
return True
+248
View File
@@ -0,0 +1,248 @@
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any
from uuid import UUID
from geoalchemy2.shape import to_shape
from shapely.geometry import GeometryCollection
from shapely.geometry.base import BaseGeometry
from shapely.ops import unary_union
from shapely.validation import make_valid
from shapely.geometry import shape
from app.core.errors import AppError
from app.models import Area, Dataset
from app.schemas.qa import QaProviderComparisonResult
from app.services.vector_operations_service import VectorOperationsService
def _extract_crs_warnings(source_dataset: Dataset, reference_dataset: Dataset) -> list[str]:
warnings: list[str] = []
for dataset, label in ((source_dataset, "candidate"), (reference_dataset, "reference")):
metadata = dataset.metadata_json
crs_assumed = None
if isinstance(metadata, dict):
crs_assumed = metadata.get("crs_assumed")
if crs_assumed:
warnings.append(f"CRS assumption is weak for {label} dataset ({dataset.id}); geometry metrics are approximate")
if dataset.crs is None:
warnings.append(f"Missing CRS on {label} dataset ({dataset.id})")
return warnings
class QaService:
SUPPORTED_GEOMETRY_TYPES = {"Polygon", "MultiPolygon"}
@staticmethod
def _load_dataset_payload(db, dataset_id: UUID, *, expected_project_id: UUID | None = None) -> tuple[Dataset, dict[str, Any], list[tuple[dict[str, Any], BaseGeometry]]]:
dataset = db.get(Dataset, dataset_id)
if not dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Dataset not found", status_code=404)
if expected_project_id is not None and dataset.project_id != expected_project_id:
raise AppError(code="INVALID_DATASET_SCOPE", message="Dataset does not belong to this project", status_code=400)
if dataset.dataset_type not in {"vector", "geojson"}:
raise AppError(code="INVALID_DATASET_TYPE", message="Dataset is not a vector dataset", status_code=400)
payload, raw_features = VectorOperationsService._load_dataset_payload(dataset)
geometries = VectorOperationsService._extract_geometries(raw_features)
return dataset, payload, geometries
@staticmethod
def _apply_area_filter(
geometries: list[tuple[dict[str, Any], BaseGeometry]],
area_geometry: BaseGeometry,
*,
dataset_id: UUID,
) -> list[tuple[dict[str, Any], BaseGeometry]]:
area_geom = area_geometry
if isinstance(area_geom, GeometryCollection):
area_geom = unary_union(area_geom.geoms)
filtered: list[tuple[dict[str, Any], BaseGeometry]] = []
for feature, feature_geometry in geometries:
clipped = feature_geometry.intersection(area_geom)
if clipped.is_empty:
continue
if not clipped.is_valid:
clipped = make_valid(clipped)
if not clipped.is_valid:
raise AppError(
code="INVALID_GEOMETRY",
message=f"Area filtering produced invalid geometry for feature in dataset {dataset_id}",
status_code=400,
)
filtered.append((feature, clipped))
return filtered
@staticmethod
def _validate_area(db, area_id: UUID | None, project_id: UUID, *, dataset_ids: tuple[UUID, UUID]) -> BaseGeometry | None:
if not area_id:
return None
area = db.get(Area, area_id)
if not area:
raise AppError(code="AREA_NOT_FOUND", message="Area not found", status_code=404)
if area.project_id != project_id:
raise AppError(code="INVALID_DATASET_SCOPE", message="Area does not belong to this project", status_code=400)
if area.id in dataset_ids:
raise AppError(code="INVALID_PARAMETERS", message="area_id must reference an area, not a dataset", status_code=400)
area_geometry = to_shape(area.geometry)
if area_geometry.is_empty:
raise AppError(code="INVALID_GEOMETRY", message="Area geometry is empty", status_code=400)
return area_geometry
@staticmethod
def _match_io_u_metrics(
source_geometries: list[tuple[dict[str, Any], BaseGeometry]],
reference_geometries: list[tuple[dict[str, Any], BaseGeometry]],
iou_threshold: float,
) -> tuple[int, int, int, list[float], list[str], bool]:
source_supported = [
(feature, geom) for feature, geom in source_geometries if geom.geom_type in QaService.SUPPORTED_GEOMETRY_TYPES
]
reference_supported = [
(feature, geom)
for feature, geom in reference_geometries
if geom.geom_type in QaService.SUPPORTED_GEOMETRY_TYPES
]
unsupported = sorted(
{
geom.geom_type
for _, geom in source_geometries + reference_geometries
if geom.geom_type not in QaService.SUPPORTED_GEOMETRY_TYPES
}
)
if not source_supported or not reference_supported:
return (
0,
len(source_supported),
len(reference_supported),
[],
[f"Unsupported geometry types: {unsupported}"] if unsupported else [],
True,
)
unmatched_reference_indices = set(range(len(reference_supported)))
matches = 0
match_iou_values: list[float] = []
false_positives = 0
for _, source_geom in source_supported:
if source_geom.area <= 0:
false_positives += 1
continue
best_iou = 0.0
best_index = None
for reference_index in list(unmatched_reference_indices):
_, reference_geom = reference_supported[reference_index]
if reference_geom.area <= 0:
unmatched_reference_indices.discard(reference_index)
continue
try:
intersection = source_geom.intersection(reference_geom)
except Exception as exc: # pragma: no cover - robustness path
raise AppError(code="GEOMETRY_OPERATION_UNSUPPORTED", message="Geometry operations failed", details={"reason": str(exc)}, status_code=422)
if intersection.is_empty:
continue
intersection_area = intersection.area
if intersection_area < 0:
intersection_area = 0.0
union_area = source_geom.area + reference_geom.area - intersection_area
if union_area <= 0:
continue
candidate_iou = intersection_area / union_area
if candidate_iou > best_iou:
best_iou = candidate_iou
best_index = reference_index
if best_index is not None and best_iou >= iou_threshold:
matches += 1
match_iou_values.append(best_iou)
unmatched_reference_indices.discard(best_index)
else:
false_positives += 1
false_negatives = len(unmatched_reference_indices)
warnings: list[str] = [f"Unsupported geometry types: {unsupported}"] if unsupported else []
return matches, false_positives, false_negatives, match_iou_values, warnings, bool(unsupported)
@staticmethod
def compare_candidate_with_reference(
db,
project_id: UUID,
candidate_dataset_id: UUID,
reference_dataset_id: UUID,
iou_threshold: float = 0.5,
area_id: UUID | None = None,
) -> QaProviderComparisonResult:
if candidate_dataset_id == reference_dataset_id:
raise AppError(code="INVALID_PARAMETERS", message="Candidate and reference dataset must differ", status_code=400)
candidate_dataset, candidate_payload, candidate_geometries = QaService._load_dataset_payload(
db,
candidate_dataset_id,
expected_project_id=project_id,
)
reference_dataset, reference_payload, reference_geometries = QaService._load_dataset_payload(
db,
reference_dataset_id,
expected_project_id=project_id,
)
area_geometry = QaService._validate_area(
db,
area_id=area_id,
project_id=project_id,
dataset_ids=(candidate_dataset_id, reference_dataset_id),
)
if area_geometry is not None:
candidate_geometries = QaService._apply_area_filter(candidate_geometries, area_geometry, dataset_id=candidate_dataset.id)
reference_geometries = QaService._apply_area_filter(reference_geometries, area_geometry, dataset_id=reference_dataset.id)
matches, false_positives, false_negatives, match_iou_values, warnings, unsupported = QaService._match_io_u_metrics(
candidate_geometries,
reference_geometries,
iou_threshold,
)
candidate_feature_count = len(candidate_payload.get("features", [])) if isinstance(candidate_payload, dict) else 0
reference_feature_count = len(reference_payload.get("features", [])) if isinstance(reference_payload, dict) else 0
mean_iou = None if not match_iou_values else sum(match_iou_values) / len(match_iou_values)
precision = None
if matches + false_positives > 0:
precision = matches / (matches + false_positives)
recall = None
if matches + false_negatives > 0:
recall = matches / (matches + false_negatives)
f1_score = None
if precision is not None and recall is not None and precision + recall > 0:
f1_score = (2 * precision * recall) / (precision + recall)
status = "unsupported" if unsupported else "ok"
return QaProviderComparisonResult(
status=status,
warnings=_extract_crs_warnings(candidate_dataset, reference_dataset) + warnings,
candidate_feature_count=candidate_feature_count,
reference_feature_count=reference_feature_count,
matches=matches,
false_positives=false_positives,
false_negatives=false_negatives,
precision=precision,
recall=recall,
f1_score=f1_score,
mean_iou=mean_iou,
iou_threshold=iou_threshold,
unsupported_geometry=unsupported,
unsupported_geometries=warnings,
generated_at=datetime.now(timezone.utc),
)
@@ -0,0 +1,47 @@
from __future__ import annotations
from uuid import UUID
from sqlalchemy.orm import Session
from app.models import Metric, QualityCheck
from app.schemas.qa import MetricRead, QualityCheckRead
class QualityCheckService:
@staticmethod
def list_quality_checks(
db: Session,
*,
project_id: UUID,
limit: int = 50,
offset: int = 0,
) -> tuple[list[QualityCheckRead], int]:
query = (
db.query(QualityCheck)
.filter(QualityCheck.project_id == project_id)
.order_by(QualityCheck.created_at.desc())
)
total = query.count()
rows = query.offset(offset).limit(limit).all()
if not rows:
return [], total
quality_check_ids = [row.id for row in rows]
metrics_by_quality_check: dict[UUID, list[MetricRead]] = {row.id: [] for row in rows}
metrics = (
db.query(Metric)
.filter(Metric.quality_check_id.in_(quality_check_ids))
.order_by(Metric.created_at.asc())
.all()
)
for metric in metrics:
if metric.quality_check_id in metrics_by_quality_check:
metrics_by_quality_check[metric.quality_check_id].append(MetricRead.model_validate(metric))
return [
QualityCheckRead.model_validate(row).model_copy(
update={"metrics": metrics_by_quality_check.get(row.id, [])}
)
for row in rows
], total
+58
View File
@@ -0,0 +1,58 @@
from __future__ import annotations
from datetime import datetime, timezone
from uuid import UUID, uuid4
from app.models import Metric, QualityCheck
class QualityService:
@staticmethod
def persist_quality_check(
db,
project_id: UUID,
reference_dataset_id: UUID,
check_type: str,
status: str,
score: float | None,
parameters: dict | None,
findings: dict | None,
*,
job_id: UUID | None = None,
analysis_run_id: UUID | None = None,
candidate_dataset_id: UUID | None = None,
metrics: dict[str, float | int | None] | None = None,
commit: bool = True,
) -> QualityCheck:
quality_check = QualityCheck(
id=uuid4(),
project_id=project_id,
job_id=job_id,
analysis_run_id=analysis_run_id,
candidate_dataset_id=candidate_dataset_id,
reference_dataset_id=reference_dataset_id,
check_type=check_type,
status=status,
score=score,
parameters_json=parameters or {},
findings_json=findings or {},
completed_at=datetime.now(timezone.utc),
)
db.add(quality_check)
for key, value in (metrics or {}).items():
db.add(
Metric(
id=uuid4(),
quality_check_id=quality_check.id,
analysis_run_id=analysis_run_id,
metric_key=key,
metric_value=float(value) if value is not None else None,
metadata_json={},
)
)
if commit:
db.commit()
db.refresh(quality_check)
return quality_check
File diff suppressed because it is too large Load Diff
+51
View File
@@ -0,0 +1,51 @@
from __future__ import annotations
from pathlib import Path
from app.core.errors import AppError
def _import_rasterio():
import importlib
rasterio = importlib.import_module("rasterio")
errors = importlib.import_module("rasterio.errors")
return rasterio, errors
def extract_raster_metadata(path: str) -> dict:
try:
rasterio, errors = _import_rasterio()
except Exception as exc: # pragma: no cover - exercised via API-level fallback tests
raise AppError(
code="RASTER_PROCESSING_UNAVAILABLE",
message="Raster processing unavailable. Install rasterio and GDAL-compatible drivers to enable raster metadata extraction.",
status_code=503,
) from exc
dataset_path = Path(path)
try:
with rasterio.open(dataset_path) as dataset:
nodata = dataset.nodata
if isinstance(nodata, (list, tuple)):
nodata_value = [None if value is None else float(value) for value in nodata]
else:
nodata_value = None if nodata is None else float(nodata)
transform = dataset.transform.to_gdal() if hasattr(dataset, "transform") else None
return {
"driver": dataset.driver,
"width": int(dataset.width),
"height": int(dataset.height),
"band_count": int(dataset.count),
"crs": str(dataset.crs) if dataset.crs else None,
"bounds": list(dataset.bounds),
"resolution": list(dataset.res),
"dtype": list(dataset.dtypes),
"nodata": nodata_value,
"transform": list(transform) if transform is not None else None,
}
except Exception as exc:
if isinstance(exc, errors.RasterioIOError):
raise AppError(code="INVALID_RASTER", message="Uploaded raster file is invalid", status_code=400) from exc
raise AppError(code="RASTER_METADATA_ERROR", message="Unable to read raster metadata", status_code=400) from exc
@@ -0,0 +1,48 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Protocol
@dataclass(frozen=True)
class SegmentationAdapterResult:
class_name: str
confidence: float | None
geometry: dict[str, Any]
bbox_json: dict[str, Any] | None = None
mask_path: str | None = None
source_tile_path: str | None = None
tile_index: int | None = None
properties_json: dict[str, Any] | None = None
provenance_json: dict[str, Any] | None = None
area_m2: float | None = None
class SegmentationAdapter(Protocol):
def segment(self, *args: Any, **kwargs: Any) -> list[SegmentationAdapterResult]:
"""Future segmentation adapters must local-import model dependencies inside execution paths."""
class FixtureSegmentationAdapter:
def segment(self, raw_segmentations: Any) -> list[SegmentationAdapterResult]:
if not isinstance(raw_segmentations, list):
return []
results: list[SegmentationAdapterResult] = []
for raw in raw_segmentations:
if not isinstance(raw, dict):
continue
results.append(
SegmentationAdapterResult(
class_name=str(raw.get("class_name") or ""),
confidence=float(raw["confidence"]) if raw.get("confidence") is not None else None,
geometry=raw.get("geometry"),
bbox_json=raw.get("bbox_json"),
mask_path=raw.get("mask_path"),
source_tile_path=raw.get("source_tile_path"),
tile_index=raw.get("tile_index"),
properties_json=raw.get("properties_json"),
provenance_json=raw.get("provenance_json"),
area_m2=raw.get("area_m2"),
)
)
return results
@@ -0,0 +1,512 @@
from __future__ import annotations
import uuid
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from geoalchemy2.shape import from_shape, to_shape
from shapely.geometry import MultiPolygon, Polygon, mapping, shape
from shapely.validation import make_valid
from app.core.config import Settings, get_settings
from app.core.errors import AppError
from app.models import AnalysisRun, Dataset, Job, Project, Segmentation, VectorFeature
from app.schemas.segmentation import (
SegmentationListResponse,
SegmentationRead,
SegmentationRunListResponse,
SegmentationRunRead,
SegmentationRunResponse,
)
from app.services.model_registry_service import ModelRegistryService
from app.services.qa_service import QaService
from app.services.quality_service import QualityService
from app.services.segmentation_adapter import FixtureSegmentationAdapter
class SegmentationService:
@staticmethod
def _now() -> datetime:
return datetime.now(UTC)
@staticmethod
def run_segmentation(
db,
project_id: uuid.UUID,
dataset_id: uuid.UUID,
model_id: str,
confidence_threshold: float,
class_filter: list[str] | None = None,
tile_manifest_path: str | None = None,
parameters_json: dict[str, Any] | None = None,
settings: Settings | None = None,
) -> SegmentationRunResponse:
parameters = dict(parameters_json or {})
resolved_settings = settings or get_settings()
project = db.get(Project, project_id)
if not project:
raise AppError(code="PROJECT_NOT_FOUND", message="Project not found", status_code=404)
dataset = db.get(Dataset, dataset_id)
if not dataset or dataset.project_id != project_id:
raise AppError(code="DATASET_NOT_FOUND", message="Dataset not found", status_code=404)
if dataset.dataset_type != "raster":
raise AppError(
code="INVALID_DATASET_TYPE",
message="Segmentation requires a raster dataset",
details={"dataset_type": dataset.dataset_type},
status_code=400,
)
model = ModelRegistryService.get_model_capability(model_id, task_type="segmentation")
if model is None:
raise AppError(code="SEGMENTATION_MODEL_NOT_FOUND", message="Segmentation model not found", status_code=404)
if model.model_id == "fixture-segmenter" and parameters.get("fixture_mode") is not True:
raise AppError(
code="FIXTURE_MODE_REQUIRED",
message="Fixture segmenter requires explicit fixture_mode=true",
status_code=400,
)
run_parameters = {
"model_id": model.model_id,
"confidence_threshold": confidence_threshold,
"class_filter": class_filter or [],
"tile_manifest_path": tile_manifest_path,
"parameters_json": parameters,
}
job = SegmentationService._create_job(db, project_id, dataset_id, run_parameters)
analysis_run = SegmentationService._create_analysis_run(db, project_id, dataset_id, job.id, model, run_parameters)
if not model.configured:
message = model.limitation_message
SegmentationService._mark_failed(
db,
analysis_run,
job,
code="SEGMENTATION_MODEL_UNAVAILABLE",
message=message,
)
return SegmentationRunResponse(
analysis_run_id=analysis_run.id,
job_id=job.id,
project_id=project_id,
dataset_id=dataset_id,
model_id=model.model_id,
status="failed",
segmentation_count=0,
error_code="SEGMENTATION_MODEL_UNAVAILABLE",
message=message,
)
if model.model_id == "fixture-segmenter":
segmentations = SegmentationService._persist_fixture_segmentations(
db=db,
project_id=project_id,
dataset_id=dataset_id,
analysis_run=analysis_run,
job=job,
model_name=model.model_id,
model_version=model.version,
raw_segmentations=parameters.get("fixture_segmentations"),
confidence_threshold=confidence_threshold,
class_filter=class_filter or [],
settings=resolved_settings,
)
SegmentationService._mark_success(db, analysis_run, job, segmentation_count=len(segmentations))
return SegmentationRunResponse(
analysis_run_id=analysis_run.id,
job_id=job.id,
project_id=project_id,
dataset_id=dataset_id,
model_id=model.model_id,
status="success",
segmentation_count=len(segmentations),
message="Fixture segmentations persisted.",
)
raise AppError(code="SEGMENTATION_MODEL_UNAVAILABLE", message="Segmentation model is unavailable", status_code=503)
@staticmethod
def get_run(db, analysis_run_id: uuid.UUID) -> SegmentationRunRead:
run = db.get(AnalysisRun, analysis_run_id)
if not run or run.analysis_type != "segmentation":
raise AppError(code="SEGMENTATION_RUN_NOT_FOUND", message="Segmentation run not found", status_code=404)
return SegmentationRunRead.model_validate(run)
@staticmethod
def list_runs(
db,
*,
project_id: uuid.UUID | None = None,
dataset_id: uuid.UUID | None = None,
) -> SegmentationRunListResponse:
query = db.query(AnalysisRun).filter(AnalysisRun.analysis_type == "segmentation")
if project_id is not None:
query = query.filter(AnalysisRun.project_id == project_id)
if dataset_id is not None:
query = query.filter(AnalysisRun.dataset_id == dataset_id)
rows = query.order_by(AnalysisRun.created_at.desc()).all()
return SegmentationRunListResponse(items=[SegmentationRunRead.model_validate(row) for row in rows], total=len(rows))
@staticmethod
def list_segmentations(
db,
analysis_run_id: uuid.UUID | None = None,
*,
dataset_id: uuid.UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
) -> SegmentationListResponse:
if analysis_run_id is not None:
run = db.get(AnalysisRun, analysis_run_id)
if not run or run.analysis_type != "segmentation":
raise AppError(code="SEGMENTATION_RUN_NOT_FOUND", message="Segmentation run not found", status_code=404)
rows = SegmentationService._query_segmentation_rows(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
)
items = [SegmentationRead.model_validate(row) for row in rows]
return SegmentationListResponse(items=items, total=len(items))
@staticmethod
def get_segmentation(db, segmentation_id: uuid.UUID) -> SegmentationRead:
segmentation = db.get(Segmentation, segmentation_id)
if not segmentation:
raise AppError(code="SEGMENTATION_NOT_FOUND", message="Segmentation not found", status_code=404)
return SegmentationRead.model_validate(segmentation)
@staticmethod
def segmentations_to_geojson(
db,
*,
analysis_run_id: uuid.UUID | None = None,
dataset_id: uuid.UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
) -> dict[str, Any]:
segmentations = SegmentationService._query_segmentation_rows(
db,
analysis_run_id=analysis_run_id,
dataset_id=dataset_id,
class_name=class_name,
min_confidence=min_confidence,
)
return {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"id": str(segmentation.id),
"properties": SegmentationService._segmentation_properties(segmentation),
"geometry": mapping(to_shape(segmentation.geometry)),
}
for segmentation in segmentations
],
}
@staticmethod
def compare_segmentations_with_reference(
db,
analysis_run_id: uuid.UUID,
reference_dataset_id: uuid.UUID,
iou_threshold: float = 0.5,
class_name: str | None = None,
min_confidence: float | None = None,
) -> dict[str, Any]:
run = db.get(AnalysisRun, analysis_run_id)
if not run or run.analysis_type != "segmentation":
raise AppError(code="SEGMENTATION_RUN_NOT_FOUND", message="Segmentation run not found", status_code=404)
reference_dataset = db.get(Dataset, reference_dataset_id)
if not reference_dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Reference dataset not found", status_code=404)
if reference_dataset.project_id != run.project_id:
raise AppError(code="INVALID_DATASET_SCOPE", message="Reference dataset does not belong to segmentation project", status_code=400)
if reference_dataset.dataset_type not in {"vector", "geojson"}:
raise AppError(code="INVALID_DATASET_TYPE", message="Reference dataset must be vector data", status_code=400)
segmentations = SegmentationService._query_segmentation_rows(
db,
analysis_run_id=analysis_run_id,
dataset_id=run.dataset_id,
class_name=class_name,
min_confidence=min_confidence,
)
if not segmentations:
raise AppError(
code="SEGMENTATIONS_NOT_FOUND",
message="Segmentation run has no persisted geometries for QA",
status_code=422,
)
references = db.query(VectorFeature).filter(VectorFeature.dataset_id == reference_dataset_id).all()
if not references:
raise AppError(
code="REFERENCE_FEATURES_NOT_FOUND",
message="Reference dataset has no persisted vector features for QA",
status_code=422,
)
candidate_geometries = [({"id": str(row.id), "class_name": row.class_name}, to_shape(row.geometry)) for row in segmentations]
reference_geometries = [({"id": str(row.id), "feature_class": row.feature_class}, to_shape(row.geometry)) for row in references]
matches, false_positives, false_negatives, match_iou_values, warnings, unsupported = QaService._match_io_u_metrics(
candidate_geometries,
reference_geometries,
iou_threshold,
)
mean_iou = None if not match_iou_values else sum(match_iou_values) / len(match_iou_values)
precision = matches / (matches + false_positives) if matches + false_positives > 0 else None
recall = matches / (matches + false_negatives) if matches + false_negatives > 0 else None
f1_score = None
if precision is not None and recall is not None:
f1_score = (2 * precision * recall) / (precision + recall) if precision + recall > 0 else 0.0
status = "unsupported" if unsupported else "ok"
quality_check = QualityService.persist_quality_check(
db=db,
project_id=run.project_id,
analysis_run_id=analysis_run_id,
candidate_dataset_id=run.dataset_id,
reference_dataset_id=reference_dataset_id,
check_type="segmentations_vs_reference",
status=status,
score=f1_score,
parameters={
"analysis_run_id": str(analysis_run_id),
"reference_dataset_id": str(reference_dataset_id),
"iou_threshold": iou_threshold,
"class_name": class_name,
"min_confidence": min_confidence,
},
findings={
"matches": matches,
"false_positives": false_positives,
"false_negatives": false_negatives,
"warnings": warnings,
"unsupported_geometry": unsupported,
},
metrics={
"precision": precision,
"recall": recall,
"f1": f1_score,
"mean_iou": mean_iou,
"false_positive_count": false_positives,
"false_negative_count": false_negatives,
},
)
return {
"status": status,
"quality_check_id": str(quality_check.id),
"analysis_run_id": str(analysis_run_id),
"reference_dataset_id": str(reference_dataset_id),
"candidate_feature_count": len(candidate_geometries),
"reference_feature_count": len(reference_geometries),
"matches": matches,
"false_positives": false_positives,
"false_negatives": false_negatives,
"precision": precision,
"recall": recall,
"f1_score": f1_score,
"mean_iou": mean_iou,
"iou_threshold": iou_threshold,
"warnings": warnings,
}
@staticmethod
def mask_artifact_path(storage_root: str, project_id: uuid.UUID, analysis_run_id: uuid.UUID, tile_index: int | None, segmentation_id: uuid.UUID) -> str:
tile_folder = f"tile_{tile_index if tile_index is not None else 0}"
return (Path(storage_root) / "masks" / str(project_id) / str(analysis_run_id) / tile_folder / f"mask_{segmentation_id}.png").as_posix()
@staticmethod
def _create_job(db, project_id: uuid.UUID, dataset_id: uuid.UUID, parameters: dict[str, Any]) -> Job:
job = Job(
id=uuid.uuid4(),
job_type="segmentation.run",
status="running",
project_id=project_id,
dataset_id=dataset_id,
input_dataset_id=dataset_id,
parameters_json=parameters,
started_at=SegmentationService._now(),
)
db.add(job)
db.commit()
db.refresh(job)
return job
@staticmethod
def _create_analysis_run(db, project_id, dataset_id, job_id, model, parameters: dict[str, Any]) -> AnalysisRun:
analysis_run = AnalysisRun(
id=uuid.uuid4(),
project_id=project_id,
dataset_id=dataset_id,
job_id=job_id,
analysis_type="segmentation",
status="running",
model_name=model.model_id,
model_version=model.version,
parameters_json=parameters,
started_at=SegmentationService._now(),
)
db.add(analysis_run)
db.commit()
db.refresh(analysis_run)
return analysis_run
@staticmethod
def _mark_failed(db, analysis_run: AnalysisRun, job: Job, code: str, message: str) -> None:
result = {"error_code": code, "message": message, "segmentation_count": 0}
analysis_run.status = "failed"
analysis_run.finished_at = SegmentationService._now()
analysis_run.error_message = message
analysis_run.result_json = result
job.status = "failed"
job.finished_at = analysis_run.finished_at
job.error_message = message
job.result_json = result
db.add(analysis_run)
db.add(job)
db.commit()
db.refresh(analysis_run)
db.refresh(job)
@staticmethod
def _mark_success(db, analysis_run: AnalysisRun, job: Job, segmentation_count: int) -> None:
result = {"segmentation_count": segmentation_count}
analysis_run.status = "success"
analysis_run.finished_at = SegmentationService._now()
analysis_run.result_json = result
job.status = "success"
job.finished_at = analysis_run.finished_at
job.result_json = result
db.add(analysis_run)
db.add(job)
db.commit()
db.refresh(analysis_run)
db.refresh(job)
@staticmethod
def _persist_fixture_segmentations(
db,
project_id: uuid.UUID,
dataset_id: uuid.UUID,
analysis_run: AnalysisRun,
job: Job,
model_name: str,
model_version: str | None,
raw_segmentations: Any,
confidence_threshold: float,
class_filter: list[str],
settings: Settings,
) -> list[Segmentation]:
if not isinstance(raw_segmentations, list):
raise AppError(code="INVALID_FIXTURE_SEGMENTATIONS", message="fixture_segmentations must be a list", status_code=400)
adapter = FixtureSegmentationAdapter()
adapter_results = adapter.segment(raw_segmentations)
if len(adapter_results) != len(raw_segmentations):
raise AppError(code="INVALID_FIXTURE_SEGMENTATION", message="Each fixture segmentation must be an object", status_code=400)
persisted: list[Segmentation] = []
allowed_classes = set(class_filter)
for raw in adapter_results:
class_name = raw.class_name
confidence = raw.confidence
if allowed_classes and class_name not in allowed_classes:
continue
if confidence is not None and confidence < confidence_threshold:
continue
if not isinstance(raw.geometry, dict):
raise AppError(code="INVALID_FIXTURE_SEGMENTATION", message="Fixture segmentation geometry is required", status_code=400)
geometry = SegmentationService._validated_multipolygon(raw.geometry)
segmentation_id = uuid.uuid4()
mask_path = raw.mask_path or SegmentationService.mask_artifact_path(
settings.storage_root,
project_id,
analysis_run.id,
raw.tile_index,
segmentation_id,
)
segmentation = Segmentation(
id=segmentation_id,
project_id=project_id,
dataset_id=dataset_id,
analysis_run_id=analysis_run.id,
job_id=job.id,
model_name=model_name,
model_version=model_version,
class_name=class_name,
confidence=confidence,
geometry=from_shape(geometry, srid=4326),
bbox_json=raw.bbox_json,
area_m2=raw.area_m2,
mask_path=mask_path,
source_tile_path=raw.source_tile_path,
tile_index=raw.tile_index,
properties_json=raw.properties_json,
provenance_json={**dict(raw.provenance_json or {}), "fixture_mode": True},
)
db.add(segmentation)
persisted.append(segmentation)
db.commit()
for segmentation in persisted:
db.refresh(segmentation)
return persisted
@staticmethod
def _validated_multipolygon(geometry_payload: dict[str, Any]) -> MultiPolygon:
try:
geometry = shape(geometry_payload)
except Exception as exc:
raise AppError(code="INVALID_FIXTURE_GEOMETRY", message="Fixture segmentation geometry must be valid GeoJSON", status_code=400) from exc
if geometry.is_empty:
raise AppError(code="INVALID_FIXTURE_GEOMETRY", message="Fixture segmentation geometry must not be empty", status_code=400)
if not geometry.is_valid:
geometry = make_valid(geometry)
if geometry.is_empty or not geometry.is_valid:
raise AppError(code="INVALID_FIXTURE_GEOMETRY", message="Fixture segmentation geometry must be valid", status_code=400)
if isinstance(geometry, Polygon):
geometry = MultiPolygon([geometry])
if not isinstance(geometry, MultiPolygon):
raise AppError(code="INVALID_FIXTURE_GEOMETRY", message="Fixture segmentation geometry must be Polygon or MultiPolygon", status_code=400)
if geometry.area <= 0:
raise AppError(code="INVALID_FIXTURE_GEOMETRY", message="Fixture segmentation geometry must have positive area", status_code=400)
return geometry
@staticmethod
def _query_segmentation_rows(
db,
*,
analysis_run_id: uuid.UUID | None = None,
dataset_id: uuid.UUID | None = None,
class_name: str | None = None,
min_confidence: float | None = None,
) -> list[Segmentation]:
query = db.query(Segmentation)
if analysis_run_id is not None:
query = query.filter(Segmentation.analysis_run_id == analysis_run_id)
if dataset_id is not None:
query = query.filter(Segmentation.dataset_id == dataset_id)
if class_name:
query = query.filter(Segmentation.class_name == class_name)
if min_confidence is not None:
query = query.filter(Segmentation.confidence >= min_confidence)
return query.order_by(Segmentation.created_at.desc()).all()
@staticmethod
def _segmentation_properties(segmentation: Segmentation) -> dict[str, Any]:
return {
"segmentation_id": str(segmentation.id),
"class_name": segmentation.class_name,
"confidence": segmentation.confidence,
"area_m2": segmentation.area_m2,
"model_name": segmentation.model_name,
"model_version": segmentation.model_version,
"analysis_run_id": str(segmentation.analysis_run_id) if segmentation.analysis_run_id else None,
"dataset_id": str(segmentation.dataset_id) if segmentation.dataset_id else None,
"job_id": str(segmentation.job_id) if segmentation.job_id else None,
"source_tile_path": segmentation.source_tile_path,
"tile_index": segmentation.tile_index,
"mask_path": segmentation.mask_path,
"bbox_json": segmentation.bbox_json,
"provenance_json": segmentation.provenance_json,
}
+136
View File
@@ -0,0 +1,136 @@
from __future__ import annotations
import hashlib
import shutil
from pathlib import Path
from typing import Any
from app.core.config import get_settings
class StorageService:
@staticmethod
def _base_dir() -> Path:
return Path(get_settings().storage_root).resolve()
@staticmethod
def normalize_dataset_type(dataset_type: str) -> str:
normalized = dataset_type.strip().lower()
if normalized == "geojson":
return "vector"
return normalized
@staticmethod
def _safe_filename(value: str) -> str:
value = value.strip().replace("\\", "/").split("/")[-1]
fallback = "upload"
if not value:
return fallback
allowed = []
for char in value:
if char.isalnum() or char in "-_ .":
allowed.append(char)
else:
allowed.append("_")
cleaned = "".join(allowed)
cleaned = cleaned.strip(" .")
return cleaned or fallback
@staticmethod
def dataset_root(project_id: str, dataset_id: str, dataset_type: str) -> Path:
return StorageService._base_dir() / "uploads" / project_id / dataset_type / dataset_id
@staticmethod
def derived_raster_root(project_id: str, dataset_id: str) -> Path:
return StorageService._base_dir() / "rasters" / "derived" / project_id / dataset_id
@staticmethod
def preview_root(project_id: str, dataset_id: str) -> Path:
return StorageService._base_dir() / "previews" / project_id / dataset_id
@staticmethod
def raster_tiles_root(project_id: str, source_dataset_id: str, tile_set_id: str) -> Path:
return StorageService._base_dir() / "tiles" / project_id / source_dataset_id / tile_set_id
@staticmethod
def dataset_file_path(
project_id: str,
dataset_id: str,
dataset_type: str,
original_filename: str,
) -> str:
safe_original = StorageService._safe_filename(original_filename)
stored_filename = f"{dataset_id}_{safe_original}"
return str(StorageService.dataset_root(project_id, dataset_id, dataset_type) / stored_filename)
@staticmethod
def calculate_checksum_sha256(content: bytes) -> str:
digest = hashlib.sha256()
digest.update(content)
return digest.hexdigest()
@staticmethod
def persist_dataset_file(
project_id: str,
dataset_id: str,
dataset_type: str,
original_filename: str,
content: bytes,
content_type: str | None,
) -> dict[str, Any]:
normalized_type = StorageService.normalize_dataset_type(dataset_type)
file_path = Path(StorageService.dataset_file_path(project_id, dataset_id, normalized_type, original_filename))
file_path.parent.mkdir(parents=True, exist_ok=True)
with file_path.open("wb") as stream:
stream.write(content)
metadata: dict[str, Any] = {
"original_filename": StorageService._safe_filename(original_filename),
"stored_filename": file_path.name,
"content_type": content_type or "application/octet-stream",
"size_bytes": len(content),
"checksum_sha256": StorageService.calculate_checksum_sha256(content),
"storage_path": str(file_path),
}
return metadata
@staticmethod
def persist_file(
storage_path: str,
content: bytes,
original_filename: str,
content_type: str | None,
) -> dict[str, Any]:
target = Path(storage_path)
target.parent.mkdir(parents=True, exist_ok=True)
with target.open("wb") as stream:
stream.write(content)
metadata: dict[str, Any] = {
"original_filename": StorageService._safe_filename(original_filename),
"stored_filename": target.name,
"content_type": content_type or "application/octet-stream",
"size_bytes": len(content),
"checksum_sha256": StorageService.calculate_checksum_sha256(content),
"storage_path": str(target),
}
return metadata
@staticmethod
def remove_dataset_file(path: str) -> None:
target = Path(path)
if target.exists():
target.unlink(missing_ok=True)
dataset_parent = target.parent
if dataset_parent.exists() and dataset_parent.is_dir():
has_files = any(dataset_parent.iterdir())
if not has_files:
shutil.rmtree(dataset_parent, ignore_errors=True)
@staticmethod
def dataset_export_path(project_id: str, dataset_id: str, filename: str) -> str:
output_dir = StorageService._base_dir() / "exports" / project_id / "datasets"
output_dir.mkdir(parents=True, exist_ok=True)
return str(output_dir / f"{dataset_id}_{StorageService._safe_filename(filename)}")
@@ -0,0 +1,65 @@
from __future__ import annotations
from typing import Any
from uuid import UUID
from geoalchemy2.shape import from_shape
from shapely.geometry import shape
from shapely.validation import make_valid
from app.core.errors import AppError
from app.models import VectorFeature
class VectorFeatureService:
@staticmethod
def persist_geojson_features(
db,
dataset_id: UUID,
payload: dict[str, Any],
feature_class: str | None = None,
*,
commit: bool = True,
) -> list[VectorFeature]:
features = payload.get("features")
if payload.get("type") != "FeatureCollection" or not isinstance(features, list):
raise AppError(code="INVALID_GEOJSON", message="GeoJSON payload must be a FeatureCollection", status_code=400)
persisted: list[VectorFeature] = []
for index, feature in enumerate(features):
if not isinstance(feature, dict):
raise AppError(code="INVALID_GEOJSON", message=f"Feature {index} must be an object", status_code=400)
geometry_payload = feature.get("geometry")
if geometry_payload is None:
continue
try:
geometry = shape(geometry_payload)
except Exception as exc:
raise AppError(code="INVALID_GEOJSON", message=f"Invalid feature geometry at index {index}", status_code=400) from exc
if geometry.is_empty:
continue
if not geometry.is_valid:
geometry = make_valid(geometry)
if geometry.is_empty or not geometry.is_valid:
raise AppError(code="INVALID_GEOMETRY", message=f"Invalid feature geometry at index {index}", status_code=400)
properties = feature.get("properties") if isinstance(feature.get("properties"), dict) else {}
source_feature_id = feature.get("id")
if source_feature_id is None:
source_feature_id = properties.get("id") or properties.get("source_feature_id")
row = VectorFeature(
dataset_id=dataset_id,
feature_class=feature_class,
source_feature_id=str(source_feature_id) if source_feature_id is not None else None,
properties_json=properties,
geometry=from_shape(geometry, srid=4326),
)
db.add(row)
persisted.append(row)
if commit:
db.commit()
for row in persisted:
db.refresh(row)
return persisted
@@ -0,0 +1,329 @@
from __future__ import annotations
import json
import uuid
from pathlib import Path
from typing import Any
from geoalchemy2.shape import to_shape
from shapely.geometry import GeometryCollection, MultiPolygon, shape
from shapely.geometry.base import BaseGeometry
from shapely.geometry import mapping
from shapely.ops import unary_union
from shapely.validation import make_valid
from sqlalchemy.orm import Session
from app.core.errors import AppError
from app.models import Area, Dataset
from app.schemas.operations import VectorOperationResult
from app.services.geojson_service import parse_geojson_payload
from app.services.storage_service import StorageService
class VectorOperationsService:
@staticmethod
def _require_vector_dataset(dataset: Dataset) -> None:
if dataset.dataset_type not in {"vector", "geojson"}:
raise AppError(code="INVALID_DATASET_TYPE", message="Dataset is not a vector dataset", status_code=400)
@staticmethod
def _load_dataset_payload(dataset: Dataset) -> tuple[dict[str, Any], list[dict[str, Any]]]:
if not dataset.storage_path:
raise AppError(code="DATASET_FILE_MISSING", message="Stored file missing", status_code=404)
path = Path(dataset.storage_path)
if not path.exists():
raise AppError(code="DATASET_FILE_MISSING", message="Stored file missing", status_code=404)
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except Exception as exc:
raise AppError(code="INVALID_GEOJSON", message="Stored dataset is not valid JSON", status_code=400) from exc
if not isinstance(payload, dict) or payload.get("type") != "FeatureCollection":
raise AppError(code="INVALID_GEOJSON", message="Dataset payload is not a FeatureCollection", status_code=400)
features = payload.get("features")
if not isinstance(features, list):
raise AppError(code="INVALID_GEOJSON", message="Dataset payload is missing features", status_code=400)
return payload, [feature for feature in features if isinstance(feature, dict)]
@staticmethod
def _extract_geometries(features: list[dict[str, Any]]) -> list[tuple[dict[str, Any], BaseGeometry]]:
geometries: list[tuple[dict[str, Any], BaseGeometry]] = []
for feature in features:
if not isinstance(feature, dict):
continue
geometry = feature.get("geometry")
if not geometry:
continue
try:
shapely_geom = shape(geometry)
except Exception as exc:
raise AppError(code="INVALID_GEOMETRY", message="Feature geometry invalid", status_code=400) from exc
if not shapely_geom.is_valid:
shapely_geom = make_valid(shapely_geom)
if not shapely_geom.is_valid:
raise AppError(code="INVALID_GEOMETRY", message="Feature geometry cannot be repaired", status_code=400)
geometries.append((feature, shapely_geom))
if not geometries:
raise AppError(code="EMPTY_VECTOR_DATASET", message="Dataset has no parseable geometries", status_code=422)
return geometries
@staticmethod
def inspect(db: Session, dataset_id: uuid.UUID) -> VectorOperationResult:
dataset = db.get(Dataset, dataset_id)
if not dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Dataset not found", status_code=404)
VectorOperationsService._require_vector_dataset(dataset)
payload, features = VectorOperationsService._load_dataset_payload(dataset)
geometries = VectorOperationsService._extract_geometries(features)
geometry_type_summary: dict[str, int] = {}
for _, geometry in geometries:
geometry_type_summary[geometry.geom_type] = geometry_type_summary.get(geometry.geom_type, 0) + 1
unioned = unary_union([geometry for _, geometry in geometries])
bounds = unioned.bounds
return VectorOperationResult(
source_dataset_id=str(dataset_id),
feature_count=len(geometries),
geometry_type_summary=geometry_type_summary,
bounds_json={"min_x": float(bounds[0]), "min_y": float(bounds[1]), "max_x": float(bounds[2]), "max_y": float(bounds[3])},
crs=payload.get("crs") if isinstance(payload.get("crs"), str) else dataset.crs,
)
@staticmethod
def bbox(db: Session, dataset_id: uuid.UUID) -> dict[str, Any]:
summary = VectorOperationsService.inspect(db, dataset_id)
return {
"dataset_id": str(dataset_id),
"bounds_json": summary.bounds_json,
"feature_count": summary.feature_count,
"crs": summary.crs,
}
@staticmethod
def stats(db: Session, dataset_id: uuid.UUID) -> dict[str, Any]:
summary = VectorOperationsService.inspect(db, dataset_id)
return {
"dataset_id": str(dataset_id),
"feature_count": summary.feature_count,
"geometry_type_summary": summary.geometry_type_summary,
"bounds_json": summary.bounds_json,
"crs": summary.crs,
}
@staticmethod
def clip_by_area(db: Session, dataset_id: uuid.UUID, area_id: uuid.UUID, output_name: str | None) -> uuid.UUID:
source_dataset = db.get(Dataset, dataset_id)
if not source_dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Dataset not found", status_code=404)
VectorOperationsService._require_vector_dataset(source_dataset)
area = db.get(Area, area_id)
if not area:
raise AppError(code="AREA_NOT_FOUND", message="Area not found", status_code=404)
if area.project_id != source_dataset.project_id:
raise AppError(code="INVALID_DATASET_SCOPE", message="Area does not belong to dataset project", status_code=400)
payload, features = VectorOperationsService._load_dataset_payload(source_dataset)
geometries = VectorOperationsService._extract_geometries(features)
area_geom = to_shape(area.geometry)
if area_geom.is_empty:
raise AppError(code="INVALID_AREA_GEOMETRY", message="Area geometry is empty", status_code=400)
if isinstance(area_geom, GeometryCollection):
area_geom = unary_union(area_geom.geoms)
if area_geom.geom_type == "MultiPolygon":
area_geom = MultiPolygon(area_geom.geoms)
if not area_geom.is_valid:
area_geom = make_valid(area_geom)
if not area_geom.is_valid:
raise AppError(code="INVALID_AREA_GEOMETRY", message="Area geometry cannot be repaired", status_code=400)
output_features: list[dict[str, Any]] = []
for feature, source_geom in geometries:
clipped = source_geom.intersection(area_geom)
if clipped.is_empty:
continue
if not clipped.is_valid:
clipped = make_valid(clipped)
if not clipped.is_valid:
raise AppError(code="INVALID_GEOMETRY", message="Clipped geometry became invalid", status_code=400)
output_features.append({
"type": "Feature",
"geometry": mapping(clipped),
"properties": feature.get("properties", {}) or {},
})
if not output_features:
raise AppError(code="VECTOR_OPERATION_EMPTY_RESULT", message="Clip operation produced no output features", status_code=422)
return VectorOperationsService._persist_derived_dataset(
db=db,
source_dataset=source_dataset,
source_id=dataset_id,
operation="clip",
feature_collection={"type": "FeatureCollection", "features": output_features},
output_name=output_name,
default_name="vector_clipped",
)
@staticmethod
def buffer(db: Session, dataset_id: uuid.UUID, distance_m: float, dissolve: bool, output_name: str | None) -> uuid.UUID:
source_dataset = db.get(Dataset, dataset_id)
if not source_dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Dataset not found", status_code=404)
VectorOperationsService._require_vector_dataset(source_dataset)
if distance_m <= 0:
raise AppError(code="INVALID_PARAMETERS", message="distance_m must be greater than 0", status_code=400)
_, features = VectorOperationsService._load_dataset_payload(source_dataset)
geometries = VectorOperationsService._extract_geometries(features)
buffered_features = [(feature, geometry.buffer(distance_m)) for feature, geometry in geometries]
output_features: list[dict[str, Any]] = []
for feature, geometry in buffered_features:
if geometry.is_empty:
continue
if not geometry.is_valid:
geometry = make_valid(geometry)
if not geometry.is_valid:
raise AppError(code="INVALID_GEOMETRY", message="Buffer geometry became invalid", status_code=400)
output_features.append({
"type": "Feature",
"geometry": mapping(geometry),
"properties": feature.get("properties", {}) or {},
})
if dissolve:
dissolved = unary_union([shape(feature["geometry"]) for feature in output_features])
output_features = [{
"type": "Feature",
"geometry": mapping(dissolved),
"properties": {"operation": "vector_buffer", "distance_m": distance_m, "dissolve": True},
}]
if not output_features:
raise AppError(code="VECTOR_OPERATION_EMPTY_RESULT", message="Buffer operation produced no output features", status_code=422)
return VectorOperationsService._persist_derived_dataset(
db=db,
source_dataset=source_dataset,
source_id=dataset_id,
operation="buffer",
feature_collection={"type": "FeatureCollection", "features": output_features},
output_name=output_name,
default_name="vector_buffered",
)
@staticmethod
def intersect(
db: Session,
source_dataset_id: uuid.UUID,
target_dataset_id: uuid.UUID,
output_name: str | None,
) -> uuid.UUID:
if source_dataset_id == target_dataset_id:
raise AppError(code="INVALID_PARAMETERS", message="other_dataset_id must be different from source dataset", status_code=400)
source_dataset = db.get(Dataset, source_dataset_id)
if not source_dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Source dataset not found", status_code=404)
VectorOperationsService._require_vector_dataset(source_dataset)
target_dataset = db.get(Dataset, target_dataset_id)
if not target_dataset:
raise AppError(code="DATASET_NOT_FOUND", message="Target dataset not found", status_code=404)
VectorOperationsService._require_vector_dataset(target_dataset)
if target_dataset.project_id != source_dataset.project_id:
raise AppError(code="INVALID_DATASET_SCOPE", message="Datasets must belong to same project", status_code=400)
source_payload, source_features = VectorOperationsService._load_dataset_payload(source_dataset)
target_payload, _ = VectorOperationsService._load_dataset_payload(target_dataset)
source_geometries = VectorOperationsService._extract_geometries(source_features)
target_geometries = VectorOperationsService._extract_geometries(target_payload.get("features", []))
target_union = unary_union([geometry for _, geometry in target_geometries])
output_features: list[dict[str, Any]] = []
for source_feature, source_geometry in source_geometries:
intersection = source_geometry.intersection(target_union)
if intersection.is_empty:
continue
if not intersection.is_valid:
intersection = make_valid(intersection)
if not intersection.is_valid:
raise AppError(code="INVALID_GEOMETRY", message="Intersection geometry became invalid", status_code=400)
output_features.append({
"type": "Feature",
"geometry": mapping(intersection),
"properties": source_feature.get("properties", {}) or {},
})
if not output_features:
raise AppError(code="VECTOR_OPERATION_EMPTY_RESULT", message="Intersection operation produced no output features", status_code=422)
return VectorOperationsService._persist_derived_dataset(
db=db,
source_dataset=source_dataset,
source_id=source_dataset_id,
operation="intersect",
feature_collection={"type": "FeatureCollection", "features": output_features},
output_name=output_name,
default_name="vector_intersect",
)
@staticmethod
def _persist_derived_dataset(
db: Session,
source_dataset: Dataset,
source_id: uuid.UUID,
operation: str,
feature_collection: dict[str, Any],
output_name: str | None,
default_name: str,
) -> uuid.UUID:
derived_id = uuid.uuid4()
output_name_value = f"{(output_name or default_name)}.geojson"
if not output_name_value.strip():
output_name_value = f"{default_name}.geojson"
stored = json.dumps(feature_collection, ensure_ascii=False, separators=(",", ":")).encode("utf-8")
storage_info = StorageService.persist_dataset_file(
project_id=str(source_dataset.project_id),
dataset_id=str(derived_id),
dataset_type="vector",
original_filename=output_name_value,
content=stored,
content_type="application/geo+json",
)
metadata = parse_geojson_payload(json.dumps(feature_collection, ensure_ascii=False, separators=(",", ":")))
derived_dataset = Dataset(
id=derived_id,
project_id=source_dataset.project_id,
area_id=source_dataset.area_id,
name=output_name_value,
dataset_type="vector",
source=f"operation:{operation}",
storage_path=storage_info["storage_path"],
original_filename=storage_info["original_filename"],
stored_filename=storage_info["stored_filename"],
content_type=storage_info["content_type"],
size_bytes=storage_info["size_bytes"],
checksum_sha256=storage_info["checksum_sha256"],
derived_from_dataset_id=source_id,
crs=metadata.get("crs"),
bounds_json=metadata.get("bounds_json"),
resolution_json=metadata.get("resolution_json"),
bands_json=metadata.get("bands_json"),
metadata_json=metadata,
status="ready",
)
db.add(derived_dataset)
db.commit()
db.refresh(derived_dataset)
return derived_id
+100
View File
@@ -0,0 +1,100 @@
from __future__ import annotations
import importlib.util
from pathlib import Path
from typing import Any
from app.core.config import Settings
from app.core.errors import AppError
class YoloDetectionAdapter:
def __init__(self, settings: Settings) -> None:
self.settings = settings
@staticmethod
def dependencies_available() -> bool:
return importlib.util.find_spec("ultralytics") is not None and importlib.util.find_spec("torch") is not None
def load_model(self, model_path: Path):
if not model_path.exists() or not model_path.is_file():
raise AppError(
code="DETECTION_MODEL_UNAVAILABLE",
message="Configured YOLO model file does not exist",
details={"model_path": str(model_path)},
status_code=503,
)
if not self.dependencies_available():
raise AppError(
code="DETECTION_DEPENDENCY_UNAVAILABLE",
message="YOLO dependencies are not installed. Install backend optional extras with geointel-backend[ai].",
status_code=503,
)
try:
from ultralytics import YOLO
except ImportError as exc:
raise AppError(
code="DETECTION_DEPENDENCY_UNAVAILABLE",
message="YOLO dependencies are not importable. Install backend optional extras with geointel-backend[ai].",
status_code=503,
) from exc
try:
return YOLO(str(model_path))
except Exception as exc:
raise AppError(
code="DETECTION_MODEL_LOAD_FAILED",
message="Configured YOLO model could not be loaded",
details={"model_path": str(model_path)},
status_code=503,
) from exc
def predict_tile(self, model, tile_path: Path, confidence_threshold: float) -> list[dict[str, Any]]:
if not tile_path.exists() or not tile_path.is_file():
raise AppError(
code="DETECTION_TILE_NOT_FOUND",
message="Tile referenced by manifest does not exist",
details={"tile_path": str(tile_path)},
status_code=422,
)
results = model.predict(
source=str(tile_path),
conf=float(confidence_threshold),
imgsz=int(self.settings.yolo_image_size),
device=self.settings.yolo_device,
verbose=False,
)
detections: list[dict[str, Any]] = []
for result in results:
names = getattr(result, "names", {}) or {}
boxes = getattr(result, "boxes", None)
if boxes is None:
continue
xyxy_values = _to_list(getattr(boxes, "xyxy", []))
confidence_values = _to_list(getattr(boxes, "conf", []))
class_values = _to_list(getattr(boxes, "cls", []))
for index, bbox in enumerate(xyxy_values):
class_id = int(class_values[index]) if index < len(class_values) else -1
detections.append(
{
"class_name": str(names.get(class_id, class_id)),
"confidence": float(confidence_values[index]) if index < len(confidence_values) else 0.0,
"bbox": [float(value) for value in bbox],
"properties": {"class_id": class_id},
}
)
return detections
def _to_list(value: Any) -> list[Any]:
if hasattr(value, "detach"):
value = value.detach()
if hasattr(value, "cpu"):
value = value.cpu()
if hasattr(value, "numpy"):
value = value.numpy()
if hasattr(value, "tolist"):
return value.tolist()
return list(value)
@@ -0,0 +1,93 @@
from __future__ import annotations
from pathlib import Path
from typing import Any, Type
from app.core.config import Settings, get_settings
from app.core.errors import AppError
from app.services.detection_service import DetectionService
from app.services.yolo_adapter import YoloDetectionAdapter
class YoloPreflightService:
@staticmethod
def run(
*,
settings: Settings | None = None,
tile_manifest_path: str | None = None,
yolo_adapter_class: Type[YoloDetectionAdapter] = YoloDetectionAdapter,
assume_dependencies: bool = False,
) -> dict[str, Any]:
resolved_settings = settings or get_settings()
result: dict[str, Any] = {
"model_id": resolved_settings.yolo_model_id,
"model_path": resolved_settings.yolo_model_path,
"tile_manifest_path": tile_manifest_path,
"status": "not_configured",
"message": "",
"checks": {
"enabled": resolved_settings.yolo_enabled,
"dependencies_available": None,
"model_path_set": None,
"model_file_exists": None,
"manifest_path_set": None,
"manifest_valid": None,
"tile_paths_exist": None,
"tile_limit_ok": None,
},
"tile_count": 0,
"max_tiles": resolved_settings.yolo_max_tiles,
"will_download_models": False,
"will_run_inference": False,
}
if not resolved_settings.yolo_enabled:
result["message"] = "YOLO is disabled. Set YOLO_ENABLED=true for configured local inference."
return result
dependencies_available = True if assume_dependencies else yolo_adapter_class.dependencies_available()
result["checks"]["dependencies_available"] = dependencies_available
if not dependencies_available:
result["status"] = "dependency_unavailable"
result["message"] = "YOLO dependencies are not installed. Install backend optional extras with geointel-backend[ai]."
return result
result["checks"]["model_path_set"] = bool(resolved_settings.yolo_model_path)
if not resolved_settings.yolo_model_path:
result["message"] = "YOLO_MODEL_PATH is not set. GeoIntel will not download model weights automatically."
return result
model_path = Path(resolved_settings.yolo_model_path).expanduser()
model_exists = model_path.exists() and model_path.is_file()
result["checks"]["model_file_exists"] = model_exists
if not model_exists:
result["message"] = "YOLO_MODEL_PATH does not point to an existing local model file."
return result
result["checks"]["manifest_path_set"] = bool(tile_manifest_path)
if not tile_manifest_path:
result["status"] = "manifest_unavailable"
result["message"] = "Configured YOLO inference requires an existing raster tile manifest path."
return result
try:
manifest = DetectionService._load_tile_manifest(tile_manifest_path, resolved_settings.yolo_max_tiles)
tile_paths = [DetectionService._resolve_tile_path(tile, Path(tile_manifest_path).expanduser()) for tile in manifest["tiles"]]
except AppError as exc:
result["status"] = "manifest_invalid"
result["message"] = exc.message
result["error_code"] = exc.code
result["checks"]["manifest_valid"] = False
if exc.code != "DETECTION_TILE_LIMIT_EXCEEDED":
result["checks"]["tile_limit_ok"] = None
else:
result["checks"]["tile_limit_ok"] = False
return result
result["checks"]["manifest_valid"] = True
result["checks"]["tile_paths_exist"] = all(path.exists() and path.is_file() for path in tile_paths)
result["checks"]["tile_limit_ok"] = len(tile_paths) <= resolved_settings.yolo_max_tiles
result["tile_count"] = len(tile_paths)
result["status"] = "ready"
result["message"] = "Configured YOLO preflight passed. No model was loaded and no inference was run."
return result
View File
View File
+58
View File
@@ -0,0 +1,58 @@
from __future__ import annotations
from typing import Any
from pyproj import Transformer
from shapely.geometry import GeometryCollection, MultiPolygon, Polygon, box, shape
from shapely.ops import transform
from shapely.validation import make_valid
def normalize_to_multipolygon(raw_geometry: dict[str, Any]) -> MultiPolygon:
geom = shape(raw_geometry)
if geom.is_empty:
raise ValueError("Geometry is empty")
if not geom.is_valid:
geom = make_valid(geom)
if not geom.is_valid:
raise ValueError("Geometry is invalid and could not be repaired")
if geom.geom_type == "Polygon":
return MultiPolygon([geom])
if geom.geom_type == "MultiPolygon":
return MultiPolygon(geom.geoms)
if isinstance(geom, GeometryCollection):
polygons = [g for g in geom.geoms if isinstance(g, Polygon)]
multipolygons = [g for g in geom.geoms if g.geom_type == "MultiPolygon"]
if not polygons and not multipolygons:
raise ValueError("Only polygon geometries are supported for AOI")
normalized = []
normalized.extend(polygons)
for mp in multipolygons:
normalized.extend(mp.geoms)
return MultiPolygon(normalized)
raise ValueError("Only Polygon or MultiPolygon geometries are accepted")
def area_bounds_multipolygon(geom: MultiPolygon):
return {
"min_x": float(geom.bounds[0]),
"min_y": float(geom.bounds[1]),
"max_x": float(geom.bounds[2]),
"max_y": float(geom.bounds[3]),
}
def area_m2(geom: MultiPolygon) -> float:
projected = transform(
Transformer.from_crs("EPSG:4326", "EPSG:31370", always_xy=True).transform,
geom,
)
return float(projected.area)
def geometry_bbox_polygon(geom: MultiPolygon):
return box(*geom.bounds)
+5
View File
@@ -0,0 +1,5 @@
from typing import Any
def envelope(payload: Any) -> dict[str, Any]:
return {"data": payload}
View File
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env sh
set -eu
echo "Waiting for database connection..."
python - <<'PY'
import time
from sqlalchemy import create_engine, text
from app.core.config import get_settings
settings = get_settings()
last_error = None
for attempt in range(1, 31):
try:
engine = create_engine(settings.database_url, pool_pre_ping=True, future=True)
with engine.connect() as connection:
connection.execute(text("SELECT 1"))
print(f"Database connection ready after attempt {attempt}.")
break
except Exception as exc:
last_error = exc
print(f"Database not ready yet ({attempt}/30): {exc}")
time.sleep(2)
else:
raise SystemExit(f"Database did not become ready: {last_error}")
PY
python -m alembic upgrade head
exec uvicorn app.main:app --host 0.0.0.0 --port 8000

Some files were not shown because too many files have changed in this diff Show More