feat: add governed nationwide AOI orchestration and CUDA enforcement
This commit is contained in:
@@ -92,7 +92,7 @@ Ultralytics/PyTorch compatibility, but it still does not run tile prediction and
|
||||
does not download weights. It cannot be combined with `--assume-dependencies`
|
||||
because that would turn the smoke into a false positive.
|
||||
|
||||
Docker and Unraid runtime support remains opt-in. Set `GEOINTEL_INSTALL_AI=true`
|
||||
Docker AI dependencies remain opt-in. Set `GEOINTEL_INSTALL_AI=true`
|
||||
at build time to install the backend `.[gis,ai]` extra into the container. Leave
|
||||
it unset or `false` for the default GIS-only image. Runtime model files should be
|
||||
mounted into the container, for example `/app/models/local-model.pt`, and enabled
|
||||
@@ -109,6 +109,11 @@ Environment variables:
|
||||
- `YOLO_MODEL_DISPLAY_NAME`
|
||||
- `YOLO_MODEL_VERSION`
|
||||
- `YOLO_DEVICE`
|
||||
- `YOLO_REQUIRE_CUDA` (set to `true` on the production server; inference then
|
||||
fails closed when CUDA is unavailable or `YOLO_DEVICE` selects CPU)
|
||||
- `YOLO_MODEL_CLASSES` (the active promoted detector is `building` only)
|
||||
- `YOLO_ENFORCE_VALIDATION_SCOPE` and `YOLO_VALIDATED_AREA_NAMES` (production
|
||||
rejects inference when the raster is not bound to a persisted validated Area)
|
||||
- `YOLO_IMAGE_SIZE`
|
||||
- `YOLO_MAX_TILES`
|
||||
- `YOLO_MAX_DETECTIONS`
|
||||
|
||||
+85
-4
@@ -275,6 +275,74 @@ Idempotently creates or returns a project Area from the exact persisted NGI
|
||||
municipality geometry. The resulting Area can be used by all existing bounded
|
||||
selection, acquisition, analysis and export contracts.
|
||||
|
||||
## Resumable AOI operations
|
||||
|
||||
### POST `/api/v1/projects/{project_id}/aoi-operations`
|
||||
|
||||
Creates one persisted parent operation and deterministic bounded partitions.
|
||||
Exactly one of `area_id` or an EPSG:4326 `bbox` is required. Partitioning is
|
||||
calculated in EPSG:31370 and clipped to the exact AOI; source side limits remain
|
||||
server concerns. Optional `coverage_zone` clips the immutable AOI snapshot to
|
||||
the persisted legal/regional scope before planning. The production worker
|
||||
automatically claims queued children; clients poll the parent instead of
|
||||
driving provider requests.
|
||||
|
||||
```json
|
||||
{
|
||||
"area_id": "optional-uuid",
|
||||
"operation_type": "acquire",
|
||||
"provider_key": "grb",
|
||||
"product_key": "buildings",
|
||||
"max_partition_side_m": null,
|
||||
"max_attempts": 3,
|
||||
"parameters_json": {"force_refresh": false}
|
||||
}
|
||||
```
|
||||
|
||||
When `max_partition_side_m` is omitted, the backend derives the limit from the
|
||||
governed provider registry. An explicit value can only make partitions smaller,
|
||||
never relax the provider budget. The response reports parent status, progress from `0.0` to `1.0`, per-status
|
||||
partition counts and child partition evidence. Stable partition keys make
|
||||
planning and completion idempotent.
|
||||
|
||||
### GET `/api/v1/projects/{project_id}/aoi-operations/{operation_id}`
|
||||
|
||||
Returns the persisted operation, aggregate progress and every child state.
|
||||
|
||||
### GET `/api/v1/projects/{project_id}/aoi-operations`
|
||||
|
||||
Lists the most recent parent operations for the project with their derived
|
||||
progress, status counts and child evidence.
|
||||
|
||||
### POST `/api/v1/projects/{project_id}/aoi-operations/{operation_id}/execute-next`
|
||||
|
||||
Claims and executes one queued partition through the registered governed
|
||||
provider. Registered executors cover `grb`, `orthophoto`, `dhmv`,
|
||||
`spw_terrain`, `official_vector`, `flood_hazard`, `thematic_raster`, `walous`,
|
||||
`bathymetry_profiles` and `mdk_bathymetry`; their existing configuration and
|
||||
zone contracts remain authoritative. Every
|
||||
execution creates a linked child Job. Transient provider failures are retried
|
||||
within the stored attempt budget; validation failures fail immediately.
|
||||
|
||||
### POST `/api/v1/projects/{project_id}/aoi-operations/{operation_id}/partitions/claim`
|
||||
|
||||
Atomically claims the next queued child using a locked, skip-locked database
|
||||
selection. Returns `data: null` when no queued child remains.
|
||||
|
||||
### PUT `/api/v1/projects/{project_id}/aoi-operations/{operation_id}/partitions/{partition_id}/checkpoint`
|
||||
|
||||
Persists provider-specific restart evidence for a running child.
|
||||
|
||||
### POST `/api/v1/projects/{project_id}/aoi-operations/{operation_id}/partitions/{partition_id}/complete`
|
||||
|
||||
Marks a running child successful or explicitly skipped. Repeating completion
|
||||
for a terminal successful/skipped child is idempotent.
|
||||
|
||||
### POST `/api/v1/projects/{project_id}/aoi-operations/{operation_id}/partitions/{partition_id}/fail`
|
||||
|
||||
Records an error and either returns the child to `queued` within its attempt
|
||||
budget or terminalizes it as `failed`.
|
||||
|
||||
## Datasets
|
||||
|
||||
### POST `/api/v1/projects/{project_id}/datasets/upload`
|
||||
@@ -1098,7 +1166,8 @@ mode, source URL, attribution, licence and limitation text.
|
||||
|
||||
Allowed result states are exactly:
|
||||
|
||||
- `operational`: a matching `ready` Dataset is materialized in the project;
|
||||
- `operational`: one matching `ready` Dataset or the spatial union of governed
|
||||
partition Datasets covers the complete selection;
|
||||
- `partial`: the governed integration exists but matching project data is
|
||||
absent or incomplete;
|
||||
- `not_configured`: an audited source has no operational adapter;
|
||||
@@ -1124,7 +1193,9 @@ Areas. Cross-region and land/sea selections remain split by zone.
|
||||
|
||||
The response returns `intersected_zones`, `outside_supported_scope`, one item
|
||||
per zone/theme combination, matching source names and IDs of actually
|
||||
materialized Datasets. An empty `themes` list requests the complete normalized
|
||||
materialized Datasets. Every materialized item also carries evidence for
|
||||
authority, source version, observation/publication time, CRS, resolution,
|
||||
coverage bbox, attribution, licence and checksum where persisted. An empty `themes` list requests the complete normalized
|
||||
vocabulary. Unknown themes fail with `COVERAGE_THEME_UNSUPPORTED`.
|
||||
|
||||
The Map workbench uses this response before a rectangle analysis. It queries
|
||||
@@ -1396,6 +1467,7 @@ Response data:
|
||||
"checks": {
|
||||
"enabled": true,
|
||||
"dependencies_available": true,
|
||||
"accelerator_ready": true,
|
||||
"model_path_set": false,
|
||||
"model_file_exists": null,
|
||||
"model_load_requested": false,
|
||||
@@ -1411,7 +1483,9 @@ Response data:
|
||||
"yolo_config_dir": "/app/storage/ultralytics",
|
||||
"torch_version": "2.12.1",
|
||||
"ultralytics_version": "8.4.88",
|
||||
"cuda_available": false
|
||||
"cuda_available": true,
|
||||
"configured_device": "cuda:0",
|
||||
"cuda_required": true
|
||||
},
|
||||
"tile_count": 0,
|
||||
"max_tiles": 100,
|
||||
@@ -1422,7 +1496,14 @@ Response data:
|
||||
|
||||
### POST `/api/v1/detection/run`
|
||||
|
||||
Creates a detection job and detection analysis run. If the requested model is unavailable, the job and analysis run are marked `failed` with `DETECTION_MODEL_UNAVAILABLE` or `DETECTION_DEPENDENCY_UNAVAILABLE`.
|
||||
Creates a detection job and detection analysis run. If the requested model is unavailable, the job and analysis run are marked `failed` with `DETECTION_MODEL_UNAVAILABLE` or `DETECTION_DEPENDENCY_UNAVAILABLE`. When `YOLO_REQUIRE_CUDA=true`, missing CUDA or a CPU device selection fails closed with `DETECTION_ACCELERATOR_UNAVAILABLE` or `DETECTION_ACCELERATOR_MISCONFIGURED`; production inference never silently falls back to CPU.
|
||||
|
||||
The configured model exposes only classes declared by `YOLO_MODEL_CLASSES`;
|
||||
the current promoted server detector is building-only. Unsupported class
|
||||
filters fail with `DETECTION_CLASS_NOT_VALIDATED`. With production scope
|
||||
enforcement enabled, the raster must belong to a persisted Area matching the
|
||||
configured validated Mol/Kempen evidence or inference fails with
|
||||
`DETECTION_VALIDATION_SCOPE_UNAVAILABLE`.
|
||||
|
||||
Request:
|
||||
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
# Audit remediation roadmap
|
||||
|
||||
Status: active, 2026-07-26
|
||||
|
||||
## Release outcome
|
||||
|
||||
GeoIntel may accept every valid AOI inside the governed Belgium and Belgian
|
||||
North Sea scope. It may only call a theme operational where bounded processing,
|
||||
source coverage, provenance, resolution and validation evidence support that
|
||||
claim. Production AI inference on the server uses its NVIDIA GPU and fails
|
||||
closed when CUDA is unavailable; CPU fallback is not an accepted production
|
||||
state.
|
||||
|
||||
## Wave 0 - Runtime truth and NVIDIA GPU (in progress)
|
||||
|
||||
- expose the NVIDIA device to the Unraid container;
|
||||
- set `YOLO_DEVICE=cuda:0` and `YOLO_REQUIRE_CUDA=true` in the server runtime;
|
||||
- make preflight and model loading reject missing CUDA instead of using CPU;
|
||||
- report configured device, CUDA requirement and accelerator readiness;
|
||||
- rebuild on Tower and capture `nvidia-smi`, CUDA-enabled PyTorch, preflight and
|
||||
one bounded inference smoke as release evidence.
|
||||
|
||||
Exit gate: the live container sees the NVIDIA GPU, `torch.cuda.is_available()`
|
||||
is true, preflight is ready, and a persisted smoke run records the configured
|
||||
CUDA device. A CPU-only image or unavailable device remains `not_configured` /
|
||||
unavailable and cannot run production inference.
|
||||
|
||||
## Wave 1 - General AOI orchestration (in progress)
|
||||
|
||||
- [x] introduce one persisted parent operation with source-specific partitions;
|
||||
- [x] derive partitions from provider side budgets supplied by the governed plan;
|
||||
- [x] support queued execution, bounded retries, checkpoints and restart recovery;
|
||||
- make partition application idempotent and retain exact request/checksum
|
||||
provenance;
|
||||
- [x] reuse source-aware vector deduplication and raster mosaic contracts and
|
||||
aggregate their Dataset identities into one parent result;
|
||||
- [x] expose one progress/result contract to the frontend system workspace.
|
||||
|
||||
Start with existing orthophoto, GRB and raster partition services; do not create
|
||||
a second provider or persistence path. Keep per-source limits internal. A source
|
||||
that cannot cover a partition returns explicit partial/not_configured evidence.
|
||||
|
||||
Exit gate: interrupted cross-region and coastal golden AOIs resume without
|
||||
duplicate rows and finish as one inspectable result.
|
||||
|
||||
## Wave 2 - Zone x theme x source coverage truth (implemented; live gate pending)
|
||||
|
||||
- [x] materialize the resolver contract for land, regions and legal maritime zones;
|
||||
- [x] evaluate partition-union spatial coverage and expose source edition,
|
||||
resolution, time, CRS, attribution, licence and checksum evidence;
|
||||
- [x] derive only `operational`, `partial`, `not_configured` or `unsupported`;
|
||||
- [x] show missing partitions and limitations in API and map states;
|
||||
- [x] prohibit UI wording that implies complete national analysis from selection
|
||||
acceptance alone.
|
||||
|
||||
Exit gate: all frozen golden areas have checksum-bound coverage evidence and no
|
||||
theme is promoted from file presence or a Mol-only success.
|
||||
|
||||
## Wave 3 - Source completion
|
||||
|
||||
- resolve North Sea bathymetry through a TLS-valid, authority-approved endpoint
|
||||
or reviewed bounded operator acquisition; never bypass TLS;
|
||||
- close configured orthophoto/nature/soil gaps for Wallonia where authoritative
|
||||
machine access permits;
|
||||
- close Brussels orthophoto gaps and retain unsupported statuses where no
|
||||
source-appropriate analytical contract exists;
|
||||
- add explicit external-catalog review evidence for the six source families
|
||||
currently requiring review.
|
||||
|
||||
Exit gate: every required theme/zone cell has current evidence and honest status;
|
||||
vertical datums remain separate and water volume remains unavailable without a
|
||||
governed compatible model.
|
||||
|
||||
## Wave 4 - Model portfolio and validation (current active model gated)
|
||||
|
||||
- [x] inventory model assets without treating presence as configuration;
|
||||
- [x] bind the current active model to persisted Mol/Kempen Area evidence;
|
||||
- [x] retain building-only semantics for the current active detector;
|
||||
- add segmentation only with a configured model, georeferencing tests and
|
||||
persisted polygon/mask evidence;
|
||||
- [x] require a model/region pair to pass reproducible holdout, hard-negative and
|
||||
QA gates pass.
|
||||
|
||||
Exit gate: UI and API operational labels are derived from validation evidence,
|
||||
not run counts; no claim that PyTorch itself is trained on themes.
|
||||
|
||||
## Wave 5 - Release proof
|
||||
|
||||
- run Mol/Kempen plus Walloon, Brussels, language-boundary, coastal and maritime
|
||||
golden workflows;
|
||||
- test fresh install, upgrade, rollback, restart/resume and sustained runtime;
|
||||
- audit OpenAPI envelopes, CRS/units, provenance, exports and frontend
|
||||
loading/empty/error states;
|
||||
- publish exact checksums, source editions, model evidence and known limitations.
|
||||
|
||||
Exit gate: `docs/DEFINITION_OF_DONE.md` and the RC freeze are satisfied. The
|
||||
product claim remains location-complete at source-native resolution, never
|
||||
literal centimetre-resolution.
|
||||
|
||||
## Execution order
|
||||
|
||||
1. Finish Wave 0 and verify it live on Tower.
|
||||
2. Implement the parent/partition state machine and one GRB/orthophoto vertical
|
||||
slice from Wave 1.
|
||||
3. Generalize that slice across compatible raster/vector providers.
|
||||
4. Deliver Wave 2 before promoting additional source cells.
|
||||
5. Run Waves 3 and 4 in parallel only where their evidence is independent.
|
||||
6. Close with Wave 5; do not advertise the audited guarantee before its gate.
|
||||
@@ -11346,3 +11346,85 @@ Validation:
|
||||
- added backend API/service tests for multilingual/NIS search and persisted-area activation;
|
||||
- added frontend tests for live search, activation and explicit optional/free-selection semantics;
|
||||
- visually inspected the 512 px application mark and 32 px favicon derivative.
|
||||
## 2026-07-26 - Audit remediation roadmap and NVIDIA GPU contract
|
||||
|
||||
Implemented:
|
||||
|
||||
- translated the platform audit into `docs/AUDIT_REMEDIATION_ROADMAP.md`, with
|
||||
gated waves for NVIDIA runtime truth, general AOI orchestration, coverage
|
||||
evidence, missing sources, model validation and release proof;
|
||||
- made the Unraid production contract explicitly NVIDIA/CUDA-based through a
|
||||
CUDA PyTorch wheel index, `gpus: all`, `YOLO_DEVICE=cuda:0` and
|
||||
`YOLO_REQUIRE_CUDA=true`;
|
||||
- added fail-closed accelerator validation to YOLO model loading and preflight,
|
||||
including explicit unavailable/misconfigured errors instead of CPU fallback;
|
||||
- extended the preflight response with accelerator readiness, configured device
|
||||
and CUDA-required state, and updated the API, AI and dependency documentation;
|
||||
- retained CPU as an allowed local-development default only when CUDA is not
|
||||
explicitly required.
|
||||
|
||||
Validation:
|
||||
|
||||
- 57 focused backend tests passed for YOLO inference, preflight and Docker
|
||||
runtime configuration;
|
||||
- backend application byte-compilation passed;
|
||||
- local Compose rendering could not run because this Windows workstation has no
|
||||
`docker` CLI. Tower rebuild, `nvidia-smi`, CUDA-enabled PyTorch preflight and
|
||||
one bounded persisted GPU inference smoke remain the live Wave 0 exit gate.
|
||||
|
||||
Known limitations and next pass:
|
||||
|
||||
- these repository changes do not prove that the deployed Tower container can
|
||||
see the physical GPU; do not claim GPU readiness until the live gate passes;
|
||||
- next implement the persisted parent/partition/checkpoint operation as the
|
||||
first Wave 1 vertical slice, reusing existing job and provider services.
|
||||
## 2026-07-26 - Audit remediation: resumable AOI federation and truthful AI runtime
|
||||
|
||||
Implemented:
|
||||
|
||||
- added migration `202607260001` with persisted AOI parent operations and
|
||||
deterministic child partitions, exact EPSG:31370 planning, zone clipping,
|
||||
checkpoints, attempt budgets, child Jobs and restart reconciliation;
|
||||
- added a production background worker that automatically claims queued work
|
||||
and dispatches ten existing governed providers without introducing a second
|
||||
Dataset persistence path;
|
||||
- aggregated child Dataset identities, completeness and source-aware vector
|
||||
deduplication/raster mosaic contracts into one parent result and exposed live
|
||||
progress plus failures in the System workspace;
|
||||
- removed the frontend overview/detail refusal for on-demand themes and routed
|
||||
regional/overview acquisition through the resumable server operation;
|
||||
- made coverage resolution accept a spatial union of bounded partitions only
|
||||
when it covers the selection, and exposed authority, edition, time, CRS,
|
||||
resolution, bbox, attribution, licence and checksum evidence per Dataset;
|
||||
- corrected the active YOLO contract to building-only, fail-closed CUDA and
|
||||
persisted Mol/Kempen Area scope. Production cannot advertise configured AI
|
||||
from a CPU runtime or execute an unvalidated class/area.
|
||||
|
||||
Validation:
|
||||
|
||||
- one Alembic head (`202607260001`) and a complete 40,888-byte offline SQL
|
||||
migration chain were generated;
|
||||
- 71 focused backend tests, 40 frontend tests, frontend typecheck and production
|
||||
build passed;
|
||||
- the broader Windows-compatible backend gate passed 1,115 tests with five
|
||||
WSL-dependent shell tests deselected because the workstation WSL VHD is
|
||||
missing; those Linux shell gates remain mandatory on Tower;
|
||||
- API contract audit passed with all 146 routes documented.
|
||||
|
||||
Live pre-deploy evidence:
|
||||
|
||||
- Tower is healthy on PostGIS 3.6 at migration `202607160001`;
|
||||
- the host has an NVIDIA GeForce RTX 4080 SUPER with 16,376 MiB and driver
|
||||
595.71.05;
|
||||
- the old container confirms the audited failure state: no Docker device
|
||||
request, `torch 2.13.0+cpu`, CUDA false and zero visible GPUs;
|
||||
- the official Vlaanderen catalog still identifies MDK Version 8, 20 m, LAT as
|
||||
live production data, but both catalogued HTTPS host variants fail strict TLS
|
||||
and HTTP does not expose the WCS path. MDK remains `not_configured`; TLS is not
|
||||
bypassed.
|
||||
|
||||
Next gate:
|
||||
|
||||
- commit/push the immutable source, deploy on Tower, run the Linux readiness and
|
||||
migration gates, prove CUDA PyTorch plus a bounded persisted inference, and
|
||||
capture golden AOI/coverage evidence.
|
||||
|
||||
@@ -318,6 +318,22 @@ Official editions take precedence over legacy rolling markers in catalog
|
||||
comparison, but direct metadata backfill, update or deletion of those legacy
|
||||
rows remains prohibited.
|
||||
|
||||
## Resumable AOI operations
|
||||
|
||||
Migration `202607260001` adds `aoi_operations` and
|
||||
`aoi_operation_partitions`. The parent stores the immutable EPSG:4326 AOI
|
||||
snapshot, original request and source-specific plan. Child rows store exact
|
||||
partition geometry, stable `(operation_id, partition_key)` identity, provider
|
||||
and product, attempt budget, checkpoint, result and child Job linkage.
|
||||
|
||||
Workers atomically claim queued child rows. Interrupted running rows return to
|
||||
queued state while attempts remain and become failed when exhausted. Provider
|
||||
output still flows through the existing DatasetService/provider contracts;
|
||||
these tables orchestrate work and never become a parallel dataset store.
|
||||
The all-in-one runtime enables one database-polling worker. Multiple future
|
||||
workers remain safe because partition claim uses `FOR UPDATE SKIP LOCKED` and
|
||||
the partition key is unique inside its parent operation.
|
||||
|
||||
## Geometry normalization
|
||||
|
||||
- User-drawn polygons arrive as EPSG:4326.
|
||||
|
||||
@@ -45,12 +45,13 @@ AI dependencies remain separate in the `ai` optional dependency group and must
|
||||
not be installed by the default Docker backend image unless an explicit AI image
|
||||
or profile is introduced later.
|
||||
|
||||
The opt-in Unraid all-in-one AI build is CPU-oriented because its documented
|
||||
runtime sets `YOLO_DEVICE=cpu`. It installs the pinned PyTorch/torchvision pair
|
||||
from PyTorch's CPU wheel index before installing the `ai` extra. This avoids
|
||||
shipping unused CUDA runtime libraries. The index and versions remain explicit
|
||||
Docker build arguments so a future, separately validated GPU image can override
|
||||
them without changing the base dependency group.
|
||||
The opt-in Unraid all-in-one AI build is NVIDIA-GPU-oriented. It installs the
|
||||
pinned PyTorch/torchvision pair from the CUDA 13.0 wheel index before installing
|
||||
the `ai` extra. The production runtime exposes the NVIDIA device, selects
|
||||
`YOLO_DEVICE=cuda:0` and sets `YOLO_REQUIRE_CUDA=true`, so missing CUDA fails
|
||||
closed instead of silently falling back to CPU. The index and versions remain
|
||||
explicit Docker build arguments and require live driver/runtime validation on
|
||||
Tower before release promotion.
|
||||
|
||||
Docker dependency metadata is copied before application source. Backend source
|
||||
changes therefore reuse the dependency layer while changes to `pyproject.toml`
|
||||
|
||||
@@ -918,3 +918,14 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Explain theme choice as map and primary-metric focus rather than a mandatory workflow gate.
|
||||
- [x] Keep narrow theme-card copy and status labels inside their cards.
|
||||
- [x] Replace the former compass icon with a scale-safe GeoIntel geo-lens mark.
|
||||
# Audit remediation (2026-07-26)
|
||||
|
||||
- [x] Freeze an audit remediation roadmap with NVIDIA GPU as a server requirement.
|
||||
- [x] Add fail-closed CUDA configuration and preflight/runtime validation.
|
||||
- [ ] Rebuild Tower with NVIDIA device exposure and capture a CUDA inference smoke.
|
||||
- [x] Implement the persisted parent/partition/checkpoint AOI operation.
|
||||
- [x] Add source-aware seam deduplication contracts and one-result Dataset aggregation across child partitions.
|
||||
- [x] Extend the zone x theme x source resolver with partition-union coverage and per-Dataset evidence.
|
||||
- [ ] Resolve governed North Sea bathymetry access without bypassing TLS.
|
||||
- [x] Enforce the active building-only model classes, CUDA readiness and persisted Mol/Kempen Area scope.
|
||||
- [ ] Add national/regional holdout evidence before expanding model classes or validated Areas.
|
||||
|
||||
Reference in New Issue
Block a user