Initial public release

2026-08-31 21:56:53 +02:00
commit faeb58ef6d
1386 changed files with 263203 additions and 0 deletions
@@ -0,0 +1,41 @@
+# Autonomous Build Charter
+
+## Mission
+Codex must build GeoIntel Kempen as a production-shaped GeoAI Workbench, not as a demo-only UI. Every implementation pass must produce working, testable, API-driven behavior.
+
+## Frozen Product Identity
+GeoIntel is a GeoAI Workbench for the Kempen focused on raster/vector processing, AI detection, segmentation, QA/QC against reference data, and geospatial exports.
+
+## Allowed Autonomy
+Codex may improve:
+- internal code organization when it keeps documented contracts intact;
+- UI microcopy and layout when it improves clarity;
+- validation and error messages;
+- tests, fixtures, logging, and developer tooling;
+- performance optimizations that do not alter outputs;
+- accessibility and keyboard navigation;
+- extra helper utilities that support documented workflows.
+
+## Not Allowed Without Explicit User Approval
+Codex must not:
+- replace FastAPI, React, TypeScript, PostgreSQL/PostGIS, or Python GIS stack;
+- introduce a different product direction such as a generic chatbot or CRUD dashboard;
+- remove GRB/QA/QC as a first-class concept;
+- make LiDAR, training studio, or MLOps part of V1 core;
+- hardcode fake results while presenting them as real processing;
+- break existing documented API contracts;
+- silently change geometry formats or CRS assumptions;
+- ignore tests because a dependency is missing.
+
+## Output Expectations Per Pass
+Every Codex pass must end with:
+1. changed files list;
+2. completed tasks;
+3. skipped tasks with reason;
+4. commands run;
+5. test results;
+6. known limitations;
+7. next recommended pass.
+
+## Quality Principle
+If a feature cannot be fully implemented in the pass, Codex must implement the durable skeleton plus honest status handling, not a fake success path.
@@ -0,0 +1,66 @@
+# Build Pass Template
+
+Use this exact structure for every Codex build pass.
+
+## Pass Name
+Example: `Pass 03 — Dataset Manager Backend`
+
+## Goal
+One sentence describing the feature outcome.
+
+## Inputs
+- Required docs:
+- Required contracts:
+- Required fixtures:
+- Previous pass dependencies:
+
+## Scope
+### Must implement
+- [ ] Item
+
+### May improve
+- [ ] Item
+
+### Must not touch
+- [ ] Item
+
+## Implementation Steps
+1. Inspect existing repo state.
+2. Confirm relevant contracts.
+3. Implement backend/domain changes.
+4. Implement frontend/API client changes if applicable.
+5. Add or update tests.
+6. Add fixtures if needed.
+7. Update docs and TODO status.
+8. Run validation commands.
+9. Produce pass summary.
+
+## Validation Commands
+```bash
+# backend
+pytest
+ruff check backend || true
+
+# frontend
+npm run build
+npm run typecheck || true
+npm run lint || true
+```
+
+## Acceptance Criteria
+- [ ] Feature works through API or UI.
+- [ ] Errors are explicit and typed.
+- [ ] No mock success path is presented as real.
+- [ ] Tests or fixtures cover the happy path and at least one failure path.
+- [ ] Documentation is updated.
+
+## Handoff Format
+```md
+# Pass Summary
+## Completed
+## Changed Files
+## Commands Run
+## Test Results
+## Known Issues
+## Next Step
+```
@@ -0,0 +1,24 @@
+# Codex Start Here
+
+Read these files first, in this exact order:
+
+1. `README.md`
+2. `AGENTS.md`
+3. `docs/18-ultra-prep/AUTONOMOUS_BUILD_CHARTER.md`
+4. `docs/V1_SCOPE_FREEZE.md`
+5. `docs/SERVICE_ARCHITECTURE.md`
+6. `docs/REPOSITORY_CONVENTIONS.md`
+7. `docs/API_CONTRACTS.md`
+8. `docs/DATABASE_IMPLEMENTATION_PLAN.md`
+9. `docs/18-ultra-prep/CRITICAL_PATH_TO_V1.md`
+10. `docs/18-ultra-prep/BUILD_PASS_TEMPLATE.md`
+
+Then execute the first incomplete pass from:
+
+- `prompts/codex/M10_PASS_SEQUENCE.md`
+
+Rules:
+- Do not skip ahead.
+- Do not start UI polish before backend contracts exist.
+- Do not fake geospatial processing; return honest pending/unavailable states.
+- Update TODO/checklists after every pass.
@@ -0,0 +1,27 @@
+# Connector Implementation Guide
+
+## Connector Interface
+Every external data connector should expose:
+- `name`
+- `capabilities`
+- `is_configured()`
+- `health_check()`
+- `fetch_by_area(area_geometry, parameters)`
+- `normalize(raw_result)`
+- `cache_key(area, parameters)`
+
+## Required Connectors
+### GRB Connector
+V1 target: fetch or register reference building polygons. If live WFS cannot be implemented immediately, build the interface and support fixture/local import honestly.
+
+### OSM Connector
+V1 target: fetch buildings/roads/water/green where possible using Overpass or local fixture fallback.
+
+### Sentinel Connector
+V2 target. In V1 it should remain disabled with clear roadmap state.
+
+## Connector Failure Behavior
+- timeout -> `CONNECTOR_TIMEOUT`
+- missing config -> `CONNECTOR_UNCONFIGURED`
+- invalid response -> `CONNECTOR_INVALID_RESPONSE`
+- no data -> valid empty result, not error
@@ -0,0 +1,35 @@
+# Critical Path to V1
+
+This is the shortest reliable path to a portfolio-ready V1.
+
+## V1 Critical Path
+1. Repo foundation and dev environment.
+2. Database and PostGIS schema.
+3. Project and area CRUD.
+4. Dataset upload and metadata extraction.
+5. Vector import and display.
+6. Raster import and metadata display.
+7. Map workbench with layer management.
+8. OSM/GRB reference ingest interface or stubbed real-data connector with honest unavailable state.
+9. Detection pipeline architecture with YOLO adapter.
+10. Detection result storage as geospatial features.
+11. QA/QC comparison engine against reference polygons.
+12. GeoJSON export.
+13. Demo scenario page.
+14. Smoke tests and documentation.
+
+## V1 Can Be Portfolio-Ready Without
+- full Sentinel automation;
+- actual production GRB WFS credentials or final endpoint if unavailable;
+- LiDAR processing;
+- model training;
+- multi-user auth;
+- perfect visual polish.
+
+## V1 Cannot Be Portfolio-Ready Without
+- a working map-driven workflow;
+- actual geospatial data models;
+- real CRS/geometry handling;
+- honest dataset status handling;
+- at least one end-to-end detection/QA workflow, even if it uses a small local demo fixture/model adapter first;
+- clear export output.
@@ -0,0 +1,24 @@
+# CRS Policy
+
+## API Boundary
+GeoJSON in API responses should be EPSG:4326.
+
+## Analytical Operations
+For Belgium/Kempen projected calculations, use EPSG:31370 where possible.
+
+## Raster Operations
+Raster operations must preserve source CRS unless a reproject operation is explicitly requested.
+
+## Required Metadata
+Every dataset must store:
+- source CRS;
+- normalized CRS string;
+- bounds in source CRS;
+- bounds in EPSG:4326;
+- transform/affine for rasters where available.
+
+## Unsupported Cases
+If CRS cannot be determined:
+- dataset can be stored as `metadata_failed` or `requires_crs`;
+- processing requiring geospatial alignment must be blocked;
+- UI must ask for CRS or show limitation.
@@ -0,0 +1,36 @@
+# Error Taxonomy
+
+All API errors must use a consistent shape.
+
+```json
+{
+  "error": {
+    "code": "DATASET_UNSUPPORTED_FORMAT",
+    "message": "The uploaded file format is not supported for this operation.",
+    "details": {},
+    "trace_id": "optional"
+  }
+}
+```
+
+## Core Error Codes
+- `VALIDATION_ERROR`
+- `NOT_FOUND`
+- `CONFLICT`
+- `FEATURE_DISABLED`
+- `DATASET_UPLOAD_FAILED`
+- `DATASET_UNSUPPORTED_FORMAT`
+- `CRS_MISSING`
+- `CRS_UNSUPPORTED`
+- `GEOMETRY_INVALID`
+- `RASTER_METADATA_FAILED`
+- `VECTOR_METADATA_FAILED`
+- `PROCESSING_JOB_FAILED`
+- `MODEL_UNAVAILABLE`
+- `INFERENCE_FAILED`
+- `REFERENCE_LAYER_MISSING`
+- `QA_MATCHING_FAILED`
+- `EXPORT_FAILED`
+
+## Frontend Requirements
+Every error code should render a useful message and suggested next action.
@@ -0,0 +1,34 @@
+# Feature Flag Strategy
+
+Feature flags prevent future modules from appearing as broken V1 features.
+
+## Required Flags
+- `ENABLE_GRB_CONNECTOR`
+- `ENABLE_OSM_CONNECTOR`
+- `ENABLE_SENTINEL_LAB`
+- `ENABLE_SAM_SEGMENTATION`
+- `ENABLE_YOLO_DETECTION`
+- `ENABLE_LIDAR_LAB`
+- `ENABLE_TRAINING_STUDIO`
+- `ENABLE_QGIS_EXPORT`
+
+## V1 Defaults
+```env
+ENABLE_GRB_CONNECTOR=true
+ENABLE_OSM_CONNECTOR=true
+ENABLE_SENTINEL_LAB=false
+ENABLE_SAM_SEGMENTATION=false
+ENABLE_YOLO_DETECTION=true
+ENABLE_LIDAR_LAB=false
+ENABLE_TRAINING_STUDIO=false
+ENABLE_QGIS_EXPORT=false
+```
+
+## UI Behavior
+Disabled features may be visible as roadmap cards, but must not look like broken tools. They should show:
+- why disabled;
+- what dependency is missing;
+- which milestone enables them.
+
+## Backend Behavior
+Disabled endpoints return a typed `FEATURE_DISABLED` error, not 404 and not silent success.
@@ -0,0 +1,14 @@
+# Final Pre-Codex Checklist
+
+Before starting tomorrow's Codex build:
+
+- [ ] Extract latest full zip.
+- [ ] Open repo root in Codex environment.
+- [ ] Confirm `.env.example` exists.
+- [ ] Confirm Docker/PostGIS plan is present.
+- [ ] Give Codex `prompts/codex/M10_MASTER_AUTONOMOUS_PROMPT.md`.
+- [ ] Tell Codex to execute only one pass at a time.
+- [ ] After each pass, require pass report.
+- [ ] Reject pass if it changes frozen scope.
+- [ ] Reject pass if it hides missing functionality behind fake success.
+- [ ] Save output as new update/full zip after meaningful passes.
@@ -0,0 +1,30 @@
+# Frontend State Machine
+
+Every async module must model these states explicitly:
+
+- `idle`
+- `loading`
+- `empty`
+- `ready`
+- `error`
+- `unavailable`
+- `disabled`
+
+## Dataset Card
+- idle: not selected
+- loading: metadata extraction running
+- empty: no datasets uploaded
+- ready: dataset metadata available
+- error: upload/extraction failed
+- unavailable: operation unsupported for this dataset type
+- disabled: feature flag off
+
+## Analysis Run
+- queued
+- running
+- succeeded
+- failed
+- cancelled
+
+## UI Rule
+Never leave users with a blank panel. Every panel must explain what is happening or what to do next.
@@ -0,0 +1,29 @@
+# Geometry Contracts
+
+## Internal Geometry Standard
+- Database geometries are stored in PostGIS.
+- Default storage CRS: EPSG:31370 for Belgian projected operations where appropriate, or EPSG:4326 for API interchange.
+- API GeoJSON is always EPSG:4326 unless explicitly stated.
+- Area calculations must use projected CRS, not raw WGS84 degrees.
+
+## Accepted Area Input
+- GeoJSON Polygon
+- GeoJSON MultiPolygon
+- drawn polygon from frontend
+- future: municipality selection
+
+## Validation Rules
+- polygon must be closed;
+- polygon must be valid;
+- polygon must have non-zero area;
+- self-intersections must be rejected or repaired explicitly;
+- extremely large areas must require confirmation or be rejected by configured max area.
+
+## Output Rules
+- include `crs` metadata if transformed;
+- include computed area in square meters;
+- include bounds;
+- include geometry validity status.
+
+## Common Trap
+Never compute area or distance in degrees.
@@ -0,0 +1,16 @@
+# Known Limitations Template
+
+Use this file format after each major build.
+
+## Current Limitations
+| Area | Limitation | Impact | Workaround | Target Milestone |
+|---|---|---|---|---|
+| Detection | YOLO weights not configured | Demo adapter only | Configure local weights | V1.x |
+
+## Data Limitations
+- GRB live connector may require endpoint details or local downloads.
+- Sentinel automation is V2.
+- LiDAR is outside V1.
+
+## Technical Debt
+List only real debt, not unfinished planned scope.
@@ -0,0 +1,34 @@
+# Model Adapter Guide
+
+## Purpose
+Detection and segmentation must be implemented behind stable adapters so V1 can support demo/local model flows while remaining ready for YOLO/SAM integration.
+
+## Detection Adapter Interface
+```python
+class DetectionAdapter:
+    name: str
+    supported_classes: list[str]
+    def is_available(self) -> bool: ...
+    def predict(self, image_tile, parameters) -> list[DetectionResult]: ...
+```
+
+## Detection Result
+Fields:
+- class_name
+- confidence
+- bbox_pixel
+- bbox_geo optional after georeferencing
+- source_tile
+- metadata
+
+## Segmentation Adapter Interface
+```python
+class SegmentationAdapter:
+    name: str
+    supported_classes: list[str]
+    def is_available(self) -> bool: ...
+    def segment(self, image_tile, parameters) -> list[SegmentationResult]: ...
+```
+
+## V1 Rule
+If real YOLO weights are not configured, use a demo adapter only when clearly labelled as demo and never present it as production AI.
@@ -0,0 +1,34 @@
+# Observability Plan
+
+## Required Logs
+- request start/end with trace id;
+- dataset upload registration;
+- metadata extraction status;
+- job lifecycle transitions;
+- model adapter selection;
+- inference start/end;
+- QA/QC matching summary;
+- export creation.
+
+## Required Health Checks
+- API health;
+- database connection;
+- PostGIS extension availability;
+- storage write check;
+- optional Redis check;
+- optional model registry check.
+
+## Diagnostics Endpoint
+`GET /system/diagnostics`
+
+Should return safe non-secret status:
+```json
+{
+  "api": "ok",
+  "database": "ok",
+  "postgis": "ok",
+  "storage": "ok",
+  "redis": "unconfigured",
+  "models": "unconfigured"
+}
+```
@@ -0,0 +1,25 @@
+# Performance Budgets
+
+## API
+- health endpoint: < 200ms local
+- project list: < 500ms for 100 projects
+- dataset metadata: < 500ms after extraction
+- synchronous upload response: should register file quickly and hand off heavy work to job queue
+
+## Raster
+- never load large rasters fully into memory for preview;
+- prefer windowed reads and overviews;
+- large tiling must run as background job.
+
+## Vector
+- use spatial indexes for intersection/QA operations;
+- simplify only for display, not source-of-truth unless explicitly stored as derived layer.
+
+## Frontend
+- initial app load should not require heavy GIS data;
+- map layers should be lazy-loaded;
+- large GeoJSON should be paged, tiled, or simplified.
+
+## Jobs
+Long-running operations must have status transitions:
+`queued -> running -> succeeded|failed|cancelled`.
@@ -0,0 +1,41 @@
+# QA/QC Matching Algorithm
+
+## Goal
+Compare AI detections/segmentations with a reference layer such as GRB buildings.
+
+## Inputs
+- predicted polygons
+- reference polygons
+- class filter
+- IoU threshold, default 0.5
+
+## Steps
+1. Validate CRS alignment.
+2. Reproject to analytical CRS.
+3. Build spatial indexes.
+4. For every predicted polygon, find candidate reference polygons by bbox intersection.
+5. Compute IoU for candidates.
+6. Assign best match above threshold greedily, one reference per prediction.
+7. Count true positives, false positives, false negatives.
+8. Compute precision, recall, F1, mean IoU.
+9. Produce unmatched prediction layer and unmatched reference layer.
+
+## Metrics
+```text
+precision = TP / (TP + FP)
+recall = TP / (TP + FN)
+f1 = 2 * precision * recall / (precision + recall)
+IoU = intersection_area / union_area
+```
+
+## Output Layers
+- matched predictions
+- false positives
+- false negatives
+- low IoU matches
+
+## Edge Cases
+- empty predictions and empty reference: score should be explicit `no_objects` not perfect success;
+- empty predictions with reference: recall 0;
+- predictions with empty reference: precision 0;
+- invalid polygons must be repaired or excluded with warning.
@@ -0,0 +1,13 @@
+# M10 Ultra Preparation Pack
+
+This folder contains the additional control layer for making GeoIntel as autonomous-build-ready as possible for Codex.
+
+Purpose:
+- remove hidden architecture ambiguity;
+- give Codex concrete implementation contracts;
+- define allowed freedom versus frozen decisions;
+- provide validation and recovery procedures;
+- prevent common GIS/AI/web-app regressions;
+- make each build pass reviewable without manual guessing.
+
+M10 does not replace the previous documentation. It adds the final execution scaffolding around it.
@@ -0,0 +1,27 @@
+# V1 Release Gate
+
+GeoIntel V1 is releasable when all gates pass.
+
+## Product Gate
+- [ ] User can create project.
+- [ ] User can create/select area.
+- [ ] User can upload at least one vector dataset.
+- [ ] User can upload/register at least one raster dataset.
+- [ ] User can see datasets on map or metadata panel.
+- [ ] User can run a detection workflow or labelled demo adapter.
+- [ ] User can compare detections to reference polygons.
+- [ ] User can export GeoJSON.
+
+## Engineering Gate
+- [ ] Backend starts from clean checkout.
+- [ ] Frontend builds from clean checkout.
+- [ ] Database migrations apply.
+- [ ] Tests run.
+- [ ] Smoke script passes.
+- [ ] Docs explain setup.
+
+## Integrity Gate
+- [ ] No fake success states.
+- [ ] CRS assumptions documented.
+- [ ] Feature-disabled states are clear.
+- [ ] Known limitations are listed.
@@ -0,0 +1,28 @@
+# Repo Hygiene Rules
+
+## Do Commit
+- source code;
+- documentation;
+- small fixtures;
+- config examples;
+- migration files;
+- test data under size limits.
+
+## Do Not Commit
+- `.env` with secrets;
+- large rasters;
+- model weights;
+- generated cache;
+- local database volumes;
+- node_modules;
+- Python virtualenvs;
+- personal paths.
+
+## Naming
+- backend modules use snake_case;
+- frontend components use PascalCase;
+- docs use uppercase topic names or numbered folders;
+- fixtures describe region and purpose.
+
+## Generated Outputs
+Generated exports should go under `exports/` and be gitignored unless they are tiny documented sample fixtures.
@@ -0,0 +1,26 @@
+# Security and Secret Handling
+
+## Secrets
+Never commit API keys, tokens, model credentials, STAC credentials, database passwords, or private URLs.
+
+## Environment Variables
+All secrets must be loaded from `.env` or deployment environment.
+
+## File Upload Safety
+- limit accepted extensions;
+- validate MIME/type where possible;
+- store uploads outside source directories;
+- generate server-side filenames;
+- never execute uploaded files;
+- reject path traversal.
+
+## External Connectors
+- log endpoint names but not credentials;
+- timeout external requests;
+- cache responses where appropriate;
+- show connector status in UI.
+
+## AI/Model Safety
+- model files must be treated as artifacts;
+- do not auto-download arbitrary executable code;
+- keep model registry metadata separate from weights.
@@ -0,0 +1,37 @@
+# UI Copy Bank
+
+## Product Language
+Use:
+- GeoAI Workbench
+- Dataset
+- Analysis Run
+- Reference Layer
+- Detection Result
+- QA/QC
+- Export
+
+Avoid:
+- magic AI
+- automatic truth
+- perfect detection
+- black box conclusions
+
+## Empty States
+### No Projects
+"Start a GeoAI project by selecting an area in the Kempen or opening the prepared demo scenario."
+
+### No Datasets
+"Upload a GeoTIFF, GeoJSON, Shapefile, or use a connector to add reference data."
+
+### No Analysis Runs
+"Run a raster, vector, detection, segmentation, or QA/QC analysis to generate geospatial outputs."
+
+## Error Suggestions
+### CRS Missing
+"This dataset has no detected CRS. Add CRS metadata before using it in spatial operations."
+
+### Model Unavailable
+"No detection model is configured. Configure YOLO weights or use the labelled demo adapter."
+
+### Reference Missing
+"QA/QC requires a reference layer such as GRB buildings or a local reference GeoJSON."