Initial public release

docs/RC_ROADMAP_BELGIUM_NORTH_SEA.md

@@ -0,0 +1,578 @@
+# Autonomous RC Roadmap: Belgium and the Belgian North Sea
+
+## Mission
+
+Turn the current GeoIntel implementation into a reproducible, demonstrable and
+release-candidate-quality map-first workbench for all of Belgium and the
+Belgian North Sea.
+
+This file is an executable runbook for Codex. It is both the implementation
+order and the release evidence index. Work continues autonomously between
+phases. A separate RC-12 phase is deliberately omitted.
+
+## Sources of truth
+
+Read in this order before changing behavior:
+
+1. `AGENTS.md`
+2. `docs/CODEX_BOOTSTRAP_PROMPT.md`
+3. `docs/RC_SCOPE_FREEZE_BELGIUM_NORTH_SEA.md`
+4. `docs/governance/GEOINTEL_CONSTITUTION.md`
+5. `docs/governance/ARCHITECTURE_INVARIANTS.md`
+6. `docs/governance/FORBIDDEN_DECISIONS.md`
+7. `docs/API_CONTRACTS.md`
+8. `docs/DATABASE_IMPLEMENTATION_PLAN.md`
+9. `docs/DATA_SPECIFICATION.md`
+10. `docs/DATA_SOURCES.md`
+11. `docs/STORAGE_ARCHITECTURE.md`
+12. `docs/DEFINITION_OF_DONE.md`
+
+## Autonomous execution contract
+
+Codex shall:
+
+- execute phases in order and continue without requesting routine approval;
+- use existing service, dataset, vector-feature, raster and job boundaries;
+- keep every provider bounded, fail-closed and provenance-complete;
+- add or update tests with every behavior change;
+- update `docs/CODEX_EXECUTION_LOG.md`, `docs/TODO.md` and `CHANGELOG.md`
+  after each completed phase;
+- run the phase gate and record exact output under
+  `storage/release-evidence/<release-id>/`;
+- commit and push only after the relevant local gate is green;
+- deploy only a commit that passed the local gate;
+- run browser and live API acceptance after deployment;
+- retain Mol/Kempen as regression references while adding Wallonia, Brussels,
+  cross-region, coast and North Sea golden areas.
+
+Codex stops only when:
+
+- an operation would destructively mutate or delete unbacked production data;
+- an external licence, credential or authority decision cannot be derived from
+  public source metadata;
+- an official endpoint fails security validation and no authoritative
+  alternative exists;
+- the same mandatory gate fails three times for the same external reason; or
+- continuing would require fabricating data, metrics, provenance or model
+  output.
+
+For a stop, write the blocker, evidence and exact resume command to the
+execution log. Continue with independent work when possible.
+
+## Release priorities
+
+### P0 - release blockers
+
+- truthful health/readiness and capability reporting;
+- current backup plus isolated restore proof;
+- stale job/run reconciliation and useful logs;
+- temporally correct detection QA;
+- production secrets, upload limits, configuration parity and rollback;
+- full CI and reproducible dependencies;
+- fresh install and upgrade proof.
+
+### P1 - national product completeness
+
+- national and maritime scope model;
+- governed source coverage matrix;
+- Wallonia, Brussels and North Sea adapters;
+- nationwide bounded selection and mixed-zone result handling;
+- historical compatibility matrix;
+- golden areas outside Flanders.
+
+### P2 - quality and maintainability
+
+- typed response models on critical routes;
+- frontend unit and browser E2E coverage;
+- loading/accessibility/performance hardening;
+- retention and operator documentation;
+- remaining bounded modularization where protected by tests.
+
+## Global gates
+
+Run after every behavior-changing phase:
+
+```bash
+python -m compileall backend/app
+cd backend && python -m pytest
+cd frontend && npm run typecheck
+cd frontend && npm run build
+bash scripts/run_readiness_check.sh
+cd backend && python -m alembic heads
+cd backend && python -m alembic upgrade head --sql
+bash -n scripts/live_migration_smoke.sh
+docker compose config
+```
+
+When Tower is reachable:
+
+```powershell
+.\scripts\deploy_tower.ps1
+```
+
+Then verify:
+
+```bash
+curl -fsS http://192.0.2.10:1202/health/live
+curl -fsS http://192.0.2.10:1202/health/ready
+curl -fsS http://192.0.2.10:1202/api/v1/system/capabilities
+```
+
+The in-app browser acceptance must cover desktop, widescreen and a narrow
+viewport. It must inspect console errors, failed requests and the complete
+map-select-analyse-export flow.
+
+## Official source baseline
+
+The roadmap starts from these verified official families. Machine endpoints,
+editions and licences must still pass source-specific probes before activation.
+
+| Scope | Candidate | Role |
+| --- | --- | --- |
+| Belgium | NGI/IGN AdminVector and Top10Vector | authoritative boundary and common topographic baseline |
+| Belgium | Statbel statistical sectors/population | authoritative statistical geometry and time series |
+| Flanders | existing Digitaal Vlaanderen/VMM/DOV/INBO/ALZ adapters | detailed authoritative regional layers |
+| Wallonia | SPW Géoportail, PICC, orthophoto, MNT and thematic services | detailed authoritative regional layers |
+| Brussels | Paradigm UrbIS and regional environmental geodata | detailed authoritative urban layers |
+| North Sea | FPS Marine Environment marine spatial plan | legal/use zones |
+| North Sea | RBINS/BMDC and Marine Atlas | marine environment and observations |
+| North Sea | MDK/Flemish Hydrography | bathymetry and nautical source candidate |
+| North Sea | EMODnet | contextual fallback only when explicitly labelled |
+
+## Phase status
+
+| Phase | State | Purpose |
+| --- | --- | --- |
+| RC-0 | complete | freeze scope and produce release evidence baseline |
+| RC-1 | complete | backup, restore and data safety |
+| RC-2 | complete | health, capabilities and stale-runtime correctness |
+| RC-3 | complete | temporal detection/QA correctness and observability |
+| RC-4 | complete | national/maritime scope and provider coverage contracts |
+| RC-5 | complete | deployment, secrets, configuration, fresh install and rollback |
+| RC-6 | complete | complete CI, dependency and supply-chain gates |
+| RC-7 | complete | critical API envelope typing and contract validation |
+| RC-8 | complete | frontend and browser E2E release journeys |
+| RC-9 | in progress | loading, accessibility and performance hardening |
+| RC-10 | pending | retention, cleanup and national data operations |
+| RC-11 | pending | final package, upgrade proof, release tag and handoff |
+
+## RC-0 - Scope freeze and evidence baseline
+
+**State: complete.** The read-only baseline command and focused tests pass.
+Runtime evidence is stored outside Git under `storage/release-evidence/`.
+
+### Work
+
+- Freeze Belgium plus Belgian North Sea scope.
+- Record current commit, migration head, dependency versions, container image,
+  database/storage sizes and readiness results.
+- Create a release-evidence manifest command.
+- Replace stale product-status wording that still treats Kempen as the product
+  boundary.
+- Add a coverage matrix schema with `operational`, `partial`,
+  `not_configured`, `unsupported`.
+
+### Likely files
+
+- `docs/RC_SCOPE_FREEZE_BELGIUM_NORTH_SEA.md`
+- `docs/RC_ROADMAP_BELGIUM_NORTH_SEA.md`
+- `docs/BUILD_STATUS.md`
+- `docs/DATA_SOURCES.md`
+- `scripts/capture_release_evidence.py`
+- `backend/tests/test_rc_release_evidence.py`
+
+### Evidence and exit
+
+- scope documents contain Belgium, all three regions and maritime zones;
+- evidence command is read-only and deterministic;
+- manifest contains commit, dirty state, Alembic head and gate status;
+- compile and focused tests pass.
+
+## RC-1 - Backup, restore and data safety
+
+**State: complete.** Tower retains the checksum-verified 1.4 GiB backup
+`rc-belgium-north-sea-fc42ea9-secure`. Its manifest records a non-default
+database password. An isolated restore reached PostGIS 3.6 and Alembic
+`202607160001`, reconciled every retained critical table count and removed the
+temporary database.
+
+### Work
+
+- Add a safe production backup command for PostgreSQL custom-format dumps.
+- Capture schema head, PostGIS version, table counts, model checksums and a
+  storage checksum inventory without duplicating all large artifacts.
+- Reject default/empty production database passwords.
+- Add backup verification with `pg_restore --list` and checksums.
+- Add an isolated restore drill using a temporary database/container; never
+  restore over production.
+- Document retention, encryption boundary and cleanup.
+
+### Likely files
+
+- `scripts/backup_release_state.sh`
+- `scripts/verify_release_backup.sh`
+- `scripts/restore_release_backup_smoke.sh`
+- `deploy/unraid/README.md`
+- `docs/ROLLBACK_AND_RECOVERY.md`
+- `backend/tests/test_rc_backup_restore_scripts.py`
+
+### Exit
+
+- current Tower database has a dated, checksum-verified backup;
+- isolated restore reaches the recorded Alembic head;
+- representative project, dataset, geometry and QA counts reconcile;
+- no production application data was mutated by the drill.
+
+## RC-2 - Truthful runtime state
+
+**State: complete.** Local compile, 963 backend tests, frontend
+typecheck/build and the full readiness gate pass. Tower reports green
+DB/PostGIS/migration/storage readiness and configured YOLO capability. Startup
+reconciled five orphaned jobs and two orphaned analysis runs; zero remain
+`running`.
+
+### Work
+
+- Split process liveness from dependency readiness.
+- Make readiness return HTTP 503 when DB, PostGIS, migration head or writable
+  storage is unavailable.
+- Derive version/build identity and AI capability from runtime configuration.
+- Make Docker use the readiness endpoint.
+- Reconcile orphaned `running` jobs and analysis runs on process startup.
+- Add request IDs and exception logging without exposing secrets or full SQL.
+
+### Likely files
+
+- `backend/app/api/routes/health.py`
+- `backend/app/schemas/health.py`
+- `backend/app/core/config.py`
+- `backend/app/core/logging.py`
+- `backend/app/main.py`
+- `backend/app/services/runtime_reconciliation_service.py`
+- Docker/nginx files and health tests
+
+### Exit
+
+- liveness remains 200 while the process runs;
+- readiness is fail-closed;
+- live capabilities match the active local YOLO state;
+- restart converts impossible orphaned work to a terminal failed state with
+  `PROCESS_INTERRUPTED`;
+- Docker reports unhealthy for a broken database.
+
+## RC-3 - Temporal QA correctness and observability
+
+**State: complete.** The local release gate passes backend compilation, all
+974 backend tests, frontend typecheck/build, the integrated readiness script,
+one Alembic head and complete offline migration SQL generation. Tower runs
+immutable build `a28f2497e8d0da7d108828875ba9ffedda3c1688`, reports PostgreSQL,
+PostGIS 3.6, migration `202607160001`, storage and configured YOLO as ready,
+and contains no orphaned running Job or AnalysisRun. Live browser acceptance
+proved that Detection Lab starts with an empty raster choice, disables its
+primary action and reports the required operator choice without console
+errors.
+
+### Work
+
+- Stop Detection Lab from silently selecting the first raster.
+- Require explicit raster/model choice.
+- Prevent current reference QA against historical imagery unless a compatible
+  reference edition/coverage is selected.
+- Persist source observation period and QA compatibility decision.
+- Add structured request/job/run logs and operator diagnostics.
+- Add a stale-runtime report and safe reconciliation command.
+
+### Exit
+
+- no detection run starts with an implicit old raster;
+- incompatible temporal QA fails with a clear code;
+- compatible current orthophoto/GRB path remains green;
+- logs correlate request, job, analysis run and quality check.
+
+## RC-4 - Belgium and North Sea coverage foundation
+
+**State: complete.** Tower persists one idempotent national workspace with
+eight distinct land/maritime areas and six ready authoritative datasets
+containing 685 features. The coverage catalog and resolver report operational,
+partial, not-configured and unsupported states honestly for Belgium, all three
+regions and the Belgian maritime zones. Live browser acceptance proved the
+national default context, coverage-only selection and responsive 390/1920 px
+layouts while unavailable detailed regional themes remain explicit.
+
+### Work
+
+1. Persist NGI authoritative Belgium land and administrative scopes.
+2. Persist territorial sea, EEZ and continental shelf as distinct scope
+   layers.
+3. Implement the coverage matrix and source authority resolver.
+4. Add bounded provider contracts for:
+   - NGI/Statbel national baseline;
+   - SPW/PICC/orthophoto/relief for Wallonia;
+   - UrbIS/environmental context for Brussels;
+   - federal marine zones and BMDC/MDK marine sources.
+5. Normalize the public theme vocabulary without discarding provider-native
+   semantics.
+6. Add Wallonia, Brussels, cross-region, coast and North Sea golden areas.
+7. Make mixed-zone selection split into bounded provider requests and merge
+   only semantically compatible metrics.
+
+### Required rules
+
+- all persistence flows through DatasetService and geospatial services;
+- no provider writes directly to feature tables;
+- no country-wide startup import;
+- no TLS bypass;
+- no implicit vertical datum conversion;
+- unsupported metrics remain unavailable.
+
+### Likely files
+
+- provider registry/base and new regional provider modules;
+- geographic scope and source coverage services;
+- dataset/source schemas;
+- map source portfolio and scope UI;
+- source, API, data and storage documentation;
+- source-specific tests and fixtures.
+
+### Exit
+
+- a user can draw within every golden area and receive an honest coverage
+  matrix;
+- at least the common national baseline is operational on land;
+- regional detailed layers are operational or explicitly partial;
+- maritime legal zones are operational;
+- bathymetry remains gated until a source passes strict acquisition evidence.
+
+## RC-5 - Production deployment and rollback
+
+**State: complete.** Tower runs a commit-plus-AI-profile immutable image with
+OCI revision/build labels. Repeated deploys reuse the exact image and preserve
+the actual prior release. A fresh install passed on isolated volumes, manual
+rollback passed against retained production mounts, and the release image
+upgraded a checksum-verified 1.4 GiB backup in a generated temporary database
+at PostGIS 3.6/Alembic `202607160001` before removing it. Production startup is
+fail-closed for default secrets and invalid upload limits, nginx/backend limits
+are aligned, and all operator-owned settings are editable in Unraid.
+
+### Work
+
+- align backend, nginx and proxy upload/time limits;
+- make every runtime setting intentionally configured, internal or documented;
+- reject production default secrets;
+- add image labels with version, commit and build time;
+- deploy immutable tags in addition to `latest`;
+- preserve previous image for one-command rollback;
+- prove fresh install against empty volumes;
+- prove upgrade from a copied current database/storage state;
+- prove rollback without schema or data loss.
+
+### Exit
+
+- Unraid edit fields cover all operator-owned settings;
+- fresh install, upgrade and rollback commands are documented and executed;
+- deployed UI exposes build identity;
+- previous image remains addressable;
+- no mutable-only release is accepted.
+
+## RC-6 - CI and supply-chain gates
+
+**State: complete.** Equivalent Gitea and GitHub workflows now enforce the
+complete readiness gate, offline migrations, Compose resolution, dependency
+audits, immutable container build, SPDX SBOM and Trivy policy. Linux/Python
+3.11 runtime and CI locks are hash-pinned and optional AI dependencies remain
+separate. The live AI image passed the container policy with zero reachable
+fixed HIGH/CRITICAL findings; its complete report and 25.7 MB SBOM are retained
+on Tower. The report deliberately preserves two timeboxed Starlette findings
+and fifteen shadowed base-layer `gosu` findings. Runtime evidence proves that
+the final `gosu` path is the audited `setpriv` wrapper running as PostgreSQL
+UID 999.
+
+### Work
+
+- make CI run backend compile/tests, frontend typecheck/build, readiness,
+  Alembic single-head/offline SQL and Docker config/build;
+- add Python and npm dependency vulnerability checks;
+- add container scan and SBOM;
+- introduce reproducible Python resolution/lock output;
+- preserve optional AI dependency separation;
+- publish gate artefacts.
+
+### Exit
+
+- a broken compile, test, build, migration or contract blocks CI;
+- dependency and container findings have severity policy;
+- a clean checkout can reproduce dependency versions and image metadata.
+
+## RC-7 - Critical API contract hardening
+
+**State: complete.** All successful JSON routes now publish concrete OpenAPI
+response schemas. The contract audit covers 124 implemented routes and 228
+component schemas; the eight non-envelope operations are the three health
+probes, four persisted PNG responses and the export download. Readiness passed
+with 1,008 backend tests, frontend typecheck/build, one Alembic head and
+offline migration SQL generation.
+
+### Work
+
+- replace `response_model=dict` first on health, projects, areas, datasets,
+  jobs, analysis runs, QA, exports and map-critical endpoints;
+- retain canonical `{data: ...}` success and canonical error envelopes;
+- add OpenAPI response validation tests;
+- document intentionally streaming/download responses separately.
+
+### Exit
+
+- all critical map-first routes have concrete response models;
+- actual payloads pass FastAPI validation;
+- API contract audit and frontend typecheck pass.
+
+## RC-8 - Release journey automation
+
+**State: complete.** Four Vitest suites protect selection, coverage, temporal
+comparison and bootstrap behavior with 12 tests. The Playwright release runner
+provisions seven deterministic land, cross-region, coastal and maritime
+golden areas in the national workbench through canonical APIs and executes the complete browser/API
+journey against the live PostGIS runtime. Evidence includes governed Mol
+metrics and provenance, compatible forest history, no-data, partial coverage,
+unsupported maritime metrics, simulated provider failure, persistent export,
+real local Ollama context and an explicit configured-YOLO run. The green run
+contains no unexpected browser-console or failed-request events.
+
+### Work
+
+- add frontend unit tests for selection, coverage, loading and temporal guards;
+- add browser E2E for:
+  - open Belgium map;
+  - select each golden area;
+  - load a theme;
+  - inspect metric/provenance;
+  - compare compatible history;
+  - run a bounded configured detection path;
+  - export;
+  - ask Ollama with persisted context;
+- include no-data, partial coverage, provider failure and unsupported metric
+  journeys.
+
+### Exit
+
+- E2E runs against a real backend/PostGIS fixture;
+- no workflow depends only on source-text assertion tests;
+- console and failed-request audits are clean.
+
+### Reproduction
+
+```bash
+npm --prefix frontend run test:unit
+bash scripts/run_rc8_release_journeys.sh \
+  http://192.0.2.10:1202 \
+  artifacts/rc8-release-journeys \
+  artifacts/rc8-golden-areas.json
+```
+
+The second command is intentionally live and mutating. It creates only missing
+golden Areas, copies the official Mol/Kempen geometry into the national
+workbench with source identifiers, uses the existing explicit demo-fixture flow for AI validation
+and writes screenshots plus `manifest.json` under the ignored `artifacts/`
+directory.
+
+## RC-9 - UX, accessibility and performance
+
+**State: complete.** The live immutable AI image passed the RC-9 browser audit
+at 390x844, 1366x768 and 2560x1080 without horizontal overflow, browser-console
+errors or failed requests. All seven workspaces expose accessible control
+names, bootstrap loading remains truthful under delayed API responses and
+coverage/map timing budgets have visible operator feedback. Repository
+readiness passed with 1,015 backend tests and 16 frontend tests.
+
+### Work
+
+- replace initial false `missing` source states with explicit loading;
+- provide accessible names, labels, keyboard focus and status announcements;
+- enforce desktop, widescreen and narrow layouts;
+- record API and map-selection budgets for golden areas;
+- virtualize or page large panels;
+- split large modules only where tests protect behavior.
+
+### Exit
+
+- no core control lacks an accessible name;
+- loading, empty, partial, error and ready states are distinct;
+- map selection remains usable at target viewports;
+- no release journey exceeds its documented budget without a visible warning.
+
+## RC-10 - Data operations and retention
+
+**State: complete.**
+
+Release image `22fb8d51a9fde39552cf06a789174441631842b6-ai`
+passed the live read-only data-operations audit. It found zero missing direct
+database artifact references, left every critical table count unchanged,
+reported 2,002 bounded cleanup candidates (218,263,512 bytes) without deleting
+them and classified disk pressure as `ok` with approximately 725 GB free.
+The 224 missing historical manifest intermediates remain an explicit
+provenance warning and are excluded from cleanup decisions. National and
+maritime freshness evidence classifies all three fixed official editions as
+`current`, with zero due sources and zero integrity issues.
+
+### Work
+
+- define lifecycle for raw, normalized, derived, export, AI and evidence
+  artifacts;
+- add dry-run-first cleanup for stale technical projects, failed work and
+  superseded caches;
+- retain immutable official source editions and release evidence;
+- report disk pressure before acquisition;
+- schedule nothing implicitly; provide explicit operator commands;
+- add source freshness reports for national/regional/maritime families.
+
+### Exit
+
+- cleanup cannot remove release evidence or current authoritative editions;
+- every destructive action requires explicit confirmation and a current
+  backup;
+- storage growth and retained provenance are reportable.
+
+## RC-11 - Final release package
+
+**State: complete.**
+
+The release-candidate gate passed 1,030 backend tests and 16 frontend tests,
+frontend typecheck/build, offline and live migration validation, isolated
+fresh install, checksum-verified backup, isolated restore/upgrade and an
+actual previous-image rollback plus forward deployment. Belgium/North Sea
+release journeys and the 390/1366/2560 UX audit passed against port 1202.
+Python/npm policy, SPDX SBOM and the container policy scan passed with zero
+reachable fixed HIGH/CRITICAL findings. The exact tagged commit, immutable
+image and detached SSH signature are bound in the external checksummed release
+manifest.
+
+### Work
+
+- rerun fresh-install and upgrade proof using the release image;
+- rerun backup/isolated restore and rollback proof;
+- execute every golden area and core browser journey;
+- generate coverage, security, dependency, migration, performance and known
+  limitation reports;
+- update operator/end-user documentation;
+- remove stale status documents from active navigation without deleting
+  historical evidence;
+- assign semantic version, create signed/checksummed release manifest, tag,
+  push and deploy the immutable image.
+
+### Final acceptance
+
+- all P0 findings are closed;
+- every mandatory global gate is green;
+- backup and isolated restore are proven;
+- one Alembic head applies to empty and upgraded PostGIS;
+- Belgium and maritime golden areas pass;
+- no unsupported metric or unavailable source appears as successful;
+- live browser acceptance passes at port 1202;
+- remaining P1/P2 limitations are explicit, bounded and non-deceptive.
+
+## Current autonomous phase
+
+RC-0 through RC-11 are complete. No separate RC-12 phase exists. Continue only
+with bounded post-RC work explicitly selected from the known limitations; do
+not reinterpret a jurisdiction's `partial`, `not_configured` or `unsupported`
+coverage as release success.