Initial public release

2026-08-31 21:56:53 +02:00
commit faeb58ef6d
1386 changed files with 263203 additions and 0 deletions
@@ -0,0 +1,230 @@
+# GeoIntel Release Runbook
+
+## Scope
+
+This runbook releases GeoIntel for Belgium and the Belgian North Sea. Mol and
+the Kempen remain regression references. A successful release never implies
+that every theme is operational in every jurisdiction; the coverage API and
+source provenance remain authoritative.
+
+The repository version is stored in `VERSION`. The current release is
+`v1.0.0`.
+
+## Mandatory preconditions
+
+- clean `main` worktree at the commit being released;
+- secure non-default PostGIS password in the Tower `.env`;
+- existing local AI model only when the AI image is enabled;
+- recent checksum-verified database dump and byte-complete storage/model snapshot;
+- Docker, `ssh-keygen`, Python 3.11, Node 20 and Bash available;
+- one Alembic head and no unsupported metric represented as successful.
+
+No command in this runbook downloads AI weights or implicitly deletes
+application data.
+
+## Repository gate
+
+```bash
+python -m compileall backend/app
+cd backend && python -m pytest
+cd ../frontend && npm run test:unit && npm run typecheck && npm run build
+cd ..
+bash scripts/run_readiness_check.sh
+cd backend && python -m alembic heads
+python -m alembic upgrade head --sql
+cd ..
+bash -n scripts/live_migration_smoke.sh
+docker compose config
+```
+
+## Immutable deployment
+
+`deploy-release.sh` first builds/reuses the candidate while the current release
+stays available. Immediately before replacement it quiesces backend writes and
+creates and checksum-verifies a database dump plus byte-complete SHA-256
+storage/model snapshots under `/mnt/user/appdata/geointel/backups`. The first
+snapshot copies every byte (using CoW reflinks when supported); later snapshots
+hard-link only checksum-identical bytes from a verified older backup, never
+from live storage. A conservative full-copy/free-space preflight runs before
+backend quiescence. Only then may the
+candidate start or run Alembic. If backup fails, the unchanged release is
+restarted; deployment stops if existing PostGIS state cannot be backed up
+consistently.
+
+The backup manifest separates provenance intentionally:
+`backup_tool_revision` is the new candidate source running the backup tool,
+while `running_image_revision` is the OCI label of the old release whose data
+is being captured. The retained Docker `image_id`, not either descriptive
+revision field, is authoritative for rollback.
+
+On the Codex workstation:
+
+```powershell
+.\scripts\deploy_tower.ps1
+```
+
+On Tower:
+
+```bash
+cd /mnt/user/appdata/geointel
+docker inspect --format \
+  '{{index .Config.Labels "org.opencontainers.image.version"}} {{index .Config.Labels "org.opencontainers.image.revision"}}' \
+  geointel
+curl -fsS http://127.0.0.1:1202/health/ready
+bash scripts/live_migration_smoke.sh
+```
+
+## Backup and recovery proof
+
+Create an immutable byte-complete backup. Initial storage/model copy and
+verification can be I/O-heavy; subsequent backups deduplicate unchanged bytes
+against the newest verified prior snapshot:
+
+The gated Unraid job keeps its controller request attached for up to twelve
+hours because an initial snapshot can exceed 100 GiB. Release runs for the same
+branch queue instead of cancelling an in-flight deploy. Snapshot copying and
+checksum verification use the lowest available best-effort CPU and I/O
+priority (`nice` and `ionice`) while the deploy lock prevents overlap. If an
+operator manually cancels the caller, inspect the controller state and
+`/mnt/user/appdata/gitea-deploy-runners/deploy.lock` before retrying; never
+assume that cancelling the HTTP client terminated the server-side operation.
+
+```bash
+bash scripts/backup_release_state.sh \
+  --container geointel \
+  --output-root /mnt/user/appdata/geointel/backups \
+  --release-id v1.0.0 \
+  --storage-path /mnt/user/appdata/geointel/storage \
+  --models-path /mnt/user/appdata/geointel/models \
+  --inventory-mode sha256
+```
+
+Verify and restore only into an automatically generated temporary database:
+
+```bash
+bash scripts/verify_release_backup.sh \
+  --backup-dir /mnt/user/appdata/geointel/backups/v1.0.0 \
+  --container geointel
+bash scripts/restore_release_backup_smoke.sh \
+  --backup-dir /mnt/user/appdata/geointel/backups/v1.0.0 \
+  --container geointel \
+  --confirm-isolated-restore
+bash scripts/verify_release_upgrade_smoke.sh \
+  --backup-dir /mnt/user/appdata/geointel/backups/v1.0.0 \
+  --container geointel \
+  --confirm-isolated-upgrade
+```
+
+The restore and upgrade scripts refuse the production database name and remove
+their generated verification database.
+
+## Fresh install, browser and data operations
+
+```bash
+bash scripts/verify_release_fresh_install.sh \
+  geointel-all-in-one:<release-commit>-ai
+bash scripts/run_rc8_release_journeys.sh \
+  http://127.0.0.1:1202 artifacts/releases/v1.0.0/rc8
+bash scripts/run_rc9_ux_audit.sh \
+  http://127.0.0.1:1202 artifacts/releases/v1.0.0/rc9
+bash scripts/run_rc10_data_operations_audit.sh \
+  artifacts/releases/v1.0.0/rc10
+```
+
+The RC10 command is read-only and runs cleanup in dry-run mode only.
+
+## Supply-chain evidence
+
+```bash
+bash scripts/audit_python_dependencies.sh
+cd frontend && npm audit --audit-level=high
+cd ..
+bash scripts/generate_container_sbom.sh \
+  geointel-all-in-one:<release-commit>-ai \
+  artifacts/releases/v1.0.0/geointel-sbom.spdx.json
+bash scripts/scan_container_image.sh \
+  geointel-all-in-one:<release-commit>-ai \
+  artifacts/releases/v1.0.0/container-vulnerabilities.json
+```
+
+The complete vulnerability report remains evidence. The executable policy
+gate fails on reachable fixed HIGH/CRITICAL findings.
+
+## Rollback proof
+
+Use the exact backup printed by the deployment. Rollback first stops the
+candidate, restores and verifies the pre-deploy PostgreSQL dump in a temporary
+proof database, then swaps database names while retaining the pre-restore
+database as a recovery point. Only then does it start the exact image ID bound
+into that backup; it never relies on a mutable global `previous` tag. It never runs an
+Alembic downgrade or starts an old image against an unknown newer schema:
+
+```bash
+bash deploy/unraid/rollback-dockerman-container.sh \
+  --backup-dir /mnt/user/appdata/geointel/backups/<predeploy-release-id> \
+  --confirm-production-database-restore
+curl -fsS http://127.0.0.1:1202/health/ready
+bash deploy/unraid/deploy-release.sh
+curl -fsS http://127.0.0.1:1202/health/ready
+```
+
+Database rollback restores persisted rows and schema. Files newly written by a
+failed candidate remain in storage as unreferenced evidence; the protected,
+recoverable quarantine flow in `DATA_OPERATIONS_RUNBOOK.md` handles those
+files without deleting source data.
+
+After the rollback has remained healthy and its retained evidence has been
+reviewed, list the recovery database printed by the script. Remove it only by
+an explicit, separately approved `dropdb` maintenance command; deployment and
+rollback never auto-delete recovery databases or backup directories. Retain at
+least the current successful predeploy backup and its predecessor. Before
+removing an older backup, run `verify_release_backup.sh` on the backups that
+remain and confirm no newer snapshot hard-links depend on operator policy for
+retention (hard-linked bytes remain allocated while any retained backup names
+them). Remove its `rollback-predeploy-*` image tag only in the same explicitly
+reviewed retention operation.
+
+## Tag and signed package
+
+Create an SSH-signed Git tag at the accepted clean commit. Use a configured
+release key; never add the private key to the repository:
+
+```bash
+git -c gpg.format=ssh \
+  -c user.signingkey=/secure/path/release-key \
+  tag -s v1.0.0 -m "GeoIntel v1.0.0"
+git push origin v1.0.0
+```
+
+Place the collected evidence files in an ignored package directory and create
+the detached SSH signature plus complete checksum inventory:
+
+```bash
+python scripts/build_release_package.py build \
+  --output-dir artifacts/releases/v1.0.0 \
+  --release-id v1.0.0 \
+  --image-name geointel-all-in-one:<release-commit>-ai \
+  --image-id sha256:<image-id> \
+  --image-revision <release-commit> \
+  --signing-key /secure/path/release-key
+python scripts/build_release_package.py verify \
+  --package-dir artifacts/releases/v1.0.0
+```
+
+The builder refuses a dirty worktree, a tag not pointing at `HEAD`, a mismatched
+image revision, missing evidence, symlinks, checksum drift or an invalid
+signature.
+
+## Safe cleanup and shutdown
+
+Stop only the temporary smoke container by its generated name; normal scripts
+already clean it automatically. To stop GeoIntel without deleting persistent
+data:
+
+```bash
+docker stop geointel
+```
+
+Do not use broad Docker volume pruning. Application cleanup remains dry-run by
+default and requires the exact confirmation, recent backup and delete ceiling
+documented in `DATA_OPERATIONS_RUNBOOK.md`.