geointel/docs/RELEASE_RUNBOOK.md

GeoIntel Release Runbook

Scope

This runbook releases GeoIntel for Belgium and the Belgian North Sea. Mol and the Kempen remain regression references. A successful release never implies that every theme is operational in every jurisdiction; the coverage API and source provenance remain authoritative.

The repository version is stored in VERSION. The current release is v1.0.0.

Mandatory preconditions

• clean main worktree at the commit being released;
• secure non-default PostGIS password in the Tower .env;
• existing local AI model only when the AI image is enabled;
• recent checksum-verified database dump and byte-complete storage/model snapshot;
• Docker, ssh-keygen, Python 3.11, Node 20 and Bash available;
• one Alembic head and no unsupported metric represented as successful.

No command in this runbook downloads AI weights or implicitly deletes application data.

Repository gate

python -m compileall backend/app
cd backend && python -m pytest
cd ../frontend && npm run test:unit && npm run typecheck && npm run build
cd ..
bash scripts/run_readiness_check.sh
cd backend && python -m alembic heads
python -m alembic upgrade head --sql
cd ..
bash -n scripts/live_migration_smoke.sh
docker compose config

Immutable deployment

deploy-release.sh first builds/reuses the candidate while the current release stays available. Immediately before replacement it quiesces backend writes and creates and checksum-verifies a database dump plus byte-complete SHA-256 storage/model snapshots under /mnt/user/appdata/geointel/backups. The first snapshot copies every byte (using CoW reflinks when supported); later snapshots hard-link only checksum-identical bytes from a verified older backup, never from live storage. A conservative full-copy/free-space preflight runs before backend quiescence. Only then may the candidate start or run Alembic. If backup fails, the unchanged release is restarted; deployment stops if existing PostGIS state cannot be backed up consistently.

The backup manifest separates provenance intentionally: backup_tool_revision is the new candidate source running the backup tool, while running_image_revision is the OCI label of the old release whose data is being captured. The retained Docker image_id, not either descriptive revision field, is authoritative for rollback.

On the Codex workstation:

.\scripts\deploy_tower.ps1

On Tower:

cd /mnt/user/appdata/geointel
docker inspect --format \
  '{{index .Config.Labels "org.opencontainers.image.version"}} {{index .Config.Labels "org.opencontainers.image.revision"}}' \
  geointel
curl -fsS http://127.0.0.1:1202/health/ready
bash scripts/live_migration_smoke.sh

Backup and recovery proof

Create an immutable byte-complete backup. Initial storage/model copy and verification can be I/O-heavy; subsequent backups deduplicate unchanged bytes against the newest verified prior snapshot:

bash scripts/backup_release_state.sh \
  --container geointel \
  --output-root /mnt/user/appdata/geointel/backups \
  --release-id v1.0.0 \
  --storage-path /mnt/user/appdata/geointel/storage \
  --models-path /mnt/user/appdata/geointel/models \
  --inventory-mode sha256

Verify and restore only into an automatically generated temporary database:

bash scripts/verify_release_backup.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/v1.0.0 \
  --container geointel
bash scripts/restore_release_backup_smoke.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/v1.0.0 \
  --container geointel \
  --confirm-isolated-restore
bash scripts/verify_release_upgrade_smoke.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/v1.0.0 \
  --container geointel \
  --confirm-isolated-upgrade

The restore and upgrade scripts refuse the production database name and remove their generated verification database.

Fresh install, browser and data operations

bash scripts/verify_release_fresh_install.sh \
  geointel-all-in-one:<release-commit>-ai
bash scripts/run_rc8_release_journeys.sh \
  http://127.0.0.1:1202 artifacts/releases/v1.0.0/rc8
bash scripts/run_rc9_ux_audit.sh \
  http://127.0.0.1:1202 artifacts/releases/v1.0.0/rc9
bash scripts/run_rc10_data_operations_audit.sh \
  artifacts/releases/v1.0.0/rc10

The RC10 command is read-only and runs cleanup in dry-run mode only.

Supply-chain evidence

bash scripts/audit_python_dependencies.sh
cd frontend && npm audit --audit-level=high
cd ..
bash scripts/generate_container_sbom.sh \
  geointel-all-in-one:<release-commit>-ai \
  artifacts/releases/v1.0.0/geointel-sbom.spdx.json
bash scripts/scan_container_image.sh \
  geointel-all-in-one:<release-commit>-ai \
  artifacts/releases/v1.0.0/container-vulnerabilities.json

The complete vulnerability report remains evidence. The executable policy gate fails on reachable fixed HIGH/CRITICAL findings.

Rollback proof

Use the exact backup printed by the deployment. Rollback first stops the candidate, restores and verifies the pre-deploy PostgreSQL dump in a temporary proof database, then swaps database names while retaining the pre-restore database as a recovery point. Only then does it start the exact image ID bound into that backup; it never relies on a mutable global previous tag. It never runs an Alembic downgrade or starts an old image against an unknown newer schema:

bash deploy/unraid/rollback-dockerman-container.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/<predeploy-release-id> \
  --confirm-production-database-restore
curl -fsS http://127.0.0.1:1202/health/ready
bash deploy/unraid/deploy-release.sh
curl -fsS http://127.0.0.1:1202/health/ready

Database rollback restores persisted rows and schema. Files newly written by a failed candidate remain in storage as unreferenced evidence; the protected, recoverable quarantine flow in DATA_OPERATIONS_RUNBOOK.md handles those files without deleting source data.

After the rollback has remained healthy and its retained evidence has been reviewed, list the recovery database printed by the script. Remove it only by an explicit, separately approved dropdb maintenance command; deployment and rollback never auto-delete recovery databases or backup directories. Retain at least the current successful predeploy backup and its predecessor. Before removing an older backup, run verify_release_backup.sh on the backups that remain and confirm no newer snapshot hard-links depend on operator policy for retention (hard-linked bytes remain allocated while any retained backup names them). Remove its rollback-predeploy-* image tag only in the same explicitly reviewed retention operation.

Tag and signed package

Create an SSH-signed Git tag at the accepted clean commit. Use a configured release key; never add the private key to the repository:

git -c gpg.format=ssh \
  -c user.signingkey=/secure/path/release-key \
  tag -s v1.0.0 -m "GeoIntel v1.0.0"
git push origin v1.0.0

Place the collected evidence files in an ignored package directory and create the detached SSH signature plus complete checksum inventory:

python scripts/build_release_package.py build \
  --output-dir artifacts/releases/v1.0.0 \
  --release-id v1.0.0 \
  --image-name geointel-all-in-one:<release-commit>-ai \
  --image-id sha256:<image-id> \
  --image-revision <release-commit> \
  --signing-key /secure/path/release-key
python scripts/build_release_package.py verify \
  --package-dir artifacts/releases/v1.0.0

The builder refuses a dirty worktree, a tag not pointing at HEAD, a mismatched image revision, missing evidence, symlinks, checksum drift or an invalid signature.

Safe cleanup and shutdown

Stop only the temporary smoke container by its generated name; normal scripts already clean it automatically. To stop GeoIntel without deleting persistent data:

docker stop geointel

Do not use broad Docker volume pruning. Application cleanup remains dry-run by default and requires the exact confirmation, recent backup and delete ceiling documented in DATA_OPERATIONS_RUNBOOK.md.