geointel/docs/RELEASE_RUNBOOK.md

# GeoIntel Release Runbook

## Scope

This runbook releases GeoIntel for Belgium and the Belgian North Sea. Mol and
the Kempen remain regression references. A successful release never implies
that every theme is operational in every jurisdiction; the coverage API and
source provenance remain authoritative.

The repository version is stored in `VERSION`. The current release is
`v1.0.0`.

## Mandatory preconditions

- clean `main` worktree at the commit being released;
- secure non-default PostGIS password in the Tower `.env`;
- existing local AI model only when the AI image is enabled;
- recent checksum-verified database dump and byte-complete storage/model snapshot;
- Docker, `ssh-keygen`, Python 3.11, Node 20 and Bash available;
- one Alembic head and no unsupported metric represented as successful.

No command in this runbook downloads AI weights or implicitly deletes
application data.

## Repository gate

```bash
python -m compileall backend/app
cd backend && python -m pytest
cd ../frontend && npm run test:unit && npm run typecheck && npm run build
cd ..
bash scripts/run_readiness_check.sh
cd backend && python -m alembic heads
python -m alembic upgrade head --sql
cd ..
bash -n scripts/live_migration_smoke.sh
docker compose config
```

## Immutable deployment

`deploy-release.sh` first builds/reuses the candidate while the current release
stays available. Immediately before replacement it quiesces backend writes and
creates and checksum-verifies a database dump plus byte-complete SHA-256
storage/model snapshots under `/mnt/user/appdata/geointel/backups`. The first
snapshot copies every byte (using CoW reflinks when supported); later snapshots
hard-link only checksum-identical bytes from a verified older backup, never
from live storage. A conservative full-copy/free-space preflight runs before
backend quiescence. Only then may the
candidate start or run Alembic. If backup fails, the unchanged release is
restarted; deployment stops if existing PostGIS state cannot be backed up
consistently.

The backup manifest separates provenance intentionally:
`backup_tool_revision` is the new candidate source running the backup tool,
while `running_image_revision` is the OCI label of the old release whose data
is being captured. The retained Docker `image_id`, not either descriptive
revision field, is authoritative for rollback.

On the Codex workstation:

```powershell
.\scripts\deploy_tower.ps1
```

On Tower:

```bash
cd /mnt/user/appdata/geointel
docker inspect --format \
  '{{index .Config.Labels "org.opencontainers.image.version"}} {{index .Config.Labels "org.opencontainers.image.revision"}}' \
  geointel
curl -fsS http://127.0.0.1:1202/health/ready
bash scripts/live_migration_smoke.sh
```

## Backup and recovery proof

Create an immutable byte-complete backup. Initial storage/model copy and
verification can be I/O-heavy; subsequent backups deduplicate unchanged bytes
against the newest verified prior snapshot:

The gated Unraid job keeps its controller request attached for up to twelve
hours because an initial snapshot can exceed 100 GiB. Release runs for the same
branch queue instead of cancelling an in-flight deploy. Snapshot copying and
checksum verification use the lowest available best-effort CPU and I/O
priority (`nice` and `ionice`) while the deploy lock prevents overlap. If an
operator manually cancels the caller, inspect the controller state and
`/mnt/user/appdata/gitea-deploy-runners/deploy.lock` before retrying; never
assume that cancelling the HTTP client terminated the server-side operation.

```bash
bash scripts/backup_release_state.sh \
  --container geointel \
  --output-root /mnt/user/appdata/geointel/backups \
  --release-id v1.0.0 \
  --storage-path /mnt/user/appdata/geointel/storage \
  --models-path /mnt/user/appdata/geointel/models \
  --inventory-mode sha256
```

Verify and restore only into an automatically generated temporary database:

```bash
bash scripts/verify_release_backup.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/v1.0.0 \
  --container geointel
bash scripts/restore_release_backup_smoke.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/v1.0.0 \
  --container geointel \
  --confirm-isolated-restore
bash scripts/verify_release_upgrade_smoke.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/v1.0.0 \
  --container geointel \
  --confirm-isolated-upgrade
```

The restore and upgrade scripts refuse the production database name and remove
their generated verification database.

## Fresh install, browser and data operations

```bash
bash scripts/verify_release_fresh_install.sh \
  geointel-all-in-one:<release-commit>-ai
bash scripts/run_rc8_release_journeys.sh \
  http://127.0.0.1:1202 artifacts/releases/v1.0.0/rc8
bash scripts/run_rc9_ux_audit.sh \
  http://127.0.0.1:1202 artifacts/releases/v1.0.0/rc9
bash scripts/run_rc10_data_operations_audit.sh \
  artifacts/releases/v1.0.0/rc10
```

The RC10 command is read-only and runs cleanup in dry-run mode only.

## Supply-chain evidence

```bash
bash scripts/audit_python_dependencies.sh
cd frontend && npm audit --audit-level=high
cd ..
bash scripts/generate_container_sbom.sh \
  geointel-all-in-one:<release-commit>-ai \
  artifacts/releases/v1.0.0/geointel-sbom.spdx.json
bash scripts/scan_container_image.sh \
  geointel-all-in-one:<release-commit>-ai \
  artifacts/releases/v1.0.0/container-vulnerabilities.json
```

The complete vulnerability report remains evidence. The executable policy
gate fails on reachable fixed HIGH/CRITICAL findings.

## Rollback proof

Use the exact backup printed by the deployment. Rollback first stops the
candidate, restores and verifies the pre-deploy PostgreSQL dump in a temporary
proof database, then swaps database names while retaining the pre-restore
database as a recovery point. Only then does it start the exact image ID bound
into that backup; it never relies on a mutable global `previous` tag. It never runs an
Alembic downgrade or starts an old image against an unknown newer schema:

```bash
bash deploy/unraid/rollback-dockerman-container.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/<predeploy-release-id> \
  --confirm-production-database-restore
curl -fsS http://127.0.0.1:1202/health/ready
bash deploy/unraid/deploy-release.sh
curl -fsS http://127.0.0.1:1202/health/ready
```

Database rollback restores persisted rows and schema. Files newly written by a
failed candidate remain in storage as unreferenced evidence; the protected,
recoverable quarantine flow in `DATA_OPERATIONS_RUNBOOK.md` handles those
files without deleting source data.

After the rollback has remained healthy and its retained evidence has been
reviewed, list the recovery database printed by the script. Remove it only by
an explicit, separately approved `dropdb` maintenance command; deployment and
rollback never auto-delete recovery databases or backup directories. Retain at
least the current successful predeploy backup and its predecessor. Before
removing an older backup, run `verify_release_backup.sh` on the backups that
remain and confirm no newer snapshot hard-links depend on operator policy for
retention (hard-linked bytes remain allocated while any retained backup names
them). Remove its `rollback-predeploy-*` image tag only in the same explicitly
reviewed retention operation.

## Tag and signed package

Create an SSH-signed Git tag at the accepted clean commit. Use a configured
release key; never add the private key to the repository:

```bash
git -c gpg.format=ssh \
  -c user.signingkey=/secure/path/release-key \
  tag -s v1.0.0 -m "GeoIntel v1.0.0"
git push origin v1.0.0
```

Place the collected evidence files in an ignored package directory and create
the detached SSH signature plus complete checksum inventory:

```bash
python scripts/build_release_package.py build \
  --output-dir artifacts/releases/v1.0.0 \
  --release-id v1.0.0 \
  --image-name geointel-all-in-one:<release-commit>-ai \
  --image-id sha256:<image-id> \
  --image-revision <release-commit> \
  --signing-key /secure/path/release-key
python scripts/build_release_package.py verify \
  --package-dir artifacts/releases/v1.0.0
```

The builder refuses a dirty worktree, a tag not pointing at `HEAD`, a mismatched
image revision, missing evidence, symlinks, checksum drift or an invalid
signature.

## Safe cleanup and shutdown

Stop only the temporary smoke container by its generated name; normal scripts
already clean it automatically. To stop GeoIntel without deleting persistent
data:

```bash
docker stop geointel
```

Do not use broad Docker volume pruning. Application cleanup remains dry-run by
default and requires the exact confirmation, recent backup and delete ceiling
documented in `DATA_OPERATIONS_RUNBOOK.md`.