fix(release): make deployment backup and rollback immutable

This commit is contained in:
Jens
2026-08-30 06:00:43 +02:00
parent a0884d64c9
commit c272220277
47 changed files with 3035 additions and 430 deletions
+46 -7
View File
@@ -15,7 +15,7 @@ The repository version is stored in `VERSION`. The current release is
- clean `main` worktree at the commit being released;
- secure non-default PostGIS password in the Tower `.env`;
- existing local AI model only when the AI image is enabled;
- recent checksum-verified backup with SHA-256 storage inventory;
- recent checksum-verified database dump and byte-complete storage/model snapshot;
- Docker, `ssh-keygen`, Python 3.11, Node 20 and Bash available;
- one Alembic head and no unsupported metric represented as successful.
@@ -39,6 +39,24 @@ docker compose config
## Immutable deployment
`deploy-release.sh` first builds/reuses the candidate while the current release
stays available. Immediately before replacement it quiesces backend writes and
creates and checksum-verifies a database dump plus byte-complete SHA-256
storage/model snapshots under `/mnt/user/appdata/geointel/backups`. The first
snapshot copies every byte (using CoW reflinks when supported); later snapshots
hard-link only checksum-identical bytes from a verified older backup, never
from live storage. A conservative full-copy/free-space preflight runs before
backend quiescence. Only then may the
candidate start or run Alembic. If backup fails, the unchanged release is
restarted; deployment stops if existing PostGIS state cannot be backed up
consistently.
The backup manifest separates provenance intentionally:
`backup_tool_revision` is the new candidate source running the backup tool,
while `running_image_revision` is the OCI label of the old release whose data
is being captured. The retained Docker `image_id`, not either descriptive
revision field, is authoritative for rollback.
On the Codex workstation:
```powershell
@@ -58,8 +76,9 @@ bash scripts/live_migration_smoke.sh
## Backup and recovery proof
Create an immutable backup. The SHA-256 inventory can take several minutes on
large storage:
Create an immutable byte-complete backup. Initial storage/model copy and
verification can be I/O-heavy; subsequent backups deduplicate unchanged bytes
against the newest verified prior snapshot:
```bash
bash scripts/backup_release_state.sh \
@@ -124,17 +143,37 @@ gate fails on reachable fixed HIGH/CRITICAL findings.
## Rollback proof
The rollback command reuses persistent paths and never downgrades Alembic:
Use the exact backup printed by the deployment. Rollback first stops the
candidate, restores and verifies the pre-deploy PostgreSQL dump in a temporary
proof database, then swaps database names while retaining the pre-restore
database as a recovery point. Only then does it start the exact image ID bound
into that backup; it never relies on a mutable global `previous` tag. It never runs an
Alembic downgrade or starts an old image against an unknown newer schema:
```bash
bash deploy/unraid/rollback-dockerman-container.sh
bash deploy/unraid/rollback-dockerman-container.sh \
--backup-dir /mnt/user/appdata/geointel/backups/<predeploy-release-id> \
--confirm-production-database-restore
curl -fsS http://127.0.0.1:1202/health/ready
bash deploy/unraid/deploy-release.sh
curl -fsS http://127.0.0.1:1202/health/ready
```
For a future backward-incompatible migration, restore the verified pre-release
backup instead of running an older image against a newer schema.
Database rollback restores persisted rows and schema. Files newly written by a
failed candidate remain in storage as unreferenced evidence; the protected,
recoverable quarantine flow in `DATA_OPERATIONS_RUNBOOK.md` handles those
files without deleting source data.
After the rollback has remained healthy and its retained evidence has been
reviewed, list the recovery database printed by the script. Remove it only by
an explicit, separately approved `dropdb` maintenance command; deployment and
rollback never auto-delete recovery databases or backup directories. Retain at
least the current successful predeploy backup and its predecessor. Before
removing an older backup, run `verify_release_backup.sh` on the backups that
remain and confirm no newer snapshot hard-links depend on operator policy for
retention (hard-linked bytes remain allocated while any retained backup names
them). Remove its `rollback-predeploy-*` image tag only in the same explicitly
reviewed retention operation.
## Tag and signed package