fix(release): make deployment backup and rollback immutable
This commit is contained in:
+46
-7
@@ -15,7 +15,7 @@ The repository version is stored in `VERSION`. The current release is
|
||||
- clean `main` worktree at the commit being released;
|
||||
- secure non-default PostGIS password in the Tower `.env`;
|
||||
- existing local AI model only when the AI image is enabled;
|
||||
- recent checksum-verified backup with SHA-256 storage inventory;
|
||||
- recent checksum-verified database dump and byte-complete storage/model snapshot;
|
||||
- Docker, `ssh-keygen`, Python 3.11, Node 20 and Bash available;
|
||||
- one Alembic head and no unsupported metric represented as successful.
|
||||
|
||||
@@ -39,6 +39,24 @@ docker compose config
|
||||
|
||||
## Immutable deployment
|
||||
|
||||
`deploy-release.sh` first builds/reuses the candidate while the current release
|
||||
stays available. Immediately before replacement it quiesces backend writes and
|
||||
creates and checksum-verifies a database dump plus byte-complete SHA-256
|
||||
storage/model snapshots under `/mnt/user/appdata/geointel/backups`. The first
|
||||
snapshot copies every byte (using CoW reflinks when supported); later snapshots
|
||||
hard-link only checksum-identical bytes from a verified older backup, never
|
||||
from live storage. A conservative full-copy/free-space preflight runs before
|
||||
backend quiescence. Only then may the
|
||||
candidate start or run Alembic. If backup fails, the unchanged release is
|
||||
restarted; deployment stops if existing PostGIS state cannot be backed up
|
||||
consistently.
|
||||
|
||||
The backup manifest separates provenance intentionally:
|
||||
`backup_tool_revision` is the new candidate source running the backup tool,
|
||||
while `running_image_revision` is the OCI label of the old release whose data
|
||||
is being captured. The retained Docker `image_id`, not either descriptive
|
||||
revision field, is authoritative for rollback.
|
||||
|
||||
On the Codex workstation:
|
||||
|
||||
```powershell
|
||||
@@ -58,8 +76,9 @@ bash scripts/live_migration_smoke.sh
|
||||
|
||||
## Backup and recovery proof
|
||||
|
||||
Create an immutable backup. The SHA-256 inventory can take several minutes on
|
||||
large storage:
|
||||
Create an immutable byte-complete backup. Initial storage/model copy and
|
||||
verification can be I/O-heavy; subsequent backups deduplicate unchanged bytes
|
||||
against the newest verified prior snapshot:
|
||||
|
||||
```bash
|
||||
bash scripts/backup_release_state.sh \
|
||||
@@ -124,17 +143,37 @@ gate fails on reachable fixed HIGH/CRITICAL findings.
|
||||
|
||||
## Rollback proof
|
||||
|
||||
The rollback command reuses persistent paths and never downgrades Alembic:
|
||||
Use the exact backup printed by the deployment. Rollback first stops the
|
||||
candidate, restores and verifies the pre-deploy PostgreSQL dump in a temporary
|
||||
proof database, then swaps database names while retaining the pre-restore
|
||||
database as a recovery point. Only then does it start the exact image ID bound
|
||||
into that backup; it never relies on a mutable global `previous` tag. It never runs an
|
||||
Alembic downgrade or starts an old image against an unknown newer schema:
|
||||
|
||||
```bash
|
||||
bash deploy/unraid/rollback-dockerman-container.sh
|
||||
bash deploy/unraid/rollback-dockerman-container.sh \
|
||||
--backup-dir /mnt/user/appdata/geointel/backups/<predeploy-release-id> \
|
||||
--confirm-production-database-restore
|
||||
curl -fsS http://127.0.0.1:1202/health/ready
|
||||
bash deploy/unraid/deploy-release.sh
|
||||
curl -fsS http://127.0.0.1:1202/health/ready
|
||||
```
|
||||
|
||||
For a future backward-incompatible migration, restore the verified pre-release
|
||||
backup instead of running an older image against a newer schema.
|
||||
Database rollback restores persisted rows and schema. Files newly written by a
|
||||
failed candidate remain in storage as unreferenced evidence; the protected,
|
||||
recoverable quarantine flow in `DATA_OPERATIONS_RUNBOOK.md` handles those
|
||||
files without deleting source data.
|
||||
|
||||
After the rollback has remained healthy and its retained evidence has been
|
||||
reviewed, list the recovery database printed by the script. Remove it only by
|
||||
an explicit, separately approved `dropdb` maintenance command; deployment and
|
||||
rollback never auto-delete recovery databases or backup directories. Retain at
|
||||
least the current successful predeploy backup and its predecessor. Before
|
||||
removing an older backup, run `verify_release_backup.sh` on the backups that
|
||||
remain and confirm no newer snapshot hard-links depend on operator policy for
|
||||
retention (hard-linked bytes remain allocated while any retained backup names
|
||||
them). Remove its `rollback-predeploy-*` image tag only in the same explicitly
|
||||
reviewed retention operation.
|
||||
|
||||
## Tag and signed package
|
||||
|
||||
|
||||
Reference in New Issue
Block a user