fix(release): make deployment backup and rollback immutable
This commit is contained in:
Jens
+36
-20
@@ -5,8 +5,10 @@ GeoIntel uses the same release gates in Gitea Actions and GitHub Actions:
|
||||
- `.gitea/workflows/release-gates.yml`
|
||||
- `.github/workflows/release-gates.yml`
|
||||
|
||||
Gitea is the operational source-control platform. The GitHub workflow is kept
|
||||
equivalent so a mirror or external review does not receive a weaker gate.
|
||||
Gitea is the operational source-control platform. Its release workflow builds
|
||||
the production AI variant and is the only workflow that can automatically
|
||||
deploy. The GitHub mirror builds and scans the same AI variant for external
|
||||
review but is not a production deployment authority.
|
||||
|
||||
## Runner requirements
|
||||
|
||||
@@ -16,11 +18,11 @@ The `ubuntu-latest` runner must provide:
|
||||
- Python 3.11 and Node 20 through the official setup actions;
|
||||
- Bash and Docker with Compose v2;
|
||||
- permission to build images and mount `/var/run/docker.sock`;
|
||||
- sufficient disk for the all-in-one GIS image and scanner databases.
|
||||
- sufficient disk for the all-in-one AI image and scanner databases.
|
||||
|
||||
The container job builds the GIS release variant only. PyTorch and
|
||||
Ultralytics remain in the optional `ai` extra and in the explicit AI image
|
||||
variant; CI does not silently make them base dependencies.
|
||||
Both container jobs build the explicit AI image, including the
|
||||
PyTorch/Ultralytics layers used on Tower. AI packages remain outside the
|
||||
standard backend lock and are pinned by Docker build arguments.
|
||||
|
||||
## Quality gate
|
||||
|
||||
@@ -33,11 +35,25 @@ cd frontend && npm ci
|
||||
```
|
||||
|
||||
It then validates the lock policy and runs the complete readiness script. The
|
||||
readiness script covers backend compile/tests, contract audits, Alembic
|
||||
single-head, frontend typecheck/build and release-script syntax. CI also
|
||||
readiness script covers Ruff, repository-layout validation, backend
|
||||
compile/tests, contract audits, Alembic single-head, frontend typecheck/build
|
||||
and release-script syntax. CI also
|
||||
renders offline migration SQL and resolved Compose configuration as retained
|
||||
evidence.
|
||||
|
||||
Pull requests run managed validation against the real `backend/` and
|
||||
`frontend/` projects plus the complete release gates. On a `main` push, the
|
||||
Unraid deploy job has explicit `needs` dependencies on quality, dependency and
|
||||
AI-container jobs. There is no separate deployment workflow: manual validation
|
||||
uses `workflow_dispatch` on this same release-gates workflow and cannot skip
|
||||
quality, dependency or AI-container jobs.
|
||||
|
||||
The deploy host requires a full controller commit SHA, builds the AI variant,
|
||||
records its Docker image/config digest, generates SBOM and Trivy evidence for
|
||||
that exact local image, and starts the immutable image ID. Deployment fails if
|
||||
the running container ID, revision label or AI label differs from the retained
|
||||
attestation.
|
||||
|
||||
## Reproducible Python lock
|
||||
|
||||
`backend/requirements-runtime.lock` and `backend/requirements-ci.lock` are
|
||||
@@ -73,12 +89,12 @@ The dependency job:
|
||||
- publishes both unfiltered and policy-filtered Python JSON reports plus the
|
||||
npm JSON report, including on failure.
|
||||
|
||||
The only current Python/container exceptions are the Starlette 2026 advisories recorded
|
||||
in `security/pip-audit-exceptions.json`. FastAPI 0.139.2 still constrains
|
||||
Starlette below 0.53 while patched releases begin at 1.x. GeoIntel applies
|
||||
request-target, form-content, route-class and Linux-runtime compensating
|
||||
controls. The exception file has a mandatory review date; readiness and CI
|
||||
fail automatically after it expires. New advisories are never auto-ignored.
|
||||
There are currently no Python or container vulnerability exceptions.
|
||||
`security/pip-audit-exceptions.json` remains as a strict, machine-readable
|
||||
registry: every future exception must identify one advisory, package, specific
|
||||
reason and expiry date. Readiness and CI fail on malformed or expired entries;
|
||||
new advisories are never auto-ignored. GeoIntel requires Starlette 1.3.1 or
|
||||
newer and therefore no longer suppresses the five 2026 Starlette advisories.
|
||||
The all-in-one image replaces the Go-based base-image `gosu` helper with a
|
||||
small `setpriv` exec wrapper and upgrades packaged setuptools/wheel metadata;
|
||||
the final runtime filesystem no longer exposes the vulnerable Go executable.
|
||||
@@ -89,8 +105,8 @@ verified to contain the audited shell wrapper. This is not a vulnerability
|
||||
exception: the raw evidence remains published and the runtime wrapper is
|
||||
exercised during live release validation.
|
||||
|
||||
The container job builds a non-AI all-in-one image and uses digest-pinned
|
||||
scanner images:
|
||||
The operational container job builds the production AI all-in-one image and
|
||||
uses digest-pinned scanner images:
|
||||
|
||||
- Syft 1.44.0 generates an SPDX JSON SBOM;
|
||||
- Trivy 0.70.0 generates a complete JSON vulnerability report;
|
||||
@@ -103,12 +119,12 @@ Run these controls on a Docker-enabled workstation:
|
||||
```bash
|
||||
docker build \
|
||||
-f deploy/unraid/Dockerfile.all-in-one \
|
||||
--build-arg GEOINTEL_INSTALL_AI=false \
|
||||
--build-arg GEOINTEL_INSTALL_AI=true \
|
||||
--build-arg GEOINTEL_BUILD_SHA=local \
|
||||
--build-arg GEOINTEL_BUILD_TIME=local \
|
||||
-t geointel-ci:local .
|
||||
bash scripts/generate_container_sbom.sh geointel-ci:local
|
||||
bash scripts/scan_container_image.sh geointel-ci:local
|
||||
-t geointel-ci:local-ai .
|
||||
bash scripts/generate_container_sbom.sh geointel-ci:local-ai
|
||||
bash scripts/scan_container_image.sh geointel-ci:local-ai
|
||||
```
|
||||
|
||||
Outputs are written below ignored `artifacts/`; scanner cache is written below
|
||||
|
||||
@@ -89,9 +89,9 @@ The normal project lifecycle archive path does not need destructive
|
||||
confirmation because it changes only `status=archived` and preserves all
|
||||
data.
|
||||
|
||||
## Destructive apply gate
|
||||
## Recoverable quarantine gate
|
||||
|
||||
First create a fresh backup with a SHA-256 storage inventory on the host:
|
||||
First create a fresh backup with a byte-complete SHA-256 storage snapshot on the host:
|
||||
|
||||
```bash
|
||||
bash scripts/backup_release_state.sh \
|
||||
@@ -112,7 +112,8 @@ bash scripts/verify_release_backup.sh \
|
||||
```
|
||||
|
||||
The Unraid runtime mounts `GEOINTEL_BACKUPS_PATH` read-only at `/app/backups`.
|
||||
Only after reviewing the dry run may an operator execute:
|
||||
Only after reviewing the dry run may an operator move the exact candidates to
|
||||
a protected quarantine:
|
||||
|
||||
```bash
|
||||
docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
|
||||
@@ -120,14 +121,34 @@ docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
|
||||
--max-delete <reviewed-candidate-count> \
|
||||
--backup-dir /app/backups/<release-id> \
|
||||
--backup-max-age-hours 24 \
|
||||
--confirm DELETE_STORAGE_ARTIFACTS \
|
||||
--confirm QUARANTINE_STORAGE_ARTIFACTS \
|
||||
--apply
|
||||
```
|
||||
|
||||
The command re-runs the audit immediately before deletion. It refuses the
|
||||
The command re-runs the audit immediately before quarantine. It refuses the
|
||||
operation when the exact token is absent, the candidate count exceeds the
|
||||
operator limit, the backup is stale/incomplete, checksums differ, the storage
|
||||
inventory is not SHA-256, or the path is outside the cleanup allowlist.
|
||||
operator limit, the backup is stale/incomplete, snapshot checksums differ, or
|
||||
the path is outside the cleanup allowlist.
|
||||
Every moved byte is hashed and retained below
|
||||
`operator-evidence/cleanup-quarantine/<operation-id>/files/`; an atomic
|
||||
manifest records its original path, quarantine path, size, checksum and
|
||||
backup identity. Cleanup uses a backup-to-quarantine hard-link state machine:
|
||||
the manifest records `planned`, `linked` and `quarantined` transitions so each
|
||||
crash window can be reconciled without losing the retained inode.
|
||||
|
||||
Restore a reviewed quarantine without overwriting any path:
|
||||
|
||||
```bash
|
||||
docker exec geointel python /app/scripts/restore_storage_quarantine.py \
|
||||
--storage-root /app/storage \
|
||||
--manifest /app/storage/operator-evidence/cleanup-quarantine/<operation-id>/manifest.json \
|
||||
--confirm RESTORE_QUARANTINED_ARTIFACTS
|
||||
```
|
||||
|
||||
Restore validates every retained checksum and never replaces an existing
|
||||
destination. It uses an exclusive hard link and reconciles both-file and
|
||||
one-file interruption states before updating the manifest, so an interrupted
|
||||
restore can be resumed and verified.
|
||||
|
||||
The older demo-export cleanup has the same gate and uses confirmation token
|
||||
`DELETE_DEMO_EXPORTS`.
|
||||
|
||||
+46
-7
@@ -15,7 +15,7 @@ The repository version is stored in `VERSION`. The current release is
|
||||
- clean `main` worktree at the commit being released;
|
||||
- secure non-default PostGIS password in the Tower `.env`;
|
||||
- existing local AI model only when the AI image is enabled;
|
||||
- recent checksum-verified backup with SHA-256 storage inventory;
|
||||
- recent checksum-verified database dump and byte-complete storage/model snapshot;
|
||||
- Docker, `ssh-keygen`, Python 3.11, Node 20 and Bash available;
|
||||
- one Alembic head and no unsupported metric represented as successful.
|
||||
|
||||
@@ -39,6 +39,24 @@ docker compose config
|
||||
|
||||
## Immutable deployment
|
||||
|
||||
`deploy-release.sh` first builds/reuses the candidate while the current release
|
||||
stays available. Immediately before replacement it quiesces backend writes and
|
||||
creates and checksum-verifies a database dump plus byte-complete SHA-256
|
||||
storage/model snapshots under `/mnt/user/appdata/geointel/backups`. The first
|
||||
snapshot copies every byte (using CoW reflinks when supported); later snapshots
|
||||
hard-link only checksum-identical bytes from a verified older backup, never
|
||||
from live storage. A conservative full-copy/free-space preflight runs before
|
||||
backend quiescence. Only then may the
|
||||
candidate start or run Alembic. If backup fails, the unchanged release is
|
||||
restarted; deployment stops if existing PostGIS state cannot be backed up
|
||||
consistently.
|
||||
|
||||
The backup manifest separates provenance intentionally:
|
||||
`backup_tool_revision` is the new candidate source running the backup tool,
|
||||
while `running_image_revision` is the OCI label of the old release whose data
|
||||
is being captured. The retained Docker `image_id`, not either descriptive
|
||||
revision field, is authoritative for rollback.
|
||||
|
||||
On the Codex workstation:
|
||||
|
||||
```powershell
|
||||
@@ -58,8 +76,9 @@ bash scripts/live_migration_smoke.sh
|
||||
|
||||
## Backup and recovery proof
|
||||
|
||||
Create an immutable backup. The SHA-256 inventory can take several minutes on
|
||||
large storage:
|
||||
Create an immutable byte-complete backup. Initial storage/model copy and
|
||||
verification can be I/O-heavy; subsequent backups deduplicate unchanged bytes
|
||||
against the newest verified prior snapshot:
|
||||
|
||||
```bash
|
||||
bash scripts/backup_release_state.sh \
|
||||
@@ -124,17 +143,37 @@ gate fails on reachable fixed HIGH/CRITICAL findings.
|
||||
|
||||
## Rollback proof
|
||||
|
||||
The rollback command reuses persistent paths and never downgrades Alembic:
|
||||
Use the exact backup printed by the deployment. Rollback first stops the
|
||||
candidate, restores and verifies the pre-deploy PostgreSQL dump in a temporary
|
||||
proof database, then swaps database names while retaining the pre-restore
|
||||
database as a recovery point. Only then does it start the exact image ID bound
|
||||
into that backup; it never relies on a mutable global `previous` tag. It never runs an
|
||||
Alembic downgrade or starts an old image against an unknown newer schema:
|
||||
|
||||
```bash
|
||||
bash deploy/unraid/rollback-dockerman-container.sh
|
||||
bash deploy/unraid/rollback-dockerman-container.sh \
|
||||
--backup-dir /mnt/user/appdata/geointel/backups/<predeploy-release-id> \
|
||||
--confirm-production-database-restore
|
||||
curl -fsS http://127.0.0.1:1202/health/ready
|
||||
bash deploy/unraid/deploy-release.sh
|
||||
curl -fsS http://127.0.0.1:1202/health/ready
|
||||
```
|
||||
|
||||
For a future backward-incompatible migration, restore the verified pre-release
|
||||
backup instead of running an older image against a newer schema.
|
||||
Database rollback restores persisted rows and schema. Files newly written by a
|
||||
failed candidate remain in storage as unreferenced evidence; the protected,
|
||||
recoverable quarantine flow in `DATA_OPERATIONS_RUNBOOK.md` handles those
|
||||
files without deleting source data.
|
||||
|
||||
After the rollback has remained healthy and its retained evidence has been
|
||||
reviewed, list the recovery database printed by the script. Remove it only by
|
||||
an explicit, separately approved `dropdb` maintenance command; deployment and
|
||||
rollback never auto-delete recovery databases or backup directories. Retain at
|
||||
least the current successful predeploy backup and its predecessor. Before
|
||||
removing an older backup, run `verify_release_backup.sh` on the backups that
|
||||
remain and confirm no newer snapshot hard-links depend on operator policy for
|
||||
retention (hard-linked bytes remain allocated while any retained backup names
|
||||
them). Remove its `rollback-predeploy-*` image tag only in the same explicitly
|
||||
reviewed retention operation.
|
||||
|
||||
## Tag and signed package
|
||||
|
||||
|
||||
@@ -28,10 +28,11 @@ The restore smoke may only create databases whose name starts with
|
||||
use `pg_restore --clean` and drops the temporary database unless an operator
|
||||
explicitly asks to retain it.
|
||||
|
||||
Storage and model files are inventoried rather than copied into the database
|
||||
dump. Release backups must therefore be paired with the persistent storage
|
||||
volume backup policy. Use `--inventory-mode sha256` for final release
|
||||
evidence.
|
||||
Storage and model files are retained as byte-complete, SHA-256 verified
|
||||
snapshots alongside the database dump. Snapshot creation rejects symlinks and
|
||||
special files. The first snapshot uses CoW reflinks when supported and falls
|
||||
back to full copies; later snapshots hard-link checksum-identical files only
|
||||
from a verified older backup, never from live storage.
|
||||
|
||||
An old persistent volume can also retain glibc collation metadata for the
|
||||
empty `postgres` and `template1` system databases. If `createdb` fails for that
|
||||
|
||||
Reference in New Issue
Block a user