fix(release): make deployment backup and rollback immutable

This commit is contained in:
Jens
2026-08-30 06:00:43 +02:00
parent a0884d64c9
commit c272220277
47 changed files with 3035 additions and 430 deletions
+2
View File
@@ -160,8 +160,10 @@ COPY scripts/migrate_runtime_model_provenance.py /app/scripts/migrate_runtime_mo
COPY scripts/archive_technical_projects.py /app/scripts/archive_technical_projects.py
COPY scripts/runtime_state_report.py /app/scripts/runtime_state_report.py
COPY scripts/release_backup_guard.py /app/scripts/release_backup_guard.py
COPY scripts/release_backup_snapshot.py /app/scripts/release_backup_snapshot.py
COPY scripts/audit_data_operations.py /app/scripts/audit_data_operations.py
COPY scripts/cleanup_storage_artifacts.py /app/scripts/cleanup_storage_artifacts.py
COPY scripts/restore_storage_quarantine.py /app/scripts/restore_storage_quarantine.py
COPY deploy/unraid/nginx-all-in-one.conf /etc/nginx/conf.d/default.conf
COPY deploy/unraid/all-in-one-start.sh /usr/local/bin/geointel-all-in-one-start
COPY --from=frontend-build /frontend/dist/ /usr/share/nginx/html/
+34 -32
View File
@@ -11,9 +11,8 @@ met de naam `geointel`, bereikbaar op `http://192.168.10.150:1202`.
Kopieer de **volledige** map `C:\Projects\geointel` naar
`/mnt/user/appdata/geointel` op de server. Verder niets uitzoeken.
`.dockerignore` regelt de rest: `node_modules/`, `.git/`, de dubbele
`geointel/`-map, `docs/`, `artifacts/` en testoutput gaan niet mee de
build-context in, ook al staan ze in de map.
`.dockerignore` regelt de rest: `node_modules/`, `.git/`, `docs/`, `artifacts/`
en testoutput gaan niet mee de build-context in, ook al staan ze in de map.
Eén waarschuwing bij het overschrijven: laat `storage/`, `postgres-data/`,
`backups/` en `models/` op de server **staan**. Dat is je bestaande data, en
@@ -71,7 +70,8 @@ die hostname in `GEOINTEL_CORS_ORIGINS` staan — anders blokkeert de browser de
API-calls vanaf het publieke adres. De backend doet geen host-validatie, dus
verder is er niets nodig aan applicatiekant.
Voor GPU-inferentie (optioneel, kan ook later):
De productie-image bevat altijd de gepinde AI-runtime. Inferentie zelf kan
uitblijven totdat een lokaal, gevalideerd model beschikbaar is:
```env
GEOINTEL_INSTALL_AI=true
@@ -81,8 +81,8 @@ YOLO_REQUIRE_CUDA=true
YOLO_MODEL_PATH=/app/models/<jouw-model>.pt
```
Laat `GEOINTEL_INSTALL_AI=false` staan als je eerst gewoon wilt dat de app
draait — dat scheelt een paar GB aan PyTorch-lagen in de build.
Laat `YOLO_ENABLED=false` zolang er geen geschikt modelbestand is. Het
release-deployscript weigert bewust een GIS-only productie-image.
---
@@ -134,12 +134,21 @@ bash deploy/unraid/deploy-release.sh
Het script:
1. ruimt een eventueel achtergebleven Compose-stack op (ook de oude
1. bindt het exacte huidige image-ID aan een unieke backup-specifieke
`rollback-predeploy-*`-tag;
2. bouwt of hergebruikt de AI-candidate terwijl de huidige release beschikbaar
blijft, legt het exacte lokale image-ID vast en maakt daarop SBOM- en
Trivy-evidence;
3. controleert vóór het pauzeren de vrije ruimte en maakt vervolgens een
byte-complete, SHA-256-geverifieerde database-, storage- en modelsnapshot in
`/mnt/user/appdata/geointel/backups`; ongewijzigde bestanden mogen alleen
vanuit een oudere geverifieerde backup worden gehardlinkt;
4. ruimt een eventueel achtergebleven Compose-stack op (ook de oude
3-container dev-stack die óók poort 1202 pakt);
2. bewaart de huidige image als `geointel-all-in-one:previous`;
3. bouwt `deploy/unraid/Dockerfile.all-in-one`;
4. start één container `geointel` met `-p 1202:80` en `--gpus all`;
5. rolt automatisch terug naar `:previous` als de healthcheck of smoke faalt.
5. start één container `geointel` met `-p 1202:80` en `--gpus all`;
6. bewijst een rollbackdump eerst in een geïsoleerde tijdelijke database,
bewaart de oude productiedatabase als herstelpad en start pas daarna
automatisch het image-ID dat cryptografisch in die pre-deploybackup staat.
De eerste build duurt lang (PostGIS + GDAL + npm build). Volgende deploys
hergebruiken de Docker-layercache.
@@ -165,24 +174,13 @@ docker logs --tail 200 geointel
---
## 7. Belangrijk: dubbele projectmap lokaal
## 7. Geretireerde dubbele projectmap
In `C:\Projects\geointel` staat een tweede, volledige kopie van het project
onder `C:\Projects\geointel\geointel\`. Die bevat dezelfde bestanden en
dezelfde datum, maar staat buiten git. Zolang die er staat:
- wordt de Docker build-context onnodig verdubbeld;
- weet je bij het bewerken van bijvoorbeeld `docker-compose.unraid.yml` niet
welke versie je te pakken hebt.
`.dockerignore` sluit hem nu uit, maar ruim hem op zodra je zeker weet dat er
niets unieks in staat. Vergelijk eerst:
```powershell
robocopy C:\Projects\geointel\geointel C:\Projects\geointel /L /E /NJH /NJS /NDL /XF *.pyc
```
Regels die als `New File` verschijnen bestaan alleen in de kopie.
De vroegere geneste mirror `C:\Projects\geointel\geointel` is geretireerd. De
immutable Git-herkomst, niet-getrackte recoverybestanden en verificatiegrens
staan in `docs/accuracy-program/13-nested-mirror-retirement.md`. Behandel die
evidence als herstelreferentie; deze handleiding vraagt geen extra kopieer- of
opruimactie.
---
@@ -190,11 +188,15 @@ Regels die als `New File` verschijnen bestaan alleen in de kopie.
```bash
cd /mnt/user/appdata/geointel
bash deploy/unraid/rollback-dockerman-container.sh
bash deploy/unraid/rollback-dockerman-container.sh \
--backup-dir /mnt/user/appdata/geointel/backups/<predeploy-release-id> \
--confirm-production-database-restore
```
Rollback hergebruikt dezelfde PostGIS- en storage-paden en draait nooit een
Alembic-downgrade.
Rollback hergebruikt dezelfde storage-paden, bewijst de geverifieerde dump
eerst geïsoleerd, wisselt daarna databases via no-clobber namen en draait nooit
een Alembic-downgrade. De oude productiedatabase blijft staan totdat een
operator haar na controle expliciet opruimt.
---
@@ -202,7 +204,7 @@ Alembic-downgrade.
| Bestand | Aanpassing |
|---|---|
| `.dockerignore` | Sluit root-`node_modules`, de dubbele `geointel/`-map, `.git`, `docs/`, `artifacts/` en testoutput uit de build-context |
| `.dockerignore` | Sluit root-`node_modules`, `.git`, `docs/`, `artifacts/` en testoutput uit de build-context |
| `deploy/unraid/deploy-release.sh` | `git rev-parse HEAD` crashte op een kopie zonder `.git`. Valt nu terug op `GEOINTEL_BUILD_SHA`, een `RELEASE_SHA`-bestand of een content-hash van de broncode |
| `deploy/unraid/deploy-release.sh` | Smoke-scripts worden op bestaan getest in plaats van op de execute-bit, die bij een Windows-kopie verloren gaat |
| `deploy/unraid/run-dockerman-container.sh` | Ruimt expliciet zowel `docker-compose.yml` (3 containers) als `docker-compose.unraid.yml` op, zodat poort 1202 gegarandeerd vrij is |
+30 -22
View File
@@ -106,27 +106,25 @@ customer or operational data. Deploy a separate demo container and storage
root for public or recruiter-facing access.
The repository deploy scripts run the same flow automatically. They validate
the Compose reference, preserve the current image as
`geointel-all-in-one:previous`, build an immutable `<commit-sha>-ai` or
`<commit-sha>-gis` tag plus `latest`, install the DockerMan metadata and start
the immutable image. An existing matching tag is reused, never rebuilt. A
failed start, live migration smoke or browser/API smoke automatically attempts
the previous image without changing the configured PostGIS or storage paths.
the Compose reference, preserve the current image under a unique
backup-specific `rollback-predeploy-*` tag, build the immutable production
`<commit-sha>-ai` tag plus `latest`, attest its exact local image ID, generate
an SBOM and enforce the Trivy policy before starting that same ID. An existing
matching tag is reused, never rebuilt. A failed start, live migration smoke or
browser/API smoke automatically attempts the previous image without changing
the configured PostGIS or storage paths.
`scripts/deploy_tower.sh` and `scripts/deploy_tower.ps1` source the remote
`.env` before building the image. That means `GEOINTEL_INSTALL_AI=true` in
`/mnt/user/appdata/geointel/.env` is enough for the automatic deploy to build
the AI-enabled image. Set `GEOINTEL_INSTALL_AI` in the local shell or pass
`-InstallAi true/false` to the PowerShell wrapper only when you intentionally
want to override the remote `.env` for that deploy.
`.env` before building the image. Production deployment requires
`GEOINTEL_INSTALL_AI=true`; an explicit false value fails closed before the
image or running container is replaced.
Database credentials are runtime configuration, not image metadata. The
all-in-one image does not bake `GEOINTEL_POSTGRES_PASSWORD` into the Dockerfile;
set it through `.env`, the Unraid template or `docker run -e`.
AI dependencies are opt-in. Leave `GEOINTEL_INSTALL_AI=false` for the default
GIS-only image. Set `GEOINTEL_INSTALL_AI=true`, mount models through
`GEOINTEL_MODELS_PATH` and configure `YOLO_ENABLED=true` plus
Production images always include the pinned AI dependencies. Mount models
through `GEOINTEL_MODELS_PATH` and configure `YOLO_ENABLED=true` plus
`YOLO_MODELS_DIR=/app/models` and `YOLO_MODEL_PATH=/app/models/<model>.pt` only
when you have a local model file.
The AI-enabled image installs PyTorch/Ultralytics plus the native OpenCV runtime
@@ -264,7 +262,8 @@ git reset --hard origin/main
bash deploy/unraid/deploy-release.sh
```
The equivalent low-level build remains available for debugging:
The equivalent low-level GIS-only build remains available only for local
debugging; it is not a production deployment path:
```bash
docker build --build-arg GEOINTEL_INSTALL_AI=${GEOINTEL_INSTALL_AI:-false} -f deploy/unraid/Dockerfile.all-in-one -t geointel-all-in-one:latest .
@@ -305,20 +304,27 @@ bash scripts/verify_release_upgrade_smoke.sh \
Return to the image that was active immediately before the latest deployment:
```bash
bash deploy/unraid/rollback-dockerman-container.sh
bash deploy/unraid/rollback-dockerman-container.sh \
--backup-dir /mnt/user/appdata/geointel/backups/<predeploy-release-id> \
--confirm-production-database-restore
```
For an older retained commit, select its immutable tag explicitly:
```bash
GEOINTEL_ROLLBACK_IMAGE=geointel-all-in-one:<commit-sha>-ai \
bash deploy/unraid/rollback-dockerman-container.sh
bash deploy/unraid/rollback-dockerman-container.sh \
--backup-dir /mnt/user/appdata/geointel/backups/<matching-predeploy-release-id> \
--confirm-production-database-restore
```
Rollback reuses the configured PostGIS and storage mounts and never runs an
Alembic downgrade. If a future release has a backward-incompatible migration,
restore its verified pre-release backup instead of forcing an older app
against a newer schema.
Rollback restores and verifies the selected dump in an isolated proof database
before any production replacement. It then swaps database names, retains the
pre-restore production database for operator recovery, reuses the configured
storage mount and never runs an Alembic downgrade or an older app against an
unknown newer schema. Remove the retained recovery database and old backup
directories only in a separately reviewed operator retention step; deployment
never deletes them automatically.
The configured upload limit is shared by FastAPI and the generated nginx
runtime configuration. Values outside `1..2048` MiB are rejected before the
@@ -339,7 +345,9 @@ docker exec geointel python /app/scripts/cleanup_storage_artifacts.py
```
The full backup, confirmation, candidate-limit and apply sequence is in
`docs/DATA_OPERATIONS_RUNBOOK.md`. GeoIntel installs no automatic cleanup
`docs/DATA_OPERATIONS_RUNBOOK.md`. Apply moves bytes to protected,
checksum-bound quarantine rather than deleting them; a separate confirmed
restore command reverses the move. GeoIntel installs no automatic cleanup
schedule.
## Safe cleanup
+372 -22
View File
@@ -26,7 +26,11 @@ if [ -n "${DEPLOY_GEOINTEL_INSTALL_AI:-}" ]; then
GEOINTEL_INSTALL_AI="$DEPLOY_GEOINTEL_INSTALL_AI"
fi
GEOINTEL_INSTALL_AI="${GEOINTEL_INSTALL_AI:-false}"
GEOINTEL_INSTALL_AI="${GEOINTEL_INSTALL_AI:-true}"
if [ "$GEOINTEL_INSTALL_AI" != "true" ]; then
echo "Production release deployment requires the gated AI image (GEOINTEL_INSTALL_AI=true)." >&2
exit 2
fi
GEOINTEL_APP_VERSION="$(tr -d '[:space:]' < VERSION)"
if ! [[ "$GEOINTEL_APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]]; then
echo "Invalid semantic version in VERSION: ${GEOINTEL_APP_VERSION}" >&2
@@ -57,7 +61,52 @@ source_tree_hash() {
}
resolve_build_sha() {
local head="" content=""
local head="" content="" controller_sha="" controller_source=""
if [ -n "${GITEA_COMMIT_SHA:-}" ]; then
controller_sha="$GITEA_COMMIT_SHA"
controller_source="GITEA_COMMIT_SHA"
fi
if [ -n "${GITHUB_SHA:-}" ]; then
if ! [[ "$GITHUB_SHA" =~ ^[0-9A-Fa-f]{40}$ ]]; then
echo "GITHUB_SHA must contain one full 40-character Git commit SHA." >&2
return 2
fi
if [ -n "$controller_sha" ] && [ "${controller_sha,,}" != "${GITHUB_SHA,,}" ]; then
echo "Controller commit variables disagree." >&2
return 2
fi
controller_sha="$GITHUB_SHA"
controller_source="${controller_source:-GITHUB_SHA}"
fi
if [ -n "$controller_sha" ]; then
if ! [[ "$controller_sha" =~ ^[0-9A-Fa-f]{40}$ ]]; then
echo "${controller_source} must contain one full 40-character Git commit SHA." >&2
return 2
fi
controller_sha="${controller_sha,,}"
if [ -n "${GEOINTEL_BUILD_SHA:-}" ] && [ "${GEOINTEL_BUILD_SHA,,}" != "$controller_sha" ]; then
echo "Explicit build revision differs from the controller revision." >&2
return 2
fi
if command -v git >/dev/null 2>&1 && git rev-parse --git-dir >/dev/null 2>&1; then
head="$(git rev-parse HEAD 2>/dev/null || true)"
if [ "${head,,}" != "$controller_sha" ]; then
echo "Prepared Git checkout does not match the controller revision." >&2
return 2
fi
if [ -n "$(git status --porcelain 2>/dev/null)" ]; then
echo "Prepared Git checkout contains changes outside the controller revision." >&2
return 2
fi
fi
printf '%s' "$controller_sha"
return 0
fi
if [ -n "${GITEA_REPOSITORY:-}" ] || [ -n "${GITHUB_REPOSITORY:-}" ]; then
echo "Automated deployment context is missing GITEA_COMMIT_SHA/GITHUB_SHA." >&2
return 2
fi
# 1. Explicit override wins.
if [ -n "${GEOINTEL_BUILD_SHA:-}" ]; then
@@ -107,7 +156,13 @@ if [ -z "$GEOINTEL_BUILD_SHA" ]; then
echo "Could not determine a build revision for this deployment." >&2
exit 2
fi
export GEOINTEL_BUILD_SHA
echo "Build revision: ${GEOINTEL_BUILD_SHA}"
GEOINTEL_RELEASE_TOKEN="$(printf '%s' "$GEOINTEL_BUILD_SHA" | tr -c 'A-Za-z0-9._-' '_' | cut -c1-48)"
if [ -z "$GEOINTEL_RELEASE_TOKEN" ]; then
echo "Could not derive a safe release evidence identifier." >&2
exit 2
fi
GEOINTEL_BUILD_TIME="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
GEOINTEL_IMAGE_REPOSITORY="${GEOINTEL_IMAGE_REPOSITORY:-geointel-all-in-one}"
if [ "$GEOINTEL_INSTALL_AI" = "true" ]; then
@@ -116,8 +171,23 @@ else
GEOINTEL_RELEASE_VARIANT="gis"
fi
GEOINTEL_RELEASE_IMAGE="${GEOINTEL_IMAGE_REPOSITORY}:${GEOINTEL_BUILD_SHA}-${GEOINTEL_RELEASE_VARIANT}"
GEOINTEL_PREVIOUS_IMAGE="${GEOINTEL_IMAGE_REPOSITORY}:previous"
FRONTEND_URL="${FRONTEND_URL:-http://127.0.0.1:${GEOINTEL_FRONTEND_PORT:-1202}}"
GEOINTEL_BACKUPS_PATH="${GEOINTEL_BACKUPS_PATH:-/mnt/user/appdata/geointel/backups}"
GEOINTEL_STORAGE_PATH="${GEOINTEL_STORAGE_PATH:-/mnt/user/appdata/geointel/storage}"
GEOINTEL_MODELS_PATH="${GEOINTEL_MODELS_PATH:-/mnt/user/appdata/geointel/models}"
GEOINTEL_POSTGIS_DATA_PATH="${GEOINTEL_POSTGIS_DATA_PATH:-/mnt/user/appdata/geointel/postgres-data}"
GEOINTEL_DEPLOY_EVIDENCE_DIR="${GEOINTEL_DEPLOY_EVIDENCE_DIR:-artifacts/release-evidence/deploy/${GEOINTEL_RELEASE_TOKEN}-ai}"
GEOINTEL_PREDEPLOY_BACKUP_DIR=""
GEOINTEL_RELEASE_IMAGE_ID=""
GEOINTEL_BACKUP_LINK_DEST=""
GEOINTEL_PREDEPLOY_ROLLBACK_TAG=""
case "$GEOINTEL_DEPLOY_EVIDENCE_DIR" in
/*|*..*)
echo "Deployment evidence directory must be repository-relative and must not contain '..'." >&2
exit 2
;;
esac
wait_for_geointel_health() {
local status=""
@@ -141,35 +211,312 @@ wait_for_geointel_health() {
start_image() {
local image="$1"
local running_image_id=""
local running_revision=""
local running_ai=""
GEOINTEL_IMAGE="$image" bash deploy/unraid/run-dockerman-container.sh
wait_for_geointel_health
running_image_id="$(docker inspect --format '{{.Image}}' geointel)"
if [ "$running_image_id" != "$image" ]; then
echo "Running container image ${running_image_id} differs from attested image ${image}." >&2
return 1
fi
running_revision="$(docker inspect --format '{{index .Config.Labels "org.opencontainers.image.revision"}}' geointel)"
running_ai="$(docker inspect --format '{{index .Config.Labels "io.geointel.ai.enabled"}}' geointel)"
if [ "$running_revision" != "$GEOINTEL_BUILD_SHA" ] || [ "$running_ai" != "true" ]; then
echo "Running container labels do not match the attested AI revision." >&2
return 1
fi
echo "Running container matches attested image: ${running_image_id}"
}
scan_release_image() {
local scanned_image_id=""
local current_image_id=""
local inspect_output="${GEOINTEL_DEPLOY_EVIDENCE_DIR}/image-inspect.json"
local sbom_output="${GEOINTEL_DEPLOY_EVIDENCE_DIR}/geointel-sbom.spdx.json"
local vulnerability_output="${GEOINTEL_DEPLOY_EVIDENCE_DIR}/geointel-container-vulnerabilities.json"
local attestation_output="${GEOINTEL_DEPLOY_EVIDENCE_DIR}/deployment-attestation.json"
scanned_image_id="$(docker image inspect --format '{{.Id}}' "$GEOINTEL_RELEASE_IMAGE")"
test -n "$scanned_image_id"
mkdir -p "$ROOT/$GEOINTEL_DEPLOY_EVIDENCE_DIR"
docker image inspect "$GEOINTEL_RELEASE_IMAGE" > "$ROOT/$inspect_output"
bash scripts/generate_container_sbom.sh "$GEOINTEL_RELEASE_IMAGE" "$sbom_output"
bash scripts/scan_container_image.sh "$GEOINTEL_RELEASE_IMAGE" "$vulnerability_output"
current_image_id="$(docker image inspect --format '{{.Id}}' "$GEOINTEL_RELEASE_IMAGE")"
if [ "$current_image_id" != "$scanned_image_id" ]; then
echo "Release image tag changed while SBOM/scan evidence was being generated." >&2
return 1
fi
test -s "$ROOT/$inspect_output"
test -s "$ROOT/$sbom_output"
test -s "$ROOT/$vulnerability_output"
GEOINTEL_RELEASE_IMAGE_ID="$scanned_image_id"
python3 - \
"$ROOT/$attestation_output" \
"$GEOINTEL_RELEASE_IMAGE" \
"$GEOINTEL_RELEASE_IMAGE_ID" \
"$GEOINTEL_BUILD_SHA" \
"$inspect_output" \
"$sbom_output" \
"$vulnerability_output" <<'PY'
import datetime
import json
import pathlib
import sys
output, image_tag, image_id, revision, inspect_path, sbom_path, vulnerability_path = sys.argv[1:]
payload = {
"schema_version": 1,
"attested_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"image_tag": image_tag,
"image_id": image_id,
"image_config_digest": image_id,
"revision": revision,
"variant": "ai",
"evidence": {
"image_inspect": inspect_path,
"sbom": sbom_path,
"vulnerabilities": vulnerability_path,
},
}
path = pathlib.Path(output)
temporary = path.with_suffix(".json.partial")
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
temporary.replace(path)
PY
test -s "$ROOT/$attestation_output"
echo "Exact deployment image scanned: ${GEOINTEL_RELEASE_IMAGE_ID}"
}
preflight_backup_capacity() {
local database_name=""
local database_user=""
local database_size_bytes=""
database_name="$(docker exec geointel sh -c 'printf %s "${POSTGRES_DB:-${GEOINTEL_POSTGRES_DB:-geointel}}"')"
database_user="$(docker exec geointel sh -c 'printf %s "${POSTGRES_USER:-${GEOINTEL_POSTGRES_USER:-geointel}}"')"
database_size_bytes="$(docker exec geointel psql -X -v ON_ERROR_STOP=1 \
-U "$database_user" -d "$database_name" -Atqc \
'SELECT pg_database_size(current_database());')"
mkdir -p "$GEOINTEL_BACKUPS_PATH"
python3 - \
"$GEOINTEL_BACKUPS_PATH" \
"$GEOINTEL_STORAGE_PATH" \
"$GEOINTEL_MODELS_PATH" \
"$database_size_bytes" <<'PY'
import os
import pathlib
import shutil
import stat
import sys
backup_root = pathlib.Path(sys.argv[1]).expanduser().resolve()
sources = [pathlib.Path(value).expanduser().resolve() for value in sys.argv[2:4]]
database_bytes = int(sys.argv[4])
def retained_bytes(root: pathlib.Path) -> int:
if not root.is_dir():
raise SystemExit(f"Mandatory snapshot source is not a directory: {root}")
total = 0
for current, directories, files in os.walk(root, topdown=True, followlinks=False):
current_path = pathlib.Path(current)
for name in [*directories, *files]:
path = current_path / name
details = path.lstat()
if stat.S_ISLNK(details.st_mode):
raise SystemExit(f"Mandatory snapshot refuses symlinked content: {path}")
if name in directories and not stat.S_ISDIR(details.st_mode):
raise SystemExit(f"Snapshot directory changed during capacity preflight: {path}")
if name in files:
if not stat.S_ISREG(details.st_mode):
raise SystemExit(f"Mandatory snapshot refuses non-regular content: {path}")
total += details.st_size
return total
source_bytes = sum(retained_bytes(source) for source in sources)
# Reflink clones are used when the backing filesystem supports them. Budget for
# a complete copy plus two uncompressed database sizes (dump and isolated
# restore/cutover recovery) so fallback still fails before the live backend is
# quiesced rather than midway through the snapshot.
required = source_bytes + (2 * database_bytes)
headroom = max(5 * 1024**3, required // 10)
free = shutil.disk_usage(backup_root).free
if free < required + headroom:
raise SystemExit(
"Insufficient free space for a fail-safe predeploy snapshot: "
f"required={required + headroom} free={free} source={source_bytes} database={database_bytes}"
)
print(
"Predeploy snapshot capacity: "
f"source_bytes={source_bytes} database_bytes={database_bytes} free_bytes={free}"
)
PY
}
select_verified_link_dest() {
local candidate=""
GEOINTEL_BACKUP_LINK_DEST=""
while IFS= read -r candidate; do
if (
cd "$candidate" \
&& sha256sum -c CHECKSUMS.sha256 >/dev/null \
&& python3 "$ROOT/scripts/release_backup_snapshot.py" verify-backup --backup-dir "$candidate"
); then
GEOINTEL_BACKUP_LINK_DEST="$candidate"
echo "Using verified prior byte snapshot as link-dest: ${candidate}"
return 0
fi
echo "Skipping unusable prior backup link-dest: ${candidate}" >&2
done < <(
python3 - "$GEOINTEL_BACKUPS_PATH" <<'PY'
import pathlib
import sys
root = pathlib.Path(sys.argv[1]).expanduser().resolve()
candidates = sorted(
(
path
for path in root.iterdir()
if path.is_dir() and not path.name.startswith(".") and (path / "manifest.json").is_file()
),
key=lambda path: path.stat().st_mtime_ns,
reverse=True,
)
for candidate in candidates:
print(candidate)
PY
)
echo "No verified prior byte snapshot found; this deployment will create a full first snapshot."
}
create_predeploy_backup() {
local container_exists="false"
local container_running="false"
local release_id=""
local backup_link_args=()
local current_image_id=""
local existing_rollback_id=""
if docker ps -a --format '{{.Names}}' | grep -Fxq geointel; then
container_exists="true"
fi
if [ "$(docker inspect -f '{{.State.Running}}' geointel 2>/dev/null || true)" = "true" ]; then
container_running="true"
fi
if [ "$container_exists" = "false" ]; then
if [ -f "$GEOINTEL_POSTGIS_DATA_PATH/PG_VERSION" ]; then
echo "PostGIS data exists without a running GeoIntel container; refusing an unbacked migration." >&2
return 1
fi
echo "No existing GeoIntel state found; pre-deploy backup is not required for this fresh install."
return 0
fi
if [ "$container_running" != "true" ]; then
echo "Existing GeoIntel container is not running; refusing deployment because a consistent backup cannot be created." >&2
return 1
fi
current_image_id="$(docker inspect --format '{{.Image}}' geointel)"
if ! [[ "$current_image_id" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "Running release does not expose one immutable Docker image ID." >&2
return 1
fi
release_id="predeploy-${GEOINTEL_RELEASE_TOKEN:0:24}-$(date -u +%Y%m%dT%H%M%SZ)-$$"
GEOINTEL_PREDEPLOY_ROLLBACK_TAG="${GEOINTEL_IMAGE_REPOSITORY}:rollback-${release_id}"
existing_rollback_id="$(docker image inspect --format '{{.Id}}' "$GEOINTEL_PREDEPLOY_ROLLBACK_TAG" 2>/dev/null || true)"
if [ -n "$existing_rollback_id" ] && [ "$existing_rollback_id" != "$current_image_id" ]; then
echo "Backup-specific rollback tag already identifies different image bytes." >&2
return 1
fi
docker tag "$current_image_id" "$GEOINTEL_PREDEPLOY_ROLLBACK_TAG"
# This conservative full-copy fallback estimate runs while the existing
# release is still healthy. Verified backup-to-backup hardlinks normally
# avoid recopying unchanged bytes, but are never assumed for this fail-closed
# capacity decision.
preflight_backup_capacity
select_verified_link_dest
if [ -n "$GEOINTEL_BACKUP_LINK_DEST" ]; then
backup_link_args=(--link-dest-backup "$GEOINTEL_BACKUP_LINK_DEST")
fi
echo "Quiescing the current backend so the rollback point cannot miss concurrent writes..."
if ! docker exec -i geointel python - <<'PY'
import os
import pathlib
import signal
import time
matches = []
for item in pathlib.Path("/proc").iterdir():
if not item.name.isdigit() or int(item.name) in {os.getpid(), os.getppid()}:
continue
try:
command = (item / "cmdline").read_bytes().replace(b"\0", b" ")
except (OSError, PermissionError):
continue
if b"uvicorn" in command and b"app.main:app" in command:
matches.append(int(item.name))
if not matches:
raise SystemExit("Could not identify the running GeoIntel backend")
for process_id in matches:
os.kill(process_id, signal.SIGTERM)
deadline = time.monotonic() + 60
remaining = matches
while remaining and time.monotonic() < deadline:
time.sleep(0.25)
remaining = [process_id for process_id in remaining if pathlib.Path(f"/proc/{process_id}").exists()]
if remaining:
raise SystemExit(f"Backend did not stop cleanly: {remaining}")
print(f"Stopped {len(matches)} backend process(es)")
PY
then
echo "Could not quiesce the current backend; refusing a potentially inconsistent backup." >&2
docker restart geointel >/dev/null || true
wait_for_geointel_health || true
return 1
fi
GEOINTEL_PREDEPLOY_BACKUP_DIR="${GEOINTEL_BACKUPS_PATH%/}/${release_id}"
echo "Creating mandatory pre-deploy backup ${release_id}..."
if ! bash scripts/backup_release_state.sh \
--container geointel \
--output-root "$GEOINTEL_BACKUPS_PATH" \
--release-id "$release_id" \
--storage-path "$GEOINTEL_STORAGE_PATH" \
--models-path "$GEOINTEL_MODELS_PATH" \
--inventory-mode sha256 \
--rollback-image-tag "$GEOINTEL_PREDEPLOY_ROLLBACK_TAG" \
"${backup_link_args[@]}" \
|| ! bash scripts/verify_release_backup.sh \
--container geointel \
--backup-dir "$GEOINTEL_PREDEPLOY_BACKUP_DIR"; then
echo "Pre-deploy backup failed; restarting the unchanged current release." >&2
docker restart geointel >/dev/null || true
wait_for_geointel_health || true
GEOINTEL_PREDEPLOY_BACKUP_DIR=""
return 1
fi
echo "Pre-deploy backup verified: ${GEOINTEL_PREDEPLOY_BACKUP_DIR}"
}
rollback_previous() {
if ! docker image inspect "$GEOINTEL_PREVIOUS_IMAGE" >/dev/null 2>&1; then
echo "Automatic rollback unavailable: ${GEOINTEL_PREVIOUS_IMAGE} does not exist." >&2
if [ -z "$GEOINTEL_PREDEPLOY_BACKUP_DIR" ]; then
echo "Automatic rollback unavailable: no verified pre-deploy database backup was created." >&2
return 1
fi
echo "Rolling back to ${GEOINTEL_PREVIOUS_IMAGE}..."
start_image "$GEOINTEL_PREVIOUS_IMAGE"
echo "Rolling back database and image to the verified pre-deploy state..."
GEOINTEL_DEPLOY_LOCK_HELD=true \
bash deploy/unraid/rollback-dockerman-container.sh \
--backup-dir "$GEOINTEL_PREDEPLOY_BACKUP_DIR" \
--confirm-production-database-restore
}
docker compose -f docker-compose.unraid.yml config >/dev/null
current_image_id="$(docker inspect --format '{{.Image}}' geointel 2>/dev/null || true)"
release_image_id="$(
docker image inspect --format '{{.Id}}' "$GEOINTEL_RELEASE_IMAGE" 2>/dev/null || true
)"
if (
[ -n "$current_image_id" ] &&
[ "$current_image_id" != "$release_image_id" ] &&
docker image inspect "$current_image_id" >/dev/null 2>&1
); then
docker tag "$current_image_id" "$GEOINTEL_PREVIOUS_IMAGE"
elif [ -n "$current_image_id" ] && [ "$current_image_id" = "$release_image_id" ]; then
echo "Current container already uses ${GEOINTEL_RELEASE_IMAGE}; preserving the existing previous image."
fi
if docker image inspect "$GEOINTEL_RELEASE_IMAGE" >/dev/null 2>&1; then
stored_revision="$(
docker image inspect \
@@ -208,7 +555,10 @@ else
.
fi
if ! start_image "$GEOINTEL_RELEASE_IMAGE"; then
scan_release_image
create_predeploy_backup
if ! start_image "$GEOINTEL_RELEASE_IMAGE_ID"; then
rollback_previous || true
exit 1
fi
@@ -32,11 +32,17 @@
<Config Name="Postgres Password" Target="GEOINTEL_POSTGRES_PASSWORD" Default="change-me-before-shared-use" Mode="" Description="Embedded PostGIS database password. Change before shared use." Type="Variable" Display="advanced" Required="true" Mask="true">change-me-before-shared-use</Config>
<Config Name="CORS Origins" Target="GEOINTEL_CORS_ORIGINS" Default="http://localhost:1202,http://127.0.0.1:1202,http://192.168.10.150:1202" Mode="" Description="Comma-separated browser origins allowed to call the backend directly." Type="Variable" Display="advanced" Required="false" Mask="false">http://localhost:1202,http://127.0.0.1:1202,http://192.168.10.150:1202</Config>
<Config Name="Max Upload MB" Target="GEOINTEL_MAX_UPLOAD_MB" Default="500" Mode="" Description="Maximum upload size in MiB enforced consistently by nginx and the backend (1-2048)." Type="Variable" Display="advanced" Required="true" Mask="false">500</Config>
<Config Name="Max In-memory Vector MB" Target="GEOINTEL_MAX_IN_MEMORY_VECTOR_MB" Default="64" Mode="" Description="Maximum decompressed vector payload processed fully in memory by the backend (1-256 MiB)." Type="Variable" Display="advanced" Required="true" Mask="false">64</Config>
<Config Name="Operator Login Enabled" Target="GEOINTEL_AUTH_ENABLED" Default="false" Mode="" Description="Require the single configured operator login before the browser may access workbench APIs." Type="Variable" Display="advanced" Required="true" Mask="false">false</Config>
<Config Name="Operator Username" Target="GEOINTEL_AUTH_USERNAME" Default="" Mode="" Description="Exact username for the single operator account." Type="Variable" Display="advanced" Required="false" Mask="false"></Config>
<Config Name="Operator Password Hash" Target="GEOINTEL_AUTH_PASSWORD_HASH" Default="" Mode="" Description="PBKDF2-SHA256 password hash. Never enter a plaintext password." Type="Variable" Display="advanced" Required="false" Mask="true"></Config>
<Config Name="Operator Session Secret" Target="GEOINTEL_AUTH_SESSION_SECRET" Default="" Mode="" Description="Random secret of at least 32 characters used only to sign browser sessions." Type="Variable" Display="advanced" Required="false" Mask="true"></Config>
<Config Name="Operator Session TTL" Target="GEOINTEL_AUTH_SESSION_TTL_SECONDS" Default="43200" Mode="" Description="Session lifetime in seconds (900-604800)." Type="Variable" Display="advanced" Required="true" Mask="false">43200</Config>
<Config Name="Public Base URL" Target="GEOINTEL_PUBLIC_BASE_URL" Default="http://localhost:1202" Mode="" Description="Public browser origin. Must be HTTPS when Authentik is configured." Type="Variable" Display="advanced" Required="true" Mask="false">http://localhost:1202</Config>
<Config Name="Authentik Issuer" Target="GEOINTEL_AUTHENTIK_ISSUER" Default="" Mode="" Description="Optional HTTPS OIDC issuer. Configure all Authentik values together." Type="Variable" Display="advanced" Required="false" Mask="false"></Config>
<Config Name="Authentik Client ID" Target="GEOINTEL_AUTHENTIK_CLIENT_ID" Default="" Mode="" Description="OIDC client identifier for GeoIntel." Type="Variable" Display="advanced" Required="false" Mask="false"></Config>
<Config Name="Authentik Client Secret" Target="GEOINTEL_AUTHENTIK_CLIENT_SECRET" Default="" Mode="" Description="OIDC client secret; never commit this value." Type="Variable" Display="advanced" Required="false" Mask="true"></Config>
<Config Name="Authentik Allowed Email" Target="GEOINTEL_AUTHENTIK_ALLOWED_EMAIL" Default="" Mode="" Description="Exact verified operator e-mail address allowed to create an operator session." Type="Variable" Display="advanced" Required="false" Mask="false"></Config>
<Config Name="Guest Demo Enabled" Target="GEOINTEL_GUEST_ACCESS_ENABLED" Default="true" Mode="" Description="Show a guest button that opens only the seeded, restricted demo workspace. Enabled by default when operator login is active; set false on non-demo instances." Type="Variable" Display="advanced" Required="true" Mask="false">true</Config>
<Config Name="Guest Display Name" Target="GEOINTEL_GUEST_DISPLAY_NAME" Default="Gast" Mode="" Description="Label shown for the temporary guest session." Type="Variable" Display="advanced" Required="true" Mask="false">Gast</Config>
<Config Name="Guest Session TTL" Target="GEOINTEL_GUEST_SESSION_TTL_SECONDS" Default="7200" Mode="" Description="Temporary guest session lifetime in seconds (900-86400)." Type="Variable" Display="advanced" Required="true" Mask="false">7200</Config>
+13 -2
View File
@@ -34,6 +34,8 @@ GEOINTEL_CORS_ORIGINS=https://geointel.itworx.tech,http://geointel.itworx.tech,h
# Upload guard in MiB. The same 1-2048 limit is applied by nginx and FastAPI.
GEOINTEL_MAX_UPLOAD_MB=500
# Maximum decompressed vector payload processed fully in memory (1-256 MiB).
GEOINTEL_MAX_IN_MEMORY_VECTOR_MB=64
GEOINTEL_AOI_WORKER_ENABLED=true
GEOINTEL_AOI_WORKER_POLL_SECONDS=2
@@ -46,6 +48,14 @@ GEOINTEL_AUTH_PASSWORD_HASH=
GEOINTEL_AUTH_SESSION_SECRET=
GEOINTEL_AUTH_SESSION_TTL_SECONDS=43200
# Optional additive Authentik OIDC login. Configure all fields together. The
# local operator credentials above remain the recovery login.
GEOINTEL_PUBLIC_BASE_URL=http://localhost:1202
GEOINTEL_AUTHENTIK_ISSUER=
GEOINTEL_AUTHENTIK_CLIENT_ID=
GEOINTEL_AUTHENTIK_CLIENT_SECRET=
GEOINTEL_AUTHENTIK_ALLOWED_EMAIL=
# Guest access is enabled by default whenever operator authentication is active.
# It opens the seeded GeoIntel demo in a temporary, API-enforced restricted
# session. Set this to false on installations containing private project data.
@@ -153,8 +163,9 @@ SPW_TERRAIN_ANALYSIS_RESOLUTION_M=5
SPW_TERRAIN_MAX_SIDE_M=20000
SPW_TERRAIN_MAX_PIXELS=12000000
# Optional configured-YOLO runtime. Keep disabled unless a local model is mounted.
GEOINTEL_INSTALL_AI=false
# Production releases always contain the pinned AI dependencies. Inference may
# remain disabled until an integrity-bound local model is mounted.
GEOINTEL_INSTALL_AI=true
YOLO_ENABLED=false
YOLO_MODELS_DIR=/app/models
YOLO_MODEL_PATH=
+43 -2
View File
@@ -1,3 +1,18 @@
geo $geointel_trusted_forwarder {
default 0;
127.0.0.0/8 1;
::1/128 1;
# The outer Nginx Proxy Manager reaches this container through Docker's
# internal bridge; public/LAN clients are not trusted forwarders.
172.16.0.0/12 1;
}
map "$geointel_trusted_forwarder:$http_x_forwarded_proto" $geointel_forwarded_proto {
default $scheme;
"1:https" https;
"1:http" http;
}
server {
listen 80;
server_name _;
@@ -5,21 +20,42 @@ server {
proxy_read_timeout 600s;
proxy_send_timeout 600s;
add_header Content-Security-Policy "frame-ancestors 'none'" always;
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;
root /usr/share/nginx/html;
index index.html;
location = /index.html {
add_header Cache-Control "no-cache";
add_header Content-Security-Policy "frame-ancestors 'none'" always;
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;
try_files /index.html =404;
}
location = /geointel-icon.svg {
add_header Cache-Control "public, max-age=3600";
add_header Content-Security-Policy "frame-ancestors 'none'" always;
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;
try_files /geointel-icon.svg =404;
}
location = /geointel-icon.png {
add_header Cache-Control "public, max-age=3600";
add_header Content-Security-Policy "frame-ancestors 'none'" always;
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;
# Preserve the stable DockerMan/public URL while the frontend keeps
# its explicit 32px and 180px icon variants.
try_files /geointel-icon-180.png =404;
@@ -27,6 +63,11 @@ server {
location /assets/ {
add_header Cache-Control "no-cache";
add_header Content-Security-Policy "frame-ancestors 'none'" always;
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;
try_files $uri =404;
}
@@ -36,7 +77,7 @@ server {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Proto $geointel_forwarded_proto;
}
location = /health {
@@ -45,7 +86,7 @@ server {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Proto $geointel_forwarded_proto;
}
location = /health/live {
+323
View File
@@ -0,0 +1,323 @@
#!/usr/bin/env bash
set -euo pipefail
# Restore the production database from a verified pre-deploy dump while the
# normal GeoIntel container is stopped. This is intentionally a separate,
# explicitly confirmed operation: starting an older image against a schema
# migrated by a newer image is not a safe rollback strategy.
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
cd "$ROOT"
GEOINTEL_DEPLOY_LOCK_FILE="${GEOINTEL_DEPLOY_LOCK_FILE:-/tmp/geointel-release-deploy.lock}"
if [ "${GEOINTEL_DEPLOY_LOCK_HELD:-false}" != "true" ]; then
command -v flock >/dev/null 2>&1 || {
echo "GeoIntel database restore requires flock to prevent concurrent deployment." >&2
exit 2
}
exec 9>"$GEOINTEL_DEPLOY_LOCK_FILE"
if ! flock -n 9; then
echo "Another GeoIntel deployment or rollback is already running." >&2
exit 3
fi
fi
BACKUP_DIR=""
CONFIRMED="false"
RESTORE_IMAGE="${GEOINTEL_ROLLBACK_IMAGE:-}"
GEOINTEL_CONTAINER_NAME="${GEOINTEL_CONTAINER_NAME:-geointel}"
usage() {
cat <<'EOF'
Usage: bash deploy/unraid/restore-predeploy-database.sh \
--backup-dir PATH --confirm-production-database-restore [options]
Stops the normal GeoIntel container, starts an isolated PostGIS recovery
container on the same persistent database path, restores the checksum-verified
custom-format dump, validates Alembic/table counts, and stops recovery again.
The caller must start the rollback image after this command succeeds.
Options:
--image IMAGE Recovery image containing PostgreSQL/PostGIS tools
--container NAME Normal application container (default: geointel)
EOF
}
while [ "$#" -gt 0 ]; do
case "$1" in
--backup-dir) BACKUP_DIR="$2"; shift 2 ;;
--confirm-production-database-restore) CONFIRMED="true"; shift ;;
--image) RESTORE_IMAGE="$2"; shift 2 ;;
--container) GEOINTEL_CONTAINER_NAME="$2"; shift 2 ;;
--help|-h) usage; exit 0 ;;
*) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
if [ "$CONFIRMED" != "true" ] || [ -z "$BACKUP_DIR" ]; then
echo "Explicit --confirm-production-database-restore and --backup-dir are required." >&2
exit 2
fi
for required in docker python3 sha256sum; do
command -v "$required" >/dev/null 2>&1 || {
echo "Missing required command: $required" >&2
exit 2
}
done
if [ -f .env ]; then
set -a
# shellcheck disable=SC1091
. ./.env
set +a
fi
GEOINTEL_BACKUPS_PATH="${GEOINTEL_BACKUPS_PATH:-/mnt/user/appdata/geointel/backups}"
GEOINTEL_POSTGIS_DATA_PATH="${GEOINTEL_POSTGIS_DATA_PATH:-/mnt/user/appdata/geointel/postgres-data}"
GEOINTEL_POSTGRES_DB="${GEOINTEL_POSTGRES_DB:-geointel}"
GEOINTEL_POSTGRES_USER="${GEOINTEL_POSTGRES_USER:-geointel}"
GEOINTEL_POSTGRES_PASSWORD="${GEOINTEL_POSTGRES_PASSWORD:-}"
if ! [[ "$GEOINTEL_POSTGRES_DB" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] \
|| ! [[ "$GEOINTEL_POSTGRES_USER" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then
echo "Configured PostGIS database and user names must be simple SQL identifiers." >&2
exit 2
fi
case "$GEOINTEL_POSTGRES_PASSWORD" in
''|geointel|postgres|password|changeme|change-me-before-shared-use)
echo "Refusing database restore with an empty or known-default PostGIS password." >&2
exit 2
;;
esac
test -f "$GEOINTEL_POSTGIS_DATA_PATH/PG_VERSION" || {
echo "Persistent PostGIS data path is not initialized: $GEOINTEL_POSTGIS_DATA_PATH" >&2
exit 3
}
GEOINTEL_BACKUPS_PATH="$(python3 -c 'import pathlib,sys; print(pathlib.Path(sys.argv[1]).expanduser().resolve())' "$GEOINTEL_BACKUPS_PATH")"
BACKUP_DIR="$(python3 -c 'import pathlib,sys; print(pathlib.Path(sys.argv[1]).expanduser().resolve())' "$BACKUP_DIR")"
python3 - "$GEOINTEL_BACKUPS_PATH" "$BACKUP_DIR" <<'PY'
import pathlib
import sys
root = pathlib.Path(sys.argv[1])
backup = pathlib.Path(sys.argv[2])
try:
backup.relative_to(root)
except ValueError as exc:
raise SystemExit(f"Backup directory must be below {root}") from exc
if backup == root:
raise SystemExit("Backup directory must identify one immutable backup")
PY
for required_file in manifest.json database.dump database.list database-metadata.tsv table-counts.tsv CHECKSUMS.sha256; do
test -s "$BACKUP_DIR/$required_file" || {
echo "Missing or empty backup artifact: $required_file" >&2
exit 3
}
done
(
cd "$BACKUP_DIR"
sha256sum -c CHECKSUMS.sha256
)
python3 "$ROOT/scripts/release_backup_snapshot.py" verify-backup --backup-dir "$BACKUP_DIR"
IFS=$'\t' read -r BACKUP_DB BACKUP_USER BACKUP_IMAGE_ID BACKUP_RELEASE_ID < <(
python3 - "$BACKUP_DIR/manifest.json" <<'PY'
import json
import pathlib
import sys
payload = json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))
if payload.get("schema_version") != 1 or payload.get("read_only_source") is not True:
raise SystemExit("Unsupported or unsafe backup manifest")
print(
f"{payload.get('database_name', '')}\t{payload.get('database_user', '')}\t"
f"{payload.get('image_id', '')}\t{payload.get('release_id', '')}"
)
PY
)
if [ "$BACKUP_DB" != "$GEOINTEL_POSTGRES_DB" ] || [ "$BACKUP_USER" != "$GEOINTEL_POSTGRES_USER" ]; then
echo "Backup database identity does not match the configured production database." >&2
exit 3
fi
if ! [[ "$BACKUP_IMAGE_ID" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "Backup manifest does not contain one immutable Docker image ID." >&2
exit 3
fi
if [ -z "$RESTORE_IMAGE" ]; then
RESTORE_IMAGE="$BACKUP_IMAGE_ID"
fi
docker image inspect "$RESTORE_IMAGE" >/dev/null
RESTORE_IMAGE_ID="$(docker image inspect --format '{{.Id}}' "$RESTORE_IMAGE")"
if [ -z "$BACKUP_IMAGE_ID" ] || [ "$BACKUP_IMAGE_ID" != "$RESTORE_IMAGE_ID" ]; then
echo "Backup image identity does not match the retained rollback image." >&2
exit 3
fi
case "$BACKUP_RELEASE_ID" in
predeploy-*) ;;
*) echo "Production rollback requires a predeploy backup." >&2; exit 3 ;;
esac
if docker ps -a --format '{{.Names}}' | grep -Fxq "$GEOINTEL_CONTAINER_NAME"; then
docker rm -f "$GEOINTEL_CONTAINER_NAME" >/dev/null
fi
RECOVERY_CONTAINER="geointel-db-restore-$(date -u +%Y%m%d%H%M%S)-$$"
RESTORE_PROOF_DB="geointel_restore_proof_$(date -u +%Y%m%d%H%M%S)_$$"
RECOVERY_DB="geointel_pre_restore_$(date -u +%Y%m%d%H%M%S)_$$"
FAILED_RESTORE_DB="geointel_failed_restore_$(date -u +%Y%m%d%H%M%S)_$$"
SWAP_COMPLETE="false"
cleanup_recovery() {
if [ "$SWAP_COMPLETE" != "true" ] \
&& [ "$(docker inspect -f '{{.State.Running}}' "$RECOVERY_CONTAINER" 2>/dev/null || true)" = "true" ]; then
docker exec "$RECOVERY_CONTAINER" dropdb --if-exists --force \
-U "$GEOINTEL_POSTGRES_USER" "$RESTORE_PROOF_DB" >/dev/null 2>&1 || true
fi
docker rm -f "$RECOVERY_CONTAINER" >/dev/null 2>&1 || true
}
trap cleanup_recovery EXIT
docker run -d \
--name "$RECOVERY_CONTAINER" \
--restart no \
-e PGDATA=/var/lib/postgresql/data \
-e PGPASSWORD="$GEOINTEL_POSTGRES_PASSWORD" \
-v "$GEOINTEL_POSTGIS_DATA_PATH:/var/lib/postgresql/data" \
-v "$BACKUP_DIR:/restore:ro" \
--entrypoint /bin/bash \
"$RESTORE_IMAGE" \
-c 'set -euo pipefail; chown postgres:postgres "$PGDATA"; exec gosu postgres postgres' \
>/dev/null
for attempt in $(seq 1 180); do
if docker exec "$RECOVERY_CONTAINER" pg_isready -h 127.0.0.1 -U "$GEOINTEL_POSTGRES_USER" -d postgres >/dev/null 2>&1; then
break
fi
if [ "$(docker inspect -f '{{.State.Running}}' "$RECOVERY_CONTAINER" 2>/dev/null || true)" != "true" ]; then
echo "Database recovery container exited before PostGIS became ready." >&2
docker logs "$RECOVERY_CONTAINER" >&2 || true
exit 4
fi
if [ "$attempt" -eq 180 ]; then
echo "PostGIS recovery did not become ready within six minutes." >&2
exit 4
fi
sleep 2
done
RESTORED_LIST="$(mktemp)"
trap 'rm -f -- "$RESTORED_LIST"; cleanup_recovery' EXIT
docker exec "$RECOVERY_CONTAINER" pg_restore --list /restore/database.dump > "$RESTORED_LIST"
cmp -s "$RESTORED_LIST" "$BACKUP_DIR/database.list" || {
echo "Recovery image reads a different PostgreSQL archive listing." >&2
exit 4
}
if ! docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d postgres -Atqc \
"SELECT 1 FROM pg_database WHERE datname = '${GEOINTEL_POSTGRES_DB}';" | grep -Fxq 1; then
echo "Configured production database does not exist; refusing replacement." >&2
exit 4
fi
for generated_database in "$RESTORE_PROOF_DB" "$RECOVERY_DB" "$FAILED_RESTORE_DB"; do
if docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d postgres -Atqc \
"SELECT 1 FROM pg_database WHERE datname = '${generated_database}';" | grep -Fxq 1; then
echo "Generated recovery database already exists: ${generated_database}" >&2
exit 4
fi
done
# Prove the complete archive in a separate database before touching production.
docker exec "$RECOVERY_CONTAINER" createdb \
-U "$GEOINTEL_POSTGRES_USER" "$RESTORE_PROOF_DB"
docker exec "$RECOVERY_CONTAINER" pg_restore \
--exit-on-error \
--no-owner \
--no-privileges \
-U "$GEOINTEL_POSTGRES_USER" \
-d "$RESTORE_PROOF_DB" \
/restore/database.dump
EXPECTED_HEAD="$(awk -F $'\t' '$1 == "alembic_head" { print $2 }' "$BACKUP_DIR/database-metadata.tsv")"
validate_restored_database() {
local database_name="$1"
local restored_head=""
restored_head="$(docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d "$database_name" -Atqc \
'SELECT version_num FROM alembic_version;')"
if [ -z "$EXPECTED_HEAD" ] || [ "$restored_head" != "$EXPECTED_HEAD" ]; then
echo "Restored Alembic head '$restored_head' differs from backup head '$EXPECTED_HEAD'." >&2
return 1
fi
while IFS=$'\t' read -r table expected; do
[[ "$table" =~ ^[a-z_]+$ ]] || {
echo "Unsafe table name in retained counts: $table" >&2
return 1
}
actual="$(docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d "$database_name" -Atqc \
"SELECT count(*) FROM public.${table};")"
if [ "$actual" != "$expected" ]; then
echo "Restored count mismatch for $table: expected $expected, got $actual." >&2
return 1
fi
done < "$BACKUP_DIR/table-counts.tsv"
}
validate_restored_database "$RESTORE_PROOF_DB"
echo "Isolated predeploy restore proof passed: ${RESTORE_PROOF_DB}"
docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d postgres -c \
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname IN ('${GEOINTEL_POSTGRES_DB}', '${RESTORE_PROOF_DB}') AND pid <> pg_backend_pid();" \
>/dev/null
docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d postgres -c \
"ALTER DATABASE ${GEOINTEL_POSTGRES_DB} RENAME TO ${RECOVERY_DB};"
if ! docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d postgres -c \
"ALTER DATABASE ${RESTORE_PROOF_DB} RENAME TO ${GEOINTEL_POSTGRES_DB};"; then
echo "Restored database cutover failed; restoring the untouched production database name." >&2
docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d postgres -c \
"ALTER DATABASE ${RECOVERY_DB} RENAME TO ${GEOINTEL_POSTGRES_DB};"
exit 4
fi
SWAP_COMPLETE="true"
if ! validate_restored_database "$GEOINTEL_POSTGRES_DB"; then
echo "Post-cutover validation failed; restoring the retained pre-restore database." >&2
docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d postgres -c \
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = '${GEOINTEL_POSTGRES_DB}' AND pid <> pg_backend_pid();" \
>/dev/null
docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d postgres -c \
"ALTER DATABASE ${GEOINTEL_POSTGRES_DB} RENAME TO ${FAILED_RESTORE_DB};"
docker exec "$RECOVERY_CONTAINER" psql -X -v ON_ERROR_STOP=1 \
-U "$GEOINTEL_POSTGRES_USER" -d postgres -c \
"ALTER DATABASE ${RECOVERY_DB} RENAME TO ${GEOINTEL_POSTGRES_DB};"
SWAP_COMPLETE="false"
echo "Original production database was restored; failed restore retained as ${FAILED_RESTORE_DB}." >&2
exit 4
fi
if [ -z "$RECOVERY_DB" ]; then
echo "Recovery database identity was not retained." >&2
exit 4
fi
while IFS=$'\t' read -r table expected; do
[[ "$table" =~ ^[a-z_]+$ ]] || {
echo "Unsafe table name in retained counts: $table" >&2
exit 4
}
done < "$BACKUP_DIR/table-counts.tsv"
rm -f -- "$RESTORED_LIST"
cleanup_recovery
trap - EXIT
echo "Production database restored and verified from: $BACKUP_DIR"
echo "Pre-restore production database retained for operator recovery as: $RECOVERY_DB"
+68 -4
View File
@@ -4,14 +4,78 @@ set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
cd "$ROOT"
GEOINTEL_ROLLBACK_IMAGE="${GEOINTEL_ROLLBACK_IMAGE:-geointel-all-in-one:previous}"
GEOINTEL_DEPLOY_LOCK_FILE="${GEOINTEL_DEPLOY_LOCK_FILE:-/tmp/geointel-release-deploy.lock}"
if [ "${GEOINTEL_DEPLOY_LOCK_HELD:-false}" != "true" ]; then
command -v flock >/dev/null 2>&1 || {
echo "GeoIntel rollback requires flock to prevent concurrent deployment." >&2
exit 2
}
exec 9>"$GEOINTEL_DEPLOY_LOCK_FILE"
if ! flock -n 9; then
echo "Another GeoIntel deployment or rollback is already running." >&2
exit 3
fi
GEOINTEL_DEPLOY_LOCK_HELD="true"
export GEOINTEL_DEPLOY_LOCK_HELD
fi
if ! docker image inspect "$GEOINTEL_ROLLBACK_IMAGE" >/dev/null 2>&1; then
echo "Rollback image does not exist: ${GEOINTEL_ROLLBACK_IMAGE}" >&2
GEOINTEL_ROLLBACK_IMAGE="${GEOINTEL_ROLLBACK_IMAGE:-}"
BACKUP_DIR="${GEOINTEL_ROLLBACK_BACKUP_DIR:-}"
CONFIRM_RESTORE="false"
usage() {
cat <<'EOF'
Usage: bash deploy/unraid/rollback-dockerman-container.sh \
--backup-dir PATH --confirm-production-database-restore
Restores the verified pre-deploy PostgreSQL dump first and only then starts the
retained previous image. Image-only rollback against an unknown migrated
schema is deliberately not supported.
EOF
}
while [ "$#" -gt 0 ]; do
case "$1" in
--backup-dir) BACKUP_DIR="$2"; shift 2 ;;
--confirm-production-database-restore) CONFIRM_RESTORE="true"; shift ;;
--help|-h) usage; exit 0 ;;
*) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
if [ -z "$BACKUP_DIR" ] || [ "$CONFIRM_RESTORE" != "true" ]; then
echo "Rollback requires a verified pre-deploy backup and explicit database-restore confirmation." >&2
usage >&2
exit 2
fi
echo "Starting rollback image ${GEOINTEL_ROLLBACK_IMAGE} without changing persistent volumes..."
restore_image_args=()
if [ -n "$GEOINTEL_ROLLBACK_IMAGE" ]; then
restore_image_args=(--image "$GEOINTEL_ROLLBACK_IMAGE")
fi
echo "Restoring the pre-deploy database before starting its checksum-bound image..."
bash deploy/unraid/restore-predeploy-database.sh \
--backup-dir "$BACKUP_DIR" \
"${restore_image_args[@]}" \
--confirm-production-database-restore
if [ -z "$GEOINTEL_ROLLBACK_IMAGE" ]; then
GEOINTEL_ROLLBACK_IMAGE="$(python3 - "$BACKUP_DIR/manifest.json" <<'PY'
import json
import pathlib
import re
import sys
image_id = json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8")).get("image_id", "")
if not re.fullmatch(r"sha256:[0-9a-f]{64}", image_id):
raise SystemExit("Backup manifest lacks an immutable rollback image ID")
print(image_id)
PY
)"
fi
docker image inspect "$GEOINTEL_ROLLBACK_IMAGE" >/dev/null
echo "Starting rollback image ${GEOINTEL_ROLLBACK_IMAGE} with the restored persistent database..."
GEOINTEL_IMAGE="$GEOINTEL_ROLLBACK_IMAGE" bash deploy/unraid/run-dockerman-container.sh
for attempt in $(seq 1 90); do
+53
View File
@@ -33,6 +33,7 @@ GEOINTEL_POSTGRES_USER="${GEOINTEL_POSTGRES_USER:-geointel}"
GEOINTEL_POSTGRES_PASSWORD="${GEOINTEL_POSTGRES_PASSWORD:-}"
GEOINTEL_CORS_ORIGINS="${GEOINTEL_CORS_ORIGINS:-http://localhost:${GEOINTEL_FRONTEND_PORT},http://127.0.0.1:${GEOINTEL_FRONTEND_PORT},http://192.168.10.150:${GEOINTEL_FRONTEND_PORT}}"
GEOINTEL_MAX_UPLOAD_MB="${GEOINTEL_MAX_UPLOAD_MB:-500}"
GEOINTEL_MAX_IN_MEMORY_VECTOR_MB="${GEOINTEL_MAX_IN_MEMORY_VECTOR_MB:-64}"
GEOINTEL_AOI_WORKER_ENABLED="${GEOINTEL_AOI_WORKER_ENABLED:-true}"
GEOINTEL_AOI_WORKER_POLL_SECONDS="${GEOINTEL_AOI_WORKER_POLL_SECONDS:-2}"
GEOINTEL_AUTH_ENABLED="${GEOINTEL_AUTH_ENABLED:-false}"
@@ -40,6 +41,11 @@ GEOINTEL_AUTH_USERNAME="${GEOINTEL_AUTH_USERNAME:-}"
GEOINTEL_AUTH_PASSWORD_HASH="${GEOINTEL_AUTH_PASSWORD_HASH:-}"
GEOINTEL_AUTH_SESSION_SECRET="${GEOINTEL_AUTH_SESSION_SECRET:-}"
GEOINTEL_AUTH_SESSION_TTL_SECONDS="${GEOINTEL_AUTH_SESSION_TTL_SECONDS:-43200}"
GEOINTEL_PUBLIC_BASE_URL="${GEOINTEL_PUBLIC_BASE_URL:-http://localhost:${GEOINTEL_FRONTEND_PORT}}"
GEOINTEL_AUTHENTIK_ISSUER="${GEOINTEL_AUTHENTIK_ISSUER:-}"
GEOINTEL_AUTHENTIK_CLIENT_ID="${GEOINTEL_AUTHENTIK_CLIENT_ID:-}"
GEOINTEL_AUTHENTIK_CLIENT_SECRET="${GEOINTEL_AUTHENTIK_CLIENT_SECRET:-}"
GEOINTEL_AUTHENTIK_ALLOWED_EMAIL="${GEOINTEL_AUTHENTIK_ALLOWED_EMAIL:-}"
GEOINTEL_GUEST_ACCESS_ENABLED="${GEOINTEL_GUEST_ACCESS_ENABLED:-true}"
GEOINTEL_GUEST_DISPLAY_NAME="${GEOINTEL_GUEST_DISPLAY_NAME:-Gast}"
GEOINTEL_GUEST_SESSION_TTL_SECONDS="${GEOINTEL_GUEST_SESSION_TTL_SECONDS:-7200}"
@@ -189,6 +195,17 @@ validate_runtime_config() {
return 2
fi
case "$GEOINTEL_MAX_IN_MEMORY_VECTOR_MB" in
''|*[!0-9]*)
echo "GEOINTEL_MAX_IN_MEMORY_VECTOR_MB must be a whole number." >&2
return 2
;;
esac
if [ "$GEOINTEL_MAX_IN_MEMORY_VECTOR_MB" -lt 1 ] || [ "$GEOINTEL_MAX_IN_MEMORY_VECTOR_MB" -gt 256 ]; then
echo "GEOINTEL_MAX_IN_MEMORY_VECTOR_MB must be between 1 and 256." >&2
return 2
fi
case "$GEOINTEL_AUTH_ENABLED" in
true|false) ;;
*)
@@ -212,6 +229,36 @@ validate_runtime_config() {
esac
fi
local authentik_count=0
local authentik_value
for authentik_value in \
"$GEOINTEL_AUTHENTIK_ISSUER" \
"$GEOINTEL_AUTHENTIK_CLIENT_ID" \
"$GEOINTEL_AUTHENTIK_CLIENT_SECRET" \
"$GEOINTEL_AUTHENTIK_ALLOWED_EMAIL"; do
if [ -n "$authentik_value" ]; then
authentik_count=$((authentik_count + 1))
fi
done
if [ "$authentik_count" -ne 0 ] && [ "$authentik_count" -ne 4 ]; then
echo "All GEOINTEL_AUTHENTIK_* values must be configured together." >&2
return 2
fi
if [ "$authentik_count" -eq 4 ]; then
if [ "$GEOINTEL_AUTH_ENABLED" != "true" ]; then
echo "GEOINTEL_AUTH_ENABLED must be true when Authentik is configured." >&2
return 2
fi
case "$GEOINTEL_AUTHENTIK_ISSUER" in
https://*) ;;
*) echo "GEOINTEL_AUTHENTIK_ISSUER must use HTTPS." >&2; return 2 ;;
esac
case "$GEOINTEL_PUBLIC_BASE_URL" in
https://*) ;;
*) echo "GEOINTEL_PUBLIC_BASE_URL must use HTTPS for Authentik." >&2; return 2 ;;
esac
fi
case "$GEOINTEL_GUEST_ACCESS_ENABLED" in
true|false) ;;
*)
@@ -314,6 +361,7 @@ docker run -d \
-e GEOINTEL_STORAGE_ROOT=/app/storage \
-e GEOINTEL_CORS_ORIGINS="$GEOINTEL_CORS_ORIGINS" \
-e GEOINTEL_MAX_UPLOAD_MB="$GEOINTEL_MAX_UPLOAD_MB" \
-e GEOINTEL_MAX_IN_MEMORY_VECTOR_MB="$GEOINTEL_MAX_IN_MEMORY_VECTOR_MB" \
-e GEOINTEL_AOI_WORKER_ENABLED="$GEOINTEL_AOI_WORKER_ENABLED" \
-e GEOINTEL_AOI_WORKER_POLL_SECONDS="$GEOINTEL_AOI_WORKER_POLL_SECONDS" \
-e GEOINTEL_AUTH_ENABLED="$GEOINTEL_AUTH_ENABLED" \
@@ -321,6 +369,11 @@ docker run -d \
-e GEOINTEL_AUTH_PASSWORD_HASH="$GEOINTEL_AUTH_PASSWORD_HASH" \
-e GEOINTEL_AUTH_SESSION_SECRET="$GEOINTEL_AUTH_SESSION_SECRET" \
-e GEOINTEL_AUTH_SESSION_TTL_SECONDS="$GEOINTEL_AUTH_SESSION_TTL_SECONDS" \
-e GEOINTEL_PUBLIC_BASE_URL="$GEOINTEL_PUBLIC_BASE_URL" \
-e GEOINTEL_AUTHENTIK_ISSUER="$GEOINTEL_AUTHENTIK_ISSUER" \
-e GEOINTEL_AUTHENTIK_CLIENT_ID="$GEOINTEL_AUTHENTIK_CLIENT_ID" \
-e GEOINTEL_AUTHENTIK_CLIENT_SECRET="$GEOINTEL_AUTHENTIK_CLIENT_SECRET" \
-e GEOINTEL_AUTHENTIK_ALLOWED_EMAIL="$GEOINTEL_AUTHENTIK_ALLOWED_EMAIL" \
-e GEOINTEL_GUEST_ACCESS_ENABLED="$GEOINTEL_GUEST_ACCESS_ENABLED" \
-e GEOINTEL_GUEST_DISPLAY_NAME="$GEOINTEL_GUEST_DISPLAY_NAME" \
-e GEOINTEL_GUEST_SESSION_TTL_SECONDS="$GEOINTEL_GUEST_SESSION_TTL_SECONDS" \