geointel/docs/ROLLBACK_AND_RECOVERY.md

Rollback and Recovery

Basisregels

Maak per milestone een full zip en update zip. Geen destructieve migrations zonder rollback-notitie.

Database rollback

Elke migration beschrijft upgrade en downgrade. Als downgrade onveilig is, noteer dit expliciet.

Storage rollback

Bestanden worden niet stilzwijgend verwijderd. Gebruik soft-delete waar mogelijk.

AI/model rollback

Elke analysis run bewaart model id, version, parameters, confidence threshold en pipeline version.

RC backup and isolated restore gate

Before any release migration, credential rotation or image promotion:

1. run scripts/backup_release_state.sh against the running all-in-one container;
2. run scripts/verify_release_backup.sh read-only;
3. run scripts/restore_release_backup_smoke.sh with --confirm-isolated-restore;
4. retain the backup directory and the separate restore-smoke JSON with the release evidence.

The restore smoke may only create databases whose name starts with geointel_restore_verify_. It refuses the production database name, does not use pg_restore --clean and drops the temporary database unless an operator explicitly asks to retain it.

Storage and model files are retained as byte-complete, SHA-256 verified snapshots alongside the database dump. Snapshot creation rejects symlinks and special files. The first snapshot uses CoW reflinks when supported and falls back to full copies; later snapshots hard-link checksum-identical files only from a verified older backup, never from live storage.

An old persistent volume can also retain glibc collation metadata for the empty postgres and template1 system databases. If createdb fails for that reason, first complete and verify the production backup, then reindex each affected system database and run ALTER DATABASE <name> REFRESH COLLATION VERSION. Never refresh before reindexing, and never treat this maintenance as a substitute for the isolated restore proof.