5.5 KiB
PostgreSQL backup and restore
Method
ModelForge owns its own backup rather than delegating to a filesystem snapshot:
pg_dump --format=custom --compress=6 --no-owner --no-privileges --serializable-deferrable
Custom format restores schema, data, sequences, constraints, indexes and extension dependencies
through pg_restore, and --serializable-deferrable takes the whole dump in one snapshot that
cannot see a partially applied transaction. Copying /var/lib/postgresql/data from a running
server is never done.
The control-plane image ships pg_dump, pg_restore and psql pinned to the same major version
as the server (PostgreSQL 17), installed from Alpine's signed 3.23 repository. The release build
upgrades repository packages before adding that client and verifies the resulting image.
Recorded identity
Every backup records the server version and version number, the major version, the database name,
the installed extensions, the backup tool version, the backup method and a redacted connection
string, alongside the Alembic revision. A restore preflight compares source and destination major
versions and refuses POSTGRES_VERSION_INCOMPATIBLE.
Historical M15 live evidence: server PostgreSQL 17.11, server_version_num 170011, extensions
plpgsql 1.0, tool pg_dump (PostgreSQL) 17.11 (Debian 17.11-1.pgdg13+2), schema
20260827_0021. The v1.1 Alpine release image independently reports pg_dump (PostgreSQL) 17.11.
Point-in-time recovery
PITR: NOT_SUPPORTED
M15 implements verified logical snapshots. WAL archiving and continuous point-in-time recovery are
deliberately out of scope for v1. The value is exported as POINT_IN_TIME_SUPPORT and rendered on
the recovery dashboard so no operator has to infer it. The practical consequence is a finite RPO
bounded by the backup interval, and the platform measures it rather than claiming zero.
Restore
A restore runs against a genuinely separate database. _same_database compares host, port and
database name, and a plan targeting the control plane's own database is refused at creation with
DESTINATION_NOT_ISOLATED. Production targets additionally require
MODELFORGE_RESTORE_ALLOW_PRODUCTION_TARGET and a DISASTER_RECOVERY mode.
pg_restore --no-owner --no-privileges --exit-on-error --single-transaction
--single-transaction means a failed restore leaves nothing behind. The destination must be empty:
DESTINATION_NOT_EMPTY is returned rather than merging into existing data, which is what makes a
retry safe.
Schema compatibility
The build walks its own Alembic chain by down_revision — filename order is not migration order —
and compares the backup's revision against it:
| Backup revision | Result |
|---|---|
| equals head | restored as is |
| known, older than head | restored, then migrated forward to head |
| unknown to this build | SCHEMA_TOO_NEW, fail closed |
| build ships no chain | SCHEMA_UNKNOWN, fail closed |
| backup records none | preflight fails |
Restoring a newer backup onto older code is never silently downgraded. The discovery resolves both
the source checkout and the installed image layout; a packaging error that hides the chain produces
SCHEMA_UNKNOWN rather than an accidental pass.
Validation
After the restore the destination is measured, not assumed: table, constraint, index and sequence counts, the Alembic head, the audit high-water mark, a bounded semantic fingerprint and its diff against the fingerprint captured at backup time, and the residual row count of every current-truth table.
The READY gate requires all of:
database_validated restored table and constraint counts meet the plan's requirement
current_truth_reconciled no stale lease, telemetry or inventory row survives
schema_current the destination is at this build's Alembic head
audit_available the audit trail restored with a non-zero sequence
audit_chain_intact every event/hash/link and the durable checkpoint verify strictly
fingerprint_compatible current m15.2, or the exact schema-0022 m15.1 migration rule
no_unknown_corruption every fingerprint difference is explained
READY always performs a fresh strict target audit verification. A resumed operation cannot reuse
the presence of a journaled VALIDATING or READY duration as that proof. Audit rows are read with
a stable (sequence, id) cursor, so a corrupt duplicate sequence split at the 500-row page boundary
is observed and rejected rather than skipped. A separate aggregate identity check requires total
rows, distinct UUIDs and distinct sequences to agree before the chain can pass; this also catches a
duplicate cursor tuple after constraints were removed from a tampered restored schema.
Live evidence from disaster rehearsal A: 121 tables, 1,759 constraints, 590 indexes, audit high-water mark 838, 108 of 112 fingerprinted tables byte-identical, and four explained differences.
Identity columns
audit_events.sequence is GENERATED BY DEFAULT AS IDENTITY, but AuditWriter always supplies the
value explicitly, so the underlying sequence is never advanced. The dump carries its SEQUENCE SET
and the restore reproduces the source state exactly — an insert that omits sequence collides
identically on both the source and the restored database. This is pre-existing M0 behaviour
faithfully preserved by the restore, not a recovery defect, and it is recorded here because a
future milestone that starts relying on the identity default must fix it in the audit writer first.