Backup and restore

Backup contents

PostgreSQL logical backup or selected physical method with documented compatibility;
Pulse configuration;
dashboard versions;
alert rules and maintenance windows;
inventory/events/incidents/audit according to policy;
non-secret datasource/channel configuration;
schema/application version;
image digests;
checksums and manifest.

Secrets:

are backed up only through an explicitly secured secret-store procedure;
never appear plaintext in portable exports;
are documented as required reattachment steps if excluded.

Backup behavior

on-demand administrator action;
scheduled optional job;
unique immutable backup ID;
temporary file + atomic finalize;
checksum;
size and duration;
success/failure audit;
retention;
destination health;
no backup only inside the source database volume.

Verification

A backup is not trusted until:

checksum passes;
archive/format opens;
manifest/schema version valid;
secret scan/redaction policy passes;
restore test has succeeded for the release.

Restore test

Use an empty isolated environment:

deploy compatible database/application version;
restore backup;
run migrations if expected;
start services;
authenticate;
verify dashboards/rules/inventory/events/incidents;
query metrics datasource through restored config;
run smoke tests;
compare key counts/checksums;
document credential reattachment.

Do not overwrite production during routine validation.

Upgrade rollback

Before schema upgrade:

create verified backup;
record current migration version and image digests;
test upgrade from prior release fixture;
test supported downgrade or forward-recovery strategy;
document compatibility window.

If down migration is unsafe, use restore/forward-fix and state this explicitly.

Supported M9-10 recovery path

Pulse migrations are forward-only. Do not manually remove rows from schema_migrations, run ad-hoc down SQL, or point an older image at a schema whose compatibility has not been proven. The supported recovery path after a failed upgrade is:

Freeze only the Pulse compose project and preserve the migration/API logs.
Record the image digests, schema_migrations rows and the verified backup ID before making another attempt.
Correct the release/configuration fault, use the same or a newer compatible Pulse image, and re-run cmd/migrate. Each migration is committed with its schema-migration record in one transaction and repeat runs are safe.
Restart the Pulse API/worker/agent, then verify health, schema migration count, dashboard/rule current-version links and the operational smoke checks.
If forward recovery cannot be completed, build a new empty isolated target, restore the last verified portable backup, reattach excluded secrets, run compatible migrations and validate it before any production cutover.

The old image/configuration is a rollback point only when its documented schema compatibility is satisfied. M9-10 verified the baseline 61cb2a4 to 11e3496 transition, a repeat migration and a PostgreSQL transaction fault rollback. It does not claim that arbitrary future down-migrations are safe.

Recovery objectives

Codex must measure and document practical RPO/RTO for the environment. Do not invent guarantees.

Baseline intent:

configuration/incident loss limited by backup schedule;
restore procedure executable by the operator;
no dependency on the failed Pulse API to access backup instructions.

M9-09 portable backup

Configure a protected, operator-owned destination with PULSE_BACKUP_DIR and an optional retention count with PULSE_BACKUP_RETENTION (default 5, maximum 100). The API exposes the following administrator-only endpoint:

GET /api/v1/system/backups lists verified backup archives.
POST /api/v1/system/backups creates a temporary archive, writes it atomically, emits a sidecar SHA-256 checksum and applies retention.

Format version 2 of the archive is a ZIP containing a manifest and deterministic JSONL entries for the approved non-secret persistence tables. Durable discovery identity in container_aliases is included so a restored worker does not assign new entity identities to unchanged containers. Runtime configuration, system_settings, notification channel configuration, secret references, notification delivery state and bounded agent_snapshots and capacity_samples runtime telemetry are excluded and must be reattached or republished through the approved procedure. The exporter rejects sensitive JSON keys rather than writing them. The manifest records the migration count, table row counts and per-entry SHA-256 values. Restore first verifies the archive and sidecar, then only restores into a database where every backed-up table is empty, inside one transaction. Current dashboard/rule version foreign keys are restored after their version rows exist. Version 1 archives predate durable container-alias coverage and are deliberately rejected instead of being accepted as complete recovery points.

The PostgreSQL clean-room test classifies every migrated application table as either portable or deliberately excluded and validates every configured export and restore column against the migrated schema. CI sets PULSE_REQUIRE_BACKUP_INTEGRATION=true and provisions separate source and restore databases, so a missing DSN or a newly unclassified table fails rather than silently skipping this release gate.

A reproducible DSN-gated test is:

$env:PULSE_TEST_DATABASE_URL = 'postgresql://...'
$env:PULSE_TEST_RESTORE_DATABASE_URL = 'postgresql://...'
go test ./internal/backup -run TestPostgreSQLBackupRestoreCleanRoom -count=1 -v

Do not put actual DSNs, passwords, secret-store values or backup bytes in the repository, evidence or shell history. The restore test uses an isolated empty environment and never overwrites production.

5.8 KiB Raw Permalink Blame History