ITWorx-Pulse-Public/docs/operations/BACKUP_RESTORE.md

# Backup and restore

## Backup contents

- PostgreSQL logical backup or selected physical method with documented compatibility;
- Pulse configuration;
- dashboard versions;
- alert rules and maintenance windows;
- inventory/events/incidents/audit according to policy;
- non-secret datasource/channel configuration;
- schema/application version;
- image digests;
- checksums and manifest.

Secrets:
- are backed up only through an explicitly secured secret-store procedure;
- never appear plaintext in portable exports;
- are documented as required reattachment steps if excluded.

## Backup behavior

- on-demand administrator action;
- scheduled optional job;
- unique immutable backup ID;
- temporary file + atomic finalize;
- checksum;
- size and duration;
- success/failure audit;
- retention;
- destination health;
- no backup only inside the source database volume.

## Verification

A backup is not trusted until:
- checksum passes;
- archive/format opens;
- manifest/schema version valid;
- secret scan/redaction policy passes;
- restore test has succeeded for the release.

## Restore test

Use an empty isolated environment:

1. deploy compatible database/application version;
2. restore backup;
3. run migrations if expected;
4. start services;
5. authenticate;
6. verify dashboards/rules/inventory/events/incidents;
7. query metrics datasource through restored config;
8. run smoke tests;
9. compare key counts/checksums;
10. document credential reattachment.

Do not overwrite production during routine validation.

## Upgrade rollback

Before schema upgrade:
- create verified backup;
- record current migration version and image digests;
- test upgrade from prior release fixture;
- test supported downgrade or forward-recovery strategy;
- document compatibility window.

If down migration is unsafe, use restore/forward-fix and state this explicitly.

### Supported M9-10 recovery path

Pulse migrations are forward-only. Do not manually remove rows from
`schema_migrations`, run ad-hoc down SQL, or point an older image at a schema
whose compatibility has not been proven. The supported recovery path after a
failed upgrade is:

1. Freeze only the Pulse compose project and preserve the migration/API logs.
2. Record the image digests, `schema_migrations` rows and the verified backup
   ID before making another attempt.
3. Correct the release/configuration fault, use the same or a newer compatible
   Pulse image, and re-run `cmd/migrate`. Each migration is committed with its
   schema-migration record in one transaction and repeat runs are safe.
4. Restart the Pulse API/worker/agent, then verify health, schema migration
   count, dashboard/rule current-version links and the operational smoke
   checks.
5. If forward recovery cannot be completed, build a new empty isolated target,
   restore the last verified portable backup, reattach excluded secrets, run
   compatible migrations and validate it before any production cutover.

The old image/configuration is a rollback point only when its documented
schema compatibility is satisfied. M9-10 verified the baseline `61cb2a4` to
`11e3496` transition, a repeat migration and a PostgreSQL transaction fault
rollback. It does not claim that arbitrary future down-migrations are safe.

## Recovery objectives

Codex must measure and document practical RPO/RTO for the environment. Do not invent guarantees.

Baseline intent:
- configuration/incident loss limited by backup schedule;
- restore procedure executable by the operator;
- no dependency on the failed Pulse API to access backup instructions.

## M9-09 portable backup

Configure a protected, operator-owned destination with `PULSE_BACKUP_DIR` and an optional retention count with `PULSE_BACKUP_RETENTION` (default 5, maximum 100). The API exposes the following administrator-only endpoint:

- `GET /api/v1/system/backups` lists verified backup archives.
- `POST /api/v1/system/backups` creates a temporary archive, writes it atomically, emits a sidecar SHA-256 checksum and applies retention.

Format version 2 of the archive is a ZIP containing a manifest and deterministic JSONL entries for the approved non-secret persistence tables. Durable discovery identity in `container_aliases` is included so a restored worker does not assign new entity identities to unchanged containers. Runtime configuration, `system_settings`, notification channel configuration, secret references, notification delivery state and bounded `agent_snapshots` and `capacity_samples` runtime telemetry are excluded and must be reattached or republished through the approved procedure. The exporter rejects sensitive JSON keys rather than writing them. The manifest records the migration count, table row counts and per-entry SHA-256 values. Restore first verifies the archive and sidecar, then only restores into a database where every backed-up table is empty, inside one transaction. Current dashboard/rule version foreign keys are restored after their version rows exist. Version 1 archives predate durable container-alias coverage and are deliberately rejected instead of being accepted as complete recovery points.

The PostgreSQL clean-room test classifies every migrated application table as
either portable or deliberately excluded and validates every configured export
and restore column against the migrated schema. CI sets
`PULSE_REQUIRE_BACKUP_INTEGRATION=true` and provisions separate source and
restore databases, so a missing DSN or a newly unclassified table fails rather
than silently skipping this release gate.

A reproducible DSN-gated test is:

```text
$env:PULSE_TEST_DATABASE_URL = 'postgresql://...'
$env:PULSE_TEST_RESTORE_DATABASE_URL = 'postgresql://...'
go test ./internal/backup -run TestPostgreSQLBackupRestoreCleanRoom -count=1 -v
```

Do not put actual DSNs, passwords, secret-store values or backup bytes in the repository, evidence or shell history. The restore test uses an isolated empty environment and never overwrites production.