2.0 KiB
ADR-0045: Chaos-safe reconciliation semantics
Status
Accepted in M16 on 2026-08-27.
Decision
Every reconciliation path converges rather than accumulates, and each subsystem declares which of three answers it gives after an interruption:
resolve the safe outcome is knowable from local state; apply it exactly once
report the safe outcome depends on external truth; surface it and stop
fail closed the artefact cannot be trusted; mark it unusable
Lifecycle resolves: an incomplete operation rolls back from its immutable snapshot, once, and a second start changes nothing. Migration reports: an interrupted cutover stays in its intermediate stage because external alias truth cannot be inferred after a crash. Backups fail closed: a backup interrupted mid-write can never become restore eligible, and a restore interrupted mid-phase requires operator review rather than resuming.
Single-use claims are atomic conditional updates whose row count decides the winner, not read-then-write checks. Idempotency keys are unique in the database, not in a queue, so a retry storm converges on one row regardless of which component retried.
Consequences
A restart storm is uneventful: four consecutive control-plane restarts under load produced no authoritative drift, and a Redis outage produced no duplicate serving job.
The cost is that some recoveries stop and wait for a human. That is deliberate — a migration cutover that guesses is how a platform ends up serving two embedding spaces, and an operator reconciling through the typed adapter is cheaper than a silent split.
"Exactly once" has to be scoped to the operation, not the action class. M15 shipped a reconciliation marker keyed on the audit action alone, which meant a backup carrying an earlier restore's reconciliation event made every later restore believe it had already reconciled. The marker is now per-operation and the readiness gate additionally refuses to pass while stale current truth survives.