42 lines
2.0 KiB
Markdown
42 lines
2.0 KiB
Markdown
# ADR-0045: Chaos-safe reconciliation semantics
|
|
|
|
## Status
|
|
|
|
Accepted in M16 on 2026-08-27.
|
|
|
|
## Decision
|
|
|
|
Every reconciliation path converges rather than accumulates, and each subsystem declares which of
|
|
three answers it gives after an interruption:
|
|
|
|
```text
|
|
resolve the safe outcome is knowable from local state; apply it exactly once
|
|
report the safe outcome depends on external truth; surface it and stop
|
|
fail closed the artefact cannot be trusted; mark it unusable
|
|
```
|
|
|
|
Lifecycle resolves: an incomplete operation rolls back from its immutable snapshot, once, and a
|
|
second start changes nothing. Migration reports: an interrupted cutover stays in its intermediate
|
|
stage because external alias truth cannot be inferred after a crash. Backups fail closed: a backup
|
|
interrupted mid-write can never become restore eligible, and a restore interrupted mid-phase
|
|
requires operator review rather than resuming.
|
|
|
|
Single-use claims are atomic conditional updates whose row count decides the winner, not
|
|
read-then-write checks. Idempotency keys are unique in the database, not in a queue, so a retry
|
|
storm converges on one row regardless of which component retried.
|
|
|
|
## Consequences
|
|
|
|
A restart storm is uneventful: four consecutive control-plane restarts under load produced no
|
|
authoritative drift, and a Redis outage produced no duplicate serving job.
|
|
|
|
The cost is that some recoveries stop and wait for a human. That is deliberate — a migration
|
|
cutover that guesses is how a platform ends up serving two embedding spaces, and an operator
|
|
reconciling through the typed adapter is cheaper than a silent split.
|
|
|
|
"Exactly once" has to be scoped to the operation, not the action class. M15 shipped a reconciliation
|
|
marker keyed on the audit action alone, which meant a backup carrying an earlier restore's
|
|
reconciliation event made every later restore believe it had already reconciled. The marker is now
|
|
per-operation and the readiness gate additionally refuses to pass while stale current truth
|
|
survives.
|