1.6 KiB
ADR 0023 — Deterministic Advisor and raw evidence matrix
Status: accepted
Decision
Model comparisons are first-class immutable evidence snapshots. Each evaluated row references an existing EvaluationRun; quality metrics and critical-case outcomes are materialized from that run and its pairwise comparison rather than accepted as caller claims. Operational, resource, migration, security and provenance evidence remain separate fields. Blocked and unknown candidates remain visible as blocked or unknown and never receive fabricated metrics.
Advisor recommendations are deterministic. The persisted policy requires project-local evidence class A for a high-confidence recommendation, treats any critical regression as a hard block, requires verified supply chain and runtime fit, and requires non-negative quality deltas. A p95 increase above 100% is an operational review ceiling, not a weighted quality score. Policy versions and rationale are persisted with each recommendation.
The only verdicts are KEEP_CURRENT, PROMOTION_ELIGIBLE, PROMOTION_NOT_RECOMMENDED and
REQUIRES_MORE_EVIDENCE. There is deliberately no automatic approval or promotion endpoint.
Evidence hierarchy
- A — local project evaluation on the pinned suite/corpus/configuration.
- B — local generic benchmark or measured preflight.
- C — independent external benchmark.
- D — upstream model card or vendor claim.
- E — downloads, likes or trending data.
Lower evidence cannot overrule project-local critical regressions without a new explicit policy and new recommendation fingerprint.