32 lines
1.6 KiB
Markdown
32 lines
1.6 KiB
Markdown
# ADR 0023 — Deterministic Advisor and raw evidence matrix
|
|
|
|
Status: accepted
|
|
|
|
## Decision
|
|
|
|
Model comparisons are first-class immutable evidence snapshots. Each evaluated row references an
|
|
existing EvaluationRun; quality metrics and critical-case outcomes are materialized from that run
|
|
and its pairwise comparison rather than accepted as caller claims. Operational, resource, migration,
|
|
security and provenance evidence remain separate fields. Blocked and unknown candidates remain
|
|
visible as blocked or unknown and never receive fabricated metrics.
|
|
|
|
Advisor recommendations are deterministic. The persisted policy requires project-local evidence
|
|
class A for a high-confidence recommendation, treats any critical regression as a hard block,
|
|
requires verified supply chain and runtime fit, and requires non-negative quality deltas. A p95
|
|
increase above 100% is an operational review ceiling, not a weighted quality score. Policy versions
|
|
and rationale are persisted with each recommendation.
|
|
|
|
The only verdicts are `KEEP_CURRENT`, `PROMOTION_ELIGIBLE`, `PROMOTION_NOT_RECOMMENDED` and
|
|
`REQUIRES_MORE_EVIDENCE`. There is deliberately no automatic approval or promotion endpoint.
|
|
|
|
## Evidence hierarchy
|
|
|
|
1. A — local project evaluation on the pinned suite/corpus/configuration.
|
|
2. B — local generic benchmark or measured preflight.
|
|
3. C — independent external benchmark.
|
|
4. D — upstream model card or vendor claim.
|
|
5. E — downloads, likes or trending data.
|
|
|
|
Lower evidence cannot overrule project-local critical regressions without a new explicit policy and
|
|
new recommendation fingerprint.
|