# 12 — Quality, linting and evaluation

## Quality philosophy

A playbook is trustworthy when its structure, safety, clarity and observed behavior are evidenced. Popularity, length and confident wording are not quality proof.

## Quality dimensions

### Scope clarity

Does the playbook define the target, permitted reading scope, modification scope, exclusions and protected behavior?

### Safety

Does it prevent destructive, secret-exposing or policy-weakening behavior appropriate to its risk?

### Verification

Does it require relevant checks and evidence rather than a narrative claim?

### Reproducibility

Can the same package version and normalized inputs reproduce the same output?

### Compatibility

Are supported stacks, required profile capabilities and limitations explicit?

### Reporting

Does the final report make outcome, evidence, risks and unresolved items reviewable?

### Efficiency

Does the task avoid unnecessary repository-wide change, redundant investigation and repeated validation without sacrificing safety?

Scores are displayed separately on a defined scale, such as Not assessed, Weak, Adequate, Strong. A combined score may be calculated for sorting but must never replace the dimensions.

## Lifecycle policy

### Draft

- schema-valid or actively being edited;
- may have lint errors;
- not recommended outside author workspace.

### Reviewed

- schema and semantic validation pass;
- no blocking prompt-lint findings on required examples;
- human editorial review complete;
- limitations documented.

### Validated

- Reviewed requirements;
- required evaluation cases pass;
- evaluation environment and fixture version recorded;
- no unresolved safety regression.

### Battle-tested

- Validated requirements;
- minimum real-world run count under policy;
- acceptable operator feedback and failure rate;
- no unaddressed severe incident;
- evidence remains recent enough for the playbook class.

### Deprecated

- replacement or rationale provided;
- historical rendering remains available;
- excluded from default recommendations.

## Static linter catalog

Suggested rule IDs:

- `PB001` missing mission
- `PB002` missing explicit scope
- `PB003` missing done-when criteria
- `PB004` missing reporting contract
- `PB005` duplicate input or step ID
- `PB006` invalid autonomy range
- `PB007` unknown template variable
- `PB008` published version without changelog
- `PB009` validated status without evidence
- `PR001` ambiguous unbounded improvement language
- `PR002` conflicting read-only and modification instruction
- `PR003` vague “best practices” without dimensions
- `PR004` asks for success claim without evidence
- `SA001` token-like value in rendered output
- `SA002` protected path in change scope
- `SA003` destructive migration without rollback/backup
- `SA004` Git push/release not explicitly authorized
- `SA005` imported content placed in policy section
- `VA001` bugfix lacks reproduction/regression step
- `VA002` implementation lacks available build/test validation
- `VA003` dependency change lacks lockfile/install/build check
- `VA004` frontend flow lacks browser verification
- `VA005` inspect playbook lacks evidence-source reporting

Every finding includes severity, location, message, rationale, remediation hint and provenance.

## Evaluation case format

An evaluation case defines:

- case ID and version;
- target playbook/version range;
- fixture repository reference and digest;
- repository profile;
- inputs and autonomy;
- expected prompt properties;
- prohibited prompt properties;
- optional future execution expectations;
- scoring rubric.

MVP can evaluate rendering and lint behavior without executing Codex. Future isolated evaluation can run tasks in disposable fixture environments.

## Static evaluation examples

- generated prompt contains all required headings;
- no secret fixture value appears;
- protected paths are rendered as no-change constraints;
- selected `observe` autonomy contains no implementation permission;
- missing test command creates warning rather than invented command;
- stack incompatibility blocks export;
- same input produces same digest;
- a conditional migration section appears only when migration input is true.

## Future execution evaluation

Fixture repositories intentionally contain known problems. Evaluation runner captures:

- task completion status;
- changed file set;
- protected-path violations;
- command exit codes;
- tests added or changed;
- artifact diffs;
- final report completeness;
- token/time/cost metadata where available;
- human review.

The runner must use isolated disposable environments and must never execute untrusted playbooks on the DevRunbook application host.

## Regression policy

A new playbook version compares against the previous version on common evaluation cases. Publication UI highlights:

- newly passing cases;
- newly failing cases;
- meaningful prompt diffs;
- new permissions or wider scope;
- changed required inputs;
- validation reductions.

Safety or validation regressions block promotion to Validated.

## User feedback

Feedback fields:

- task was understandable;
- Codex stayed within scope;
- validation was sufficient;
- follow-up prompts were required;
- result solved the intended problem;
- free-form note;
- optional execution evidence.

Feedback is not silently converted into evaluation evidence. It is a separate signal with abuse and privacy controls in future community features.