174 lines
5.3 KiB
Markdown
174 lines
5.3 KiB
Markdown
# 12 — Quality, linting and evaluation
|
|
|
|
## Quality philosophy
|
|
|
|
A playbook is trustworthy when its structure, safety, clarity and observed behavior are evidenced. Popularity, length and confident wording are not quality proof.
|
|
|
|
## Quality dimensions
|
|
|
|
### Scope clarity
|
|
|
|
Does the playbook define the target, permitted reading scope, modification scope, exclusions and protected behavior?
|
|
|
|
### Safety
|
|
|
|
Does it prevent destructive, secret-exposing or policy-weakening behavior appropriate to its risk?
|
|
|
|
### Verification
|
|
|
|
Does it require relevant checks and evidence rather than a narrative claim?
|
|
|
|
### Reproducibility
|
|
|
|
Can the same package version and normalized inputs reproduce the same output?
|
|
|
|
### Compatibility
|
|
|
|
Are supported stacks, required profile capabilities and limitations explicit?
|
|
|
|
### Reporting
|
|
|
|
Does the final report make outcome, evidence, risks and unresolved items reviewable?
|
|
|
|
### Efficiency
|
|
|
|
Does the task avoid unnecessary repository-wide change, redundant investigation and repeated validation without sacrificing safety?
|
|
|
|
Scores are displayed separately on a defined scale, such as Not assessed, Weak, Adequate, Strong. A combined score may be calculated for sorting but must never replace the dimensions.
|
|
|
|
## Lifecycle policy
|
|
|
|
### Draft
|
|
|
|
- schema-valid or actively being edited;
|
|
- may have lint errors;
|
|
- not recommended outside author workspace.
|
|
|
|
### Reviewed
|
|
|
|
- schema and semantic validation pass;
|
|
- no blocking prompt-lint findings on required examples;
|
|
- human editorial review complete;
|
|
- limitations documented.
|
|
|
|
### Validated
|
|
|
|
- Reviewed requirements;
|
|
- required evaluation cases pass;
|
|
- evaluation environment and fixture version recorded;
|
|
- no unresolved safety regression.
|
|
|
|
### Battle-tested
|
|
|
|
- Validated requirements;
|
|
- minimum real-world run count under policy;
|
|
- acceptable operator feedback and failure rate;
|
|
- no unaddressed severe incident;
|
|
- evidence remains recent enough for the playbook class.
|
|
|
|
### Deprecated
|
|
|
|
- replacement or rationale provided;
|
|
- historical rendering remains available;
|
|
- excluded from default recommendations.
|
|
|
|
## Static linter catalog
|
|
|
|
Suggested rule IDs:
|
|
|
|
- `PB001` missing mission
|
|
- `PB002` missing explicit scope
|
|
- `PB003` missing done-when criteria
|
|
- `PB004` missing reporting contract
|
|
- `PB005` duplicate input or step ID
|
|
- `PB006` invalid autonomy range
|
|
- `PB007` unknown template variable
|
|
- `PB008` published version without changelog
|
|
- `PB009` validated status without evidence
|
|
- `PR001` ambiguous unbounded improvement language
|
|
- `PR002` conflicting read-only and modification instruction
|
|
- `PR003` vague “best practices” without dimensions
|
|
- `PR004` asks for success claim without evidence
|
|
- `SA001` token-like value in rendered output
|
|
- `SA002` protected path in change scope
|
|
- `SA003` destructive migration without rollback/backup
|
|
- `SA004` Git push/release not explicitly authorized
|
|
- `SA005` imported content placed in policy section
|
|
- `VA001` bugfix lacks reproduction/regression step
|
|
- `VA002` implementation lacks available build/test validation
|
|
- `VA003` dependency change lacks lockfile/install/build check
|
|
- `VA004` frontend flow lacks browser verification
|
|
- `VA005` inspect playbook lacks evidence-source reporting
|
|
|
|
Every finding includes severity, location, message, rationale, remediation hint and provenance.
|
|
|
|
## Evaluation case format
|
|
|
|
An evaluation case defines:
|
|
|
|
- case ID and version;
|
|
- target playbook/version range;
|
|
- fixture repository reference and digest;
|
|
- repository profile;
|
|
- inputs and autonomy;
|
|
- expected prompt properties;
|
|
- prohibited prompt properties;
|
|
- optional future execution expectations;
|
|
- scoring rubric.
|
|
|
|
MVP can evaluate rendering and lint behavior without executing Codex. Future isolated evaluation can run tasks in disposable fixture environments.
|
|
|
|
## Static evaluation examples
|
|
|
|
- generated prompt contains all required headings;
|
|
- no secret fixture value appears;
|
|
- protected paths are rendered as no-change constraints;
|
|
- selected `observe` autonomy contains no implementation permission;
|
|
- missing test command creates warning rather than invented command;
|
|
- stack incompatibility blocks export;
|
|
- same input produces same digest;
|
|
- a conditional migration section appears only when migration input is true.
|
|
|
|
## Future execution evaluation
|
|
|
|
Fixture repositories intentionally contain known problems. Evaluation runner captures:
|
|
|
|
- task completion status;
|
|
- changed file set;
|
|
- protected-path violations;
|
|
- command exit codes;
|
|
- tests added or changed;
|
|
- artifact diffs;
|
|
- final report completeness;
|
|
- token/time/cost metadata where available;
|
|
- human review.
|
|
|
|
The runner must use isolated disposable environments and must never execute untrusted playbooks on the DevRunbook application host.
|
|
|
|
## Regression policy
|
|
|
|
A new playbook version compares against the previous version on common evaluation cases. Publication UI highlights:
|
|
|
|
- newly passing cases;
|
|
- newly failing cases;
|
|
- meaningful prompt diffs;
|
|
- new permissions or wider scope;
|
|
- changed required inputs;
|
|
- validation reductions.
|
|
|
|
Safety or validation regressions block promotion to Validated.
|
|
|
|
## User feedback
|
|
|
|
Feedback fields:
|
|
|
|
- task was understandable;
|
|
- Codex stayed within scope;
|
|
- validation was sufficient;
|
|
- follow-up prompts were required;
|
|
- result solved the intended problem;
|
|
- free-form note;
|
|
- optional execution evidence.
|
|
|
|
Feedback is not silently converted into evaluation evidence. It is a separate signal with abuse and privacy controls in future community features.
|