Publish DevRunbook source

docs/15-test-strategy.md

@@ -0,0 +1,188 @@
+# 15 — Test strategy
+
+## Build-pack contract test
+
+Before application tests, run `python3 scripts/validate_pack.py`. It validates all nine JSON Schemas, 28 P0 runtime packages, six normative examples, the 72-entry catalog, package inventories, evaluation references, canonical fixture digests, required OpenAPI coverage, reference SQL tables, internal documentation references, secret-like files and 28 golden rendered prompts against the executable reference composer. CI must run the same script without a reduced local variant.
+
+## Test pyramid
+
+### Unit tests
+
+Fast deterministic tests for:
+
+- schema semantic rules;
+- canonicalization and digests;
+- autonomy range and policy precedence;
+- compatibility resolution;
+- scope merging;
+- prompt block rendering;
+- byte-identical golden prompt conformance for all 28 P0 examples;
+- linter rules;
+- redaction;
+- path/archive safety;
+- finding rules;
+- authorization policies.
+
+### Integration tests
+
+Use a real disposable PostgreSQL instance for:
+
+- migrations;
+- immutable version enforcement;
+- idempotent imports;
+- full-text search and filters;
+- profile revisioning;
+- run generation transactions;
+- job leasing/retry;
+- artifact metadata;
+- cross-workspace authorization.
+
+Use a controlled fake HTTP Gitea server and optional real-version compatibility environment for:
+
+- pagination;
+- permission differences;
+- rate limits;
+- version/capability detection;
+- timeouts and errors;
+- redirect/SSRF controls;
+- content-size enforcement.
+
+### Contract tests
+
+- OpenAPI schema matches route behavior;
+- Playbook Package examples validate against published JSON Schema;
+- Run Pack manifest round-trip;
+- repository-profile export/import round-trip;
+- Gitea adapter normalized contract.
+
+### Browser tests
+
+Critical Playwright flows:
+
+1. browse, search and filter library;
+2. open playbook and start composition;
+3. create manual repository profile;
+4. compose with profile and protected path;
+5. resolve linter error;
+6. generate immutable run;
+7. copy and export Markdown;
+8. generate and verify Run Pack;
+9. import private playbook draft;
+10. configure fake Gitea, import repository snapshot and open recommendation;
+11. theme, keyboard navigation and reduced-motion behavior;
+12. permission boundary in future multi-user fixture.
+
+### Visual verification
+
+Use stable screenshots for selected high-value states, not every component. Verify:
+
+- command center;
+- library in both card and dense modes;
+- playbook detail;
+- composer at desktop, laptop and narrow widths;
+- repository workspace;
+- Prompt Lab diff;
+- dark and light themes;
+- error/degraded states.
+
+Visual snapshots do not replace semantic browser assertions.
+
+## Fixture strategy
+
+### Package fixtures
+
+- minimal valid quick playbook;
+- full guided playbook;
+- run-pack playbook;
+- unknown template variable;
+- autonomy range error;
+- malicious archive path;
+- symlink package;
+- secret-like template value;
+- duplicate semantic version with changed digest.
+
+### Repository-profile fixtures
+
+- TypeScript monorepo;
+- .NET service;
+- Python application;
+- Docker/Unraid self-hosted app;
+- mixed repository with conflicting package managers;
+- profile missing test command;
+- protected runtime data directory.
+
+### Gitea fixtures
+
+- full permissions;
+- limited permissions;
+- old/limited capability response;
+- rate limited;
+- unreachable;
+- private HTTP explicitly permitted;
+- malicious redirect;
+- oversized file;
+- repository containing prompt-injection text.
+
+## Security tests
+
+- cross-workspace IDOR attempts;
+- token leakage in logs/errors/responses;
+- XSS through Markdown, YAML and repository evidence;
+- CSRF on state-changing endpoints;
+- SSRF to localhost, metadata and DNS rebinding fixtures;
+- ZIP slip, Unicode path tricks, Windows reserved names and symlink escape;
+- decompression bomb limits;
+- template injection and unsafe helper access;
+- secrets redaction false negatives on representative patterns;
+- session revocation.
+
+## Performance tests
+
+Dataset:
+
+- 10,000 playbook versions;
+- 1,000 playbook identities;
+- 500 repository profiles;
+- 50,000 generated runs for history pagination;
+- realistic tags and full-text distributions.
+
+Measure:
+
+- search P50/P95/P99;
+- composer preview latency;
+- final run transaction latency;
+- Run Pack generation;
+- Gitea synchronization under pagination;
+- worker throughput and job starvation;
+- memory use during malicious archive rejection.
+
+## Clean-room test
+
+From a clean machine or disposable VM/container environment:
+
+1. clone release tag;
+2. copy documented environment template;
+3. launch PostgreSQL and application;
+4. apply migrations;
+5. confirm built-in playbooks import;
+6. create first user;
+7. complete a manual-profile composition and export;
+8. restart all containers;
+9. confirm data and artifacts persist;
+10. back up, delete deployment state and restore;
+11. repeat core flow.
+
+## Release gate
+
+Mandatory:
+
+- format, lint and typecheck;
+- unit and integration suites;
+- schema/example and golden-render validation;
+- production build;
+- critical Playwright suite;
+- security scanner and secret scan;
+- migration test;
+- container health test;
+- no unexplained skipped critical test;
+- `CURRENT_STATE.md` and release notes updated.