# 15 — Test strategy

## Build-pack contract test

Before application tests, run `python3 scripts/validate_pack.py`. It validates all nine JSON Schemas, 28 P0 runtime packages, six normative examples, the 72-entry catalog, package inventories, evaluation references, canonical fixture digests, required OpenAPI coverage, reference SQL tables, internal documentation references, secret-like files and 28 golden rendered prompts against the executable reference composer. CI must run the same script without a reduced local variant.

## Test pyramid

### Unit tests

Fast deterministic tests for:

- schema semantic rules;
- canonicalization and digests;
- autonomy range and policy precedence;
- compatibility resolution;
- scope merging;
- prompt block rendering;
- byte-identical golden prompt conformance for all 28 P0 examples;
- linter rules;
- redaction;
- path/archive safety;
- finding rules;
- authorization policies.

### Integration tests

Use a real disposable PostgreSQL instance for:

- migrations;
- immutable version enforcement;
- idempotent imports;
- full-text search and filters;
- profile revisioning;
- run generation transactions;
- job leasing/retry;
- artifact metadata;
- cross-workspace authorization.

Use a controlled fake HTTP Gitea server and optional real-version compatibility environment for:

- pagination;
- permission differences;
- rate limits;
- version/capability detection;
- timeouts and errors;
- redirect/SSRF controls;
- content-size enforcement.

### Contract tests

- OpenAPI schema matches route behavior;
- Playbook Package examples validate against published JSON Schema;
- Run Pack manifest round-trip;
- repository-profile export/import round-trip;
- Gitea adapter normalized contract.

### Browser tests

Critical Playwright flows:

1. browse, search and filter library;
2. open playbook and start composition;
3. create manual repository profile;
4. compose with profile and protected path;
5. resolve linter error;
6. generate immutable run;
7. copy and export Markdown;
8. generate and verify Run Pack;
9. import private playbook draft;
10. configure fake Gitea, import repository snapshot and open recommendation;
11. theme, keyboard navigation and reduced-motion behavior;
12. permission boundary in future multi-user fixture.

### Visual verification

Use stable screenshots for selected high-value states, not every component. Verify:

- command center;
- library in both card and dense modes;
- playbook detail;
- composer at desktop, laptop and narrow widths;
- repository workspace;
- Prompt Lab diff;
- dark and light themes;
- error/degraded states.

Visual snapshots do not replace semantic browser assertions.

## Fixture strategy

### Package fixtures

- minimal valid quick playbook;
- full guided playbook;
- run-pack playbook;
- unknown template variable;
- autonomy range error;
- malicious archive path;
- symlink package;
- secret-like template value;
- duplicate semantic version with changed digest.

### Repository-profile fixtures

- TypeScript monorepo;
- .NET service;
- Python application;
- Docker/Unraid self-hosted app;
- mixed repository with conflicting package managers;
- profile missing test command;
- protected runtime data directory.

### Gitea fixtures

- full permissions;
- limited permissions;
- old/limited capability response;
- rate limited;
- unreachable;
- private HTTP explicitly permitted;
- malicious redirect;
- oversized file;
- repository containing prompt-injection text.

## Security tests

- cross-workspace IDOR attempts;
- token leakage in logs/errors/responses;
- XSS through Markdown, YAML and repository evidence;
- CSRF on state-changing endpoints;
- SSRF to localhost, metadata and DNS rebinding fixtures;
- ZIP slip, Unicode path tricks, Windows reserved names and symlink escape;
- decompression bomb limits;
- template injection and unsafe helper access;
- secrets redaction false negatives on representative patterns;
- session revocation.

## Performance tests

Dataset:

- 10,000 playbook versions;
- 1,000 playbook identities;
- 500 repository profiles;
- 50,000 generated runs for history pagination;
- realistic tags and full-text distributions.

Measure:

- search P50/P95/P99;
- composer preview latency;
- final run transaction latency;
- Run Pack generation;
- Gitea synchronization under pagination;
- worker throughput and job starvation;
- memory use during malicious archive rejection.

## Clean-room test

From a clean machine or disposable VM/container environment:

1. clone release tag;
2. copy documented environment template;
3. launch PostgreSQL and application;
4. apply migrations;
5. confirm built-in playbooks import;
6. create first user;
7. complete a manual-profile composition and export;
8. restart all containers;
9. confirm data and artifacts persist;
10. back up, delete deployment state and restore;
11. repeat core flow.

## Release gate

Mandatory:

- format, lint and typecheck;
- unit and integration suites;
- schema/example and golden-render validation;
- production build;
- critical Playwright suite;
- security scanner and secret scan;
- migration test;
- container health test;
- no unexplained skipped critical test;
- `CURRENT_STATE.md` and release notes updated.