DevRunbook-Public/docs/15-test-strategy.md

15 — Test strategy

Build-pack contract test

Before application tests, run python3 scripts/validate_pack.py. It validates all nine JSON Schemas, 28 P0 runtime packages, six normative examples, the 72-entry catalog, package inventories, evaluation references, canonical fixture digests, required OpenAPI coverage, reference SQL tables, internal documentation references, secret-like files and 28 golden rendered prompts against the executable reference composer. CI must run the same script without a reduced local variant.

Test pyramid

Unit tests

Fast deterministic tests for:

• schema semantic rules;
• canonicalization and digests;
• autonomy range and policy precedence;
• compatibility resolution;
• scope merging;
• prompt block rendering;
• byte-identical golden prompt conformance for all 28 P0 examples;
• linter rules;
• redaction;
• path/archive safety;
• finding rules;
• authorization policies.

Integration tests

Use a real disposable PostgreSQL instance for:

• migrations;
• immutable version enforcement;
• idempotent imports;
• full-text search and filters;
• profile revisioning;
• run generation transactions;
• job leasing/retry;
• artifact metadata;
• cross-workspace authorization.

Use a controlled fake HTTP Gitea server and optional real-version compatibility environment for:

• pagination;
• permission differences;
• rate limits;
• version/capability detection;
• timeouts and errors;
• redirect/SSRF controls;
• content-size enforcement.

Contract tests

• OpenAPI schema matches route behavior;
• Playbook Package examples validate against published JSON Schema;
• Run Pack manifest round-trip;
• repository-profile export/import round-trip;
• Gitea adapter normalized contract.

Browser tests

Critical Playwright flows:

1. browse, search and filter library;
2. open playbook and start composition;
3. create manual repository profile;
4. compose with profile and protected path;
5. resolve linter error;
6. generate immutable run;
7. copy and export Markdown;
8. generate and verify Run Pack;
9. import private playbook draft;
10. configure fake Gitea, import repository snapshot and open recommendation;
11. theme, keyboard navigation and reduced-motion behavior;
12. permission boundary in future multi-user fixture.

Visual verification

Use stable screenshots for selected high-value states, not every component. Verify:

• command center;
• library in both card and dense modes;
• playbook detail;
• composer at desktop, laptop and narrow widths;
• repository workspace;
• Prompt Lab diff;
• dark and light themes;
• error/degraded states.

Visual snapshots do not replace semantic browser assertions.

Fixture strategy

Package fixtures

• minimal valid quick playbook;
• full guided playbook;
• run-pack playbook;
• unknown template variable;
• autonomy range error;
• malicious archive path;
• symlink package;
• secret-like template value;
• duplicate semantic version with changed digest.

Repository-profile fixtures

• TypeScript monorepo;
• .NET service;
• Python application;
• Docker/Unraid self-hosted app;
• mixed repository with conflicting package managers;
• profile missing test command;
• protected runtime data directory.

Gitea fixtures

• full permissions;
• limited permissions;
• old/limited capability response;
• rate limited;
• unreachable;
• private HTTP explicitly permitted;
• malicious redirect;
• oversized file;
• repository containing prompt-injection text.

Security tests

• cross-workspace IDOR attempts;
• token leakage in logs/errors/responses;
• XSS through Markdown, YAML and repository evidence;
• CSRF on state-changing endpoints;
• SSRF to localhost, metadata and DNS rebinding fixtures;
• ZIP slip, Unicode path tricks, Windows reserved names and symlink escape;
• decompression bomb limits;
• template injection and unsafe helper access;
• secrets redaction false negatives on representative patterns;
• session revocation.

Performance tests

Dataset:

• 10,000 playbook versions;
• 1,000 playbook identities;
• 500 repository profiles;
• 50,000 generated runs for history pagination;
• realistic tags and full-text distributions.

Measure:

• search P50/P95/P99;
• composer preview latency;
• final run transaction latency;
• Run Pack generation;
• Gitea synchronization under pagination;
• worker throughput and job starvation;
• memory use during malicious archive rejection.

Clean-room test

From a clean machine or disposable VM/container environment:

1. clone release tag;
2. copy documented environment template;
3. launch PostgreSQL and application;
4. apply migrations;
5. confirm built-in playbooks import;
6. create first user;
7. complete a manual-profile composition and export;
8. restart all containers;
9. confirm data and artifacts persist;
10. back up, delete deployment state and restore;
11. repeat core flow.

Release gate

Mandatory:

• format, lint and typecheck;
• unit and integration suites;
• schema/example and golden-render validation;
• production build;
• critical Playwright suite;
• security scanner and secret scan;
• migration test;
• container health test;
• no unexplained skipped critical test;
• CURRENT_STATE.md and release notes updated.