189 lines
5.1 KiB
Markdown
189 lines
5.1 KiB
Markdown
# 15 — Test strategy
|
|
|
|
## Build-pack contract test
|
|
|
|
Before application tests, run `python3 scripts/validate_pack.py`. It validates all nine JSON Schemas, 28 P0 runtime packages, six normative examples, the 72-entry catalog, package inventories, evaluation references, canonical fixture digests, required OpenAPI coverage, reference SQL tables, internal documentation references, secret-like files and 28 golden rendered prompts against the executable reference composer. CI must run the same script without a reduced local variant.
|
|
|
|
## Test pyramid
|
|
|
|
### Unit tests
|
|
|
|
Fast deterministic tests for:
|
|
|
|
- schema semantic rules;
|
|
- canonicalization and digests;
|
|
- autonomy range and policy precedence;
|
|
- compatibility resolution;
|
|
- scope merging;
|
|
- prompt block rendering;
|
|
- byte-identical golden prompt conformance for all 28 P0 examples;
|
|
- linter rules;
|
|
- redaction;
|
|
- path/archive safety;
|
|
- finding rules;
|
|
- authorization policies.
|
|
|
|
### Integration tests
|
|
|
|
Use a real disposable PostgreSQL instance for:
|
|
|
|
- migrations;
|
|
- immutable version enforcement;
|
|
- idempotent imports;
|
|
- full-text search and filters;
|
|
- profile revisioning;
|
|
- run generation transactions;
|
|
- job leasing/retry;
|
|
- artifact metadata;
|
|
- cross-workspace authorization.
|
|
|
|
Use a controlled fake HTTP Gitea server and optional real-version compatibility environment for:
|
|
|
|
- pagination;
|
|
- permission differences;
|
|
- rate limits;
|
|
- version/capability detection;
|
|
- timeouts and errors;
|
|
- redirect/SSRF controls;
|
|
- content-size enforcement.
|
|
|
|
### Contract tests
|
|
|
|
- OpenAPI schema matches route behavior;
|
|
- Playbook Package examples validate against published JSON Schema;
|
|
- Run Pack manifest round-trip;
|
|
- repository-profile export/import round-trip;
|
|
- Gitea adapter normalized contract.
|
|
|
|
### Browser tests
|
|
|
|
Critical Playwright flows:
|
|
|
|
1. browse, search and filter library;
|
|
2. open playbook and start composition;
|
|
3. create manual repository profile;
|
|
4. compose with profile and protected path;
|
|
5. resolve linter error;
|
|
6. generate immutable run;
|
|
7. copy and export Markdown;
|
|
8. generate and verify Run Pack;
|
|
9. import private playbook draft;
|
|
10. configure fake Gitea, import repository snapshot and open recommendation;
|
|
11. theme, keyboard navigation and reduced-motion behavior;
|
|
12. permission boundary in future multi-user fixture.
|
|
|
|
### Visual verification
|
|
|
|
Use stable screenshots for selected high-value states, not every component. Verify:
|
|
|
|
- command center;
|
|
- library in both card and dense modes;
|
|
- playbook detail;
|
|
- composer at desktop, laptop and narrow widths;
|
|
- repository workspace;
|
|
- Prompt Lab diff;
|
|
- dark and light themes;
|
|
- error/degraded states.
|
|
|
|
Visual snapshots do not replace semantic browser assertions.
|
|
|
|
## Fixture strategy
|
|
|
|
### Package fixtures
|
|
|
|
- minimal valid quick playbook;
|
|
- full guided playbook;
|
|
- run-pack playbook;
|
|
- unknown template variable;
|
|
- autonomy range error;
|
|
- malicious archive path;
|
|
- symlink package;
|
|
- secret-like template value;
|
|
- duplicate semantic version with changed digest.
|
|
|
|
### Repository-profile fixtures
|
|
|
|
- TypeScript monorepo;
|
|
- .NET service;
|
|
- Python application;
|
|
- Docker/Unraid self-hosted app;
|
|
- mixed repository with conflicting package managers;
|
|
- profile missing test command;
|
|
- protected runtime data directory.
|
|
|
|
### Gitea fixtures
|
|
|
|
- full permissions;
|
|
- limited permissions;
|
|
- old/limited capability response;
|
|
- rate limited;
|
|
- unreachable;
|
|
- private HTTP explicitly permitted;
|
|
- malicious redirect;
|
|
- oversized file;
|
|
- repository containing prompt-injection text.
|
|
|
|
## Security tests
|
|
|
|
- cross-workspace IDOR attempts;
|
|
- token leakage in logs/errors/responses;
|
|
- XSS through Markdown, YAML and repository evidence;
|
|
- CSRF on state-changing endpoints;
|
|
- SSRF to localhost, metadata and DNS rebinding fixtures;
|
|
- ZIP slip, Unicode path tricks, Windows reserved names and symlink escape;
|
|
- decompression bomb limits;
|
|
- template injection and unsafe helper access;
|
|
- secrets redaction false negatives on representative patterns;
|
|
- session revocation.
|
|
|
|
## Performance tests
|
|
|
|
Dataset:
|
|
|
|
- 10,000 playbook versions;
|
|
- 1,000 playbook identities;
|
|
- 500 repository profiles;
|
|
- 50,000 generated runs for history pagination;
|
|
- realistic tags and full-text distributions.
|
|
|
|
Measure:
|
|
|
|
- search P50/P95/P99;
|
|
- composer preview latency;
|
|
- final run transaction latency;
|
|
- Run Pack generation;
|
|
- Gitea synchronization under pagination;
|
|
- worker throughput and job starvation;
|
|
- memory use during malicious archive rejection.
|
|
|
|
## Clean-room test
|
|
|
|
From a clean machine or disposable VM/container environment:
|
|
|
|
1. clone release tag;
|
|
2. copy documented environment template;
|
|
3. launch PostgreSQL and application;
|
|
4. apply migrations;
|
|
5. confirm built-in playbooks import;
|
|
6. create first user;
|
|
7. complete a manual-profile composition and export;
|
|
8. restart all containers;
|
|
9. confirm data and artifacts persist;
|
|
10. back up, delete deployment state and restore;
|
|
11. repeat core flow.
|
|
|
|
## Release gate
|
|
|
|
Mandatory:
|
|
|
|
- format, lint and typecheck;
|
|
- unit and integration suites;
|
|
- schema/example and golden-render validation;
|
|
- production build;
|
|
- critical Playwright suite;
|
|
- security scanner and secret scan;
|
|
- migration test;
|
|
- container health test;
|
|
- no unexplained skipped critical test;
|
|
- `CURRENT_STATE.md` and release notes updated.
|