Publish ITWorx Pulse source

docs/engineering/TEST_STRATEGY.md

@@ -0,0 +1,137 @@
+# Test strategy
+
+## Test pyramid and evidence
+
+Tests prove behavior at the cheapest reliable level, but critical workflows require end-to-end proof.
+
+## 1. Unit tests
+
+Required for:
+- status aggregation;
+- freshness/staleness;
+- metric transformations and units;
+- query limit calculations;
+- inventory identity/reconciliation;
+- dashboard validation/migration/editor reducer;
+- alert state machine, hysteresis and suppression;
+- incident correlation rules;
+- authorization policy;
+- probe target validation/SSRF rules;
+- formatting/localization.
+
+Use deterministic clocks and fixtures.
+
+## 2. Contract tests
+
+For:
+- Prometheus responses/errors/warnings;
+- Unraid API capabilities and payload variants;
+- agent protocol;
+- OIDC claims;
+- notification connectors;
+- the isolated real-stack gate (`scripts/integration-smoke.ps1`) for collector → PostgreSQL → API → UI and alert → webhook delivery;
+- dashboard/live/event schemas.
+
+Captured fixtures must be redacted and versioned.
+
+## 3. Integration tests
+
+Use real PostgreSQL through Testcontainers or equivalent.
+
+Cover:
+- migrations from empty and prior versions;
+- transaction/concurrency;
+- optimistic locking;
+- discovery/reconciliation;
+- alert evaluation/outbox;
+- backup/restore;
+- API authorization;
+- WebSocket persistence/reconnect interactions where practical.
+
+Prometheus and Unraid can use deterministic simulators/fake servers, plus optional non-destructive live contract checks.
+
+## 4. End-to-end browser tests
+
+Playwright core flows:
+- login/session;
+- overview healthy/degraded/unknown;
+- dashboard create/edit/drag/resize/config/save/reload;
+- version restore and conflict;
+- time range and cross-filter;
+- entity drill-down;
+- alert acknowledge/silence;
+- incident view/note;
+- mobile navigation;
+- wallboard reconnect;
+- permission differences;
+- source failure and recovery.
+
+Run desktop and mobile viewports. Add a wallboard viewport.
+
+## 5. Accessibility
+
+Automated checks on every core route/state:
+- desktop and mobile;
+- keyboard flow;
+- focus after modal/drawer/drag alternative;
+- status without color;
+- chart summary/alternative;
+- reduced motion.
+
+Manual spot checks for screen-reader naming and dashboard editor keyboard behavior.
+
+## 6. Performance/load/soak
+
+- API benchmarks and P95 load tests.
+- WebSocket clients/subscriptions/fan-out.
+- Query dedup/cache behavior.
+- 24-hour wallboard heap/subscription/resource soak.
+- 150 container/40 disk/300 probe fixture scale.
+- slow Prometheus/database/agent recovery.
+- frontend rendering with maximum supported widgets.
+
+## 7. Security
+
+- RBAC matrix per endpoint and WebSocket message.
+- OIDC state/nonce/issuer/audience.
+- CSRF/cookie/origin.
+- SSRF, redirect, DNS rebinding, metadata targets.
+- XSS through names/events/import/Markdown.
+- query template and raw query limits.
+- rate/body/message limits.
+- secret scan and diagnostic redaction.
+- dependency/image/static scan.
+- compose mounts/capabilities/network exposure.
+- backup contents.
+
+## 8. Failure simulation
+
+Use `fixtures/scenarios/`:
+- stale Prometheus;
+- source disconnect;
+- CPU saturation;
+- memory pressure/OOM;
+- container restart loop;
+- service down while container runs;
+- disk temperature;
+- SMART warning;
+- cache/pool pressure;
+- array degradation fixture;
+- DNS/gateway failure;
+- UPS on battery;
+- WebSocket slow client/reconnect;
+- database restart.
+
+Never induce destructive real faults.
+
+## 9. Gate behavior
+
+A failing required test:
+- keeps task/milestone incomplete;
+- is diagnosed and repaired;
+- may be quarantined only for a proven external nondeterministic issue, with owner, expiry and alternate evidence;
+- is never simply deleted or skipped.
+
+## 10. Clean-room
+
+Final release is built/deployed from a clean checkout using documented inputs, no developer `.env`, caches or untracked files. Migrations, seed/default dashboards, auth config, health, smoke and restart are verified.