ITWorx-Pulse-Public/docs/engineering/TEST_STRATEGY.md

Test strategy

Test pyramid and evidence

Tests prove behavior at the cheapest reliable level, but critical workflows require end-to-end proof.

1. Unit tests

Required for:

• status aggregation;
• freshness/staleness;
• metric transformations and units;
• query limit calculations;
• inventory identity/reconciliation;
• dashboard validation/migration/editor reducer;
• alert state machine, hysteresis and suppression;
• incident correlation rules;
• authorization policy;
• probe target validation/SSRF rules;
• formatting/localization.

Use deterministic clocks and fixtures.

2. Contract tests

For:

• Prometheus responses/errors/warnings;
• Unraid API capabilities and payload variants;
• agent protocol;
• OIDC claims;
• notification connectors;
• the isolated real-stack gate (scripts/integration-smoke.ps1) for collector → PostgreSQL → API → UI and alert → webhook delivery;
• dashboard/live/event schemas.

Captured fixtures must be redacted and versioned.

3. Integration tests

Use real PostgreSQL through Testcontainers or equivalent.

Cover:

• migrations from empty and prior versions;
• transaction/concurrency;
• optimistic locking;
• discovery/reconciliation;
• alert evaluation/outbox;
• backup/restore;
• API authorization;
• WebSocket persistence/reconnect interactions where practical.

Prometheus and Unraid can use deterministic simulators/fake servers, plus optional non-destructive live contract checks.

4. End-to-end browser tests

Playwright core flows:

• login/session;
• overview healthy/degraded/unknown;
• dashboard create/edit/drag/resize/config/save/reload;
• version restore and conflict;
• time range and cross-filter;
• entity drill-down;
• alert acknowledge/silence;
• incident view/note;
• mobile navigation;
• wallboard reconnect;
• permission differences;
• source failure and recovery.

Run desktop and mobile viewports. Add a wallboard viewport.

5. Accessibility

Automated checks on every core route/state:

• desktop and mobile;
• keyboard flow;
• focus after modal/drawer/drag alternative;
• status without color;
• chart summary/alternative;
• reduced motion.

Manual spot checks for screen-reader naming and dashboard editor keyboard behavior.

6. Performance/load/soak

• API benchmarks and P95 load tests.
• WebSocket clients/subscriptions/fan-out.
• Query dedup/cache behavior.
• 24-hour wallboard heap/subscription/resource soak.
• 150 container/40 disk/300 probe fixture scale.
• slow Prometheus/database/agent recovery.
• frontend rendering with maximum supported widgets.

7. Security

• RBAC matrix per endpoint and WebSocket message.
• OIDC state/nonce/issuer/audience.
• CSRF/cookie/origin.
• SSRF, redirect, DNS rebinding, metadata targets.
• XSS through names/events/import/Markdown.
• query template and raw query limits.
• rate/body/message limits.
• secret scan and diagnostic redaction.
• dependency/image/static scan.
• compose mounts/capabilities/network exposure.
• backup contents.

8. Failure simulation

Use fixtures/scenarios/:

• stale Prometheus;
• source disconnect;
• CPU saturation;
• memory pressure/OOM;
• container restart loop;
• service down while container runs;
• disk temperature;
• SMART warning;
• cache/pool pressure;
• array degradation fixture;
• DNS/gateway failure;
• UPS on battery;
• WebSocket slow client/reconnect;
• database restart.

Never induce destructive real faults.

9. Gate behavior

A failing required test:

• keeps task/milestone incomplete;
• is diagnosed and repaired;
• may be quarantined only for a proven external nondeterministic issue, with owner, expiry and alternate evidence;
• is never simply deleted or skipped.

10. Clean-room

Final release is built/deployed from a clean checkout using documented inputs, no developer .env, caches or untracked files. Migrations, seed/default dashboards, auth config, health, smoke and restart are verified.