ITWorx Pulse release export
bd774932d5
Public source validation / validate (push) Failing after 3m8s
138 lines
3.8 KiB
Markdown
138 lines
3.8 KiB
Markdown
# Test strategy
|
|
|
|
## Test pyramid and evidence
|
|
|
|
Tests prove behavior at the cheapest reliable level, but critical workflows require end-to-end proof.
|
|
|
|
## 1. Unit tests
|
|
|
|
Required for:
|
|
- status aggregation;
|
|
- freshness/staleness;
|
|
- metric transformations and units;
|
|
- query limit calculations;
|
|
- inventory identity/reconciliation;
|
|
- dashboard validation/migration/editor reducer;
|
|
- alert state machine, hysteresis and suppression;
|
|
- incident correlation rules;
|
|
- authorization policy;
|
|
- probe target validation/SSRF rules;
|
|
- formatting/localization.
|
|
|
|
Use deterministic clocks and fixtures.
|
|
|
|
## 2. Contract tests
|
|
|
|
For:
|
|
- Prometheus responses/errors/warnings;
|
|
- Unraid API capabilities and payload variants;
|
|
- agent protocol;
|
|
- OIDC claims;
|
|
- notification connectors;
|
|
- the isolated real-stack gate (`scripts/integration-smoke.ps1`) for collector → PostgreSQL → API → UI and alert → webhook delivery;
|
|
- dashboard/live/event schemas.
|
|
|
|
Captured fixtures must be redacted and versioned.
|
|
|
|
## 3. Integration tests
|
|
|
|
Use real PostgreSQL through Testcontainers or equivalent.
|
|
|
|
Cover:
|
|
- migrations from empty and prior versions;
|
|
- transaction/concurrency;
|
|
- optimistic locking;
|
|
- discovery/reconciliation;
|
|
- alert evaluation/outbox;
|
|
- backup/restore;
|
|
- API authorization;
|
|
- WebSocket persistence/reconnect interactions where practical.
|
|
|
|
Prometheus and Unraid can use deterministic simulators/fake servers, plus optional non-destructive live contract checks.
|
|
|
|
## 4. End-to-end browser tests
|
|
|
|
Playwright core flows:
|
|
- login/session;
|
|
- overview healthy/degraded/unknown;
|
|
- dashboard create/edit/drag/resize/config/save/reload;
|
|
- version restore and conflict;
|
|
- time range and cross-filter;
|
|
- entity drill-down;
|
|
- alert acknowledge/silence;
|
|
- incident view/note;
|
|
- mobile navigation;
|
|
- wallboard reconnect;
|
|
- permission differences;
|
|
- source failure and recovery.
|
|
|
|
Run desktop and mobile viewports. Add a wallboard viewport.
|
|
|
|
## 5. Accessibility
|
|
|
|
Automated checks on every core route/state:
|
|
- desktop and mobile;
|
|
- keyboard flow;
|
|
- focus after modal/drawer/drag alternative;
|
|
- status without color;
|
|
- chart summary/alternative;
|
|
- reduced motion.
|
|
|
|
Manual spot checks for screen-reader naming and dashboard editor keyboard behavior.
|
|
|
|
## 6. Performance/load/soak
|
|
|
|
- API benchmarks and P95 load tests.
|
|
- WebSocket clients/subscriptions/fan-out.
|
|
- Query dedup/cache behavior.
|
|
- 24-hour wallboard heap/subscription/resource soak.
|
|
- 150 container/40 disk/300 probe fixture scale.
|
|
- slow Prometheus/database/agent recovery.
|
|
- frontend rendering with maximum supported widgets.
|
|
|
|
## 7. Security
|
|
|
|
- RBAC matrix per endpoint and WebSocket message.
|
|
- OIDC state/nonce/issuer/audience.
|
|
- CSRF/cookie/origin.
|
|
- SSRF, redirect, DNS rebinding, metadata targets.
|
|
- XSS through names/events/import/Markdown.
|
|
- query template and raw query limits.
|
|
- rate/body/message limits.
|
|
- secret scan and diagnostic redaction.
|
|
- dependency/image/static scan.
|
|
- compose mounts/capabilities/network exposure.
|
|
- backup contents.
|
|
|
|
## 8. Failure simulation
|
|
|
|
Use `fixtures/scenarios/`:
|
|
- stale Prometheus;
|
|
- source disconnect;
|
|
- CPU saturation;
|
|
- memory pressure/OOM;
|
|
- container restart loop;
|
|
- service down while container runs;
|
|
- disk temperature;
|
|
- SMART warning;
|
|
- cache/pool pressure;
|
|
- array degradation fixture;
|
|
- DNS/gateway failure;
|
|
- UPS on battery;
|
|
- WebSocket slow client/reconnect;
|
|
- database restart.
|
|
|
|
Never induce destructive real faults.
|
|
|
|
## 9. Gate behavior
|
|
|
|
A failing required test:
|
|
- keeps task/milestone incomplete;
|
|
- is diagnosed and repaired;
|
|
- may be quarantined only for a proven external nondeterministic issue, with owner, expiry and alternate evidence;
|
|
- is never simply deleted or skipped.
|
|
|
|
## 10. Clean-room
|
|
|
|
Final release is built/deployed from a clean checkout using documented inputs, no developer `.env`, caches or untracked files. Migrations, seed/default dashboards, auth config, health, smoke and restart are verified.
|