Files
ForgeFlow/docs/TEST_MATRIX.md
T
NuklearRabbit aa4895912a
ForgeFlow quality gate / quality (push) Canceled after 0s
docs: record production readiness evidence
2026-07-29 20:06:14 +02:00

6.4 KiB
Raw Blame History

Test matrix

Automated baseline (0.10.x)

The quality chain contains more than 230 Node and browser acceptance cases. The latest Windows source run completed without failures and retains one explicitly Bash-dependent skip. npm run coverage enforces 75% lines/statements/functions and 60% branches; the measured hardening baseline is 78.75% statements/lines, 79.68% functions and 59.25% branches before the final ConfigStore additions. See COVERAGE_POLICY.md for the non-gamed branch policy.

npm run quality is the local equivalent of .gitea/workflows/quality.yml and runs source verification, ESLint, the complete suite and coverage on Node 22 LTS. Production dependencies are separately checked with npm audit --omit=dev --audit-level=high.

Server safety and reconciliation

  • inventory discovery is read-only and byte-stable for configuration;
  • reconciliation requires a content-addressed preview plan and recovery snapshot;
  • automatic linking requires unique exact provenance/runtime identity;
  • server-pull verification checks Gitea branch, read-only deploy-key ID, pinned host/key fingerprints, remote/live SHA, Compose evidence, runtime and health;
  • a fresh access verification is mandatory immediately before server-pull deploy;
  • writable or missing deploy keys fail closed.

Renderer regression matrix

Playwright runs 36 cases across 1120×720, 1440×900 and 1920×1080, dark and light themes, reduced motion, and simulated 100%, 125% and 150% Windows scaling. It checks console/page errors, accessible names, labels, heading structure, horizontal overflow, viewport containment, dialogs, keyboard focus, updater and deployment failure evidence. CI retains screenshots, video, trace, console JSON, DOM HTML and fixture context on failure.

Git and repository behavior

  • porcelain v2 ordinary and rename parsing;
  • HTTPS and SCP-style remote matching;
  • real temporary bare remote: status, diff, selected commit and push;
  • real temporary bare remote: commit-only, branch creation/publication and remote-SHA ancestry verification;
  • real stash creation, listing, pop and untracked-file restoration;
  • repository monitor baseline, change detection and pause/resume;
  • safe repository folder-name derivation from HTTPS and SSH clone URLs;
  • automatic target construction beneath the project root;
  • missing, empty and matching-checkout clone target handling;
  • different repository, ordinary non-empty folder and file conflict rejection.

Gitea and deployment behavior

  • Gitea URL/credential validation;
  • Actions run normalization across payload shapes;
  • optional query-filter compatibility retry;
  • runs-to-tasks fallback;
  • newest matching run selection;
  • repository workflow contents lookup and 404 behavior;
  • deployment terminal-status mapping;
  • controlled dispatch inputs that cannot be overridden by profile data;
  • exact post-workflow SHA and request-ID verification;
  • rollback input allowlisting and exact current previous-SHA enforcement;
  • complete deployment preflight with Git, workflow, Actions, status and health mocks.

Security and diagnostics

  • repository path traversal and absolute-path rejection;
  • workflow filename, branch, environment and full-SHA validation;
  • clone protocol and embedded-password rejection;
  • runtime token, authorization, query token, URL credential and private-key redaction;
  • camelCase and nested sensitive-key removal;
  • home-path aliasing;
  • deterministic strict-privacy identifier hashing;
  • required versus optional preflight blocking behavior;
  • system preflight before credentials are entered;
  • structured JSONL diagnostic writes;
  • support-bundle strict privacy and secret exclusion;
  • ZIP structure, deflate payloads and CRC validation.
  • Windows npm command-shim discovery through npm_execpath and cmd.exe;
  • normal direct npm discovery on non-Windows systems.

Static source quality gate

npm run verify checks:

  • all required source, documentation and server-template files;
  • JavaScript syntax across the project;
  • package version and required scripts;
  • desktop packaging metadata and icons;
  • Bash syntax for the server entry point;
  • status JSON parsing;
  • required setup-guide sections;
  • renderer entry hooks.

Manual before a real production release

  • setup wizard against the installed Gitea version;
  • repository discovery on the target Windows system;
  • token persistence through Windows credential protection;
  • HTTPS and/or SSH Git authentication;
  • actual Actions dispatch, run resolution and job visibility;
  • runner label and repository trust scope;
  • server target-file ownership/mode enforcement;
  • status endpoint through the real reverse proxy;
  • deployment lock, failed healthcheck and rollback;
  • diagnostic ZIP inspection after a deliberately failed deployment;
  • locally test-signed installer, portable, helper and uninstaller fixtures with RFC 3161 timestamp plus wrong-publisher, missing-timestamp and tamper rejection;
  • keyboard-only and screen-reader smoke test.

Renderer smoke target

The standalone demo should be checked at minimum at:

  • 1120 × 720;
  • 1440 × 900;
  • 1920 × 1080.

Required views now include setup readiness, dashboard, repository workspace, deployment preflight, active run, success/failure and Diagnostics.

v0.8 functional acceptance

  • real-repository partial hunk staging without staging the remaining changes;
  • guided merge-conflict resolution and safe continue/abort actions;
  • Gitea pull-request creation and protected-branch inspection;
  • shell-free editor and terminal argument-template expansion;
  • deployment freezes, maintenance windows, mandatory release notes and reasoned overrides;
  • authenticated encrypted configuration backup without credentials or operation history;
  • append-only audit JSONL and CSV export;
  • desktop notification, tray and close-to-tray preference integration;
  • read-only-by-default end-to-end Gitea Actions acceptance harness with explicit deploy/rollback flags;
  • interactive demo verification for repository quick actions, hunk staging and pull-request dialogs.

v0.4 additions

  • bounded independently scrollable changed-file layout;
  • explicit commit-message and selection readiness contract;
  • ITWorx.tech asset integration;
  • semantic update-version comparison;
  • exact-SHA Gitea update manifest lookup;
  • update repository path-injection rejection;
  • SSH host-key fingerprint helper;
  • remote shell quoting;
  • Unraid folder and Compose path escape rejection;
  • server inspection payload decoding;
  • SSH deployment preflight summary behavior.