Test strategy

Status: partly implemented. Unit, end-to-end and package smoke tests run in CI. See implementation status.

Every change ships with tests at the right level. The strategy is designed so that a green ci-gate means the packaged app installs, starts, stays within its performance budgets and keeps its security guarantees on Windows, macOS and Linux, and that it updates through NSIS on Windows and AppImage on Linux; release runs add the updater tests for every package format. The decision record is ADR 0022 (test strategy); see the decision record index. Every M1 acceptance criterion is an automated test unless marked otherwise; the M1 scope is described in the roadmap.

Test levels

Level	Tool	Scope	Runs
Unit	Vitest	shared, kernel, domain and module logic with in-memory adapters. Coverage gate: at least 90% of lines in domain and kernel, at least 80% overall. The threshold is real: CI fails below it.	Every pull request, 3 operating systems
Integration	Vitest (Node)	Platform adapters against the real file system and SQLite; migrations with golden profiles; policy providers against real sources (Windows: an HKCU test key; macOS: a plist in /Library/Managed Preferences written with sudo on the runner; Linux: JSON in a temporary directory); the IPC router including property-based fuzzing with fast-check.	Every pull request, 3 operating systems
Component	Vitest Browser Mode (Chromium)	Web components of the shell and the internal pages, accessibility checks with axe-core.	Every pull request
End-to-end	Playwright _electron	Runs against the packaged --dir build, which differs from the shipped build only in the nodeCliInspect fuse. Covers start-up, navigation (local fixture server, no third-party sites), all 4 tab bar positions, drag and drop between windows and workspaces, groups and pins, workspaces with isolated cookies, themes (import, export and live editor preview), settings search, export and policy locks, changing shortcuts, lazy session restore, private windows, downloads, history, bookmarks, switching between English and German, keyboard operation and axe checks.	Every pull request on ubuntu-24.04, windows-2025 and macos-15
Package smoke	Shell scripts and the --smoke-test mode	Every package is installed silently (NSIS /S, msiexec /i … /qn, mounting the dmg and copying the app, AppImage, apt install ./*.deb on Ubuntu 22.04 and 24.04, dnf install on Fedora). The app starts, loads the shell and a local page, writes a JSON report (versions, modules, timings, sandbox status, fuses) and exits with code 0. The package is then uninstalled cleanly.	Every pull request (unsigned) and every release (signed where signing secrets exist), x64 and arm64
Updater	Local test feed (HTTP server on 127.0.0.1); after a release, the live feed and repositories	Two builds, X-test.1 and X-test.2, signed with a test key. The mirror is set through the real policy mechanism, so production builds need no test backdoor. Checks detection, delta download, signature, installation on quit, restart, restored session and migrated data. Negative cases: wrong signature, hash mismatch, downgrade, expired manifest. Rollback: X-test.2 with a deliberate boot failure, after which the boot guard restores X-test.1. deb and rpm: a local APT or DNF repository in a container, with apt upgrade and dnf upgrade.	Every pull request (Linux AppImage and Windows NSIS), every release (all formats)
Performance	Own harness	Budgets and regressions against the main baseline (see performance.md).	Every pull request, 3 operating systems
Security	CodeQL, osv-scanner, dependency-review, zizmor, OpenSSF Scorecard, secret scanning with push protection, own tests	zizmor checks the workflows. Own tests check the webPreferences of every created WebContents, the fuse state of the packaged binary, the Content Security Policy of the app:// pages and the navigation locks of the shell (see security.md).	Every pull request and weekly
Documentation	TypeDoc, Markdown lint, Mermaid render check, lychee, Astro build	TypeDoc runs with treatWarningsAsErrors and validation.notDocumented, so a missing doc comment on a public API is an error. lychee checks links, and the website must build.	Every pull request
i18n	Script	Every key exists in all locales, there are no orphaned keys, and ICU message arguments match across locales.	Every pull request

Module tests live in each module’s __tests__/ folder. The other suites will live in app/tests/: e2e/, perf/ (including the bare Electron reference), smoke/, updater/, screenshots/ and fixtures/ (local sites and golden profiles).

Special cases

• Sandbox tests. The AppArmor and user-namespace tests will run on the host runner ubuntu-24.04 with sysctl kernel.apparmor_restrict_unprivileged_userns=1, not in a container. The expected result: deb and rpm packages run with the Chromium sandbox, and the AppImage shows its fix page. Containers are used only for installation, dependencies and upgrade order.
• Drag and drop between windows. Playwright mouse actions are bound to a single page (as of October 2026). The hit-test and drag-session logic is therefore covered by unit tests, and the end-to-end tests drive the drag across two windows with webContents.sendInputEvent.
• Documentation coverage. TypeDoc will run with entryPointStrategy: expand over app/src, so every public class and method is checked for a doc comment.
• One fuse apart. Playwright needs the nodeCliInspect fuse enabled (as of October 2026), while the shipped build disables it. The end-to-end tests therefore run on the test build, and the shipped package is checked through its built-in self-test (--smoke-test) and by reading its fuses.
• Live update proof. After every release, the previous release must update itself to the new one on a real installation, through the live feed and the live package repositories. Its packages come from its release assets and are checked against its SHA256SUMS before they run. That check only catches damaged downloads; the assets of a published release cannot change. It runs in its normal configuration, without policies: it reads the feed named in brand.json, follows the channel of its version and checks 30 to 60 seconds after the start. The harness only adds a fresh profile and the self-test report (--smoke-test), which do not touch the update path. The previous release is the newest earlier release of the same channel. The first stable release has none and starts from the newest beta, because betas follow the beta channel, which also receives stable releases.
  • It runs in release.yml after the pages job rather than in pages.yml: it belongs to a release, whose version and predecessor the release plan knows, and it must start only after the deployment and verify-live have confirmed the new feed. Weekly runs and deployments without a release do not repeat it, which leaves a gap: they rebuild the feed and the package repositories from the same, immutable releases, but with the feed and repository tools of their own commit. Until the next release, only verify-live checks that output: its signatures, downloads and fresh installs from the repositories, but no installed release that reads the feed.
  • The harness recognizes a downloaded update, a failed check and a disabled updater by the log messages of the installed release, and it starts that release in its self-test mode. Both must therefore stay compatible from one release to the next.
  • macOS is left out until the macOS updater test exists. The validation run proved Squirrel.Mac updates with the self-signed certificate only on a runner that trusts that certificate for code signing, and release packages carry that signature only when the release environment holds it, so a live proof would be the first test of that path instead of the last.

Manual checks per minor release

A few checks need people. Maintainers perform them once per minor release, not per patch, and record the result as described in WORKFLOW.md. They include the screen reader smoke test with NVDA, VoiceOver and Orca, the import of the ADMX templates into Microsoft Intune, and the design review of the generated screenshot gallery. The complete list is kept in WORKFLOW.md only.

Rules

• No skipping. Tests are never skipped or disabled, and there is no quarantine.
• Zero retries. Playwright runs with 0 retries in CI. A flaky test is fixed immediately and takes priority over other work. The one automatic rerun that the performance harness allows before a budget check fails is part of its measurement method, not a test retry (see performance.md).
• Own fixtures. Tests use their own test pages, served locally: deterministic and free of third-party brands. The test package carries content blocker data built from the test filters in app/tests/e2e/fixtures/blocker/, never from the real lists, and the browser starts with --no-first-run, so the setup guide opens only in the tests of the setup guide.
• Golden profiles from the release preparation step. The golden profiles for migration tests will be created by pnpm release:prepare in the release pull request that a maintainer authors, not by a workflow (see data.md).

Implementation status

Running in CI on every pull request today:

• Unit: Vitest on Ubuntu, Windows and macOS with the coverage gates above.
• i18n: a unit test instead of a script checks the catalogs of the UI and of the browser process: every key in every language, no orphaned keys, no empty messages, the same placeholders as in English and the same languages in both.
• End-to-end: Playwright against the packaged --dir test build on the three runners. The harness starts the browser itself, waits for the self-test report that the shell has rendered, and then connects: over the DevTools protocol to the pages and through the Node.js inspector to the main process. Connecting only after the shell is ready avoids a race in _electron.launch, which can miss the commit of an internal page that loads before it has attached. It covers start-up and the Content Security Policy of the shell, navigation and history, the keyboard shortcuts of the running system in the shell and in pages, the back and forward buttons of a mouse, the isolation of web content from internal pages and APIs, default-denied permissions, tabs and popups that keep their opener, new windows that keep their place, private windows that keep nothing, and the context menus in English and German, which the tests read and use from the main process instead of showing them.
• Package smoke: NSIS and MSI (Windows x64, on separate runners), NSIS (Windows arm64), dmg (macOS arm64 and x64), AppImage and deb (Linux x64 and arm64) and rpm (Fedora container). Each package is installed silently, its fuses are read, the self-test runs, and NSIS, MSI and deb are uninstalled again. On a host that restricts user namespaces, the AppImage without its AppArmor profile must open the sandbox help page; with the profile, it must run sandboxed. Installers and uninstallers that do not finish within five minutes fail the job, and a failed MSI installation prints the end of its log. After each self-test, the browser and every process it started are stopped, and the browser’s output reaches the job through pipes, so a leftover helper process cannot keep the step running. The tests write their output to a file; a watchdog ends them after 10 minutes, and on Windows the step then lists every process started meanwhile, with parent and command line, and stops what the tests left behind (tools/package/leftovers.ps1). The step ends well before the job’s own time limit, because GitHub keeps no log of a job that reaches it.
• Updater: an update from build 1 to build 2 through a local signed feed on Windows (NSIS) and Linux (AppImage), after build 1 refused a tampered feed, and the rollback from a build 3 that fails to start (see updates.md).
• Package repositories: installation, upgrade order (~beta before the release) and removal through the APT and DNF repositories in containers of six distributions.
• Performance: the harness checks the first budgets against the bare reference (see performance.md).
• Documentation: Markdown lint, the strict TypeDoc check of every doc comment in app/src, and the website build, all in pnpm verify.
• Tooling: node --test runs the tests of the repository tools, such as the statistics of the performance harness.
• Security: zizmor and actionlint check the workflows.
• Releases: every release runs the complete CI on the merged commit. After each Pages deployment, verify-live checks the signatures of the live feed and repositories and every download, and installs both channels from the live APT and DNF repositories.
• Live update proof: after the deployment of a release, the live-update jobs download the previous release (tools/release/download.ts), which tools/release/plan.ts chose, and check it against its SHA256SUMS. On Windows (NSIS) and Linux (AppImage), app/tests/updater/run.ts --live installs it and waits until the live index lists the new release; the installed release must then download the new one at its first check and install it when it quits, and the new release must pass its self-test. On Linux, app/tests/linux-repos/live.sh --upgrade installs the deb and rpm files of the previous release on Ubuntu and Fedora, which register the live repository of their channel, and the system must upgrade them to the new release. The first release has no previous one, so the jobs are skipped.

Not implemented yet: integration tests against real policy sources, component and accessibility tests, the remaining updater cases (hash mismatch, downgrade and expiry, which unit tests cover so far, and macOS, also in the live update proof), smoke tests on further distributions and the Mermaid render check. The link check runs in pages.yml and covers the links within the site.