#Run tests

#Basic usage

Run YAML test cases from the project root:

pnpm exec midscene-test

#Specify a test case directory or file

With a config, a file or directory argument filters the selected cases. Full-suite, single-file, and subdirectory runs share the same project configuration.

Run these commands from the project root:

pnpm exec midscene-test
pnpm exec midscene-test ./cases/order.yaml
pnpm exec midscene-test ./cases/smoke

They share the configured Nodes, Execution Projects, and default output location.

Configuration and path resolution

Default discovery checks cwd only. When running from a child directory, select the project config explicitly:

cd cases
pnpm exec midscene-test ./order.yaml --config ../midscene.config.ts

The parent directory containing the config becomes the test root. Default output remains in that directory's midscene_run.

Without a config, a file argument collects the selected YAML, and a directory argument discovers YAML files in that directory. Paths outside cwd are supported. With no argument, discovery starts at cwd. Configs beside the selected input are not loaded. Default artifacts stay under cwd/midscene_run.

Without a config, no Nodes are registered. YAML that references Nodes still needs a config that registers them. Legacy tasks/flow YAML is run with midscene, not midscene-test.

File and tag selection

Scaffolded projects use files.include to select .yaml and .yml files under cases/. Without files, the framework discovers YAML files recursively under the test root, ignoring node_modules, .git, and run artifact directories.

When a config is loaded, the positional path further narrows the selection. --project, file inclusion/exclusion rules, and tags still apply, including to explicitly named files.

With a positional path, the command fails with a selection error if no Project has matching cases. An empty selection in one Project does not prevent other Projects from running.

Without a positional path, existing behavior is preserved: if files exist but tags exclude every case, the command can succeed. Other errors, such as failure to discover files, still fail the run.

Switching projects and SDK calls

To run another project, change into its directory or select its config with --config. A directory argument does not change config discovery.

The SDK accepts these options:

• projectRoot: the default config discovery directory; defaults to cwd.
• testPath: a file or directory filter, resolved relative to cwd.
• configPath: an explicit config, resolved relative to cwd. Its directory becomes the test root.

These rules apply to test runs. The nodes subcommand retains its existing directory and config resolution rules.

#Migrate config discovery

Default config discovery checks the current working directory (cwd) only. It does not search parent directories or switch configs when you specify a test directory.

If an existing command relies on a directory argument to load another project's config:

pnpm exec midscene-test ./other-project

Select that config explicitly:

pnpm exec midscene-test --config ./other-project/midscene.config.ts

Alternatively, change into other-project and run pnpm exec midscene-test. To run one file from the original working directory:

pnpm exec midscene-test ./other-project/cases/order.yaml --config ./other-project/midscene.config.ts

Check these paths when upgrading:

• files.include, files.exclude, and output.reportDir resolve from the config directory. If your config is in config/ and your tests are in a sibling cases/ directory, move the config to their common parent and set files.include to ['cases/**/*.yaml'].
• Default artifacts go to midscene_run inside the config directory. Update CI artifact collection paths, or move the config to the project root.
• A relative --config path now resolves from cwd, even when you specify a test file or directory. Relative test input and --result-dir paths also resolve from cwd.

Without a config, YAML discovery and collection remain available, but referenced Nodes must be registered through a config. This does not enable arbitrary YAML execution without configuration.

#Filter execution targets and configuration files

If a project defines multiple platforms or environments, you can use --project to run specific Execution Projects, or use --config to specify a custom configuration file:

# Run only the android-smoke and ios-regression Execution Projects
pnpm exec midscene-test --project android-smoke --project ios-regression

# Run tests with a specific configuration file
pnpm exec midscene-test --config ./config/midscene.config.ts

When generating the Node Spec for a specific Execution Project, you can also use --project:

pnpm exec midscene-test nodes --project android-smoke

#Output and exit codes

#Terminal output mode

By default, the terminal prints summarized execution results grouped by Case. To view detailed step diagnostics, use the verbose reporter. Run --help to view all supported CLI options:

# Show detailed step diagnostics
pnpm exec midscene-test --reporter verbose

# View all CLI options
pnpm exec midscene-test --help

#Result files and report paths

Each run generates two types of output artifacts. Their exact paths are printed at the end of the terminal output as Summary: and Report::

• Result files: Saved in midscene_run/test-results/<runId>/ by default, including the readable result.md, machine-readable summary.json, and diagnostic records. Set --result-dir to change the results root; relative paths use the command's working directory:

  pnpm exec midscene-test --result-dir ./test-results
• HTML report: Saved in midscene_run/report/ by default. Open it directly in a browser to inspect case outcomes, screenshots, errors, and replay. Set output.reportDir to change the report directory; relative paths use the Test project as their base.

Keep the complete results directory for troubleshooting or programmatic analysis. Use the summary.json fields produced by your installed version.

#Artifact directory

The default artifact directory is midscene_run/ in the Test project. Reports are in report/; results and diagnostic records are in test-results/. Set the MIDSCENE_RUN_DIR environment variable to change the artifact directory. Relative paths use the Test project as their base:

MIDSCENE_RUN_DIR="./artifacts" pnpm exec midscene-test

In CI:

• Update collection rules from .midscene/test-results to midscene_run/test-results, or your configured results directory.
• With the Midscene GitHub Action, set run-directory or MIDSCENE_RUN_DIR. Execution and artifact collection use the same directory.
• Collect files using the printed Summary: / Report: paths or the SDK's summaryPath, markdownPath, and reportPath. See below for sharing reports.

#Exit codes

You can determine the overall outcome of a test run from the CLI exit code:

• 0: All selected cases and lifecycle hooks executed successfully.
• 1: An error occurred during execution (including case step failures, YAML parsing or collection errors, and project-level or file-level lifecycle errors). Even if all case steps pass, the exit code is still 1 if an exception occurs during the cleanup phase.

#Report archiving and URL deep links

#Report archiving

Reports default to a single HTML file: midscene_run/report/midscene-e2e-<runId>.html. To choose the delivery format, set output.reportFormat in midscene.config.ts:

output: {
  reportFormat: 'single-html', // default: open the HTML directly, including offline
  // reportFormat: 'html-and-external-assets',
}

When using html-and-external-assets, preserve the entire report directory when archiving or uploading, and view it over HTTP. See Run in GitHub Actions for CI artifact configuration.

#Link to a specific step (URL hash)

When sharing reports in CI notifications, chat bots, or PR comments, use the runner-step URL hash parameter to jump directly to a target Step. Beyond explicit Step IDs, reports support portable selectors:

Example: Jump directly to the last failed Step in a report:

https://example.com/report/index.html#runner-step=last-error

If the Step includes Agent visual replay, append runner-trace=page to open the Agent trace in full-screen view:

https://example.com/report/index.html#runner-step=last-error&runner-trace=page

To limit selection to a specific case, combine with runner-page=case, runner-project=<projectId>, and runner-case=<caseKey>. Use the report's internal projectId and the composite Case Key <projectId>:<documentId>:<caseId>. URL-encode each parameter value, for example with URLSearchParams. To obtain these values, open the target case in the report and copy its URL, then add runner-step and, if needed, runner-trace.

#Environment loading

Midscene Test reads model configurations and runtime parameters from system environment variables (process.env). You can provide environment variables based on your workflow:

#Local development with .env files

• Scaffolded projects (ready out of the box): Projects created via midscene-test create already load the .env file from the configuration directory in midscene.config.ts. Simply create a .env file in the project root to get started.
• Handwritten configuration: When writing a configuration file manually, import dotenv at the top of midscene.config.ts:

  import 'dotenv/config';

When loaded through dotenv, existing environment variables in your shell take precedence over values in the .env file by default.

#CI / CD environments

In continuous integration (such as GitHub Actions) or automated pipelines, you do not need a .env file. Simply inject API keys or credentials directly as pipeline environment variables.

#Model connectivity check (model verify)

When running midscene-test model verify to test model connectivity, the command automatically reads .env from the current working directory. To facilitate quick local checks, it prioritizes values in .env over matching shell environment variables. When troubleshooting, ensure the settings in .env match the model configuration used during test runs.

#Run in GitHub Actions

Midscene provides midscene-action to run Midscene Test in GitHub Actions.

• Automatically add test results to the workflow Summary.
• Save HTML reports and test data as artifacts for download.
• Optionally publish reports to Cloudflare Workers for online previews.

For setup instructions, configuration parameters, and complete examples, see midscene-action on GitHub Marketplace.

#Show test results with a badge

The official Midscene Test badge service renders pass/fail counts as an SVG badge that can be embedded in any Markdown file, such as your repository README:

![Midscene Test: 42 passed, 3 failed](https://badge.midscenejs.com/e2e?pass=42&fail=3)

A two-line card style is also available, with light and dark themes:

The counts come entirely from the pass and fail query parameters; the service does not read CI data or reports, so you update the URL or generate it in your workflow after each run. When running inside GitHub Actions, the job Summary already includes a badge generated from the current run's results, and the same line appears in result.md, ready to copy.

For all supported parameters and styles — including flat, large, card, the dark theme, and the static "E2E by Midscene.js" attribution badge — see the badge service homepage, which provides live previews and ready-to-paste Markdown.