> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.

# Run tests

## Basic usage {#run-tests}

Run YAML test cases from the project root:

```bash
pnpm exec midscene-test
```

### Specify a test case directory or file

With a config, a file or directory argument filters the selected cases. Full-suite, single-file, and subdirectory runs share the same project configuration.

Run these commands from the project root:

```bash
pnpm exec midscene-test
pnpm exec midscene-test ./cases/order.yaml
pnpm exec midscene-test ./cases/smoke
```

They share the configured Nodes, Execution Projects, and default output location.

**Configuration and path resolution**

| Input                      | Resolution rule                                                                                                                           |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Default config             | Search cwd only for `midscene.config.ts` or `midscene.config.mjs`.                                                                        |
| `--config`                 | Resolve relative paths from cwd; absolute paths are also supported.                                                                       |
| Test root                  | Use the config directory, or cwd when no config is found.                                                                                 |
| File or directory argument | Resolve from cwd. Filter configured cases when a config exists; otherwise discover and collect cases from the selected file or directory. |
| Paths in config            | Resolve `files.include`, `files.exclude`, and `output.reportDir` from the test root.                                                      |
| `--result-dir`             | Resolve relative paths from cwd.                                                                                                          |

Default discovery checks cwd only. When running from a child directory, select the project config explicitly:

```bash
cd cases
pnpm exec midscene-test ./order.yaml --config ../midscene.config.ts
```

The parent directory containing the config becomes the test root. Default output remains in that directory's `midscene_run`.

Without a config, a file argument collects the selected YAML, and a directory argument discovers YAML files in that directory. Paths outside cwd are supported. With no argument, discovery starts at cwd. Configs beside the selected input are not loaded. Default artifacts stay under `cwd/midscene_run`.

Without a config, no Nodes are registered. YAML that references Nodes still needs a config that registers them. Legacy `tasks/flow` YAML is run with `midscene`, not `midscene-test`.

**File and tag selection**

Scaffolded projects use `files.include` to select `.yaml` and `.yml` files under `cases/`. Without `files`, the framework discovers YAML files recursively under the test root, ignoring `node_modules`, `.git`, and run artifact directories.

When a config is loaded, the positional path further narrows the selection. `--project`, file inclusion/exclusion rules, and tags still apply, including to explicitly named files.

With a positional path, the command fails with a selection error if no Project has matching cases. An empty selection in one Project does not prevent other Projects from running.

Without a positional path, existing behavior is preserved: if files exist but tags exclude every case, the command can succeed. Other errors, such as failure to discover files, still fail the run.

**Switching projects and SDK calls**

To run another project, change into its directory or select its config with `--config`. A directory argument does not change config discovery.

The SDK accepts these options:

- `projectRoot`: the default config discovery directory; defaults to `cwd`.
- `testPath`: a file or directory filter, resolved relative to `cwd`.
- `configPath`: an explicit config, resolved relative to `cwd`. Its directory becomes the test root.

These rules apply to test runs. The `nodes` subcommand retains its existing directory and config resolution rules.

### Migrate config discovery

Default config discovery checks the current working directory (cwd) only. It does not search parent directories or switch configs when you specify a test directory.

If an existing command relies on a directory argument to load another project's config:

```bash
pnpm exec midscene-test ./other-project
```

Select that config explicitly:

```bash
pnpm exec midscene-test --config ./other-project/midscene.config.ts
```

Alternatively, change into `other-project` and run `pnpm exec midscene-test`. To run one file from the original working directory:

```bash
pnpm exec midscene-test ./other-project/cases/order.yaml --config ./other-project/midscene.config.ts
```

Check these paths when upgrading:

- `files.include`, `files.exclude`, and `output.reportDir` resolve from the config directory. If your config is in `config/` and your tests are in a sibling `cases/` directory, move the config to their common parent and set `files.include` to `['cases/**/*.yaml']`.
- Default artifacts go to `midscene_run` inside the config directory. Update CI artifact collection paths, or move the config to the project root.
- A relative `--config` path now resolves from cwd, even when you specify a test file or directory. Relative test input and `--result-dir` paths also resolve from cwd.

Without a config, YAML discovery and collection remain available, but referenced Nodes must be registered through a config. This does not enable arbitrary YAML execution without configuration.

### Filter execution targets and configuration files

If a project defines multiple platforms or environments, you can use `--project` to run specific Execution Projects, or use `--config` to specify a custom configuration file:

```bash
# Run only the android-smoke and ios-regression Execution Projects
pnpm exec midscene-test --project android-smoke --project ios-regression

# Run tests with a specific configuration file
pnpm exec midscene-test --config ./config/midscene.config.ts
```

When generating the Node Spec for a specific Execution Project, you can also use `--project`:

```bash
pnpm exec midscene-test nodes --project android-smoke
```

## Output and exit codes

### Terminal output mode

By default, the terminal prints summarized execution results grouped by Case. To view detailed step diagnostics, use the `verbose` reporter. Run `--help` to view all supported CLI options:

```bash
# Show detailed step diagnostics
pnpm exec midscene-test --reporter verbose

# View all CLI options
pnpm exec midscene-test --help
```

### Result files and report paths

Each run generates two types of output artifacts. Their exact paths are printed at the end of the terminal output as `Summary:` and `Report:`:

- **Result files**: Saved in `midscene_run/test-results/<runId>/` by default, including the readable `result.md`, machine-readable `summary.json`, and diagnostic records. Set `--result-dir` to change the results root; relative paths use the command's working directory:
  ```bash
  pnpm exec midscene-test --result-dir ./test-results
  ```
- **HTML report**: Saved in `midscene_run/report/` by default. Open it directly in a browser to inspect case outcomes, screenshots, errors, and replay. Set `output.reportDir` to change the report directory; relative paths use the Test project as their base.

Keep the complete results directory for troubleshooting or programmatic analysis. Use the `summary.json` fields produced by your installed version.

### Artifact directory {#unified-artifact-root-and-migration}

The default artifact directory is `midscene_run/` in the Test project. Reports are in `report/`; results and diagnostic records are in `test-results/`. Set the `MIDSCENE_RUN_DIR` environment variable to change the artifact directory. Relative paths use the Test project as their base:

```bash
MIDSCENE_RUN_DIR="./artifacts" pnpm exec midscene-test
```

In CI:

- Update collection rules from `.midscene/test-results` to `midscene_run/test-results`, or your configured results directory.
- With the [Midscene GitHub Action](https://github.com/web-infra-dev/midscene-action), set `run-directory` or `MIDSCENE_RUN_DIR`. Execution and artifact collection use the same directory.
- Collect files using the printed `Summary:` / `Report:` paths or the SDK's `summaryPath`, `markdownPath`, and `reportPath`. See below for sharing reports.

### Exit codes

You can determine the overall outcome of a test run from the CLI exit code:

- `0`: All selected cases and lifecycle hooks executed successfully.
- `1`: An error occurred during execution (including case step failures, YAML parsing or collection errors, and project-level or file-level lifecycle errors). Even if all case steps pass, the exit code is still `1` if an exception occurs during the cleanup phase.

## Report archiving and URL deep links

### Report archiving

Reports default to a single HTML file: `midscene_run/report/midscene-e2e-<runId>.html`. To choose the delivery format, set `output.reportFormat` in `midscene.config.ts`:

```ts
output: {
  reportFormat: 'single-html', // default: open the HTML directly, including offline
  // reportFormat: 'html-and-external-assets',
}
```

When using `html-and-external-assets`, preserve the entire report directory when archiving or uploading, and view it over HTTP. See [Run in GitHub Actions](#run-in-github-actions) for CI artifact configuration.

### Link to a specific step (URL hash)

When sharing reports in CI notifications, chat bots, or PR comments, use the `runner-step` URL hash parameter to jump directly to a target Step. Beyond explicit Step IDs, reports support portable selectors:

| Selector      | Target                                                               |
| ------------- | -------------------------------------------------------------------- |
| `first`       | Earliest started Step                                                |
| `last`        | Most recently completed Step                                         |
| `first-error` | Earliest started failed Step, including failed retry attempts        |
| `last-error`  | Most recently completed failed Step, including failed retry attempts |

Example: Jump directly to the last failed Step in a report:

```text
https://example.com/report/index.html#runner-step=last-error
```

If the Step includes Agent visual replay, append `runner-trace=page` to open the Agent trace in full-screen view:

```text
https://example.com/report/index.html#runner-step=last-error&runner-trace=page
```

To limit selection to a specific case, combine with `runner-page=case`, `runner-project=<projectId>`, and `runner-case=<caseKey>`. Use the report's internal `projectId` and the composite Case Key `<projectId>:<documentId>:<caseId>`. URL-encode each parameter value, for example with `URLSearchParams`. To obtain these values, open the target case in the report and copy its URL, then add `runner-step` and, if needed, `runner-trace`.

## Environment loading {#environment-loading}

Midscene Test reads model configurations and runtime parameters from system environment variables (`process.env`). You can provide environment variables based on your workflow:

### Local development with .env files

- **Scaffolded projects (ready out of the box)**: Projects created via `midscene-test create` already load the `.env` file from the configuration directory in `midscene.config.ts`. Simply create a `.env` file in the project root to get started.
- **Handwritten configuration**: When writing a configuration file manually, import `dotenv` at the top of `midscene.config.ts`:
  ```ts
  import 'dotenv/config';
  ```

When loaded through `dotenv`, existing environment variables in your shell take precedence over values in the `.env` file by default.

### CI / CD environments

In continuous integration (such as GitHub Actions) or automated pipelines, you do not need a `.env` file. Simply inject API keys or credentials directly as pipeline environment variables.

### Model connectivity check (model verify)

When running `midscene-test model verify` to test model connectivity, the command automatically reads `.env` from the current working directory. To facilitate quick local checks, it prioritizes values in `.env` over matching shell environment variables. When troubleshooting, ensure the settings in `.env` match the model configuration used during test runs.

## Run in GitHub Actions

Midscene provides [midscene-action](https://github.com/marketplace/actions/midscene-action) to run Midscene Test in GitHub Actions.

- Automatically add test results to the workflow **Summary**.
- Save HTML reports and test data as **artifacts** for download.
- Optionally publish reports to **Cloudflare Workers** for online previews.

For setup instructions, configuration parameters, and complete examples, see [midscene-action on GitHub Marketplace](https://github.com/marketplace/actions/midscene-action).

### Show test results with a badge

The official [Midscene Test badge service](https://badge.midscenejs.com) renders pass/fail counts as an SVG badge that can be embedded in any Markdown file, such as your repository README:

```md
![Midscene Test: 42 passed, 3 failed](https://badge.midscenejs.com/e2e?pass=42&fail=3)
```

![Midscene Test: 42 passed, 3 failed](https://badge.midscenejs.com/e2e?pass=42\&fail=3)

A two-line card style is also available, with light and dark themes:

![Midscene Test: 42 passed, 3 failed](https://badge.midscenejs.com/e2e?pass=42\&fail=3\&style=card\&theme=light)

The counts come entirely from the `pass` and `fail` query parameters; the service does not read CI data or reports, so you update the URL or generate it in your workflow after each run. When running inside GitHub Actions, the job Summary already includes a badge generated from the current run's results, and the same line appears in `result.md`, ready to copy.

For all supported parameters and styles — including `flat`, `large`, `card`, the dark theme, and the static "E2E by Midscene.js" attribution badge — see the [badge service homepage](https://badge.midscenejs.com), which provides live previews and ready-to-paste Markdown.