> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.

# See interfaces like a human
E2E testing in natural language

Midscene.js provides a vision-based GUI Agent and a natural-language E2E testing framework for web, mobile, and desktop apps.

**15k+**Github Stars

[**MIT**Fully open source](https://github.com/web-infra-dev/midscene/blob/main/LICENSE)

**#2**No.2 in Github trending

## Describe steps and expectations in natural language {#test-intent}
Describe what a user wants to do and what they should see. Express the same intent in YAML or in your existing JavaScript tests.

**YAML · @midscene/test**

```yaml
beforeEach:
  - gotoUrl: https://your-shop.example

cases:
  - name: "Find headphones under $100"
    steps:
      - aiAct: "Search for headphones and filter the results to prices under $100"
      - aiWaitFor: "The filtered search results are visible"
      - aiAssert: "There are products in the search results, and each costs less than $100"
```

**Rstest + JavaScript**

```javascript
import { test } from '@rstest/core';
import { PlaywrightAgent } from '@midscene/web/playwright';

// Model configuration, page setup, and cleanup omitted
test("Find headphones under $100", async () => {
  const agent = new PlaywrightAgent(page);
  await agent.aiAct("Search for headphones and filter the results to prices under $100");
  await agent.aiWaitFor("The filtered search results are visible");
  await agent.aiAssert("There are products in the search results, and each costs less than $100");
}, 180_000);
```

**Vitest + JavaScript**

```javascript
import { test } from 'vitest';
import { PlaywrightAgent } from '@midscene/web/playwright';

// Model configuration, page setup, and cleanup omitted
test("Find headphones under $100", async () => {
  const agent = new PlaywrightAgent(page);
  await agent.aiAct("Search for headphones and filter the results to prices under $100");
  await agent.aiWaitFor("The filtered search results are visible");
  await agent.aiAssert("There are products in the search results, and each costs less than $100");
}, 180_000);
```

![](/images/backgrounds/section-divider-light.svg)![](/images/backgrounds/section-divider-dark.svg)

## A GUI Agent built for testing {#gui-agent}
Midscene’s GUI Agent brings visual actions and assertions to your existing JavaScript tests. We recommend [Playwright Test](/integrate-with-playwright.md) for web apps and [Rstest](/sdk/rstest.md) or [Vitest](/sdk/vitest.md) for mobile and desktop apps.
### See, act, and verify like a human {#visual-capabilities}
E2E testing verifies complete workflows from the user’s perspective, so Midscene tests the way a user does.

1. **Observe**Read screenshots, not the DOM
1. **Act**Find the target, then tap, type, or scroll
1. **Verify**Assert on what users actually see

[See real-world examples](/showcases)

### One API, across platforms {#platforms}
Visual localization does not depend on the UI rendering stack, so the same API works on web, mobile, desktop, and canvas-based interfaces.

[**Web**`@midscene/web`](/integrate-with-playwright)[**Android**`@midscene/android`](/platforms/android)[**iOS**`@midscene/ios`](/platforms/ios)

Same API`await agent.**aiAct**('Search for headphones')`

[**HarmonyOS**`@midscene/harmony`](/platforms/harmonyos)[**Desktop**`@midscene/computer`](/platforms/desktop)[**Custom interfaces**`@midscene/core`](/integrate-with-any-interface)

### Model options and cost {#model-options}

[![](/images/models/supported-models-light.svg)![](/images/models/supported-models-dark.svg)### Multiple model options

Choose from Doubao, DeepSeek, Qwen, GPT, Gemini, Kimi, and more to fit your tasks and budget.

View model configuration](/model-selection)[Benchmark example · AppControlBench

**$0.59**/ 60 tasks

Total model API cost### Low model API cost

In this AppControlBench run, Doubao Seed 2.1 Turbo passed 58 of 60 tasks at a total model API cost of $0.59.

View benchmark report](/app-control-bench-report)

### Benchmarks {#benchmarks}
- **AndroidWorld**: Pass\@1 of 93.1% and Pass\@3 of 97.4%. [Read the benchmark report](/android-world-benchmark-report.md)
- **MobileWorld**: Pass\@1 of 78.6%, completing 92 of 117 tasks. [Read the benchmark report](/mobile-world-benchmark-report.md)
- **AppControlBench**: Pass\@1 of 96.7%, passing 58 of 60 tasks. [Read the benchmark report](/app-control-bench-report.md)
## A new E2E testing framework designed for natural language {#midscene-test}
Midscene Test (`@midscene/test`) is a natural-language E2E testing framework. Describe actions and expectations in YAML, and keep logic that needs precise control in TypeScript. [Get started with Midscene Test](/midscene-test/use.md)
### AI and business operations in one YAML file {#ai-and-nodes}
Each YAML step calls a Node: an AI, platform, or custom business operation. Compose them into a test case.

- AI**AI operations**APIs such as aiAct and aiAssert read the screen and follow natural-language instructions.
- Business**Business operations**Custom Nodes use TypeScript to prepare test data, check APIs, and clean up resources.

cases/refund.yaml

````
1beforeEach:2  - order.prepare: { status: paid }Business3  - gotoUrl: ${shopUrl}/orders/latestBusiness45cases:6  - name: A paid order supports a full refund7    steps:8      - aiAct: Apply for a full refund and submitAI9      - aiAssert: The page shows that the refund was submittedAI10      - order.checkRefund: { status: refunding }Business1112afterEach:13  - clearCookies: {}Business
````

### Test cases maintained by people and AI {#nodes-and-spec}
Wrap reusable business logic in custom Nodes. Midscene Test generates a Markdown Node Spec listing every Node and its inputs, so QA engineers and AI agents can write cases without reading TypeScript. [Learn how to develop custom Nodes](/midscene-test/extend.md)

1. nodes/order.ts

```typescript
defineNode({
  name: 'order.prepare',
  description: "Create a test order",
  inputSchema: z.strictObject({
    status: z.literal('paid'),
  }),
  async execute({ input }) {
    await orders.create(input);
  },
});
```

1. midscene-node-spec.md

````markdown
### order.prepare
Create a test order
- status: "paid"
````

1. case.yaml

```yaml
steps:
  - order.prepare:
      status: paid
```

### Test framework capabilities {#framework-capabilities}
Midscene Test also includes the capabilities expected of a test framework: lifecycle hooks and cleanup, retries and timeouts, concurrent execution with isolation, and step-level reports.

[ProjectsetuponTeardown

cases/order.yamlfileSetupbeforeAllafterAll

Case 1beforeEachaiActaiAssertafterEach

Case 2beforeEachaiActaiAssertafterEach

Timeout or Ctrl+CCleanup still runs### Lifecycle hooks and cleanup

Prepare and clean up resources at the project, file, and case level with setup, fileSetup, beforeAll, beforeEach, and their counterparts. Cleanup still runs after failures, timeouts, or Ctrl+C.

Project lifecycle](/midscene-test/execution#project-lifecycle)[A paid order supports a full refundretry: 1

**Attempt 1**aiAssert timed out after 120 s

afterEach

Retry after cleanup

**Attempt 2**All steps passed

afterEach

testTimeoutcontinue-on-errorbail### Retries and timeouts

Retry failed cases; each attempt reruns beforeEach, steps, and afterEach. Set step timeouts, keep going after a failed step with continue-on-error, and stop early with bail.

Handling failures](/midscene-test/execution#handling-failures)[RunningmaxConcurrentFiles: 3

- `search.yaml`Own page + Agent
- `cart.yaml`Own page + Agent
- `refund.yaml`Own page + Agent

Queued`profile.yaml``orders.yaml`### Concurrency and isolation

Run YAML files concurrently with maxConcurrentFiles. fileSetup gives each file its own page, Agent, or device, so concurrent files do not share state.

Concurrency and isolation](/midscene-test/execution#concurrency-and-isolation)[report.html#runner-step=last-error

A paid order supports a full refund

1. order.prepare{ status: paid }
1. aiActApply for a full refund and submit
1. aiAssertThe page shows that the refund was submitted
1. order.checkRefundNot run

ScreenshotAgent replay

Error

For scripts and AI agentssummary.jsonresult.md### Step-level reports

The HTML report records screenshots, inputs, outputs, errors, and Agent replay for every step. A #runner-step=last-error link opens the latest failed step from CI notifications or PR comments, and summary.json and result.md are written for scripts and AI agents.

Output and reports](/midscene-test/running#output-and-exit-codes)

## Who is using Midscene {#users}

[![](/images/icons/bytedance-color.svg)ByteDance](https://www.bytedance.com)[![](/images/users/alibaba-color.svg)Alibaba](https://www.alibaba.com)[![](/images/users/ctrip-color.svg)Ctrip](https://www.ctrip.com)[![AVATR](/images/users/avatr-wordmark.svg)](https://www.avatr.com)[![](/images/users/xiaomi-color.svg)Xiaomi](https://www.mi.com)[![iQIYI](/images/users/iqiyi-color.svg)](https://www.iqiyi.com)[![Bilibili](/images/users/bilibili-color.svg)](https://www.bilibili.com)[![Autel](/images/users/autel-color.svg)](https://autelenergy.com)[![](/images/users/rome-os.svg)Rome OS](https://github.com/rome-os/rome)[![](/images/users/volcengine.png)Volcengine](https://www.volcengine.com)[![](/images/users/douyin-color.svg)Douyin](https://www.douyin.com)[![](/images/users/tiktok-color.svg)TikTok](https://www.tiktok.com)[![](/images/users/lark-color.svg)Lark](https://www.larksuite.com)[![](/images/users/soda-music-color.svg)Soda Music](https://www.douyin.com/qishui)[![](/images/users/doubao-color.png)Doubao](https://www.doubao.com)[![Dongchedi](/images/users/dongchedi-color.svg)](https://www.dongchedi.com)

## Next steps {#next-steps}
- **Understand the basics**: Learn how the GUI Agent works and explore its core capabilities in [The Basics](/basics.md)
- **Get started**: Answer a few questions in [Get started](/get-started.md) and complete a first run on your platform
- **Create a test project**: Write and run YAML test cases with [Midscene Test](/midscene-test/use.md)
- **Write tests with code**: Integrate with [Playwright Test](/integrate-with-playwright.md) for web apps or [Rstest](/sdk/rstest.md) or [Vitest](/sdk/vitest.md) for mobile and desktop apps
- **Run standalone automation workflows**: [Write scripts with the JavaScript SDK](/sdk/standalone.md)
To let AI coding agents operate your UI, explore [Midscene Skills](/skills.md).