# Midscene.js - AI UI Automation and E2E Testing > Bring AI vision to E2E testing with a newly designed test framework for web, mobile, and desktop apps. ## Others - [BDD-style scripts with Gherkin](/advanced/bdd-style-scripts-with-gherkin.md): This feature is available starting in Midscene 1.10.BDD-related capabilities are still in Beta. The API may change in future releases. Gherkin is a plain-text syntax for describing behavior examples. It is commonly used in BDD (Behavior-Driven Development) workflows. A BDD scenario usually contains three kinds of steps: Given describes the precondition, When describes the user action, and Then describes the expected result. For example, when you need to test adding a todo item, you can write the case as the following Gherkin script. Longer GUI cases can also be written in one Scenario. You can organize the steps by business phase, and each phase can use a group of Given, When, and Then. This style is useful when one business scenario naturally contains several UI phases, such as cart review, shipping selection, payment, and final confirmation. In AI-driven GUI automation, Gherkin is a natural fit for describing operation cases. It keeps the readability of natural language while providing a stable step structure. This structure also helps models generate more consistent cases. See the official Cucumber Gherkin reference for the full Gherkin syntax. - [Midscene AndroidWorld Benchmark Report](/android-world-benchmark-report.md): This is Midscene's test report for the AndroidWorld benchmark. In this run, Midscene achieved Pass@1 93.10%, Pass@2 95.69%, and Pass@3 97.41%. - [Midscene AppControlBench Benchmark Report](/app-control-bench-report.md): This is Midscene's benchmark report for AppControlBench. In this evaluation, each model ran the same set of 60 tasks, with the following results: Note: Doubao Seed 2.1 Turbo costs use Volcengine's official pricing: CNY ¥3 / M input tokens, ¥0.6 / M cached input tokens, and ¥15 / M output tokens. Other model costs use OpenRouter pricing. USD values use ¥6.8 per $1. The results show that the Midscene + GUI vision approach is highly competitive in both cost and pass rate. - [Automate with scripts in YAML](/automate-with-scripts-in-yaml.md): In most cases, developers write automation scripts just to perform some smoke tests, like checking for the appearance of certain content or verifying that a key user path is accessible. In such situations, maintaining a large test project is unnecessary. Midscene offers a way to perform automation using .yaml files, which helps you focus on the script itself rather than the testing framework. This allows any team member to write automation scripts without needing to learn any API. Here is an example. By reading its content, you should be able to understand how it works. - [Awesome Midscene](/awesome-midscene.md): A curated list of community projects that extend Midscene.js capabilities across different platforms and programming languages. - [The Basics](/basics.md): This page explains Midscene's vision-based approach, usage options, and key APIs to help you understand and use the GUI Agent. To try it first, go to Get started. - [Introducing Instant Actions and Deep Think](/blog-introducing-instant-actions-and-deep-think.md): From Midscene v0.14.0, we have introduced two new features: Instant Actions and Deep Think. - [Chrome Bridge environment setup](/bridge-mode.md): Chrome Bridge enables Node.js scripts to control your desktop Chrome browser through the Midscene Chrome extension, allowing you to reuse existing login sessions, cookies, and browser extensions. The Node.js process runs the Bridge Server and handles AI model calls, while the Chrome extension connects to that server and executes operations on browser tabs. - [Caching AI Planning and DOM Localization](/caching.md): Midscene supports caching two kinds of data: AI Planning steps and matched element localization information. The former can be used across automation workflows to reduce AI model calls and improve execution efficiency; for web automation, the DOM localization part (XPath) in the latter can significantly reduce repeated locating overhead, but it is currently web-only and has certain limitations. Effect With caching hit, time cost is significantly reduced. For example, in the following case, execution time was reduced from 51 seconds to 28 seconds. before after - [Changelog](/changelog.md): {/* release-notes:start */} - [Use a ChatGPT subscription](/chatgpt-subscription.md): Authorize Midscene through Sign in with ChatGPT (SIWC) to call GPT models using your ChatGPT subscription. This guide walks you through sign-in, configuration, and connection verification. A ChatGPT subscription and a model API key are two ways to connect Midscene. You can choose either based on your model payment plan. - [Consume Report Files](/consume-report-file.md): Midscene HTML report files capture the full execution history of a single Agent, making them useful for replay and debugging. Starting in v1.7.0, you can extract raw screenshots and JSON data from a report file, or convert the report into Markdown so other tools can consume it. Both split and to-markdown export images to screenshots/ and reuse repeated images. Use the image paths provided in the exported content. - [Data privacy](/data-privacy.md): ⁠Midscene.js is an open-source project (GitHub: Midscene) under the MIT license. You can see all the codes in the public repository. The Midscene project does not collect your screenshots, API keys, or access tokens through the SDK. During AI steps, the SDK sends the required screenshots, prompts, and page context directly to the model service you configure. That service’s privacy policy governs how it stores and uses this data. By default, Midscene saves reports and logs on the machine running the task, under midscene_run/report and midscene_run/log. If your scripts, application screens, or execution results contain sensitive information such as plaintext tokens or cookies, that information may also appear in reports and logs. Manage access and retention, and remove sensitive content before sharing. In CI, console logs and uploaded artifacts may also be visible to people with access to the job. If you prefer building Midscene.js and its Chrome Extension in your own environment instead of using the published versions, you can refer to the Contributing Guide to find building instructions. - [FAQ](/faq.md) - [Get started](/get-started.md): Answer three questions to open the setup steps for your platform, way of use, and model. Each path ends with a first run whose result you can inspect. - [Integrate with any interface](/integrate-with-any-interface.md): You can use Midscene Agent to control any interface—such as IoT devices, in-house apps, and in-vehicle displays—by implementing a UI operation class that conforms to AbstractInterface. After implementing the UI operation class, you get the full capabilities of Midscene Agent: the TypeScript GUI Automation Agent SDK, supporting integration with any interfacethe playground for debuggingcontrolling the interface with YAML scriptsSkills support through CLI commands - [Integrate with Playwright Test](/integrate-with-playwright.md): With PlaywrightAiFixture, you can use Midscene's visual interaction and assertion capabilities directly in an existing Playwright Test project. Playwright Test organizes and runs the tests, while Midscene interacts with the page and generates execution reports. - [Web / Puppeteer SDK](/integrate-with-puppeteer.md): @midscene/web/puppeteer provides Agents bound to a Puppeteer page or Browser. Both inherit the base Agent’s AI methods and accept Web configuration options. - [LLMs.txt documentation](/llm-txt.md): How to get tools like Cursor, Windstatic, GitHub Copilot, ChatGPT, and Claude to understand Midscene.js. We support LLMs.txt files for making the Midscene.js documentation available to large language models. - [MCP integration has been retired](/mcp.md): Midscene no longer ships MCP servers. Use Skills to let AI coding agents drive Midscene through the platform CLIs. If you still need the MCP server packages, pin Midscene to 1.9.8. This is the final version that includes MCP support. Remove any agent configuration that references the retired MCP packages: @midscene/web-bridge-mcp@midscene/android-mcp@midscene/ios-mcp@midscene/harmony-mcp@midscene/computer-mcp@midscene/mcp If your previous MCP configuration set MIDSCENE_MCP_CHROME_PATH, move that value to MIDSCENE_CHROME_PATH for Skills and CLI usage. The old variable is still accepted as a temporary migration alias. For code-level automation, use the JavaScript SDK, YAML runner, or the platform CLIs listed in Skills. - [Built-in Nodes across platforms](/midscene-test/builtin-nodes.md): Midscene Test refers to each callable capability in YAML test cases and lifecycle hooks as a Node. When you register an official Agent in midscene.config.ts using createMidsceneNodes({ agentClass, getAgent }), the framework injects general AI capabilities and official built-in Nodes for the respective platform. This guide catalogs available built-in Nodes, their input structures, and common usage patterns across platforms. - [Configure test projects](/midscene-test/configuration.md): midscene.config.ts is the project configuration file for Midscene Test. Use it to configure browser or device initialization, register Nodes for test cases to call, and set execution options such as concurrency and timeouts. This guide covers configuration structure, Agent and platform integration, multiple execution projects, and file resource isolation. - [Execution lifecycle and concurrency](/midscene-test/execution.md) - [Develop custom Nodes](/midscene-test/extend.md): Custom Nodes turn business operations into steps that test authors can call from YAML. This guide covers defining and validating inputs, sharing data between Nodes, and handling execution and cleanup. For platform setup, Agent integration, and execution settings, see Configure test projects. Register a Node: define an operation, call it from YAML, and verify registration through the generated Node Spec.Node execution arguments: distinguish business inputs from framework step settings.Share context across Nodes: create, read, and reset shared test data.Advanced usage: return results, handle errors and cancellation, and clean up resources. - [Midscene Test overview](/midscene-test/overview.md) - [Run tests](/midscene-test/running.md) - [Create a Midscene Test E2E project](/midscene-test/use.md): Create a project, run the TodoMVC example, then change a case. This guide follows the Web starter; mobile and desktop users can select their platform template. For a first run on a specific platform, Get started walks you through device, model, and project setup for your combination and ends with a passing example. - [Author YAML cases](/midscene-test/writing.md) - [Migrate to Midscene Test](/migrate-to-midscene-test.md): Legacy YAML and native Midscene Test share the same underlying execution kernel and report page, but they have separate user-facing entries. The midscene command owns legacy tasks/flow files, YAML batch configuration, and legacy CLI options. The midscene-test command owns native cases/steps files, TypeScript or JavaScript project configuration, lifecycle hooks, and Nodes. There are two migration paths: Upgrade in place: keep legacy cases, batch configuration, directory layout, and the midscene command unchanged. This adopts the shared execution kernel and new report without changing the input contract.Native migration: rewrite complete files as Test Documents, Cases, and Steps; replace batch YAML with project configuration; then switch those files to midscene-test. The complete legacy field reference remains available in YAML script runner and Workflow in YAML format for maintaining and migrating existing projects. - [Midscene MobileWorld Benchmark Report](/mobile-world-benchmark-report.md): This is Midscene's test report for the MobileWorld benchmark. In this run, Midscene evaluated only 117 GUI-ONLY tasks and achieved Pass@1 78.63% (92/117). - [Use a model API key](/model-api-setup.md): Connect Midscene using an API key from a model provider. This guide covers model configuration and connection verification. If you already have a ChatGPT subscription, you can also follow Use a ChatGPT subscription. The model configuration shown on this page consists of environment variables. Provide it according to how you use Midscene: In Playground, paste the configuration text from this page directly into the settings page.With the SDK or CLI, load the configuration as described in Set environment variables. For advanced parameter definitions, see Model advanced configuration and debugging. - [Model advanced configuration and debugging](/model-config.md): This page covers advanced model settings, connection troubleshooting, logs, and tracing. - [Model Selection Guide](/model-selection.md): This guide introduces the models supported by Midscene, two ways to connect them, and when to use one Default model or combine multiple models. - [Android environment setup](/platforms/android.md): Midscene connects to and controls Android devices over adb, supporting UI automation across native apps, hybrid apps, and system interfaces. This guide prepares the Android environment: install adb and connect the device. Playground, Midscene Test, and the JavaScript SDK all use this connection. - [Desktop environment setup](/platforms/desktop.md): Midscene automates desktop applications across Windows, macOS, and Linux using native mouse and keyboard controls or Remote Desktop Protocol (RDP), with built-in multi-display support. It works with Electron, Qt, native desktop apps, and cross-application workflows. This guide prepares the desktop environment: grant screen capture and input permissions, and install the dependencies Linux needs. Playground, Midscene Test, and the JavaScript SDK all rely on them. - [HarmonyOS environment setup](/platforms/harmonyos.md): Midscene connects to and controls HarmonyOS NEXT devices through HarmonyOS Device Connector (HDC), supporting UI automation across native apps, atomic services, and system interfaces. This guide prepares the HarmonyOS environment: install HDC and connect the device. Playground, Midscene Test, and the JavaScript SDK all use this connection. - [Platform environment setup](/platforms/index.md): Each platform guide prepares one environment: the connection tools, device connection, and system permissions that Midscene needs. Playground, Midscene Test, and the JavaScript SDK all use the same environment. Each guide ends with a Verify section that checks the environment in Playground. Midscene uses multimodal vision models to understand screenshots, operating directly on the rendered UI rather than relying on the underlying DOM or native view hierarchy. This allows the same interaction and assertion approach to work across native apps and cross-platform frameworks alike. - [iOS environment setup](/platforms/ios.md): Midscene connects to and controls iOS devices through WebDriverAgent (WDA), supporting UI automation across native apps, hybrid apps, and system interfaces. This guide prepares the iOS environment: build and start WDA, then make it reachable from the computer that runs Midscene. Playground, Midscene Test, and the JavaScript SDK all use this connection. - [Web environment setup](/platforms/web.md): Midscene controls web pages in one of three browsers: the Midscene Chrome extension, a browser launched by Playwright or Puppeteer, or your desktop Chrome through Chrome Bridge. This guide prepares each of them. Prepare only the one you plan to use. - [Provide application knowledge to a GUI Agent](/provide-application-knowledge-to-gui-agent.md): When using a GUI Agent, developers often want to describe a goal and let the Agent handle the actions. But a clear goal does not mean the Agent already has the application knowledge needed to complete it. This article uses a practical example to explain how to supply relevant application knowledge and express your intent clearly, helping a GUI Agent execute tasks more reliably. - [Shared Agent APIs](/reference/index.md): This page documents the shared APIs inherited by all platform Agents—covering configuration options, interaction methods, data extraction, element location, assertions and waiting, continuous observation, workflow execution, reporting and usage metrics, as well as lifecycle and runtime configuration. - [Android JavaScript SDK](/sdk/android.md): @midscene/android controls Android devices over adb. AndroidAgent inherits all AI methods from the base Agent. - [Chrome Bridge SDK](/sdk/chrome-bridge.md): AgentOverChromeBridge controls desktop Chrome through the Midscene Chrome extension and inherits all AI methods from the base Agent. Complete the Chrome Bridge environment setup before connecting. - [Computer JavaScript SDK](/sdk/desktop.md): @midscene/computer provides Device implementations for local desktops (ComputerDevice) and remote RDP desktops (RDPDevice), along with ComputerAgent, which inherits all AI methods from the base Agent. - [HarmonyOS JavaScript SDK](/sdk/harmonyos.md): @midscene/harmony controls HarmonyOS devices over HDC. HarmonyAgent inherits all AI methods from the base Agent. - [iOS JavaScript SDK](/sdk/ios.md): @midscene/ios controls iOS devices through WebDriverAgent (WDA). IOSAgent inherits all AI methods from the base Agent. - [Web / Playwright SDK](/sdk/playwright.md): @midscene/web/playwright provides Agents bound to a Playwright page or BrowserContext. Both inherit the base Agent’s AI methods and accept Web configuration options. - [Integrate with Rstest](/sdk/rstest.md): We recommend Rstest or Vitest for writing JavaScript and TypeScript tests on Android, iOS, HarmonyOS, and desktop. This guide covers Rstest. Rstest organizes test cases, runs lifecycle hooks, and aggregates test results, while Midscene Agents connect to devices, interact with the UI, and perform assertions. - [Write standalone automation scripts](/sdk/standalone.md): You can call the Midscene SDK directly from a Node.js script to operate apps, extract data, or automate everyday workflows without a test runner. Your script controls the execution order, branching logic, error handling, and resource cleanup. - [Integrate with Vitest](/sdk/vitest.md): We recommend Rstest or Vitest for writing JavaScript and TypeScript tests on Android, iOS, HarmonyOS, and desktop. This guide covers Vitest. Vitest organizes test cases, runs lifecycle hooks, and aggregates test results, while Midscene Agents connect to devices, interact with the UI, and perform assertions. - [showcases-android](/showcases-android.md) - [showcases-computer](/showcases-computer.md) - [showcases-harmony](/showcases-harmony.md) - [showcases-ios](/showcases-ios.md) - [showcases-web](/showcases-web.md) - [Showcases](/showcases.md): This page introduces cross-platform GUI Agent, E2E test, and community showcases built with Midscene. - [Control any platform with Skills](/skills.md): Agent Skills are a format for extending AI coding agents with specialized capabilities. Midscene provides Agent Skills that let AI coding tools (like Claude Code, Cline, etc.) drive UI automation through CLI commands. Skills work by running CLI commands directly in the terminal. The AI agent acts as the brain: it takes screenshots, analyzes the UI, and decides which actions to perform next. - [YAML script runner](/yaml-script-runner.md): Midscene defines a YAML-based scripting format so you can quickly author automation scripts, then run them from the command line without extra setup. For more details on YAML scripts, see Automate with scripts in YAML. For example, you can write a YAML script like this: Run it with one command: The CLI prints execution progress and generates a visual report when it finishes, while keeping setup simple. - [Get started: Web · Playground · ChatGPT subscription](/get-started/web-playground-chatgpt.md): Step-by-step setup for Midscene on Web with Playground and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: Web · Midscene Test · ChatGPT subscription](/get-started/web-test-chatgpt.md): Step-by-step setup for Midscene on Web with Midscene Test and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: Web · Playground · Model API Key](/get-started/web-playground-api-key.md): Step-by-step setup for Midscene on Web with Playground and a Model API Key, ending with a first run whose result you can inspect. - [Get started: Web · Midscene Test · Model API Key](/get-started/web-test-api-key.md): Step-by-step setup for Midscene on Web with Midscene Test and a Model API Key, ending with a first run whose result you can inspect. - [Get started: Web · Playwright Test · ChatGPT subscription](/get-started/web-sdk-chatgpt.md): Step-by-step setup for Midscene on Web with Playwright Test and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: Web · Playwright Test · Model API Key](/get-started/web-sdk-api-key.md): Step-by-step setup for Midscene on Web with Playwright Test and a Model API Key, ending with a first run whose result you can inspect. - [Get started: Android · Playground · ChatGPT subscription](/get-started/android-playground-chatgpt.md): Step-by-step setup for Midscene on Android with Playground and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: Android · Playground · Model API Key](/get-started/android-playground-api-key.md): Step-by-step setup for Midscene on Android with Playground and a Model API Key, ending with a first run whose result you can inspect. - [Get started: Android · Midscene Test · Model API Key](/get-started/android-test-api-key.md): Step-by-step setup for Midscene on Android with Midscene Test and a Model API Key, ending with a first run whose result you can inspect. - [Get started: Android · Rstest / Vitest · ChatGPT subscription](/get-started/android-sdk-chatgpt.md): Step-by-step setup for Midscene on Android with Rstest / Vitest and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: Android · Rstest / Vitest · Model API Key](/get-started/android-sdk-api-key.md): Step-by-step setup for Midscene on Android with Rstest / Vitest and a Model API Key, ending with a first run whose result you can inspect. - [Get started: iOS · Playground · ChatGPT subscription](/get-started/ios-playground-chatgpt.md): Step-by-step setup for Midscene on iOS with Playground and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: iOS · Playground · Model API Key](/get-started/ios-playground-api-key.md): Step-by-step setup for Midscene on iOS with Playground and a Model API Key, ending with a first run whose result you can inspect. - [Get started: iOS · Midscene Test · ChatGPT subscription](/get-started/ios-test-chatgpt.md): Step-by-step setup for Midscene on iOS with Midscene Test and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: iOS · Midscene Test · Model API Key](/get-started/ios-test-api-key.md): Step-by-step setup for Midscene on iOS with Midscene Test and a Model API Key, ending with a first run whose result you can inspect. - [Get started: iOS · Rstest / Vitest · ChatGPT subscription](/get-started/ios-sdk-chatgpt.md): Step-by-step setup for Midscene on iOS with Rstest / Vitest and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: iOS · Rstest / Vitest · Model API Key](/get-started/ios-sdk-api-key.md): Step-by-step setup for Midscene on iOS with Rstest / Vitest and a Model API Key, ending with a first run whose result you can inspect. - [Get started: HarmonyOS · Playground · ChatGPT subscription](/get-started/harmonyos-playground-chatgpt.md): Step-by-step setup for Midscene on HarmonyOS with Playground and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: HarmonyOS · Playground · Model API Key](/get-started/harmonyos-playground-api-key.md): Step-by-step setup for Midscene on HarmonyOS with Playground and a Model API Key, ending with a first run whose result you can inspect. - [Get started: HarmonyOS · Midscene Test · ChatGPT subscription](/get-started/harmonyos-test-chatgpt.md): Step-by-step setup for Midscene on HarmonyOS with Midscene Test and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: HarmonyOS · Midscene Test · Model API Key](/get-started/harmonyos-test-api-key.md): Step-by-step setup for Midscene on HarmonyOS with Midscene Test and a Model API Key, ending with a first run whose result you can inspect. - [Get started: HarmonyOS · Rstest / Vitest · ChatGPT subscription](/get-started/harmonyos-sdk-chatgpt.md): Step-by-step setup for Midscene on HarmonyOS with Rstest / Vitest and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: HarmonyOS · Rstest / Vitest · Model API Key](/get-started/harmonyos-sdk-api-key.md): Step-by-step setup for Midscene on HarmonyOS with Rstest / Vitest and a Model API Key, ending with a first run whose result you can inspect. - [Get started: Desktop · Playground · ChatGPT subscription](/get-started/desktop-playground-chatgpt.md): Step-by-step setup for Midscene on Desktop with Playground and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: Desktop · Playground · Model API Key](/get-started/desktop-playground-api-key.md): Step-by-step setup for Midscene on Desktop with Playground and a Model API Key, ending with a first run whose result you can inspect. - [Get started: Desktop · Midscene Test · Model API Key](/get-started/desktop-test-api-key.md): Step-by-step setup for Midscene on Desktop with Midscene Test and a Model API Key, ending with a first run whose result you can inspect. - [Get started: Desktop · Midscene Test · ChatGPT subscription](/get-started/desktop-test-chatgpt.md): Step-by-step setup for Midscene on Desktop with Midscene Test and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: Desktop · Rstest / Vitest · ChatGPT subscription](/get-started/desktop-sdk-chatgpt.md): Step-by-step setup for Midscene on Desktop with Rstest / Vitest and a ChatGPT subscription, ending with a first run whose result you can inspect. - [Get started: Desktop · Rstest / Vitest · Model API Key](/get-started/desktop-sdk-api-key.md): Step-by-step setup for Midscene on Desktop with Rstest / Vitest and a Model API Key, ending with a first run whose result you can inspect. - [Get started: Android · Midscene Test · ChatGPT subscription](/get-started/android-test-chatgpt.md): Step-by-step setup for Midscene on Android with Midscene Test and a ChatGPT subscription, ending with a first run whose result you can inspect.