> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # Control any platform with Skills [Agent Skills](https://github.com/anthropics/skills) are a format for extending AI coding agents with specialized capabilities. Midscene provides Agent Skills that let AI coding tools (like Claude Code, Cline, etc.) drive UI automation through CLI commands. Skills work by running CLI commands directly in the terminal. The AI agent acts as the brain: it takes screenshots, analyzes the UI, and decides which actions to perform next. ## Supported platforms | Skill | Package | CLI command | Description | | --------------------------- | -------------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | Browser Automation | `@midscene/web` | `npx @midscene/web` | Browser automation with three modes: default Puppeteer headless, `--bridge` to use your own Chrome, `--cdp ` to connect via CDP | | Desktop Computer Automation | `@midscene/computer` | `npx @midscene/computer` | macOS, Windows, Linux desktop control | | Android Device Automation | `@midscene/android` | `npx @midscene/android` | Android device control via ADB | | iOS Device Automation | `@midscene/ios` | `npx @midscene/ios` | iOS device control via WebDriverAgent | | HarmonyOS Device Automation | `@midscene/harmony` | `npx @midscene/harmony` | HarmonyOS device control via HDC | ## Installation Make sure [Node.js](https://nodejs.org) is installed, then run: ```bash # General installation npx skills add web-infra-dev/midscene-skills # Claude Code npx skills add web-infra-dev/midscene-skills -a claude-code # OpenClaw npx skills add web-infra-dev/midscene-skills -a openclaw ``` Skills repository: [github.com/web-infra-dev/midscene-skills](https://github.com/web-infra-dev/midscene-skills) ## Model configuration Midscene skills require a multimodal model with strong UI localization. Configure the following environment variables — either as system environment variables or in a `.env` file in the current working directory (Midscene loads `.env` automatically): ```bash MIDSCENE_MODEL_API_KEY="your-api-key" MIDSCENE_MODEL_NAME="model-name" MIDSCENE_MODEL_BASE_URL="https://..." MIDSCENE_MODEL_FAMILY="family-identifier" ``` For supported models and configuration details, see [Use a model API key](/model-api-setup.md). ## Use skills Once installed, just describe the task in natural language to your AI coding agent. It picks the right Skill, runs the CLI, reads the screenshots, and decides what to do next — for example: > Open the photo app and tell me the first photo in the album. ## Example: Coding Agent self-verifies after writing code In this example, we ask Claude Code to develop an Electron Todo app, and after writing the code, it uses the `desktop-computer-automation` Skill to launch the app, interact with the UI, and take screenshots to verify the feature works as expected — no manual intervention or test scripts needed. **Prompt:** ``` Build an Electron Todo app with add, toggle, and delete functionality. After development, launch the app and verify with desktop automation: add 3 todos, check one off, delete one, and take a screenshot to confirm the final state is correct. ``` The coding agent autonomously completes the entire workflow: write the Todo component → launch the Electron app → connect to the desktop → take screenshots to understand the UI → interact via natural language → take screenshots to verify results. The developer only describes the intent, and Skills give the agent the ability to "see the screen and move the mouse", letting it verify its own code just like a human would. ## More use cases Skills go beyond local desktop testing. By combining different Skills, you can cover a wide range of automation scenarios: - **Desktop app testing** — Verify functionality of Electron, Qt, WPF and other desktop applications - **Remote computer control** — Operate applications on remote machines via remote desktop connections for remote ops and debugging - **Mobile app testing** — Use `@midscene/android` and `@midscene/ios` Skills to test mobile apps on real devices or simulators - **Cross-app workflows** — Chain operations across multiple apps, e.g. fetch data from browser → paste into Excel → take screenshot and send to Slack - **CI/CD integration** — Run desktop automation in headless mode on Linux CI via Xvfb, no physical display needed - **Daily task automation** — Batch form filling, scheduled screenshot monitoring, automatic file organization, etc. ## Browser connection options (optional) In default Puppeteer mode, you can override the default `1440x800` viewport with `--viewport-width ` and `--viewport-height `. These flags are only supported in default Puppeteer mode, not in `--bridge` or `--cdp` mode. In CDP mode, use a separate `--extra-http-header 'Name:Value'` option for each HTTP header added to page requests. Repeat the option on every separate CLI command that may issue requests because each command creates a new CDP session: ```bash npx @midscene/web connect \ --cdp ws://127.0.0.1:9222/devtools/browser \ --extra-http-header 'x-use-ppe:1' \ --extra-http-header 'x-tt-env:ppe_example' \ --url https://example.com npx @midscene/web act \ --cdp ws://127.0.0.1:9222/devtools/browser \ --extra-http-header 'x-use-ppe:1' \ --extra-http-header 'x-tt-env:ppe_example' \ --prompt "click the button" ``` The headers are applied before `connect --url` navigates, so the initial document request includes them. Avoid putting sensitive authentication values directly in shell history. If Chrome is installed in a non-standard location, set `MIDSCENE_CHROME_PATH` to the Chrome executable path. `MIDSCENE_MCP_CHROME_PATH` is still accepted as a temporary migration alias. ## Switch browser tabs Use `tabs` to list open tabs and `switch-tab --id ` to select one. Save the original tab ID before clicking a link that opens a new tab, then list tabs again to find the new tab: ```bash npx @midscene/web tabs # After opening the new tab: npx @midscene/web tabs npx @midscene/web switch-tab --id npx @midscene/web take_screenshot # Return to the original tab after working in the new tab: npx @midscene/web switch-tab --id ``` `tabs` returns a JSON array with `id`, `title`, `url`, and `active`. `active` identifies the page controlled by Midscene. Switching selects an existing tab without navigating or reloading it, preserving page state. Missing or closed IDs produce an error. If a site opens a tab asynchronously, run `tabs` again after it appears. Keep the connection mode consistent across commands. In default Puppeteer and CDP modes, subsequent commands use the selected tab. These modes preserve newly opened tabs without switching to them automatically. In CDP mode, pass `--cdp ` on each command, or set `MIDSCENE_CDP_ENDPOINT` and pass `--cdp`. In Bridge mode, pass `--bridge` on each command and use an updated Midscene Chrome extension. Switching activates the selected tab in the current Chrome window. Subsequent commands connect to the foreground tab, so manually switching tabs also changes which page the next command controls. `disconnect` releases the connection. In default Puppeteer mode, `close` closes the entire browser, including all tabs. ## More Please refer to the [Skills Repository](https://github.com/web-infra-dev/midscene-skills) for more details.