> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.

# Integrate with any interface

You can use Midscene Agent to control any interface—such as IoT devices, in-house apps, and in-vehicle displays—by implementing a UI operation class that conforms to `AbstractInterface`.

After implementing the UI operation class, you get the full capabilities of Midscene Agent:

- the TypeScript GUI Automation Agent SDK, supporting integration with any interface
- the playground for debugging
- controlling the interface with YAML scripts
- Skills support through CLI commands

## Demo and community project

We have prepared a demo project for you to learn how to define your own interface class. It's highly recommended to check it out.

- [Demo Project](https://github.com/web-infra-dev/midscene-example/tree/main/custom-interface) - A simple demo project that shows how to define your own interface class

- [Android (adb) Agent](https://github.com/web-infra-dev/midscene/blob/main/packages/android/src/device.ts) - This is the Android (adb) Agent for Midscene that implements this feature

- [iOS (WebDriverAgent) Agent](https://github.com/web-infra-dev/midscene/blob/main/packages/ios/src/device.ts) - This is the iOS (WebDriverAgent) Agent for Midscene that implements this feature

There are also some community projects that use this feature:

- [midscene-ios](https://github.com/lhuanyu/midscene-ios) - A project driving the OSX "iPhone Mirroring" app with Midscene

## Set up API keys for model

The following example uses `qwen3.7-plus` through Alibaba Cloud:

```bash
export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
export MIDSCENE_MODEL_API_KEY="your-api-key"
export MIDSCENE_MODEL_NAME="qwen3.7-plus"
export MIDSCENE_MODEL_FAMILY="qwen3"
```

Replace `your-api-key` with your API Key.

> To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Use a model API key](/model-api-setup.md).

For all configuration options, see [Model configuration](/model-config.md).

## Implement your own interface class

### Key concepts

- The `AbstractInterface` class: a predefined abstract class that can connect to the Midscene Agent
- The **action space**: a set of actions that describe the actions that can be performed on the interface. This will affect how the AI model plans the actions and executes them

### Step 1. Clone and setup from the demo project

We provide a demo project that runs all the features of this document below. It's the fastest way to get started.

```bash
# prepare the environment
git clone https://github.com/web-infra-dev/midscene-example.git
cd midscene-example/custom-interface
npm install
npm run build

# run the demo
npm run demo
```

### Step 2. Implement your interface class

Define a class that extends the `AbstractInterface` class, and implement the required methods.

You can get the sample implementation from the [`./src/sample-device.ts`](https://github.com/web-infra-dev/midscene-example/blob/main/custom-interface/src/sample-device.ts) file. Let's take a glance at it.

```typescript
import type { DeviceAction, Size } from '@midscene/core';
import { getMidsceneLocationSchema, z } from '@midscene/core';
import {
  type AbstractInterface,
  defineAction,
  defineActionTap,
  defineActionInput,
  // ... other action imports
} from '@midscene/core/device';

export interface SampleDeviceConfig {
  deviceName?: string;
  width?: number;
  height?: number;
}

/**
 * SampleDevice - A template implementation of AbstractInterface
 */
export class SampleDevice implements AbstractInterface {
  interfaceType = 'sample-device';
  private config: Required<SampleDeviceConfig>;

  constructor(config: SampleDeviceConfig = {}) {
    this.config = {
      deviceName: config.deviceName || 'Sample Device',
      width: config.width || 1920,
      height: config.height || 1080,
    };
  }

  /**
   * Required: Take a screenshot and return base64 string
   */
  async screenshotBase64(): Promise<string> {
    // TODO: Implement actual screenshot capture
    console.log('📸 Taking screenshot...');
    return 'data:image/png;base64,...'; // Your screenshot implementation
  }

  /**
   * Required: Get interface dimensions
   * The width and height here refer to the logical size of the interface, not considering the device pixel ratio (dpr). The coordinates obtained from actions like defineActionTap are also based on this logical coordinate system. You can convert logical coordinates to physical coordinates in your action implementations if needed.
   */
  async size(): Promise<Size> {
    return {
      width: this.config.width,
      height: this.config.height,
    };
  }

  /**
   * Required: Define available actions for AI model
   */
  actionSpace(): DeviceAction[] {
    return [
      // Basic tap action
      defineActionTap(async (param) => {
        // TODO: Implement tap at param.locate.center coordinates
        await this.performTap(param.locate.center[0], param.locate.center[1]);
      }),

      // Text input action  
      defineActionInput(async (param) => {
        // TODO: Implement text input
        await this.performInput(param.locate.center[0], param.locate.center[1], param.value);
      }),

      // Custom action example
      defineAction({
        name: 'CustomAction',
        description: 'Your custom device-specific action',
        paramSchema: z.object({
          locate: getMidsceneLocationSchema(),
          // ... custom parameters
        }),
        call: async (param) => {
          // TODO: Implement custom action
        },
      }),
    ];
  }

  async destroy(): Promise<void> {
    // TODO: Cleanup resources
  }

  // Private implementation methods
  private async performTap(x: number, y: number): Promise<void> {
    // TODO: Your actual tap implementation
  }

  private async performInput(x: number, y: number, text: string): Promise<void> {
    // TODO: Your actual input implementation  
  }
}
```

The key methods that you need to implement are:

- `screenshotBase64()`, `size()`: help the AI model to get the context of the interface
- `actionSpace()`: an array of `DeviceAction` objects defining the actions that can be performed on the interface. AI model will use these actions to perform the actions. Midscene has provided a set of predefined action spaces for the most common interfaces and devices. And there is also a method to define any custom action.

Use these commands to run the agent:

- `npm run build` to rebuild the agent
- `npm run demo` to run the agent with javascript
- `npm run demo:yaml` to run the agent with yaml script

### Step 3. Test the agent with the playground

Attach a playground server to the agent, and you can test the agent in the web browser.

```ts
import 'dotenv/config'; // read Midscene environment variables from .env file
import { playgroundForAgent } from '@midscene/playground';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// instantiate device and agent
const device = new SampleDevice();
await device.launch();
const agent = new Agent(device);

// launch playground
const server = await playgroundForAgent(agent).launch();

// close playground
await sleep(10 * 60 * 1000);
await server.close();
console.log('Playground closed!');
```

### Step 4. Add Skill support (optional)

[Agent Skills](https://github.com/anthropics/skills) let AI coding assistants (Claude Code, Cline, etc.) drive your custom interface through CLI commands. Learn more in the [Skills documentation](/skills.md).

Add a CLI entry file to your npm package (e.g., `./src/cli.ts`):

```ts
#!/usr/bin/env node
import { runSkillCLI } from '@midscene/core/skill';
import { SampleDevice } from './sample-device';

runSkillCLI({
  DeviceClass: SampleDevice,
  scriptName: 'my-device',
});
```

Then add a `bin` field to your `package.json`:

```json
{
  "bin": {
    "my-device": "./dist/cli.js"
  }
}
```

Once published, AI coding assistants can control your custom interface via `npx my-device`.

For Skill authoring guidelines and more examples, see the [midscene-skills](https://github.com/web-infra-dev/midscene-skills) repository.

### Step 5. Release the npm package, and let your users use it

The agent and interface class have been exported in `./index.ts` file. Now you can publish it to npm.

Fill the `name` and `version` in the `package.json` file, and then run the following command:

```bash
npm publish
```

A typical usage of your npm package is like this:

```typescript
import 'dotenv/config'; // read Midscene environment variables from .env file
import { playgroundForAgent } from '@midscene/playground';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// instantiate device and agent
const device = new SampleDevice();
await device.launch();
const agent = new Agent(device);

await agent.aiAct('click the button');
```

### Step 6. Invoke your class in Midscene CLI and YAML script

Write a yaml script with the `interface` section to invoke your class.

```yaml
interface:
  module: 'my-pkg-name'
  # export: 'MyDeviceClass' # use this if this is a named export

config: 
  output: './data.json'
```

This config works same as this:

```typescript
import MyDeviceClass from 'my-pkg-name';
const device = new MyDeviceClass();
const agent = new Agent(device, {
  output: './data.json',
});
```

Other fields in the yaml script are the same as the [yaml script](/automate-with-scripts-in-yaml.md).

## API reference

### Agent constructor

```typescript
import { Agent } from '@midscene/core/agent';
import type { AbstractInterface } from '@midscene/core';
```

Create an agent by pairing your custom `AbstractInterface` implementation with the standard constructor:

```typescript
const device = new SampleDevice({
  deviceName: 'Demo Panel',
  width: 1920,
  height: 1080,
});

const agent = new Agent(device, {
  aiContexts: {
    aiAct: 'Dismiss pop-ups before every task.',
  },
  generateReport: true,
  customActions: [], // optional additional custom DeviceAction list
});
```

- `device: AbstractInterface` (required): Any class that fulfills `screenshotBase64`, `size`, and `actionSpace`. This is where you translate Midscene actions into real I/O calls for your hardware or desktop app.
- `options?: PageAgentOpt`: Shares the same option bag as the browser and mobile agents described in the [API constructors](/reference.md#common-parameters). Commonly used fields include `generateReport`, `reportFileName`, `aiContexts`, `cacheId`, `modelConfig`, `createOpenAIClient`, `customActions`, and `onTaskStartTip`. For guidance on writing and supplying context, see [Provide application knowledge to a GUI Agent](/provide-application-knowledge-to-gui-agent.md).
- The resulting agent instantly unlocks the regular automation surfaces: `aiAct`/`aiTap` APIs, YAML runner (`interface` block), [playground](#playgroundforagent-function), Skills CLI support, and reporting pipeline.

### `AbstractInterface` class

```typescript
import { AbstractInterface } from '@midscene/core';
```

`AbstractInterface` is the key class for the agent to control the interface.

These are the required methods that you need to implement:

- `interfaceType: string`: define a name for the interface, this will not be provided to the AI model
- `screenshotBase64(): Promise<string>`: take a screenshot of the interface and return the base64 string with the `'data:image/` prefix
- `size(): Promise<Size>`: the size of the interface, which is an object with the `width` and `height` properties
- `actionSpace(): DeviceAction[] | Promise<DeviceAction[]>`: the action space of the interface, which is an array of `DeviceAction` objects. Use predefined actions or define any custom action.

Type signatures:

```ts
import type { DeviceAction, Size, UIContext } from '@midscene/core';
import type { ElementNode } from '@midscene/shared/extractor';

abstract class AbstractInterface {
  // Required
  abstract interfaceType: string;
  abstract screenshotBase64(): Promise<string>;
  abstract size(): Promise<Size>;
  abstract actionSpace(): DeviceAction[] | Promise<DeviceAction[]>;

  // Optional lifecycle/hooks
  abstract destroy?(): Promise<void>;
  abstract describe?(): string;
  abstract beforeInvokeAction?(actionName: string, param: any): Promise<void>;
  abstract afterInvokeAction?(actionName: string, param: any): Promise<void>;
}
```

These are the optional methods that you can implement:

- `destroy?(): Promise<void>`: destroy the interface
- `describe?(): string`: describe the interface, this may be used for the report and the playground. But it will not be provided to the AI model.
- `beforeInvokeAction?(actionName: string, param: any): Promise<void>`: a hook function before invoking an action in action space
- `afterInvokeAction?(actionName: string, param: any): Promise<void>`: a hook function after invoking an action

### The action space

Action space is the set of actions that can be performed on the interface. AI model will use these actions to perform the actions. All the descriptions and parameter schemas of the actions will be provided to the AI model.

To help you easily define the action space, Midscene has provided a set of predefined action spaces for the most common interfaces and devices. And there is also a method to define any custom action.

This is how you can import the utils to define the action space:

```typescript
import {
	type ActionTapParam,
	defineAction,
	defineActionTap,
} from "@midscene/core/device";
```

#### The predefined actions

These are the predefined action spaces for the most common interfaces and devices. You can expose them to the customized interface by implementing the call method of the action.

You can find the parameters of the actions in the type definition of these functions.

- `defineActionTap()`: define the tap action. This is also the function to invoke for the `aiTap` method.
- `defineActionDoubleClick()`: define the double click action
- `defineActionInput()`: define the input action. This is also the function to invoke for the `aiInput` method. This is also the function to invoke for the `aiInput` method.
- `defineActionKeyboardPress()`: define the keyboard press action. This is also the function to invoke for the `aiKeyboardPress` method.
- `defineActionScroll()`: define the scroll action. This is also the function to invoke for the `aiScroll` method.
- `defineActionDragAndDrop()`: define the drag and drop action
- `defineActionLongPress()`: define the long press action
- `defineActionSwipe()`: define the swipe action

#### Define a custom action

You can define your own action by using the `defineAction()` function. You can also use this method to define more actions for the [PuppeteerAgent](/integrate-with-puppeteer.md), [AgentOverChromeBridge](/sdk/chrome-bridge.md#constructor), and [AndroidAgent](/sdk/android.md).

API Signature:

```typescript
import type { ExecutorContext } from "@midscene/core";
import { defineAction } from "@midscene/core/device";

defineAction(
  {
    name: string,
    description: string,
    paramSchema: z.ZodType<T>;
    planningFeedbackMaxLength?: number | 'unlimited';
    call: (
      param: z.infer<z.ZodType<T>>,
      context?: ExecutorContext,
    ) => Promise<void>;
  }
)
```

- `name`: the name of the action, AI model will use this name to invoke the action
- `description`: the description of the action, AI model will use this description to understand what the action is doing. For complex actions, you can provide a more detailed example here.
- `paramSchema`: the [Zod](https://www.npmjs.com/package/zod) schema of the parameters of the action, AI model will help to fill the parameters according to this schema
- `planningFeedbackMaxLength`: optional character limit for `context.task.planningFeedback`. Omit it to use the core default of 500 characters, set a number to use an action-specific limit, or set `'unlimited'` to disable truncation for this action.
- `call`: the function to invoke the action, you can get the parameters from the `param` parameter which conforms to the `paramSchema`
- `context`: optional execution context. The action's return value populates `task.output` (for your API callers and the report) and is **not** sent to the planner. To pass a planner-facing summary to the next planning round, set `context.task.planningFeedback`; `planningFeedbackMaxLength` controls whether and where it is truncated.

Example:

```typescript
defineAction({
  name: 'MyAction',
  description: 'My action',
  paramSchema: z.object({
    name: z.string(),
  }),
  call: async (param) => {
    console.log(param.name);
  },
});
```

If you want to get a param about the location of some element, you can use the `getMidsceneLocationSchema()` function to get the specific zod schema.

A more complex example about defining a custom action:

```typescript
import { getMidsceneLocationSchema } from "@midscene/core/device";

defineAction({
  name: 'LaunchApp',
  description: 'A an app on screen',
  paramSchema: z.object({
    name: z.string().describe('The name of the app to launch'),
    locate: getMidsceneLocationSchema().describe('The app icon to be launched'),
  }),
  call: async (param) => {
    console.log(`launching app: ${param.name}, ui located at: ${JSON.stringify(param.locate.center)}`);
  },
});
```

### `playgroundForAgent` function

```typescript
import { playgroundForAgent } from '@midscene/playground';
```

The `playgroundForAgent` function creates a playground launcher for a specific Agent, allowing you to test and debug your custom interface Agent in a web browser.

#### Function signature

```typescript
function playgroundForAgent(agent: Agent): {
  launch(options?: LaunchPlaygroundOptions): Promise<LaunchPlaygroundResult>
}
```

#### Parameters

- `agent: Agent`: The Agent instance to launch the playground for

#### Return value

Returns an object containing a `launch` method.

#### `launch` method options

```typescript
interface LaunchPlaygroundOptions {
  /**
   * Port to start the playground server on
   * @default 5800
   */
  port?: number;

  /**
   * Whether to automatically open the playground in browser
   * @default true
   */
  openBrowser?: boolean;

  /**
   * Custom browser command to open playground
   * @default 'open' on macOS, 'start' on Windows, 'xdg-open' on Linux
   */
  browserCommand?: string;

  /**
   * Whether to show server logs
   * @default true
   */
  verbose?: boolean;

  /**
   * Unique identifier for the playground server instance
   * Same ID shares playground chat history
   * @default undefined (generates random UUID)
   */
  id?: string;
}
```

#### `launch` method return value

```typescript
interface LaunchPlaygroundResult {
  /**
   * The playground server instance
   */
  server: PlaygroundServer;

  /**
   * The server port
   */
  port: number;

  /**
   * The server host
   */
  host: string;

  /**
   * Function to close the playground
   */
  close: () => Promise<void>;
}
```

#### Usage example

```typescript
import 'dotenv/config';
import { playgroundForAgent } from '@midscene/playground';
import { SampleDevice } from './sample-device';
import { Agent } from '@midscene/core/agent';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// Create device and agent instances
const device = new SampleDevice();
const agent = new Agent(device);

// Launch playground
const result = await playgroundForAgent(agent).launch({
  port: 5800,
  openBrowser: true,
  verbose: true
});

console.log(`Playground started: http://${result.host}:${result.port}`);

// Close playground when needed
await sleep(10 * 60 * 1000); // Wait 10 minutes
await result.close();
console.log('Playground closed!');
```

## FAQ

**My interface-controller is general-purpose; can it be included in this document?**

Yes, we are happy to gather creative projects and list them in this document.

[Raise an issue](https://github.com/web-infra-dev/midscene/issues) to us when it's ready.