#Integrate with Android (adb)

After connecting the Android device with adb, you can use Midscene javascript SDK to control Android devices.

#Preparation

#Install Node.js

Install Node.js 18 or higher.

#Prepare API Key

Prepare an API Key for a Vision Language (VL) model.

You can check the models and configurations supported by Midscene.js in the Model Strategy documentation.

#Install adb

adb is a command-line tool that allows you to communicate with Android devices. There are two ways to install adb:

• Method 1: Install using Android Studio
• Method 2: Install using Android Command Line Tools

Verify that adb is installed successfully:

adb --version

When you see the following output, it means adb is installed successfully:

Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64) 

#Set the ANDROID_HOME environment variable

Refer to Android Environment Variables to set the ANDROID_HOME environment variable.

Verify that the ANDROID_HOME variable is set successfully:

echo $ANDROID_HOME

When the above command has output, it means the ANDROID_HOME variable is set successfully:

/Users/your_username/Library/Android/sdk

#Connect Android Device

In the developer options of your Android device, enable 'USB debugging'. If 'USB debugging (Security settings)' exists, enable it as well. Then connect your Android device using a USB cable.

Verify the connection:

adb devices -l

When you see the following output, it means the connection is successful:

List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3

#Set up API keys for model

Set your model configs into the environment variables. You may refer to Model strategy for more details.

export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

For more configuration details, please refer to Model strategy and Model configuration.

#Integrate Midscene

#Step 1: Install dependencies

npm install @midscene/android --save-dev

yarn add @midscene/android --save-dev

pnpm add @midscene/android --save-dev

bun add @midscene/android --save-dev

deno add npm:@midscene/android --save-dev

#Step 2: Write scripts

Let's take a simple example: search for headphones on eBay using the browser in the Android device. （Of course, you can also use any other apps on the Android device.）

Write the following code, and save it as ./demo.ts

import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new AndroidDevice(devices[0].udid);

    // 👀 init Midscene agent
    const agent = new AndroidAgent(device, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.',
    });
    await device.connect();

    // 👀 open browser and navigate to ebay.com (Please ensure that the current page has a browser app)
    await agent.aiAction('open browser and navigate to ebay.com');

    await sleep(5000);

    // 👀 type keywords, perform a search
    await agent.aiAction('type "Headphones" in search box, hit Enter');

    // 👀 wait for loading completed
    await agent.aiWaitFor('There is at least one headphone product');
    // or you can use a normal sleep:
    // await sleep(5000);

    // 👀 understand the page content, extract data
    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find item in list and corresponding price',
    );
    console.log('headphones in stock', items);

    // 👀 assert by AI
    await agent.aiAssert('There is a category filter on the left');
  })(),
);

#Step 3: Run

Using tsx to run

# run
npx tsx demo.ts

After a while, you will see the following output:

[
{
  itemTitle: 'Beats by Dr. Dre Studio Buds Totally Wireless Noise Cancelling In Ear + OPEN BOX',
  price: 505.15
},
{
  itemTitle: 'Skullcandy Indy Truly Wireless Earbuds-Headphones Green Mint',
  price: 186.69
}
]

#Step 4: View the report

After the above command executes successfully, the console will output: Midscene - report file updated: /path/to/report/some_id.html. You can open this file in a browser to view the report.

#Constructor and Interface

#AndroidDevice Constructor

The AndroidDevice constructor supports the following parameters:

• deviceId: string - The device id
• opts?: AndroidDeviceOpt - Optional, the options for the AndroidDevice
  • autoDismissKeyboard?: boolean - Optional, whether to dismiss the keyboard after inputting. (Default: true)
  • keyboardDismissStrategy?: 'esc-first' | 'back-first' - Optional, the strategy to dismiss the keyboard. 'esc-first' tries ESC key first, then back key if needed. 'back-first' tries back key first, then ESC key if needed. (Default: 'esc-first')
  • androidAdbPath?: string - Optional, the path to the adb executable.
  • remoteAdbHost?: string - Optional, the remote adb host.
  • remoteAdbPort?: number - Optional, the remote adb port.
  • imeStrategy?: 'always-yadb' | 'yadb-for-non-ascii' - Optional, controls when Midscene invokes yadb for text input. 'yadb-for-non-ascii' (default) automatically uses yadb for Unicode characters (including Latin Unicode like ö, é, ñ), Chinese, Japanese, and format specifiers (like %s, %d), while pure ASCII text uses the faster native adb input text. 'always-yadb' forces yadb for all text input, providing maximum compatibility but slightly slower for pure ASCII. (Default: 'yadb-for-non-ascii')
  • displayId?: number - Optional, the display id to use. (Default: undefined, means use the current display)
  • screenshotResizeScale?: number - Optional, controls the size of the screenshot Midscene sends to the AI model. Default is 1 / devicePixelRatio, so a 1200×800 display with a device pixel ratio of 3 sends an image of roughly 400×267 to the model. Adjusting this value manually is not recommended.
  • alwaysRefreshScreenInfo?: boolean - Optional, whether to re-fetch screen size and orientation information every time. Default is false (uses cache for better performance). Set to true if the device may rotate or you need real-time screen information.

#Additional Android Agent Interfaces

Except the common agent interfaces in API Reference, AndroidAgent also provides some other interfaces:

#agent.launch()

Launch a webpage or native page.

• Type

function launch(uri: string): Promise<void>;

• Parameters:

  • uri: string - The uri to open, can be a webpage url or a native app's package name or activity name, if the activity name exists, it should be separated by / (e.g. com.android.settings/.Settings).

• Return Value:

  • Returns a Promise that resolves to void when the page is opened.

• Examples:

import { AndroidAgent, AndroidDevice } from '@midscene/android';

const device = new AndroidDevice('s4ey59');
const agent = new AndroidAgent(device);

await agent.launch('https://www.ebay.com'); // open a webpage
await agent.launch('com.android.settings'); // open a native page
await agent.launch('com.android.settings/.Settings'); // open a native page

#agent.runAdbShell()

Execute a command through adb shell on the connected device.

Note: This method wraps adb shell and forwards the command directly to the device.

• Type

function runAdbShell(command: string): Promise<string>;

• Parameters:

  • command: string - The adb shell command to execute.

• Return Value:

  • Promise<string> - Returns a Promise that resolves to the command output.

• Examples:

import { AndroidAgent, AndroidDevice } from '@midscene/android';

const device = new AndroidDevice('s4ey59');
const agent = new AndroidAgent(device);
await device.connect();

const result = await agent.runAdbShell('dumpsys battery');
// Equivalent to running `adb shell dumpsys battery`
console.log(result);

#agentFromAdbDevice()

Create a AndroidAgent from a connected adb device.

• Type

function agentFromAdbDevice(
  deviceId?: string,
  opts?: PageAgentOpt,
): Promise<AndroidAgent>;

• Parameters:

  • deviceId?: string - Optional, the adb device id to connect. If not provided, the first connected device will be used.
  • opts?: PageAgentOpt & AndroidDeviceOpt - Optional, the options for the AndroidAgent, PageAgentOpt refer to constructor, AndroidDeviceOpt refer to AndroidDevice constructor.

• Return Value:

  • Promise<AndroidAgent> Returns a Promise that resolves to an AndroidAgent.

• Examples:

import { agentFromAdbDevice } from '@midscene/android';

const agent = await agentFromAdbDevice('s4ey59'); // create a AndroidAgent from a specific adb device
const agent = await agentFromAdbDevice(); // no deviceId, use the first connected device

#getConnectedDevices()

Get all connected Android devices.

• Type

function getConnectedDevices(): Promise<Device[]>;
interface Device {
  /**
   * The device udid.
   */
  udid: string;
  /**
   * Current device state, as it is visible in
   * _adb devices -l_ output.
   */
  state: string;
  port?: number;
}

• Return Value:

  • Promise<Device[]> Returns a Promise that resolves to an array of Device.

• Examples:

import { agentFromAdbDevice, getConnectedDevices } from '@midscene/android';

const devices = await getConnectedDevices();
console.log(devices);
const agent = await agentFromAdbDevice(devices[0].udid);

#Extending Custom Interaction Actions

Use the customActions option to extend the agent's action space with your own actions defined via defineAction. When provided, these actions will be appended to the built-in ones so the agent can call them during planning.

import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { AndroidAgent, AndroidDevice } from '@midscene/android';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z
      .number()
      .int()
      .positive()
      .describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
    // carry out your clicking logic using locate + count
  },
});

const device = new AndroidDevice('your-device-id', {
  customActions: [ContinuousClick],
});
const agent = new AndroidAgent(device);

await agent.aiAction('click the red button five times');

Check Integrate with any interface for more details about defining custom actions.

#More

• For all the APIs on the Agent, please refer to API Reference.

#FAQ

#Why can't I control the device even though I've connected it?

A typical error message is:

Error:
Exception occurred while executing 'tap':
java.lang.SecurityException: Injecting input events requires the caller (or the source of the instrumentation, if any) to have the INJECT_EVENTS permission.

Please check if the device is unlocked in the developer options of the system settings.

#How to use a custom adb path, remote adb host and port?

You can use the MIDSCENE_ADB_PATH environment variable to specify the path to the adb executable, MIDSCENE_ADB_REMOTE_HOST environment variable to specify the remote adb host, MIDSCENE_ADB_REMOTE_PORT environment variable to specify the remote adb port.

export MIDSCENE_ADB_PATH=/path/to/adb
export MIDSCENE_ADB_REMOTE_HOST=192.168.1.100
export MIDSCENE_ADB_REMOTE_PORT=5037

Additionally, you can also specify the adb path, remote adb host and port through the AndroidDevice constructor.

const device = new AndroidDevice('s4ey59', {
  androidAdbPath: '/path/to/adb',
  remoteAdbHost: '192.168.1.100',
  remoteAdbPort: 5037,
});