#Desktop environment setup

Midscene automates desktop applications across Windows, macOS, and Linux using native mouse and keyboard controls or Remote Desktop Protocol (RDP), with built-in multi-display support. It works with Electron, Qt, native desktop apps, and cross-application workflows.

This guide prepares the desktop environment: grant screen capture and input permissions, and install the dependencies Linux needs. Playground, Midscene Test, and the JavaScript SDK all rely on them.

#Prepare the desktop environment

Midscene requires screen capture and mouse/keyboard control permissions on the target desktop. Configure local desktop control or a remote Windows desktop (RDP) below.

#Local desktop permissions and dependencies

• macOS: Grant two separate system permissions to the terminal or IDE that launches Playground or runs your scripts (such as Terminal, iTerm2, VS Code, Cursor, or WebStorm):
  1. Accessibility (for mouse and keyboard control): Enable your terminal or IDE in System Settings > Privacy & Security > Accessibility (see nut.js macOS setup).
  2. Screen Recording (for desktop screenshots): Enable your terminal or IDE in System Settings > Privacy & Security > Screen Recording. After granting access, completely quit and relaunch the terminal or IDE before starting Playground.
• Windows: No extra setup is required for standard applications. However, Windows User Interface Privilege Isolation (UIPI) blocks non-elevated processes from injecting mouse clicks or keystrokes into windows running as Administrator (the cursor still moves to the target coordinates, but clicks and key presses are silently dropped). Prefer running the target app without Administrator privileges; if it must run elevated, launch the terminal or Node.js process running Midscene as Administrator as well (see Windows: clicks have no effect on some apps).
• Linux (desktop GUI): Install ImageMagick for screen capture support.
• Linux (headless server / CI): To run desktop automation on a headless Linux host (such as GitHub Actions), install Xvfb and ImageMagick, then enable headless mode to start a virtual display automatically:

# Install dependencies
sudo apt-get install -y xvfb x11-xserver-utils imagemagick

When using Xvfb, make sure the target application is started inside the same virtual display environment (not needed on machines with a physical display).

#Remote Windows desktop over RDP (optional)

To automate a remote Windows desktop over RDP, install FreeRDP on the machine running Midscene and enable Remote Desktop on the target Windows host.

Playground controls local displays only. To connect to a remote Windows session in code, see RDP options in the Desktop JavaScript SDK.

#Verify

When you grant the permissions and install the dependencies above, verify the environment in Playground. Open Get started, where Desktop and Playground are already selected, choose how to connect a model, and follow the steps to run one aiAct action and one aiAssert assertion. A passing assertion confirms that screen capture, mouse and keyboard input, and model calls work together.

#FAQ

#macOS: Script cannot move the mouse or press keys

Controlling the mouse and keyboard on macOS requires Accessibility permissions. Open System Settings > Privacy & Security > Accessibility and make sure the toggle is enabled for the terminal or IDE running your script (such as Terminal, iTerm2, VS Code, Cursor, or WebStorm).

If the toggle is already enabled but input still does not work, remove the app from the Accessibility list (using the - button), add it back, and restart the app—macOS occasionally caches stale permission signatures after app updates.

#Windows: Clicks have no effect on some apps

If the mouse cursor moves to the right coordinates but clicks or keystrokes have no effect on a specific application window (while working normally in other apps like a browser or Notepad), check whether the target application is running as Administrator (elevated). Windows User Interface Privilege Isolation (UIPI) silently drops input events sent from a standard-privilege process to an elevated window without throwing an error.

How to fix it:

1. Recommended: Run the target application with standard user privileges (uncheck "Run this program as an administrator" in its shortcut or compatibility properties).
2. Alternative: If the target application must run as Administrator, launch the terminal (such as PowerShell, Windows Terminal, or your IDE) running Midscene as Administrator as well so both processes share the same privilege level.

Note: System-level shortcuts such as Win+Tab are handled directly by the Windows shell and still work even when UIPI blocks in-window input, which is why global shortcuts may work while clicks inside the app do not. Whenever the cursor reaches the right spot but clicks do nothing, check for privilege elevation first.

#Linux: Screenshots or interactions fail on a headless server

Headless Linux environments (such as CI containers) have no physical display. Install Xvfb and ImageMagick, and enable headless mode when creating the Agent:

sudo apt-get install -y xvfb x11-xserver-utils imagemagick

Enable in code:

const agent = await agentForComputer({ headless: true });

Or enable via an environment variable:

MIDSCENE_COMPUTER_HEADLESS_LINUX=true npx tsx example.ts