mcp-pyautogui

This MCP server exposes automated GUI control capabilities through PyAutoGUI, allowing you to move the mouse, perform clicks and drags, type text, press keys and hotkeys, take screenshots, locate images on screen, and query screen information. It is designed to be invoked via the MCP protocol, enabling other tools or agents to request GUI actions in a standardized way. The server is built to be cross-platform, so you can run it on Windows, macOS, or Linux, provided you have a compatible GUI environment. To operate it locally, start the server using the UV-based runner described in the installation section, then use MCP clients or the MCP Inspector to issue tool calls and inspect results. The included tools map to PyAutoGUI functionality, giving you programmatic control over mouse position and movement, keyboard input, and screen operations like screenshot capture and image matching.

When using the server, you can execute mouse control actions (move, click, drag, get position), keyboard actions (type text, press keys, combo hotkeys), and screen operations (take screenshots, determine screen size, locate image positions on screen, read pixel colors). The MCP Inspector provides a browser-based debugging interface to step through calls and observe responses, which is especially helpful for validating tool parameters and ensuring the expected GUI behavior before integrating into automation workflows.

Prerequisites: - Python 3.12+ - PyAutoGUI (and other dependencies installed automatically by the package)

Install steps:

1. Ensure Python 3.12+ is installed on your system and is accessible in your PATH.
2. Install the MCP PyAutoGUI server package from PyPI:

pip install mcp-pyautogui-server

3. (Optional but recommended) Prepare your environment for GUI automation (e.g., ensure macOS / Windows accessibility permissions are granted if needed).

4. Start the server using the UV-based runner as shown in the development/documentation section:

uv --directory /path/to/mcp-pyautogui-server run mcp-pyautogui-server

If you are using a published setup, you can also run via UV with the published package name in your environment by configuring the appropriate command (e.g., uvx mcp-pyautogui-server).

Notes: The README describes two configurations—an unpublished/development setup using uv with a directory path, and a published setup using uvx with just the package name. Choose the approach that matches your deployment scenario.

Environment and configuration tips: - The server relies on PyAutoGUI, so you need a valid GUI session. On headless servers, GUI support is required or use a virtual display. - macOS requires screen recording accessibility permissions for PyAutoGUI to simulate input; grant permissions in System Preferences under Security & Privacy. - If running from a remote session, ensure the display is accessible and PyAutoGUI can interact with the GUI surface. - For debugging and development, the MCP Inspector (invoked via npm) can connect to the running UV-based server to inspect tools and responses:

npx @modelcontextprotocol/inspector uv --directory /path/to/mcp-pyautogui-server run mcp-pyautogui-server

• If you modify the server or its configuration, re-run the UV command to pick up changes. - Common issues include missing dependencies, permission errors when controlling the GUI, or incorrect tool parameters; use the MCP Inspector to validate calls and responses step-by-step.