mcp-pyautogui

The mcp-pyautogui MCP server exposes PyAutoGUI automation capabilities over stdio, allowing an AI assistant or client to control the host machine's mouse, keyboard, and screen. Once running, you can issue JSON-RPC style commands such as click, type_text, take_screenshot, get_mouse_position, and more to automate desktop tasks. The server provides a comprehensive set of tools for interacting with the desktop, including mouse movement, clicks (left, right, double), keyboard input (typing and hotkeys), screen capture, and OS information, with consistent boolean success responses or string tuples for position and size data. It is designed to be easy to integrate with Claude Desktop or any MCP-compatible client that can send and receive JSON requests over stdio.

To use it, install the Python package, start the server, and connect a client configured to the server’s stdio transport. Typical usage involves sending commands like click(100, 200), type_text("hello"), take_screenshot("shot.png"), or get_os(), and then handling the returned results. The server’s command list includes full movement and interaction controls (move_to, drag_to, double_click, scroll), text input (type_text, press_key, hotkey), and utility queries (get_mouse_position, get_screen_size, get_os). If you’re testing automation flows, you can begin with simple actions such as clicking at a coordinate, typing a short string, and capturing a screenshot to validate the sequence.

Prerequisites

• Python 3.11+
• Access to a shell with pip available
• (Optional) A GUI-enabled environment if you plan to run on headless systems with a virtual display configured

Installation steps

1. Install the MCP PyAutoGUI package from PyPI

   • Command: pip install mcp-pyautogui

2. Verify the CLI entry point is available

   • Command: mcp-pyautogui --version
   • If the entry point is not on PATH, locate the module and run via Python: python -m mcp_pyautogui --help

3. Run the MCP server (stdio transport)

   • Command: python -m mcp_pyautogui

4. Determine the executable location for configuration (optional but recommended for Claude Desktop)

   • On macOS/Linux, you might locate the script with: which mcp-pyautogui
   • Use the resulting path in your Claude Desktop configuration if you prefer a direct executable path.

5. Sample configuration for Claude Desktop (mcp-pyautogui as an MCP server)

   • Ensure your claude_desktop_config.json includes the server name and proper executable path if not relying on PATH: { "mcpServers": { "mcp-pyautogui": { "command": "/path/to/python", "args": ["-m", "mcp_pyautogui"] } } }

Notes and tips:

• The server uses stdio transport; ensure the client you connect from supports JSON-RPC style commands.
• PyAutoGUI requires appropriate permissions on some OSes (e.g., accessibility permissions on macOS). Grant necessary permissions to the terminal or IDE running the server.
• If you run headless or in containers, you may need a virtual display (e.g., Xvfb on Linux) to enable GUI interactions.
• The command list in the README documents all available tools; typical workflows start with get_os(), get_mouse_position(), and take_screenshot() to validate environment before performing automated actions.
• When using hotkey strings, follow the documented format (space-separated keys) e.g., hotkey("ctrl c").
• For reliability, confirm that the host desktop environment is ready and that any required GUI elements are accessible before running automation sequences.