ScreenPilot

ScreenPilot exposes a set of screen automation capabilities to an MCP-enabled environment, allowing a language model to take control of your desktop by capturing the screen, moving the mouse, typing text, and performing keyboard shortcuts. The server integrates tools for Screen Capture, Mouse Control, Keyboard Actions, Scrolling, Element Detection, and Action Sequences. You can leverage these tools to automate GUI tasks, gather visual context, and interact with applications in a programmable way. Once the MCP server is running, you can issue tool invocations from your LLM-driven workflows to capture screenshots, search for UI elements, perform clicks and drags, and input text or key presses as part of a larger automation script.

Prerequisites:

• Python 3.12 installed on your system
• Git installed
• Access to the ScreenPilot project repository

Installation steps:

1. Clone the repository:

git clone https://github.com/Mtehabsim/ScreenPilot.git

2. Create a Python virtual environment:

python -m venv venv

3. Activate the virtual environment:

• Windows:

venv\Scripts\activate

• macOS/Linux:

source venv/bin/activate

4. Install required packages:

pip install -r requirements.txt

5. Prepare the MCP configuration file that points to the ScreenPilot main script. Example (adjust paths to your environment):

{
  "mcpServers": {
    "device-controll": {
      "command": "pathToEnv\\venv\\Scripts\\python.exe",
      "args": [
        "pathToProject\\ScreenPilot\\main.py"
      ]
    }
  }
}

6. Save the configuration file in a location accessible to Claude AI Desktop (or your MCP client) and ensure the paths reflect your setup.
7. Run the MCP server as configured and connect your MCP client (e.g., Claude AI Desktop) to start issuing tool commands.

Tips and common issues:

• Ensure the Python virtual environment is activated before running the server to guarantee dependencies are available.
• The configuration example uses Windows-style paths. If you are on macOS/Linux, adapt paths accordingly (e.g., /home/user/... and forward slashes).
• If the server cannot locate main.py, verify the path in the args exactly matches the location of ScreenPilot/main.py.
• You may need additional system permissions for screen capture and GUI automation depending on your OS (e.g., macOS privacy settings for Screen Recording and Accessibility).
• The tool set includes Screen Capture, Mouse Control, Keyboard Actions, Scrolling, Element Detection, and Action Sequences. Combine these tools to implement robust automation pipelines.