mirroir

mirroir-mcp is an MCP server that enables AI agents to interact with a macOS device by exposing the screen contents, tapping, typing, and gestures through an AI-driven agent workflow. It uses describe_screen to provide text elements and tap coordinates, while the agent decides the next action (tap, type_text, swipe) and the server executes it. The system is designed around a component-driven explorer that matches screen regions against predefined UI patterns, allowing reliable interactions across apps and interfaces. The server supports advanced capabilities like icon detection, component calibration, and an extensible Skill framework, plus a test runner to deterministically verify skills and workflows in CI or local development. To use it, run the server via the recommended package manager (npx mirroir-mcp) and connect your MCP clients (Claude, Cursor, OpenAI Codex, etc.) to issue natural language or scripted instructions that translate into screen actions on the connected macOS device.

Prerequisites:

• macOS 15+ with a connected iPhone for mirroring
• Node.js and npm/yarn installed (for npx execution)
• Internet access to fetch the mirroir-mcp package

Installation steps:

1. Install Node.js (if not already installed)

   • macOS: using the official installer from nodejs.org or via a package manager

2. Install and run mirroir-mcp via npx

npx -y mirroir-mcp install

This will install the MCP server and any necessary dependencies. You can also run the server directly with:

npx -y mirroir-mcp

3. Alternative installation methods mentioned in the project docs:

• Homebrew:

brew tap jfarcand/tap && brew install mirroir-mcp

Notes:

• The first time you take a screenshot, macOS will prompt for Screen Recording and Accessibility permissions. Grant both.
• Per-client setup examples are provided in the docs for Claude Code, GitHub Copilot, Cursor, and OpenAI Codex, showing how to add the mirroir-mcp server to various clients.

Tips and common issues:

• After startup, the server logs its effective configuration values to a startup dump (mirroir.log) at ~/.mirroir-mcp/mirroir.log for troubleshooting.
• Icon detection support has been added via CoreML models placed in ~/.mirroir-mcp/models/. The describe_screen flow now recognizes icons, buttons, and toggles in addition to OCR text.
• You can calibrate components with calibrate_component to fine-tune the UI pattern matching (useful for multi-row elements or complex layouts).
• For multi-client setups, use per-client configuration blocks similar to the examples in the docs (JSON blocks shown in per-client setup details).
• If you encounter permission prompts, ensure Screen Recording and Accessibility permissions are granted for the user running the MCP server.
• The server supports 32 tools and an extensible Skills system (placed in ~/.mirroir-mcp/skills), plus a test runner to deterministically verify skills.
• The MCP server provides an internal test runner with options for JUnit output and verbose logs to aid CI integration.