This MCP server exposes a set of primitive computer-control actions (launching applications, reading content, clicking, typing text, and pressing keys) that you can orchestrate with an AI agent. The server is the Rust-based backend for the Computer Use SDK, allowing you to build custom agent workflows that automate desktop tasks on macOS. You can interact with the server through provided client interfaces (CLI and Web) to issue high-level commands and compose them into automation pipelines. Use cases include automating repetitive desktop work, integrating with other MCP servers to pull data or trigger actions, and scheduling workflows to run periodically.

To get started, run the backend server and then use one of the client interfaces. The repository provides two client options: a CLI interface for debugging and a Next.js web app for a user-friendly UI. The CLI lets you send commands directly from the terminal, while the web app offers a browser-based workflow editor and execution environment. You can build custom agent workflows that combine the primitive actions into higher-level tasks, such as filling out forms, reading messages, or controlling other applications, then save and cron those workflows for automatic execution.

Prerequisites:

• Rust toolchain (rustc, cargo)
• Node.js and npm (for the CLI and web UI)

Install and run the backend:

1. Clone the repository git clone https://github.com/m13v/computer-use-ai-sdk.git cd MCP-server-client-computer-use-ai-sdk

2. Install Rust toolchain if you don’t have it curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

   Restart your shell or source your profile as instructed

3. Build/run the server (OS-level backend) cd mcp-server-os-level cargo run --bin server

Install and run the CLI interface (optional):

1. Move to the CLI client folder cd mcp-client-cli-interface

2. Install dependencies npm install

3. Set your Anthropic API key (required by the CLI for the AI model) export ANTHROPIC_API_KEY=sk-YourAnthropicKey

   On Windows: set ANTHROPIC_API_KEY=sk-YourAnthropicKey

4. Run the CLI client npx tsx main.ts

Install and run the Web app interface (optional):

1. Move to the web client folder cd mcp-client-nextjs

2. Install dependencies npm install

3. Set API key for the environment echo "ANTHROPIC_API_KEY=sk-YourAnthropicKey" > .env

4. Start the dev server npm run dev

Visit the local web page that starts after the dev server boots and begin building agent workflows.

Tips and common issues:

• The system relies on Anthropic API keys for the AI-driven components; ensure ANTHROPIC_API_KEY is set in your environment for the CLI or that .env includes ANTHROPIC_API_KEY for the web UI.
• The backend targets macOS and interacts with native desktop elements. Ensure you run the backend on a macOS machine where you want to automate tasks.
• When using the CLI, keep the terminal session open to keep the server running; the web UI runs as a separate development server.
• If you encounter permission prompts on macOS for accessibility, grant the necessary permissions to the terminal/IDE where you run the server and clients.
• The supported primitives (launch apps, read content, click, enter text, press keys) can be composed into complex workflows and scheduled with cron-like mechanisms.
• You can integrate this MCP server with other MCP servers to chain capabilities, such as feeding data into a spreadsheet or reacting to events from other automation agents.