MCPSecBench

MCPSecBench is a comprehensive security benchmark and playground for testing Model Context Protocols (MCP). The repository provides a suite of Python scripts that simulate both normal and malicious MCP servers, along with a client to interact with different backends (OpenAI, Claude, Cursor) and a set of testing utilities. The primary entry point is main.py, which orchestrates automated tests including tool and server injections, server squatting scenarios, and MITM-style evaluations. To use it, launch the main Python script via your MCP runner (the MCP server in this project is designed to be run with uvx in a Python environment). Ensure you provide your API keys (OpenAI/Anthropic) as environment variables so the client can access the LLM services during tests. The repository also includes: addserver.py (a normal computation server), maliciousadd.py (a malicious server), download.py (a server for signature checks), squatting.py (malicious server for server name squatting), mitm.py (Man-in-the-Middle attack simulator), index.js (DNS rebinding attack), and claude_desktop_config.json / cursor_config.json for specific desktop integrations. You can extend the client (client.py) to work with Claude Desktop, OpenAI, or Cursor, and you can adapt the tests to your ML models or MCP deployments by adjusting the mode and protection flags when invoking main.py.

To run the benchmark, start by launching the remote servers (e.g., download.py, and any other test servers you need) under uvx, then run the MCPSecBench orchestrator (main.py) with the appropriate mode/protection settings. The README indicates a CLI-like workflow where you can switch between Claude, OpenAI, or Cursor modes and enable different protection levels, enabling a wide range of MCP security scenarios.

Prerequisites:\n- Python 3.10+\n- A Python environment (virtualenv suggested)\n- OpenAI and/or Anthropic API access if you plan to test with these providers\n- Optional: Claude Desktop or Cursor setups if you want desktop integrations\n\nStep-by-step installation:\n1) Create and activate a virtual environment:\n python -m venv venv\n source venv/bin/activate # Linux/macOS\n venv\Scripts\activate # Windows\n\n2) Install required Python dependencies (per the README, dependencies include Starlette, Pydantic, mcp[cli], Anthropic, OpenAI, aiohttp, pyautogui, pyperclip, etc.):\n uv add starlette pydantic pydantic_settings mcp[cli] anthropic aiohttp openai pyautogui pyperclip\n # You may need to install system dependencies for pyautogui (e.g., xdotool, scrot, etc.) on your OS as noted in the README.\n\n3) Ensure you have the necessary environment variables for API access:\n export OPENAI_API_KEY=your-openai-key # Linux/macOS\n export ANTHROPIC_API_KEY=your-anthropic-key # Linux/macOS\n # On Windows use: set OPENAI_API_KEY=... and set ANTHROPIC_API_KEY=...\n\n4) Install any additional tools or servers you plan to run (e.g., Claude Desktop, Cursor configurations) by following the repository’s setup notes for those components.\n\n5) Verify Python dependencies are satisfied by running a quick import test, e.g., python -c "import starlette; print(starlette.version)".\n\n6) You’re ready to launch MCPSecBench using the MCP runner (uvx) as described in the next section.

Notes and tips:\n- The benchmark relies on several scripts that simulate both benign and malicious MCP servers (e.g., maliciousadd.py, squatting.py, mitm.py). Ensure you review and adjust paths and base directories (as recommended in the README) before running tests.\n- Some components (notably pyautogui) may require additional system dependencies; on Linux you might need apt install packages to enable GUI automation.\n- When testing tool name or server name squatting, Claude Desktop/OpenAI handlers may pick the last server with the same name; ensure the ordering matches your intended test scenario.\n- Always start remote servers before launching the foreground client orchestrator to avoid connection failures.\n- If you encounter path or configuration issues, double-check the basepath settings referenced in malicious_add.py and replace with your actual paths.\n- For clean test runs, delete /tmp/state.json between runs as indicated in the README.