mcpbr

mcpbr is an MCP server benchmarking tool that lets you measure and compare the effectiveness of an MCP server against a baseline agent using real GitHub SWE-bench issues. The server is implemented as a Python package that you install from PyPI and run via the mcpbr CLI. After installation, you can quickly start a run to benchmark a server by specifying the number of tasks to evaluate. The tool handles task selection, execution, and results aggregation, providing metrics that help you quantify improvements or regressions when enabling MCP-based tooling in your agent's decision process.

To use it, install the package with pip install mcpbr and then run mcpbr run -n <num_tasks> to perform a benchmark on the default SWE-bench tasks. The run command executes the server against real GitHub issues, comparing it to a baseline agent and producing reproducible results with pinned dependencies via Docker containers in other configurations. The focus is on providing hard numbers for apples-to-apples comparisons of tool-assisted versus baseline agent performance.

Prerequisites:

• Python 3.11+ installed on your system
• pip available

Installation and quick test (one-liner):

• curl -sSL https://raw.githubusercontent.com/greynewell/mcpbr/main/install.sh | bash

Manual installation and run:

• pip install mcpbr
• mcpbr run -n 1

Alternative if you prefer building from source:

• git clone https://github.com/greynewell/mcpbr.git
• cd mcpbr
• python -m pip install -e .
• mcpbr run -n 1

Notes and tips:

• The benchmark relies on real SWE-bench GitHub issues, so ensure you have network access and any required permissions to fetch data.
• Docker-based runs can pin dependencies for reproducible results; consider using the Docker images referenced in the project if you need isolated environments.
• If you encounter issues with Python dependencies, ensure you are using Python 3.11+ as indicated by the project badges.
• The mcpbr CLI uses a standard MCP workflow; you can adjust task count (-n) to scale the benchmarking workload as needed.
• When integrating into CI, consider using the one-liner install script for quick tests or pinning a specific version of mcpbr via pip.