mcp-coder-bench

mcp-coder-bench is a Rust-based benchmarking tool designed to evaluate how different MCP servers impact token usage, cost, and task completion when used with Claude Code. It supports parallel execution, multiple runtimes, and rich output formats to help you quantify the benefits of MCP-enabled workflows. The tool collects metrics such as token usage (inputs, outputs, and cache operations), estimated cost, wall time, tool calls, and task success rate, then provides statistical analyses (confidence intervals, Welch's t-test, and effect sizes) to compare scenarios. You can run benchmarks across multiple scenarios, analyze results, and compare different MCP configurations side by side. The Quick Start demonstrates initializing a configuration, validating it, running benchmarks, analyzing results, and comparing scenarios to determine the impact of MCPs on your development tasks.

Prerequisites:

• Rust 1.75+ (for building from source)
• Docker or Podman (for containerized benchmarking)
• ANTHROPIC_API_KEY environment variable (for Claude access)

Install from source or binary:

Option A: Build from source

# Ensure you have Rust and cargo installed
cargo install --path .

Option B: Build release locally (optional)

cargo build --release

Verify installation:

mcp-coder-bench --version

Environment setup (example):

export ANTHROPIC_API_KEY=your_api_key_here
export RUST_LOG=info

Notes:

• The tool can auto-detect container runtimes (Docker/Podman) if installed.
• If you prefer a containerized workflow, ensure your environment has access to the required images.

Tips and gotchas:

• Ensure ANTHROPIC_API_KEY is set to enable Claude access; without it, benchmark tasks may fail when invoking MCP-enabled prompts.
• For reproducible results, use the workspace isolation and reset strategies described in the configuration (e.g., copy strategy).
• The tool supports multiple output formats (JSON, CSV, Markdown, HTML) and can generate visualizations; use --format and --stats options during analyze to tailor reports.
• If you run in parallel, monitor resource usage (CPU/memory) to avoid contention that could skew timing metrics.
• When using container runtimes auto-detection, ensure your environment has the necessary permissions to run containers without prompts (e.g., sudo not required).
• Always validate configuration with mcp-coder-bench validate before a full run to catch connectivity or image issues early.