mcp -tester

The MCP Server Tester is a dedicated, implementation-agnostic CLI tool for validating and testing MCP servers and their tooling. It provides three focused commands: tools to verify tool calls against direct MCP API interactions, evals to ensure LLMs can reliably discover and use your tools, and compliance to check alignment with evolving MCP specifications. You can run it directly against your MCP server configuration to perform end-to-end checks and catch regressions early. The tester is designed to work without API keys for tooling tests, but evaluating LLM behavior requires an API key (e.g., ANTHROPIC_API_KEY) for the evals phase.

To use it, start by ensuring your server configuration file is ready (for example filesystem-server-config.json). Then run the tester commands with your server config to execute the desired test suite:

• Direct tool testing: run tests against your server’s tooling using direct MCP calls.
• LLM evaluation testing: run prompts and assess whether the LLMs use your tools correctly and consistently.
• Compliance testing: validate your server against the MCP spec as it evolves (WIP).

Example workflows include verifying a tool like write_file by calling it directly, and then validating via an eval that an LLM can call write_file to produce the same outcome. The tool suite is designed to be extensible with additional tests and tool definitions as your MCP server grows in capabilities.

Prerequisites:

• Node.js (14.x or newer) and npm/yarn installed
• Basic familiarity with MCP server config files and tool/test YAMLs

Step-by-step installation:

1. Install Node.js and npm if not already installed. See https://nodejs.org/ for platform-specific instructions.

2. Install or ensure npx is available (comes with npm). No global install is required for npm 9+ users when using npx directly.

3. Install the MCP Server Tester package locally (or rely on npx to fetch it on first run):

# Optional: install locally
npm install -g mcp-server-tester

# or rely on npx (no local install needed):
npx -y mcp-server-tester

4. Prepare your MCP server config file (e.g., filesystem-server-config.json) and test files (tool-tests.yaml, eval-tests.yaml) as described in the README.

5. Run a basic tools test to verify your server wiring:

npx mcp-server-tester tools tool-tests.yaml --server-config filesystem-server-config.json

6. When ready to test LLM integration, export your API key and run evals:

export ANTHROPIC_API_KEY="your-key"
npx mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json

You can also run compliance tests when the MCP tester supports it:

npx mcp-server-tester compliance --server-config filesystem-server-config.json

Tips and common issues:

• Environment variables: For evals, ensure ANTHROPIC_API_KEY (or the appropriate key for your provider) is set in your environment before running evals.
• Tool discovery: Your tool definitions must be discoverable by the tester (e.g., via the tools section in tool-tests.yaml).
• Server config alignment: The tester expects a filesystem-like example configuration (e.g., a server labeled 'filesystem' with a path-to-server script). Adjust your server-config.json accordingly to reflect your MCP server’s setup.
• Version management: The MCP spec evolves; the compliance command is marked as WIP. Use it to catch regressions but rely on your own spec interpretation for edge cases.
• Debugging: If a test fails, inspect the detailed test output and compare it against the expected tool calls or prompts to identify mismatches in required tools or call sequences.