mcp -tester

The @gleanwork/mcp-server-tester package provides a testing framework for MCP servers using Playwright fixtures. It exposes a high-level API to drive an MCP server, call tools, and assert responses with a collection of built-in matchers. You can use the provided mcp fixture to connect to your MCP server (via stdio or HTTP) and then perform operations such as listing available tools, invoking specific tools with arguments, and validating outputs against schemas or plain text. This makes it straightforward to write deterministic, CI-friendly tests that verify tool discoverability, response formats, and overall conformance to the MCP specification. The framework also supports evaluation datasets and an LLM-hosted evaluation workflow for testing tool usability and discoverability in more dynamic scenarios.

To get started, install the tester in your project, then write Playwright tests that import the mcp fixture. For example, you can call tools like read_file, verify response content, and ensure the server exposes the expected tools. The tester also ships with matchers such as toContainToolText, toMatchToolSchema, toMatchToolPattern, and more, enabling expressive assertions about tool behavior and responses.

Prerequisites:

• Node.js 22+ (as required by the package)
• npm or yarn

Install the MCP server tester and required test tooling:

npm install --save-dev @gleanwork/mcp-server-tester @playwright/test zod

Optional (for LLM-based assertions or LLM host mode):

npm install --save-dev @anthropic-ai/sdk

If you’re starting from scratch, you can also use npx to initialize a project scaffold provided by the package:

npx mcp-server-tester init

This will set up a basic test environment and example tests that you can customize for your MCP server.

Tips and notes:

• This tester is designed for MCP servers and relies on Playwright test fixtures. Use the mcp fixture to connect to your MCP server via stdio or HTTP.
• Available assertions include containsText, schema, regex, snapshot, judge, and toolsTriggered. Use them to validate tool outputs, input schemas, and LLM-driven evaluations.
• If you plan to use LLM-based judgments or LLM-host mode, you’ll want to install the Anthropic SDK package and configure provider and model details as described in the docs.
• The Quick Start shows using npx to run the tester. For local development, you can install locally and reference the package in your test files.
• Ensure Node.js 22+ is present in your environment, and that your CI environment is configured to run Playwright tests (e.g., setting up browsers if needed).