mcp -evaluations-skills

This MCP server provides an automated evaluation workflow for MCP servers, focusing on tool discovery, health checks, functional testing, and error handling. It exposes an API-based interface that you can exercise with curl (and jq for JSON parsing) to verify health, list available tools, call tools with valid arguments, and deliberately trigger errors to assess resilience. To use it, first ensure the evaluator server is running and reachable at its configured host and port (default http://localhost:3030). The quickstart workflow demonstrates how to check health, list tools, invoke a tool, and generate an error to test the quality checks. You can script these calls to automate repeated evaluations as part of a CI workflow and use the evaluation rubric to gauge performance over time.

Prerequisites:

• curl
• jq
• Optional: bun or node for local automation

Install steps:

1. Clone the MCP Evaluations Skill repository git clone https://github.com/your-org/mcp-evaluations-skills.git cd mcp-evaluations-skills

2. Install dependencies

   • If a package.json is present (Node.js): npm install
   • If using Bun (optional): bun install

3. Run the MCP Evaluations Skill server

   • If a start script is defined: npm run start
   • Or run directly with Node: node server.js

4. Verify the server is running (default port 3030) curl -s http://localhost:3030/health

Note: If you configure a different port, ensure you update your requests accordingly.

Tips and considerations:

• The evaluator expects health at /health and tools discovery/calling via /mcp endpoint. If you customize ports or paths, keep consistency across tests.
• Use curl + jq for quick validation of responses; the README examples demonstrate typical payloads for tools/list and tools/call.
• Environment variables such as PORT and LOG_LEVEL can help with logging and running in containers or CI environments.
• If your server uses a different start command or a Docker image, adjust the mcp_config accordingly and ensure the port is exposed.
• Common issues: server not reachable (check network/port), invalid tool name or arguments (ensure correct input shapes), and JSON-RPC formatting errors.
• For CI automation, you can script the health check, tools/list, and a sample tools/call sequence to automatically verify tool availability and basic correctness.