the -company

TheMCPCompany is a benchmark and MCP server framework for evaluating tool-calling agents across a wide set of REST APIs. It provides thousands of task-specific tools packaged as MCP servers that agents can call to interact with real-world services. The goal is to assess how well agents can discover, select, and compose tools to solve complex tasks without relying on browser-based interactions. To get started, run the MCP server container (e.g., via Docker) and then deploy or connect to the evaluation environment (OpenHands) and the agent runner. The repository references additional instructions for running TheAgentCompany tasks and Azure task setups, as well as a dedicated README for running MCP servers. The key capabilities you’ll find are tool retrieval, tool execution through REST endpoints, and ground-truth tool annotations used to evaluate agent accuracy and efficiency. Once the MCP server is up, you can query available tools, invoke them with structured prompts, and observe the results returned by each tool invocation to steer subsequent agent reasoning and actions.

Prerequisites:

• Docker installed and running on your machine (or a compatible container runtime)
• Git installed
• Optional: access credentials if your MCP server image requires authentication

Steps:

1. Clone the repository that hosts TheMCPCompany tools and evaluation setup (the MCP server assets referenced by the README).
2. Authenticate to any required registries or services as documented in the repository (if applicable).
3. Build or pull the MCP server image. If using Docker as suggested, you can pull the prebuilt image or build from a provided Dockerfile:
   • docker pull the-mcp-company/mcp-server:latest
   • or, if you have a Dockerfile, run: docker build -t the-mcp-company/mcp-server:latest .
4. Run the MCP server container with required environment variables (PORT, API keys, etc.). Example:
   • docker run -i -p 8080:8080 -e MCP_API_KEY=your-api-key the-mcp-company/mcp-server:latest
5. Verify the server is reachable at the configured port (e.g., http://localhost:8080).
6. Follow any additional setup steps described in the repository for connecting agents and evaluation tools (OpenHands, mcp_servers README, and Azure task setup if needed).

Notes and tips:

• The MCP server exposes a large set of task-specific tools. Tool retrieval performance depends on the underlying indexing and querying strategy used by the server; ensure your agent is configured to query and filter through relevant tools efficiently.
• If you encounter connectivity issues, check container networking, port mappings, and environment variable names (e.g., MCP_API_KEY, PORT).
• Use the provided OpenHands and evaluation harnesses to run end-to-end experiments and compare agent performance against ground-truth tools.
• Review the mcp_servers/README.md and OpenHands/README.md in the repository for exact commands and dependencies tailored to this project.
• If you need to customize tool discovery or add new REST-based tools, you’ll typically extend the MCP server configuration with new tool endpoints and update the tool catalog accordingly.