patronus

Patronus MCP Server exposes a standardized interface to manage Patronus-powered evaluations, batch evaluations, experiments, and evaluator management. You can initialize the server with an API key either by passing it on the command line when starting the server or by setting the PATRONUS_API_KEY environment variable. Once running, you can interact with endpoints like initialize, evaluate, batch_evaluate, run_experiment, list_evaluator_info, and create_criteria through the MCP framework. The server supports single evaluations via a RemoteEvaluatorConfig, batch evaluations with multiple evaluators, running experiments with datasets and custom evaluators, and introspection of available evaluators and criteria for easier configuration. The tooling is designed to plug into Patronus-enabled workflows and can be exercised via the provided test Live script or programmatic calls to mcp.call_tool. In practice, you would initialize with your project and API key, perform evaluations against prompts and references, and progressively run experiments to compare evaluator configurations and adapters across datasets.

Prerequisites:

• Python 3.8+ (or as required by the project)
• Git
• Optional: virtual environment tools (venv is fine)

1. Clone the repository

git clone https://github.com/yourusername/patronus-mcp-server.git
cd patronus-mcp-server

2. Create and activate a virtual environment

• Unix/macOS

python -m venv .venv
source .venv/bin/activate

• Windows

python -m venv .venv
.venv\Scripts\activate

3. Install main and development dependencies

uv pip install -e .
uv pip install -e ".[dev]"

4. Run the server (example)

export PATRONUS_API_KEY=your_api_key_here   # or provide via --api-key when starting
python src/patronus_mcp/server.py

Tips and caveats:

• Provide your Patronus API key either via the command line (as part of the server startup) or through the PATRONUS_API_KEY environment variable.
• The server supports single and batch evaluations using RemoteEvaluatorConfig, and supports custom evaluators via adapter classes for experiments.
• For experiments, you can mix remote evaluators with custom adapters; ensure any custom adapter/module paths are importable in the runtime environment.
• If you run tests (tests/test_live.py), you may pass the API key on the command line or export it in the environment to exercise the endpoints.
• When debugging, inspect logs printed by the server to confirm endpoints are wired correctly and that evaluator configurations are loaded as expected.