web-eval-agent

The web-eval-agent MCP server provides an automated web evaluation workflow that lets you run a browser-based agent to navigate, test, and report on your web application directly from your development environment. This toolset includes web_eval_agent, which drives a browser to perform UX-focused tests, capture console logs, and collect network activity, returning a rich report you can inspect inside your editor or in your API responses. It also includes setup_browser_state to initialize and reuse login state across runs. You can trigger these tools from your IDE chat or command prompts, for example asking the agent to navigate to a URL, perform a signup flow, or verify a particular user journey, and then retrieve a detailed report with steps, screenshots, and logs.

Prerequisites:

• A development environment with a code editor and internet access
• Node.js and npm (for related tooling if needed)
• UV (Astral UV) for managing MCP tools
• Playwright in the environment (for browser automation)

Installation steps:

1. Get an API key from operative.sh/mcp and prepare your OPERATIVE_API_KEY.
2. Install UV (follow the official UV installation instructions):

curl -LsSf https://astral.sh/uv/install.sh | sh

3. Install Playwright and any required browsers:

npm install -g playwright
uvx --with playwright playwright install --with-deps

4. Add the MCP server configuration to your editor or project using the provided JSON, replacing <YOUR_KEY> with your actual API key:

"web-eval-agent": {
  "command": "uvx",
  "args": [
    "--refresh-package",
    "webEvalAgent",
    "--from",
    "git+https://github.com/Operative-Sh/web-eval-agent.git",
    "webEvalAgent"
  ],
  "env": {
    "OPERATIVE_API_KEY": "<YOUR_KEY>"
  }
}

5. Restart your editor and follow the Quick Start flow to initialize and start using the web_eval_agent tools.

Tips and common issues:

• You’ll need an OPERATIVE_API_KEY to unlock the full capabilities; store it securely and do not commit to source control.
• The web-eval-agent relies on Playwright to drive a browser; ensure browsers are installed and accessible in your environment.
• If updates are not reflected in your editor, run uv cache clean or re-run the MCP install steps to pull the latest package.
• You can trigger tools via IDE chat or through prompts, such as: "Test my app at http://localhost:3000. Use web-eval-agent." The agent will navigate, interact with elements, capture logs, and generate a UX report.
• Ensure headless_browser is set as needed (default is non-headless) to inspect UI during development.
• The setup_browser_state tool can help you sign in once and reuse cookies/local-storage for subsequent runs.