webscraping-ai

The WebScraping.AI MCP Server integrates WebScraping.AI capabilities into your MCP workflow, enabling chat-based questions about page content, structured data extraction, and full-page content retrieval with optional JavaScript rendering. It exposes a suite of tools you can invoke through the MCP interface to extract specific data from web pages, render pages as HTML, or obtain plain text from visible content. The server supports proxy selection, JS rendering, device emulation, and per-request controls such as timeouts and wait-for selectors, making it suitable for building complex data-gathering pipelines and connected AI agents.

Prerequisites:

• Node.js and npm installed on your machine
• An API key from WebScraping.AI

Installation steps:

1. Install the MCP server package globally or run via npx as described in the README:

env WEBSCRAPING_AI_API_KEY=your_api_key npx -y webscraping-ai-mcp

2. If you prefer a manual setup, clone the repository and install dependencies:

# Clone the repository
git clone https://github.com/webscraping-ai/webscraping-ai-mcp-server.git
cd webscraping-ai-mcp-server

# Install dependencies
npm install

# Run
npm start

3. Configure in Cursor (optional): create a .cursor/mcp.json with the following server entry, replacing the API key and optional settings as needed:

{
  "servers": {
    "webscraping-ai": {
      "type": "command",
      "command": "npx -y webscraping-ai-mcp",
      "env": {
        "WEBSCRAPING_AI_API_KEY": "your-api-key",
        "WEBSCRAPING_AI_CONCURRENCY_LIMIT": "5",
        "WEBSCRAPING_AI_ENABLE_CONTENT_SANDBOXING": "true"
      }
    }
  }
}

Notes and tips:

• Set WEBSCRAPING_AI_API_KEY to your WebScraping.AI API key; without it, requests will fail.
• WEBSCRAPING_AI_CONCURRENCY_LIMIT controls parallel requests; adjust based on your plan and rate limits.
• Enable WEBSCRAPING_AI_ENABLE_CONTENT_SANDBOXING to wrap scraped content and reduce prompt injection risk when feeding results to LLMs.
• You can specify default proxies, JS rendering behavior, and timeouts per-request to tailor scraping to each task.
• If you encounter Windows command issues, wrap the command using cmd /c as noted in the README example.
• The MCP server exposes tools for questions, field extraction, full HTML with JS rendering, text extraction, and CSS selector-based content selection; combine these tools to build robust scrapers and data pipelines.