scrapfly

Scrapfly MCP Server connects your AI assistant to live web data by routing queries through Scrapfly's real-time web scraping capabilities. Once connected, your AI can request current page content, prices, listings, news, and other web data as if it were performing a live browser visit, with the ability to bypass anti-bot protections and render dynamic pages. The server exposes tools to scrape live data, extract structured JSON from pages using AI-powered extraction, capture visual screenshots, and manage data extraction pipelines. This enables your AI to respond with up-to-date information from any website, rather than relying solely on static training data.

To use Scrapfly MCP, configure the MCP client in your environment (via NPX or through your IDEs) to point at the Scrapfly MCP endpoint at https://mcp.scrapfly.io/mcp. Once connected, you can issue prompts that request live content, structured data, or screenshots. The integration supports a variety of workflows, including real-time price checks, product listings, news aggregation, and automated data extraction tasks, all delivered in a format suitable for downstream AI processing.

Prerequisites:

• Node.js and npm installed on your system
• Access to run NPX commands (no global install required)
• Optional Scrapfly API key if you plan to use authenticated endpoints

Step 1: Install Node.js (if not already installed)

• Visit https://nodejs.org/ and follow the install instructions for your OS.

Step 2: Run the MCP server using NPX

• This uses the mcp-remote package to connect to Scrapfly's MCP endpoint.

Example:

# Basic startup using NPX (no local install required)
npx mcp-remote https://mcp.scrapfly.io/mcp

Step 3: Provide credentials if required

• If Scrapfly requires an API key, export it as an environment variable before starting:

export SCRAPFLY_API_KEY=your-scrapfly-api-key
npx mcp-remote https://mcp.scrapfly.io/mcp

Step 4: Alternative configuration via mcp.config.json (Cloud Configuration) You can also configure the MCP server in a config file and point your MCP client to it, for example:

{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": ["mcp-remote", "https://mcp.scrapfly.io/mcp"]
    }
  }
}

Step 5: Verify connectivity

• Ensure the MCP client can reach https://mcp.scrapfly.io/mcp and that any required API keys are valid. Test a simple data fetch to confirm live scraping works as expected.

Tips and common issues:

• Ensure you have network access to Scrapfly endpoints and any required proxies are configured if your environment restricts direct outbound access.
• If you encounter authentication errors, double-check SCRAPFLY_API_KEY and related credentials in your environment variables.
• For heavy or rate-limited scraping tasks, consider implementing batching or throttling in your AI prompts and/or using Scrapfly's extraction pipelines to reduce repeated page loads.
• The mcp-remote package acts as a bridge; you can also pin a specific version if you need stability across environments.
• If you plan to deploy in CI/CD, use a dedicated environment with secure storage for API keys and avoid committing secrets in config files.