crawlbase

Crawlbase MCP serves as a bridge between AI agents and live web content. It exposes tools that fetch and render real-time web data, including raw HTML, clean Markdown, and visual screenshots, by leveraging Crawlbase's crawling infrastructure. After configuring tokens, you can invoke commands within your MCP-enabled client (such as Claude, Cursor, or Windsurf) to request live content and have it returned in structured formats that are easier for your agents to consume. The available actions you’ll typically use are crawl to fetch HTML, crawl_markdown to extract and deliver clean, structured Markdown, and crawl_screenshot to capture page visuals. This enables AI workflows to query fresh information, render pages, or obtain compact, human-friendly representations of live websites.

To start, ensure you have tokens from Crawlbase. Then run the MCP server using your client’s standard MCP configuration (for example, in Claude or Cursor you’ll point to the npm runable MCP via npx @crawlbase/mcp@latest with the appropriate environment variables). When you issue a prompt that involves web data, instruct the agent to call crawl, crawl_markdown, or crawl_screenshot as needed, and the MCP server will handle crawling, rendering, and returning the data to your agent for further reasoning or display.

Prerequisites:

• Node.js and npm/yarn installed on your machine
• Access tokens from Crawlbase (CRAWLBASE_TOKEN and CRAWLBASE_JS_TOKEN)
• Basic familiarity with your MCP client (Claude, Cursor, Windsurf) configuration

Install steps:

1. Install the MCP server package (via npx as shown in the readme):

   • Ensure you have an environment with Node.js installed.

2. Run the MCP server in your environment:

   • Example (HTTP token-based run):
     • Set environment variables:
       • CRAWLBASE_TOKEN=your_token
       • CRAWLBASE_JS_TOKEN=your_js_token
     • npm install (if needed) and then start via your client configuration as shown in the readme, e.g., using npx @crawlbase/mcp@latest

3. Verify the server is up:

   • Use the HTTP endpoints described in the readme if you choose HTTP mode (POST /mcp, GET /health).

4. Integrate with your MCP client:

   • For Claude/Cursor/Windsurf, add the crawlbase entry in mcpServers with type stdio and the same npm command and tokens as described in the configuration examples.

Note: In HTTP mode, you can share a single server across multiple tenants by supplying per-request tokens in headers (X-Crawlbase-Token and X-Crawlbase-JS-Token).

Tips and considerations:

• Use per-request authentication in HTTP mode to support multi-tenant workloads via headers. Environment variables act as defaults; headers override them per request.
• Tokens are required for both HTML and JS-rendered data; use CRAWLBASE_TOKEN for HTML and CRAWLBASE_JS_TOKEN for JavaScript-rendered content or screenshots.
• The MCP server exposes tools to fetch raw HTML (crawl), convert content to Markdown (crawl_markdown), and capture rendered pages as screenshots (crawl_screenshot).
• When configuring in IDEs (Claude, Cursor, Windsurf), keep token values secure and avoid committing them to version control.
• If you run into rate limits or token issues, verify token scopes on Crawlbase dashboard and ensure you’re using the latest @crawlbase/mcp package.
• The example configuration uses npx to fetch the latest MCP package; pin versions if you need reproducible builds.