crawl4ai

Crawl4AI MCP Server provides powerful web search and content extraction capabilities tuned for LLM workflows. It exposes a search tool that can query multiple engines (DuckDuckGo by default, with optional Google integration via API keys) and a read_url tool that fetches web pages and returns optimized content in formats suitable for LLM processing, including markdown with citations, simplified markdown, raw HTML to markdown, and references sections. Use these tools to gather diverse sources and convert them into concise, structured outputs that preserve traceability to original URLs.

To use the server, first start it in a Python environment as described in the installation steps. Then call the search tool with a query, optionally specifying the engine and number of results. You can request multiple engines simultaneously for broader coverage. For read_url, provide a URL and choose the preferred format for output (for example, markdown_with_citations to maintain inline references). The server will filter out non-essential content, preserve the article body and key points, and keep URL references so your AI assistant can trace back to sources. These tools together enable robust web information retrieval and content transformation tailored for LLM consumption.

Prerequisites:

• Python 3.9 or newer
• Optional virtual environment tool (e.g., venv)

Installation steps:

1. Clone the repository:

git clone https://github.com/yourusername/crawl4ai-mcp-server.git
cd crawl4ai-mcp-server

2. Create and activate a virtual environment:

python -m venv crawl4ai_env
# Linux/macOS
source crawl4ai_env/bin/activate
# Windows
.\crawl4ai_env\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

4. Install Playwright browsers (for web automation if used by the server):

playwright install

5. Run the MCP server (development):

python -m crawl4ai_mcp_server

6. Optional: If you plan to integrate via Smithery (Claude desktop or other clients) see installation via Smithery in the README:

npx -y @smithery/cli install @weidwonder/crawl4ai-mcp-server --client claude

Notes:

• Ensure you have API keys configured for Google search if you intend to use the Google engine (see config.json example in the README).
• The repository uses a FastMCP-based asynchronous design for performance.

Tips and caveats:

• If you enable Google search, populate the API key and Custom Search Engine (cse_id) in config.json as shown in README examples.
• Keep Python virtual environments isolated to avoid dependency conflicts.
• The default content format for output is markdown_with_citations to maximize traceability; you can switch formats in read_url calls.
• For long-running searches, consider paging results with num_results to manage token usage and latency.
• If encountering environment issues with Playwright, ensure all browser dependencies are installed and compatible with your OS.
• The project uses FastMCP for high-performance asynchronous operations; monitor resource usage under heavy concurrent requests.