cortex-scout

CortexScout is a Rust-based search and web extraction engine that can operate as an MCP stdio server with an optional HTTP interface. It provides capabilities for web search, token-efficient fetching, bounded crawling, schema-driven extraction, anti-bot handling with Chromium CDP rendering, and HITL fallbacks when needed. The MCP interface exposes a range of tools such as web_search, web_fetch, web_crawl, extract_fields, memory_search, and deep_research, enabling you to perform multi-hop research, structured extractions, and robust retrieval under challenging protections. To integrate, configure CortexScout as an MCP server and issue requests through the MCP protocol; you can also run the optional HTTP server to monitor health and direct traffic with a URL endpoint. Tools like visual_scout for HITL visuals and non_robot_search for last-resort rendering provide a safety net for protected targets.

Prerequisites:

• Rust toolchain (rustup, cargo)
• A Unix-like environment (Linux/macOS) or Windows with WSL

Installation steps:

1. Install Rust and cargo if not already installed:

# Rust installation (eligible for all platforms)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
export PATH="$HOME/.cargo/bin:$PATH"

2. Clone the repository and navigate to the mcp-server directory:

git clone https://github.com/cortex-works/cortex-scout.git
cd cortex-scout/mcp-server

3. Build the project (release for production):

cargo build --release

4. Run the MCP server via the configured command (as defined in MCP config):

# The MCP integration will typically invoke the binary cortex-scout-mcp with the environment and arguments specified in mcp_config

5. Optional: verify the HTTP server is up if enabled by running:

curl http://localhost:5000/health

Notes and tips:

• Use RUST_LOG=warn to avoid overwhelming MCP clients with logs on stderr.
• Adjust HTTP_TIMEOUT_SECS and MAX_CONTENT_CHARS to balance latency and data volume for your workloads.
• If using HITL or deep research features, ensure LANCEDB_URI points to a writable LanceDB instance location.
• For Windows environments, consider using the alternative command format that does not rely on the env wrapper; see docs/IDE_SETUP.md for details.
• The OPENAI/OpenRouter/LLM integration for deep research can be enabled by passing environment variables like DEEP_RESEARCH_ENABLED and related synthesis settings when launching the MCP server via the same config structure.
• Always validate your IP_LIST_PATH and PROXY_SOURCE_PATH files to ensure proper proxy rotation and request pacing.