web-research

This MCP server provides internet research capabilities for AI assistants, including web searching via DuckDuckGo, scraping website content with flexible selectors, and fetching raw URL content in multiple formats. It exposes three tools: web_search, scrape_website, and fetch_url. You can integrate these into your MCP client configuration and then invoke the corresponding tools through your agent prompts. The server is built with TypeScript and the MCP SDK, and relies on standard HTTP requests with support for timeouts and content length limits to keep responses efficient and reliable.

To use the tools, configure the MCP client to point to the web-research-server as shown in the provided examples. Then call web_search with a query and optional limit to retrieve structured results (title, URL, snippet). Use scrape_website with a URL and optional flags to extract content, links, and images, with options for relative-to-absolute URL conversion and content length controls. Use fetch_url to retrieve raw content from a URL, specifying the desired format (text, json, or html) and receiving status codes and headers. These capabilities allow your agents to gather information, summarize pages, or collect data for downstream processing.

Example workflows include: performing a web search for recent developments, scraping a target site to extract article content and links for indexing, and fetching API endpoints to obtain structured data for integration into a knowledge base.

Prerequisites:

• Node.js 16 or higher
• npm (or yarn)

Setup steps:

1. Clone or download the repository: git clone <repository-url> cd <repository-directory>

2. Install dependencies: npm install

3. Build the server: npm run build

4. Run the server (local development): npm run start

5. Add the server to your MCP client configuration (example provided in the README). If you deploy to a different environment, ensure the path in the command args points to the built entry file (build/index.js).

Tips and considerations:

• Ensure Node.js version compatibility (16+) in your deployment environment.
• If you encounter timeouts, adjust server timeouts or request-level timeouts in your MCP client configuration.
• The server uses axios for HTTP requests and cheerio for HTML parsing; network access and external site behavior may affect results.
• Content length limits are in place to optimize performance; adjust as needed based on your use case.
• Validate URLs before fetching and consider adding retry/ backoff strategies at the client level.
• If you customize build paths, keep the entry file path consistent with the npm script outputs (build/index.js).
• No automatic security approvals are performed by the server; you should implement appropriate security checks in your client workflow.