mcp-llms-txt-explorer

The MCP LLMS.txt Explorer is a server that helps you discover and analyze websites that implement the llms.txt standard. It exposes tools to inspect whether a website provides llms.txt or llms-full.txt files and to parse and validate their contents, giving you structured data about compliant sites. The primary tools are check_website and list_websites: check_website accepts a domain URL and returns file locations and validation status, while list_websites returns a catalog of known llms.txt-enabled sites and can be filtered by file type to show llms.txt versus llms-full.txt results. You can integrate this server into your MCP workflow to quickly audit websites and gather metadata for downstream tasks.

To use the server effectively, start it through your configured command (for example, using Node to run the built index). Then issue commands to check individual websites or to enumerate supported sites, leveraging any MCP client you prefer. Documentation and tooling in the repo describe how to invoke the check_website and list_websites capabilities from your MCP client and how to interpret the structured responses.

Prerequisites:

• Node.js installed (recommended version aligned with the project, e.g., v14+ or v16+).
• pnpm installed for development workflow (npm install -g pnpm).
• Git installed to clone the repository.

Manual installation steps:

1. Clone the repository: git clone https://github.com/thedaviddias/mcp-llms-txt-explorer.git cd mcp-llms-txt-explorer
2. Install dependencies: pnpm install
3. Build the server: pnpm run build
4. Run the server locally (example): node build/index.js

Optional Smithery-based installation:

• Use Smithery to install the server as a client integration, following the instructions on the Smithery page for this MCP server.

Configuration for Claude Desktop or MCP clients:

• Prepare your mcpServers configuration as shown in the README, specifying either a direct node invocation to the built index.js or an npx-based approach for on-demand execution.

Tips and common considerations:

• The server communicates over stdio, so debugging can be challenging; consider using the MCP Inspector tooling for debugging sessions.
• If you prefer not to run a local Node build, you can opt for an npx-based invocation as shown in the README, which downloads and executes the package on demand.
• Ensure the build output path in the mcp_config matches your actual runtime path (e.g., build/index.js).
• When configuring Claude Desktop or other MCP clients, you can switch between launching via node directly or via npx for easier deployment, depending on your environment.
• Environment variables can be added to the mcp_config if your deployment requires specific runtime settings (example placeholder provided).