extract-llms-docs

llm.energy exposes an MCP server that automates extraction and packaging of llms.txt and install.md, turning web documentation into agent-friendly formats. The server offers a suite of tools for AI agents to fetch, validate, discover, and organize documentation from a wide set of websites. Core capabilities include extracting llms.txt content from URLs, validating that a URL supports llms.txt, discovering documentation URLs for a domain, listing known sites with llms.txt, and performing searches within the curated site directory. The MCP tools are designed to be integrated with common MCP clients (for example Claude Desktop or Cursor) so agents can autonomously pull up-to-date documentation for use in prompting, context-building, or task scripting. The server also provides an install.md generator to create executable installation instructions from GitHub repos or docs pages, enabling smooth onboarding for automation pipelines.

Prerequisites:

• Node.js (recommended current LTS) and npm or pnpm installed on your system
• Internet access to fetch packages from npm

Option A: Run via MCP client using npx (recommended for quick start)

1. Ensure Node.js is installed
2. Start the MCP server by configuring your MCP client to load the server:

{
  "mcpServers": {
    "llm-energy": {
      "command": "npx",
      "args": ["-y", "@llm-energy/mcp-server"]
    }
  }
}

3. Launch the MCP client with this configuration; the server will be installed and run on demand by npx.

Option B: Install locally (if you want a persistent local process)

1. Install the MCP server package globally for easy access (optional):

npm install -g @llm-energy/mcp-server

2. Run the MCP server with your preferred invocation (example using npx is shown above for convenience).
3. Point your MCP client to the running server configuration.

Option C: Use pnpm for development workflow (as in the project repository)

1. Clone the repository and install dependencies:

git clone https://github.com/nirholas/extract-llms-docs.git
cd extract-llms-docs
pnpm install

2. Run the development server:

pnpm dev

3. The MCP server can be integrated via the provided configuration in the repository README.

Notes:

• The recommended production command is to rely on npx to fetch the latest @llm-energy/mcp-server on first run.
• If you maintain your own MCP client configuration, ensure your environment allows network access to npm registry during first run.

Tips and troubleshooting:

• If npx prompts for installation, accept to fetch the latest MCP server package.
• Ensure the MCP client configuration JSON matches the expected shape, with the server key under mcpServers.
• The llm-energy MCP server exposes tools such as extract_docs, validate_url, verify_llms_txt, discover_documentation_urls, list_sites, and search_sites. Use these endpoints to build robust automation pipelines for documentation extraction and validation.
• When running in restricted environments, consider pinning the package version (e.g., @llm-energy/mcp-server@1.x) to avoid unexpected updates.
• Monitor network egress and API rate limits if you process many URLs in batch mode.
• If you encounter permission errors with npx, try running with a user-level npm cache or use a local installation approach.