llama _mcp_proxy

This MCP Proxy acts as a bridge between your llama-server instance and MCP-compatible tool servers. It intercepts chat completion requests, injects a dynamically generated system prompt that lists available tools and their descriptions, and enables the language model to call tools iteratively within a single user turn. When the LLM indicates a tool call, the proxy parses the call, executes the corresponding MCP tool via its configured client, streams progress and results back to the client, and feeds the tool results back to the LLM to continue the conversation. It also preserves streaming of the LLM’s generated messages (Server-Sent Events), including intermediate statuses like [Executing tool...] and [Tool Result: ...]. To use it, point your LLM UI to the proxy URL (for example http://localhost:9090) and ensure your llama-server is running (default http://localhost:8080). The proxy will handle tool usage automatically once you’ve configured one or more MCP tool servers via mcp-config.json (e.g., search1api).

Prerequisites:

• Node.js version 18 or higher
• npm (comes with Node.js)
• A running llama-server instance
• One or more MCP tool servers you want the LLM to access

Step-by-step:

1. Prepare your project directory and install dependencies

mkdir llama-server-mcp-proxy
cd llama-server-mcp-proxy
npm init -y
npm install @modelcontextprotocol/sdk

2. Create the proxy script (e.g., llama-server-mcp-proxy.js) and implement the proxy logic as described in the README. If you already have the script, place it here.
3. Create mcp-config.json describing your MCP tool servers (see the example in the README).

{
  "mcpServers": {
    "search1api": {
      "command": "npx",
      "args": [
        "search1api-mcp",
        "--port",
        "0",
        "--api-key",
        "YOUR_SEARCH1API_KEY_PLACEHOLDER"
      ],
      "env": {
        "DEBUG": "true"
      }
    }
  }
}

4. Start the proxy

npm start

Or run directly with Node if you have the script named accordingly:

node llama-server-mcp-proxy.js

5. (Optional) set environment variables to customize behavior (PORT, LLAMA_SERVER_URL, MCP_CONFIG_PATH, MCP_PROXY_DEBUG, LLAMA_SERVER_TIMEOUT).

Tips and notes:

• Keep your mcp-config.json secure because it may contain API keys. Add it to .gitignore.
• The proxy defaults to http://localhost:8080 for llama-server and port 9090 for the proxy; adjust with PORT and LLAMA_SERVER_URL.
• If a tool server requires authentication, supply appropriate API keys in mcp-config.json (as shown in the example).
• For debugging, enable MCP_PROXY_DEBUG to get verbose logs; watch for [Executing tool...] and [Tool Result: ...] messages to understand the flow.
• If you encounter connection errors to a tool server, ensure the tooling server is listening on the port selected by the proxy (port 0 lets the server pick a free port).
• You can add additional MCP tool servers in mcp-config.json as needed, each with its own command, arguments, and environment variables.