flowllm

FlowLLM provides HTTP and MCP (Model Context Protocol) services built around LLMs, embeddings, and vector stores. It enables you to expose LLM-powered capabilities as MCP tools that MCP-compatible clients can discover and invoke, alongside a REST API for general usage. FlowLLM auto-generates services so you can run a single MCP-enabled endpoint and test flows via a minimal setup. Once started, you can register new Ops, compose them into flows, and serve them through MCP tools or HTTP endpoints. The included token counting support helps you estimate usage, and the architecture supports both synchronous JSON HTTP responses and streaming responses where appropriate.

Prerequisites:

• Python 3.10+
• Git
• Internet access to install dependencies

Install from PyPI (recommended):

pip install flowllm

Install from source (editable mode):

git clone https://github.com/flowllm-ai/flowllm.git
cd flowllm
pip install -e .

Configure environment variables (examples):

# API keys and base URLs
export FLOW_LLM_API_KEY=your-api-key-here
export FLOW_LLM_BASE_URL=https://api.example.com/v1
export FLOW_EMBEDDING_API_KEY=your-embedding-api-key-here
export FLOW_EMBEDDING_BASE_URL=https://embed.example.com/v1

Run the MCP server:

# Run via the module entry point (as configured in mcp_config)
python -m flowllm

Optional: for development, install dependencies and run tests if provided in the repo.

Tips and considerations:

• Ensure your API keys and base URLs are correctly configured in the environment to enable LLM and embedding services.
• The MCP service will register as tools for MCP-compatible clients; you can discover and invoke tools defined in your FlowLLM configuration.
• If you plan heavy concurrent usage, consider configuring token_count and backends (base, OpenAI, HF) as described in the project docs to get accurate usage metrics.
• For debugging, FlowLLM supports a CMD/service mode to run a single Op from the command line; this is useful for quick tests of a specific operation.
• Check the project docs for guidance on YAML configuration of flows, input/output schemas, and how to compose Ops with serial (>>) and parallel (|) operators.