unsloth

Unsloth MCP Server integrates the Unsloth library to accelerate fine-tuning and inference for large language models. The server exposes tooling to check installation, discover supported models, load and fine-tune models with 4-bit quantization, and generate text or export models to formats like GGUF or Hugging Face. With the node-based server running, you can interact with the tools via the provided MCP endpoints, invoking operations such as verifying your setup, listing models, loading a model with optional 4-bit quantization, performing LoRA/QLoRA-based fine-tuning, generating text from a prompt, and exporting tuned models for deployment. The available tools include: check_installation, list_supported_models, load_model, finetune_model, generate_text, and export_model. Each tool takes a specific set of arguments (e.g., model_name, dataset_name, output_dir, etc.) to control the behavior and resource usage of the Unsloth integration.

Prerequisites:\n- Node.js and npm installed on your system.\n- Python 3.10-3.12 and Python tooling available (for Unsloth usage and optional dependencies).\n- Access to a CUDA-enabled GPU if you plan to fine-tune large models locally.\n\nStep-by-step installation:\n1) Install Unsloth in your Python environment (the MCP server assumes Unsloth is installed for model operations):\nbash\npip install unsloth\n\n2) Clone or obtain the Unsloth MCP server repository and navigate to the server directory (unsloth-server).\nbash\ncd unsloth-server\n\n3) Install server dependencies and build the server:\nbash\nnpm install\nnpm run build\n\n4) Ensure the MCP configuration references the built server entry (build/index.js) and set any needed environment variables (e.g., HUGGINGFACE_TOKEN) in your MCP settings. The example below shows how to wire it into MCP settings:\njson\n{\n "mcpServers": {\n "unsloth-server": {\n "command": "node",\n "args": ["/path/to/unsloth-server/build/index.js"],\n "env": {\n "HUGGINGFACE_TOKEN": "your_token_here"\n },\n "disabled": false,\n "autoApprove": []\n }\n }\n}\n

Tips and considerations:\n- HUGGINGFACE_TOKEN is optional but may be required to access private models or certain repositories. Include it in the MCP config if needed.\n- Ensure CUDA toolkit and driver compatibility with your PyTorch version if you plan to run GPU-accelerated fine-tuning.\n- Supported models include Llama, Mistral, Phi, Gemma, and related variants; use list_supported_models to discover compatibility.\n- When exporting models, you can choose formats like gguf, huggingface, etc., and adjust quantization_bits as needed (default 4).\n- If you encounter CUDA OOM errors, reduce batch size, enable 4-bit quantization, and consider gradient checkpointing or shorter max_seq_length.\n- Verify that your environment has the necessary Python packages aligned with your PyTorch and CUDA versions to avoid import errors.\n- For production deployments, consider pinning exact versions of Unsloth and dependencies to ensure reproducibility.