BioRAG-AI-Agent

BioRAG-AI-Agent is a Retrieval-Augmented Generation (RAG) system that queries PubMed metadata via an MCP server, embeds retrieved records, and performs semantic search to extract structured biomedical metadata. Users interact by providing a natural language request, which is transformed into a PubMed-compatible query by an OpenAI model, then executed against PubMed through the MCP endpoint. The server returns PubMed metadata (PMID, title, abstract), which the RAG pipeline encodes into vector space, stores in FAISS, and couples with classification logic to filter for in-scope studies. For downstream usage, the system can extract drug names, indications, sponsor information, and NCT IDs from retrieved abstracts, producing structured JSONL outputs for integration with your knowledge base.

Prerequisites:

• Python 3.8+ installed on your system
• Access to PubMed via NCBI endpoints
• API keys for OpenAI and any embedding service you use
• FAISS installed or available in your environment

1. Setup environment

• Create a virtual environment: python -m venv env source env/bin/activate # on Unix/macOS env\Scripts\activate # on Windows

2. Install dependencies

• Make sure you have the project files cloned. Then: pip install -r requirements.txt

3. Configure environment variables

• Set required keys (example): export PUBMED_API_KEY=your-pubmed-api-key export OPENAI_API_KEY=your-openai-api-key export FAISS_INDEX_PATH=/path/to/faiss/index export EMBEDDING_MODEL=text-embedding-3-small export CLASSIFICATION_MODEL=gpt-3.5-turbo

4. Run the MCP server wrapper (example)

• Ensure the server entrypoint module exists (biorag_ai_agent.server): python -m biorag_ai_agent.server

5. Verify service

• Access the MCP endpoint at http://127.0.0.1:8000/mcp and ensure responses return PMID, title, and abstract.

Notes:

• If you use a containerized setup, adapt the commands to your Docker/OCI workflow and ensure environment variables are passed into the container.
• If embedding models differ in dimension, align them before indexing (or use PCA) as described in the README.

Tips and common issues:

• Ensure OpenAI API access and rate limits align with batching (the system uses batched processing for speed).
• When mixing embedding backends, keep consistent embedding dimensions to avoid FAISS dimension mismatches.
• Always deduplicate by SHA-256 hash before indexing to prevent duplicates.
• If PubMed access is rate-limited, implement your own local caching or backoff strategy and respect NCBI’s rate limits.
• Validate environment variables in your runtime shell or via a .env file loaded at startup.
• The MCP server exposes endpoints at /mcp; you can extend or customize the system prompt and classification thresholds via the SYSTEM_PROMPT and related config.

Environment variables:

• PUBMED_API_KEY: API key for PubMed data access
• OPENAI_API_KEY: Key for OpenAI models used in PubMed query generation and classification
• FAISS_INDEX_PATH: Path to the local FAISS index
• EMBEDDING_MODEL: Embedding model name/identifier
• CLASSIFICATION_MODEL: LLM used for classification and extraction
• SYSTEM_PROMPT: System prompt used for classification and extraction