rust-crates-analytics

Rust Crates Analytics lets you explore and analyze the crates.io Rust ecosystem locally. It provides an incremental ingestion pipeline that loads data into DuckDB-backed schemas, a Streamlit dashboard for quick visualizations, and an MCP HTTP server that lets you interact with the data programmatically or via AI agents. You can also run dbt transformations to enforce data quality and contracts. To use the MCP server, start the Python-based server via the MCP entry point and then connect your client (SQL or API) to the local endpoint. The project is designed so you can query the staging and raw data directly using a DuckDB-compatible client, or hook up an AI agent to interact with the data through the MCP server.

Key capabilities include:

• An HTTP MCP server that exposes a DuckDB-backed data layer for analysis and querying.
• A Streamlit dashboard with prebuilt visualizations for immediate insight into the crates.io data.
• An ELT pipeline with incremental updates for version_downloads and full refreshes for other tables, plus backfill options for historical data.
• dbt-based data quality checks and contracts to ensure data integrity during transformations.
• Easy integration with AI agents via MCP, enabling natural language querying and exploration of the Rust crates ecosystem.

Prerequisites:

• Python (recommended 3.8+)
• uv (the universal viewer/runner for this project)
• Sufficient disk space (at least 15 GB free)

Installation steps:

1. Clone or download the repository to your local machine.
2. Install and set up uv according to its documentation if you haven’t already.
3. Install Python dependencies for the project (if a virtual environment is provided, activate it).
4. Synchronize the uv project and set up dependencies:

uv sync
uv run setup.py

This will:

• Synchronize the Python environment and dependencies
• Download the crates.io database dump (approx. 5 GB)
• Create DuckDB schemas
• Load ~3 months of data (~5 GB)
• Run dbt transformations and tests (dbt build)

If you want to refresh data later:

```bash
uv run update.py

To backfill historical data or run the visualization dashboard:

uv run scripts/ingest_vd_archives.py --backfill-to-date <YYYY-MM-DD>

or

uv run scripts/ingest_vd_archives.py --backfill-days <INT>

To launch the Streamlit dashboard for orphan IDs:

uv run streamlit run visualization/orphan_dashboard.py

To start the MCP HTTP server for querying the data:

uv run mcp/mcp_duckdb_http.py

Optionally, add MCP connections with Claude or other agents as described in the project documentation.

Tips and notes:

• Ensure you have at least 15 GB free and sufficient CPU/RAM for running DuckDB and dbt transformations.
• The MCP server exposes a DuckDB-backed HTTP endpoint; you can connect any DuckDB-compatible client or AI agent to it.
• If you encounter issues with Python dependencies, consider recreating a fresh virtual environment and re-running uv sync/setup.
• The backfill and incremental loading strategies are designed to minimize long-running full-table refreshes; monitor dbt tests to ensure data quality.
• When using the MCP with AI agents, you can load prebuilt prompt contexts to accelerate discovery and exploration of the Rust Crates Analytics dataset.