spark_history_mcp_server
MCP server from zjt-peekaboo/spark_history_mcp_server
claude mcp add --transport stdio zjt-peekaboo-spark_history_mcp_server docker run -i spark-history-mcp-server:latest \ --env MCP_SSE_ENDPOINT="http://localhost:18888/sse" \ --env SPARK_HISTORY_SERVER_URL="http://<spark-history-server-host>:18080"
How to use
This MCP server provides a set of Spark History Server utility tools for inspecting and diagnosing Spark applications. The tools allow you to list applications, examine environments and executors, fetch details about specific applications, stages, and jobs, and identify bottlenecks or slow SQL queries. The included endpoints expose an MCP SSE endpoint at http://localhost:18888/sse for streaming updates as you interact with the server. Use the provided tools to gather structured insights about Spark job performance and resource usage, enabling targeted optimizations and historical trend analysis.
To use the tools, first ensure the server is running via the MCP SSE endpoint. Then you can query for: a list of applications, detailed application info, per-stage or per-executor metrics, and bottleneck analyses. The tool set is designed to help you compare environments between jobs, compare performance metrics, and pinpoint where tasks or stages may be underperforming. Typical workflows include listing applications, selecting a specific application, and then drilling into stages, jobs, and SQL queries to identify slow components or recurring bottlenecks.
How to install
Prerequisites:
- Docker installed and running
- Access to the Spark History Server instance
- Network access from the MCP server container to the Spark History Server
Installation steps:
-
Pull and run the MCP server docker image (adjust image name if different):
docker pull spark-history-mcp-server:latest docker run -d --name spark_history_mcp_server -p 18888:18888 -e SPARK_HISTORY_SERVER_URL=http://<spark-history-server-host>:18080 -e MCP_SSE_ENDPOINT=http://localhost:18888/sse spark-history-mcp-server:latest
-
Verify the MCP SSE endpoint is available:
-
If you need to customize environment variables, run with additional -e settings or mount a config file as needed per your deployment environment.
Notes:
- Replace placeholder hostnames with your actual Spark History Server host and port.
- Ensure network policies allow traffic between the MCP container and the Spark History Server.
Additional notes
Tips and common issues:
- Ensure the Spark History Server is reachable from the MCP container; use stable network names or IPs.
- If the SSE endpoint does not respond, check firewall rules and that the Spark History Server is running on the expected port.
- The tools expose endpoints for listing and retrieving detailed Spark job information; for large clusters, consider paginating list calls where supported.
- Environment variable SPARK_HISTORY_SERVER_URL is used to point the MCP server to the Spark History Server; ensure it is accurate for your deployment.
- If you encounter authentication issues, verify any required credentials for accessing Spark History Server logs or endpoints and adjust the MCP server's access settings accordingly.
