skunit

skUnit is a semantic testing framework for .NET designed to verify AI-powered interactions and MCP (Model Context Protocol) tool integrations. It lets you author tests in readable Markdown to exercise chat-style interactions, validate JSON payloads, exercise function calls, and even test multi-turn conversations. When used as an MCP server, skUnit can be started as a background service and queried by test clients to verify that your tools or APIs under MCP specification behave as expected. You can pair it with a test harness that provides a chat client and a suite of scenarios, then run those scenarios through the MCP server to ensure tool invocations, semantic checks, and function calls are correctly handled and validated.

Within your MCP workflow, you can:

• Run MCP-style scenario tests that verify function invocations and their parameters, as well as semantic checks on responses.
• Validate complex JSON responses and partial matches using JsonCheck-like assertions.
• Execute multi-turn conversations to ensure consistency across turns and to catch hallucinations or misalignments in tool calls.
• Integrate scenario-driven tests with your existing CI to gate changes based on robust scenario outcomes.

To use the MCP server, start the server with the configured DOTNET command, then use your MCP client to list available tools, build scenarios, and run them against the server. The client can discover tools exposed by your environment and feed them into scenario runners for validation.

Prerequisites:

• .NET SDK (recommended latest LTS, e.g., .NET 6/7/8 as appropriate for your skUnit build)
• Access to the MCP client tooling you plan to use with this server

Step-by-step:

1. Clone or download the skUnit repository and locate the MCP server project (the skUnit server build).
2. Restore dependencies for the server project: dotnet restore path/to/skUnit.Server.csproj
3. Build the MCP server: dotnet build path/to/skUnit.Server.csproj
4. Run the MCP server locally (see mcp_config for the exact command): dotnet run --project path/to/skUnit.Server.csproj
5. Verify the server is reachable from your MCP client (default port 5000 as per config): http://localhost:5000/

Optional configuration:

• Set credentials and endpoints for your AI provider (Azure OpenAI, OpenAI, etc.) via environment variables as shown in mcp_config.
• Adjust ASPNETCORE_URLS or port mappings if you want to run behind a reverse proxy or in a container.

If you plan to containerize, you can wrap the server in a Docker image using a standard .NET runtime base image and copy the server into the container, then expose port 5000 unless you override via environment/config.

Tips and common issues:

• Ensure your AI provider credentials are correctly configured in the environment where the MCP server runs. Missing or invalid API keys will cause failures in tool invocations.
• If tool discovery fails within MCP, verify that the skUnit server exposes the expected endpoints and that the MCP client can reach the server URL and port from the test environment.
• For flaky semantic assertions, consider enabling ScenarioRunOptions (TotalRuns and MinSuccessRate) to introduce statistical robustness in CI.
• When running in CI, pin the .NET SDK version to avoid mismatches that could cause build failures.
• If you run multiple MCP servers in tandem, ensure there are unique ports or proper container networking to avoid port conflicts.