CI regression gate for MCP servers: run a golden set of tool calls, diff against a baseline, fail the build when outputs get worse
A CI regression gate for MCP servers. It calls your server's tools, compares the results to a saved baseline, and fails the build when the output gets worse, even if the schema did not change.
Read the write-up: MCP server testing in CI: the bug no schema check can see.
An MCP server's tool names and input schemas can stay identical while what a tool returns quietly gets worse: a refactor breaks a code path, a dependency bump changes behavior. Nothing crashes, the agent using it just gets worse answers.
Other tools cover parts of this. The official Inspector's CLI runs scripted single-run assertions. mcp-server-diff compares declared schemas and says it does not test output correctness. A few other early tools record golden outputs and diff them too, for example vexyo and cisco-open/mcptoolkit-test. mcp-eval-gate is another take on golden-output regression.
1. You describe how to reach your server and list the tool calls that matter, with the output you expect from each, in a goldenset.yaml. 2. run --update-baseline calls every tool and records the results as the baseline, once, while everything is known to be good. 3. run calls the same tools again and compares against the baseline. It prints a table and a diff, and exits 1 if anything failed or got worse, so any CI job can gate on it.
Then point server at your own server and replace the cases. --update-baseline refuses to record while any case fails, since a baseline should hold known-good output.
From the project README.
Add the radar badge to your README — it shows your project was picked up by MCP Radar and links to this page:
[](https://mcp.liqiwa.com/s/Umer-2612--mcp-eval-gate.html)
AI-native HL7 v2 toolkit with generation, validation, HL7→FHIR conversion, MCP & Claude Code integration. Bringing healthcare interoperability into the AI-native developer ecosystem
hirotomasato/yowesGenerate realistic teacher documents (ID cards, licenses, letters) for 13 countries via MCP.
rokbenko/quackd🦆🧠 Give your small robot a brain. Tell a Microduck, a Reachy Mini, a LeRobot arm or any ROS base what you want in plain language. An LLM (Claude, OpenAI, Gemini, Grok, or a local model via Ollama, vLL
lotchuazzz-crypto/papergraph-mcpTurn arXiv and LaTeX mathematical papers into theorem dependency graphs for AI agents through MCP.
Stupidoodle/swissdevjobs-cliSearch & apply to ~4,700 salary-transparent tech jobs across 7 countries (🇨🇭🇩🇪🇬🇧🇺🇸🇨🇦🇳🇱🇫🇷) from your terminal or AI agent — zero-dependency Python CLI + MCP server + Claude Code plugin
bybit-exchange/kaasTurn scattered notes, docs and transcripts into a queryable Markdown wiki — an LLM knowledge-base compiler with MCP access, no embeddings, self-hosted.
The top new MCP servers of the week, every Monday. No spam, unsubscribe anytime.