Open-source CLI and MCP server for evaluating AI applications and gating releases with reliability evidence.
The release-confidence layer for AI-agent changes.
Autoeval is an open-source CLI and MCP server for evaluating AI applications, inspecting reliability evidence, and gating releases from development and CI workflows.
Follow this path from a repository visit to a repeatable release gate:
1. Meet the requirements and configure Plumloom. 2. Install Autoeval. 3. Log in and run the quickstart. 4. Run your first file-driven evaluation. 5. Add a suite gate to CI. 6. Inspect results, refine evaluations, and repeat.
- Node.js 22.13 or newer - pnpm 11 - A Plumloom API key beginning with plsk - AUTOEVALAPIBASEURL - A supported OS credential store for interactive login persistence
Complete these two setup steps in Plumloom before running Autoeval.
Autoeval needs access to a model to run evaluations.
Profile avatar → Settings → AI Providers & Models
Plumloom currently supports OpenAI and Together.ai API keys. More providers are coming soon.
Create an API key for Autoeval. Use it to authenticate from your terminal, CI, or other development tools.
The examples below use the installed autoeval binary. When running directly from this repository, replace autoeval with:
When you are ready to use your own evaluation files, run:
Then continue with the file-driven workflow below.
4. Run your first file-driven evaluation
An evaluation file uses the configured-run JSON accepted by autoeval eval validate. Conversation and agent-trace files can keep their artifacts and reference documents in separate files:
From the project README.
Add the radar badge to your README — it shows your project was picked up by MCP Radar and links to this page:
[](https://mcp.liqiwa.com/s/AlchemyInCode--plumloom-autoeval-oss.html)
ChatGPT thinks. Codex works. Use ChatGPT as the planning brain while keeping the Codex harness.
shinthink/blitzstrike⚡ Blitz Strike — a universal MCP penetration-testing toolbelt. Structured methodology: reconnaissance & attack-surface mapping, source-to-sink analysis, and live validation. 57 escalation chains, 130-
0xwilliamortiz/openclaude-improvedruns anywhere. uses anything
halofyai/halofyHalofy is the open access and governance layer for AI agents across your organization. Identity, policy, provenance, audit, and signed erasure.
CaptureGrubEnchant/SolidWorksSolidWorks MCP Server connects an AI assistant to a running SolidWorks instance. Sketch, extrude, fillet, export STEP/STL, generate macros.
arielshad/3d-asset-server3d assets api and mcp
The top new MCP servers of the week, every Monday. No spam, unsubscribe anytime.