Vibe Coding with DeepEval

Although DeepEval is great as an AI quality validation suite — pytest assertions, regression gates, CI/CD failure tracking — that's only half the use case.

The other half is using the same evals during development: your coding agent runs them, reads the failing metrics and traces, and uses the results to decide what to change next in your agent, RAG pipeline, or chatbot. Then re-runs to confirm.

In short: DeepEval helps you vibe code your agent without vibe coding your agents.

The Loop

Vibe coding with DeepEval is a feedback loop between your eval suite and your coding agent:

1. Define a dataset, or let DeepEval generate one from your docs, traces, or existing examples.
2. Add an eval suite that calls your agent against that dataset and scores the outputs with the metrics you care about.
3. Let your coding agent run the suite, read the failures, and make targeted changes to the relevant prompts, retrieval logic, tools, or application code.
4. Re-run the same evals until the scores and metric reasons show that the behavior has improved.

A trace from deepeval test run gives the coding agent more than a pass/fail result. It includes scores, span-level context, and metric reasons, so a failure can be traced back to the part of the system that produced it.

Claude Code v2.1.19

>eval the refund agent and fix any regressions

⏺Bash(deepeval test run agents/checkout.py)⎿faithfulness 0.64 ⚠

⏺Edit(agents/retriever.py)⎿scoped to active refund policies

⏺Bash(deepeval test run agents/checkout.py)⎿faithfulness 0.98 ✓

●All metrics green — ready to commit.

>Try “ship it”

? for shortcuts

For example, if a run reports faithfulness 0.64, the agent can open the retriever span that produced the off-source claim, narrow retrieval to active refund policies, and re-run the eval to confirm the fix. The workflow is similar to a tight unit-test cycle, except the assertions are scored model outputs and the runner is your coding agent.

Under the Hood

When the Agent Skill is installed and you say "add evals to this repo and fix the failing ones", your coding agent doesn't invent an evaluation framework — it shells out to DeepEval's CLI. Concretely, every iteration round walks through these stages, each backed by a single CLI command documented in the CLI reference:

1. Load (or generate) the dataset

The agent first looks for an existing dataset under tests/evals/, on Confident AI, or as a Hugging Face dataset.

If none exists, it generates one with deepeval generate. That single command synthesizes goldens from your docs, contexts, scratch, or existing goldens — single-turn or multi-turn — without any custom Python:

deepeval generate \
  --method docs \
  --variation single-turn \
  --documents ./docs \
  --output-dir ./tests/evals \
  --file-name .dataset

The generated .dataset.json is committed to the repo. Future runs reuse it; new edge cases append to it.

2. Build the eval suite

The skill ships pytest templates for the four common shapes — single-turn end-to-end, multi-turn end-to-end, single-turn component-level, plus a shared conftest.py. The agent picks the closest template, fills placeholders (dataset path, app entrypoint, metrics, thresholds), and writes a committed file like tests/evals/test_<app>.py. No throwaway scripts, no hidden goldens — the suite reruns without an agent.

The metrics it picks are not invented either; they come from the 50+ metrics catalog — GEval, AnswerRelevancyMetric, FaithfulnessMetric, ToolCorrectnessMetric, ConversationalGEval, etc. — each with a default threshold and a reason field the agent can read.

3. Run the suite

Now the loop's heartbeat: test run. Same command every round, no flake from rerunning a UI: