🎉 NEW: Persistent local storage with SQLite. Read the post →

Introduction to LLM Evaluation Metrics

deepeval offers 50+ SOTA, ready-to-use metrics for you to quickly get started with. Essentially, while a test case represents the thing you're trying to measure, the metric acts as the ruler for specific criteria of interest.

Quick Summary

Almost all predefined metrics on deepeval use LLM-as-a-judge, with techniques such as QAG (question-answer-generation), DAG (deep acyclic graphs), and G-Eval. Most score test cases representing atomic interactions, while trajectory metrics score the complete ordered trace produced by an AI agent.

All of deepeval's metrics output a score between 0-1 based on its corresponding equation, as well as score reasoning. A metric is only successful if the evaluation score is equal to or greater than threshold, which is defaulted to 0.5 for all metrics.

Custom metrics allow you to define your custom criteria using SOTA implementations of LLM-as-a-Judge metrics in everyday language:

  • G-Eval
  • DAG (Deep Acyclic Graph)
  • Conversational G-Eval
  • Conversational DAG
  • Arena G-Eval
  • Do it yourself, 100% self-coded metrics (e.g. if you want to use BLEU, ROUGE)

You should aim to have at least one custom metric in your LLM evals pipeline.

Agentic metrics evaluate AI agents at two different scopes:

Trajectory metrics analyze the complete ordered chain of decisions and actions captured through LLM tracing:

  • Task Completion — whether the agent successfully accomplished its task.
  • Step Efficiency — whether the agent avoided unnecessary or redundant steps.
  • Plan Adherence — whether the agent followed its generated plan.
  • Plan Quality — whether the generated plan was logical, complete, and efficient.

Component-level action metrics evaluate one LLM decision about tool selection and arguments inside that trajectory:

  • Tool Correctness — whether the agent selected the correct tools.
  • Argument Correctness — whether it supplied the correct arguments to those tools.

Use trajectory-based evaluation for overall execution quality and component-level evaluation to diagnose individual actions.

RAG (retrieval augmented generation) metrics focus on the retriever and generator components independently.

  • Retriever:

    • Contextual Relevancy
    • Contextual Precision
    • Contextual Recall

  • Generator:

    • Answer Relevancy
    • Faithfulness

Multi-turn metrics' main use case are for evaluating chatbots and uses a ConversationalTestCase instead. They include:

  • Knowledge Retention
  • Role Adherence
  • Conversation Completeness
  • Conversation Relevancy

Multi-turn metrics evaluates conversations as a whole and takes prior context into consideration when doing so.

Safety metrics concerns more on LLM security. They include:

  • Bias
  • Toxicity
  • Non-Advice
  • Misuse
  • PIILeakage
  • Role Violation

For those looking for a full-blown LLM red teaming orchestration frameowork, checkout DeepTeam. DeepTeam is deepeval but for red teaming LLMs specifically.

Metrics in deepeval are multi-modal by default, metrics targeting images are metrics that definitely expects an image in the test case. They include:

  • Image Coherence
  • Image Helpfulness
  • Image Reference
  • Text-to-Image
  • Image-Editing

Note that multi-modal metrics requires MLLMImages in LLMTestCases.

Not use case specific, but still useful for some use cases:

  • Hallucination
  • Json Correctness
  • Summarization
  • Ragas

Metrics can score your app's black-box result, an agent's complete trajectory, or an individual component. This first example runs an end-to-end evaluation by providing metrics and test cases:

main.py
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

evaluate(
    metrics=[AnswerRelevancyMetric()],
    test_cases=[LLMTestCase(input="What's `deepeval`?", actual_output="Your favorite eval framework's favorite evals framework.")]
)

If you're logged into Confident AI before running an evaluation (deepeval login or deepeval view in the CLI), you'll also get entire testing reports on the platform:

Run evaluations on Confident AI

Get sharable test reports for every metric you run with deepeval.

Explore EnterpriseE

More information on everything can be found on the Confident AI evaluation docs.

Why deepeval Metrics?

Apart from the variety of metrics offered, deepeval's metrics are a step up to other implementations because they:

  • Are research-backed LLM-as-as-Judge (GEval)
  • One of the most used in the world (20 million+ daily evaluations)
  • Make deterministic metric scores possible (when using DAGMetric)
  • Are extra reliable as LLMs are only used for extremely confined tasks during evaluation to greatly reduce stochasticity and flakiness in scores
  • Provide a comprehensive reason for the scores computed
  • Integrated 100% with Confident AI

Create Your First Metric

Custom Metrics

deepeval provides G-Eval, a state-of-the-art LLM evaluation framework for anyone to create a custom LLM-evaluated metric using natural language. G-Eval is available for all single-turn, multi-turn, and multimodal evals.

from deepeval.test_case import LLMTestCase, SingleTurnParams
from deepeval.metrics import GEval

test_case = LLMTestCase(input="...", actual_output="...", expected_output="...")
correctness = GEval(
    name="Correctness",
    criteria="Correctness - determine if the actual output is correct according to the expected output.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
    strict_mode=True
)

correctness.measure(test_case)
print(correctness.score, correctness.reason)
from deepeval.test_case import Turn, MultiTurnParams, ConversationalTestCase
from deepeval.metrics import ConversationalGEval

convo_test_case = ConversationalTestCase(turns=[Turn(role="...", content="..."), Turn(role="...", content="...")])
professionalism_metric = ConversationalGEval(
    name="Professionalism",
    criteria="Determine whether the assistant has acted professionally based on the content."
    evaluation_params=[MultiTurnParams.CONTENT],
    strict_mode=True
)

professionalism_metric.measure(convo_test_case)
print(professionalism_metric.score, professionalism_metric.reason)

Under the hood, deepeval first generates a series of evaluation steps, before using these steps in conjunction with information in an LLMTestCase for evaluation. For more information, visit the G-Eval documentation page.

Default Metrics

deepeval includes six metrics for evaluating AI agents. Choose them based on the scope you need:

Trajectory metrics evaluate how the full execution works together:

  • Task Completion: Assesses whether the agent successfully completed its task.
  • Step Efficiency: Assesses whether the agent completed the task without unnecessary or redundant steps.
  • Plan Adherence: Assesses whether the agent followed its generated plan during execution.
  • Plan Quality: Assesses whether the generated plan was logical, complete, and efficient.

Component-level action metrics evaluate individual LLM tool-calling decisions:

  • Tool Correctness: Assesses whether the agent selected the correct tools.
  • Argument Correctness: Assesses whether the agent supplied the correct arguments to those tools.

Trajectory metrics require tracing because they analyze the complete ordered trace. Pass them to evals_iterator() when running the agent:

main.py
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import observe

@observe()
def trip_planner_agent(input):

    @observe()
    def itinerary_generator(destination, days):
        return ["Eiffel Tower", "Louvre Museum", "Montmartre"][:days]

    return itinerary_generator("Paris", 2)

dataset = EvaluationDataset(goldens=[Golden(input="Plan a two-day trip to Paris")])

for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric(threshold=0.5)]):
    trip_planner_agent(golden.input)

The most used RAG metrics include:

  • Answer Relevancy: Evaluates if the generated answer is relevant to the user query
  • Faithfulness: Measures if the generated answer is factually consistent with the provided context
  • Contextual Relevancy: Assesses if the retrieved context is relevant to the user query
  • Contextual Recall: Evaluates if the retrieved context contains all relevant information
  • Contextual Precision: Measures if the retrieved context is precise and focused

Which can be simply imported from the deepeval.metrics module:

main.py
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(input="...", actual_output="...")
relevancy = AnswerRelevancyMetric(threshold=0.5)

relevancy.measure(test_case)
print(relevancy.score, relevancy.reason)

Chatbots require "conversational" (or multi-turn) metrics and they include:

  • Conversation Completeness: Evaluates if conversation satisfy user needs.
  • Conversation Relevancy: Measures if the generated outputs are relevant to user inputs.
  • Role Adherence: Assesses if the chatbot stays in character throughout a conversation.
  • Knowledge Retention: Evaluates if the chatbot is able to retain knowledge learnt throughout a conversation.

You'll need to also use ConversationalTestCases instead of regular LLMTestCase for conversational metrics:

main.py
from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import ConversationalGEval

convo_test_case = ConversationalTestCase(turns=[Turn(role="...", content="..."), Turn(role="...", content="...")])
role_adherence = RoleAdherenceMetric(threshold=0.5)

role_adherence.measure(convo_test_case)
print(role_adherence.score, role_adherence.reason)
from deepeval.test_case import LLMTestCase, MLLMImage
from deepeval.metrics import ImageCoherenceMetric

test_case = LLMTestCase(input=f"What does this image say? {MLLMImage(...)}", actual_output="No idea!")
image_coherence = ImageCoherenceMetric(threshold=0.5)

image_coherence.measure(test_case)
print(image_coherence.score, image_coherence.reason)
from deepeval.test_case import LLMTestCase
from deepeval.metrics import BiasMetric

test_case = LLMTestCase(input="...", actual_output="...")
bias = BiasMetric(threshold=0.5)

bias.measure(test_case)
print(bias.score, bias.reason)

Choosing Your Metrics

These are the metric categories to consider when choosing your metrics:

  • Custom metrics are use case specific and architecture agnostic:
    • G-Eval – best for subjective criteria like correctness, coherence, or tone; easy to set up.
    • DAG – decision-tree metric for objective or mixed criteria (e.g., verify format before tone).
    • Start with G-Eval for simplicity; use DAG for more control. You can also subclass BaseMetric to create your own.
  • Generic metrics are system specific and use case agnostic:
    • Agent trajectory metrics: evaluate task completion, execution efficiency, planning, and plan adherence across the complete trace
    • Agent component metrics: evaluate tool selection and argument generation at individual action steps
    • RAG metrics: measures retriever and generator separately
    • Multi-turn metrics: measure overall dialogue quality
    • Combine these for multi-component LLM systems.
  • Reference vs. Referenceless:
    • Reference-based metrics need ground truth (e.g., contextual recall or tool correctness).
    • Referenceless metrics work without labeled data, ideal for online or production evaluation.
    • Check each metric’s docs for required parameters.

When deciding on metrics, no matter how tempting, try to limit yourself to no more than 5 metrics, with this breakdown:

  • 2-3 generic, system-specific metrics (e.g. task completion for agents, contextual precision for RAG)
  • 1-2 custom, use case-specific metrics (e.g. helpfulness for a medical chatbot, format correctness for summarization)

The goal is to force yourself to prioritize and clearly define your evaluation criteria. This will not only help you use deepeval, but also help you understand what you care most about in your LLM application.

Here are some additional ideas if you're not sure:

  • AI agents: Start with TaskCompletionMetric for overall trajectory quality, add StepEfficiencyMetric, PlanAdherenceMetric, or PlanQualityMetric when the execution path matters, and use ToolCorrectnessMetric or ArgumentCorrectnessMetric to diagnose individual LLM tool-calling decisions
  • RAG: Focus on the AnswerRelevancyMetric (evaluates actual_output alignment with the input) and FaithfulnessMetric (checks for hallucinations against retrieval_context)
  • Chatbots: Implement a ConversationCompletenessMetric to assess overall conversation quality
  • Custom Requirements: When standard metrics don't fit your needs, create custom evaluations with G-Eval or DAG frameworks

In some cases, where your LLM model is doing most of the heavy lifting, it is not uncommon to have more use case specific metrics.

Configure LLM Judges

You can use ANY LLM judge in deepeval, including OpenAI, Azure OpenAI, Ollama, Anthropic, Gemini, LiteLLM, etc. You can also wrap your own LLM API in deepeval's DeepEvalBaseLLM class to use ANY model of your choice. Click here for full guide.

To use OpenAI for deepeval's LLM metrics, supply your OPENAI_API_KEY in the CLI:

export OPENAI_API_KEY=<your-openai-api-key>

Alternatively, if you're working in a notebook environment (Jupyter or Colab), set your OPENAI_API_KEY in a cell:

%env OPENAI_API_KEY=<your-openai-api-key>

deepeval also allows you to use Azure OpenAI for metrics that are evaluated using an LLM. Run the following command in the CLI to configure your deepeval environment to use Azure OpenAI for all LLM-based metrics.

deepeval set-azure-openai \
    --base-url=<endpoint> \ # e.g. https://example-resource.azure.openai.com/
    --model=<model_name> \ # e.g. gpt-4.1
    --deployment-name=<deployment_name> \  # e.g. Test Deployment
    --api-version=<api_version> \ # e.g. 2025-01-01-preview
    --model-version=<model_version> # e.g. 2024-11-20

Note that the model-version is optional. If you ever wish to stop using Azure OpenAI and move back to regular OpenAI, simply run:

deepeval unset-azure-openai

To use Ollama models for your metrics, run deepeval set-ollama --model=<model> in your CLI. For example:

deepeval set-ollama --model=deepseek-r1:1.5b

Optionally, you can specify the base URL of your local Ollama model instance if you've defined a custom port. The default base URL is set to http://localhost:11434.

deepeval set-ollama --model=deepseek-r1:1.5b \
    --base-url="http://localhost:11434"

To stop using your local Ollama model and move back to OpenAI, run:

deepeval unset-ollama

To use Gemini models with deepeval, run the following command in your CLI.

deepeval set-gemini \
    --model=<model_name> # e.g. "gemini-2.0-flash-001"

deepeval allows you to use ANY custom LLM for evaluation. This includes LLMs from langchain's chat model integrations, Hugging Face's transformers library, or even LLMs in GGML format.

This includes any of your favorite models such as:

  • Azure OpenAI
  • Claude via AWS Bedrock
  • Google Vertex AI
  • Mistral 7B

All the examples can be found here, but down below is a quick example of a custom Azure OpenAI model through langchain's AzureChatOpenAI module for evaluation:

from langchain_openai import AzureChatOpenAI
from deepeval.models.base_model import DeepEvalBaseLLM

class AzureOpenAI(DeepEvalBaseLLM):
    def __init__(
        self,
        model
    ):
        self.model = model

    def load_model(self):
        return self.model

    def generate(self, prompt: str) ->