Assertions & metrics
Assertions are used to compare the LLM output against expected values or conditions. While assertions are not required to run an eval, they are a useful way to automate your analysis.
Different types of assertions can be used to validate the output in various ways, such as checking for equality, JSON structure, similarity, or custom functions.
Red team strategies can grade responses during an attack. The final assertion check reuses a strategy grade only when the configured strategy, plugin, assertion, and recorded input/output match the returned result. Stale grades and older results without this binding receive fresh grading. When a new grade is needed, red team assertions use the final transformed attack input, then the provider-reported prompt (a nonempty string or the last user message in a chat array), then the last saved user message, falling back to the original test prompt when none is available. Prior user/assistant conversation is included as grading context, except for Hydra and Goblin, which grade the current turn without adding history. System and tool messages are not treated as user input.
When no final strategy prompt is available, a provider-reported chat array without a usable user message uses the original prompt fallback instead of unrelated saved messages. Historical strategy grading usage is counted once across matching assertions, including assertion sets. Replayed grading responses count their full token usage as cached.
For data-exfil red team assertions, valid page tracking can supply deterministic evidence of exfiltration. Missing or malformed tracking identifiers leave model grading in place and do not automatically pass the test.
In machine learning, "Accuracy" is a metric that measures the proportion of correct predictions made by a model out of the total number of predictions. With promptfoo, accuracy is defined as the proportion of prompts that produce the expected or desired output.
Using assertions
To use assertions in your test cases, add an assert property to the test case with an array of assertion objects. Each assertion object should have a type property indicating the assertion type and any additional properties required for that assertion type.
For a quick reference to available checks, jump to Assertion types.
Example:
tests:
- description: 'Test if output is equal to the expected value'
vars:
example: 'Hello, World!'
assert:
- type: equals
value: 'Hello, World!'
Assertion properties
| Property | Type | Required | Description |
|---|---|---|---|
| type | string | Yes | Type of assertion |
| value | string | No | The expected value, if applicable |
| threshold | number | No | The threshold value, applicable only to certain types such as similar, cost, javascript, python, ruby |
| weight | number | No | How heavily to weigh the assertion. Defaults to 1.0 |
| provider | string | No | Some assertions (similarity, llm-rubric, model-graded-*) require an LLM provider |
| rubricPrompt | string | string[] | No | Model-graded LLM prompt |
| config | object | No | External mapping of arbitrary strings to values passed to custom javascript/python/ruby assertions |
| transform | string | Function | No | Process the output before running the assertion. Accepts a string expression, file:// reference, or a function when using the Node.js package. See Transformations |
| metric | string | No | Tag that appears in the web UI as a named metric |
| contextTransform | string | Function | No | Javascript expression or function to dynamically construct context for context-based assertions. See Context Transform for more details. |
Grouping assertions via Assertion Sets
Assertions can be grouped together using an assert-set.
Example:
tests:
- description: 'Test that the output is cheap and fast'
vars:
example: 'Hello, World!'
assert:
- type: assert-set
assert:
- type: cost
threshold: 0.001
- type: latency
threshold: 200
In the above example if all assertions of the assert-set pass the entire assert-set passes.
There are cases where you may only need a certain number of assertions to pass. Here you can use threshold.
Example - if one of two assertions need to pass or 50%:
tests:
- description: 'Test that the output is cheap or fast'
vars:
example: 'Hello, World!'
assert:
- type: assert-set
threshold: 0.5
assert:
- type: cost
threshold: 0.001
- type: latency
threshold: 200
Assertion Set properties
| Property | Type | Required | Description |
|---|---|---|---|
| type | string | Yes | Must be assert-set |
| assert | array of asserts | Yes | Assertions to be run for the set |
| threshold | number | No | Success threshold for the assert-set. Ex. 1 out of 4 equal weights assertions need to pass. Threshold should be 0.25 |
| weight | number | No | How heavily to weigh the assertion set within test assertions. Defaults to 1.0 |
| metric | string | No | Metric name for this assertion set within the test |
Assertion types
Deterministic eval metrics
These metrics are programmatic tests that are run on LLM output. See all details
| Assertion Type | Returns true if... |
|---|---|
| equals | output matches exactly |
| contains | output contains a string or number as text |
| icontains | output contains a string or number as text, case insensitive |
| regex | output matches regex |
| starts-with | output starts with string |
| contains-any | output contains any of the listed substrings |
| contains-all | output contains all list of substrings |
| icontains-any | output contains any of the listed substrings, case insensitive |
| icontains-all | output contains all list of substrings, case insensitive |
| is-json | output is valid json (optional json schema validation) |
| contains-json | output contains valid json (optional json schema validation) |
| contains-html | output contains HTML content |
| is-html | output is valid HTML |
| is-sql | output is a non-empty valid SQL statement |
| contains-sql | output is valid SQL or contains a valid SQL code block |
| is-xml | output is a supported well-formed XML document containing any required elements |
| contains-xml | output contains valid xml fragment(s) |
| is-refusal | the provider reports a refusal or the output indicates one |
| javascript | provided Javascript function validates the output |
| python | provided Python function validates the output |
| ruby | provided Ruby function validates the output |
| webhook | webhook returns a boolean pass and an optional score from 0 to 1 |
| rouge-l | Rouge-L (LCS) score is at least the threshold (default 0.75) |
| rouge-n | Rouge-N score is above a given threshold (default 0.75) |
| rouge-s | Rouge-S (skip-bigram) score is at least the threshold (default 0.75) |
| bleu | BLEU >= threshold (default 0.5); blank references are ignored |
| gleu | GLEU >= threshold (default 0.5); empty output scores 0 |
| levenshtein | Levenshtein distance is below a threshold |
| latency | Latency is below a threshold (milliseconds) |
| meteor | METEOR score is above a given threshold (default 0.5); requires natural |
| perplexity | Perplexity is below a threshold |
| perplexity-score | Normalized perplexity |
| cost | Cost is below a threshold, or recorded as a zero-weight named metric |
| is-valid-function-call | Ensure that the function call matches the function's JSON schema |
| is-valid-openai-function-call | Ensure that the function call matches the function's JSON schema |
| is-valid-openai-tools-call | Ensure all tool calls match the tools JSON schema |
| tool-call-f1 | Tool names meet the F1 threshold; parser limit exhaustion fails |
| trace-span-count | Count spans matching names and attributes with min/max thresholds |
| trace-span-duration | Check durations of spans selected by name and attributes, including percentiles |
| trace-error-spans | Check error counts and rates in spans selected by name and attributes |
| skill-used | Ensure normalized provider skill metadata includes expected skills |
| trajectory:tool-used | Ensure a traced agent trajectory used specific tools |
| trajectory:tool-args-match | Ensure traced tool calls used the expected arguments |
| trajectory:tool-sequence | Ensure traced tool usage happened in the expected order |
| trajectory:step-count | Count normalized trajectory steps by type or name pattern |
| guardrails | Evaluate the target's normalized input or output guardrail signal |
Single-line JavaScript assertions return the final expression after variable declarations and ignore semicolons inside comments. The final expression can use optional chaining and division, call an async function, consume a generator's yielded result, or return a grading result object. Multiline JavaScript assertions require an explicit return.
Every test type can be negated by prepending not-. For example, not-equals or not-regex.
For tool-call validity assertions, not- inverts a validation verdict. Tool configuration errors and unexpected validator failures remain failures.
When a not-javascript assertion returns a full grading result, its reason is preserved verbatim, including an empty string, for either inverted outcome. For not-python and not-ruby, a negated failure keeps its custom reason, falling back to a generic message if it is empty; a negated pass reports Assertion passed. All three preserve the script's score.
The search-rubric and not-search-rubric assertions require a rubric value that renders to a string.
For not-classifier, not-search-rubric, not-factuality (also not-model-graded-factuality), and not-model-graded-closedqa, a grader error or missing verdict remains a failure with score 0. Negation only inverts a valid grading result.
Model-assisted eval metrics
These metrics are model-assisted, and rely on LLMs or other machine learning models.
See Model-graded evals, classification, and similarity docs for more information.
| Assertion Type | Method |
|---|---|
| similar | Embeddings and cosine similarity are above a threshold |
| classifier | Run LLM output through a classifier |
| moderation | Check output against safety policies and include provider-reported usage metrics |
| llm-rubric | Grade text, images, or audio against a rubric with a compatible model |
| g-eval | Chain-of-thought evaluation based on custom criteria using the G-Eval framework |
| answer-relevance | Ensure that LLM output is related to original query (default threshold 0.5) |
| context-faithfulness | Ensure that LLM output uses the context (default threshold 0.5) |
| context-recall | Ensure that ground truth appears in context (default threshold 0.5) |
| context-relevance | Ensure that context is relevant to original query (default threshold 0.5) |
| conversation-relevance | Ensure that responses remain relevant throughout a conversation |
| trajectory:goal-success | Use an LLM judge to decide whether the traced agent run achieved its goal |
| factuality | LLM output adheres to the given facts, using Factuality method from OpenAI eval |
| model-graded-factuality | Alias for factuality |
| model-graded-closedqa | LLM output adheres to given criteria, using Closed QA method from OpenAI eval |
| pi | Alternative scoring approach that uses a dedicated model for evaluating criteria |
| select-best | Compare multiple outputs for a test case and pick the best one |
| max-score | Select output with highest aggregate score from other assertions |
Weighted assertions
In some cases, you might want to assign different weights to your assertions depending on their importance. The weight property is a number that determines the relative importance of the assertion. The default weight is 1.
The final score of the test case is calculated as the weighted average of the scores of all assertions, where the weights are the weight values of the assertions. Averaging does not rescale the scores: a negative similarity or an unbounded dot product retains its original scale.
Here's an example:
tests:
assert:
- type: equals
value: 'Hello world'
weight: 2
- type: contains
value: 'world'
weight: 1
In this example, the equals assertion is twice as important as the contains assertion.
If the LLM output is Goodbye world, the equals assertion fails but the contains assertion passes, and the final score is 0.33 (1/3).
Setting a score requirement
Test cases support an optional threshold property. If set, the pass/fail status of a test case is determined by whether the combined weighted score of all assertions is greater than or equal to the threshold value. This can override individual assertion verdicts. Omit the test-level threshold to require every nonzero-weight assertion to pass.
For example:
tests:
threshold: 0.5
assert:
- type: equals
value: 'Hello world'
weight: 2
- type: contains
value: 'world'
weight: 1
If the LLM outputs Goodbye world, the equals assertion fails but the contains assertion passes and the final score is 0.33. Because this is below the 0.5 threshold, the test case fails. If the threshold were lowered to 0.2, the test case would succeed.
A threshold of 0 passes only when the combined score is nonnegative; negative scores can still fail. The same applies to an assert-set threshold. To collect a metric without requiring that individual assertion to pass, set weight: 0 on the assertion instead.
Setting weight: 0 on an individual assertion makes it pass and excludes its score from the aggregate. On an assert-set, zero weight excludes the set's score, but the set can still fail based on its children or its own threshold. Named metrics retain their measured values in both cases, with unit weight used in place of zero for metric aggregation.
Custom assertion scoring
By default, test cases use weighted averaging to combine assertion scores. You can define custom scoring functions to implement more complex logic, such as:
- Failing if any critical metric falls below a threshold
- Implementing non-linear scoring combinations
- Mapping metrics to a common scale before combining them
- Using different scoring logic for different test cases
Prerequisites
Custom scoring functions require named metrics. Each assertion must have a metric field:
assert:
- type: equals
value: 'Hello'
metric: accuracy
- type: contains
value: 'world'
metric: completeness
Configuration
Define scoring functions at two levels:
defaultTest:
assertScoringFunction: file://scoring.js # Global default
tests:
- description: 'Custom scoring for this test'
assertScoringFunction: file://custom.js # Test-specific override
The scoring function can be JavaScript or Python, referenced with file:// prefix. For named exports, use file://path/to/file.js:functionName.
Function Interface
type ScoringFunction = (
namedScores: Record<string, number>, // Weighted averages; may be nonfinite after aggregation
context: {
threshold?: number; // Test case threshold if set
tokensUsed