Testing prompts with Mocha/Chai

promptfoo can be integrated with test frameworks like Mocha and assertion libraries like Chai in order to evaluate prompts as part of existing testing and CI workflows.

This guide includes examples that show how to create Mocha test cases for desired prompt quality using semantic similarity and LLM grading.

For more information on supported checks, see Assertions & Metrics documentation.

Prerequisites

Before you begin, make sure you have the following node packages installed:

• mocha: npm install --save-dev mocha
• chai: npm install --save-dev chai
• promptfoo: npm install --save-dev promptfoo

Creating custom chai assertions

First, we'll create custom chai assertions:

• toMatchSemanticSimilarity: Compares two strings for semantic similarity.
• toPassLLMRubric: Checks if a string meets the specified LLM Rubric criteria.
• toMatchFactuality: Checks if a string meets the specified factuality criteria.
• toMatchClosedQA: Checks if a string meets the specified question-answering criteria.

Create a new file called assertions.js and add the following:

Javascript

import { Assertion } from 'chai';
import { assertions } from 'promptfoo';

const { matchesSimilarity, matchesLlmRubric } = assertions;

Assertion.addAsyncMethod('toMatchSemanticSimilarity', async function (expected, threshold = 0.8) {
  const received = this._obj;
  const result = await matchesSimilarity(received, expected, threshold);
  const pass = received === expected || result.pass;

  this.assert(
    pass,
    `expected #{this} to match semantic similarity with #{exp}, but it did not. Reason: ${result.reason}`,
    `expected #{this} not to match semantic similarity with #{exp}`,
    expected,
  );
});

Assertion.addAsyncMethod('toPassLLMRubric', async function (expected, gradingConfig) {
  const received = this._obj;
  const gradingResult = await matchesLlmRubric(expected, received, gradingConfig);

  this.assert(
    gradingResult.pass,
    `expected #{this} to pass LLM Rubric with #{exp}, but it did not. Reason: ${gradingResult.reason}`,
    `expected #{this} not to pass LLM Rubric with #{exp}`,
    expected,
  );
});

Assertion.addAsyncMethod('toMatchFactuality', async function (input, expected, gradingConfig) {
  const received = this._obj;
  const gradingResult = await matchesFactuality(input, expected, received, gradingConfig);

  this.assert(
    gradingResult.pass,
    `expected #{this} to match factuality with #{exp}, but it did not. Reason: ${gradingResult.reason}`,
    `expected #{this} not to match factuality with #{exp}`,
    expected,
  );
});

Assertion.addAsyncMethod('toMatchClosedQA', async function (input, expected, gradingConfig) {
  const received = this._obj;
  const gradingResult = await matchesClosedQa(input, expected, received, gradingConfig);

  this.assert(
    gradingResult.pass,
    `expected #{this} to match ClosedQA with #{exp}, but it did not. Reason: ${gradingResult.reason}`,
    `expected #{this} not to match ClosedQA with #{exp}`,
    expected,
  );
});