Skip to main content

Using Promptfoo in n8n Workflows

This guide shows how to run Promptfoo evaluations from an n8n workflow so you can:

  • schedule nightly or ad‑hoc LLM tests,
  • gate downstream steps (Slack/Teams alerts, merge approvals, etc.) on pass‑rates, and
  • publish rich results links generated by Promptfoo.

Prerequisites​

WhatWhy
Self‑hosted n8n ≥ v1 (Docker or bare‑metal)Gives access to the “Execute Command” node.
Promptfoo CLI available in the container/hostNeeded to run promptfoo eval.
(Optional) LLM provider API keys set as environment variables or n8n credentialsExample: OPENAI_API_KEY, ANTHROPIC_API_KEY, …
(Optional) Slack / email / GitHub nodes in the same workflowFor notifications or comments once the eval finishes.

The easiest way is to bake Promptfoo into your n8n image so every workflow run already has the CLI:

# Dockerfile
FROM n8nio/n8n:latest          # or a fixed tag
USER root                      # gain perms to install packages
RUN npm install -g promptfoo   # installs CLI system‑wide
ENV PROMPTFOO_RUNNING_IN_DOCKER=1
USER node                      # drop back to non‑root

Update docker‑compose.yml:

services:
  n8n:
    build: .
    env_file: .env # where your OPENAI_API_KEY lives
    volumes:
      - ./data:/data # prompts & configs live here

If you prefer not to rebuild the image you can install Promptfoo on the fly inside the Execute Command node, but that adds 10‑15 s to every execution.

Basic “Run & Alert” workflow​

Below is the minimal pattern most teams start with:

#NodePurpose
1Trigger (Cron or Webhook)Decide when to evaluate (nightly, on Git push webhook …).
2Execute CommandRuns Promptfoo and emits raw stdout / stderr.
3Code / Set nodeParses the resulting JSON, extracts pass/fail counts & share‑URL.
4IF nodeBranches on “failures > 0”.
5Slack / Email / GitHubSends alert or PR comment when the gate fails.

Execute Command node configuration​

promptfoo eval \
 -c /data/promptfooconfig.yaml \
 --prompts "/data/prompts/**/*.json" \
 --output /tmp/pf-results.json \
 --share --fail-on-error
cat /tmp/pf-results.json

Set the working directory to /data (mount it with Docker volume) and set it to execute once (one run per trigger).

The node writes a machine‑readable results file and prints it to stdout, so the next node can simply JSON.parse($json["stdout"]).

info

The Execute Command node that we rely on is only available in self‑hosted n8n. n8n Cloud does not expose it yet.

Sample “Parse & alert” snippet (Code node, TypeScript)​

// Input: raw JSON string from previous node
const output = JSON.parse(items[0].json.stdout as string);

const { successes, failures } = output.results.stats;
items[0].json.passRate = successes / (successes + failures);
items[0].json.failures = failures;
items[0].json.shareUrl = output.shareableUrl;

return items;

An IF node can then route execution:

  • failures = 0 → take green path (maybe just archive the results).
  • failures > 0 → post to Slack or comment on the pull request.

Evaluating n8n AI Agent prompts and outputs​

If your goal is to test the prompt inside an n8n AI Agent / OpenAI node (not just run Promptfoo from a workflow), treat the n8n node like any other app contract:

  1. Put the agent prompt in a file,
  2. Map incoming n8n fields to tests.vars, and
  3. Assert on the exact JSON or tool-call shape that downstream n8n nodes expect.

This works well when you want to regression-test an agent before wiring it into a larger workflow.

Validate JSON that downstream n8n nodes consume​

If your agent is supposed to emit structured data for a Set, Code, Switch, or HTTP Request node, validate the payload directly.

promptfooconfig.yaml
prompts:
  - file://./prompts/n8n-support-router.txt

providers:
  - openai:gpt-5-mini

tests