Skip to main content

OpenAI Agents API

The openai:agents-api provider evaluates the OpenAI Agents API, which runs agents using OpenAI's managed Codex harness. Each provider call creates a fresh session, waits for the root agent's turn to finish, collects its final answer, and deletes the session.

For applications built with the JavaScript @openai/agents package, use the Agents SDK provider (openai:agents:*).

Setup​

Use an OpenAI project API key with api.agents.read, api.agents.write, and api.responses.write permissions:

export OPENAI_API_KEY=your-api-key

Promptfoo adds the required OpenAI-Beta: agents=v1 header. No additional SDK package is required. Keep your API key outside the agent's sandbox.

promptfooconfig.yaml
providers:
  - id: openai:agents-api:gpt-6-astra
    config:
      agent:
        instructions: Answer concisely.
        reasoning:
          effort: low
      environment:
        type: none
prompts:
  - '{{question}}'
tests:
  - vars:
      question: What is six times seven? Answer with just the number.
    assert:
      - type: equals
        value: '42'

Hosted sandboxes and tools​

Set environment.type: openai_hosted to let the agent execute commands and work with files. The environment object accepts the API's hosted environment settings, including network policy, packages, initial files, skills, and plugins. Configure remote MCP servers and web search in agent.tools using the Agents API tool format.

providers:
  - id: openai:agents-api:gpt-6-astra
    config