Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Observe and actbreadcrumb arrow Agent Observabilitybreadcrumb arrow Referencebreadcrumb arrow Explore evaluation API
Grafana Cloud

Evaluation API

The Grafana Agent Observability evaluation control plane API manages evaluators, rules, guard rules, saved conversations, test suites, experiments, and judge provider discovery. Offline experiment runs can be recorded by SDKs, CI jobs, and benchmark processes.

Before you begin

  • Enable Agent Observability on your Grafana Cloud stack. Refer to Set up Agent Observability
  • For configuration changes, use a Grafana identity with permission to manage the relevant resource. Refer to Security and access controls
  • For experiment runs and exported scores, create a Grafana Cloud Access Policy Token with the sigil:write scope

Choose credentials

The credentials you use depend on what you change.

  • Configuration, such as evaluators, rules, templates, guards, saved conversations, collections, and test suites, changes only through Grafana. Use the Agent Observability app or gcx. The SDK’s TestSuitesClient also goes through Grafana, with a Grafana service account token. Each acts as a Grafana user or service account, so the app’s role-based access control (RBAC) permissions apply. A request that calls the API directly with a Cloud Access Policy Token can’t change configuration.
  • Experiment runs and exported scores that SDKs, CI jobs, and benchmark processes send use a Cloud Access Policy Token with the sigil:write scope. Refer to Collect credentials from Grafana Cloud.

Manage evaluators

Use evaluator endpoints to create and list scoring definitions.

MethodPathDescription
GET/api/v1/eval/evaluatorsList all evaluators.
POST/api/v1/eval/evaluatorsCreate an evaluator.
GET/api/v1/eval/evaluators/{id}Get an evaluator by ID.
DELETE/api/v1/eval/evaluators/{id}Delete an evaluator.
GET/api/v1/eval/predefined/evaluatorsList predefined evaluators.
POST/api/v1/eval/predefined/evaluators/{id}:forkFork a predefined evaluator.

To update an existing evaluator, send POST /api/v1/eval/evaluators again with the same evaluator_id and a new version (YYYY-MM-DD or YYYY-MM-DD.N). That request stores another snapshot.

Manage rules

Use rule endpoints to attach evaluators to matching generation or conversation traffic.

MethodPathDescription
GET/api/v1/eval/rulesList all evaluation rules.
POST/api/v1/eval/rulesCreate a rule.
GET/api/v1/eval/rules/{id}Get a rule by ID.
PATCH/api/v1/eval/rules/{id}Update a rule.
DELETE/api/v1/eval/rules/{id}Delete a rule.
POST/api/v1/eval/rules:previewPreview rule matches.
GET/api/v1/eval/rules/{id}/actionsList rule actions.
POST/api/v1/eval/rules/{id}/actionsCreate a rule action.
GET/api/v1/eval/rules/{id}/actions/{action_id}Get a rule action.
PATCH/api/v1/eval/rules/{id}/actions/{action_id}Update a rule action.