---
title: "Evaluation API | Grafana Cloud documentation"
description: "API endpoints for managing evaluators, rules, guards, saved conversations, test suites, experiments, and judge providers in Agent Observability evaluation."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Evaluation API

> Note
> 
> Agent Observability Evaluations is currently in [public preview](/docs/release-life-cycle/). Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.

The Grafana Agent Observability evaluation control plane API manages evaluators, rules, guard rules, saved conversations, test suites, experiments, and judge provider discovery. Experiments are available in public preview for offline runs recorded by SDKs, CI jobs, and benchmark processes.

## Manage evaluators

Use evaluator endpoints to create and list scoring definitions.

Expand table

| Method   | Path                                           | Description                  |
|----------|------------------------------------------------|------------------------------|
| `GET`    | `/api/v1/eval/evaluators`                      | List all evaluators.         |
| `POST`   | `/api/v1/eval/evaluators`                      | Create an evaluator.         |
| `GET`    | `/api/v1/eval/evaluators/{id}`                 | Get an evaluator by ID.      |
| `DELETE` | `/api/v1/eval/evaluators/{id}`                 | Delete an evaluator.         |
| `GET`    | `/api/v1/eval/predefined/evaluators`           | List predefined evaluators.  |
| `POST`   | `/api/v1/eval/predefined/evaluators/{id}:fork` | Fork a predefined evaluator. |

## Manage rules

Use rule endpoints to attach evaluators to matching generation or conversation traffic.

Expand table

| Method   | Path                                          | Description                |
|----------|-----------------------------------------------|----------------------------|
| `GET`    | `/api/v1/eval/rules`                          | List all evaluation rules. |
| `POST`   | `/api/v1/eval/rules`                          | Create a rule.             |
| `GET`    | `/api/v1/eval/rules/{id}`                     | Get a rule by ID.          |
| `PATCH`  | `/api/v1/eval/rules/{id}`                     | Update a rule.             |
| `DELETE` | `/api/v1/eval/rules/{id}`                     | Delete a rule.             |
| `POST`   | `/api/v1/eval/rules:preview`                  | Preview rule matches.      |
| `GET`    | `/api/v1/eval/rules/{id}/actions`             | List rule actions.         |
| `POST`   | `/api/v1/eval/rules/{id}/actions`             | Create a rule action.      |
| `GET`    | `/api/v1/eval/rules/{id}/actions/{action_id}` | Get a rule action.         |
| `PATCH`  | `/api/v1/eval/rules/{id}/actions/{action_id}` | Update a rule action.      |
| `DELETE` | `/api/v1/eval/rules/{id}/actions/{action_id}` | Delete a rule action.      |

## Discover judge providers

Use judge provider endpoints to discover configured judge providers and models.

Expand table

| Method | Path                           | Description                      |
|--------|--------------------------------|----------------------------------|
| `GET`  | `/api/v1/eval/judge/providers` | List discovered judge providers. |
| `GET`  | `/api/v1/eval/judge/models`    | List models for a provider.      |

## Manage templates

Use template endpoints to manage reusable evaluator templates and versions.

Expand table

| Method   | Path                                             | Description                   |
|----------|--------------------------------------------------|-------------------------------|
| `GET`    | `/api/v1/eval/templates`                         | List evaluator templates.     |
| `POST`   | `/api/v1/eval/templates`                         | Create an evaluator template. |
| `GET`    | `/api/v1/eval/templates/{id}`                    | Get a template by ID.         |
| `DELETE` | `/api/v1/eval/templates/{id}`                    | Delete a template.            |
| `POST`   | `/api/v1/eval/templates/{id}:fork`               | Fork a template.              |
| `GET`    | `/api/v1/eval/templates/{id}/versions`           | List template versions.       |
| `POST`   | `/api/v1/eval/templates/{id}/versions`           | Publish a template version.   |
| `GET`    | `/api/v1/eval/templates/{id}/versions/{version}` | Get a template version.       |

## Manage guards

Use guard endpoints to evaluate requests synchronously and manage guard rules.

Expand table

| Method   | Path                           | Description                                |
|----------|--------------------------------|--------------------------------------------|
| `POST`   | `/api/v1/hooks:evaluate`       | Evaluate a request against enabled guards. |
| `GET`    | `/api/v1/eval/hook-rules`      | List all guard rules.                      |
| `POST`   | `/api/v1/eval/hook-rules`      | Create a guard rule.                       |
| `GET`    | `/api/v1/eval/hook-rules/{id}` | Get a guard rule by ID.                    |
| `PUT`    | `/api/v1/eval/hook-rules/{id}` | Update a guard rule.                       |
| `DELETE` | `/api/v1/eval/hook-rules/{id}` | Delete a guard rule.                       |

### Hook evaluation request

JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```json
{
  "phase": "preflight",
  "context": {
    "agent_name": "my-agent",
    "model": { "provider": "openai", "name": "gpt-4o" },
    "tags": { "env": "production" }
  },
  "input": {
    "messages": [
      { "role": "user", "parts": [{ "kind": "text", "text": "Hello" }] }
    ],
    "system_prompt": "You are a helpful assistant.",
    "tools": []
  }
}
```

### Hook evaluation response

JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```json
{
  "action": "allow",
  "transformed_input": null,
  "evaluations": []
}
```

The `action` field is `allow` or `deny`. When any guard rule applied redactions, `transformed_input` contains the sanitized copy of the input. Each entry in `evaluations` reports a per-rule outcome with `rule_id`, `evaluator_id`, `passed`, and `latency_ms`.

### Guard rule configuration

JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```json
{
  "rule_id": "block_tools",
  "enabled": true,
  "phase": "preflight",
  "priority": 5,
  "selector": "all",
  "match": {},
  "evaluator_ids": [],
  "action_on_fail": "deny",
  "tool_filter": { "blocked_names": ["shell_exec", "danger_*"] },
  "redact": {
    "patterns": [
      { "id": "ssn", "regex": "\\b\\d{3}-\\d{2}-\\d{4}\\b" }
    ]
  }
}
```

Each guard rule must include at least one of these fields: `evaluator_ids`, `redact.patterns`, or `tool_filter.blocked_names`. Redaction matches are always replaced with the fixed token `redact`; free-form replacement text is not accepted.

## Export scores

Use score ingest to export externally computed scores from SDKs, CI jobs, and benchmark processes.

Expand table

| Transport | Endpoint                     |
|-----------|------------------------------|
| HTTP      | `POST /api/v1/scores:export` |

Scores are idempotent by `score_id`. Resubmitting the same score ID returns a duplicate item result instead of creating another score. A score must include either `generation_id` or `trial_id`. Scores with `experiment_id` are rejected if the experiment doesn’t exist for the authenticated tenant.

## Manage saved conversations

Use saved conversation endpoints to bookmark telemetry conversations or create manual conversations for review and experiments.

Expand table

| Method   | Path                                                | Description                                       |
|----------|-----------------------------------------------------|---------------------------------------------------|
| `GET`    | `/api/v1/eval/saved-conversations`                  | List saved conversations.                         |
| `POST`   | `/api/v1/eval/saved-conversations`                  | Save a telemetry conversation.                    |
| `POST`   | `/api/v1/eval/saved-conversations:manual`           | Create a manual conversation.                     |
| `GET`    | `/api/v1/eval/saved-conversations/{id}`             | Get a saved conversation.                         |
| `DELETE` | `/api/v1/eval/saved-conversations/{id}`             | Delete a saved conversation.                      |
| `GET`    | `/api/v1/eval/saved-conversations/{id}/collections` | List collections containing a saved conversation. |

## Manage collections

Use collection endpoints to group saved conversations for review or promotion into test cases.

Expand table

| Method   | Path                                               | Description                 |
|----------|----------------------------------------------------|-----------------------------|
| `GET`    | `/api/v1/eval/collections`                         | List collections.           |
| `POST`   | `/api/v1/eval/collections`                         | Create a collection.        |
| `GET`    | `/api/v1/eval/collections/{id}`                    | Get a collection.           |
| `PATCH`  | `/api/v1/eval/collections/{id}`                    | Update a collection.        |
| `DELETE` | `/api/v1/eval/collections/{id}`                    | Delete a collection.        |
| `GET`    | `/api/v1/eval/collections/{id}/members`            | List collection members.    |
| `POST`   | `/api/v1/eval/collections/{id}/members`            | Add collection members.     |
| `DELETE` | `/api/v1/eval/collections/{id}/members/{saved_id}` | Remove a collection member. |

## Manage test suites

Use test suite endpoints to manage versioned datasets for repeatable offline experiments. Add or edit cases in a draft, then publish the version to make it a stable input for experiment runs.

Expand table

| Method   | Path                                                                               | Description                        |
|----------|------------------------------------------------------------------------------------|------------------------------------|
| `GET`    | `/api/v1/eval/test-suites`                                                         | List test suites.                  |
| `POST`   | `/api/v1/eval/test-suites`                                                         | Create a test suite.               |
| `GET`    | `/api/v1/eval/test-suites/{suite_id}`                                              | Get a test suite and its versions. |
| `PATCH`  | `/api/v1/eval/test-suites/{suite_id}`                                              | Update test suite metadata.        |
| `POST`   | `/api/v1/eval/test-suites/{suite_id}/versions`                                     | Create a draft version.            |
| `GET`    | `/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-cases`                | List test cases in a version.      |
| `POST`   | `/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-cases`                | Create or update a test case.      |
| `GET`    | `/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-cases/{test_case_id}` | Get a test case.                   |
| `PATCH`  | `/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-cases/{test_case_id}` | Update a test case.                |
| `DELETE` | `/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-cases/{test_case_id}` | Delete a test case.                |
| `POST`   | `/api/v1/eval/test-suites/{suite_id}/versions/{version}:publish`                   | Publish a suite version.           |

Published versions are immutable. Create a new draft version when you need to change the suite. The Agent Observability SDK exposes these operations through `TestSuitesClient` and can synchronize local YAML or JSON suites with stored suite versions.

## Read experiment results

Use the Agent Observability SDK to read the scores and aggregated report for an offline experiment.

Expand table

| Method | Path                                   | Description                           |
|--------|----------------------------------------|---------------------------------------|
| `GET`  | `/api/v1/eval/experiments/{id}/scores` | List experiment scores.               |
| `GET`  | `/api/v1/eval/experiments/{id}/report` | Get the aggregated experiment report. |

These reads use the same tenant API credentials as experiment run ingest. In Python, use `experiment.report()` to retrieve the aggregated report and `experiment.client.list_scores()` to retrieve scores.

## Ingest experiment runs

Use experiment run ingest endpoints from SDKs, CI jobs, and benchmark processes that execute test cases. Prefer the Agent Observability SDK, which manages these calls and finalizes experiments automatically. These endpoints use tenant API credentials.

Expand table

| Method  | Path                                                                         | Description                         |
|---------|------------------------------------------------------------------------------|-------------------------------------|
| `POST`  | `/api/v1/experiment-runs:upsert`                                             | Create or resume an experiment.     |
| `POST`  | `/api/v1/experiment-runs/{experiment_id}:finalize`                           | Finalize an experiment.             |
| `POST`  | `/api/v1/experiment-runs/{experiment_id}/trials`                             | Create or update a test-case trial. |
| `PATCH` | `/api/v1/experiment-runs/{experiment_id}/trials/{trial_id}`                  | Update a test-case trial.           |
| `POST`  | `/api/v1/experiment-runs/{experiment_id}/trials/{trial_id}/artifacts:upload` | Upload a trial artifact.            |

After you finalize an experiment, Agent Observability builds its report in the background.

Set `planned_trial_count` when you create the run if the total number of attempts is known. Agent Observability uses this value to show progress for an active experiment.

## Configure evaluator types

### LLM judge

JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```json
{
  "kind": "llm_judge",
  "config": {
    "provider": "openai",
    "model": "gpt-4o-mini",
    "system_prompt": "You are a quality evaluator.",
    "user_prompt": "Rate this response:\n{{agent_response}}",
    "max_tokens": 100,
    "temperature": 0.0,
    "timeout_ms": 30000
  }
}
```

### JSON schema

JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```json
{
  "kind": "json_schema",
  "config": {
    "schema": {
      "type": "object",
      "required": ["answer"],
      "properties": {
        "answer": { "type": "string" }
      }
    },
    "target": "response"
  }
}
```

The optional `target` field sets the text to evaluate: `response` (default), `input`, or `system_prompt`.

### Regex

JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```json
{
  "kind": "regex",
  "config": {
    "pattern": "^\\d+$",
    "reject": false,
    "target": "response"
  }
}
```

### Heuristic

JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```json
{
  "kind": "heuristic",
  "config": {
    "version": "v2",
    "target": "response",
    "root": {
      "kind": "group",
      "operator": "and",
      "rules": [
        { "kind": "rule", "type": "not_empty" },
        { "kind": "rule", "type": "min_length", "value": 10 }
      ]
    }
  }
}
```

## Choose rule selectors

Expand table

| Selector                    | Description                                          |
|-----------------------------|------------------------------------------------------|
| `user_visible_turn`         | Assistant text responses without tool calls.         |
| `all_assistant_generations` | Any agent output.                                    |
| `tool_call_steps`           | Generations containing tool calls.                   |
| `errored_generations`       | Generations with `call_error`.                       |
| `conversation`              | Full conversations after the configured idle window. |

## Integrate rule alerts

Rules can include an `alert_rule_uids` field that stores the UIDs of Grafana alert rules created for the evaluation. In Agent Observability, add alerts from the evaluation creation or configuration flow by setting a pass-rate threshold and contact point. Alerts are created as non-provisioned, so you can edit them in the Grafana Alerting UI afterward.