Documentation Grafana Cloud Observe and act Agent Observability Reference Explore evaluation API

Evaluation API

The Grafana Agent Observability evaluation control plane API manages evaluators, rules, guard rules, saved conversations, test suites, experiments, and judge provider discovery. Experiments are available in public preview for offline runs recorded by SDKs, CI jobs, and benchmark processes.

Manage evaluators

Use evaluator endpoints to create and list scoring definitions.

Manage rules

Use rule endpoints to attach evaluators to matching generation or conversation traffic.

Discover judge providers

Use judge provider endpoints to discover configured judge providers and models.

Manage templates

Use template endpoints to manage reusable evaluator templates and versions.

Manage guards

Use guard endpoints to evaluate requests synchronously and manage guard rules.

Hook evaluation request

JSON Copy

{
  "phase": "preflight",
  "context": {
    "agent_name": "my-agent",
    "model": { "provider": "openai", "name": "gpt-4o" },
    "tags": { "env": "production" }
  },
  "input": {
    "messages": [
      { "role": "user", "parts": [{ "kind": "text", "text": "Hello" }] }
    ],
    "system_prompt": "You are a helpful assistant.",
    "tools": []
  }
}

Hook evaluation response

JSON Copy

{
  "action": "allow",
  "transformed_input": null,
  "evaluations": []
}

The action field is allow or deny. When any guard rule applied redactions, transformed_input contains the sanitized copy of the input. Each entry in evaluations reports a per-rule outcome with rule_id, evaluator_id, passed, and latency_ms.

Guard rule configuration

JSON Copy

{
  "rule_id": "block_tools",
  "enabled": true,
  "phase": "preflight",
  "priority": 5,
  "selector": "all",
  "match": {},
  "evaluator_ids": [],
  "action_on_fail": "deny",
  "tool_filter": { "blocked_names": ["shell_exec", "danger_*"] },
  "redact": {
    "patterns": [
      { "id": "ssn", "regex": "\\b\\d{3}-\\d{2}-\\d{4}\\b" }
    ]
  }
}

Each guard rule must include at least one of these fields: evaluator_ids, redact.patterns, or tool_filter.blocked_names. Redaction matches are always replaced with the fixed token redact; free-form replacement text is not accepted.

Export scores

Use score ingest to export externally computed scores from SDKs, CI jobs, and benchmark processes.

Scores are idempotent by score_id. Resubmitting the same score ID returns a duplicate item result instead of creating another score. A score must include either generation_id or trial_id. Scores with experiment_id are rejected if the experiment doesn’t exist for the authenticated tenant.

Manage saved conversations

Use saved conversation endpoints to bookmark telemetry conversations or create manual conversations for review and experiments.

Manage collections

Use collection endpoints to group saved conversations for review or promotion into test cases.

Manage test suites

Use test suite endpoints to manage versioned datasets for repeatable offline experiments. Add or edit cases in a draft, then publish the version to make it a stable input for experiment runs.

Published versions are immutable. Create a new draft version when you need to change the suite. The Agent Observability SDK exposes these operations through TestSuitesClient and can synchronize local YAML or JSON suites with stored suite versions.

Read experiment results

Use the Agent Observability SDK to read the scores and aggregated report for an offline experiment.

These reads use the same tenant API credentials as experiment run ingest. In Python, use experiment.report() to retrieve the aggregated report and experiment.client.list_scores() to retrieve scores.

Ingest experiment runs

Use experiment run ingest endpoints from SDKs, CI jobs, and benchmark processes that execute test cases. Prefer the Agent Observability SDK, which manages these calls and finalizes experiments automatically. These endpoints use tenant API credentials.

After you finalize an experiment, Agent Observability builds its report in the background.

Set planned_trial_count when you create the run if the total number of attempts is known. Agent Observability uses this value to show progress for an active experiment.

Configure evaluator types

LLM judge

JSON Copy

{
  "kind": "llm_judge",
  "config": {
    "provider": "openai",
    "model": "gpt-4o-mini",
    "system_prompt": "You are a quality evaluator.",
    "user_prompt": "Rate this response:\n{{agent_response}}",
    "max_tokens": 100,
    "temperature": 0.0,
    "timeout_ms": 30000
  }
}

JSON schema

JSON Copy

{
  "kind": "json_schema",
  "config": {
    "schema": {
      "type": "object",
      "required": ["answer"],
      "properties": {
        "answer": { "type": "string" }
      }
    },
    "target": "response"
  }
}

The optional target field sets the text to evaluate: response (default), input, or system_prompt.

Regex

JSON Copy

{
  "kind": "regex",
  "config": {
    "pattern": "^\\d+$",
    "reject": false,
    "target": "response"
  }
}

Heuristic

JSON Copy

{
  "kind": "heuristic",
  "config": {
    "version": "v2",
    "target": "response",
    "root": {
      "kind": "group",
      "operator": "and",
      "rules": [
        { "kind": "rule", "type": "not_empty" },
        { "kind": "rule", "type": "min_length", "value": 10 }
      ]
    }
  }
}

Choose rule selectors

Integrate rule alerts

Rules can include an alert_rule_uids field that stores the UIDs of Grafana alert rules created for the evaluation. In Agent Observability, add alerts from the evaluation creation or configuration flow by setting a pass-rate threshold and contact point. Alerts are created as non-provisioned, so you can edit them in the Grafana Alerting UI afterward.