Documentation Grafana Cloud Observe and act Agent Observability API and SDKs Evaluation

Evaluation API

Configure evaluators, rules, and templates, discover judge providers, and export scores.

Before you begin

• Enable Agent Observability on your Grafana Cloud stack. Refer to Set up Agent Observability in Grafana Cloud.
• For configuration changes, use a Grafana identity with permission to manage the relevant resource. Refer to Security and access controls.
• For experiment runs or exported scores, create a Grafana Cloud Access Policy Token with the sigil:write scope.

For credential requirements, refer to Choose credentials.

Manage evaluators

Use evaluator endpoints to create and list scoring definitions.

To update an existing evaluator, send POST /api/v1/eval/evaluators again with the same evaluator_id and a new version (YYYY-MM-DD or YYYY-MM-DD.N). That request stores another snapshot.

Manage rules

Use rule endpoints to attach evaluators to matching generation or conversation traffic.

Discover judge providers

Use judge provider endpoints to discover configured judge providers and models.

Manage templates

Use template endpoints to manage reusable evaluator templates and versions.

Configure evaluator types

Choose an evaluator type for the content and scoring criteria you want to check.

LLM judge

Use an LLM judge to score responses against prompt-based criteria:

JSON Copy

{
  "kind": "llm_judge",
  "config": {
    "provider": "openai",
    "model": "gpt-4o-mini",
    "system_prompt": "You are a quality evaluator.",
    "user_prompt": "Rate this response:\n{{agent_response}}",
    "timeout_ms": 120000
  }
}

You don’t configure the LLM judge’s output-token limit. Agent Observability sets it to 16384 tokens or the model’s supported output limit, whichever is lower. A max_tokens value in an existing evaluator configuration doesn’t change this limit.

The optional temperature setting uses the model provider’s default when omitted. The UI doesn’t display Max tokens or Temperature and removes both settings when you save an evaluator or fork a template. Set temperature only if the selected model supports it; an explicit 0 isn’t the same as leaving it unset.

The default timeout_ms is 120000 (120 seconds) per judge attempt. Increase it if your chosen reasoning model needs more time. For prompt and model settings, refer to Configure evaluators and evaluation rules.

Set reasoning_effort only when the selected provider and model support it. For available values and omission behavior, refer to LLM judge settings.

JSON schema

JSON Copy

{
  "kind": "json_schema",
  "config": {
    "schema": {
      "type": "object",
      "required": ["answer"],
      "properties": {
        "answer": { "type": "string" }
      }
    },
    "target": "response"
  }
}

The optional target field sets the text to evaluate: response (default), latest_user_message, input, or system_prompt.

Regex

JSON Copy

{
  "kind": "regex",
  "config": {
    "pattern": "^\\d+$",
    "reject": false,
    "target": "response"
  }
}

Heuristic

JSON Copy

{
  "kind": "heuristic",
  "config": {
    "version": "v2",
    "target": "response",
    "root": {
      "kind": "group",
      "operator": "and",
      "rules": [
        { "kind": "rule", "type": "not_empty" },
        { "kind": "rule", "type": "min_length", "value": 10 }
      ]
    }
  }
}

Choose rule selectors

Choose rule execution mode

Evaluation rules can run attached evaluators in parallel or sequential mode. In parallel mode, every evaluator runs for each matched item. In sequential mode, evaluators run in order, and each evaluator must pass before the next evaluator is queued.

Use sequential mode to put a low-cost gate in front of a more expensive evaluator. The first evaluator in a sequential rule must define a pass condition, such as pass_threshold, pass_value, or pass_match, so Agent Observability knows whether to continue. Gate outcomes are tracked separately from final quality pass-rate reporting.

Integrate rule alerts

Rules can include an alert_rule_uids field that stores the UIDs of Grafana alert rules created for the evaluation. In Agent Observability, add alerts from the evaluation creation or configuration flow by setting a pass-rate threshold and contact point. Alerts are created as non-provisioned, so you can edit them in the Grafana Alerting UI afterward.

Export scores

Use score ingest to export externally computed scores from SDKs, CI jobs, and benchmark processes.

Scores are idempotent by score_id, which must be at most 128 characters. Resubmitting the same score ID returns a duplicate item result instead of creating another score. A score must include either generation_id or trial_id. Scores with experiment_id are rejected if the experiment doesn’t exist for the authenticated tenant.

Validation failures remain descriptive. Internal lookup or storage failures use internal server error; error_id=<id> in the existing result schema, with one shared ID for internal failures in the batch.

Next steps

• Set up and monitor online evaluation
• Explore the experiments API