Documentation Grafana Cloud Observe and act Agent Observability Evaluate and improve Evaluation settings

Configure evaluators and evaluation rules

Use this page to tune evaluator settings and traffic selection. To create your first evaluation and review its results, refer to Set up and monitor online evaluation. Evaluators define the scoring logic, and evaluations define which traffic to score.

Before you begin

• Send generation data to Agent Observability. Refer to Get started with Grafana Cloud.
• You have the Agent Observability Admin role.
• For LLM judge evaluators, choose a provider and model available in your stack. You don’t need a judge provider for JSON schema, regex, or heuristic evaluators.

Choose a judge provider

In Grafana Cloud, you use Grafana-configured judge providers. When you create an LLM judge evaluator, select an available provider and model. You don’t supply your own provider credentials.

LLM judges send the content being scored to the selected provider. For details, refer to Privacy.

Handle an exhausted judge allowance

When your stack reaches its monthly LLM judge allowance, Evaluations displays Monthly LLM judge allowance reached. LLM judge evaluations stop running, but non-LLM evaluations aren’t affected.

Your allowance tracks estimated judge spend using catalog prices, including long-context rates when the judge’s input exceeds the model’s threshold. Long prompts can consume the allowance faster when those rates are higher. Your custom model prices don’t change this calculation.

Click Contact support to request a higher limit, or wait until the allowance resets at the start of the next month (UTC).

Create evaluators

Use Agent Observability or the evaluation API to create evaluators. Four evaluator types are available:

LLM judge

An LLM judge uses an LLM to score generations based on criteria you define in a prompt template.

Key settings:

• provider and model: the LLM to use for judging. In the UI, choose a model from the LLM Model field on evaluator creation and editing, version publishing, template creation, and the test panel:
  • The field lists all models across every configured provider with vendor marks and human-readable names, for example, Claude Haiku 4.5 or GPT-4o Mini.
  • While the catalog loads, the field displays a loading indicator and still accepts a typed model.
  • When models from different providers share a display name, the provider name appears in parentheses. When models from the same provider share a name, the list retains the snapshot date or version suffix (falling back to the raw model ID if there’s no suffix).
  • Leave the field set to Default (or clear the selection) to use your organization’s configured default judge.
  • If a model isn’t listed in the catalog, or if a provider’s model list is unavailable or returns no models, type a custom PROVIDER/MODEL identifier, for example, openai/gpt-4o.
  • When you fork an LLM judge template, use the Model override field to select a model or keep the template default. Non-judge evaluator templates don’t show this field.
  • When you select a Jev model, the form displays guidance to configure typed decisions (a boolean output, a string output with 2–255 allowed values, or a numeric output with integer bounds spanning 2–10 levels). Numeric outputs need a labeled Scoring rubric, for example, 1: harmless; 2: review; 3: urgent.
• system_prompt and user_prompt: prompt templates with variables.
• max_tokens: set by Sigil. A value in a stored config is ignored. Judges run at the model’s output limit, up to 16384.
• temperature: optional, evaluation API only. When omitted, the model provider’s default applies (for example, 1.0 for OpenAI and Anthropic). The UI doesn’t display a Temperature field, and saving an evaluator in the UI removes the setting. Set it only if the selected model supports it; an explicit 0 isn’t the same as leaving it unset.
• reasoning_effort: optional none, minimal, low, medium, high, or xhigh. none disables reasoning; omission leaves it unspecified. Provider APIs validate model support.
• timeout_ms: deadline for each judge attempt, in milliseconds. The default is 120000. Judges run at the model’s output limit, so a slow reasoning model can need more; raise timeout_ms for it.

JSON schema

Validates that the agent response matches a JSON schema. Returns true or false.

Regex

Checks the agent response against one or more regex patterns. Use reject: true to invert the match.

Heuristic

Applies a rule tree with AND/OR logic. Supported checks: not_empty, contains, not_contains, min_length, max_length.

Evaluator descriptions

When you create an evaluator in the Agent Observability UI, you don’t enter a description manually. Agent Observability generates a one-sentence description from the configuration the first time you save:

• When Grafana Assistant is available and the evaluator doesn’t have a description yet, it drafts a summary from the prompt templates, patterns, schemas, or heuristic rules and score contract.
• When a description is already stored, or Grafana Assistant is unavailable, times out after 30 seconds, or fails to generate text, Agent Observability keeps the stored description or applies a fallback sentence for that evaluator kind, for example, Rates the evaluated content with an LLM judge. or Flags content that matches the configured pattern.

While drafting the description, the save button displays Generating description…, and controls such as Cancel and Back are temporarily disabled. Leaving the form while generation is in progress cancels the save.

The generated description appears in the evaluator library and alongside evaluation scores. When you edit an existing evaluator, saving keeps the existing description without regenerating it. When drafting an LLM judge system prompt, Grafana Assistant references the stored description. Evaluator templates continue to provide an editable description field.

Choose the content to check

JSON schema, regex, and heuristic evaluators evaluate the agent response by default. You can choose another field instead:

Under Content to check, select the Generation field for a JSON Schema or Regex evaluator. Heuristic evaluators select a field for each condition, so one check can combine multiple generation fields. In evaluator config JSON, continue to use the target field shown above.

Your application can send the system prompt in either of two ways, depending on the SDK and framework you use: in the generation’s own system prompt field, or as a message with the system role inside the chat history. The system_prompt target and the {{system_prompt}} template variable accept both. They read the system prompt field first, and the text of the system messages when that field is empty.

Use this for lightweight detection of injected content in generation input without requiring an LLM judge.

Configure outputs and pass verdicts

Under Judge output (for LLM judges) or Output (for deterministic and human review evaluators), configure the emitted score and pass criteria:

1. In Output key, enter the name of the score field, for example, score.
2. Under Output type, select Number, Boolean, or String. Evaluator kinds with fixed schemas, such as JSON schema or regex, set this to Boolean.
   • For numeric outputs, enter Min score and Max score (defaults to 1 and 10 for new evaluators). Changing the score bounds clamps the pass threshold to stay within the range.
   • For LLM judges using TypeSafe Jev models, numeric outputs require integer bounds spanning 2–10 levels and a labeled Scoring rubric, for example, 1: low; 2: medium; 3: high. For evaluators that already have a stored output description, the Output description field remains visible and editable.
   • For string outputs, enter a comma-separated list of accepted values in Allowed values. Keep this list short, because each value becomes a metric label in Results.
3. In the Passing result panel, toggle Mark output as pass or fail on or off:
   • When disabled, Agent Observability records outputs neutrally without a pass or fail verdict, and omits the pass rule (pass_threshold, pass_match, or pass_value) from the evaluator configuration. Forking an evaluator or template that has neutral outputs preserves pass/fail marking as off.
   • When enabled, configure the pass rule for your output type:
     • Numeric outputs: enter a threshold in Pass at or above (defaults to 7 for new evaluators) or drag the synchronized score slider between the minimum and maximum scores. Scores at or above the threshold pass. Existing numeric outputs without score bounds keep their cutoff without the slider.
     • String outputs: click allowed values under Choose which values pass to toggle passing status. Selected values display a checkmark. If you update the Allowed values list, Agent Observability automatically removes any passing selections that are no longer allowed. Existing passing values remain visible and removable when the evaluator has no allowed-values list.
     • Boolean outputs: select true or false under Choose which result passes to select the passing outcome. Each choice displays whether the value counts as Pass or Fail.

Use template variables

LLM judge prompts support three explicit variable scopes. The prompt editor puts the scope that matches the rule trigger first:

{{conversation.transcript_detailed}} is the conversation transcript with each tool call’s arguments and each tool result, so a judge can check the agent’s claims against what its tools returned. Reasoning and media are replaced by omission markers.

{{turn.transcript}}, {{conversation.transcript}} and {{conversation.transcript_detailed}} hold up to 32,000 bytes. A longer transcript is shortened from the middle of long messages, the agent’s earlier messages first. If that is not enough, whole messages are left out, user messages last and spread across the conversation, so the turns that remain still cover all of it. The latest exchange is always kept, and each cut is marked.

Use {{turn.tool_calls}} instead of {{generation.tool_calls}} when evaluating a final response: the final generation usually contains the response, while the tool calls occurred in earlier generations in that turn.

For User-visible response judges that use turn or conversation variables, capture related generations with the same conversation ID. If Agent Observability can’t load the required context or find the selected generation in that conversation, the evaluation reports an error instead of scoring only the final generation. The playground follows the same rule. Check that the conversation contains the expected generations before running the evaluator again.

Existing unscoped variables such as {{agent_response}} and {{tool_calls}}, underscore turn variables such as {{turn_tool_calls}}, and assistant_* aliases remain supported. Prefer scoped variables in new prompts because their meaning does not change with the trigger.

Configure evaluation traffic

An evaluation connects one or more evaluators to generation or conversation traffic. Configure:

• Matching conditions: which traffic to evaluate, by agent name, agent version, operation, model, provider, mode, tags, errors, and conversation.tool_used for targeting conversations where a tool was called. An empty Agent name condition is ready by default. For the tool filter, Any matches when one pattern hits a tool used in the conversation, and All matches when every pattern does.
• Trigger: when evaluation runs:
  • User-visible response (user_visible_turn): when the agent produces final text for a user request. Use turn.* variables to include its intermediate tool steps.
  • Every agent generation (all_assistant_generations): after any agent model call.
  • Tool call requested (tool_call_steps): when a generation calls tools.
  • Generation failed (errored_generations): when a generation ends with an error.
  • Conversation idle (conversation): after the configured idle window.
• Sampling rate: percentage of matching traffic to evaluate, for example, a 10% rate evaluates approximately one in ten matching items.
• Evaluators: the evaluators to run.

The legacy turn selector remains accepted for existing API clients and stored rules, but it is not a separate authoring choice because it fires at the same event as user_visible_turn.

Create an evaluation

For the setup flow, refer to Create an evaluation. You can build an evaluation manually or use Grafana Assistant. To review existing evaluators first, open Evaluations > New evaluation and click See evaluator library.

Expose tags on evaluation results

Under When to evaluate, use Tags to expose for filtering to select up to 10 generation tag keys. These tags let you filter or break down evaluation score metrics; they don’t change which traffic matches the evaluation.

Each key appears in Prometheus with the agento11y_eval_tag_ prefix, for example, team becomes agento11y_eval_tag_team. Keep the number of distinct values small to limit metric series. For conversation-scope rules, Agent Observability uses the value from the last generation in the conversation that matches the rule.

Use score conditions to trigger actions

An evaluation action adds matching conversations to a saved collection. Use it to send failures, examples that need review, or scores that meet a threshold to a review queue.

Choose one of these triggers when you add an action:

• When passes: runs when every evaluator produces a passing verdict.
• When fails: runs when every evaluator produces a failing verdict.
• When output value: runs when one evaluator output matches a value that you choose.

For an output-value trigger, select the evaluator and output key, then choose a comparison and value. Numeric outputs support equality and greater-than or less-than comparisons. String outputs support equality and contains. Boolean outputs support equality. Use this trigger for a specific score, such as adding conversations whose toxicity score is greater than a threshold, even when the evaluator does not produce a pass/fail verdict.

Preview an evaluation before you save it

When you create or configure an evaluation, the rule builder shows a Playground beside the form so you can check your rule before you save it. The playground has two tabs.

What gets evaluated

This tab is a volume simulation: it applies your current criteria and sample rate to recent traffic (for example, the last 6 hours). It is not a forecast of future volume. On high-volume windows, the preview is based on a sample of that traffic. In that case, projected generation volume is labelled as an estimate, and actual volume may differ if the traffic mix changed. Expand Sample conversations to see matching traffic grouped into conversations, then open a conversation to see each generation and whether it would have been evaluated, matched but sampled out, or did not match. Select an agent, model, or tag on a generation to add it to your match criteria. When the rule includes LLM judges, the playground also shows an approximate projected cost per day based on sample-rendered judge prompts, model pricing, and projected evaluation volume; real spend varies with transcript length and judge completions.

When the selector is conversation, the playground shows conversation-level simulation only: each card reports whether the conversation would be evaluated (one evaluation per matching conversation after the idle window) or was sampled out, and links out to the full conversation instead of a per-generation drill-down. For a capped scan, the playground reports conversations selected for evaluation after sampling but does not project hourly or daily unique-conversation volume.

Use this tab to tune your selector, match filters, and sampling rate without waiting for scores to arrive asynchronously.

How it’s evaluated

This tab shows what the LLM judge receives for a selected generation — or a selected conversation when the rule selector is conversation. Choose which attached evaluator to test, pick a Source (Matched traffic from the volume preview by default, or Saved conversations), then pick a generation or conversation. Inspect the rendered judge prompt with template variables filled in and highlighted. You can test different judge models using the LLM Model field before running the evaluation. Run evaluator evaluates the selection with that evaluator’s work-in-progress config so you can check pass/fail and rationale before saving. If the test indicates that the judge model ran out of tokens before finishing its response, shorten the prompt or ask the judge for a more compact answer.

The generation picker lists only generations eligible for your trigger, in chronological order. For example, Tool call requested lists generations with tool calls, and Generation failed lists generations with errors. This applies to both matched traffic and saved conversations.

Click Open conversation to inspect the selected target’s full conversation in a new tab. Your unsaved evaluator changes stay in the builder.

Gate LLM judges with deterministic evaluators

When a rule contains more than one evaluator, choose Sequential gate to run them in the displayed order. Each evaluator must pass before the next evaluator runs. Use a deterministic evaluator first to filter out generations that don’t need an LLM judge.

Configure a pass condition for every output of each non-final evaluator. If a gate doesn’t pass, later evaluators don’t run. If a gate has an execution error, the evaluation retries according to its normal retry policy.

Choose Parallel when you want every evaluator to score the same matching traffic independently.

Gate evaluator scores control whether later evaluators run, but quality pass-rate reporting separates gate outcomes from the final quality evaluator outcomes.

Evaluate complete user turns

Turn-scope evaluators run when the agent produces its final user-visible response. They include the generations after the previous user-visible response through the final response, so a judge can evaluate a multi-step tool loop without including earlier user requests or waiting for the conversation to become idle.

Select the User-visible response trigger and use {{turn.transcript}}, {{turn.agent_sequence}}, {{turn.tool_calls}}, or {{turn.tool_results}} in an LLM judge prompt. Use generation.* when you intentionally need only the final model call, or conversation.* when earlier turns also matter.

Existing prompts that use turn_* or unscoped variables continue to work. Prefer explicit dotted scopes for new prompts; they make the input boundary visible and let the editor warn about likely mistakes.

Evaluate conversations

Conversation-scope evaluators score a full conversation after it has been idle for a configured window. Use them when the quality signal depends on the complete interaction instead of a single generation, for example, whether the agent resolved the user’s request or followed a multi-turn policy.

Select the Conversation idle trigger and set the idle window. In API configuration, use the conversation selector and min_idle_seconds. Agent Observability waits until no new generations arrive for that conversation during the window, then runs the evaluator with the full conversation transcript.

Use the {{conversation.transcript}} template variable in LLM judge prompts to evaluate the complete interaction.

Existing conversation-scope LLM judge prompts can also use {{conversation_transcript}} to include the full interaction. Generation-specific variables, such as {{agent_response}}, aren’t populated for conversation-scope evaluations.

The compact transcript removes conversation history replayed in later model inputs, so the judge doesn’t receive the same history for every call. It keeps genuinely repeated requests and new responses. This doesn’t change your stored generations.

Next steps

• Monitor evaluation results
• Run an experiment with the SDK
• Explore the evaluation API