--- title: "Configure evaluators and evaluation rules | Grafana Cloud documentation" description: "Set up LLM judges, evaluator types, rules, and providers for continuous quality scoring in Agent Observability." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Configure evaluators and evaluation rules Use this page to tune evaluator settings and traffic selection. To create your first evaluation and review its results, refer to [Set up and monitor online evaluation](/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/evaluation). Evaluators define the scoring logic, and evaluations define which traffic to score. ## Before you begin - Send generation data to Agent Observability. Refer to [Get started with Grafana Cloud](/docs/grafana-cloud/observe-and-act/agent-observability/get-started/grafana-cloud). - You have the Agent Observability Admin role. - For LLM judge evaluators, choose a provider and model available in your stack. You don’t need a judge provider for JSON schema, regex, or heuristic evaluators. ## Choose a judge provider In Grafana Cloud, you use Grafana-configured judge providers. When you create an LLM judge evaluator, select an available provider and model. You don’t supply your own provider credentials. LLM judges send the content being scored to the selected provider. For details, refer to [Privacy](/docs/grafana-cloud/observe-and-act/agent-observability/privacy-and-access/privacy). ## Handle an exhausted judge allowance When your stack reaches its monthly LLM judge allowance, **Evaluations** displays **Monthly LLM judge allowance reached**. LLM judge evaluations stop running, but non-LLM evaluations aren’t affected. Your allowance tracks estimated judge spend using catalog prices, including long-context rates when the judge’s input exceeds the model’s threshold. Long prompts can consume the allowance faster when those rates are higher. Your custom model prices don’t change this calculation. Click **Contact support** to request a higher limit, or wait until the allowance resets at the start of the next month (UTC). ## Create evaluators Use Agent Observability or the evaluation API to create evaluators. Four evaluator types are available: ### LLM judge An LLM judge uses an LLM to score generations based on criteria you define in a prompt template. Key settings: - `provider` and `model`: the LLM to use for judging. In the UI, choose a model from the **LLM Model** field on evaluator creation and editing, version publishing, template creation, and the test panel: - The field lists all models across every configured provider with vendor marks and human-readable names, for example, `Claude Haiku 4.5` or `GPT-4o Mini`. - While the catalog loads, the field displays a loading indicator and still accepts a typed model. - When models from different providers share a display name, the provider name appears in parentheses. When models from the same provider share a name, the list retains the snapshot date or version suffix (falling back to the raw model ID if there’s no suffix). - Leave the field set to **Default** (or clear the selection) to use your organization’s configured default judge. - If a model isn’t listed in the catalog, or if a provider’s model list is unavailable or returns no models, type a custom *PROVIDER*/*MODEL* identifier, for example, `openai/gpt-4o`. - When you fork an LLM judge template, use the **Model override** field to select a model or keep the template default. Non-judge evaluator templates don’t show this field. - When you select a Jev model, the form displays guidance to configure typed decisions (a boolean output, a string output with 2–255 allowed values, or a numeric output with integer bounds spanning 2–10 levels). Numeric outputs need a labeled **Scoring rubric**, for example, `1: harmless; 2: review; 3: urgent`. - `system_prompt` and `user_prompt`: prompt templates with variables. - `max_tokens`: set by Sigil. A value in a stored config is ignored. Judges run at the model’s output limit, up to `16384`. - `temperature`: optional, evaluation API only. When omitted, the model provider’s default applies (for example, `1.0` for OpenAI and Anthropic). The UI doesn’t display a **Temperature** field, and saving an evaluator in the UI removes the setting. Set it only if the selected model supports it; an explicit `0` isn’t the same as leaving it unset. - `reasoning_effort`: optional `none`, `minimal`, `low`, `medium`, `high`, or `xhigh`. `none` disables reasoning; omission leaves it unspecified. Provider APIs validate model support. - `timeout_ms`: deadline for each judge attempt, in milliseconds. The default is `120000`. Judges run at the model’s output limit, so a slow reasoning model can need more; raise `timeout_ms` for it. ### JSON schema Validates that the agent response matches a JSON schema. Returns `true` or `false`. ### Regex Checks the agent response against one or more regex patterns. Use `reject: true` to invert the match. ### Heuristic Applies a rule tree with AND/OR logic. Supported checks: `not_empty`, `contains`, `not_contains`, `min_length`, `max_length`. ## Evaluator descriptions When you create an evaluator in the Agent Observability UI, you don’t enter a description manually. Agent Observability generates a one-sentence description from the configuration the first time you save: - When Grafana Assistant is available and the evaluator doesn’t have a description yet, it drafts a summary from the prompt templates, patterns, schemas, or heuristic rules and score contract. - When a description is already stored, or Grafana Assistant is unavailable, times out after 30 seconds, or fails to generate text, Agent Observability keeps the stored description or applies a fallback sentence for that evaluator kind, for example, `Rates the evaluated content with an LLM judge.` or `Flags content that matches the configured pattern.` While drafting the description, the save button displays **Generating description…**, and controls such as **Cancel** and **Back** are temporarily disabled. Leaving the form while generation is in progress cancels the save. The generated description appears in the evaluator library and alongside evaluation scores. When you edit an existing evaluator, saving keeps the existing description without regenerating it. When drafting an LLM judge system prompt, Grafana Assistant references the stored description. Evaluator templates continue to provide an editable description field. ## Choose the content to check JSON schema, regex, and heuristic evaluators evaluate the agent response by default. You can choose another field instead: Expand table | Target | Description | |-----------------------|------------------------------------------------------------------------------------------------------------------------------| | `response` | Assistant response text (default). | | `latest_user_message` | Only the most recent user-role text on this generation. Earlier user turns, assistant messages, and tool input are excluded. | | `input` | User, agent, and system messages already stored on this generation, plus tool calls. | | `system_prompt` | The system prompt. | Under **Content to check**, select the **Generation field** for a JSON Schema or Regex evaluator. Heuristic evaluators select a field for each condition, so one check can combine multiple generation fields. In evaluator config JSON, continue to use the `target` field shown above. Your application can send the system prompt in either of two ways, depending on the SDK and framework you use: in the generation’s own system prompt field, or as a message with the `system` role inside the chat history. The `system_prompt` target and the `{{system_prompt}}` template variable accept both. They read the system prompt field first, and the text of the system messages when that field is empty. Use this for lightweight detection of injected content in generation input without requiring an LLM judge. ## Configure outputs and pass verdicts Under **Judge output** (for LLM judges) or **Output** (for deterministic and human review evaluators), configure the emitted score and pass criteria: 1. In **Output key**, enter the name of the score field, for example, `score`. 2. Under **Output type**, select **Number**, **Boolean**, or **String**. Evaluator kinds with fixed schemas, such as JSON schema or regex, set this to **Boolean**. - For numeric outputs, enter **Min score** and **Max score** (defaults to 1 and 10 for new evaluators). Changing the score bounds clamps the pass threshold to stay within the range. - For LLM judges using TypeSafe Jev models, numeric outputs require integer bounds spanning 2–10 levels and a labeled **Scoring rubric**, for example, `1: low; 2: medium; 3: high`. For evaluators that already have a stored output description, the **Output description** field remains visible and editable. - For string outputs, enter a comma-separated list of accepted values in **Allowed values**. Keep this list short, because each value becomes a metric label in Results. 3. In the **Passing result** panel, toggle **Mark output as pass or fail** on or off: - When disabled, Agent Observability records outputs neutrally without a pass or fail verdict, and omits the pass rule (`pass_threshold`, `pass_match`, or `pass_value`) from the evaluator configuration. Forking an evaluator or template that has neutral outputs preserves pass/fail marking as off. - When enabled, configure the pass rule for your output type: - **Numeric outputs**: enter a threshold in **Pass at or above** (defaults to 7 for new evaluators) or drag the synchronized score slider between the minimum and maximum scores. Scores at or above the threshold pass. Existing numeric outputs without score bounds keep their cutoff without the slider. - **String outputs**: click allowed values under **Choose which values pass** to toggle passing status. Selected values display a checkmark. If you update the **Allowed values** list, Agent Observability automatically removes any passing selections that are no longer allowed. Existing passing values remain visible and removable when the evaluator has no allowed-values list. - **Boolean outputs**: select **true** or **false** under **Choose which result passes** to select the passing outcome. Each choice displays whether the value counts as **Pass** or **Fail**. ## Use template variables LLM judge prompts support three explicit variable scopes. The prompt editor puts the scope that matches the rule trigger first: Expand table | Scope | Example variables | Content | |--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------| | Generation | `{{generation.latest_user_message}}`, `{{generation.agent_response}}`, `{{generation.agent_thinking}}`, `{{generation.agent_sequence}}`, `{{generation.system_prompt}}`, `{{generation.tool_calls}}`, `{{generation.tool_results}}`, `{{generation.tools}}`, `{{generation.stop_reason}}`, `{{generation.call_error}}` | Only the model call selected by the trigger. | | Turn | `{{turn.transcript}}`, `{{turn.latest_user_message}}`, `{{turn.agent_response}}`, `{{turn.agent_sequence}}`, `{{turn.tool_calls}}`, `{{turn.tool_results}}`, `{{turn.tools}}` | The user request, intermediate tool steps, and final response. Available for the **User-visible response** trigger. | | Conversation | `{{conversation.transcript}}`, `{{conversation.transcript_detailed}}`, `{{conversation.latest_user_message}}`, `{{conversation.agent_response}}`, `{{conversation.agent_sequence}}`, `{{conversation.tool_calls}}`, `{{conversation.tool_results}}`, `{{conversation.tools}}` | All captured turns. Available for user-visible-response and conversation-idle evaluation. | `{{conversation.transcript_detailed}}` is the conversation transcript with each tool call’s arguments and each tool result, so a judge can check the agent’s claims against what its tools returned. Reasoning and media are replaced by omission markers. `{{turn.transcript}}`, `{{conversation.transcript}}` and `{{conversation.transcript_detailed}}` hold up to 32,000 bytes. A longer transcript is shortened from the middle of long messages, the agent’s earlier messages first. If that is not enough, whole messages are left out, user messages last and spread across the conversation, so the turns that remain still cover all of it. The latest exchange is always kept, and each cut is marked. Use `{{turn.tool_calls}}` instead of `{{generation.tool_calls}}` when evaluating a final response: the final generation usually contains the response, while the tool calls occurred in earlier generations in that turn. For **User-visible response** judges that use turn or conversation variables, capture related generations with the same conversation ID. If Agent Observability can’t load the required context or find the selected generation in that conversation, the evaluation reports an error instead of scoring only the final generation. The playground follows the same rule. Check that the conversation contains the expected generations before running the evaluator again. Existing unscoped variables such as `{{agent_response}}` and `{{tool_calls}}`, underscore turn variables such as `{{turn_tool_calls}}`, and `assistant_*` aliases remain supported. Prefer scoped variables in new prompts because their meaning does not change with the trigger. ## Configure evaluation traffic An evaluation connects one or more evaluators to generation or conversation traffic. Configure: - **Matching conditions**: which traffic to evaluate, by agent name, agent version, operation, model, provider, mode, tags, errors, and `conversation.tool_used` for targeting conversations where a tool was called. An empty **Agent name** condition is ready by default. For the tool filter, **Any** matches when one pattern hits a tool used in the conversation, and **All** matches when every pattern does. - **Trigger**: when evaluation runs: - **User-visible response** (`user_visible_turn`): when the agent produces final text for a user request. Use `turn.*` variables to include its intermediate tool steps. - **Every agent generation** (`all_assistant_generations`): after any agent model call. - **Tool call requested** (`tool_call_steps`): when a generation calls tools. - **Generation failed** (`errored_generations`): when a generation ends with an error. - **Conversation idle** (`conversation`): after the configured idle window. - **Sampling rate**: percentage of matching traffic to evaluate, for example, a 10% rate evaluates approximately one in ten matching items. - **Evaluators**: the evaluators to run. The legacy `turn` selector remains accepted for existing API clients and stored rules, but it is not a separate authoring choice because it fires at the same event as `user_visible_turn`. ## Create an evaluation For the setup flow, refer to [Create an evaluation](/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/evaluation/#create-an-evaluation). You can build an evaluation manually or use Grafana Assistant. To review existing evaluators first, open **Evaluations > New evaluation** and click **See evaluator library**. ## Expose tags on evaluation results Under **When to evaluate**, use **Tags to expose for filtering** to select up to 10 generation tag keys. These tags let you filter or break down evaluation score metrics; they don’t change which traffic matches the evaluation. Each key appears in Prometheus with the `agento11y_eval_tag_` prefix, for example, `team` becomes `agento11y_eval_tag_team`. Keep the number of distinct values small to limit metric series. For conversation-scope rules, Agent Observability uses the value from the last generation in the conversation that matches the rule. ## Use score conditions to trigger actions An evaluation action adds matching conversations to a saved collection. Use it to send failures, examples that need review, or scores that meet a threshold to a review queue. Choose one of these triggers when you add an action: - **When passes**: runs when every evaluator produces a passing verdict. - **When fails**: runs when every evaluator produces a failing verdict. - **When output value**: runs when one evaluator output matches a value that you choose. For an output-value trigger, select the evaluator and output key, then choose a comparison and value. Numeric outputs support equality and greater-than or less-than comparisons. String outputs support equality and contains. Boolean outputs support equality. Use this trigger for a specific score, such as adding conversations whose toxicity score is greater than a threshold, even when the evaluator does not produce a pass/fail verdict. ## Preview an evaluation before you save it When you create or configure an evaluation, the rule builder shows a **Playground** beside the form so you can check your rule before you save it. The playground has two tabs. ### What gets evaluated This tab is a volume simulation: it applies your current criteria and sample rate to recent traffic (for example, the last 6 hours). It is not a forecast of future volume. On high-volume windows, the preview is based on a sample of that traffic. In that case, projected generation volume is labelled as an estimate, and actual volume may differ if the traffic mix changed. Expand **Sample conversations** to see matching traffic grouped into conversations, then open a conversation to see each generation and whether it would have been evaluated, matched but sampled out, or did not match. Select an agent, model, or tag on a generation to add it to your match criteria. When the rule includes LLM judges, the playground also shows an approximate projected cost per day based on sample-rendered judge prompts, model pricing, and projected evaluation volume; real spend varies with transcript length and judge completions. When the selector is `conversation`, the playground shows conversation-level simulation only: each card reports whether the conversation would be evaluated (one evaluation per matching conversation after the idle window) or was sampled out, and links out to the full conversation instead of a per-generation drill-down. For a capped scan, the playground reports conversations selected for evaluation after sampling but does not project hourly or daily unique-conversation volume. Use this tab to tune your selector, match filters, and sampling rate without waiting for scores to arrive asynchronously. ### How it’s evaluated This tab shows what the LLM judge receives for a selected generation — or a selected conversation when the rule selector is `conversation`. Choose which attached evaluator to test, pick a **Source** (**Matched traffic** from the volume preview by default, or **Saved conversations**), then pick a generation or conversation. Inspect the rendered judge prompt with template variables filled in and highlighted. You can test different judge models using the **LLM Model** field before running the evaluation. **Run evaluator** evaluates the selection with that evaluator’s work-in-progress config so you can check pass/fail and rationale before saving. If the test indicates that the judge model ran out of tokens before finishing its response, shorten the prompt or ask the judge for a more compact answer. The generation picker lists only generations eligible for your trigger, in chronological order. For example, **Tool call requested** lists generations with tool calls, and **Generation failed** lists generations with errors. This applies to both matched traffic and saved conversations. Click **Open conversation** to inspect the selected target’s full conversation in a new tab. Your unsaved evaluator changes stay in the builder. ## Gate LLM judges with deterministic evaluators When a rule contains more than one evaluator, choose **Sequential gate** to run them in the displayed order. Each evaluator must pass before the next evaluator runs. Use a deterministic evaluator first to filter out generations that don’t need an LLM judge. Configure a pass condition for every output of each non-final evaluator. If a gate doesn’t pass, later evaluators don’t run. If a gate has an execution error, the evaluation retries according to its normal retry policy. Choose **Parallel** when you want every evaluator to score the same matching traffic independently. Gate evaluator scores control whether later evaluators run, but quality pass-rate reporting separates gate outcomes from the final quality evaluator outcomes. ## Evaluate complete user turns Turn-scope evaluators run when the agent produces its final user-visible response. They include the generations after the previous user-visible response through the final response, so a judge can evaluate a multi-step tool loop without including earlier user requests or waiting for the conversation to become idle. Select the **User-visible response** trigger and use `{{turn.transcript}}`, `{{turn.agent_sequence}}`, `{{turn.tool_calls}}`, or `{{turn.tool_results}}` in an LLM judge prompt. Use `generation.*` when you intentionally need only the final model call, or `conversation.*` when earlier turns also matter. Existing prompts that use `turn_*` or unscoped variables continue to work. Prefer explicit dotted scopes for new prompts; they make the input boundary visible and let the editor warn about likely mistakes. ## Evaluate conversations Conversation-scope evaluators score a full conversation after it has been idle for a configured window. Use them when the quality signal depends on the complete interaction instead of a single generation, for example, whether the agent resolved the user’s request or followed a multi-turn policy. Select the **Conversation idle** trigger and set the idle window. In API configuration, use the `conversation` selector and `min_idle_seconds`. Agent Observability waits until no new generations arrive for that conversation during the window, then runs the evaluator with the full conversation transcript. Use the `{{conversation.transcript}}` template variable in LLM judge prompts to evaluate the complete interaction. Existing conversation-scope LLM judge prompts can also use `{{conversation_transcript}}` to include the full interaction. Generation-specific variables, such as `{{agent_response}}`, aren’t populated for conversation-scope evaluations. The compact transcript removes conversation history replayed in later model inputs, so the judge doesn’t receive the same history for every call. It keeps genuinely repeated requests and new responses. This doesn’t change your stored generations. ## Next steps - [Monitor evaluation results](/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/evaluation/#monitor-results) - [Run an experiment with the SDK](/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/run-an-experiment) - [Explore the evaluation API](/docs/grafana-cloud/observe-and-act/agent-observability/reference/evaluation)