Documentation Grafana Cloud Observe and act Agent Observability Configure Tune evaluation settings

Configure online evaluation

Online evaluation continuously scores live generation traffic. Evaluators define the scoring logic, and evaluations define which traffic to score.

Create evaluators

Use Agent Observability or the evaluation API to create evaluators. Four evaluator types are available:

LLM judge

An LLM judge uses an LLM to score generations based on criteria you define in a prompt template.

Key settings:

• provider and model: the LLM to use for judging.
• system_prompt and user_prompt: prompt templates with variables.
• max_tokens, temperature, and timeout_ms: generation controls.

JSON schema

Validates that the agent response matches a JSON schema. Returns true or false.

Regex

Checks the agent response against one or more regex patterns. Use reject: true to invert the match.

Heuristic

Applies a rule tree with AND/OR logic. Supported checks: not_empty, contains, not_contains, min_length, max_length.

Set an evaluation target

JSON schema, regex, and heuristic evaluators evaluate the agent response by default. You can choose another field instead:

Set the target in Evaluate against when you create a non-LLM-judge evaluator, or set the target field in the evaluator config JSON.

Your application can send the system prompt in either of two ways, depending on the SDK and framework you use: in the generation’s own system prompt field, or as a message with the system role inside the chat history. The system_prompt target and the {{system_prompt}} template variable accept both. They read the system prompt field first, and the text of the system messages when that field is empty.

Use this for lightweight detection of injected content in generation input without requiring an LLM judge.

Configure pass verdicts

Boolean outputs from heuristic, regex, JSON schema, and LLM judge evaluators record a pass or fail verdict only when you explicitly configure a pass_value on the output key. When pass_value is omitted, the score is recorded without a verdict.

Use template variables

LLM judge prompts support these template variables:

Configure evaluation traffic

An evaluation connects one or more evaluators to generation or conversation traffic. Configure:

• Selector: which generations to evaluate:
  • user_visible_turn: agent text responses without tool calls.
  • all_assistant_generations: any agent output.
  • tool_call_steps: generations with tool calls.
  • errored_generations: generations with errors.
  • conversation: the full conversation after an idle window.
• Match filters: additional criteria to narrow the selection (agent name, model, mode, tags, errors, and conversation.tool_used for targeting conversations where a specific tool was called).
• Sampling rate: percentage of matching generations to evaluate.
• Evaluators: the evaluators to run.

Create an evaluation

Open Evaluations → New evaluation. Choose how to start:

• Build with Assistant — describe the quality you care about; Grafana Assistant creates the evaluator and evaluation.
• See evaluator library — open the evaluator library to review what your workspace already has.
• Create new evaluation — fill the rule and judge by hand in the shared builder.

Scoring starts on new traffic after you create the evaluation; historical generations are not backfilled.

Preview an evaluation before you save it

When you create or configure an evaluation, the rule builder shows a Playground beside the form so you can check your rule before you save it. The playground has two tabs.

What gets evaluated

This tab is a volume simulation: it applies your current criteria and sample rate to recent traffic (for example, the last 6 hours). It is not a forecast of future volume. On high-volume windows, the preview is based on a sample of that traffic. In that case, projected generation volume is labelled as an estimate, and actual volume may differ if the traffic mix changed. Matching traffic is grouped into conversations. Open a conversation to see each generation and whether it would have been evaluated, matched but sampled out, or did not match. Select an agent, model, or tag on a generation to add it to your match criteria. When the rule includes LLM judges, the playground also shows an approximate projected cost per day based on sample-rendered judge prompts, model pricing, and projected evaluation volume; real spend varies with transcript length and judge completions.

When the selector is conversation, the playground shows conversation-level simulation only: each card reports whether the conversation would be evaluated (one evaluation per matching conversation after the idle window) or was sampled out, and links out to the full conversation instead of a per-generation drill-down. For a capped scan, the playground reports conversations selected for evaluation after sampling but does not project hourly or daily unique-conversation volume.

Use this tab to tune your selector, match filters, and sampling rate without waiting for scores to arrive asynchronously.

How it’s evaluated

This tab shows what the LLM judge receives for a selected generation — or a selected conversation when the rule selector is conversation. Choose which attached evaluator to test, pick a Source (Matched traffic from the volume preview by default, or Saved conversations), then pick a generation or conversation. Inspect the rendered judge prompt with template variables filled in and highlighted. Run evaluator evaluates the selection with that evaluator’s work-in-progress config so you can check pass/fail and rationale before saving.

Gate LLM judges with deterministic evaluators

When a rule contains more than one evaluator, choose Sequential gate to run them in the displayed order. Each evaluator must pass before the next evaluator runs. Use a deterministic evaluator first to filter out generations that don’t need an LLM judge.

Configure a pass condition for every output of each non-final evaluator. If a gate doesn’t pass, later evaluators don’t run. If a gate has an execution error, the evaluation retries according to its normal retry policy.

Choose Parallel when you want every evaluator to score the same matching traffic independently.

Evaluate conversations

Conversation-scope rules evaluate the complete conversation instead of one generation. Set the selector to conversation and configure min_idle_seconds so Agent Observability waits for the interaction to settle before it scores the transcript.

Conversation-scope LLM judge prompts can use {{conversation_transcript}} to include the full interaction. Generation-specific variables, such as {{agent_response}}, aren’t populated for conversation-scope evaluations.

Next steps

• Set up evaluation end-to-end
• Use Agent Observability