Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Observe and actbreadcrumb arrow Agent Observabilitybreadcrumb arrow Configurebreadcrumb arrow Tune evaluation settings
Grafana Cloud

Configure online evaluation

Note

Agent Observability Evaluations is currently in public preview. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.

Online evaluation continuously scores live generation traffic. Evaluators define the scoring logic, and evaluations define which traffic to score.

Create evaluators

Use Agent Observability or the evaluation API to create evaluators. Four evaluator types are available:

LLM judge

An LLM judge uses an LLM to score generations based on criteria you define in a prompt template.

Key settings:

  • provider and model: the LLM to use for judging.
  • system_prompt and user_prompt: prompt templates with variables.
  • max_tokens, temperature, and timeout_ms: generation controls.

JSON schema

Validates that the agent response matches a JSON schema. Returns true or false.

Regex

Checks the agent response against one or more regex patterns. Use reject: true to invert the match.

Heuristic

Applies a rule tree with AND/OR logic. Supported checks: not_empty, contains, not_contains, min_length, max_length.

Set an evaluation target

JSON schema, regex, and heuristic evaluators evaluate the agent response by default. You can choose another field instead:

TargetDescription
responseAssistant response text (default).
inputUser input text.
system_promptThe system prompt.

Set the target in Evaluate against when you create a non-LLM-judge evaluator, or set the target field in the evaluator config JSON.

Use this for lightweight detection of injected content in generation input without requiring an LLM judge.

Configure pass verdicts

Boolean outputs from heuristic, regex, JSON schema, and LLM judge evaluators record a pass or fail verdict only when you explicitly configure a pass_value on the output key. When pass_value is omitted, the score is recorded without a verdict.

Use template variables

LLM judge prompts support these template variables:

VariableContent
{{latest_user_message}}Most recent user message
{{user_history}}All user messages
{{agent_response}}Agent output
{{agent_thinking}}Thinking/reasoning content
{{agent_sequence}}Ordered agent output parts
{{system_prompt}}System prompt
{{tool_calls}}Tool call details
{{tool_results}}Tool result details
{{tools}}Available tool definitions
{{call_error}}Error information
{{conversation_transcript}}Full conversation transcript for conversation-scope evaluators

Configure evaluation traffic

An evaluation connects one or more evaluators to generation or conversation traffic. Configure:

  • Selector: which generations to evaluate:
    • user_visible_turn: agent text responses without tool calls.
    • all_assistant_generations: any agent output.
    • tool_call_steps: generations with tool calls.
    • errored_generations: generations with errors.
    • conversation: the full conversation after an idle window.
  • Match filters: additional criteria to narrow the selection (agent name, model, mode, tags, errors, and conversation.tool_used for targeting conversations where a specific tool was called).
  • Sampling rate: percentage of matching generations to evaluate.
  • Evaluators: the evaluators to run.

Evaluate conversations

Conversation-scope rules evaluate the complete conversation instead of one generation. Set the selector to conversation and configure min_idle_seconds so Agent Observability waits for the interaction to settle before it scores the transcript.

Conversation-scope LLM judge prompts can use {{conversation_transcript}} to include the full interaction. Generation-specific variables, such as {{agent_response}}, aren’t populated for conversation-scope evaluations.

Next steps