Use Agent Observability with Assistant

Use Grafana Assistant with Grafana Agent Observability to inspect agent conversations and model calls, define quality checks, and review evaluation scores. An evaluator defines how to score a response. An evaluation rule selects the traffic to score and the evaluators to run.

Before you begin

Enable Grafana Assistant and install and configure Agent Observability on your stack. You need access to the agent telemetry. To create or update evaluators and rules, you also need permission to manage evaluation settings, and online evaluations must be available for your stack.

Capture representative conversations before you create a rule. Checks that depend on tool results or system instructions need that information in the captured telemetry.

Inspect agent conversations

Ask Assistant to find conversations for an agent and time range. You can inspect individual model calls, errors, token usage, and captured messages and tool results. Assistant can also list agents and their versions.

Find conversations for my checkout agent from the last hour and inspect the model calls that failed.

Use the captured evidence to identify a specific behavior you want to check. Assistant can read existing evaluators, rules, and scores before recommending a new evaluation.

Choose an evaluator

Start with one focused question, for example, whether the final response accurately describes the tool results.

Review recent conversations for my checkout agent and recommend a check for responses that contradict tool results.

You can start from an evaluation template or a built-in evaluator. Ask Assistant to create an editable copy for your stack before attaching it to a rule. You can customize the judge provider and model when they’re available.

For a custom LLM judge, describe the scoring criteria separately from the evidence it reads. Assistant supports evidence templates for conversations, user-visible turns, individual model calls, tool calls, and failed calls. Choose evidence that matches the rule’s scope.

Specify what counts as a passing result. For a numeric score, include its range and passing threshold, for example, a score from 1 to 10 with 7 or higher passing. Simple checks for text, regular expressions, or JSON structure don’t need a judge model call.

Apply a rule to live traffic

Ask Assistant to configure the evaluator for the agent and traffic you want to monitor. Review and approve each configuration change before it takes effect.

Create a rule for my checkout agent’s user-visible turns using the tool-result check. Score 20% of matching traffic.

Rules score new matching traffic asynchronously. They don’t backfill historical conversations. LLM judges make model calls, so start with a sample and increase it when the results are useful.

For checks that need the entire conversation, specify how long the conversation should be idle before evaluation. For checks about a tool call or a failed model call, use that scope rather than user-visible turns.

Review and refine results

Use the links in Assistant’s response to open the evaluator or rule in Agent Observability. Ask Assistant to inspect scores and compare examples that pass and fail before you expand the rule.

Show failing scores for my checkout agent’s tool-result rule from the last day and compare them with passing examples.

If new conversations appear but no scores do, check that the rule is enabled and that its agent filter and traffic scope match the captured data. You can ask Assistant to update an existing rule instead of creating a duplicate.

Evaluation scores are review signals, not proof of compliance or a guarantee that an agent is safe. Use the underlying conversation and tool evidence to interpret them.