Documentation Grafana Cloud Observe and act Agent Observability Guides Score quality

Set up online evaluation

Online evaluation lets you score your agents’ live production traffic automatically. This guide walks you through creating your first evaluation.

Before you begin

• If you plan to use an LLM judge, configure at least one judge provider. Refer to Configure evaluation for provider setup.
• Your instrumented agent is sending traffic to Agent Observability.
• You have the Agento11y Admin role.

Understand evaluations and evaluators

An evaluator defines how to score an agent response. You can reuse the same evaluator in multiple evaluations. Available evaluator types include:

• LLM judge scores responses against natural-language criteria.
• JSON schema validates structured output.
• Regex checks text for a pattern.
• Heuristic applies deterministic rules, such as length or value checks.

An evaluation applies one or more evaluators to selected production traffic. It includes the selector, match criteria, sampling rate, evaluators, alerts, and actions. You configure these together in the evaluation setup flow.

Create an evaluation

The setup flow creates the evaluators and rule needed to start scoring:

1. In Agent Observability, go to Evaluations.
2. Click New evaluation.
3. Select Create manually, or select Use Assistant to describe the evaluation you want.
4. Under Choose evaluators, select evaluators from your library or create one from a template or from scratch.
5. Under Where to apply, choose the traffic, match criteria, agents, and sampling rate.
6. Optionally add alerts or actions.
7. Under Review & start, verify the configuration and click Start scoring.

To reuse or edit evaluator definitions, click Browse all evaluators under Choose evaluators.

Write an LLM judge prompt

For an LLM judge evaluator, write a prompt that describes the scoring criteria. Use template variables to inject generation data:

text Copy

You are evaluating the quality of an AI agent response.

User message:
{{latest_user_message}}

Agent response:
{{agent_response}}

Rate the response quality on a scale of 1-5, where 1 is poor and 5 is excellent.
Consider accuracy, helpfulness, and clarity.

Return only the numeric score.

Select the traffic to evaluate

The selector determines which part of your agent’s traffic an evaluation can score:

• User-visible turn: evaluates final agent text shown to a user.
• All agent generations: evaluates every model generation, including intermediate steps.
• Tool call steps: evaluates generations in which the model calls tools.
• Errored generations: evaluates generations that failed with a call error.
• Conversation: evaluates the complete conversation after its configured idle window.

Use match criteria to narrow the selection by agent, agent version, operation, model, provider, mode, tool, or tag. The sampling rate controls what fraction of matching traffic is evaluated. For example, a 10% rate evaluates approximately one in ten matching items.

Evaluate whole conversations

Conversation-scope evaluators score a full conversation after it has been idle for a configured window. Use them when the quality signal depends on the complete interaction instead of a single generation, for example, whether the agent resolved the user’s request or followed a multi-turn policy.

When conversation-scope evaluation is enabled, select Conversation as the rule selector and set the idle window. Agent Observability waits until no new generations arrive for that conversation during the window, then runs the evaluator with the full conversation transcript.

Use the {{conversation_transcript}} template variable in LLM judge prompts to evaluate the complete interaction.

Set the evaluation target

For JSON schema, regex, and heuristic evaluators, you can choose which text the evaluator analyzes:

• Response (default): the assistant’s output text.
• Input: the user’s input text.
• System prompt: the system prompt.

Select the target in Evaluate against when you create or edit an evaluator. Use this to detect prompt injection or validate input structure without an LLM judge.

Create alerts

Add an alert when you want evaluation results to notify a team:

1. Create an evaluation, or open an existing evaluation and click Configure.
2. In Alerts, click Add alert.
3. Set a pass-rate threshold (for example, alert when fewer than 90% of evaluations pass).
4. Choose a contact point for notifications.
5. Start the evaluation or save your changes.

Agent Observability creates a Grafana alert rule that you can edit later in Grafana Alerting.

Route conversations with actions

Actions automatically add matching conversations to collections for review, annotation, or promotion into a test suite. An action runs only after every evaluator attached to the evaluation produces the selected pass or fail verdict.

For example, to route failed conversations to a collection:

1. Create an evaluation, or open an existing evaluation and click Configure.
2. In Actions, click Add action.
3. For Trigger, select When fails.
4. For Collections, select or create one or more collections.
5. Start the evaluation or save your changes.

When every evaluator fails, Agent Observability saves the conversation and adds it to each selected collection. Select When passes instead to collect successful conversations.

Actions require evaluators that produce pass or fail verdicts. For numeric or string evaluator outputs, configure a pass threshold or matching value before attaching an action.

Monitor results

Evaluation scores appear in several places:

• Conversation detail: scores displayed next to each evaluated generation, with pass/fail/neutral counts in the metrics bar.
• Evaluation overview: live stats, configured evaluations, and score distributions.
• Analytics dashboards: quality metrics alongside cost and performance data.

Use the evaluation dashboard to identify quality regressions after prompt changes, compare evaluator results across agent versions, and spot patterns in low-scoring generations.

Send custom code-based scores

Use custom code-based evaluators when your scoring logic needs to run outside Agent Observability, such as a proprietary classifier, a deterministic business-rule check, or an evaluation service that already runs in your infrastructure. When your application has a generation ID and an evaluation result, post the score to POST /api/v1/scores:export on your stack-specific Agent Observability API URL. Use a stable score_id for retries, set generation_id to the generation being evaluated, and identify your evaluator with evaluator_id, evaluator_version, and score_key so the result appears consistently in conversation details and quality reporting.

For the API URL, instance ID, and access token setup, refer to Use Agent Observability on Grafana Cloud.

Bash Copy

curl -X POST "${AGENTO11Y_ENDPOINT}/api/v1/scores:export" \
  -H 'Content-Type: application/json' \
  -u "${AGENTO11Y_AUTH_TENANT_ID}:${AGENTO11Y_AUTH_TOKEN}" \
  -d '{
    "scores": [
      {
        "score_id": "sc_custom_helpfulness_gen_123",
        "generation_id": "gen_123",
        "evaluator_id": "custom.helpfulness",
        "evaluator_version": "2026-06-22",
        "score_key": "helpfulness",
        "value": { "number": 0.92 },
        "passed": true,
        "explanation": "The response answered the user question directly.",
        "source": { "kind": "external_api", "id": "custom-eval-worker" }
      }
    ]
  }'

Online scores help you find production failures that belong in your regression suite. From a conversation, use Promote to test case to add the scenario to a suite draft. You can also use an action to route failed conversations into a collection and promote related conversations together.

Run the resulting suite against proposed prompt, model, tool, or agent changes before deployment. For the full workflow and a Python SDK example, refer to Run an experiment.

Next steps

• Use built-in dashboards
• Configure evaluator types
• Run an experiment