Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Observe and actbreadcrumb arrow Agent Observabilitybreadcrumb arrow Evaluate and improvebreadcrumb arrow Evaluations
Grafana Cloud

Set up and monitor online evaluation

Online evaluation lets you score your agents’ live production traffic automatically. Create an evaluation and review its results first. Then use the sections below to refine scoring, route conversations, or add alerts. For detailed settings, refer to Configure evaluators and evaluation rules.

Before you begin

  • If you plan to use an LLM judge, choose a provider and model available in your stack. Refer to Choose a judge provider.
  • Your instrumented agent is sending traffic to Agent Observability.
  • You have the Agent Observability Admin role.

Understand evaluations and evaluators

An evaluator defines how to score an agent response. You can reuse the same evaluator in multiple evaluations. Available evaluator types include:

  • LLM judge scores responses against natural-language criteria.
  • JSON schema validates structured output.
  • Regex checks text for a pattern.
  • Heuristic applies deterministic rules, for example, length or value checks.

For generated summaries and template descriptions, refer to Evaluator descriptions.

An evaluation applies one or more evaluators to selected production traffic. It includes the selector, match criteria, sampling rate, evaluators, alerts, SLOs, and actions. You configure these together in the evaluation setup flow.

Create an evaluation

The setup flow creates the evaluators and rule needed to start scoring:

  1. In Agent Observability, go to Evaluations.
  2. Click New evaluation.
  3. Click Create a new evaluation manually.
  4. Under When to evaluate, choose the traffic, match criteria, agents, and sampling rate.
  5. Under What to evaluate, click Add evaluator. Select Use existing evaluator to choose from your library or built-in templates, or select Create new evaluator. For LLM judge model options, refer to LLM judge settings.
  6. Under What happens with the results?, optionally add alerts, SLOs, or actions.
  7. Click Create evaluation, review the summary, and click Create evaluation again to confirm.

To use Grafana Assistant instead, describe the evaluation under Build with Assistant and click Create.

Scoring starts on new traffic. Historical generations aren’t backfilled.

Before you start an evaluation, review its estimated match volume. For volume estimates and testing an evaluator, refer to Preview an evaluation before you save it.

Monitor results

Evaluation scores appear in several places:

  • Conversation detail: scores displayed next to each evaluated generation, with pass/fail/neutral counts in the metrics bar.
  • Evaluation overview: live stats, configured evaluations, and score distributions.
  • Analytics dashboards: quality metrics alongside cost and performance data.

When you query evaluation execution, duration, or score metrics in Prometheus, eval_ai_request_provider and eval_ai_request_model identify the LLM judge. The gen_ai_request_provider and gen_ai_request_model labels identify the agent generation being evaluated. Use the judge labels to compare evaluators without confusing their models with your agents’ models.

Use the evaluation dashboard to identify quality regressions after prompt changes, compare evaluator results across agent versions, and spot patterns in low-scoring generations.

The AI analysis bar appears directly below the KPI summary band on the Evaluations overview page. It evaluates active filters, execution counts, average pass rates, failed scores, and estimated judge costs against the previous period to surface:

  • Evaluations with the lowest pass rates.
  • The largest pass-rate drops (in percentage points) compared to the previous period.
  • The most expensive evaluations by judge cost.
  • Enabled evaluations with no executions in the selected time range.

Click the AI analysis bar to expand or collapse findings. When collapsed, the header displays a preview of the first finding. Click Regenerate to refresh the analysis for current filters. Click any finding to open Grafana Assistant and explain it.

The bar requires Grafana Assistant; it doesn’t appear if Grafana Assistant is disabled. While queries are loading or if no evaluations match your filters, the bar stays mounted and waits for data before generating insights. If a metric query fails or data is unavailable, the analysis labels it as no data instead of reporting a measured zero, omits unavailable previous-period comparisons, and distinguishes missing rule metrics from evaluations with zero executions.

Use Exclude selected in the Provider, Model, or Agent filter to omit selected traffic from the results view. This doesn’t change which traffic your evaluation rules score. For instructions, refer to Include or exclude traffic.

Write an LLM judge prompt

For an LLM judge evaluator, write a prompt that describes the scoring criteria. Use template variables to inject generation data:

text 
You are evaluating the quality of an AI agent response.

<evaluated_content>
<user_message>
{{generation.latest_user_message}}
</user_message>
<agent_response>
{{generation.agent_response}}
</agent_response>
</evaluated_content>

Rate the response quality on a scale of 1-5, where 1 is poor and 5 is excellent.
Consider accuracy, helpfulness, and clarity.

Keep evaluator instructions in the system prompt and template variables in the user prompt. Agent Observability escapes and bounds variable values, requests structured output from the judge provider, and validates the returned score against your configured output type, range, or enum. Agent Observability derives pass or fail from your configured threshold, value, or match; the judge model cannot set it directly. For details on output setup, score ranges, and pass/fail marking, refer to Configure outputs and pass verdicts.

Choose when to evaluate

Choose a trigger, matching conditions, and sampling rate under When to evaluate. For the available settings, refer to Configure evaluation traffic. To add score metric filters without changing traffic matching, refer to Expose tags on evaluation results.

Choose evaluator order

Run evaluators independently or use a deterministic check before an LLM judge. For execution modes and pass conditions, refer to Gate LLM judges with deterministic evaluators.

Evaluate whole conversations

Use conversation evaluation when quality depends on the complete interaction. For the idle window and transcript settings, refer to Evaluate conversations.

Evaluate complete user turns

Use turn evaluation to score a user request and its intermediate tool steps when the final response arrives. For trigger and prompt settings, refer to Evaluate complete user turns.

Choose the content to check

Choose the generation field that a JSON schema, regex, or heuristic evaluator checks. For supported fields and how to select them, refer to Choose the content to check.

Route conversations with actions

Actions automatically add matching conversations to collections for review, annotation, or promotion into a test suite. For available triggers, refer to Use score conditions to trigger actions.

For example, to route failed conversations to a collection:

  1. Create an evaluation, or open an existing evaluation and click Edit evaluation.
  2. In Actions, click Add action.
  3. For Trigger, select When fails.
  4. For Collections, select or create one or more collections.
  5. Click Create evaluation and confirm, or click Save changes for an existing evaluation.

When every evaluator fails, Agent Observability saves the conversation and adds it to each selected collection. Select When passes instead to collect successful conversations.

For When fails or When passes, configure a pass condition on the evaluator outputs. For verdict settings, refer to Configure outputs and pass verdicts.

Create alerts

Add an alert when you want evaluation results to notify a team:

  1. Create an evaluation, or open an existing evaluation and click Edit evaluation.
  2. In Alerts, click Add alert.
  3. Set a pass-rate threshold, for example, alert when fewer than 90% of evaluations pass.
  4. Choose a contact point for notifications.
  5. Click Create evaluation and confirm, or click Save changes for an existing evaluation.

Agent Observability creates a Grafana alert rule that you can edit later in Grafana Alerting.

If Add alert is unavailable, its tooltip explains whether Grafana Alerting is disabled or you don’t have permission to create alerts. Ask your Grafana administrator to enable Grafana Alerting or grant the required permissions.

Create an SLO

When the Grafana SLO app is installed, you can track an evaluation’s pass rate against a target as a service level objective (SLO):

  1. Create an evaluation, or open an existing evaluation and click Edit evaluation.
  2. In SLO, click Add SLO. The button is available only when you have permission to create SLOs and the evaluation produces a pass or fail verdict (at least one evaluator has a pass condition).
  3. Set a Target pass rate. For evaluations with at least 50 scores in the last 28 days, Agent Observability shows the current pass rate and prefills a recommended target; otherwise enter one manually.
  4. Optionally turn on Create burn-rate alert and add labels.
  5. Click Create evaluation and confirm, or click Save changes for an existing evaluation.

Agent Observability creates a provisioned SLO in the Grafana SLO app, measured over a rolling 28-day window, and labels it with agento11y_eval_slo="true" and agento11y_rule set to your evaluation ID. Deleting the evaluation also deletes its SLO. Dashboard and SLO rule open the generated dashboard and the SLO definition. You can also edit or delete provisioned SLOs from the SLO app.

SLO alerts are routed via Grafana notification policies. Match agento11y_eval_slo="true" and grafana_slo_severity, or any extra labels you add, to deliver them.

To create SLOs for other agent behaviors, for example, request success rate or latency, or to view the Agent Observability SLOs you already have, open Analytics > SLOs. Describe the objective, or select a starting point, then click Build with Assistant. Grafana Assistant discovers the metrics, confirms a target with you, and creates the SLO in the Grafana SLO app.

Use the AI Act readiness templates

The evaluator template library includes four AI Act readiness templates:

  • Fundamental rights harm flags an evident unjustified discriminatory or fundamental-rights harm in a captured trace.
  • Human oversight flags a high-impact recommendation presented without a human-review, escalation, or uncertainty cue.
  • Decision explanation checks whether an applicable high-impact recommendation explains its basis, limits, and review path.
  • Tool action safety checks captured tool activity for an evident unsafe or unauthorised action, including following instructions embedded in untrusted content.

These are forkable LLM-judge templates. Use them only with the system prompt, user history, tool calls, tool results, and final response captured for the agent. Start with a conservative sample rate and route failures to a review collection before using an alert or enforcement action.

The templates are evidence signals, not a finding of AI Act compliance. In particular, they cannot establish representative data governance, aggregate disparate impact, a UI disclosure shown to a user, or whether a human actually reviewed a decision. Assess those through versioned offline suites, aggregate analysis, deployment controls, and legal review as appropriate for the system’s intended use and risk classification.

Use the HIPAA readiness templates

The evaluator template library includes two HIPAA readiness templates. They are relevant only when the agent is used by a HIPAA covered entity or business associate and the captured trace contains the required prompt and tool telemetry:

  • PHI unauthorized disclosure fails only when the final response exposes identifiable health information and the trace does not show an authorized recipient or supported purpose. PHI combines an identity with information about health, health care, or payment for health care; generic PII is not enough.
  • PHI minimum necessary flags a clear case where captured tool activity or the final response accesses, uses, or discloses more PHI fields, people, records, or detail than the stated task requires.

Use the minimum-necessary template as a conservative review signal: it returns a failure only for a clear excess and passes when no PHI is involved, the scope is proportionate, or telemetry cannot establish scope. Neither template proves HIPAA coverage, authorization, legal compliance, clinical appropriateness, access-control enforcement, audit controls, or a business-associate agreement. Start with sampled review and fork the templates to reflect the organization’s permitted-use and authorization policy.

Detect unredacted secrets

The evaluator template library includes four deterministic, response-scoped regex templates:

  • Private-key leak fails when a response contains a PEM or PGP private-key block.
  • Known secret format leak fails when a response contains a recognized high-confidence format, such as a bearer token or a provider access key. Its patterns are adapted from the SDK’s Gitleaks-style redaction table.
  • Labelled credential leak fails when a response assigns a value to a recognizable password, secret, token, or key field. This is deliberately broader and should begin as a review signal.
  • Connection-string credential leak fails when a response contains embedded credentials in a database, cache, or message-broker connection string.

These templates only flag recognizable formats; they do not prove a value is valid, and they cannot detect every secret. They intentionally do not use a generic high-entropy pattern because that produces too many false positives. Fork and tune them for the credentials your application actually handles. Use a preflight redaction guard if the objective is to prevent a secret reaching the model; use these evaluators to detect a secret that appeared in captured output.

Assess agentic safety and tool-result quality

The evaluator template library also includes three trace-aware LLM-judge templates. They use the system prompt, user history, tool calls, tool results, and final response:

  • Prompt injection resistance fails only when the agent materially follows an attempted instruction override, exfiltration request, untrusted tool instruction, or approval-boundary bypass.
  • Tool-result groundedness fails when the final response claims an outcome absent from or contradicted by captured tool results, or presents a failed tool call as a success.
  • External-action authorization fails when captured tool use sends, deletes, deploys, changes access, or performs another consequential external action without apparent user authorization.

Each is a captured-trace signal, not evidence of a complete security control. Enable them only where the required prompt and tool telemetry is captured, start with review actions, and fork them to reflect your organization’s authorization and policy boundaries.

Review rule edit history

Use edit history to check who changes an evaluation rule and compare saved configurations.

  1. In Evaluations, open an evaluation and click Edit evaluation. If you have read-only access, click View evaluation.
  2. Click View edit history, the history icon below the evaluation name.
  3. In Recent edits, review the editor, date, and changed fields, for example, Sample rate, Evaluators, and Match criteria.
  4. Select the checkboxes for two edits in Compare. Under Compare edits, review the side-by-side comparison of the saved rule configuration after each edit. Clear a checkbox to choose a different edit.

The drawer also shows who created the rule and when. You can review the latest 10 recorded edits. If No edits recorded yet appears, history starts when you next save a rule change; earlier changes aren’t available. You need at least two recorded edits to compare configurations.

Find evaluation resources

In Evaluations, click More to open related resources:

  • View evaluators opens your evaluator library.
  • View evaluator templates opens the templates you can use to create evaluators.
  • View evaluation SLOs opens the Grafana SLO app filtered to evaluation SLOs. This option appears when the app is installed and enabled.
  • View evaluation alerts opens Grafana Alerting filtered to evaluation alerts. This option appears when Grafana Alerting is enabled.

Older alerts might not appear in the filtered list until you save an update to them through Agent Observability.

Delete an evaluator

To remove an evaluator you no longer use, open it from the evaluator library and open its edit page. Click Delete, then click Delete in the confirmation dialog. You can’t undo this action.

If an evaluation rule uses the evaluator, remove it from every referencing rule before deleting it. Delete is also unavailable while Agent Observability checks rule usage or can’t load the rules. Read the tooltip for the reason, and reload the page if the usage check fails.

Send custom code-based scores

Use custom code-based evaluators when your scoring logic needs to run outside Agent Observability, for example, a proprietary classifier, a deterministic business-rule check, or an evaluation service that already runs in your infrastructure. When your application has a generation ID and an evaluation result, post the score to POST /api/v1/scores:export on your stack-specific Agent Observability API URL. Use a stable score_id for retries, set generation_id to the generation being evaluated, and identify your evaluator with evaluator_id, evaluator_version, and score_key so the result appears consistently in conversation details and quality reporting.

For the API URL, instance ID, and access token setup, refer to Use Agent Observability on Grafana Cloud.

Bash 
curl -X POST "<AGENTO11Y_ENDPOINT>/api/v1/scores:export" \
  -H 'Content-Type: application/json' \
  -u "<AGENTO11Y_AUTH_TENANT_ID>:<AGENTO11Y_AUTH_TOKEN>" \
  -d '{
    "scores": [
      {
        "score_id": "sc_custom_helpfulness_gen_123",
        "generation_id": "gen_123",
        "evaluator_id": "custom.helpfulness",
        "evaluator_version": "2026-06-22",
        "score_key": "helpfulness",
        "value": { "number": 0.92 },
        "passed": true,
        "explanation": "The response answered the user question directly.",
        "source": { "kind": "external_api", "id": "custom-eval-worker" }
      }
    ]
  }'

Online scores help you find production failures that belong in your regression suite. From a conversation, use Promote to test case to add the scenario to a suite draft. You can also use an action to route failed conversations into a collection and promote related conversations together.

Run the resulting suite against proposed prompt, model, tool, or agent changes before deployment. For the full workflow and a Python SDK example, refer to Run an experiment with the SDK.

Next steps