---
title: "Manage test suites and experiments | Grafana Cloud documentation"
description: "Learn how experiments connect production conversations, test suites, agent evaluations, and delivery workflows in Grafana Agent Observability."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Manage test suites and experiments

Experiments in Grafana Agent Observability help you measure whether a change to an agent improves quality without introducing regressions in cost, latency, or other behaviors. This is offline evaluation: you run it for benchmarking, CI, or release checks before a new or updated agent reaches production.

Your test process runs an agent against known test cases. An experiment records the resulting generations, scores, usage, and artifacts in one report. Experiments don’t replace live monitoring. [Use online evaluation](/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/evaluation) to detect behavior in live production traffic, and use experiments to test a controlled change against known scenarios before that change reaches users.

## Before you begin

- Agent Observability is enabled on your Grafana Cloud stack and receiving generation data.
- You have the Agento11y Reader role to view experiments, or the Agento11y Admin role to manage experiments and test suites.
- If you run experiments from code, configure the Agent Observability SDK and experiment credentials.

## Understand experiment concepts

Experiments organize repeatable inputs and their results:

Expand table

| Object     | Purpose                                                                                                                                                                                                                                    |
|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Test case  | Defines one scenario with a stable ID, input, and optional expected result, rubric, tags, or reference artifacts.                                                                                                                          |
| Test suite | Groups a named, versioned set of test cases.                                                                                                                                                                                               |
| Experiment | Records one run of an agent candidate against a test suite or another fixed dataset.                                                                                                                                                       |
| Trial      | Records one attempt at one test case. Multiple attempts let you measure non-deterministic behavior with pass@k and pass^k.                                                                                                                 |
| Score      | Records an evaluator result for a trial. A `primary_verdict` score supplies the headline verdict; an unannotated `final` score remains the legacy fallback, while diagnostic scores capture supporting checks without affecting pass rate. |
| Artifact   | Preserves supporting output, for example, a rendered dashboard, structured evaluator details, or a comparison.                                                                                                                             |

A trial can link to the generation, conversation, and trace produced by the agent. These links let you move from an aggregate regression to the exact transcript, tool calls, evaluator explanation, and telemetry that produced it.

Use stable test case IDs across experiments. Agent Observability uses the shared IDs to align baseline and candidate results in side-by-side comparisons.

## Interpret trial verdicts

Each trial has at most one headline score. A score with `report_role: primary_verdict` takes precedence over an unannotated `final` score. Without a primary verdict, the report uses the latest `final` score that has no report role.

Scores with `report_role: diagnostic` remain available for inspection but don’t affect the trial verdict, pass rate, pass@k, or pass^k, even when their score key is `final`. Assign only one primary verdict per trial. Multiple primary verdicts cause a report conflict instead of selecting one arbitrarily.

## Understand the experiment lifecycle

An experiment moves through five stages:

1. **Create the experiment.** An API call creates the experiment and establishes the experiment ID associated with the test suite version and candidate configuration. Each trial, score, and telemetry link uses this ID.
2. **Run each test case.** The test process loops over the suite, invokes the agent, and records each test case attempt as a trial under the experiment.
3. **Connect telemetry and scores.** The instrumented agent emits its normal generation, conversation, trace, token, and cost telemetry. The test process links that telemetry to the trial and records the trial’s scores.
4. **Finalize the experiment.** After the test process finishes, it finalizes the experiment as completed or failed. Finalization closes the experiment to further trial writes.
5. **Build the report.** Agent Observability aggregates the trials, scores, usage, and artifacts in the background. The experiment can briefly show a pending result. When the result is ready, you can review it in the UI and compare it with a baseline.

## Build and version test suites

Start with the tasks the agent must reliably complete. Add the expected result or grading rubric where it makes the desired behavior clear, then add failures found through production monitoring.

You can author test cases directly in **Experiments > Test suites**, promote a conversation from its detail page, promote the saved conversations in a collection into a suite draft, or synchronize a source-controlled YAML or JSON suite with the Agent Observability SDK.

When you save a test case or clone cases into a new draft, Agent Observability sanitizes recognized secrets and email addresses in case content, including inputs and expected results. Review the saved content before publishing to make sure redaction hasn’t changed what the case tests. This isn’t comprehensive PII detection: identifiers and referenced artifacts aren’t sanitized, and previously published cases aren’t rewritten.

Edit a draft while the dataset is changing. Publish the suite version when you want a stable input set for comparable experiments. When you need to change a published suite, create a new draft and publish a new version. Keep the suite ID and test case IDs stable so reports can distinguish dataset changes from candidate changes.

## Run and compare experiments

Your code or benchmark process executes the test suite. Use the Agent Observability SDK to connect that process to the experiment lifecycle and reuse telemetry from your instrumented agent.

Run the same published suite version against each candidate you want to compare. Candidate metadata, for example, agent version, Git commit, model, or prompt version, makes each run identifiable. The comparison view aligns shared test case IDs and shows changes in scores, pass rate, tokens, cost, and artifacts.

From **Experiments > Runs**, select two or more runs and click **Compare runs**. The comparison opens a shareable URL with the selected runs and baseline. You can add or remove runs, change the baseline, and compare more than two runs side by side. Deltas are measured against the selected baseline.

Expand a test case row to compare individual criteria and score explanations across runs. Criteria that regress appear before those that improve, followed by unchanged criteria. Click **see more** when an explanation is truncated.

For repeated test cases, the collapsed row shows the mean score and average duration and cost across attempts. The aggregate passes when at least one completed attempt has a verdict and all completed attempts with verdicts pass. Completed attempts without a verdict don’t count toward that pass rate.

In the expanded row, select an attempt independently for each run to inspect its scores, explanations, and artifacts. This selection doesn’t change the aggregate score or delta in the collapsed row.

The run list supports search, suite, tag, owner, and time-range filters. Search matches run names, experiment IDs, suite IDs, and tags across all runs that match your filters, not just the runs already loaded. You don’t need to load more runs before searching. Filters are saved in the URL so you can share the same view with another user. A trend panel charts the latest runs for the selected suite or tags, and the list shows each run’s score, pass rate, tokens, cost, and duration.

Open a run to inspect each test case result. When a trial captures a conversation or trace, use **Open conversation** or **Open trace** to jump from the experiment result to the underlying telemetry.

Compare runs from the same suite version when possible. If selected runs span different suites, Agent Observability compares only test cases with matching IDs, so confirm the matching rows represent the same scenario.

Grafana’s [o11y-bench project](https://github.com/grafana/o11y-bench), built on Harbor, demonstrates this workflow for observability and SRE agents. Its task definitions form a versioned test suite, and each model and reasoning configuration publishes its trials, verifier scores, usage, and artifacts as an Agent Observability experiment.

## Interpret experiment costs

If your test runner supplies a trial cost, Agent Observability keeps that value. Otherwise, it calculates the cost from the trial’s linked conversation usage when available.

For each model in that conversation, the calculation uses your configured price in force at that model’s latest generation, or the catalog price when no configured price applies. Adding a price after a run doesn’t apply that new price to the earlier run. Your configured prices can also cover models that aren’t in the catalog. For setup instructions, refer to [Configure your model prices](/docs/grafana-cloud/observe-and-act/agent-observability/observe-and-troubleshoot/model-prices).

These costs describe your agent’s calls. The monthly LLM judge allowance uses catalog prices, not your configured model prices.

## Authenticate experiment operations

Experiment ingest and Grafana control plane operations use separate credentials:

Expand table

| Endpoint                                                                                                                 | Credential or permission                                                                                                                           |
|--------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
| `AGENTO11Y_ENDPOINT`: generation ingest, experiment runs, scores, artifacts, and experiment result reads made by the SDK | Grafana Cloud access policy token with `sigil:write`, set in `AGENTO11Y_AUTH_TOKEN`. Set `AGENTO11Y_AUTH_TENANT_ID` to the stack ID when required. |
| `AGENTO11Y_CONTROL_ENDPOINT`: read stored test suites                                                                    | Grafana service-account token with the Agento11y Data Reader or Agento11y Reader role, set in `AGENTO11Y_SERVICE_ACCOUNT_TOKEN`.                   |
| `AGENTO11Y_CONTROL_ENDPOINT`: create, update, or publish stored test suites                                              | Grafana service-account token with the Agento11y Admin role.                                                                                       |

In the UI, users need the same roles: `Agento11y Data Reader` or `Agento11y Reader` to view experiments, and `Agento11y Admin` to manage experiments and test suites.

## Keep experiment traffic separate

Experiment telemetry is excluded from online evaluation by default. To let generations tagged with the experiment ID match online evaluation rules, set `online_evaluations_enabled` to `true` when you create the run. You can’t change this setting for an existing run. Keep the default to leave offline test traffic out of production quality metrics.

## Remove finished experiments

With the Agent Observability Admin role, you can delete runs you no longer need:

1. Open **Experiments** and find **Experiment Runs**.
2. Select the checkboxes for up to 25 completed, failed, or canceled runs.
3. Click **Delete**.
4. Review the selected runs and click **Delete** to confirm.

Deletion permanently removes the selected runs and their artifacts. You can’t delete a running experiment. Cancel or finalize it first.

## Next steps

- [Run an experiment with the SDK](/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/run-an-experiment)
- [Inspect and save production conversations](/docs/grafana-cloud/observe-and-act/agent-observability/observe-and-troubleshoot/conversations/#save-conversations)
- [Explore the experiments API](/docs/grafana-cloud/observe-and-act/agent-observability/reference/experiments)