Experiments

Experiments in Grafana Agent Observability help you measure whether a change to an agent improves quality without introducing regressions in cost, latency, or other behaviors. This is offline evaluation: you run it for benchmarking, CI, or release checks before a new or updated agent reaches production.

Your test process runs an agent against known test cases. An experiment records the resulting generations, scores, usage, and artifacts in one report. Experiments don’t replace live monitoring. Use online evaluation to detect behavior in live production traffic, and use experiments to test a controlled change against known scenarios before that change reaches users.

Before you begin

Agent Observability is enabled on your Grafana Cloud stack and receiving generation data.
You have the Agento11y Reader role to view experiments, or the Agento11y Admin role to manage experiments and test suites.
If you run experiments from code, configure the Agent Observability SDK and experiment credentials.

Understand experiment concepts

Experiments organize repeatable inputs and their results:

Object	Purpose
Test case	Defines one scenario with a stable ID, input, and optional expected result, rubric, tags, or reference artifacts.
Test suite	Groups a named, versioned set of test cases.
Experiment	Records one run of an agent candidate against a test suite or another fixed dataset.
Trial	Records one attempt at one test case. Multiple attempts let you measure non-deterministic behavior with pass@k and pass^k.
Score	Records an evaluator result for a trial. A `primary_verdict` score supplies the headline verdict; an unannotated `final` score remains the legacy fallback, while diagnostic scores capture supporting checks without affecting pass rate.
Artifact	Preserves supporting output, for example, a rendered dashboard, structured evaluator details, or a comparison.

A trial can link to the generation, conversation, and trace produced by the agent. These links let you move from an aggregate regression to the exact transcript, tool calls, evaluator explanation, and telemetry that produced it.

Use stable test case IDs across experiments. Agent Observability uses the shared IDs to align baseline and candidate results in side-by-side comparisons.

Understand the experiment lifecycle

An experiment moves through five stages:

Create the experiment. An API call creates the experiment and establishes the experiment ID associated with the test suite version and candidate configuration. Each trial, score, and telemetry link uses this ID.
Run each test case. The test process loops over the suite, invokes the agent, and records each test case attempt as a trial under the experiment.
Connect telemetry and scores. The instrumented agent emits its normal generation, conversation, trace, token, and cost telemetry. The test process links that telemetry to the trial and records the trial’s scores.
Finalize the experiment. After the test process finishes, it finalizes the experiment as completed or failed. Finalization closes the experiment to further trial writes.
Build the report. Agent Observability aggregates the trials, scores, usage, and artifacts in the background. The experiment can briefly show a pending result. When the result is ready, you can review it in the UI and compare it with a baseline.

Build and version test suites

Start with the tasks the agent must reliably complete. Add the expected result or grading rubric where it makes the desired behavior clear, then add failures found through production monitoring.

You can author test cases directly in Experiments > Test suites, promote a conversation from its detail page, promote the saved conversations in a collection into a suite draft, or synchronize a source-controlled YAML or JSON suite with the Agent Observability SDK.

Edit a draft while the dataset is changing. Publish the suite version when you want a stable input set for comparable experiments. When you need to change a published suite, create a new draft and publish a new version. Keep the suite ID and test case IDs stable so reports can distinguish dataset changes from candidate changes.

Run and compare experiments

Your code or benchmark process executes the test suite. Use the Agent Observability SDK to connect that process to the experiment lifecycle and reuse telemetry from your instrumented agent.

Run the same published suite version against each candidate you want to compare. Candidate metadata, for example, agent version, Git commit, model, or prompt version, makes each run identifiable. The comparison view aligns shared test case IDs and shows changes in scores, pass rate, tokens, cost, and artifacts.

From Experiments > Runs, select two or more runs and click Compare runs. The comparison opens a shareable URL with the selected runs and baseline. You can add or remove runs, change the baseline, and compare more than two runs side by side. Deltas are measured against the selected baseline.

The run list supports search, suite, tag, owner, and time-range filters. Filters are saved in the URL so you can share the same view with another user. A trend panel charts the latest runs for the selected suite or tags, and the list shows each run’s score, pass rate, tokens, cost, and duration.

Open a run to inspect each test case result. When a trial captures a conversation or trace, use Open conversation or Open trace to jump from the experiment result to the underlying telemetry.

Compare runs from the same suite version when possible. If selected runs span different suites, Agent Observability compares only test cases with matching IDs, so confirm the matching rows represent the same scenario.

Grafana’s o11y-bench project, built on Harbor, demonstrates this workflow for observability and SRE agents. Its task definitions form a versioned test suite, and each model and reasoning configuration publishes its trials, verifier scores, usage, and artifacts as an Agent Observability experiment.

Authenticate experiment operations

Experiment ingest and Grafana control plane operations use separate credentials:

Endpoint	Credential or permission
`AGENTO11Y_ENDPOINT`: generation ingest, experiment runs, scores, artifacts, and experiment result reads made by the SDK	Grafana Cloud access policy token with `sigil:write`, set in `AGENTO11Y_AUTH_TOKEN`. Set `AGENTO11Y_AUTH_TENANT_ID` to the stack ID when required.
`AGENTO11Y_CONTROL_ENDPOINT`: read stored test suites	Grafana service-account token with the Agento11y Data Reader or Agento11y Reader role, set in `AGENTO11Y_SERVICE_ACCOUNT_TOKEN`.
`AGENTO11Y_CONTROL_ENDPOINT`: create, update, or publish stored test suites	Grafana service-account token with the Agento11y Admin role.

In the UI, users need the same roles: Agento11y Data Reader or Agento11y Reader to view experiments, and Agento11y Admin to manage experiments and test suites.