Experiments
Note
is currently in public preview. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.
Experiments in Grafana Agent Observability help you measure whether a change to an agent improves quality without introducing regressions in cost, latency, or other behaviors. This is offline evaluation: you run it for benchmarking, CI, or release checks before a new or updated agent reaches production.
Your test process runs an agent against known test cases. An experiment records the resulting generations, scores, usage, and artifacts in one report. Experiments don’t replace live monitoring. Use online evaluation to detect behavior in live production traffic, and use experiments to test a controlled change against known scenarios before that change reaches users.
Understand experiment concepts
Experiments organize repeatable inputs and their results:
A trial can link to the generation, conversation, and trace produced by the agent. These links let you move from an aggregate regression to the exact transcript, tool calls, evaluator explanation, and telemetry that produced it.
Use stable test case IDs across experiments. Agent Observability uses the shared IDs to align baseline and candidate results in side-by-side comparisons.
Understand the experiment lifecycle
An experiment moves through five stages:
- Create the experiment. An API call creates the experiment and establishes the experiment ID associated with the test suite version and candidate configuration. Each trial, score, and telemetry link uses this ID.
- Run each test case. The test process loops over the suite, invokes the agent, and records each test case attempt as a trial under the experiment.
- Connect telemetry and scores. The instrumented agent emits its normal generation, conversation, trace, token, and cost telemetry. The test process links that telemetry to the trial and records the trial’s scores.
- Finalize the experiment. After the test process finishes, it finalizes the experiment as completed or failed. Finalization closes the experiment to further trial writes.
- Build the report. Agent Observability aggregates the trials, scores, usage, and artifacts in the background. The experiment can briefly show a pending result. When the result is ready, you can review it in the UI and compare it with a baseline.
Build and version test suites
Start with the tasks the agent must reliably complete. Add the expected result or grading rubric where it makes the desired behavior clear, then add failures found through production monitoring.
You can author test cases directly in Experiments > Test suites, promote a conversation from its detail page, promote the saved conversations in a collection into a suite draft, or synchronize a source-controlled YAML or JSON suite with the Agent Observability SDK.
Edit a draft while the dataset is changing. Publish the suite version when you want a stable input set for comparable experiments. When you need to change a published suite, create a new draft and publish a new version. Keep the suite ID and test case IDs stable so reports can distinguish dataset changes from candidate changes.
Run and compare experiments
Your code or benchmark process executes the test suite. Use the Agent Observability SDK to connect that process to the experiment lifecycle and reuse telemetry from your instrumented agent.
Run the same published suite version against each candidate you want to compare. Candidate metadata, for example, agent version, Git commit, model, or prompt version, makes each run identifiable. The comparison view aligns shared test case IDs and shows changes in scores, pass rate, tokens, cost, and artifacts.
Grafana’s o11y-bench project, built on Harbor, demonstrates this workflow for observability and SRE agents. Its task definitions form a versioned test suite, and each model and reasoning configuration publishes its trials, verifier scores, usage, and artifacts as an Agent Observability experiment.
Authenticate experiment operations
Experiment ingest and Grafana control plane operations use separate credentials:
In the UI, users need the same roles: Agento11y Data Reader or Agento11y Reader to view experiments, and Agento11y Admin to manage experiments and test suites.
