---
title: "Run an experiment | Grafana Cloud documentation"
description: "Run test cases against an instrumented agent and optionally associate the results with a versioned test suite."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Run an experiment

> Note
> 
> Grafana Agent Observability experiments is currently in [public preview](/docs/release-life-cycle/). Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.

Use the Agent Observability SDK to run test cases against an instrumented agent. You can start with cases defined directly in your test code and optionally associate the experiment with a stored test suite. This guide focuses on implementing the test loop. For the data model and lifecycle, refer to [Experiments](/docs/grafana-cloud/observe-and-act/agent-observability/guides/experiments).

## Before you begin

- [Instrument your agent](/docs/grafana-cloud/observe-and-act/agent-observability/guides/instrument-agents) with the Agent Observability SDK.
- To associate the experiment with a stored test suite, configure a test suite and publish it under **Experiments > Test suites** or with the SDK.
- Configure the ingest credentials described in [Understand experiments](/docs/grafana-cloud/observe-and-act/agent-observability/guides/experiments/#authenticate-experiment-operations).
- If you use a stored test suite, also configure the control plane credentials.

Experiments reuse the generations, conversations, traces, token usage, and cost emitted by normal instrumentation. You don’t need a separate instrumentation path.

## Choose an SDK

Experiment APIs are available in the Python and Go [Agent Observability SDKs](https://github.com/grafana/agento11y):

- Python uses the `agento11y.experiments` package. Use `experiment`, `experiment.trial`, `bind_generation`, and `final_score` to manage the run. Use `experiment_from_suite` when running a stored suite.
- Go uses the `github.com/grafana/agento11y/go/agento11y/experiments` package. Use `WithExperiment`, `WithTrial`, `BindGeneration`, and `FinalScore`, or `WithExperimentFromSuite` for a stored suite.

Both SDKs can pull a stored suite, record trials and scores, associate existing agent telemetry, upload artifacts, and finalize the experiment.

## Run an experiment with Python

Start an experiment with test cases defined by your test code. Wrap each existing agent invocation in a trial, then bind the generation emitted by the instrumented agent:

Python ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```python
from agento11y import experiments

cases = [
    {
        "id": "reset-api-token",
        "input": "How do I replace an expired API token?",
        "expected": "Revoke the expired token and create a new one.",
    },
]

with experiments.experiment(
    name="PR 412 support regression",
    candidate={
        "agent_name": "support-agent",
        "git_sha": "abc123",
        "model_name": "gpt-5",
    },
    planned_trial_count=len(cases),
) as experiment:
    for case in cases:
        with experiment.trial(case["id"]) as trial:
            result = run_agent(case["input"])

            trial.bind_generation(
                result.generation_id,
                conversation_id=result.conversation_id,
            )

            passed = evaluate(result.output, case["expected"])
            trial.final_score(
                passed,
                explanation="Checked the expected support behavior.",
            )
```

Replace `run_agent` with your existing instrumented agent invocation. Replace `evaluate` with a deterministic check or LLM judge appropriate for the test case.

The outer context creates and finalizes the experiment. The inner context creates and completes each trial. If the test loop raises an exception, the SDK finalizes the experiment as failed.

## Add artifacts

Attach supporting files to a trial, such as screenshots, additional transcript JSON, or grader output. Use `path` for any file or `data` for a JSON-serializable object:

Python ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```python
trial.artifact("screenshot", path="artifacts/result.png")
trial.artifact("transcript", data=transcript)
```

The artifacts appear with the trial in the experiment report.

## Associate an experiment with a test suite

A stored test suite gives test cases stable identities and versioned inputs. Upload and publish the suite, then create the experiment from the exact version returned by the SDK:

Python ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```python
from agento11y import experiments

suite = experiments.TestSuite.from_yaml("evals/support-regression.yaml")

suites = experiments.TestSuitesClient()
pushed = suites.push_suite(
    suite,
    publish=True,
    changelog="Add API token regression coverage",
)

with experiments.experiment_from_suite(
    pushed.suite_id,
    version=pushed.suite_version,
    name="PR 412 support regression",
    candidate={
        "agent_name": "support-agent",
        "git_sha": "abc123",
        "model_name": "gpt-5",
    },
    planned_trial_count=len(pushed.suite.cases),
) as experiment:
    for case in experiment.suite.cases:
        with experiment.trial(case) as trial:
            result = run_agent(case.input)
            trial.bind_generation(
                result.generation_id,
                conversation_id=result.conversation_id,
            )
            trial.final_score(evaluate(result.output, case.expected))
```

`push_suite` creates or updates a draft, uploads the local test cases, and publishes an immutable suite version. `experiment_from_suite` pulls that version and records its suite ID and version on the experiment and trials.

Associating an experiment with a stored suite lets you:

- View the experiments run against that suite over time.
- Preserve the exact test case input and expected behavior used by each run.
- Inspect each trial’s agent output, conversation, trace, scores, usage, and artifacts alongside its test case.
- Compare baseline and candidate runs by stable test case ID.
- Reuse the same published suite across local development and CI.

Publishing a suite requires an `AGENTO11Y_SERVICE_ACCOUNT_TOKEN` with the Agento11y Admin role. Refer to [Authenticate experiment operations](/docs/grafana-cloud/observe-and-act/agent-observability/guides/experiments/#authenticate-experiment-operations) for the required control plane and ingest credentials.

## Configure experiment progress

Set `planned_trial_count` when you know how many attempts the test process will run. Agent Observability uses it to display progress while the experiment is running.

Count attempts rather than unique test cases. For example, a suite with 20 cases and three attempts per case has 60 planned trials:

Python ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```python
with experiments.experiment_from_suite(
    "support-regression",
    version="v3",
    name="Support regression",
    planned_trial_count=60,
) as experiment:
    for case in experiment.suite.cases:
        for attempt in range(1, 4):
            with experiment.trial(case, attempt=attempt) as trial:
                result = run_agent(case.input)
                trial.bind_generation(
                    result.generation_id,
                    conversation_id=result.conversation_id,
                )
                trial.final_score(evaluate(result.output, case.expected))
```

Omit `planned_trial_count` when the number isn’t known before the experiment starts. It affects the displayed progress, not which trials the SDK runs.

## Review the result

Open **Experiments** in Agent Observability to inspect the report. Each test case links to its trials, scores, conversation, trace, usage, and artifacts.

Run another experiment with the same suite version to compare a candidate with the baseline. Because both experiments use the same test case IDs, the report can align changes in quality, cost, tokens, and latency.

Online evaluation can automatically find production failures to add to the next suite version. Refer to [Set up online evaluation](/docs/grafana-cloud/observe-and-act/agent-observability/guides/evaluation) to score live traffic and route failed conversations into a collection.

## Next steps

- [Understand experiments](/docs/grafana-cloud/observe-and-act/agent-observability/guides/experiments)
- [Set up online evaluation](/docs/grafana-cloud/observe-and-act/agent-observability/guides/evaluation)
- [Explore the evaluation API](/docs/grafana-cloud/observe-and-act/agent-observability/reference/evaluation-api)