Documentation Grafana Cloud AI and machine learning Grafana Agent Observability Experiment concepts

Understand experiment concepts

Experiments in Grafana Agent Observability help you benchmark an agent, compare models or prompts, and enforce evaluation gates during development and delivery.

Understand shared evaluation concepts

Online and offline evaluations use several common concepts:

• A conversation contains the traces and metrics for one agent session, including user turns, agent responses, and tool calls.
• A collection is a selected group of conversations.
• An evaluator checks an LLM generation. It can use an LLM judge with a prompt and rubric or a deterministic check, for example, a regular expression that returns a pass or fail verdict.

Understand offline experiment concepts

Offline experiments add versioned test inputs and repeatable executions:

• A test case defines one test scenario with a stable ID, versioned input, and an optional expected output or verification rubric.
• A test suite is a versioned collection of test cases and associated metadata, for example, tags.
• A test trial records one execution of a test case, including its test-case snapshot, attempt number, optional conversation or trace ID, scores, and artifacts.
• An experiment executes a test suite and combines its test trials into a result you can compare with other runs.

Use experiments in your development lifecycle

Run the same suite against different agent, model, or prompt versions to compare results over time. Use the recorded trials and artifacts to investigate regressions, and use the final experiment result as an automated delivery gate.