Evaluate and improve

Use online evaluation to score production traffic and experiments to compare changes before release. You can use production conversations as test cases for those experiments.

Choose your workflow

Start with the task you want to complete:

Set up and monitor online evaluation to create an evaluation and review its scores.
Configure evaluators and evaluation rules to tune evaluator settings, traffic selection, and sampling.
Manage test suites and experiments to build and version test suites and compare results.
Run an experiment with the SDK to execute test cases with the SDK.

Connect production feedback to experiments

Use the existing workflows to investigate a failure and check a change:

Inspect and save conversations, or route evaluation results into collections.
Build and version a test suite from those examples.
Run an experiment with the SDK and compare results.
Monitor online evaluation results as production traffic arrives.

Set up and monitor online evaluation
Create evaluators to continuously score agent quality on live production traffic.
Configure evaluators and evaluation rules
Set up LLM judges, evaluator types, rules, and providers for continuous quality scoring in Agent Observability.
Manage test suites and experiments
Learn how experiments connect production conversations, test suites, agent evaluations, and delivery workflows in Grafana Agent Observability.
Run an experiment with the SDK
Run test cases against an instrumented agent and optionally associate the results with a versioned test suite.