Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Observe and actbreadcrumb arrow Agent Observabilitybreadcrumb arrow API and SDKsbreadcrumb arrow Experiments
Grafana Cloud

Experiments API

Manage saved conversations, collections, and test suites, submit experiment runs, and read results.

Before you begin

  • Enable Agent Observability on your Grafana Cloud stack. Refer to Set up Agent Observability in Grafana Cloud.
  • For configuration changes, use a Grafana identity with permission to manage the relevant resource. Refer to Security and access controls.
  • For experiment runs or exported scores, create a Grafana Cloud Access Policy Token with the sigil:write scope.

For credential requirements, refer to Choose credentials.

Manage saved conversations

Use saved conversation endpoints to bookmark telemetry conversations or create manual conversations for review and experiments.

MethodPathDescription
GET/api/v1/eval/saved-conversationsList saved conversations.
POST/api/v1/eval/saved-conversationsSave a telemetry conversation.
POST/api/v1/eval/saved-conversations:manualCreate a manual conversation.
GET/api/v1/eval/saved-conversations/{id}Get a saved conversation.
DELETE/api/v1/eval/saved-conversations/{id}Delete a saved conversation.
GET/api/v1/eval/saved-conversations/{id}/collectionsList collections containing a saved conversation.

Manage collections

Use collection endpoints to group saved conversations for review or promotion into test cases.

MethodPathDescription
GET/api/v1/eval/collectionsList collections.
POST/api/v1/eval/collectionsCreate a collection.
GET/api/v1/eval/collections/{id}Get a collection.
PATCH/api/v1/eval/collections/{id}Update a collection.
DELETE/api/v1/eval/collections/{id}Delete a collection.
GET/api/v1/eval/collections/{id}/membersList collection members.
POST/api/v1/eval/collections/{id}/membersAdd collection members.
DELETE/api/v1/eval/collections/{id}/members/{saved_id}Remove a collection member.

Manage test suites

Use test suite endpoints to manage versioned datasets for repeatable offline experiments. Add or edit cases in a draft, then publish the version to make it a stable input for experiment runs.

MethodPathDescription
GET/api/v1/eval/test-suitesList test suites.
POST/api/v1/eval/test-suitesCreate a test suite.
GET/api/v1/eval/test-suites/{suite_id}Get a test suite and its versions.
PATCH/api/v1/eval/test-suites/{suite_id}Update test suite metadata.
POST/api/v1/eval/test-suites/{suite_id}/versionsCreate a draft version.
GET/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-casesList test cases in a version.
POST/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-casesCreate or update a test case.
GET/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-cases/{test_case_id}Get a test case.
PATCH/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-cases/{test_case_id}Update a test case.
DELETE/api/v1/eval/test-suites/{suite_id}/versions/{version}/test-cases/{test_case_id}Delete a test case.
POST/api/v1/eval/test-suites/{suite_id}/versions/{version}:publishPublish a suite version.

Published versions are immutable. Create a new draft version when you need to change the suite. The Agent Observability SDK exposes these operations through TestSuitesClient and can synchronize local YAML or JSON suites with stored suite versions.

Ingest experiment runs

Use experiment run ingest endpoints from SDKs, CI jobs, and benchmark processes that execute test cases. Prefer the Agent Observability SDK, which manages these calls and finalizes experiments automatically. These endpoints use tenant API credentials.

When a process needs to set machine ownership for experiment runs, trials, or artifacts, use the X-Agento11y-Ingest-Actor request header. Some routes still accept the deprecated X-Sigil-Ingest-Actor spelling for compatibility, but new integrations should use the X-Agento11y-* header name.

MethodPathDescription
POST/api/v1/experiment-runs:upsertCreate or resume an experiment.
POST/api/v1/experiment-runs/{experiment_id}:finalizeFinalize an experiment.
POST/api/v1/experiment-runs/{experiment_id}/trialsCreate or update a test-case trial.
PATCH/api/v1/experiment-runs/{experiment_id}/trials/{trial_id}Update a test-case trial.
POST/api/v1/experiment-runs/{experiment_id}/trials/{trial_id}:evaluateQueue evaluator scoring for a trial.
GET/api/v1/experiment-runs/{experiment_id}/trials/{trial_id}/evaluations/{evaluation_id}Get trial evaluation status.
POST/api/v1/experiment-runs/{experiment_id}/trials/{trial_id}/artifacts:uploadUpload a trial artifact.

After you finalize an experiment, Agent Observability builds its report in the background.

Set planned_trial_count when you create the run if the total number of attempts is known. Agent Observability uses this value to show progress for an active experiment.

Use trial evaluation endpoints when you want Agent Observability to run an existing evaluator against a recorded trial conversation on demand. The trigger request accepts evaluator_id and an optional evaluator_version. The response includes an evaluation_id, status, attempts, and any error details. Poll the status endpoint until the evaluation reaches a terminal state.

You can also set report_role to primary_verdict or diagnostic in the trigger request. The role applies to this trial evaluation’s score, not to the evaluator definition. The response includes the role when you supply it. For evaluator-ID-only requests, if you omit report_role, the score has no report role. For headline selection and pass-rate behavior, refer to Interpret trial verdicts.

For experiment traffic isolation and the immutable online_evaluations_enabled setting, refer to Keep experiment traffic separate.

Read experiment results

Use the Agent Observability SDK to read the scores and aggregated report for an offline experiment.

MethodPathDescription
GET/api/v1/eval/experiments/{id}/scoresList experiment scores.
GET/api/v1/eval/experiments/{id}/reportGet the aggregated experiment report.

These reads use the same tenant API credentials as experiment run ingest. In Python, use experiment.report() to retrieve the aggregated report and experiment.client.list_scores() to retrieve scores.

Next steps