Evaluation API
The Grafana Agent Observability evaluation control plane API manages evaluators, rules, guard rules, saved conversations, test suites, experiments, and judge provider discovery. Offline experiment runs can be recorded by SDKs, CI jobs, and benchmark processes.
Manage evaluators
Use evaluator endpoints to create and list scoring definitions.
Manage rules
Use rule endpoints to attach evaluators to matching generation or conversation traffic.
Discover judge providers
Use judge provider endpoints to discover configured judge providers and models.
Manage templates
Use template endpoints to manage reusable evaluator templates and versions.
Manage guards
Use guard endpoints to evaluate requests synchronously and manage guard rules.
Hook evaluation request
{
"phase": "preflight",
"context": {
"agent_name": "my-agent",
"model": { "provider": "openai", "name": "gpt-4o" },
"tags": { "env": "production" }
},
"input": {
"messages": [
{ "role": "user", "parts": [{ "kind": "text", "text": "Hello" }] }
],
"system_prompt": "You are a helpful assistant.",
"tools": []
}
}Hook evaluation response
{
"action": "allow",
"transformed_input": null,
"evaluations": []
}The action field is allow or deny. When any guard rule applied redactions, transformed_input contains the sanitized copy of the input. Each entry in evaluations reports a per-rule outcome with rule_id, evaluator_id, passed, and latency_ms.
Guard rule configuration
{
"rule_id": "block_tools",
"enabled": true,
"phase": "preflight",
"priority": 5,
"selector": "all",
"match": {},
"evaluator_ids": [],
"action_on_fail": "deny",
"tool_filter": { "blocked_names": ["shell_exec", "danger_*"] },
"redact": {
"patterns": [
{ "id": "ssn", "regex": "\\b\\d{3}-\\d{2}-\\d{4}\\b" }
]
}
}Each guard rule must include at least one of these fields: evaluator_ids, redact.patterns, or tool_filter.blocked_names. Redaction matches are always replaced with the fixed token [REDACTED]; free-form replacement text is not accepted.
Export scores
Use score ingest to export externally computed scores from SDKs, CI jobs, and benchmark processes.
Scores are idempotent by score_id, which must be at most 128 characters. Resubmitting the same score ID returns a duplicate item result instead of creating another score. A score must include either generation_id or trial_id. Scores with experiment_id are rejected if the experiment doesn’t exist for the authenticated tenant.
Validation failures remain descriptive. Internal lookup or storage failures use
internal server error; error_id=<id> in the existing result schema, with one
shared ID for internal failures in the batch.
Manage saved conversations
Use saved conversation endpoints to bookmark telemetry conversations or create manual conversations for review and experiments.
Manage collections
Use collection endpoints to group saved conversations for review or promotion into test cases.
Manage test suites
Use test suite endpoints to manage versioned datasets for repeatable offline experiments. Add or edit cases in a draft, then publish the version to make it a stable input for experiment runs.
Published versions are immutable. Create a new draft version when you need to change the suite. The Agent Observability SDK exposes these operations through TestSuitesClient and can synchronize local YAML or JSON suites with stored suite versions.
Read experiment results
Use the Agent Observability SDK to read the scores and aggregated report for an offline experiment.
These reads use the same tenant API credentials as experiment run ingest. In Python, use experiment.report() to retrieve the aggregated report and experiment.client.list_scores() to retrieve scores.
Ingest experiment runs
Use experiment run ingest endpoints from SDKs, CI jobs, and benchmark processes that execute test cases. Prefer the Agent Observability SDK, which manages these calls and finalizes experiments automatically. These endpoints use tenant API credentials.
When a process needs to set machine ownership for experiment runs, trials, or artifacts, use the X-Agento11y-Ingest-Actor request header. Some routes still accept the deprecated X-Sigil-Ingest-Actor spelling for compatibility, but new integrations should use the X-Agento11y-* header name.
After you finalize an experiment, Agent Observability builds its report in the background.
Set planned_trial_count when you create the run if the total number of attempts is known. Agent Observability uses this value to show progress for an active experiment.
Use trial evaluation endpoints when you want Agent Observability to run an existing evaluator against a recorded trial conversation on demand. The trigger request accepts evaluator_id and an optional evaluator_version. The response includes an evaluation_id, status, attempts, and any error details. Poll the status endpoint until the evaluation reaches a terminal state.
Configure evaluator types
LLM judge
{
"kind": "llm_judge",
"config": {
"provider": "openai",
"model": "gpt-4o-mini",
"system_prompt": "You are a quality evaluator.",
"user_prompt": "Rate this response:\n{{agent_response}}",
"max_tokens": 100,
"temperature": 0.0,
"timeout_ms": 30000
}
}JSON schema
{
"kind": "json_schema",
"config": {
"schema": {
"type": "object",
"required": ["answer"],
"properties": {
"answer": { "type": "string" }
}
},
"target": "response"
}
}The optional target field sets the text to evaluate: response (default), input, or system_prompt.
Regex
{
"kind": "regex",
"config": {
"pattern": "^\\d+$",
"reject": false,
"target": "response"
}
}Heuristic
{
"kind": "heuristic",
"config": {
"version": "v2",
"target": "response",
"root": {
"kind": "group",
"operator": "and",
"rules": [
{ "kind": "rule", "type": "not_empty" },
{ "kind": "rule", "type": "min_length", "value": 10 }
]
}
}
}Choose rule selectors
Choose rule execution mode
Evaluation rules can run attached evaluators in parallel or sequential mode. In parallel mode, every evaluator runs for each matched item. In sequential mode, evaluators run in order, and each evaluator must pass before the next evaluator is queued.
Use sequential mode to put a low-cost gate in front of a more expensive evaluator. The first evaluator in a sequential rule must define a pass condition, such as pass_threshold, pass_value, or pass_match, so Agent Observability knows whether to continue. Gate outcomes are tracked separately from final quality pass-rate reporting.
Integrate rule alerts
Rules can include an alert_rule_uids field that stores the UIDs of Grafana alert rules created for the evaluation. In Agent Observability, add alerts from the evaluation creation or configuration flow by setting a pass-rate threshold and contact point. Alerts are created as non-provisioned, so you can edit them in the Grafana Alerting UI afterward.
