--- title: "Evaluation API | Grafana Cloud documentation" description: "Configure evaluators, rules, and templates, discover judge providers, and export scores." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Evaluation API Configure evaluators, rules, and templates, discover judge providers, and export scores. ## Before you begin - Enable Agent Observability on your Grafana Cloud stack. Refer to [Set up Agent Observability in Grafana Cloud](/docs/grafana-cloud/observe-and-act/agent-observability/get-started/grafana-cloud). - For configuration changes, use a Grafana identity with permission to manage the relevant resource. Refer to [Security and access controls](/docs/grafana-cloud/observe-and-act/agent-observability/privacy-and-access/security). - For experiment runs or exported scores, create a Grafana Cloud Access Policy Token with the `sigil:write` scope. For credential requirements, refer to [Choose credentials](/docs/grafana-cloud/observe-and-act/agent-observability/reference/#choose-credentials). ## Manage evaluators Use evaluator endpoints to create and list scoring definitions. Expand table | Method | Path | Description | |----------|------------------------------------------------|------------------------------| | `GET` | `/api/v1/eval/evaluators` | List all evaluators. | | `POST` | `/api/v1/eval/evaluators` | Create an evaluator. | | `GET` | `/api/v1/eval/evaluators/{id}` | Get an evaluator by ID. | | `DELETE` | `/api/v1/eval/evaluators/{id}` | Delete an evaluator. | | `GET` | `/api/v1/eval/predefined/evaluators` | List predefined evaluators. | | `POST` | `/api/v1/eval/predefined/evaluators/{id}:fork` | Fork a predefined evaluator. | To update an existing evaluator, send `POST /api/v1/eval/evaluators` again with the same `evaluator_id` and a new version (`YYYY-MM-DD` or `YYYY-MM-DD.N`). That request stores another snapshot. ## Manage rules Use rule endpoints to attach evaluators to matching generation or conversation traffic. Expand table | Method | Path | Description | |----------|-----------------------------------------------|----------------------------| | `GET` | `/api/v1/eval/rules` | List all evaluation rules. | | `POST` | `/api/v1/eval/rules` | Create a rule. | | `GET` | `/api/v1/eval/rules/{id}` | Get a rule by ID. | | `PATCH` | `/api/v1/eval/rules/{id}` | Update a rule. | | `DELETE` | `/api/v1/eval/rules/{id}` | Delete a rule. | | `POST` | `/api/v1/eval/rules:preview` | Preview rule matches. | | `GET` | `/api/v1/eval/rules/{id}/actions` | List rule actions. | | `POST` | `/api/v1/eval/rules/{id}/actions` | Create a rule action. | | `GET` | `/api/v1/eval/rules/{id}/actions/{action_id}` | Get a rule action. | | `PATCH` | `/api/v1/eval/rules/{id}/actions/{action_id}` | Update a rule action. | | `DELETE` | `/api/v1/eval/rules/{id}/actions/{action_id}` | Delete a rule action. | ## Discover judge providers Use judge provider endpoints to discover configured judge providers and models. Expand table | Method | Path | Description | |--------|--------------------------------|----------------------------------| | `GET` | `/api/v1/eval/judge/providers` | List discovered judge providers. | | `GET` | `/api/v1/eval/judge/models` | List models for a provider. | ## Manage templates Use template endpoints to manage reusable evaluator templates and versions. Expand table | Method | Path | Description | |----------|--------------------------------------------------|-------------------------------| | `GET` | `/api/v1/eval/templates` | List evaluator templates. | | `POST` | `/api/v1/eval/templates` | Create an evaluator template. | | `GET` | `/api/v1/eval/templates/{id}` | Get a template by ID. | | `DELETE` | `/api/v1/eval/templates/{id}` | Delete a template. | | `POST` | `/api/v1/eval/templates/{id}:fork` | Fork a template. | | `GET` | `/api/v1/eval/templates/{id}/versions` | List template versions. | | `POST` | `/api/v1/eval/templates/{id}/versions` | Publish a template version. | | `GET` | `/api/v1/eval/templates/{id}/versions/{version}` | Get a template version. | ## Configure evaluator types Choose an evaluator type for the content and scoring criteria you want to check. ### LLM judge Use an LLM judge to score responses against prompt-based criteria: JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "kind": "llm_judge", "config": { "provider": "openai", "model": "gpt-4o-mini", "system_prompt": "You are a quality evaluator.", "user_prompt": "Rate this response:\n{{agent_response}}", "timeout_ms": 120000 } } ``` You don’t configure the LLM judge’s output-token limit. Agent Observability sets it to `16384` tokens or the model’s supported output limit, whichever is lower. A `max_tokens` value in an existing evaluator configuration doesn’t change this limit. The optional `temperature` setting uses the model provider’s default when omitted. The UI doesn’t display **Max tokens** or **Temperature** and removes both settings when you save an evaluator or fork a template. Set `temperature` only if the selected model supports it; an explicit `0` isn’t the same as leaving it unset. The default `timeout_ms` is `120000` (120 seconds) per judge attempt. Increase it if your chosen reasoning model needs more time. For prompt and model settings, refer to [Configure evaluators and evaluation rules](/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/configure-evaluation). Set `reasoning_effort` only when the selected provider and model support it. For available values and omission behavior, refer to [LLM judge settings](/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/configure-evaluation/#llm-judge). ### JSON schema JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "kind": "json_schema", "config": { "schema": { "type": "object", "required": ["answer"], "properties": { "answer": { "type": "string" } } }, "target": "response" } } ``` The optional `target` field sets the text to evaluate: `response` (default), `latest_user_message`, `input`, or `system_prompt`. ### Regex JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "kind": "regex", "config": { "pattern": "^\\d+$", "reject": false, "target": "response" } } ``` ### Heuristic JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "kind": "heuristic", "config": { "version": "v2", "target": "response", "root": { "kind": "group", "operator": "and", "rules": [ { "kind": "rule", "type": "not_empty" }, { "kind": "rule", "type": "min_length", "value": 10 } ] } } } ``` ## Choose rule selectors Expand table | Selector | Description | |-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------| | `user_visible_turn` | Final user-visible responses. Use explicit `turn.*` prompt variables to include the complete tool loop. | | `turn` | Legacy compatibility value for complete-turn rules. It fires at the same event as `user_visible_turn`; do not use it for new rules. | | `all_assistant_generations` | Any agent output. | | `tool_call_steps` | Generations containing tool calls. | | `errored_generations` | Generations with `call_error`. | | `conversation` | Full conversations after the configured idle window. | ## Choose rule execution mode Evaluation rules can run attached evaluators in `parallel` or `sequential` mode. In `parallel` mode, every evaluator runs for each matched item. In `sequential` mode, evaluators run in order, and each evaluator must pass before the next evaluator is queued. Use `sequential` mode to put a low-cost gate in front of a more expensive evaluator. The first evaluator in a sequential rule must define a pass condition, such as `pass_threshold`, `pass_value`, or `pass_match`, so Agent Observability knows whether to continue. Gate outcomes are tracked separately from final quality pass-rate reporting. ## Integrate rule alerts Rules can include an `alert_rule_uids` field that stores the UIDs of Grafana alert rules created for the evaluation. In Agent Observability, add alerts from the evaluation creation or configuration flow by setting a pass-rate threshold and contact point. Alerts are created as non-provisioned, so you can edit them in the Grafana Alerting UI afterward. ## Export scores Use score ingest to export externally computed scores from SDKs, CI jobs, and benchmark processes. Expand table | Transport | Endpoint | |-----------|------------------------------| | HTTP | `POST /api/v1/scores:export` | Scores are idempotent by `score_id`, which must be at most 128 characters. Resubmitting the same score ID returns a duplicate item result instead of creating another score. A score must include either `generation_id` or `trial_id`. Scores with `experiment_id` are rejected if the experiment doesn’t exist for the authenticated tenant. Validation failures remain descriptive. Internal lookup or storage failures use `internal server error; error_id=` in the existing result schema, with one shared ID for internal failures in the batch. ## Next steps - [Set up and monitor online evaluation](/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/evaluation) - [Explore the experiments API](/docs/grafana-cloud/observe-and-act/agent-observability/reference/experiments)