Documentation Grafana Cloud AI and machine learning Grafana AI Observability Configure Tune evaluation settings

Configure online evaluation

Online evaluation continuously scores live generation traffic. You configure evaluators that define how to score, and rules that define which generations to evaluate.

Enable the eval worker

Set these environment variables (or Helm values) to enable evaluation:

Push evaluation metrics

AI Observability can push per-tenant evaluation metrics to a Prometheus-compatible remote-write endpoint. This lets you query evaluation pass rates and score distributions in Grafana alongside your other metrics.

Configure judge providers

The querier discovers judge providers and judge-model dropdown options from environment variables. The eval worker enforces the same provider and allowlist config when evaluations execute.

Allowlist values are comma-separated model IDs. Use the same IDs returned by the judge model API or shown in the UI dropdown.

Empty or unset allowlist variables mean all models returned by that provider remain eligible.

In split deployments, keep allowlist env vars aligned across querier and eval-worker so the UI matches execution behavior.

Bedrock supports both Anthropic models and non-Anthropic models (for example, Mistral). Anthropic models use the Anthropic Messages API, while non-Anthropic models use the model-agnostic Bedrock Converse API. AI Observability routes model IDs automatically based on the anthropic. prefix.

Anthropic on Vertex dynamically lists available Claude models from the Vertex AI model catalog. You don’t need to specify model IDs manually.

Create evaluators

Use the AI Observability plugin UI or the evaluation API to create evaluators. Four evaluator types are available:

LLM judge

Uses an LLM to score generations based on criteria you define in a prompt template.

Key settings:

• provider and model — the LLM to use for judging.
• system_prompt and user_prompt — prompt templates with variables.
• max_tokens, temperature, timeout_ms — generation controls.

JSON schema

Validates that the assistant response matches a JSON schema. Returns true or false.

Regex

Checks the assistant response against one or more regex patterns. Use reject: true to invert the match.

Heuristic

Applies a rule tree with AND/OR logic. Supported checks: not_empty, contains, not_contains, min_length, max_length.

Evaluation target

JSON schema, regex, and heuristic evaluators evaluate the assistant response by default. You can change the evaluation target to evaluate other fields instead:

Set the target in the Evaluate against dropdown when you create a non-LLM-judge evaluator, or set the target field in the evaluator config JSON.

This is useful for lightweight detection of injected content in generation input without requiring an LLM judge.

Pass verdict

Bool-type evaluators (heuristic, regex, JSON schema, and LLM judge bool outputs) record a pass/fail verdict only when you explicitly configure a pass_value on the output key. When pass_value is omitted, the score is recorded but no verdict is determined.

Template variables

LLM judge prompts support these template variables:

Create rules

Rules connect evaluators to generation traffic. Each rule has:

• Selector — which generations to evaluate:
  • user_visible_turn — assistant text responses without tool calls.
  • all_assistant_generations — any assistant output.
  • tool_call_steps — generations with tool calls.
  • errored_generations — generations with errors.
• Match filters — additional criteria to narrow the selection.
• Sampling rate — percentage of matching generations to evaluate.
• Evaluator — the evaluator to run.

Next steps

• Set up evaluation end-to-end
• Configure deployment options