---
title: "Set up human review | Grafana Cloud documentation"
description: "Let people answer questions about a sample of conversations and compare their answers with judge scores."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Set up human review

Human review lets people answer a few questions about a sample of conversations. Their answers appear next to judge scores, so you can check whether your evaluators agree with people. You can export every answer as CSV.

Human review is in public preview.

## Before you begin

- To create questions and review queues, you need the Agent Observability Admin role.
- To answer questions, you need the Agent Observability Feedback Writer or Admin role.
- Every role with data access can view queue progress and download answers.

For more information about roles, refer to [Configure RBAC](https://grafana.com/docs/grafana-cloud/observe-and-act/agent-observability/privacy-and-access/security/#configure-rbac).

## Understand questions and review queues

A question is an evaluator of kind **Human review**. Agent Observability never runs it. People answer it about the whole conversation.

A review queue is a collection of conversations plus a rubric: the questions reviewers answer for each conversation. A rubric holds up to 50 questions. Each question gets one answer per conversation in a queue, so two reviewers can’t answer the same conversation. You can use human review questions only in review queues, not in online evaluations, guards, experiments, or test suites.

Each question has one of three answer types:

- **Number**: reviewers pick a value on an integer scale. Scales of up to 11 values show buttons; larger scales show a number field.
- **Boolean**: reviewers pick **Yes** or **No**.
- **String**: reviewers pick one value from the allowed values you define.

Answers are values, not free text. Reviewers can add one optional note each time they submit.

A question can have a pass condition, the same as other evaluators. Without one, Agent Observability keeps the answers but doesn’t count them toward the pass rate.

## Create a question

Create one question for each thing you want reviewers to judge.

1. In **Evaluations**, click **More** and select **View evaluators**.
2. Click **Create Evaluator** and select **Create manually**.
3. In **Evaluator ID**, enter a unique ID, for example, `support.helpfulness`.
4. In **Evaluator kind**, select **Human review**.
5. In **Question**, enter what reviewers answer about the conversation.
6. Optional: In **Instructions**, add guidance that reviewers see next to the question.
7. Under **Output type**, select **Number**, **Boolean**, or **String**, and configure the pass condition. For details, refer to [Configure outputs and pass verdicts](https://grafana.com/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/configure-evaluation/#configure-outputs-and-pass-verdicts).
8. Click **Create**.

## Create a review queue

A review queue starts from a collection. Add conversations to a collection when you [save conversations](https://grafana.com/docs/grafana-cloud/observe-and-act/agent-observability/observe-and-troubleshoot/conversations/#save-conversations), or [route them with evaluation actions](https://grafana.com/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/evaluation/#route-conversations-with-actions).

1. Go to **Evaluations** and click **Reviews**.
2. Click **New review queue**.
3. In **Collection**, pick or create the collection to review.
4. Under **Rubric**, pick the questions reviewers answer.
5. Click **Start reviewing this collection**.

To change the questions later, open the queue, click the menu next to **CSV**, and select **Edit rubric**.

To stop reviewing a collection, select **Stop reviewing** in the same menu. The queue disappears from **Reviews**. Agent Observability keeps its conversations and every answer already given.

## Review conversations

Reviewers answer every question in the rubric for one conversation at a time.

1. Go to **Evaluations** and click **Reviews**.
2. Click a queue to open it.
3. Click **Start reviewing**, or **Continue reviewing** if the queue already has reviewed conversations. The next conversation to review opens with the review panel.
4. Answer every question. You can submit once every question has an answer.
5. Optional: Add a note that explains your answers.
6. Click **Submit and next**, or press `Cmd+Enter` (Mac) or `Ctrl+Enter` (Linux/Windows).

You can also open the review panel from any conversation in a queue: click **Review** in the conversation header.

### Remove a conversation without generations

A conversation that a guard blocked before the model call has no generations, so it can’t be reviewed. The review panel shows **Remove from queue** for it. Removing the conversation takes it out of the queue’s collection. Agent Observability keeps the conversation.

You need the Agent Observability Admin role to remove a conversation.

## Monitor results and export answers

The queue page shows the progress of the queue, the pass rate of each question, and the result of each reviewed conversation. Reviewer answers also appear next to judge scores on the conversation page. A review covers the whole conversation, so Agent Observability attributes it to the agent that gives the final answer.

To export the answers, open the queue and click **CSV**. The file has one row per answer, with these columns: `conversation_id`, `generation_id`, `agent_name`, `agent_version`, `evaluator_id`, `evaluator_version`, `score_key`, `value`, `passed`, `reviewer`, `note`, and `created_at`.

## Edit or delete a question

Editing a question creates a new version. Queues that use it ask the new version again, and the CSV keeps the answers to earlier versions.

You can’t delete a question while a queue uses it. The question’s page lists the queues that use it.

## Next steps

- [Configure evaluators and evaluation rules](https://grafana.com/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/configure-evaluation)
- [Set up and monitor online evaluation](https://grafana.com/docs/grafana-cloud/observe-and-act/agent-observability/evaluate-and-improve/evaluation)