---
title: "Monitor agents with analytics | Grafana Cloud documentation"
description: "Monitor agent activity, performance, cost, and quality using the Agent Observability analytics dashboards."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Monitor agents with analytics

Agent Observability includes built-in analytics that visualize agent activity, performance, cost, and quality. The analytics views use Prometheus metrics and Agent Observability data to surface actionable insights.

## Before you begin

- Configure your agent to export generations and OpenTelemetry metrics to Grafana Cloud. For instructions, refer to [Get started with Grafana Cloud](/docs/grafana-cloud/observe-and-act/agent-observability/get-started/grafana-cloud).
- Select a time range that includes your agent traffic.

## Access dashboards

Select **Analytics** in Agent Observability. Under **Explore by area**, expand:

- **Agents**: generation counts, latency, time to first token, errors, tokens, and estimated cost by agent.
- **Usage and cost**: token usage, estimated cost, cache efficiency, and conversation activity.
- **Evaluations**: evaluation scores, pass rates, and quality trends.
- **Tools**: tool call frequency, duration, and error rates.

Use the time picker to set the query window. The refresh picker lets you set an auto-refresh interval, for example, 30s, 1m, or 5m. The selected interval is saved for the session.

## Find items that need attention

The **Needs attention** section highlights agents, conversations, tools, and evaluations that may need review. Open **Health overview settings** and select **Sudden changes** to compare with the previous time window, or **High levels** to check absolute values. Turn on **Show only entities that need attention** to hide healthy cards.

Open a card to investigate its supporting data in the relevant Agent Observability view.

## Include or exclude traffic

Use the **Provider**, **Model**, and **Agent** filters to focus on specific traffic. Select one or more values in each menu. By default, the filter includes the selected values.

Dropdown options cascade to reflect your selections:

- Selecting a provider limits **Model** options to models observed for that provider.
- Selecting a provider and model limits **Agent** options to agents that match both selections.

In coding-agent analytics views, **Provider**, **Model**, and **Agent** dropdown options are scoped to metrics from known coding-agent launchers. Repository, user, and added label filters use the same coding-agent scope. This prevents unrelated application names or unused provider and model combinations from appearing. Other analytics pages provide unscoped options across all agents.

To hide those values instead, select **Exclude selected** in that filter’s menu. Close the menu to apply your selection. Each filter has its own mode, so you can include one provider while excluding a model or agent. Clear **Exclude selected** to return to inclusion mode.

These controls also apply when you browse conversations or review evaluation results. Your selections and exclusion modes stay in the URL when you share the view.

## Break a chart down by one or more dimensions

Each chart has a **Break down** switch that splits a single line into one line per dimension. Turn it on and choose dimensions from the picker next to it.

The picker lists common dimensions such as **Agent**, **Model**, and **Provider** first, followed by every other Prometheus label present on your `gen_ai_client_*` metrics in the current time range. Both kinds work the same way, so you can compare staging against production, or one team’s spend against another’s.

Select more than one dimension to see their combinations. Grouping by **Agent** and `deployment_environment` gives you one line per agent per environment, named `checkout-agent / production`.

You can combine up to three dimensions. Each one multiplies the number of lines on the chart, so the picker stops offering more once you reach the limit. If a chart becomes hard to read, remove a dimension or narrow the filters above it.

A label appears in the picker only if your SDK or collector attaches it to the metrics. The simplest way to add one is to set resource attributes on the service that emits the telemetry:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=ai
```

Dots become underscores in Prometheus label names, so `deployment.environment` appears as `deployment_environment`.

You can also set client tags in the SDK, either with the `AGENTO11Y_TAGS` environment variable or the `tags` field in the client configuration:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
export AGENTO11Y_TAGS=team=ai,env=production
```

Each client tag becomes a label named `agento11y_tag_<KEY>`, for example, `agento11y_tag_team`. The SDK also stores the value on every generation.

For coding-agent sessions, refer to [Attribute cost to a user, repository, or branch](/docs/grafana-cloud/observe-and-act/agent-observability/get-started/instrument-coding-agents/#attribute-cost-to-a-user-repository-or-branch) to add user, repository, and branch labels automatically.

Keep the number of distinct values small, because each value adds a metric series. For the behavior of each mechanism, refer to [Client tags](/docs/grafana-cloud/observe-and-act/agent-observability/instrumentation/sdk/#client-tags).

Tags that you pass on a single generation, and generation metadata, aren’t metric labels, so they don’t appear in the picker. Filter conversations by `metadata.<KEY>` instead.

Your selection is remembered per chart area.

## Identify performance issues

Use the performance dashboard to spot problems:

- **High latency**: filter by agent or model to find slow generations. Drill into traces for specific conversations to identify bottlenecks.
- **Error spikes**: the error rate panel shows failures over time. Click through to conversations with errors to inspect the `call_error` payload.
- **Slow time to first token**: for streaming agents, the TTFT panel reveals which models or prompts have poor streaming performance.

## Inspect tool failures

On the **Tools** page (**Analytics** &gt; **Tools**), review tool executions, error rates, and latency.

The **AI analysis** bar appears directly below the summary KPI band. It evaluates visible tool metrics against the previous period, including any active provider, model, agent, and label filters, to highlight key anomalies:

- Sharp execution-volume shifts and concentration risk across top tools.
- Tools with the highest error rates and error counts.
- Outlier tools with the slowest P95 latency.

Click the **AI analysis** bar to expand or collapse findings. When collapsed, the header displays a preview of the first finding. Click **Regenerate** to refresh the analysis for current filters and data. Click any finding to open Grafana Assistant and explain it.

The bar requires Grafana Assistant; it doesn’t appear if Grafana Assistant is disabled. While queries are loading or if no tools match your filters, the bar stays mounted and waits for data before generating insights. If a metric query fails or data is unavailable, the analysis labels it as `no data` instead of reporting a measured zero, and omits unavailable previous-period comparisons.

Select **Error count** to chart the number of failed tool executions instead of the percentage shown by **Error rate**. You can also select **Error count** in the **Tools** area under **Analytics**; its table ranks tools by the highest error count.

Click **Tool**, **Executions**, **Exec %**, **Errors**, **Error rate**, or **Latency** to sort the table. Click the same header again to reverse the order. The sort menu and column headers control the same ordering.

## Optimize costs

The tokens and cost dashboard helps you find optimization opportunities:

- **Cost by model**: compare estimated cost across models and providers. Consider switching expensive calls to cheaper models where quality is acceptable.
- **Cache efficiency**: the cache read ratio shows how effectively prompt caching reduces token usage. Low cache rates may indicate prompts that change too frequently.
- **Token usage trends**: spot unexpected increases in token usage that may indicate prompt regression or unnecessary verbosity.

Select **Cache hit rate** in **Usage and cost** to compare prompt cache reuse over time. Break down by **Agent**, **Model**, or **Provider** to find differences. If you select **Token type**, the chart shows cache read and cache write token counts instead of a percentage. The conversations table also shows **Cache hit rate** so you can inspect individual sessions.

The overview cost total covers your selected time range. It uses recorded per-request costs where available and estimates earlier periods from token usage and catalog prices. **Agents** totals and **Needs attention** cost charts use the same approach. Recorded costs include your configured model prices. If no recorded costs are available, Agent Observability estimates the whole range. Token-based estimates don’t account for per-request long-context pricing tiers and can understate costs for long prompts.

Cost charts mark the transition from token-based estimates to recorded costs. A step at that marker doesn’t necessarily mean spending changed. Changing chart time buckets doesn’t change the cost total for the selected range.

When **Cost shown at list prices** appears, your selected filter or breakdown requires token-based estimates instead of recorded costs. Configured model prices and long-context pricing don’t apply to these estimates. A notice below the filters applies to every cost figure on the page; a notice on a chart applies only to that breakdown. Remove the named filter or breakdown to return to recorded costs where available. For details, refer to [Configure your model prices](/docs/grafana-cloud/observe-and-act/agent-observability/observe-and-troubleshoot/model-prices).

When you ask Grafana Assistant about costs from **Analytics** or **Agents**, it receives the page’s cost source and selected scope. Recorded costs include your configured prices. Estimates for older periods or unsupported labels use public list prices instead; check which source the answer uses before comparing figures.

## Track quality

The quality dashboard visualizes evaluation scores alongside operational metrics:

- **Score trends**: monitor if quality improves or degrades after agent version changes.
- **Score distributions**: identify if responses cluster around high or low scores.
- **Correlation**: compare quality scores with latency and cost to find the right balance.

## Use Prometheus metrics directly

If you need custom dashboards, query the Agent Observability OpenTelemetry metrics in Prometheus:

Expand table

| Metric                                                | Description                                                |
|-------------------------------------------------------|------------------------------------------------------------|
| `gen_ai_client_operation_duration_seconds`            | LLM call duration histogram.                               |
| `gen_ai_client_operation_time_to_first_chunk_seconds` | Streaming latency histogram for the Go `otel` export mode. |
| `gen_ai_client_time_to_first_token_seconds`           | Streaming latency histogram for all other SDK modes.       |
| `gen_ai_client_token_usage`                           | Token consumption histogram.                               |
| `gen_ai_client_tool_calls_per_operation_count`        | Tool calls per generation.                                 |

For custom cost queries, use these stack-scoped metrics from your configured Prometheus data source:

Expand table

| Metric                                    | Description                                                                               |
|-------------------------------------------|-------------------------------------------------------------------------------------------|
| `agento11y_generation_cost_usd_total`     | Cumulative estimated generation cost in USD, calculated when generations arrive.          |
| `agento11y_model_price_usd_per_1k_tokens` | Observed price in USD per 1,000 tokens, by provider, model, token type, and context tier. |
| `agento11y_model_price_request_usd`       | Observed fee in USD per request, by provider and model.                                   |

Multiplying aggregated token usage by current observed prices doesn’t reproduce recorded costs: historical rates, long-context tiers, and per-request fees can differ. Cost estimates aren’t provider invoices.

## Reuse dashboard panels

On panels with **More actions**, click **Open in Explore** to open the Prometheus query with the selected time range. Click **Add panel to dashboard** to copy the panel into a Grafana dashboard. Adding panels requires a configured Prometheus data source, dashboard create or write permission, and Grafana’s add-to-dashboard integration.

Some panels also offer **Create alert** or links to related conversations and agents. Alert creation requires Grafana Alerting, alert-creation permission, and Grafana’s create-alert integration.

## Ask scoped analytics questions

When Grafana Assistant is available, ask about the analytics page’s selected time range and filters, for example, cost, latency, errors, or tool usage. The page supplies this scope and its query context to Assistant.

A conversation can contain several model calls. Specify whether you want conversation totals or model-call totals so the answer uses the measure you need.

## Open conversations from exemplars

Dashboard exemplars connect metric spikes to example conversations. When exemplars are enabled, histogram panels can show exemplar markers with links that open the matching conversation.

Use exemplar links to move from a latency or error spike to the conversation that produced it. This feature requires exemplars to be enabled for your stack.

## Set up alerts

Create Grafana alerts on Agent Observability metrics to proactively catch issues:

- Alert on error rate exceeding a threshold.
- Alert on p95 latency exceeding SLO targets.
- Alert on estimated cost per day exceeding budget.
- Alert on evaluation score drops below a quality threshold.

Configure alerts in Grafana using the standard alerting workflow with the Prometheus data source.

## Schedule recurring reports

Use reports to receive recurring summaries of your agents’ cost, latency, errors, or quality. You need Grafana Assistant available in your stack and permission to create automations.

1. On the Agent Observability analytics page, click **Add report**.
2. Describe the summary you want, for example, a weekly cost report, then click **Continue in Assistant**. You can also select a template under **Starting points**.
3. Review the report instructions, schedule, and delivery settings in Grafana Assistant, then create the automation.

Describing a report or selecting a template opens a draft; it doesn’t schedule a report until you create the automation. Click **Manage automations** to manage schedules, delivery, and run history in Grafana Assistant.

The **Weekly cost digest** compares recorded costs for the last seven days with the previous seven days. These costs use your configured model prices where available and catalog prices otherwise. Unlike dashboard totals, this report doesn’t estimate older periods from token usage. If it can’t read recorded costs, it reports **No data** instead of estimating spend.

## Next steps

- [Optimize cost and performance](/docs/grafana-cloud/observe-and-act/agent-observability/observe-and-troubleshoot/cost-optimization)
- [Browse and debug conversations](/docs/grafana-cloud/observe-and-act/agent-observability/observe-and-troubleshoot/conversations)