Monitor agents with analytics
Agent Observability includes built-in analytics that visualize agent activity, performance, cost, and quality. The analytics views use Prometheus metrics and Agent Observability data to surface actionable insights.
Before you begin
- Configure your agent to export generations and OpenTelemetry metrics to Grafana Cloud. For instructions, refer to Get started with Grafana Cloud.
- Select a time range that includes your agent traffic.
Access dashboards
Select Analytics in Agent Observability. Under Explore by area, expand:
- Agents: generation counts, latency, time to first token, errors, tokens, and estimated cost by agent.
- Usage and cost: token usage, estimated cost, cache efficiency, and conversation activity.
- Evaluations: evaluation scores, pass rates, and quality trends.
- Tools: tool call frequency, duration, and error rates.
Use the time picker to set the query window. The refresh picker lets you set an auto-refresh interval, for example, 30s, 1m, or 5m. The selected interval is saved for the session.
Find items that need attention
The Needs attention section highlights agents, conversations, tools, and evaluations that may need review. Open Health overview settings and select Sudden changes to compare with the previous time window, or High levels to check absolute values. Turn on Show only entities that need attention to hide healthy cards.
Open a card to investigate its supporting data in the relevant Agent Observability view.
Include or exclude traffic
Use the Provider, Model, and Agent filters to focus on specific traffic. Select one or more values in each menu. By default, the filter includes the selected values.
Dropdown options cascade to reflect your selections:
- Selecting a provider limits Model options to models observed for that provider.
- Selecting a provider and model limits Agent options to agents that match both selections.
In coding-agent analytics views, Provider, Model, and Agent dropdown options are scoped to metrics from known coding-agent launchers. Repository, user, and added label filters use the same coding-agent scope. This prevents unrelated application names or unused provider and model combinations from appearing. Other analytics pages provide unscoped options across all agents.
To hide those values instead, select Exclude selected in that filter’s menu. Close the menu to apply your selection. Each filter has its own mode, so you can include one provider while excluding a model or agent. Clear Exclude selected to return to inclusion mode.
These controls also apply when you browse conversations or review evaluation results. Your selections and exclusion modes stay in the URL when you share the view.
Break a chart down by one or more dimensions
Each chart has a Break down switch that splits a single line into one line per dimension. Turn it on and choose dimensions from the picker next to it.
The picker lists common dimensions such as Agent, Model, and Provider first, followed by every other Prometheus label present on your gen_ai_client_* metrics in the current time range. Both kinds work the same way, so you can compare staging against production, or one team’s spend against another’s.
Select more than one dimension to see their combinations. Grouping by Agent and deployment_environment gives you one line per agent per environment, named checkout-agent / production.
You can combine up to three dimensions. Each one multiplies the number of lines on the chart, so the picker stops offering more once you reach the limit. If a chart becomes hard to read, remove a dimension or narrow the filters above it.
A label appears in the picker only if your SDK or collector attaches it to the metrics. The simplest way to add one is to set resource attributes on the service that emits the telemetry:
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=aiDots become underscores in Prometheus label names, so deployment.environment appears as deployment_environment.
You can also set client tags in the SDK, either with the AGENTO11Y_TAGS environment variable or the tags field in the client configuration:
export AGENTO11Y_TAGS=team=ai,env=productionEach client tag becomes a label named agento11y_tag_<KEY>, for example, agento11y_tag_team. The SDK also stores the value on every generation.
For coding-agent sessions, refer to Attribute cost to a user, repository, or branch to add user, repository, and branch labels automatically.
Keep the number of distinct values small, because each value adds a metric series. For the behavior of each mechanism, refer to Client tags.
Tags that you pass on a single generation, and generation metadata, aren’t metric labels, so they don’t appear in the picker. Filter conversations by metadata.<KEY> instead.
Your selection is remembered per chart area.
Identify performance issues
Use the performance dashboard to spot problems:
- High latency: filter by agent or model to find slow generations. Drill into traces for specific conversations to identify bottlenecks.
- Error spikes: the error rate panel shows failures over time. Click through to conversations with errors to inspect the
call_errorpayload. - Slow time to first token: for streaming agents, the TTFT panel reveals which models or prompts have poor streaming performance.
Inspect tool failures
On the Tools page (Analytics > Tools), review tool executions, error rates, and latency.
The AI analysis bar appears directly below the summary KPI band. It evaluates visible tool metrics against the previous period, including any active provider, model, agent, and label filters, to highlight key anomalies:
- Sharp execution-volume shifts and concentration risk across top tools.
- Tools with the highest error rates and error counts.
- Outlier tools with the slowest P95 latency.
Click the AI analysis bar to expand or collapse findings. When collapsed, the header displays a preview of the first finding. Click Regenerate to refresh the analysis for current filters and data. Click any finding to open Grafana Assistant and explain it.
The bar requires Grafana Assistant; it doesn’t appear if Grafana Assistant is disabled. While queries are loading or if no tools match your filters, the bar stays mounted and waits for data before generating insights. If a metric query fails or data is unavailable, the analysis labels it as no data instead of reporting a measured zero, and omits unavailable previous-period comparisons.
Select Error count to chart the number of failed tool executions instead of the percentage shown by Error rate. You can also select Error count in the Tools area under Analytics; its table ranks tools by the highest error count.
Click Tool, Executions, Exec %, Errors, Error rate, or Latency to sort the table. Click the same header again to reverse the order. The sort menu and column headers control the same ordering.
Optimize costs
The tokens and cost dashboard helps you find optimization opportunities:
- Cost by model: compare estimated cost across models and providers. Consider switching expensive calls to cheaper models where quality is acceptable.
- Cache efficiency: the cache read ratio shows how effectively prompt caching reduces token usage. Low cache rates may indicate prompts that change too frequently.
- Token usage trends: spot unexpected increases in token usage that may indicate prompt regression or unnecessary verbosity.
Select Cache hit rate in Usage and cost to compare prompt cache reuse over time. Break down by Agent, Model, or Provider to find differences. If you select Token type, the chart shows cache read and cache write token counts instead of a percentage. The conversations table also shows Cache hit rate so you can inspect individual sessions.
The overview cost total covers your selected time range. It uses recorded per-request costs where available and estimates earlier periods from token usage and catalog prices. Agents totals and Needs attention cost charts use the same approach. Recorded costs include your configured model prices. If no recorded costs are available, Agent Observability estimates the whole range. Token-based estimates don’t account for per-request long-context pricing tiers and can understate costs for long prompts.
Cost charts mark the transition from token-based estimates to recorded costs. A step at that marker doesn’t necessarily mean spending changed. Changing chart time buckets doesn’t change the cost total for the selected range.
When Cost shown at list prices appears, your selected filter or breakdown requires token-based estimates instead of recorded costs. Configured model prices and long-context pricing don’t apply to these estimates. A notice below the filters applies to every cost figure on the page; a notice on a chart applies only to that breakdown. Remove the named filter or breakdown to return to recorded costs where available. For details, refer to Configure your model prices.
When you ask Grafana Assistant about costs from Analytics or Agents, it receives the page’s cost source and selected scope. Recorded costs include your configured prices. Estimates for older periods or unsupported labels use public list prices instead; check which source the answer uses before comparing figures.
Track quality
The quality dashboard visualizes evaluation scores alongside operational metrics:
- Score trends: monitor if quality improves or degrades after agent version changes.
- Score distributions: identify if responses cluster around high or low scores.
- Correlation: compare quality scores with latency and cost to find the right balance.
Use Prometheus metrics directly
If you need custom dashboards, query the Agent Observability OpenTelemetry metrics in Prometheus:
For custom cost queries, use these stack-scoped metrics from your configured Prometheus data source:
Multiplying aggregated token usage by current observed prices doesn’t reproduce recorded costs: historical rates, long-context tiers, and per-request fees can differ. Cost estimates aren’t provider invoices.
Reuse dashboard panels
On panels with More actions, click Open in Explore to open the Prometheus query with the selected time range. Click Add panel to dashboard to copy the panel into a Grafana dashboard. Adding panels requires a configured Prometheus data source, dashboard create or write permission, and Grafana’s add-to-dashboard integration.
Some panels also offer Create alert or links to related conversations and agents. Alert creation requires Grafana Alerting, alert-creation permission, and Grafana’s create-alert integration.
Ask scoped analytics questions
When Grafana Assistant is available, ask about the analytics page’s selected time range and filters, for example, cost, latency, errors, or tool usage. The page supplies this scope and its query context to Assistant.
A conversation can contain several model calls. Specify whether you want conversation totals or model-call totals so the answer uses the measure you need.
Open conversations from exemplars
Dashboard exemplars connect metric spikes to example conversations. When exemplars are enabled, histogram panels can show exemplar markers with links that open the matching conversation.
Use exemplar links to move from a latency or error spike to the conversation that produced it. This feature requires exemplars to be enabled for your stack.
Set up alerts
Create Grafana alerts on Agent Observability metrics to proactively catch issues:
- Alert on error rate exceeding a threshold.
- Alert on p95 latency exceeding SLO targets.
- Alert on estimated cost per day exceeding budget.
- Alert on evaluation score drops below a quality threshold.
Configure alerts in Grafana using the standard alerting workflow with the Prometheus data source.
Schedule recurring reports
Use reports to receive recurring summaries of your agents’ cost, latency, errors, or quality. You need Grafana Assistant available in your stack and permission to create automations.
- On the Agent Observability analytics page, click Add report.
- Describe the summary you want, for example, a weekly cost report, then click Continue in Assistant. You can also select a template under Starting points.
- Review the report instructions, schedule, and delivery settings in Grafana Assistant, then create the automation.
Describing a report or selecting a template opens a draft; it doesn’t schedule a report until you create the automation. Click Manage automations to manage schedules, delivery, and run history in Grafana Assistant.
The Weekly cost digest compares recorded costs for the last seven days with the previous seven days. These costs use your configured model prices where available and catalog prices otherwise. Unlike dashboard totals, this report doesn’t estimate older periods from token usage. If it can’t read recorded costs, it reports No data instead of estimating spend.
Next steps
Was this page helpful?
Related resources from Grafana Labs
