Concepts on Grafana Labs

Traces and telemetry

Tue, 28 Jul 2026 16:42:04 +0000

Traces and telemetry

Metrics, logs, traces, and profiles form the pillars of observability. Correlating between the four pillars of observability helps create a holistic view of your application and infrastructure.

Metrics

Metrics provide a high level picture of the state of a system. Metrics are the foundation of alerts because metrics are numeric values and can be compared against known thresholds. Alerts constantly run in the background and trigger when a value is outside of an expected range. This is typically the first sign that something is going on and are where discovery first starts. Metrics indicate that something is happening.

Logs

Logs provide an audit trail of activity from a single process that create informational context. Logs act as atomic events, detailing what’s occurring in the services in your application. Whereas metrics are quantitative (numeric) and structured, logs are qualitative (textual) and unstructured or semi-structured. They offer a higher degree of detail, but also at the expense of creating significantly higher data volumes. Logs let you know what’s happening to your application.

Traces

Traces add further to the observability picture by telling you what happens at each step or action in a data pathway. Traces provide a map of where something is going wrong. A trace provides a graphic representation of how long each step in the data flow pathway takes to complete. For example, how long a HTTP request, a database lookup, or a call to a third party service takes. It can show where requests initiate and finish, as well as how your system responds. This data helps you locate problem areas and assess their impact, often in places you never would have anticipated or found without this ability to trace the request flow.

Profiles

Profiles help you understand how your applications utilize compute resources such as CPU time and memory. This helps identify specific lines of code or functions to optimize and improve performance and efficiency.

Why traces?

Metrics in themselves aren’t sufficient to find the root cause and solve complex issues. The same can be said for logs, which can contain a significant amount of information but lack the context of the interactions and dependencies between the different components of your complex environment. Each pillar of observability has its own unique strength when it comes to root causing issues. To get the most value of your observability strategy, you need to be able to correlate them.

Traces have the unique ability to show relationships between services. They help identify which services are upstream from your service, which is helpful when you want to understand which services might be negatively impacted by problems in your service. Traces also help identify which services are downstream from your service. This is valuable since your application relies on their downstream services, and problems with those services may be the cause of elevated errors or latency reported by your service. For example, you can directly see the failing database and all impacted failing edge endpoints.

Using traces and exemplars, you can go from a metric data point and get to an associated trace.

Or from traces to logs:

And vice versa, from logs to traces.

Trace structure

Tue, 08 Jul 2025 20:53:47 +0000

Trace structure

Tempo and AI

Thu, 02 Jul 2026 12:47:11 +0000

Tempo and AI

Tempo exposes a Model Context Protocol (MCP) server and LLM-optimized API endpoints that let AI agents query traces, compute metrics, and discover attributes directly. Grafana also provides command-line tools that connect agents to the broader observability platform.

Model Context Protocol server

The MCP server at /api/mcp lets agents search for traces with TraceQL, retrieve a trace by ID, compute metrics from span data, and discover available attributes. The server also serves TraceQL syntax documentation as MCP resources, so agents can look up query syntax on demand instead of relying on training data.

The MCP server uses the streamable-http transport and supports the same authentication and multi-tenancy as other Tempo API endpoints.

Warning
The MCP server returns tracing data to the calling agent, which may forward it to an LLM provider. Consider the content of your tracing data and your organization’s policies before connecting agents to this endpoint.

To try it locally, refer to the MCP server quick start.

LLM-optimized API responses

The trace by ID v2 and tag values v2 endpoints accept an Accept: application/vnd.grafana.llm header. This returns a simplified JSON format that strips unnecessary detail and reduces token usage, so agents can process larger traces within their context window.

Caution
The simplified LLM format is subject to change and shouldn’t be relied on for programmatic use.

Use this header for interactive or experimental agent use. For stable integrations, use the MCP server or request the standard JSON or protobuf formats.

Tempo features relevant to AI workflows

These Tempo features aren’t AI-specific, but they directly improve what agents can do with tracing data.

TraceQL metrics at general availability

The MCP server’s traceql-metrics-instant and traceql-metrics-range tools let agents compute rate, error, and duration metrics directly from trace data. Because TraceQL metrics moved from experimental to GA in Tempo 3.0, these tools query a production-ready engine.

OR conditions for tag autocomplete

The search tags v2 and search tag values v2 APIs support OR conditions, so autocomplete requests can match multiple values in a single call instead of making separate requests. When an agent discovers available attributes through the MCP tools, fewer round-trips mean faster, cheaper exploration of what’s queryable.

Control what agents can access

Before you open up trace data to agents through the MCP server or API endpoints, you can control what’s available. If your traces contain personally identifiable information or security tokens, use the Tempo CLI to remove affected traces from object storage first.

Redact traces removes traces containing sensitive data from object storage. To remove specific traces directly from the CLI, drop traces by ID.

Grafana tools for AI workflows

These capabilities aren’t specific to Tempo. You have them by working in the Grafana product family, and they apply across the platform. Their AI value is composability: an agent that starts in Tempo’s trace data can pivot outward to act on Grafana resources, chain investigations across other signals, and look up current documentation.

For additional information about AI capabilities in Grafana, refer to the AI and machine learning documentation.

`gcx`

gcx is a CLI for managing Grafana resources, including dashboards, data sources, alerting rules, and Grafana Cloud products like Synthetic Monitoring, SLO, and Adaptive Telemetry. It works with Grafana Cloud, Grafana Enterprise, and Grafana OSS (v12 or later).

An agent can use gcx alongside the Tempo MCP server to act on what it finds in traces. For example, after identifying a latency issue, the agent could query related Prometheus metrics or inspect alerting rules through gcx without leaving the terminal.

Grafana Assistant CLI

Grafana Assistant is an LLM-powered tool built into Grafana Cloud that queries data, builds dashboards, and helps you understand errors using natural language. It requires a Grafana instance where Grafana Assistant is enabled (for example, a Grafana Cloud stack).

The assistant-cli connects agents to Grafana Assistant through the Agent-to-Agent (A2A) API. This lets an agent chain investigations across observability signals from the terminal. For example, an agent could find a failing trace in Tempo, then use the assistant-cli to correlate the failure with error logs or query related metrics.

Documentation as Markdown

Grafana documentation is available as Markdown, so agents can fetch current reference material instead of relying on potentially outdated training data.

Next steps

Set up the MCP server to give agents access to your tracing data.
Install gcx to manage Grafana resources from the terminal or an agent workflow.
Install the assistant-cli to connect agents to Grafana Assistant.

Glossary

Tue, 02 Jun 2026 16:22:08 +0200

Glossary

The following terms are often used when discussing traces.

Active series: A time series that receives new data points or samples.
Cardinality: The total combination of key/value pairs, such as labels and label values for a given metric series or log stream, and how many unique combinations they generate.

Child span: A span that is nested within a parent span. Each child span represents a sub-operation or downstream call within the broader operation represented by its parent. A child span can itself be a parent of other child spans.
Data source: A basic storage for data such as a database, a flat file, or even live references or measurements from a device. A file, database, or service that provides data. For example, trace data is imported into Grafana by configuring and enabling a Tempo data source.

Event attribute: A key-value pair attached to a span event, which represents a unique point in time during the span’s duration. For more information, refer to Span events in the OpenTelemetry documentation.
Exemplar: Any data that serves as a detailed example of one of the observations aggregated into a metric. An exemplar contains the observed value together with an optional timestamp and arbitrary trace IDs, which are typically used to reference a trace.

Intrinsics: The core, built-in fields that are fundamental to the identity and lifecycle of spans and traces. Intrinsic fields include name, duration, status, and kind. These fields are defined by the OpenTelemetry specification and are always present.

Link attribute: A key-value pair attached to a span link. A span link associates one span with one or more causally related spans. For more information, refer to Span Links in the OpenTelemetry documentation.
Log: Chronological events, usually text-based, allowing for the diagnosis of problems. Logs can provide informational context, such as detailed records of all events during user interactions, for example, when events happen, who used the system, status messages, etc.
Metric: A number that helps an operator understand the state of a system, such as the number of active users, error count, average response time, and more.

Resource attribute: A key-value pair that represents information about the entity producing a span, such as the cluster name, namespace, Pod, or container name.

Root span: The first span in a trace, representing the initial request or operation. A root span has no parent span and serves as the top of the span tree.

Semantic attribute: A standardized naming scheme for attributes shared across languages, frameworks, and runtimes, as defined by the OpenTelemetry specification. For more information, refer to Trace Semantic Conventions in the OpenTelemetry documentation.
Span: A unit of work done within a trace.

Span attribute: A key-value pair that contains metadata to annotate a span with information about the operation it tracks. For example, a span tracking an “add to cart” operation might include the user ID, item ID, and cart ID as span attributes.
Spanset: The group of spans for each individual trace returned from a TraceQL query.
Trace: A trace represents the whole journey of a request or an action as it moves through all the nodes of a distributed system, especially containerized applications or microservices architectures. Traces are composed of spans. For more information, refer to What is Distributed Tracing?.