Documentation Grafana Cloud Instrument and send data Traces Use traces in Grafana

Use traces in Grafana

Using Grafana Cloud Traces, you can search for traces, generate metrics from spans, and link your tracing data with logs, metrics, and profiles.

Use Explore Traces to investigate tracing data

Explore Traces helps you visualize insights from your Tempo traces data. Using the app, you can:

• Use Rate, Errors, and Duration (RED) metrics derived from traces to investigate issues
• Uncover related issues and monitor changes over time
• Browse automatic visualizations of your data based on its characteristics
• Do all of this without writing TraceQL queries

Expand your observability journey and learn about the Explore apps suite.

Search for traces

Search for traces using common dimensions such as time range, duration, span tags, service names, etc. Use the Explore trace view to quickly diagnose errors and high latency events in your system.

Use trace search results as panels in dashboards

You can embed tracing panels and visualizations in dashboards. Your queries can be saved as panels. For more information, refer to the Traces Visualization documentation.

For example dashboards, visit play.grafana.org.

• Traces and basic operations
• Grafana Explore with a Tempo data source

Use TraceQL to query data and generate metrics

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces.

Using Grafana Explore, you can search traces. The default traces search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results return faster because the queries limit what is searched.

If you are using Cloud Traces, you can construct queries using the TraceQL query editor or use the Search query type (preview feature).

For details about how queries are constructed, read the TraceQL documentation.

TraceQL metrics queries

TraceQL language provides metrics queries as an experimental feature. Metric queries extend trace queries by applying an aggregate function to trace query results. For example: { span:name = "foo" } | rate() by (span:status) This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs.

Explore Traces is powered by metrics queries.

For more information about available queries, refer to TraceQL metrics queries.

Generate metrics from spans

RED metrics can drive service graphs and other ready-to-go visualizations of your span data. RED metrics represent:

• Rate, the number of requests per second
• Errors, the number of those requests that are failing
• Duration, the amount of time those requests take

For more information about RED method, refer to The RED Method: how to instrument your services.

To enable metrics-generator, refer to Enable metrics-generator.

These metrics exist in your Hosted Metrics instance and can also be easily used to generate powerful custom dashboards.

Metrics automatically generate exemplars as well which allows easy metrics to trace linking. Exemplars are GA in Grafana Cloud so you can also push your own.

Service graph view

Service graph view displays a table of request rate, error rate, and duration metrics (RED) calculated from your incoming spans. It also includes a node graph view built from your spans.

To use the service graph view, you need to enable service graphs and span metrics. After it’s enabled, this pre-configured view is immediately available in Explore > Service Graphs.

Refer to service graph view documentation for further information.

Link traces and logs

If you’re already doing request/response logging with trace IDs, they can be easily extracted from logs to jump directly to your traces.

In the other direction, you can configure Grafana Cloud to create a link from an individual span to your Loki logs. If you see a long-running span or a span with errors, you can immediately jump to the logs of the process causing the error.

Refer to Set up and use tracing to get started.

Link traces and metrics

Grafana can correlate different signals by adding the functionality to link between traces and metrics. Trace to metrics lets you navigate from a trace span to a selected data source. Using trace to metrics, you can quickly see trends or aggregated data related to each span.

For example, you can use span attributes to metric labels by using the $__tags keyword to convert span attributes to metrics labels.

To set up Trace to metrics for your data source, refer to Trace to metric configuration.

Link traces and profiles

Using Trace to profiles, you can use Grafana’s ability to correlate different signals by adding the functionality to link between traces and profiles. Refer to the Traces to profiles documentation for configuration instructions.