Use traces to find solutions on Grafana Labs

Identify bottlenecks and establish SLOs

Thu, 09 Apr 2026 14:59:14 +0000

Identify bottlenecks and establish SLOs

Traces give a window into an application’s performance, and the insights that go along with that. Using traces, you can discover bottlenecks and potential optimizations that can decrease latency and response times and increase application throughput.

Meet Handy Site Corp

Handy Site Corp, a fake website company, runs an ecommerce application that includes user authentication, a product catalog, order management, payment processing, and other services.

Define realistic SLOs

Handy Site’s engineers start by establishing service level objectives, or SLOs, around latency ensure that customers have a good experience when trying to complete the checkout process. To do this, they use metrics generated from their span data.

Their service level objective should be a realistic target based on previous history during times of normal operation. Once they’ve agreed upon their service level objective, they set up alerts to signal risk of failing to meet that objective.

Utilize span metrics to define your SLO and SLI

After evaluating options, they decide to use span metrics as a service-level indicator (SLI) to measure SLO compliance.

Tempo can generate metrics using the metrics-generator component. These metrics are created based on spans from incoming traces and demonstrate immediate usefulness with respect to application flow and overview. This includes rate, error, and duration (RED) metrics.

Span metrics also make it easy to use exemplars. An exemplar serves as a detailed example of one of the observations aggregated into a metric. An exemplar contains the observed value together with an optional timestamp and arbitrary trace IDs, which are typically used to reference a trace. Since traces and metrics co-exist in the metrics-generator, exemplars can be automatically added to those metrics, allowing you to quickly jump from a metric showing aggregate latency over time into an individual trace that represents a low, medium, or high latency request. Similarly, you can quickly jump from a metric showing error rate over time into an individual erroring trace.

Monitor latency

Handy Site decides they’re most interested in monitoring the latency of requests processed by their checkout service and want to set an objective that 99.5% of requests in a given month should complete within 2 seconds. To define a service-level indicator (SLI) that they can use to track their progress against their objective, they use the traces_spanmetrics_latency metric with the proper label selectors, such as service name = checkoutservice. The metrics-generator adds a default set of labels to the metrics it generates, including span_kind and status_code. If they want to calculate checkout service latency per endpoint or per version of the software, they could change the configuration of the Tempo metrics-generator to add these custom dimensions as labels to their span metrics.

With all of this in place, Handy Site opens the Grafana SLO application and follows the setup flow to establish an SLI for their checkout service around the traces_spanmetrics_latency metric. They can be alerted to degradations in service quality that directly impact their end user experience. SLO-based alerting also ensures that they don’t suffer from noisy alerts. Alerts are only triggered when the value of the SLI is such that the team is in danger of missing their SLO.

Diagnose errors with traces

Thu, 09 Apr 2026 14:59:14 +0000

Diagnose errors with traces

Traces allow you to quickly diagnose errors in your application, ensuring that you can perform Root Cause Analysis (RCA) on request failures. Trace visualizations help you determine the spans in which errors occur, along with the context behind those errors, leading to a lower mean time to repair.

Meet Handy Site Corp

Handy Site Corp, a fake website company, runs an e-commerce application that includes user authentication, a product catalog, order management, payment processing, and other services.

Handy Site’s operations team receives several alerts for their error service-level objectives, or SLOs, for monitored endpoints in their services. Using their Grafana dashboards, they notice that there are several issues. The dashboard provides the percentages of errors:

More than 5% of the requests from users are resulting in an error across several endpoints, such as /beholder, /owlbear, and /illithid, which causes a degradation in performance and usability. It’s imperative for the operations team at Handy Site to quickly troubleshoot the issue. The elevated error rates indicate that the Handy Site is unable to provide valid responses to their users’ requests, which in addition to threatening the operation team’s SLO error budget also affects profitability overall for Handy Site.

Use TraceQL to query data

Tempo has a traces-first query language, TraceQL, that provides a unique toolset for selecting and searching tracing data. TraceQL can match traces based on span and resource attributes, duration, and ancestor<>descendant relationships. The language also can compute aggregate statistics, such as rate over a set of spans.

Handy Site’s has instrumented their services and applications for tracing, so they can use TraceQL as a debugging tool. Using three TraceQL queries, the team identifies and validates the root cause of the issue.

Find HTTP errors

The top-level service, mythical-requester receives requests and returns responses to users. When it receives a request, it calls numerous downstream services whose responses it relies on to send a response to the user request. Using Grafana Explore, the operations team starts with a simple TraceQL query to find all traces from this top-level service, where an HTTP response code sent to a user is 400 or above. Status codes in this range include Forbidden, Not found, Unauthorized, and other client and server errors.

{ resource.service.name = "mythical-requester" && span.http.status_code >= 400 } | select(span.http.target)

Adding the select statement to this query (after the |) ensures that the query response includes not only the set of matched spans, but also the http.target attribute (for example, the endpoints for the SaaS service) for each of those spans.

Pinpoint the error

Looking at the set of returned spans, the most concerning ones are those with the HTTP 500 status code. Those are internal server errors.

The team decides to use structural operators to follow an error chain from the top-level mythical-requester service to any descendant spans that also have an error status. Descendant spans can be any span that’s descended from the parent span, such as a child or a further child at any depth. Using this query, the team can pinpoint the downstream service that might be causing the issue. The query below says “Find spans where status = error that are descendants of spans from the mythical-requester service that have status code 500.”

{ resource.service.name = "mythical-requester" && span.http.status_code = 500 } >> { status = error }

Expanding the span with errors that’s a descendant of the span for the mythical-server service shows the team that there is a problem with the data inserted into the database. Specifically, the service is passing a null value for a column in a database table where null values are invalid.

Verify root cause

After identifying the specific cause of this internal server error, the team wants to know if there are errors in any database operations other than the null INSERT error found above. Their updated query uses a negated regular expression to find any spans where the database statement either doesn’t exist, or doesn’t start with an INSERT clause. This should expose any other issues causing an internal server error and filter out the class of issues that they already diagnosed.

{ resource.service.name = "mythical-requester" && span.http.status_code = 500 } >> { status = error && span.db.statement !~ "INSERT.*" }

This query yields no results, suggesting that the root cause of the issues the operations team are seeing is exclusively due to the failing database INSERT statement. At this point, they can roll back to a known working version of the service, or deploy a fix to ensure that the service rejects null data appropriately. After it completes, the team marks the issue resolved. The Handy team’s error rate SLI should return back to acceptable levels.

Solve problems with TraceQL metrics queries

Thu, 09 Apr 2026 14:59:14 +0000

Solve problems with TraceQL metrics queries

Caution
TraceQL metrics is an public preview feature. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available TraceQL metrics are enabled by default in Grafana Cloud.

You can query data generated by TraceQL metrics in a similar way that you would query results stored in Prometheus, Grafana Mimir, or other Prometheus-compatible Time-Series-Database (TSDB). TraceQL metrics queries allows you to calculate metrics on trace span data on-the-fly with Tempo (your tracing database), without requiring a time-series-database like Prometheus.

This page provides an example of how you can investigate the rate of incoming requests using both PromQL and TraceQL.

RED metrics and queries

The Tempo metrics-generator emits metrics with pre-configured labels for Rate, Error, and Duration (RED) metrics and service graph edges. Generated metric labels vary, but always include the service name. For example, in service graph metrics, as a client or a server type, or both. For more information, refer to the metrics-generator documentation.

You can use these metrics to get an overview of application performance. The metrics can be directly correlated to the trace spans that are available for querying.

TraceQL metrics allow a user to query metrics from traces directly from Tempo instead of requiring the metrics-generator component and an accompanying TSDB.

Note
TraceQL metrics are constrained to a 24-hour range window, and aren’t available as a Grafana Managed Alerts source. For any metrics that you want to query over longer time ranges, use for alerting, or retain for more than 30 days, use the metrics-generator to store these metrics in Prometheus, Mimir, or other Prometheus-compatible TSDB and continue to use PromQL for querying.

Investigate the rate of incoming requests

Let’s say that you want to know how many requests are being serviced both by your application, but also by each service that comprises your application. This allows you to ensure that your application scales appropriately, can help with capacity planning, and can show you which services may be having problems and are taking up load in fail-over scenarios. In PromQL, these values are calculated over counters that increase each time a service is called. These metrics provide the Rate (R) in RED.

If you are familiar with PromQL, then you’re used to constructing queries. You can create an equivalent queries in TraceQL. Here’s the two queries for the different data sources (PromQL for Mimir and TraceQL for Tempo), shown side by side over a 6 hour time-range.

How the query looks in PromQL

The Tempo metrics-generator outputs a metric, traces_spanmetrics_calls_total, a counter that increases each time a named span in a service is called. RED data generated by the metrics-generator includes the service name and span kind. You can use this to only show call counts when a service was called externally by filtering via the SERVER span kind, thus showing the total number of times the service has been called.

You can use the PromQL rate() and sum() functions to examine the counter and determine the per-second rate of calls occurring, summing them by each service. In addition to only looking at spans of kind=server, you can also focus on spans coming from a particular Kubernetes namespace (ditl-demo-prod).

sum by (service_name)(rate(traces_spanmetrics_calls_total{service_namespace="ditl-demo-prod", span_kind="SPAN_KIND_SERVER"}[2m]))

How the query looks in TraceQL

TraceQL metrics queries let you similarly examine a particular subset of your spans. As in the example above, you can start by filtering down to spans that occur in a particular Kubernetes namespace (ditl-demo-prod), and are of kind SERVER. That resulting set of spans is piped to the TraceQL rate function, which then calculates the rate (in spans/sec) at which spans matches your filters are received. By adding the by (resource.service.name) term, the query returns spans per second rates per service, rather than an aggregate across all services.

{ resource.service.namespace="ditl-demo-prod" && kind=server } | rate() by (resource.service.name)