An issue occurs while searching for traces in Grafana Explore. The Service Name and Span Name drop down lists are empty, and there is a
No options found message.
HTTP requests to Tempo query frontend endpoint at
/api/search/tag/service.name/values would respond with an empty set.
The introduction of a cap on the size of tags causes this issue.
max_bytes_per_tag_values_query causes the return of an empty result
when a query exceeds the configured value.
There are two main solutions to this issue:
- Reduce the cardinality of tags pushed to Tempo. Reducing the number of unique tag values will reduce the size returned by a tag search query.
- Increase the
max_bytes_per_tag_values_queryparameter in the overrides block of your Tempo configuration to a value as high as 50MB.
Related Tempo resources
Be the first to learn about exciting next-generation features in Grafana 8.0, be inspired by what community members are building, and attend expert-led sessions and workshops on Grafana, Prometheus, Loki logs, and more.
Getting started with tracing and Grafana Tempo
In this session, we will use an instrumented application to demonstrate how to use logs and Prometheus exemplars to find traces effectively in Tempo. The demo will hit on the basics of operating Tempo and reveal how Tempo allows us to scale tracing as far as possible with less operational cost and complexity than ever before.