Configure Grafana Mimir tracing

Grafana Mimir uses Jaeger to implement distributed tracing. Jaeger is a valuable tool for troubleshooting the behavior of Grafana Mimir in production.

Dependencies

Set up Jaeger deployment to collect and store traces from Grafana Mimir. A deployment includes either the Jaeger all-in-one binary, or a distributed system of agents, collectors, and queriers. If you run Grafana Mimir on Kubernetes, refer to Jaeger Kubernetes.

Configuration

To configure Grafana Mimir to send traces, perform the following steps:

Set the JAEGER_AGENT_HOST environment variable in all components to point to the Jaeger agent.
Enable sampling in the appropriate components:
- The ingester and ruler self-initiate traces and should have sampling explicitly enabled.
- Sampling for the distributor and query-frontend can be enabled in Grafana Mimir or in an upstream service, like a proxy or gateway running in front of Grafana Mimir.

To enable sampling in Grafana Mimir components you can specify either JAEGER_SAMPLER_MANAGER_HOST_PORT for remote sampling, or JAEGER_SAMPLER_TYPE and JAEGER_SAMPLER_PARAM to manually set sampling configuration. Refer to Jaeger Client Go documentation for the full list of environment variables you can configure.

Note that you must specify one of JAEGER_AGENT_HOST or JAEGER_SAMPLER_MANAGER_HOST_PORT in each component for Jaeger to be enabled, even if you plan to use the default values.