Configuring Grafana Mimir tracing
Grafana Mimir uses Jaeger to implement distributed tracing. Jaeger is a valuable tool for troubleshooting the behavior of Grafana Mimir in production.
Set up Jaeger deployment to collect and store traces from Grafana Mimir. A deployment includes either the Jaeger all-in-one binary, or a distributed system of agents, collectors, and queriers. If you run Grafana Mimir on Kubernetes, refer to Jaeger Kubernetes.
To configure Grafana Mimir to send traces, perform the following steps:
- Set the
JAEGER_AGENT_HOSTenvironment variable in all components to point to the Jaeger agent.
- Enable sampling in the appropriate components:
- The ingester and ruler self-initiate traces and should have sampling explicitly enabled.
- Sampling for the distributor and query-frontend can be enabled in Grafana Mimir or in an upstream service, like a proxy or gateway running in front of Grafana Mimir.
To enable sampling in Grafana Mimir components you can specify either
JAEGER_SAMPLER_MANAGER_HOST_PORT for remote sampling, or
JAEGER_SAMPLER_PARAM to manually set sampling
configuration. Refer to Jaeger Client Go
for the full list of environment variables you can configure.
Note that you must specify one of
JAEGER_SAMPLER_MANAGER_HOST_PORT in each component for Jaeger to be enabled,
even if you plan to use the default values.