Use exemplars with Grafana Mimir on Grafana Labs

About Grafana Mimir exemplars

Thu, 14 Aug 2025 18:42:07 +0000

About Grafana Mimir exemplars

An exemplar is a specific trace representative of a repeated pattern of data in a given time interval. It helps you identify higher cardinality metadata from specific events within time series data. To learn more about exemplars and how they can help you isolate and troubleshoot problems with your systems, see Introduction to exemplars.

Grafana Mimir includes the ability to store exemplars in-memory. Exemplar storage in Grafana Mimir is implemented similarly to how it is in Prometheus. Exemplars are stored as a fixed size circular buffer that stores exemplars in memory for all series.

The limits property can be used to control the size of the circular buffer by the number of exemplars. For reference, an exemplar with just a traceID=<jaeger-trace-id> uses roughly 100 bytes of memory via the in-memory exemplar storage. If the exemplar storage is enabled, Grafana Mimir will also append the exemplars to WAL for local persistence (for WAL duration).

Before you begin using exemplars with Grafana Mimir

Thu, 14 Aug 2025 18:42:07 +0000

Before you begin using exemplars with Grafana Mimir

Follow the checklist to ensure that your application is generating metrics, traces, and exemplars.

Verify that your application is using the official Prometheus client libraries.
Ensure that the client library you choose is emitting metrics in OpenMetrics format by referencing its documentation. For the Prometheus Go client library, for example, this requires you to set EnableOpenMetrics to true. For the Java library, follow its instructions on setting the proper header format.
Obtain the trace ID for the current request and include the trace ID in calls to emit metrics.
- For histograms, use the ObserveWithExemplar method to emit the trace ID along with a value for the histogram. These functions are from the Go library but you can find similar functions in the other libraries.
- For counters, use the AddWithExemplar method to emit the trace ID along with a counter increment.
Verify that metrics are being generated with exemplars by running the following command in a shell: curl -H "Accept: application/openmetrics-text" http://<your application>/metrics | grep -i "traceid".
Configure your Prometheus server or Grafana Agent to store and send exemplars.
- To configure Grafana Agent to send exemplars:
  1. Confirm that the Agent is scraping exemplars by verifying that the prometheus_remote_storage_exemplars_total metric is a non-zero value.
  2. Add the option send_exemplars: true under the remote_write configuration block in the Grafana Agent configuration file.
- To configure a Prometheus server to send exemplars:
  1. Run Prometheus with the --enable-feature=exemplar-storage flag.
  2. Confirm that Prometheus is scraping exemplars by verifying that the prometheus_remote_storage_exemplars_total metric is a non-zero value.
  3. Add the option send_exemplars: true under the remote_write configuration block in the Prometheus configuration file.

Store exemplars in Grafana Mimir

Thu, 14 Aug 2025 18:42:07 +0000

Store exemplars in Grafana Mimir

You can enable exemplar storage in Grafana Mimir and view the resulting data in Grafana. While exemplars can be enabled for all tenants at once or for only specific tenants, we recommend enabling and enforcing limits for exemplars for only specific tenants.

Enable exemplars globally

In the Grafana Mimir configuration file, set the limits.max_global_exemplars_per_user value. Start with a relatively low number (100,000) and adjust it if needed.

A partial Grafana Mimir configuration file with max_global_exemplars_per_user set globally looks as follows:
YAML
```
limits:
  max_global_exemplars_per_user: 100000
```
Save and deploy the configuration file.
Perform a rolling update of Grafana Mimir components.

Enable exemplars for a specific tenant

Ensure Grafana Mimir uses a runtime configuration file by verifying that the flag -runtime-config.file is set to a non-null value. For more information about supported runtime configuration, refer to Runtime configuration file
In the runtime configuration file, set the overrides.<TENANT>.max_global_exemplars_per_user value. Start with a relatively low number (100,000) and adjust it if needed.

A partial runtime configuration file with max_global_exemplars_per_user set for a tenant called “tenant-a” would look as follows:
YAML
```
overrides:
  "tenant-a":
    max_global_exemplars_per_user: 100000
```
Save and deploy the runtime configuration file.

After the -runtime-config.reload-period has elapsed, components reload the runtime configuration file and use the updated configuration.

View exemplar data in Grafana Explore

Thu, 14 Aug 2025 18:42:07 +0000

View exemplar data in Grafana Explore

Grafana Explore visualizes exemplar traces as highlighted points alongside metrics data. For more information, see View exemplar data topic in Grafana OSS documentation.