Exemplars on Grafana Labs

Prerequisites

Mon, 17 Mar 2025 14:18:51 +0100

Before you begin

Follow the checklist to ensure that your application is generating metrics, traces, and exemplars.

Verify that your application is using the official Prometheus client libraries.
Ensure that the client library you choose is emitting metrics in OpenMetrics format by referencing its documentation. For the Prometheus Go client library, for example, this requires you to set EnableOpenMetrics to true. For the Java library, follow its instructions on setting the proper header format.
Obtain the trace ID for the current request and include the trace ID in calls to emit metrics.
- For histograms, use the ObserveWithExemplar method to emit the trace ID along with a value for the histogram. These functions are from the Go library but you can find similar functions in the other libraries.
- For counters, use the AddWithExemplar method to emit the trace ID along with a counter increment.
Verify that metrics are being generated with exemplars by running the following command in a shell: curl -H "Accept: application/openmetrics-text" http://<your application>/metrics | grep -i "traceid".

Enable exemplars

Mon, 17 Mar 2025 14:18:51 +0100

Enable exemplars in GEM

You can enable exemplar storage in Grafana Enterprise Metrics and view the resulting data in Grafana. While exemplars can be enabled for all tenants at once or for only specific tenants, we recommend enabling and enforcing limits for exemplars for only specific tenants.

Adjust per tenant exemplar limits via the UI

Open the Grafana Enterprise Metrics plugin in your Grafana instance.
In the tenants’ view, select the tenant where you want to adjust the exemplar limit.
Click the “edit” icon for the limit max_global_exemplars_per_user parameter to edit the value. We recommend starting with a relatively low number (10,000) and adjusting it if needed.

The new limit takes effect automatically in all GEM components in five (5) minutes.

Enable exemplars globally

Open the Grafana Enterprise Metrics configuration file.
In the limits config section, set the max_global_exemplars_per_user value. We recommend starting with a relatively low number (10,000) and adjusting it if needed.
Restart all GEM components for the changes to effect.

View exemplars

Mon, 17 Mar 2025 14:18:51 +0100

View exemplar data

Explore visualizes exemplar traces as highlighted stars alongside metrics data. For more information, see View exemplar data topic in Grafana OSS documentation.