Configure and use exemplars

An exemplar is a specific trace representative of a repeated pattern of data in a given time interval. It helps you identify higher cardinality metadata from specific events within time series data.

To learn more about exemplars and how they can help you isolate and troubleshoot problems with your systems, refer to Introduction to exemplars.

Grafana Cloud includes the ability to store exemplars in memory. Exemplar storage in Grafana Cloud is implemented similarly to how it is in Prometheus. Exemplars are stored as a fixed-size circular buffer that stores exemplars in memory for all series.

Exemplars are only supported for the Prometheus data source.

You need the following configured for exemplars to work:

Before you begin

Exemplars require the following capabilities:

Your app must be generating exemplars and linking them to trace IDs.
Your app must export OpenMetrics format; exemplars aren’t supported in Prometheus format.
For Prometheus, enable exemplars with --enable-feature=exemplar-storage. Grafana Alloy doesn’t require this flag.
The receiving end, such as Grafana Cloud, must have exemplars enabled for the tenant.
Dashboards must use TimeSeries panel, not the older Graph panel.
Dashboard queries must have Exemplars toggled on.

Follow the checklist to ensure that your application is generating metrics, traces, and exemplars.

Verify that your application is using the official Prometheus client libraries.
Ensure that the client library you choose is emitting metrics in OpenMetrics format by referencing its documentation. For the Prometheus Go client library, for example, this requires you to set EnableOpenMetrics to true. For the Java library, follow its instructions on setting the proper header format.
Obtain the trace ID for the current request and include the trace ID in calls to emit metrics.
- For histograms, use the ObserveWithExemplar method to emit the trace ID along with a value for the histogram. These functions are from the Go library but you can find similar functions in the other libraries.
- For counters, use the AddWithExemplar method to emit the trace ID along with a counter increment.
Verify that metrics are being generated with exemplars by running the following command in a shell: curl -H "Accept: application/openmetrics-text" http://<your application>/metrics | grep -i "traceid".

Configure Grafana Alloy for exemplars

Your intermediate agent, Grafana Alloy stores exemplars by default. You can also use hosted Prometheus as your intermediate agent.

To configure Alloy to send metrics including exemplars to Grafana Cloud:

Confirm that Alloy scrapes exemplars by verifying that either prometheus_remote_write_wal_exemplars_appended_total or prometheus_remote_storage_exemplars_in_total is non-zero.
Add the option send_exemplars = true under the endpoint block to forward those exemplars from Alloy to Grafana Cloud.

This example block configures Alloy to scrape a local Prometheus or Mimir server and send exemplars to Grafana Cloud using prometheus.remote_write with send_exemplars enabled.

prometheus.scrape "self" {
  targets = [
    { "__address__" = "localhost:9090" },
  ]

  // Send scraped metrics to the remote_write component below
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  // Each endpoint block is a single destination. Add more blocks to send to multiple locations.

  endpoint {
    url            = "https://prometheus-xxx.grafana.net/api/prom/push"
    send_exemplars = true

    basic_auth {
      username = "<PROMETHEUS_USERNAME>"
      password = "<GRAFANA_CLOUD_API_KEY>"
    }
  }
}

View exemplars

Explore visualizes exemplar traces as highlighted stars alongside metrics data.

For more examples, refer to Introduction to exemplars.

Troubleshoot exemplars

If you are having trouble with exemplars, you can troubleshoot by checking the following:

Verify that your application is generating exemplars and linking them to trace IDs.
Verify that your application is exporting OpenMetrics format; exemplars aren’t supported in Prometheus format.
Verify that your application is using the official Prometheus client libraries.
Verify that the client library you choose is emitting metrics in OpenMetrics format by referencing its documentation.
Verify that metrics are being generated with exemplars by running the following command in a shell: curl -H "Accept: application/openmetrics-text" http://<your application>/metrics | grep -i "traceid".

Exemplar links to traces do not work and return a 404

Check that the exemplar data source is correct. Go to the data source settings and make sure that the data source used for Exemplars is the correct traces data source. If they have multiple Tempo data sources, it’s possible that they may be pointing to the wrong one.

Exemplars are generated by the metrics generator. It extracts the trace IDs and attaches them to the metrics as exemplars. The metrics generator does this immediately, and processes spans as they arrive. The tail sampling component usually has a decision period (the default is 30s), meaning that there is a wait before sampling decisions are made about whether or not to keep the trace. It may be possible that they’re seeing the errors because the exemplars are referencing some trace IDs that don’t exist in storage because they have been dropped.

Exemplars are generated in a separate pipeline before tail sampling occurs. The reason they’re not seeing exemplars in Grafana is because the traces that are linked to those specific data points are being dropped by the tail sampler as they don’t match the policies that are being used. This is not a bug; we cannot guarantee that the trace for an exemplar has been sampled.

Exemplars not appearing on graphs

Verify that your panel type is Time series and Exemplars toggle is enabled. Verify that your application is emitting OpenMetrics with exemplars linked to trace IDs. Verify that you have set send_exemplars = true in prometheus.remote_write endpoint in Alloy. Verify that exemplars exist by running the following command in a shell: curl -H "Accept: application/openmetrics-text" http://<your application>/metrics | grep -i "traceid".

Exemplar links to traces do not work and return a 404

Ensure the Exemplar trace data source points to the correct Tempo data source.
Be aware tail sampling can drop traces after exemplars are generated; not all exemplars have a retained trace.