Exemplars in Grafana Cloud

An exemplar is a specific trace representative of a repeated pattern of data in a given time interval. It helps you identify higher cardinality metadata from specific events within time series data.

To learn more about exemplars and how they can help you isolate and troubleshoot problems with your systems, refer to Introduction to exemplars.

Grafana Cloud includes the ability to store exemplars in memory. Exemplar storage in Grafana Cloud is implemented similarly to how it is in Prometheus. Exemplars are stored as a fixed-size circular buffer that stores exemplars in memory for all series.

Exemplars are only supported for the Prometheus data source.

You need the following things working For exemplars to work

Before you begin

Exemplars require the following capabilities:

Your app must be generating exemplars and linking them to trace IDs.
Your app must export OpenMetrics format; exemplars aren’t supported in Prometheus format.
For Prometheus, enable exemplars with --enable-feature=exemplar-storage. Grafana Alloy doesn’t require this flag.
The receiving end, such as Grafana Cloud, must have exemplars enabled for the tenant.
Dashboards must use TimeSeries panel, not the older Graph panel.
Dashboard queries must have Exemplars toggled on.

Follow the checklist to ensure that your application is generating metrics, traces, and exemplars.

Verify that your application is using the official Prometheus client libraries.
Ensure that the client library you choose is emitting metrics in OpenMetrics format by referencing its documentation. For the Prometheus Go client library, for example, this requires you to set EnableOpenMetrics to true. For the Java library, follow its instructions on setting the proper header format.
Obtain the trace ID for the current request and include the trace ID in calls to emit metrics.
- For histograms, use the ObserveWithExemplar method to emit the trace ID along with a value for the histogram. These functions are from the Go library but you can find similar functions in the other libraries.
- For counters, use the AddWithExemplar method to emit the trace ID along with a counter increment.
Verify that metrics are being generated with exemplars by running the following command in a shell: curl -H "Accept: application/openmetrics-text" http://<your application>/metrics | grep -i "traceid".

Configure Grafana Alloy for exemplars

Your intermediate agent, Grafana Alloy stores exemplars by default. You can also use hosted Prometheus as your intermediate agent.

To configure Alloy to send metrics including exemplars to Grafana Cloud:

Confirm that Alloy scrapes exemplars by verifying that either prometheus_remote_write_wal_exemplars_appended_total or prometheus_remote_storage_exemplars_in_total is non-zero.
Add the option send_exemplars = true under the endpoint block to forward those exemplars from Alloy to Grafana Cloud.

This example block configures Alloy to scrape a local Prometheus or Mimir server and send exemplars to Grafana Cloud using prometheus.remote_write with send_exemplars enabled.

prometheus.scrape "self" {
  targets = [
    { "__address__" = "localhost:9090" },
  ]

  // Send scraped metrics to the remote_write component below
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  // Each endpoint block is a single destination. Add more blocks to send to multiple locations.

  endpoint {
    url            = "https://prometheus-xxx.grafana.net/api/prom/push"
    send_exemplars = true

    basic_auth {
      username = "<PROMETHEUS_USERNAME>"
      password = "<GRAFANA_CLOUD_API_KEY>"
    }
  }
}