Menu
Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Instrument and send databreadcrumb arrow Tracesbreadcrumb arrow Exemplars
Grafana Cloud RSS

Exemplars in Grafana Cloud

An exemplar is a specific trace representative of a repeated pattern of data in a given time interval. It helps you identify higher cardinality metadata from specific events within time series data.

To learn more about exemplars and how they can help you isolate and troubleshoot problems with your systems, refer to Introduction to exemplars.

Grafana Cloud includes the ability to store exemplars in memory. Exemplar storage in Grafana Cloud is implemented similarly to how it is in Prometheus. Exemplars are stored as a fixed-size circular buffer that stores exemplars in memory for all series.

You need the following things working For exemplars to work

Before you begin

Exemplars require the following capabilities:

  • Your app must be generating exemplars (and linking them to trace IDs).
  • Your app must export OpenMetrics format; exemplars are not supported in Prometheus format.
  • The scraping agent, such as Grafana Agent or Prometheus, must have exemplars enabled (--enable-feature=exemplar-storage).
  • The scraping agent must have send_exemplars: true in remote_write configuration block.
  • The receiving end, such as Grafana Cloud, must have exemplars enabled for the tenant.
  • Dashboards must use TimeSeries panel, not the older ‘Graph’ panel.
  • Dashboard queries must have ‘Exemplars’ toggled on.

Follow the checklist to ensure that your application is generating metrics, traces, and exemplars.

  • Verify that your application is using the official Prometheus client libraries.
  • Ensure that the client library you choose is emitting metrics in OpenMetrics format by referencing its documentation. For the Prometheus Go client library, for example, this requires you to set EnableOpenMetrics to true. For the Java library, follow its instructions on setting the proper header format.
  • Obtain the trace ID for the current request and include the trace ID in calls to emit metrics.
    • For histograms, use the ObserveWithExemplar method to emit the trace ID along with a value for the histogram. These functions are from the Go library but you can find similar functions in the other libraries.
    • For counters, use the AddWithExemplar method to emit the trace ID along with a counter increment.
  • Verify that metrics are being generated with exemplars by running the following command in a shell: curl -H "Accept: application/openmetrics-text" http://<your application>/metrics | grep -i "traceid".

Configure Grafana Agent for exemplars

Caution

Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Your intermediate agent, Grafana Agent stores exemplars by default. You can also use hosted Prometheus as your intermediate agent.

To configure Grafana Agent to send traces to Grafana Cloud:

  1. Confirm that exemplars are being scraped by the Agent by verifying that the prometheus_remote_storage_exemplars_total metric is a non-zero value.
  2. Add the option send_exemplars: true under the remote_write configuration block to forward those exemplars from the Agent to Grafana Cloud.

View exemplars

Explore visualizes exemplar traces as highlighted stars alongside metrics data.

For more examples, refer to Introduction to exemplars.