About exemplars in GEM
An exemplar is a specific trace representative of a repeated pattern of data in a given time interval. It helps you identify higher cardinality metadata from specific events within time series data. To learn more about exemplars and how they can help you isolate and troubleshoot problems with your systems, see Introduction to exemplars.
Grafana Enterprise Metrics includes the ability to store exemplars in-memory. Exemplar storage in GEM is implemented similarly to how it is in Prometheus. Exemplars are stored as a fixed size circular buffer that stores exemplars in memory for all series.
The limits_config property can be used to control the size of the circular buffer by the number of exemplars. For reference, an exemplar with just a
traceID=<jaeger-trace-id> uses roughly 100 bytes of memory via the in-memory exemplar storage. If the exemplar storage is enabled, GEM will also append the exemplars to WAL for local persistence (for WAL duration).
Related Enterprise Metrics resources
Running Prometheus-as-a-service with Grafana Enterprise Metrics
Introducing Grafana Enterprise Metrics (GEM), a simple and scalable Prometheus service that is seamless to use, simple to maintain, and supported by Grafana Labs.
How Robinhood scaled from 100M to 700M time series with Grafana Enterprise Metrics
In this GrafanaCONline session, the Robinhood team tells how GME (GameStop) led to GEM (Grafana Enterprise Metrics).
Benchmarking Grafana Enterprise Metrics for horizontally scaling Prometheus up to 500 million active series
We stress-tested GEM to show how it horizontally scaled. One takeaway: Hardware usage scales linearly up to 500 million active series.