This is documentation for the next version of Grafana Mimir documentation. For the latest stable release, go to the latest version.

Documentation Grafana Mimir Send data Native histograms Native histograms standard buckets

Send native histograms with exponential buckets to Grafana Mimir

Prometheus native histograms with exponential buckets, also known officially as native histograms with standard schemas, is a sample type in the Prometheus ecosystem that makes it possible to produce, store, and query a high-resolution histogram of observations.

Native histograms with exponential buckets are different from classic Prometheus histograms in a number of ways:

• Exponential bucket boundaries are calculated by a formula that depends on the scale, or resolution, of the native histogram and are not user-defined. The calculation produces exponentially increasing bucket boundaries. For details, refer to Exponential bucket boundary calculation.
• Exponential bucket boundaries might change dynamically if the observations result in too many buckets. For details, refer to Limit the number of buckets.
• Exponential bucket counters only count observations inside the bucket boundaries, whereas the classic histogram buckets only have an upper bound called le and count all observations in the bucket and all lower buckets cumulatively.
• An instance of a native histogram metric only requires a single time series, because the buckets, sum of observations, and the count of observations are stored in a single sample type called native histogram rather than in separate time series using the float sample type. Thus, there are no <metric>_bucket, <metric>_sum, and <metric>_count series. There is only a <metric> time series.
• Querying native histograms via the Prometheus query language (PromQL) uses a different syntax. For details, refer to Visualize native histograms and functions.

For an introduction to native histograms, watch the Native Histograms in Prometheus presentation.

This document provides a practical implementation guide for using native histograms with exponential schemas in Grafana Mimir. For comprehensive guidance, refer to the official specification in the Prometheus documentation.

Advantages and disadvantages

There are advantages and disadvantages of using native histograms with exponential buckets compared to the classic Prometheus histograms. For more information, refer to the Prometheus Native Histograms in Production video.

Advantages

• Simpler instrumentation: You don’t need to think about bucket boundaries because they are created automatically.
• Better resolution in practice: Custom bucket layouts are usually not high resolution.
• Increased compatibility: Native histograms with exponential buckets are compatible with each other. They have an automatic layout, which makes them easy to combine.

Disadvantages

• Observations might be distributed in a way that is not a good fit for the exponential bucket schema, such as sound pressure measured in decibels, which are already logarithmic.
• If converting from an externally represented histogram with specific bucket boundaries, there is generally no precise match with the exponential bucket boundaries of the native histogram, in which case you need to use interpolation.
• There is no way to set an arbitrary bucket boundary, such as one that is particularly interesting for an SLO definition. Generally, ratios of observations above or below a given threshold have to be estimated by interpolation, rather than being precise in the case for a classic histogram with a configured bucket boundary at a given threshold.

The preceding problems are mitigated by high resolution, which native histograms with exponential buckets can provide at a much lower resource cost compared to classic histograms.

Instrument application with Prometheus client libraries

The following examples have some reasonable defaults to define a new native histogram with exponential buckets metric. The examples use the Go client library version 1.16 and the Java client library 1.0.

Gojava

Copy

Go Copy

histogram := prometheus.NewHistogram(
   prometheus.HistogramOpts{
      Name: "request_latency_seconds",
      Help: "Histogram of request latency in seconds",
      NativeHistogramBucketFactor: 1.1,
      NativeHistogramMaxBucketNumber: 100,
      NativeHistogramMinResetDuration: 1*time.Hour,
})

java Copy

static final Histogram requestLatency = Histogram.build()
     .name("requests_latency_seconds")
     .help("Histogram of request latency in seconds")
     .nativeOnly()
     .nativeInitialSchema(3)
     .nativeMaxNumberOfBuckets(100)
     .nativeResetDuration(1, TimeUnit.HOURS)
     .register();

In Go, the NativeHistogramBucketFactor option sets an upper limit of the relative growth from one exponential bucket to the next. The value 1.1 means that an exponential bucket is at most 10% wider than the next smaller exponential bucket. The currently supported values range from 1.0027 or 0.27% up to 65536 or 655%. For a more detailed explanation, refer to Exponential bucket boundary calculation.

Some of the resulting exponential buckets for factor 1.1 rounded to two decimal places are:

…, (0.84, 0.92], (0.92, 1], (1, 1.09], (1.09, 1.19], (1.19, 1.30], …

…, (76.1, 83], (83, 91], (91, 99], …

…, (512, 558], (558, 608], (608, 663], …

In Java, .nativeInitialSchema using a schema value of 3 results in the same exponential bucket boundaries. For more information about the schema supported in Java, consult the documentation for nativeInitialSchema.

The value of NativeHistogramMaxBucketNumber/nativeMaxNumberOfBuckets limits the number of exponential buckets produced by the observations. This can be especially useful if the receiver side is limiting the number of buckets that can be sent. For more information about the bucket limit, refer to Limit the number of buckets.

The duration in NativeHistogramMinResetDuration/nativeResetDuration prohibits automatic counter resets inside that period. Counter resets are related to the bucket limit. For more information, refer to Limit the number of buckets.

Scrape and send native histograms with Prometheus

Use Prometheus version 3.00 or later.

1. To enable scraping native histograms from the application, you need to enable the native histograms feature via a feature flag on the command line:

   Bash Copy

   prometheus --enable-feature=native-histograms

   This flag makes Prometheus detect and scrape native histograms with exponential buckets over the PrometheusProto scrape protocol and ignore the classic histogram version of metrics that have native histograms defined as well. Classic histograms without native histogram definitions are not affected.

   Note

   Native histograms don’t have a textual presentation on the application’s /metrics endpoint. Therefore, Prometheus negotiates a Protobuf protocol transfer in these cases.

2. To keep scraping the classic histogram version of native histogram metrics, you need to set always_scrape_classic_histograms to true in your scrape jobs.

   Note

   In Prometheus versions 3.0 and earlier, the always_scrape_classic_histograms setting is called scrape_classic_histograms. Use the setting name that corresponds to the version of Prometheus you’re using.

   For example, to get both classic and native histograms, use the following configuration:

   YAML Copy

   scrape_configs:
     - job_name: myapp
       always_scrape_classic_histograms: true

3. To send native histograms to a Prometheus remote-write compatible receiver, for example Grafana Cloud Metrics or Grafana Mimir, set send_native_histograms to true in the remote-write configuration. For example:

   YAML Copy

   remote_write:
     - url: http://.../api/prom/push
       send_native_histograms: true

Scrape and send native histograms with Grafana Alloy

Use Grafana Alloy version 1.11 or later.

1. To scrape native histograms, set the scrape_native_histograms argument in the prometheus.scrape component to true.

   Copy

   scrape_native_histograms = true

   Warning

   In Grafana Alloy versions 1.10 and earlier, setting scrape_native_histograms to true isn’t required. For more information, refer to Release notes for Grafana Alloy.

   Note

   For now, native histograms are only available through the Prometheus Protobuf exposition format. Make sure to either not override the argument scrape_protocols or that the first item is PrometheusProto.

2. To scrape classic histograms in addition to native histograms, set scrape_classic_histograms to true.

   Copy

   scrape_native_histograms = true
   scrape_classic_histograms = true

   For more information, refer to prometheus.scrape in the Grafana Alloy documentation.

3. To send native histograms to a Prometheus remote-write compatible receiver, such as Grafana Cloud Metrics or Grafana Mimir, set the send_native_histograms argument to true in the prometheus.remote_write component. For example:

   Copy

   prometheus.remote_write "mimir" {
     endpoint {
       url = "http://.../api/prom/push"
       send_native_histograms = true
     }
   }

Migrate from classic histograms

To ease the migration process, you can keep the custom bucket definition of a classic histogram and add native histogram exponential buckets at the same time.

1. Add the native histogram exponential buckets definition to an existing histogram in the instrumentation.

2. If the existing histogram doesn’t have buckets defined, add the default buckets to keep the classic histogram.

   The following code examples have both classic and native histograms defined for the same metric:

   Gojava

   Copy

   Go Copy

   histogram := prometheus.NewHistogram(
      prometheus.HistogramOpts{
          Name: "request_latency_seconds",
          Help: "Histogram of request latency in seconds",
          Buckets: prometheus.DefBuckets,  // If buckets weren't already defined.
          NativeHistogramBucketFactor: 1.1,
          NativeHistogramMaxBucketNumber: 100,
          NativeHistogramMinResetDuration: 1*time.Hour,
   })

   java Copy

   static final Histogram requestLatency = Histogram.build()
      .name("requests_latency_seconds")
      .help("Histogram of request latency in seconds")
      .classicUpperBounds(Histogram.Builder.DEFAULT_CLASSIC_UPPER_BOUNDS)  // If upper bounds weren't already defined.
      .nativeInitialSchema(3)
      .nativeMaxNumberOfBuckets(100)
      .nativeResetDuration(1, TimeUnit.HOURS)
      .register();

3. Let Prometheus or Grafana Alloy scrape both classic and native histograms with exponential buckets for metrics that have both defined.

4. Send native histograms to remote write along with the existing classic histograms.

5. Modify dashboards to use the native histograms metrics. Refer to Visualize native histograms for more information.

   Use one of the following strategies to update dashboards.

   • (Recommended) Add new dashboards with the new native histograms queries. This solution requires looking at different dashboards for data before and after the migration, until data before the migration is removed due to passing its retention time. You can publish the new dashboard when sufficient time has passed to serve users with the new data.

   • Add a dashboard variable to your dashboard to enable switching between classic histograms and native histograms. There isn’t support for selectively enabling and disabling queries in Grafana (issue 79848). As a workaround, add the dashboard variable latency_metrics, for example, and assign it a value of either -1 or 1. Then, add the following two queries to the panel:

     Copy

     (<classic_query>) and on() (vector($latency_metrics) == 1)

     Copy

     (<native_query>) and on() (vector($latency_metrics) == -1)

     Where classic_query is the original query and native_query is the same query using native histograms query syntax placed inside parentheses. Grafana Mimir dashboards use this technique. For an example, refer to the Overview dashboard in the Grafana Mimir repository.

     This solution allows users to switch between classic histograms and native histograms without going to a different dashboard.

   • Replace the existing classic histograms queries with modified queries. For example, replace:

     Copy

     <classic_query>

     with

     Copy

     <native_query> or <classic_query>

     Where classic_query is the original query and native_query is the same query using native histograms query syntax.

     Warning

     Using the PromQL operator or can lead to unexpected results. For example, if a query uses a range of seven days, such as sum(rate(http_request_duration_seconds[7d])), then this query returns a value as soon as there are two native histograms samples present before the end time specified in the query. In this case, the seven day rate is calculated from a couple of minutes, rather than seven days, worth of data. This results in an inaccuracy in the graph around the time you started scraping native histograms.

6. Start adding new recording rules and alerts to use native histograms. Do not remove the existing recording rules and alerts at this time.

7. It is important to keep scraping both classic and native histograms for at least the period of the longest range in your recording rules and alerts, plus one day. This is the minimum amount of time, but it’s recommended to keep scraping both sample types until the new rules and alerts can be verified.

   For example, if you have an alert that calculates the rate of requests, such as sum(rate(http_request_duration_seconds[7d])), this query looks at the data from the last seven days plus the Prometheus lookback period. When you start sending native histograms, the data isn’t there for the entire seven days, and therefore, the results might be unreliable for alerting.

8. After configuring native histogram collection, choose one of the following ways to stop collecting classic histograms.

   • Remove the custom bucket definition, Buckets/classicUpperBounds, from the instrumentation. In Java, also use the nativeOnly() option. Refer to the examples in Instrument application with Prometheus client libraries.
   • Drop the classic histogram series with Prometheus relabeling or Grafana Alloy prometheus.relabel at the time of scraping.
   • Stop scraping the classic histogram version of metrics. This option applies to all metrics of a scrape target.

9. Clean up recording rules and alerts by deleting the classic histogram version of the rule or alert.

Exponential bucket boundary calculation

This section assumes that you are familiar with basic algebra. Native histogram exponential bucket boundaries are calculated from an exponential formula with a base of 2.

Native histogram with exponential buckets samples have three different kind of buckets, for any observed value the value is counted towards one kind of bucket.

• A zero bucket, which contains the count of observations whose absolute value is smaller or equal to the zero threshold.

• Positive buckets, which contain the count of observations with a positive value that is greater than the lower bound and less or equal to the upper bound of a bucket.

where the index can be a positive or negative integer resulting in boundaries above 1 and fractions below 1. The schema either directly specified out of [-4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8] at instrumentation time or it is the largest number chosen from the list in such way that

for example for factor 1.1:

Table of schema to factor:

• Negative buckets, which contain the count of observations with a negative value that is smaller than the upper bound and greater than or equal to the lower bound of a bucket.

where the schema is chosen as above.

Limit the number of buckets

Emitting and storing a potentially unlimited number of buckets isn’t practical, as higher resolution increases the storage costs with diminishing returns.

You can limit the number of buckets in Grafana Mimir or Grafana Cloud, Prometheus, or in application instrumentation.

Limit the number of buckets in Grafana Mimir or Grafana Cloud

To limit the number of buckets in all native histograms ingested by Grafana Mimir or Grafana Cloud, set the tenant limit max_native_histogram_buckets in Grafana Mimir or submit a support request for Grafana Cloud.

Native histograms that have a higher bucket count than the limit are converted to a native histogram with a lower resolution by merging buckets to reduce the number of buckets. In some rare cases, if the buckets are too widely spread out, merging them isn’t possible and the native histogram is rejected.

Limit the number of buckets in Prometheus

To limit the number of buckets in native histograms scraped by a scrape configuration, set the parameter native_histogram_bucket_limit in the scrape configuration.

Native histograms that have a higher bucket count than the limit are converted to a native histogram with a lower resolution by merging buckets to reduce the number of buckets. In some rare cases, if the buckets are too widely spread out, merging them isn’t possible and the native histogram is rejected.

Limit the number of buckets in application instrumentation

The instrumentation libraries of Prometheus have automation to keep the number of exponential buckets down, provided that the maximum bucket number option is used, such as NativeHistogramMaxBucketNumber in Go.

After the set maximum is exceeded, the following strategy is enacted:

1. First, if the last reset, or the creation, of the histogram is at least the minimum reset duration, then the whole histogram is reset to its initial state, including classic buckets. This only works if the minimum reset duration was set (NativeHistogramMinResetDuration in Go).

2. If less time has passed, or if the minimum reset duration is zero, no reset is performed. Instead, the zero threshold is increased sufficiently to reduce the number of exponential buckets to or below the maximum bucket number, but not to more than the maximum zero threshold (NativeHistogramMaxZeroThreshold in Go). Thus, if the threshold is at or above the maximum threshold already, nothing happens at this step.

3. After that, if the number of exponential buckets still exceeds the maximum bucket number, the resolution of the histogram is reduced by doubling the width of all the exponential buckets up to a growth factor between one bucket to the next of 2^(2^4) = 65536. Refer to Exponential bucket boundary calculation).

4. Any increased zero threshold or reduced resolution is reset back to their original values once the minimum reset duration has passed since the last reset or the creation of the histogram.