Prometheus native histograms in Grafana Cloud: More precise, easier to use, and better compatibility
Histograms help you monitor and visualize the distribution of values for key metrics, such as response times or request sizes of a service. They’re frequently used to gain insights into data patterns, anomalies, and trends, making them an important tool for observability.
We want to make sure you get the most from your visualizations, which is why we are excited to announce the public preview of Prometheus native histograms in Grafana Cloud. (They’re also available as an experimental feature in Grafana Mimir OSS and Grafana Enterprise Metrics.) Compared to the classic Prometheus histograms, Prometheus native histograms—native histograms, for short—offer higher resolution and precision. They’re also easier to instrument and you can use them to combine and manipulate histograms in queries and in Grafana.
Plus, native histograms provide compatibility with other formats, such as OpenTelemetry exponential histograms and Datadog distributions. There’s a lot to dig into, so let’s walk through what you need to know to get started with Prometheus native histograms in Grafana Cloud.
Prometheus histograms: native vs. classic
Introduced in Prometheus in 2022, native histograms are a data type intended to eventually replace classic histograms.To illustrate the difference in how they’re visualized in Grafana, the image below shows native histograms in action, with a view of the heatmap and histogram of the response times for a real application.
Compare that to the same application viewed with classic histograms:
Use cases
Native histograms offer two modes: a compatibility mode called “native histograms with custom buckets” (see the Future development section for more information) and a mode called “native histograms with standard exponential buckets,” which is the main focus of this blog post.
You can use native histograms with standard exponential buckets for the following use cases:
- You’re unsure about what the distribution of your values are and don’t want to spend time on choosing buckets.
- You want to catch outliers and long-tail distribution values.
- You want to add and subtract different histogram metrics.
- You want to avoid consistency problems arising from the fact that classic histograms are represented by multiple independent time series.
However, there are some trade-offs when using native histograms, compared to classic histograms:
- Comparing standard exponential buckets with some threshold (e.g., in an alert) includes an approximation, the same way it would if the threshold doesn’t align perfectly with a custom bucket boundary.
- When replacing existing classic histograms, you need to do a migration, which includes changing the instrumentation and how you query the data.
When instrumenting for native histograms with standard exponential buckets, the resulting time series looks like this:
request_duration_seconds{}Since all information is in a single series, the way to query the data is different than querying classic histograms. To get the rate of change of the overall count of the histogram, you would write this PromQL expression:
histogram_count(sum(rate(request_duration_seconds[$__rate_interval])))And to get the 90% quantile, you would write:
histogram_quantile(0.9, sum(rate(request_duration_seconds[$__rate_interval])))For more information about the syntax and differences from classic histograms, see our documentation on visualizing native histograms. You can also migrate from classic histograms to native histograms.
Classic histograms comparison
For comparison, let’s go back to the classic histogram visualization we showed before:
We can see that the system handled around 1,000 responses between 0 and 0.005 seconds (or 5ms). In other words, the bucket count for the 0 to 0.005 bucket was 1,000. With classic histograms, the bucket boundaries must be defined during the design of the application, so the boundaries might be customized depending on the use case. In this example, the custom bucket boundary definitions resulted in these time series:
request_duration_seconds_count{}
request_duration_seconds_sum{}
request_duration_seconds_bucket{le="0.005"}
request_duration_seconds_bucket{le="0.01"}
...These separate time series mean that to get the rate of change of the overall count of the histogram, you could write this PromQL expression:
sum(rate(request_duration_seconds_count[$__rate_interval]))And to get the 90% quantile, you must write:
histogram_quantile(0.9, sum by (le) (rate(request_duration_seconds_bucket[$__rate_interval])))Getting started with native histograms in Grafana Cloud
There are several ways to try out native histograms in Grafana Cloud.
If you just want to send a few native histograms to Grafana Cloud without many dependencies, you can post some samples in JSON, as shown in the following section. Alternatively, if you are familiar with OpenTelemetry or Prometheus, you can skip ahead to the corresponding sections.
Post JSON
It is possible to use any HTTP testing tool—curl, for example—to send OpenTelemetry exponential histograms, which are compatible with native histograms with standard exponential buckets, to Grafana Cloud. This example uses a script that assumes you are using Unix/Linux with Bash and curl installed.
To send samples, you’ll need some information specific to your Grafana Cloud stack:
- Sign into your Grafana instance. (If you don’t already have one, you can sign up for a forever-free account today.)
- Go to the Home page in the left-side menu.
- In the middle of that page, there’s an Account section. Click Stacks on the right. This takes you to the Grafana Cloud portal.
- In the Overview section, select or click Launch to open a Grafana Cloud stack.
- You can also get to this point by directly logging into grafana.com with your account.
- With a stack selected, click Configure from the OpenTelemetry tile.
- Note the OTLP Endpoint information: URL, instance ID (just numbers), and API token.
- Before running the provided script, set some environment variables to the following values:
export URL="https://..."
export INSTANCEID="..."
export APITOKEN="..."The script creates the metric called “blog_request_duration_seconds”. On the off chance that you already have a metric with this name, you can customize the first word by setting PREFIX.
export PREFIX="blog"Then, copy the following script into a file called “generate.sh”:
#!/usr/bin/env bash
if [[ "${INSTANCEID}" == "" ]]; then
echo "INSTANCEID for you Grafana Cloud instance is not set"
exit 1
fi
if [[ "${APITOKEN}" == "" ]]; then
echo "APITOKEN is not set"
exit 1
fi
if [[ "${URL}" == "" ]]; then
echo "URL for the OTLP endpoint is not set"
exit 1
fi
PREFIX=${PREFIX:-blog}
# Default to 10 minutes
TIMETRAVEL=${TIMETRAVEL:-600}
# Emulated scrape interval 15s
SCRAPE=15
GAUSS=("2" "3" "4" "5" "7" "9" "12" "15" "19" "24" "31" "38" "47" "57" "70" "85" "103" "123" "147" "174" "205" "241" "281" "327" "377" "433" "494" "561" "634" "712" "795" "884" "977" "1074" "1174" "1277" "1381" "1486" "1590" "1692" "1790" "1885" "1973" "2055" "2129" "2193" "2247" "2290" "2321" "2340" "2346" "2340" "2321" "2290" "2247" "2193" "2129" "2055" "1973" "1885" "1790" "1692" "1590" "1486" "1381" "1277" "1174" "1074" "977" "884" "795" "712" "634" "561" "494" "433" "377" "327" "281" "241" "205" "174" "147" "123" "103" "85" "70" "57" "47" "38" "31" "24" "19" "15" "12" "9" "7" "5" "4" "3")
steps=$((TIMETRAVEL/SCRAPE))
mins=$((TIMETRAVEL/60))
echo "Writing ${steps} histograms for the past ${mins} minutes at ${SCRAPE}s intervals"
timestampS=$(date +%s)
timestampNanoS=$(date +%N)
timestampS=$((timestampS-TIMETRAVEL))
values=("${GAUSS[@]}")
for i in $(seq 1 ${steps}) ; do
echo "Step: ${i} of ${steps}"
bucketCounts=""
count=0
first="true"
for g in "${!GAUSS[@]}" ; do
inc=$(($RANDOM%${GAUSS[$g]}))
values[$g]=$((${values[$g]}+inc))
count=$((count+${values[$g]}))
if [[ "${first}" == "true" ]]; then
first="false"
else
bucketCounts+=","
fi
bucketCounts+="${values[$g]}"
done
# echo "BucketCounts: ${bucketCounts}"
# echo "Count: ${count}"
curl -f -X POST -H "Authorization: Bearer ${INSTANCEID}:${APITOKEN}" -H "Content-Type: application/json" -d @- "${URL}/v1/metrics" <<EOF
{
"resourceMetrics": [
{
"resource": {
"attributes": [
{
"key": "service.name",
"value": {
"stringValue": "grafana-blog"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "1"
}
}
]
},
"scopeMetrics": [
{
"scope": {
"name": "grafana.blog",
"version": "1.0.0",
"attributes": [
{
"key": "my.scope.attribute",
"value": {
"stringValue": "some scope attribute"
}
}
]
},
"metrics": [
{
"name": "${PREFIX}.request.duration",
"unit": "s",
"description": "I am an Exponential Histogram",
"exponentialHistogram": {
"aggregationTemporality": 2,
"dataPoints": [
{
"startTimeUnixNano": "${timestampS}${timestampNanoS}",
"timeUnixNano": "${timestampS}${timestampNanoS}",
"count": ${count},
"sum": 10,
"scale": 3,
"zeroCount": 0,
"positive": {
"offset": -50,
"bucketCounts": [${bucketCounts}]
},
"min": 0,
"max": 5,
"zeroThreshold": 0,
"attributes": [
{
"key": "my.exponential.histogram.attr",
"value": {
"stringValue": "some value"
}
}
]
}
]
}
}
]
}
]
}
]
}
EOF
if [[ $? -ne 0 ]]; then
echo "Failed to send data"
exit 1
fi
timestampS=$((timestampS+SCRAPE))
sleep 1
doneMake the file executable with the following command:
chmod +x generate.shRun the script to push a few data points into Grafana Cloud:
./generate.shIf the script produces errors, check that you have Bash and curl installed and that you’ve correctly set the URL, instance ID, and API token.
OpenTelemetry
If you are already using OpenTelemetry exponential histograms in your application, you can start sending OpenTelemetry metrics to Grafana Cloud with no further configuration required. For more details on how to set up sending OpenTelemetry metrics, follow the instructions in our documentation about sending data to the Grafana Cloud OLTP endpoint.
If you have not instrumented your application with exponential histograms but have instrumented with classic histograms, you can modify the type of histogram that is exposed by using a “ view” to use exponential histograms. You can also add new exponential histograms in a similar way. For more information, see the documentation.
For example, in the Go language, here’s how to add a view that turns a histogram called “my_histogram” into an exponential histogram:
v := sdkmetric.NewView(
sdkmetric.Instrument{
Name: "my_histogram",
Kind: sdkmetric.InstrumentKindHistogram,
},
sdkmetric.Stream{
Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{
MaxSize: 160,
NoMinMax: true,
MaxScale: 20
},
}
)Prometheus
If you are already using native histograms with standard exponential buckets in your application, then there are some steps to take to scrape and send native histograms to Grafana Cloud. If you are scraping with Prometheus, for example, you have to enable the native histograms feature and instruct remote write to send native histograms. For more information, see our docs on scraping and sending native histograms with Prometheus.
If you haven’t instrumented your application with native histograms with standard exponential buckets, add a new histogram in the instrumentation or find an existing one. To use native histograms with standard exponential buckets, add the options to enable the functionality in the instrumentation. For more information, see our docs on instrumenting application with Prometheus client libraries.
This Go language example shows how to create and register a new native histogram with standard exponential buckets called “request_duration_seconds”:
histogram := prometheus.NewHistogram(
prometheus.HistogramOpts{
Name: "request_duration_seconds",
Help: "Histogram of request latency in seconds",
NativeHistogramBucketFactor: 1.1,
NativeHistogramMaxBucketNumber: 100,
NativeHistogramMinResetDuration: 1*time.Hour,
})Compatibility
Native histograms are compatible with most features in Prometheus, Grafana Mimir, and Grafana, including support for out-of-order samples, remote write, remote read, and so on.
Native histograms are a new data type in Prometheus. Therefore, some PromQL operations are not available for them. Attempting such an operation results in a warning from the query engine. For more information, see the Prometheus documentation on querying.
Native histograms are also fully compatible with OpenTelemetry. You can convert native histograms with standard exponential schema and OpenTelemetry exponential histograms to one another. There are only minor differences between the two formats. For example, OpenTelemetry exponential histograms can store the minimum and maximum value of the observations, whereas native histograms don’t. Instead, in Prometheus, these are estimated by calculating the 0 and 100th percentiles.
Prometheus, Mimir, and Grafana Cloud support an OTLP endpoint that automatically converts OpenTelemetry exponential histograms into native histograms.
The Prometheus remote write exporter in the OpenTelemetry Collector defaults to exporting OpenTelemetry exponential histograms into native histograms with standard exponential buckets. This means that any data model that can map histograms to OpenTelemetry exponential histograms works.
Additionally, the OpenTelemetry Collector includes exporters and receivers to get data into and out of the OpenTelemetry format. This allows for compatibility with a variety of different metric sources and backends. For an example, check out the Datadog receiver that we developed, which supports the conversion of Datadog distribution metrics into exponential histograms, which you can then ingest as native histograms.
Grafana Alloy supports Prometheus native histograms as well as OpenTelemetry exponential histograms when it is used as an OpenTelemetry Collector.
Pricing
Effective May 1, 2025, native histograms are billed differently from regular time series, which will affect invoices due in June. A native histogram series is counted as multiple active series equal to the number of active buckets multiplied by 0.25.
For example, let’s say you have a classic histogram with 10 buckets. That would be billed as 12 active series because each bucket is its own time series—even if the count in the bucket is zero—plus there’s the overall count and sum as separate time series.
For native histograms the empty or “count equals zero” buckets are not billed for, and neither is the sum and overall count. For example, using 40 buckets in a native histogram would yield 40 * 0.25, or 10 active billable series. This means that for the same cost, you get better resolution and, therefore, more value for your money.
For more information, refer to our docs on understanding your Grafana Cloud Metrics invoice.
Future development
We are currently working on a new mode for native histograms called “native histograms with custom buckets.” With this mode, you can store classic histograms directly in native histograms and retain the exact bucket boundaries that you prescribe.
Here are some benefits of native histograms with custom buckets:
- Both native histograms with standard exponential buckets and custom buckets use the same syntax in queries.
- Both can avoid consistency issues arising from classic histograms being stored in multiple independent time series.
- You can convert OpenTelemetry (explicit) histograms into native histograms with custom buckets, further improving compatibility with OpenTelemetry.
- To use native histograms with custom buckets, you don’t need to change the instrumentation. Therefore, this feature is available for legacy applications as well.
The only downside might be the need to migrate the queries and dashboards that use the existing classic histograms.
Due to these benefits, it is expected that native histograms fully replace classic histograms in the future.
Questions or feedback?
Don’t hesitate to contact us via a support ticket or in our community channels, such as the #mimir channel in the Grafana Labs Community Slack workspace.
Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!
