Grafana dashboards: tips for optimizing query performance
Even with a powerful database or visualization layer, performance can suffer if queries aren’t optimized or system settings aren’t tuned.
The new Mimir Query Engine in Grafana Cloud improves query efficiency, but there are still best practices you can follow to keep dashboards fast and responsive—whether your data source is hosted in Grafana Cloud or running on-premises.
If your Grafana Cloud dashboards are slow when querying, use the steps outlined in this guide to speed them up and achieve fast load times.
1. PromQL query optimization
Efficient queries are the foundation of good performance. In general, avoid fetching more series or datapoints than you need. The following tips will help you simplify your queries for faster performance and better results.
Use selectors
Filter by labels to limit the series you fetch.
Example:
http_requests_total{job="api", status="500"}
This is better than querying the entire http_requests_total metric.
Avoid large range windows
Large windows force Grafana Cloud Metrics, which is powered by Mimir, to process massive amounts of data. Use shorter windows that meet your needs (e.g., [1h] instead of [24h]), or use recording rules for long-range aggregations (more on that later).
Consolidate queries
If you’re running multiple similar queries, it’s often faster to combine them into one.
Instead of:
rate(http_requests_total{status="200"}[5m])
rate(http_requests_total{status="500"}[5m])
Do:
sum by (status) (rate(http_requests_total{status=~”200|500”[5m]))
Avoid high-cardinality labels
High cardinality can lead to all sorts of challenges. For example, queries grouped by labels with many unique values are expensive (e.g. user_id).
Bad:
sum by (user_id) (rate(requests_total[5m]))
Better:
sum by (user_role) (rate(requests_total[5m]))
Minimize use of regular expressions
Regex filters can slow down queries depending on their complexity. If you rely heavily on regex, consider refining your labeling strategy upstream (e.g. with prometheus.relabel in Grafana Alloy).
Avoid excessive label_replace / label_join
Using label_replace or label_join frequently can degrade performance. It’s better to define the needed labels upstream (at collection or ingestion time) if possible.
2. Dashboard best practices
Dashboards are a great way to consolidate data and share intuitive visuals that can help you track performance and resolve issues faster. Grafana offers seemingly endless options for customizing dashboards to fit your needs, but it's also important to be economical with your setup if you want to get the best, fastest results.
Share query results across panels
When multiple panels visualize the same or similar data, avoid re-running identical queries. Instead, reuse one panel’s result as input for others. This reduces redundant queries to Mimir and speeds up dashboard load times.
Consolidate panels where possible
Having multiple panels each querying different aspects of the same metric is a common anti-pattern that causes poor overall dashboard performance.
Anti-pattern:
- One panel for Server A Status
- One panel for Server B Status
Better Approach:
- Use a single panel with a single query:
status_total{job="servers"}
Adjust dashboard refresh frequency
There are times when it’s beneficial to configure dashboards to auto refresh and fetch the latest results—for example, if you have a dashboard on a TV that is constantly viewed by people in a network or security operations center. However, if the queries in the dashboard are complex or take a long time to run though, this can lead to poor performance.
Set the auto refresh interval to a longer period of time or disable auto refresh entirely if it’s impacting your dashboard’s performance.
3. Grafana Cloud configuration options
Grafana Cloud offers a number of system-wide settings and options that can improve the end user experience on the platform. These options may be faster to implement than making dashboard or query-level changes, but they should be used in conjunction with query and dashboard changes for best results.
Use recording rules
Recording rules allow you to precompute expensive expressions and save the output as a new time series metric. Because the rule is continuously evaluated (rather than recalculated on each dashboard load), dashboards can fetch results much faster.
Pro tip: Grafana has two kinds of recording rules: our newer Grafana-managed recording rules and a more classic datasource-managed recording rules. We recommend using the Grafana-managed recording rules when possible as they support non-Prometheus format data sources such as MySQL.
Enable incremental querying
Grafana supports incremental querying (beta) for Prometheus data sources. With it enabled, subsequent queries only fetch new data points rather than re-fetching the entire time range.
Enable data source caching
Grafana Cloud supports query caching for data sources, which enables better query performance. If a query is run repeatedly, the query result will be stored in the Grafana cache rather than being fetched from the data source.
This functionality is already included out of the box in Grafana Cloud Metrics, Logs, and Traces, so you’re already getting the benefits of caching without any additional configuration there. However, if you have an on-premises Prometheus data source connected to Grafana Cloud or other databases such as MySQL, data source caching can be beneficial.
What’s next?
Improving the performance of your queries and dashboards is crucial for enhancing the end-user experience in Grafana. After optimizing performance, your next step should be to refine the overall dashboard design, which includes the layout, color scheme, and visualization formatting.
For more information on mastering these design aspects, you can watch the Grafana dashboard design webinar.
Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!
