Prometheus data source on Grafana Labs

Configure the Prometheus data source

Tue, 23 Jun 2026 13:06:40 +0000

Configure the Prometheus data source

This document provides instructions for configuring the Prometheus data source and explains the available configuration options. Grafana includes built-in support for Prometheus, so you don’t need to install a plugin. For general information on adding a data source to Grafana, refer to Add a data source.

Before you begin

You need the Organization administrator role to configure the data source. You can also configure it via YAML using Grafana provisioning or using Terraform.
Know which type of Prometheus-compatible database you’re connecting to (Prometheus, Mimir, Cortex, or Thanos), as the configuration options vary by type.
Have your Prometheus server URL ready. If using TLS/SSL, gather any necessary security certificates and client keys.
If using Basic authentication, have your username and password ready.

Add the data source

To add the Prometheus data source, complete the following steps:

Click Connections in the left-side menu.
Under Connections, click Add new connection.
Enter Prometheus in the search bar.
Click Prometheus data source.
Click Add new data source in the upper right.

Grafana takes you to the Settings tab where you set up your Prometheus configuration.

Configuration options

The following sections describe the configuration options available on the Settings tab. These sections follow the order in which they appear in the UI.

Name and default

Name - The data source name. Sets the name you use to refer to the data source in panels and queries. Examples: prometheus-1, prom-metrics.
Default - Toggle to select as the default data source in dashboard panels.

Connection

Prometheus server URL - The URL of your Prometheus server.
If Prometheus is running locally, use http://localhost:9090. If it’s hosted on a networked server, provide the server’s URL along with the port where Prometheus is running. Example: http://prometheus.example.orgname:9090.

Note
When running Grafana and Prometheus in separate containers, localhost refers to each container’s own network namespace. This means that localhost:9090 points to port 9090 inside the Grafana container, not on the host machine.

Use the IP address of the Prometheus container, or the hostname if you are using Docker Compose. Alternatively, you can use http://host.docker.internal:9090 to reference the host machine.

Authentication

There are three authentication options for the Prometheus data source.

Basic authentication - The most common authentication method.
- User - The username you use to connect to the data source.
- Password - The password you use to connect to the data source.
Forward OAuth identity - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source.
No authentication - Allows access to the data source without any authentication.

Additional authentication methods (Azure AD, AWS SigV4) are available depending on your deployment:

Warning
Azure AD and SigV4 authentication on the core Prometheus data source are deprecated in Grafana 13. Existing data sources using these methods are automatically migrated to dedicated plugins on startup. For new setups, use the Azure Monitor Managed Service for Prometheus or Amazon Managed Service for Prometheus plugins instead. For migration details, refer to Azure authentication (deprecated) or AWS authentication (deprecated).

Azure AD authentication - Available in Grafana Enterprise and self-managed instances where azure_auth_enabled = true is configured. On Grafana Cloud, this option requires a server-side feature flag that isn’t enabled by default — contact Grafana Support to request it be enabled for your stack. Refer to Azure authentication settings for configuration details.
AWS SigV4 authentication - Available in self-managed Grafana instances where sigv4_auth_enabled = true is configured. On Grafana Cloud, this option isn’t available by default — contact Grafana Support to request it be enabled for your stack.

Note
If Azure AD or SigV4 options don’t appear in the authentication drop-down, the required feature flag isn’t enabled on your instance. For Grafana Cloud, submit a support request. For self-managed instances, update the Grafana configuration file.

TLS settings

Note
Use TLS (Transport Layer Security) for an additional layer of security when working with Prometheus. For information on setting up TLS encryption with Prometheus refer to Securing Prometheus API and UI Endpoints Using TLS Encryption. You must add TLS settings to your Prometheus configuration file prior to setting these options in Grafana.

Add self-signed certificate - Check the box to authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file. Required for verifying self-signed TLS certificates.
- CA certificate - Add your certificate.
TLS client authentication - Check the box to enable TLS client authentication.
- Server name - Add the server name, which is used to verify the hostname on the returned certificate.
- Client certificate - The client certificate is generated from a Certificate Authority or is self-signed. Follow the instructions of the CA to download the certificate file.
- Client key - Add your client key, which can also be generated from a CA or be self-signed. The client key encrypts data between the client and server.
Skip TLS verify - Toggle on to bypass TLS certificate validation. Skipping TLS certificate validation is not recommended unless absolutely necessary or for testing purposes.

HTTP headers

Pass along additional information and metadata about the request or response.

Header - Add a custom header. This allows custom headers to be passed based on the needs of your Prometheus instance.
Value - The value of the header.

Advanced HTTP settings

Allowed cookies - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.
Timeout - The HTTP request timeout in seconds.

Alerting

Manage alerts via Alerting UI - Toggled on by default. This enables data source-managed rules in Grafana Alerting for this data source. For Mimir, it enables managing data source-managed rules and alerts. For Prometheus, it only supports viewing existing rules and alerts, which are displayed as data source-managed. Change this by setting the default_manage_alerts_ui_toggle option in the grafana.ini configuration file.
Allow as recording rules target - Toggled on by default. This allows the data source to be selected as a target destination for writing Grafana-managed recording rules. When enabled, this data source appears in the target data source list when creating or importing recording rules. Change this by setting the default_allow_recording_rules_target_alerts_ui_toggle option in the grafana.ini configuration file.

Interval behavior

Scrape interval - Sets the standard scrape and evaluation interval in Prometheus. The default is 15s. Set it to match the typical scrape and evaluation interval in your Prometheus configuration file. If you set a higher value than your Prometheus configuration, Grafana evaluates data at this interval, resulting in less data points.
Query timeout - Sets the Prometheus query timeout. The default is 60s. Without a timeout, complex or inefficient queries can run indefinitely, consuming CPU and memory resources.

Query editor

Default editor - Sets the default query editor. Options are Builder or Code. Builder mode helps you build queries using a visual interface. Code mode is geared for the experienced Prometheus user with prior expertise in PromQL. For more details on editor types, refer to Prometheus query editor. You can switch between editors in the query editor UI.
Disable metrics lookup - Toggle on to disable the metrics chooser and metric and label support in the query field’s autocomplete. This can improve performance for large Prometheus instances.

Performance

Prometheus type - Select the type of your Prometheus-compatible database. Setting this incorrectly may cause unexpected behavior when querying metrics and labels. Cortex is end-of-life; if you’re running Cortex, consider migrating to Mimir.

Capability	Prometheus	Mimir	Thanos	Cortex
Regex label matching in variable queries	No	Yes	No	Yes
Metadata API (metric type/help)	Yes (2.4+)	Yes	No	Yes
Exemplars	Yes (2.26+)	Yes	No	No
Data source-managed alert rules (read/write)	Read only	Read/Write	Read only	Read/Write
Recording rules target (write-back)	Yes	Yes	No	Yes
LBAC (Team-based access control)	No	Yes (Cloud/GEM)	No	No
Native histograms	Yes (2.40+)	Yes	No	No

Grafana uses this setting to determine which API endpoints and features to enable. For example, when set to Mimir, Grafana uses regular expression-optimized label queries that significantly improve autocomplete and variable loading performance for large metric sets.

Note
Team-based Label-Based Access Control (LBAC) for the Prometheus data source requires the backend to be Grafana Cloud Metrics (Mimir) or Grafana Enterprise Metrics (GEM). LBAC doesn’t work with external Prometheus-compatible endpoints such as Google Managed Prometheus, self-hosted Prometheus, or Thanos, even if you enable the teamHttpHeadersMimir setting. The LBAC enforcement relies on Mimir-specific HTTP headers that other backends don’t support.

Cache level - Sets the browser caching level for editor queries. Options: Low, Medium, High, or None. Higher cache settings are recommended for high-cardinality data sources.
Incremental querying (beta) - Toggle on to enable incremental querying. Instead of always requesting fresh data from the Prometheus instance, Grafana caches query results and only fetches new records. This helps reduce database and network load.
- Query overlap window - Specify a duration (for example, 10m, 120s, or 0s). The default is 10m. This buffer is added to each incremental request to account for delayed data ingestion.
Disable recording rules (beta) - Toggle on to disable recording rules. When disabled, Grafana won’t fetch and parse recording rules from Prometheus, improving dashboard performance by reducing processing overhead.

Other

Custom query parameters - Add custom parameters to the Prometheus query URL for more control over query execution. Examples: timeout, partial_response, dedup, or max_source_resolution. Join multiple parameters with &.
HTTP method - Select either the POST or GET HTTP method to query your data source. POST is recommended and selected by default, as it supports larger queries. Select GET if your network restricts POST requests.
Series limit - Maximum number of returned series. The limit applies to all resources (metrics, labels, and values) for both endpoints (series and labels). Leave empty to use the default limit (40000). Set to 0 to disable the limit — this may cause performance issues.
Use series endpoint - Toggle on to use the series endpoint (/api/v1/series) with the match[] parameter instead of the label values endpoint (/api/v1/label/<label_name>/values). The label values endpoint is generally more performant, but the series endpoint supports the POST method.

Exemplars

Support for exemplars is available only for the Prometheus data source. An exemplar is a trace that represents a specific measurement taken within a given time interval. For more information, refer to Introduction to exemplars.

Click the + sign to add exemplars.

Internal link - Toggle on to enable an internal link. This displays the data source selector, where you can choose the backend tracing data store for your exemplar data.
URL - (Visible if you disable Internal link) Defines the external link’s URL trace backend. You can interpolate the value from the field by using the ${__value.raw} macro.
Data source - (Visible when Internal link is enabled) Select the tracing data source that the exemplar links to.
URL label - Adds a custom display label to override the value of the Label name field.
Label name - The name of the field in the labels object used to obtain the traceID property.
Remove exemplar link - Click the X to remove existing links.

You can add multiple exemplar configurations.

Private data source connect

Note
Private data source connect (PDC) is only available for Grafana Cloud users.

PDC allows you to establish a private, secured connection between a Grafana Cloud instance and data sources within a private network without opening the network to inbound traffic.

Private data source connect network - Select the PDC network where your data source is available.

PDC supports both querying and writing to Prometheus-compatible data sources. This means Grafana-managed recording rules can write their results to a Prometheus or Mimir instance behind a PDC connection.

If you use PDC with SigV4 (AWS Signature Version 4 Authentication), the PDC agent must allow internet egress to sts.<region>.amazonaws.com:443.

Click Manage private data source connect networks to view your PDC configuration details.

For more information, refer to Private data source connect (PDC) and Configure PDC.

For troubleshooting PDC connectivity issues (DNS resolution, “host unreachable” errors, or parallel connection tuning), refer to PDC connectivity errors.

Save and test

After you have configured your Prometheus data source options, click Save & test at the bottom to test your data source connection.

You should see a confirmation message:

Successfully queried the Prometheus API.

You can also remove a connection by clicking Delete.

Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system, or use Terraform with the Grafana provider.

Note
After you have provisioned a data source, you cannot edit it through the UI.

Provision with YAML

For more information about provisioning, and for available configuration options, refer to Provision Grafana.

Example Prometheus data source configuration:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    jsonData:
      httpMethod: POST
      manageAlerts: true
      allowAsRecordingRulesTarget: true
      prometheusType: Prometheus
      prometheusVersion: 3.3.0
      cacheLevel: 'High'
      disableRecordingRules: false
      seriesEndpoint: false
      timeInterval: 10s
      incrementalQueryOverlapWindow: 10m
      exemplarTraceIdDestinations:
        - datasourceUid: my_jaeger_uid
          name: traceID
        - name: traceID
          url: 'http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Jaeger%22,%7B%22query%22:%22$${__value.raw}%22%7D%5D'

Provision with Terraform

You can configure the Prometheus data source using Terraform with the Grafana Terraform provider.

For more information about provisioning resources with Terraform, refer to Grafana as code using Terraform.

The following example creates a Prometheus data source with common settings:

resource "grafana_data_source" "prometheus" {
  name = "Prometheus"
  type = "prometheus"
  url  = "http://localhost:9090"

  json_data_encoded = jsonencode({
    httpMethod                    = "POST"
    manageAlerts                  = true
    prometheusType                = "Prometheus"
    prometheusVersion             = "3.3.0"
    cacheLevel                    = "High"
    disableRecordingRules         = false
    incrementalQuerying           = true
    incrementalQueryOverlapWindow = "10m"
    timeInterval                  = "15s"
    exemplarTraceIdDestinations = [{
      datasourceUid = "my_tempo_uid"
      name          = "traceID"
    }]
  })
}

The following example creates a Prometheus data source with basic authentication:

resource "grafana_data_source" "prometheus_auth" {
  name = "Prometheus (authenticated)"
  type = "prometheus"
  url  = "https://prometheus.example.com:9090"

  basic_auth_enabled  = true
  basic_auth_username = "grafana"

  json_data_encoded = jsonencode({
    httpMethod     = "POST"
    prometheusType = "Mimir"
    timeInterval   = "15s"
  })

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = var.prometheus_password
  })
}

For all available configuration options, refer to the Grafana provider data source resource documentation.

Azure authentication settings (deprecated)

Warning
Azure AD authentication on the core Prometheus data source is deprecated in Grafana 13. Data sources using this method are automatically migrated to the dedicated Azure Monitor Managed Service for Prometheus plugin on startup. For migration details, refer to Azure authentication (deprecated).

If you still need to configure Azure AD authentication on the core Prometheus data source (for example, on older Grafana versions), refer to Configure Azure Active Directory (AD) authentication.

In Grafana Enterprise or self-managed OSS, update the .ini configuration file. Refer to Configuration file location to locate your .ini file.

Add the following setting in the [auth] section:

[auth]
azure_auth_enabled = true

Note
If you are using Azure authentication, don’t enable Forward OAuth identity. Both methods use the same HTTP authorization headers, and the OAuth token will override your Azure credentials.

Troubleshooting

If you encounter issues after configuring your Prometheus data source, refer to Troubleshoot Prometheus data source issues.

Prometheus query editor

Tue, 23 Jun 2026 13:06:40 +0000

Prometheus query editor

The Prometheus query editor lets you write PromQL queries against your Prometheus-compatible data sources. It includes a visual query builder for constructing queries without writing PromQL, a code editor with autocomplete and syntax highlighting, and configurable output formats for different visualization types.

You can access the query editor from the Explore page or from any dashboard panel by clicking the panel title and selecting Edit. For general documentation on querying data sources in Grafana, refer to Query and transform data. For more information about PromQL, see Querying Prometheus.

The query editor has two modes:

Builder mode — Visual interface for constructing queries without writing PromQL directly.
Code mode — Text editor with autocomplete for writing PromQL directly.

Grafana synchronizes both modes, allowing you to switch between them. Grafana also displays a warning message if it detects an issue with the query while switching modes.

Query options

The following options are available in both Builder and Code modes. They control how the query is executed and how results are displayed.

Legend

Controls the display name for time series in the panel legend.

Auto — Displays unique labels only, hiding labels that are common across all returned series. This produces cleaner legends when many series share the same labels.
Verbose — Displays all label names for every series.
Custom — Lets you define the legend using label templates. For example, {{hostname}} is replaced with the value of the hostname label. To switch to a different legend mode, clear the input and click outside the field.

Min step

Sets the minimum interval between data points returned by the query. For example, setting this to 1h means data is returned at hourly intervals at minimum. This setting supports the $__interval and $__rate_interval macros.

Note
The time range of the query is aligned to the step size, which may adjust the actual start and end times of the returned data.

Format

Determines how query results are interpreted and displayed in a panel.

Time series — The default format. Returns data as time-indexed series suitable for graph panels.
Table — Displays data in a tabular format. Works only in a Table panel.
Heatmap — Displays histogram-type metrics in a Heatmap panel by converting cumulative histograms to regular ones and sorting the series by bucket boundary.

Type

Determines the query type.

Both — The default. Runs both a Range query and an Instant query and returns combined results.
Range — Returns a set of time series where each series includes multiple data points over the selected time range. Use this for graph visualizations (lines, bars, points, stacked).
Instant — Returns a single data point per series (the most recent value within the selected time range). Use this for stat panels, tables, or gauges. To visualize instant query results in a time series panel, add a field override with the Transform property set to Constant. For more information, refer to Time Series Transform option.

Note
Grafana adjusts the query time range to align with the dynamically calculated step interval. This ensures consistent metric visualization and supports Prometheus result caching. However, this alignment can cause minor visual differences, such as a slight gap at the graph’s right edge or a shifted start time. For example, a 15s step aligns timestamps to Unix times divisible by 15 seconds. A 1w Min step aligns the range to the start of the week (Thursday at 00:00 UTC in Prometheus).

Exemplars

Toggle on to include exemplars in the graph. Exemplars link aggregated metric data to specific trace examples, letting you jump from a spike directly to a relevant trace. For more information, see Introduction to exemplars.

Note
Exemplars are not available with Instant query types.

Builder mode

Builder mode helps you build queries using a visual interface. This option is best for users who have limited experience with PromQL.

Builder mode contains the following components:

Kick start your query — Click to view predefined operation patterns that help you quickly build queries with multiple operations:
- Rate query starters
- Histogram query starters
- Binary query starters
Click the arrow next to each to see the available options.
Explain — Toggle on to display a step-by-step explanation of all query components and operations.

Select a metric

Metric — Select a metric from the drop-down. The data source provides available metrics based on the selected time range. You can type to search and filter the list. Click the book icon to open Metrics explorer.
Label filters — Select labels and values to filter the metric. Use the + button to add filters and the x button to remove them. When a metric is selected, the data source requests available labels and their values from the server.

Add operations

Click + Operations to add operations to your query. The query editor groups operations into the following categories:

Aggregations — sum, avg, count, min, max, etc.
Range functions — rate, increase, irate, avg_over_time, etc.
Functions — abs, ceil, histogram_quantile, etc.
Binary operations — arithmetic and comparison operators.
Trigonometric functions — acos, asin, atan, etc.
Time functions — time, timestamp, day_of_week, etc.

All operations display function parameters beneath the operation header. Some operations allow you to apply specific labels (for example, by or without clauses).

If you add an operation in a way that would create an invalid query, the query editor automatically places it in the correct position to maintain a valid query structure.

Hints

The query editor can detect which operations are most appropriate for certain selected metrics. When it does, it displays a hint next to the + Operations button. Click the hint to add the suggested operation.

Metrics explorer

Click the book icon next to the metric selector to open the Metrics explorer. It displays all metrics in a paginated list showing the name, type, and description for each metric.

The following options are available under Additional Settings:

Include description in search — Search by both name and description.
Include results with no metadata — Include metrics that lack type or description metadata.
Disable text wrap — Disable text wrapping for long metric names.
Enable regex search — Filter metric names by regular expression, which uses an additional API call.

Note
The Metrics explorer (Builder mode) and Metrics browser (Code mode) are separate components. The Metrics explorer does not browse labels, but the Metrics browser can display all labels on a metric.

Code mode

Code mode is for experienced Prometheus users who prefer writing PromQL directly. For more information about PromQL, see Querying Prometheus.

Code mode provides:

Autocomplete — Suggests metrics, labels, functions, aggregations, and keywords as you type. The drop-down includes documentation for suggested items where available.
Syntax highlighting — Color-codes PromQL syntax for readability.
Metrics browser — Click the arrow next to Metrics browser to open it.

Metrics browser

The Metrics browser helps you build basic queries by exploring available metrics and labels.

Step 1 — Select a metric. The browser narrows available labels to those applicable to the metric.
Step 2 — Select one or more labels.
Step 3 — Select values for each label to tighten the query scope.
Step 4 — Choose an action:
- Use query — Insert the selector into the query editor.
- Use as rate query — Insert the selector wrapped in rate(...[$__rate_interval]).
- Validate selector — Verify the selector is valid and display the number of matching series.
- Clear — Reset your selections.

Note
If you don’t remember the exact metric name, start by selecting a few labels to filter the list and narrow down your options.

All lists in the Metrics browser include a search field. In the Values section, a single search field filters across all selected labels.

Common query patterns

The following examples show frequently used PromQL patterns. Each example includes the PromQL expression and guidance on which query options to use.

Request rate per service

Calculate the per-second rate of HTTP requests, broken down by service:

sum(rate(http_requests_total[$__rate_interval])) by (service)

Option	Setting
Legend	`{{service}}` (Custom)
Type	Range
Min step	Leave empty (uses `$__interval`)

Builder mode steps: Select http_requests_total → Add operation Range functions > Rate → Add operation Aggregations > Sum → Set by label to service.

Error rate percentage

Calculate the percentage of requests that returned 5xx errors:

sum(rate(http_requests_total{status=~"5.."}[$__rate_interval])) / sum(rate(http_requests_total[$__rate_interval])) * 100

Option	Setting
Legend	`Error rate %` (Custom)
Type	Range
Format	Time series

Histogram quantile (p95 latency)

Calculate the 95th percentile request duration from a histogram metric:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[$__rate_interval])) by (le))

Option	Setting
Legend	`p95 latency` (Custom)
Type	Range

Builder mode steps: Select http_request_duration_seconds_bucket → Add Range functions > Rate → Add Aggregations > Sum with by label le → Add Functions > Histogram quantile with value 0.95.

Aggregation by label (CPU usage per instance)

Calculate average CPU usage percentage, grouped by instance:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval])) * 100)

Option	Setting
Legend	`{{instance}}` (Custom)
Type	Range
Min step	`15s` (match your scrape interval)

Multi-query expressions (available memory percentage)

Use multiple queries and expressions to calculate derived values. In the query editor, add multiple queries (A, B) and a math expression (C):

Query A — Total memory:

node_memory_MemTotal_bytes

Query B — Available memory:

node_memory_MemAvailable_bytes

Expression C — Percentage available (click + Expression → Math):

$B / $A * 100

Set queries A and B to Type: Instant and hide them from visualization (click the eye icon). Display only expression C.

Alert-compatible query (target down)

A simple alert query that fires when a scrape target is down:

up{job="my-service"} == 0

Option	Setting
Type	Both
Format	Time series

Note
Alert queries don’t support template variables ($variable). Use fixed label values when writing queries intended for alert rules.

Use the query inspector

The query inspector helps you debug queries that return unexpected results or no data. To open it:

Click Query inspector in the panel edit view (below the query editor).
Review the following tabs:
- Query — Shows the exact request sent to Prometheus, including the evaluated PromQL expression, time range, and step interval. Use this to verify that template variables resolved correctly.
- Data — Shows the raw response data. If empty, your query matched no series.
- Stats — Shows request timing and response size.

Common debugging steps:

No data returned — Check the Query tab to see if template variables resolved to empty values. Try the query directly in the Prometheus built-in expression browser (/graph) to rule out Grafana-specific issues.
Fewer series than expected — Check that your label filters aren’t too restrictive. Remove filters one at a time to isolate which filter is excluding data.
“Too many data points” or timeout — Reduce the time range, increase the Min step, or add aggregations to reduce cardinality. Refer to Query high-cardinality data.

Query high-cardinality data

High-cardinality metrics (those with many unique label combinations) can cause queries to timeout or exceed memory limits when queried over long time ranges. Follow these practices to query high-cardinality data effectively:

Aggregate first, then filter — Use sum(), avg(), or count() to reduce the number of series before applying other operations. For example, sum(rate(metric[5m])) by (service) is far cheaper than querying all individual pod-level series.
Use recording rules for repeated queries — If a dashboard panel queries the same expensive expression on every load, create a Prometheus recording rule to pre-compute the result.
Scope with template variables — Use dashboard template variables to select a specific namespace, cluster, or job rather than querying all labels at once.
Increase Min step for overview panels — For panels showing trends over days or weeks, set a higher Min step (for example, 5m or 15m) to reduce the number of data points requested.
Set Max data points — Use the Max data points query option to cap resolution for wide time ranges.
Use Adaptive Metrics (Grafana Cloud) — Adaptive Metrics automatically reduces the cardinality of metrics that aren’t queried at full resolution.

If your queries hit memory or sample limits, refer to Memory limit exceeded for high-cardinality queries.

Expected PromQL behaviors

The following behaviors are frequently misinterpreted as bugs but are expected Prometheus behavior.

`increase()` returns fractional values on integer counters

increase() uses linear interpolation to estimate the increase over the specified time window. Because scrape timestamps rarely align exactly with the window boundaries, the result is interpolated — which produces fractional values even for integer counters.

This is expected. If you need integer results, wrap the expression in ceil() or floor():

ceil(increase(http_requests_total[5m]))

`rate()` appears to grow over time

If rate() shows an ever-increasing value instead of a steady per-second rate, the most common cause is multiple instances writing to the same time series without unique distinguishing labels. Prometheus merges these into a single series with an artificially growing counter.

To fix this, ensure every scrape target has unique labels (for example, instance, pod, or node). Then aggregate explicitly:

sum(rate(http_requests_total[5m])) by (job)

If you see this after a deployment or scaling event, verify your service discovery is assigning unique instance labels to each target.

Counter reset spikes after pod restarts

When a monitored process restarts, its counters reset to zero. Prometheus handles this with counter reset detection — rate() and increase() account for resets and don’t produce negative values. However, you may still see a brief spike at the reset point because the first scrape after a restart reports the full counter value accumulated since the restart.

To minimize the visual impact:

Use rate() over a window that spans multiple scrape intervals (for example, $__rate_interval) so the spike is averaged out.
For critical dashboards, apply smoothing with a longer range vector or use avg_over_time() on top of rate().
If you’re seeing large spikes from frequent pod restarts, investigate the restart frequency rather than trying to hide the spikes in queries.

Label cardinality causes “too many time series” errors

If adding a label filter doesn’t reduce the result count, you may have high-cardinality labels (such as request_id or user_id) on your metrics. Use the Prometheus TSDB status page (/tsdb-status) or the Grafana Metrics explorer to identify high-cardinality label combinations, then either drop unnecessary labels at scrape time or use aggregation in your queries.

Template variables — Use variables like $namespace or $job to create dynamic, reusable dashboards.
Annotations — Overlay Prometheus events on your panels.
Alerting — Create alert rules from your Prometheus queries.
Troubleshooting — Resolve common query errors and performance issues.

Prometheus template variables

Tue, 23 Jun 2026 13:06:40 +0000

Prometheus template variables

Template variables let you create dynamic, reusable dashboards by replacing hard-coded values (such as server names, namespaces, or job labels) with selectable variables. Grafana displays these variables as dropdown menus at the top of the dashboard, letting viewers change the displayed data without editing queries.

For an introduction to templating and template variables, refer to Templating and Add and manage variables.

Query variable types

Query variables query Prometheus to populate dropdown values. When creating a query variable, select a Prometheus data source and choose a query type:

Query Type	Required inputs	Description	Example
Label names	`metric` (optional)	Returns all label names, optionally filtered by metric regular expression.	Metric: `http_requests_total` → returns `job`, `instance`, `method`, `status`, etc.
Label values	`label` (required), `metric` (optional)	Returns values for a specific label, optionally filtered by metric.	Label: `job`, Metric: `http_requests_total` → returns `api-server`, `web`, `worker`
Metrics	`metric` (optional)	Returns metric names matching the specified regular expression.	Metric: `node_.*` → returns `node_cpu_seconds_total`, `node_memory_MemFree_bytes`, etc.
Query result	`query` (required)	Runs a PromQL query and returns the results as variable values.	`query_result(up{job="prometheus"})`
Series query	`metric`, `label`, or both	Returns time series matching the specified metric and/or label selectors.	Metric: `http_requests_total`, Label: `job="api"`
Classic query	query string	Deprecated. Legacy syntax using functions like `label_values(metric, label)`.	`label_values(http_requests_total, job)`

For details on metric names, label names, and label values, refer to the Prometheus data model.

Query type examples

Label values — Populate a dropdown with all jobs:

Create a new variable with Type: Query.
Select your Prometheus data source.
Set Query type to Label values.
Set Label to job.
Leave Metric empty to query across all metrics.

The variable dropdown now shows all unique job label values.

Label values — Filtered by metric:

Set Label to instance and Metric to node_cpu_seconds_total to show only instances that report CPU metrics.

Metrics — Find available metrics by pattern:

Set Query type to Metrics and enter http_.*_total in the Metric field to populate the dropdown with all HTTP counter metrics.

Query result — Dynamic top-N filtering:

Set Query type to Query result and enter:

query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))

Set Regex to /"([^"]+)"/ to extract the instance values from the result.

Set Refresh to On time range change so the top 5 instances update as you change the dashboard time range.

Query options

Option	Description
Data source	The Prometheus data source to query.
Regex	Optional regular expression to extract a portion of the returned values. Use capture groups — for example, `/.instance="([^"]+)"./` extracts the instance label value from a series string.
Sort	Sort order for dropdown values: `Disabled`, `Alphabetical (asc)`, `Alphabetical (desc)`, `Numerical (asc)`, `Numerical (desc)`, `Alphabetical (case-insensitive, asc)`, `Alphabetical (case-insensitive, desc)`.
Refresh	When to update values: `On dashboard load` or `On time range change`. Use `On time range change` for variables that depend on `$__range`.

Selection options

Multi-value — Allows selecting multiple values at once. Grafana joins them with a pipe (|) for regular expression matching.
Include All option — Adds an “All” option that selects every value. Combined with multi-value, this generates a regular expression like value1|value2|value3.

Note
When Multi-value or Include All is enabled, use =~ (regular expression match) instead of = (exact match) in your queries, since the variable value becomes a regex pattern.

Example with multi-value:

rate(http_requests_total{job=~"$job"}[$__rate_interval])

Use interval and range variables

You can use global built-in variables in query variable definitions:

Variable	Description
`$__interval`	Calculated interval based on time range and panel width.
`$__interval_ms`	Same as `$__interval` in milliseconds.
`$__range`	Duration of the current dashboard time range (for example, `1h`).
`$__range_s`	Duration in seconds.
`$__range_ms`	Duration in milliseconds.

For details, refer to Global built-in variables.

The label_values function (Classic query type) doesn’t support these variables. Use Query result type with query_result() instead:

query_result(max_over_time(up{job="$job"}[${__range_s}s]) == 1)

Set the variable’s Refresh to On time range change to update values when the time range changes.

Use `$__rate_interval`

$__rate_interval is a Grafana-specific variable designed for use with rate() and increase(). It guarantees a range window large enough to capture at least four scrape samples, preventing gaps or inaccuracies in results.

Always use $__rate_interval instead of a fixed interval or $__interval:

rate(http_requests_total[$__rate_interval])

Not:

rate(http_requests_total[5m])       # breaks at different zoom levels
rate(http_requests_total[$__interval])  # can be too small for rate()

How `$__rate_interval` is calculated

max($__interval + scrape_interval, 4 * scrape_interval)

Where scrape_interval is:

The per-query Min step setting, if set.
Otherwise, the data source’s Scrape interval setting (under Interval behavior in the data source configuration).

The panel-level min interval is affected by the resolution setting and doesn’t factor into this calculation.

Configure `$__rate_interval` correctly

For $__rate_interval to produce reliable results, the scrape interval must match your actual Prometheus scrape configuration:

Open the Prometheus data source configuration.
Under Interval behavior, set the Scrape interval to match the scrape_interval in your Prometheus configuration file (for example, 30s or 1m).
If different targets have different scrape intervals, set the data source scrape interval to the longest interval in use, or use the per-query Min step to override on specific panels.

Common pitfalls

Missing or incorrect scrape interval setting: If the data source scrape interval is left at the default 15s but your actual Prometheus scrape interval is 60s, $__rate_interval calculates too small a window. This causes rate() to return no data because there aren’t enough data points in the window.
Different values in edit mode versus dashboard: When editing a query, the panel displays at full width. On the dashboard, the panel may be narrower, which increases $__interval and therefore $__rate_interval. Queries that work in edit mode may produce different results (or gaps) on the dashboard.
LBAC-enabled data sources: Data sources using Label-Based Access Control (LBAC) may not inherit the scrape interval setting from the parent data source. Set the Min step explicitly on each query panel to ensure a correct $__rate_interval calculation.
Recording rules with fixed intervals: If you use $__rate_interval in a recording rule query, the interval depends on the evaluation context. For recording rules, use a fixed interval (for example, [5m]) rather than $__rate_interval.

For troubleshooting $__rate_interval issues, refer to $__rate_interval returns no data or incorrect values.

For additional background, refer to $__rate_interval for Prometheus rate queries that just work.

Variable syntax

The Prometheus data source supports three variable syntaxes:

Syntax	Example	Use case
`$varname`	`rate(http_requests_total{job=~"$job"}[$__rate_interval])`	Simple, readable. Cannot be used mid-word.
`${varname}`	`rate(http_requests_total{job=~"${job}"}[$__rate_interval])`	Use when the variable is adjacent to other text (for example, `${env}-cluster`).
`[[varname]]`	`rate(http_requests_total{job=~"[[job]]"}[$__rate_interval])`	Legacy syntax. Supported for backward compatibility.

Note
If Multi-value or Include All is enabled, the variable value becomes a regular expression pattern (for example, value1|value2). Use =~ instead of = in your label matchers.

Filters variable

Prometheus supports the Filters variable type (formerly called “ad hoc filters”), which lets dashboard viewers dynamically add label filters without editing queries.

Note
The Filter and Group by feature extends the Filters variable by adding grouping support for Prometheus and Loki data sources. For more information, refer to Filter and Group by.

To set up a Filters variable:

Create a new variable with Type: Filters.
Select your Prometheus data source.
Save the dashboard.

After you add the variable, a filter bar appears at the top of the dashboard. Viewers can add filters by selecting a label, operator (=, !=, =~, !~), and value. Grafana automatically applies these filters to all Prometheus queries on the dashboard.

Example: A viewer adds the filter namespace = production. All queries on the dashboard now include {namespace="production"} without any query modifications.

Note
Filters are applied to all queries using the selected data source. You cannot selectively apply them to specific panels.

Query editor — Use variables in PromQL queries.
Annotations — Use template variables in annotation queries.
Troubleshooting — Resolve variable-related query issues.

Prometheus annotations

Tue, 23 Jun 2026 13:06:40 +0000

Prometheus annotations

Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use Prometheus as a data source for annotations to display events such as deployments, alerts, or threshold crossings on your visualizations.

For general information about annotations, refer to Annotate visualizations.

Before you begin

Before creating Prometheus annotations, ensure you have:

A configured Prometheus data source in Grafana
Metrics in Prometheus that represent the events you want to annotate
Read access to the Prometheus instance

Create an annotation query

To add a Prometheus annotation to your dashboard:

Navigate to your dashboard and click Dashboard settings (gear icon).
Select Annotations in the left menu.
Click Add annotation query.
Enter a Name for the annotation.
Select your Prometheus data source from the Data source drop-down.
Enter a PromQL expression in the query field.
Set the Min step to control annotation density (a larger step means fewer annotations).
Configure the field mappings to control what appears in the annotation tooltip.
Optionally, select a Color for the annotation markers to distinguish different annotation types visually.
Click Save dashboard.

How Prometheus annotations work

Prometheus annotations work differently from SQL-based annotations. Instead of querying a table of events, you write a PromQL expression that returns time-series data. Grafana converts the query results into annotation events using these rules:

Grafana executes the PromQL query as a range query over the dashboard’s time window.
Every data point returned creates an annotation. There is no automatic filtering of zero values — if you only want annotations at specific moments, your PromQL expression must filter the results (for example, using > 0 or the ALERTS metric).
Grafana uses the field mapping configuration to determine what text, title, and tags to display for each annotation.
If the query returns multiple time series, each series produces its own set of annotations.

Note
Because every returned data point creates an annotation, queries that return continuous data (like node_cpu_seconds_total) will produce an annotation at every step interval, flooding your dashboard. Always use expressions that return data only at the moments you want annotated.

Field mappings

After entering your PromQL expression, use the field mapping drop-downs to control how query results are displayed as annotations. Grafana shows mapping options for:

Field	Description	Default behavior
Time	The timestamp for the annotation.	Uses the first time-type field (always present).
TimeEnd	An end timestamp for range annotations, which display as a shaded region instead of a vertical line.	Not set (produces point annotations).
Title	Short label displayed on the annotation marker.	Not set.
Text	The annotation description displayed when you hover over it.	Uses the first string-type field, or the metric/label display name if configured.
Tags	Comma-separated tags for the annotation. Helps categorize and filter annotations.	Not set.

To configure field mappings, select the appropriate field name from each drop-down, or enter a fixed text value.

Example annotation queries

The following examples show common annotation patterns with Prometheus.

Alert-based annotations using `ALERTS`

The most common and reliable way to create Prometheus annotations. Prometheus automatically generates an ALERTS metric for all configured alerting rules.

ALERTS{alertstate="firing"}

This creates an annotation at every step interval where an alert is firing. The ALERTS metric includes labels such as:

alertname — The name of the alerting rule
alertstate — Either firing or pending
Any labels defined on the alerting rule

To limit to specific alerts or severity levels:

ALERTS{alertname="HighCPUUsage", severity="critical"}

Configure the field mappings:

Text: alertname (displays the alert name on hover)
Tags: severity (allows filtering by severity)

Service restart annotations

Display annotations when a process restarts. The changes() function detects when a value changes, and > 0 ensures annotations only appear at the moment of change:

changes(process_start_time_seconds{job="myservice"}[5m]) > 0

Deployment annotations

If you track deployments by pushing a timestamp metric via Pushgateway or a recording rule:

changes(deployment_timestamp_seconds{environment="production"}[10m]) > 0

Configure the field mappings:

Text: environment (shows which environment was deployed)
Tags: environment

Threshold crossing annotations

Display annotations when available memory drops below 10%:

node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1

Note
Comparison operators in PromQL act as filters — they only return data points where the condition is true. This means the expression above only returns data when memory is below 10%, and annotations only appear at those times. There’s no need to add an outer > 0 wrapper.

Scaling event annotations

Display annotations when the number of running pods changes:

changes(kube_deployment_status_replicas{deployment="my-app"}[5m]) > 0

Error spike annotations

Display annotations when the error rate exceeds a threshold:

sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) > 0.05

This creates annotations when the error rate exceeds 5%.

Version or build change annotations

If you expose build metadata using a metric like build_info (common in Go services), annotate when the version changes:

changes(build_info{job="myservice"}[10m]) > 0

Configure the field mappings:

Text: version (shows the new version in the annotation tooltip)
Tags: job

If your build info metric uses a version label (for example, build_info{version="1.2.3"}), the annotation tooltip displays the version that was deployed.

Control annotation density

The Min step setting controls how many data points the query returns, which directly affects how many annotations appear. A larger step means fewer annotations:

Min step 1m — Up to one annotation per minute (good for short time ranges).
Min step 5m — Up to one annotation per 5 minutes (good for day-range dashboards).
Min step 1h — Up to one annotation per hour (good for week-range dashboards).

If your dashboard shows too many annotation markers, increase the Min step or add more specific filters to your query.

Use template variables in annotations

You can use template variables in your annotation queries to filter annotations based on dashboard variable selections:

ALERTS{alertstate="firing", instance=~"$instance"}

changes(process_start_time_seconds{job="$job"}[5m]) > 0

Template variables in annotations are resolved at query time using the current dashboard variable values.

Range annotations

Range annotations display as shaded regions rather than vertical lines, representing events with duration. To create range annotations, the query result must include a TimeEnd field mapping.

Because PromQL doesn’t natively return start/end time pairs, range annotations with Prometheus are limited to scenarios where you have separate metrics for start and end times (for example, via recording rules or custom metrics pushed to Pushgateway). For most use cases, point annotations (using the examples in this page) are the practical approach.

If your annotations aren’t appearing or you encounter errors, refer to Troubleshoot Prometheus data source issues.

Prometheus alerting

Tue, 23 Jun 2026 13:06:40 +0000

Prometheus alerting

You can use Grafana Alerting with Prometheus to create alerts based on your time-series data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

For general information about Grafana Alerting, refer to Grafana Alerting.

Before you begin

Before creating alerts with Prometheus, ensure you have:

A Prometheus data source configured in Grafana
Appropriate permissions to create alert rules
Understanding of the PromQL metrics you want to monitor

Alert rule types

Prometheus supports two alerting workflows in Grafana:

Type	Description
Grafana-managed alert rules	Alert rules defined and evaluated within Grafana, using Prometheus as the query data source. You create and manage these entirely in the Grafana Alerting UI.
Data source-managed rules	Alerting rules defined in Prometheus itself (in `prometheus.yml` or rule files). When Manage alerts via Alerting UI is enabled in the data source configuration, Grafana displays these existing rules in the Alerting UI. For Prometheus (unlike Mimir), this is read-only.

Create a Grafana-managed alert rule

To create a Grafana-managed alert rule using Prometheus:

Navigate to Alerting > Alert rules.
Click New alert rule.
Enter a name for the alert rule.
Select your Prometheus data source.
Write a PromQL query in the query editor.
Configure the alert condition (for example, when the last value is above a threshold).
Set the evaluation interval and pending period.
Configure notifications and labels.
Click Save rule.

For detailed instructions, refer to Create a Grafana-managed alert rule.

View data source-managed rules

When Manage alerts via Alerting UI is enabled in the Prometheus data source configuration, Grafana fetches and displays alerting rules defined in Prometheus. These appear in the Alerting UI alongside Grafana-managed rules but are marked as data source-managed.

For Prometheus data sources, this view is read-only. To modify these rules, update your Prometheus rule files directly.

Note
For Mimir and Cortex data sources, the Alerting UI supports both viewing and creating data source-managed rules. Prometheus only supports viewing.

Evaluation groups and intervals

Alert rules are organized into evaluation groups. Each group has an evaluation interval that determines how frequently the rules in that group are evaluated. For example, an evaluation interval of 1m means the alert query runs every 60 seconds.

The pending period determines how long a condition must be continuously true before the alert fires. For example, with a 1-minute evaluation interval and a 5-minute pending period, the condition must be true for 5 consecutive evaluations before firing.

Choose evaluation intervals based on your use case:

15s–30s — Critical infrastructure alerts where fast detection matters.
1m — Standard monitoring alerts (recommended default).
5m — Non-urgent or noisy metrics where you want to reduce evaluation load.

Example alert queries

The following examples show common alerting scenarios with Prometheus. Each example shows the PromQL query and how to configure the alert condition.

Alert on high CPU usage

Monitor CPU usage and alert when it exceeds 90%:

Query A:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval])) * 100)

Condition: Set the threshold to alert when the last value of A is above 90.

This approach separates the metric query from the threshold, making it easier to adjust the threshold later without editing the PromQL.

Alert on high memory usage

Monitor memory usage across nodes:

Query A:

(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

Condition: Alert when the last value of A exceeds 85.

Alert on high error rate

Monitor HTTP error rates per service:

Query A:

sum(rate(http_requests_total{status=~"5.."}[$__rate_interval])) by (job)
  /
sum(rate(http_requests_total[$__rate_interval])) by (job)
  * 100

Condition: Alert when the last value of A exceeds 5 (meaning error rate above 5%).

Alert on target down

Monitor whether Prometheus scrape targets are reachable:

Query A:

up{job="myservice"}

Condition: Alert when the last value of A is below 1.

Alert when a metric disappears

Use absent() to detect when a metric stops being scraped entirely — for example, when a service crashes and no longer reports metrics:

Query A:

absent(up{job="myservice"})

Condition: Alert when the last value of A equals 1 (the absent() function returns 1 when the metric is missing, and nothing when the metric exists).

For detecting staleness over a time window (metric exists but hasn’t reported recently):

absent_over_time(up{job="myservice"}[5m])

Multi-condition alert (high latency AND high traffic)

Use multiple queries and expressions to alert only when multiple conditions are true simultaneously. This reduces noise by avoiding alerts during low-traffic periods.

Query A — P95 latency:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="api"}[$__rate_interval])) by (le))

Query B — Request rate:

sum(rate(http_requests_total{job="api"}[$__rate_interval]))

Expression C — Math (both conditions must be true):

$A > 2 && $B > 100

Condition: Alert when C has a value (it only returns data when both latency exceeds 2 seconds AND request rate exceeds 100 req/s).

Recording rules

Grafana-managed recording rules pre-compute expensive PromQL expressions on a schedule and write the results back to a Prometheus-compatible data source as a new metric. This lets you query the pre-aggregated metric instead of re-evaluating the expensive expression on every dashboard load or alert evaluation.

When to use recording rules

Dashboard panels that query the same expensive expression repeatedly — pre-compute it once and query the result.
Alert rules on complex expressions — simplify the alert query by alerting on the pre-aggregated metric.
High-cardinality aggregations — reduce thousands of series to a handful of pre-computed series.

Set up Grafana-managed recording rules for Prometheus

Enable the data source as a recording rules target: In the Prometheus data source configuration, verify that Allow as recording rules target is toggled on (it’s on by default). This allows Grafana to write recording rule results back to this instance.
Verify write access: The Prometheus-compatible backend must support remote write. Grafana Cloud Metrics (Mimir) and self-hosted Mimir support this natively. Standard Prometheus requires the --web.enable-remote-write-receiver flag (Prometheus 2.33+).
Create a recording rule:
1. Navigate to Alerting > Alert rules.
2. Click New alert rule.
3. Select Recording rule as the rule type (under the Grafana-managed section).
4. Enter the PromQL expression you want to pre-compute:
  promql
```
sum(rate(http_requests_total[$__rate_interval])) by (service)
```
5. Enter a metric name for the result (for example, service:http_requests:rate5m). Follow the Prometheus recording rule naming convention: level:metric:operations.
6. Select the Target data source — the Prometheus instance where results will be written.
7. Set the evaluation interval (for example, every 1 minute).
8. Click Save rule.
Query the recorded metric: After the first evaluation, the new metric is available for dashboards and alerts:
promql
```
service:http_requests:rate5m{service="api"}
```

Recording rules with PDC

If your Prometheus instance is behind Private data source connect (PDC), Grafana can write recording rule results through the PDC tunnel. No additional configuration is needed — PDC supports both reads and writes.

Limitations

Recording rules require the target data source to support remote write. Standard Prometheus needs the --web.enable-remote-write-receiver flag.
Thanos does not support recording rules as a write target (refer to the Prometheus type comparison).
Recording rule evaluation uses the configured evaluation interval, not dashboard time ranges. Use a fixed range vector (for example, [5m]) rather than $__rate_interval.

For more information, refer to Create Grafana-managed recording rules.

Limitations

When using Prometheus with Grafana Alerting, be aware of the following limitations.

Template variables not supported

Alert queries don’t support template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like $instance or $job aren’t resolved.

If your dashboard query uses template variables, create a separate query for alerting with hard-coded values or use label matchers directly.

Query complexity

Complex queries with many nested functions or large result sets may timeout or fail to evaluate. Simplify queries for alerting by:

Reducing the time range used in range vectors
Using appropriate aggregation to limit the number of returned series
Adding label filters to narrow the data scanned
Using recording rules to pre-compute expensive expressions

OAuth token handling differs between Explore and Alerting

When using OAuth-authenticated Prometheus endpoints (Google Managed Prometheus, Azure Managed Prometheus), queries may succeed in Explore and dashboards but fail intermittently during alert evaluation. This happens because the alerting backend handles token refresh differently from the interactive query path.

If you’re using GCP, consider the datasource-syncer pattern — a sidecar process that refreshes OAuth tokens and updates the data source credentials on a schedule shorter than the token lifetime.

For detailed troubleshooting steps, refer to OAuth token expiration errors.

Data source-managed rules are read-only

Grafana can display Prometheus alerting rules but can’t create or modify them through the UI. To manage Prometheus-native alerting rules, edit your Prometheus rule files directly and reload the configuration.

Configure alert state for execution errors

By default, when a Grafana-managed alert rule encounters an execution error or timeout (such as a network blip, i/o timeout, or a transient 502 from Prometheus), the rule enters an Error state — which fires the alert. This can cause false alarms and spam on-call teams when the underlying issue is a brief connectivity interruption rather than a genuine threshold breach.

To prevent false positives from transient errors, configure the Alert state if execution error or timeout setting on each alert rule:

Open the alert rule for editing.
In the alert conditions section, locate Alert state if execution error or timeout.
Change the value from Alerting (default) to one of:
- Keep Last State — The alert retains its previous state (firing or normal) until a successful evaluation occurs. This is the recommended setting for most Prometheus alert rules.
- OK — The alert is set to normal during the error, preventing it from firing.
Click Save rule.

Note
If your alert rules frequently enter an error state, investigate the root cause (network stability, Prometheus resource limits, query timeout settings) rather than relying solely on this setting to suppress notifications.

Common transient errors that trigger this behavior include:

sse.dependencyError or sse.dataQueryError in alert state history
“context deadline exceeded” or “i/o timeout” messages
HTTP 502 or 500 responses from the Prometheus server

For more details on troubleshooting these errors, refer to Troubleshoot Prometheus data source issues.

Best practices

Follow these best practices when creating Prometheus alerts:

Configure error state handling: Set Alert state if execution error or timeout to Keep Last State to prevent transient backend errors from triggering false alarms.
Use $__rate_interval: When using rate() or increase() in alert queries, use $__rate_interval to ensure the range window is always large enough relative to the scrape interval. Grafana resolves this variable based on the evaluation interval and scrape interval configuration.
Add label filters: Include specific label matchers to focus on relevant data and improve query performance.
Set realistic pending periods: Use the pending period to avoid alerting on brief spikes. For example, set a 5-minute pending period so the condition must persist before firing.
Test queries first: Verify your query returns expected results in Explore before creating an alert.
Use meaningful names: Give alert rules descriptive names that indicate what they monitor and the severity.
Pre-aggregate with recording rules: For complex or frequently evaluated expressions, create recording rules and alert on the pre-aggregated metric.
Use absent() for availability monitoring: Detect when metrics stop being reported, which often indicates a crashed or unresponsive service.

If you encounter errors when creating or evaluating alert rules, refer to Troubleshoot Prometheus data source issues.

Troubleshoot Prometheus data source issues

Tue, 23 Jun 2026 13:06:40 +0000

Troubleshoot Prometheus data source issues

This document provides troubleshooting information for common errors you may encounter when using the Prometheus data source in Grafana.

Connection errors

The following errors occur when Grafana cannot establish or maintain a connection to Prometheus.

Failed to connect to Prometheus

Error message: “There was an error returned querying the Prometheus API”

Cause: Grafana cannot establish a network connection to the Prometheus server.

Solution:

Verify that the Prometheus server URL is correct in the data source configuration.
Check that Prometheus is running and accessible from the Grafana server.
Ensure the URL includes the protocol (http:// or https://).
Verify the port is correct (the Prometheus default port is 9090).
Ensure there are no firewall rules blocking the connection.
If Grafana and Prometheus are running in separate containers, use the container IP address or hostname instead of localhost.
For Grafana Cloud, ensure you have configured Private data source connect if your Prometheus instance is not publicly accessible.

Request timed out

Error message: “context deadline exceeded” or “request timeout”

Cause: The connection to Prometheus timed out before receiving a response.

Solution:

Check the network latency between Grafana and Prometheus.
Verify that Prometheus is not overloaded or experiencing performance issues.
Increase the Query timeout setting in the data source configuration under Interval behavior.
Check the Grafana server timeout configuration for server-level timeout settings.
Reduce the time range or complexity of your query.
Check if any network devices (load balancers, proxies) are timing out the connection.

Failed to parse data source URL

Error message: “Failed to parse data source URL”

Cause: The URL entered in the data source configuration is not valid.

Solution:

Verify the URL format is correct (for example, http://localhost:9090 or https://prometheus.example.com:9090).
Ensure the URL includes the protocol (http:// or https://).
Remove any trailing slashes or invalid characters from the URL.

Data doesn’t appear in Metrics Drilldown or Explore

Symptom: You’ve successfully tested the data source connection, but no metric data appears in Explore or Metrics Drilldown.

Cause: The wrong data source is selected, or the data source name doesn’t match expectations.

Solution:

Verify you’ve selected the correct data source from the data source drop-down menu.
When using remote_write to send metrics to Grafana Cloud, the data source name follows the convention grafanacloud-<stackname>-prom.
Check that metrics are being actively scraped by querying up in the Explore view.
If using Grafana Cloud, verify the remote_write endpoint URL and credentials are correct.

PDC connectivity errors

Error messages: “host unreachable”, “EOF”, “network unreachable”, “connection reset by peer”, “dial tcp: lookup … no such host”

Symptom: Prometheus queries fail intermittently or consistently when using Private data source connect (PDC) to reach a Prometheus instance behind a private network. The data source test may pass occasionally but queries fail under load.

Cause: PDC tunnels traffic through an SSH connection between Grafana Cloud and your PDC agent. Connectivity failures are most commonly caused by DNS resolution issues, network configuration on the customer side, or the PDC agent’s default connection limits being too low for the query volume.

Solutions:

Verify DNS resolution from the PDC agent host. The PDC agent must be able to resolve the Prometheus hostname from the machine it runs on. Run nslookup or dig for the Prometheus URL from the agent host to confirm.
Check network connectivity from the agent. Ensure the PDC agent can reach the Prometheus endpoint directly (for example, curl http://prometheus-host:9090/-/healthy from the agent machine).
Increase parallel SSH connections. The PDC agent defaults to 1 parallel SSH connection, which can bottleneck under load from multiple alert evaluations or dashboard queries. Increase this by setting the --ssh-connections flag (or PDC_SSH_CONNECTIONS environment variable) to a higher value (for example, 4 or 8):
sh
```
pdc-agent --ssh-connections=4
```
Check firewall rules. Ensure the PDC agent’s outbound SSH connection to Grafana Cloud isn’t being interrupted by firewall rules, NAT gateways, or idle connection timeouts.
Verify the PDC agent is running and healthy. Check agent logs for connection errors or restarts. The agent must maintain a persistent connection to Grafana Cloud.
Check for idle timeout issues. If the connection drops after periods of inactivity, configure TCP keepalives on the host or add a keepalive setting to the PDC agent configuration.

Note
PDC connectivity issues are almost always caused by networking on the customer side (DNS, firewall rules, routing), not by Grafana Cloud. The data source test passing doesn’t guarantee sustained connectivity under load — it only verifies a single query succeeds.

For general PDC setup and configuration, refer to Private data source connect (PDC) and Configure PDC.

Certificate verification failed

Error message: “x509: certificate signed by unknown authority” or “certificate verify failed”

Cause: Grafana cannot verify the TLS certificate presented by Prometheus.

Solution:

If using a self-signed certificate, enable Add self-signed certificate in the TLS settings and add your CA certificate.
Verify the certificate chain is complete and valid.
Ensure the certificate has not expired.
As a temporary workaround for testing, enable Skip TLS verify (not recommended for production).

TLS handshake error

Error message: “TLS: handshake failure” or “connection reset”

Cause: The TLS handshake between Grafana and Prometheus failed.

Solution:

Verify that Prometheus is configured to use TLS.
Check that the TLS version and cipher suites are compatible.
If using client certificates, ensure they are correctly configured in the TLS client authentication section.
Verify the server name matches the certificate’s Common Name or Subject Alternative Name.

Authentication errors

The following errors occur when there are issues with authentication credentials or permissions.

Unauthorized (401)

Error message: “401 Unauthorized” or “Authorization failed”

Cause: The authentication credentials are invalid or missing.

Solution:

Verify that the username and password are correct if using basic authentication.
Check that the authentication method selected matches your Prometheus configuration.
If using a reverse proxy with authentication, verify the credentials are correct.
For AWS SigV4 authentication, verify the IAM credentials and permissions. Alternatively, consider using the Amazon Managed Service for Prometheus data source for simplified AWS authentication.

OAuth token expiration errors (GCP and Azure)

Error messages: “ACCESS_TOKEN_EXPIRED”, “401 Unauthorized” in alerting but not in Explore

Symptom: Queries in Explore and dashboards work correctly, but alert rule evaluations fail intermittently with 401 errors. This is most common with Google Managed Prometheus (GMP) and Azure-managed Prometheus endpoints using OAuth/OIDC authentication.

Cause: The Grafana alerting backend and the interactive query path (Explore, dashboards) handle credential refreshes differently. The alerting evaluator can use a cached OAuth token beyond its expiry window due to a token staleness check issue in the Prometheus data source. This causes alerting to fail with expired credentials while interactive queries succeed because they trigger a fresh token exchange.

Solutions:

For Google Managed Prometheus (GMP):

Use the GCP datasource-syncer pattern: run a sidecar process that refreshes OAuth tokens and updates the Grafana data source credentials on a schedule shorter than the token lifetime (typically every 45 minutes for a 60-minute token). This ensures the stored token is always valid when the alerting backend uses it.
Alternatively, use the Google Cloud Monitor data source with its built-in GCP credential management rather than pointing the core Prometheus data source at a GMP endpoint.
If running on GKE with Workload Identity, ensure the Kubernetes service account token refresh is functioning and the projected token volume is mounted correctly.

For Azure Managed Prometheus:

Verify the Azure AD app registration’s client secret hasn’t expired.
If using Managed Identity, ensure the Grafana instance’s identity has the Monitoring Data Reader role on the Azure Monitor workspace.
Check that Forward OAuth identity is not enabled alongside Azure AD authentication — both use the same HTTP authorization headers and conflict with each other.

General steps:

Check the Grafana server logs for token refresh errors around the time alerts fail.
Verify the data source test (Save & test) passes — this confirms current credentials are valid but doesn’t guarantee the alerting backend has a fresh token.
If the issue persists, set Alert state if execution error or timeout to Keep Last State to prevent false alarms while investigating. Refer to Configure alert state for execution errors.

Note
This token caching behavior is a known issue that has received code fixes in recent Grafana releases. If you’re experiencing this on an older Grafana version, upgrading may resolve it. Check the Grafana changelog for relevant fixes.

LBAC not restricting data on non-Mimir backends

Symptom: You’ve enabled teamHttpHeadersMimir and configured Team LBAC rules, but users can still see all metrics regardless of their team assignments.

Cause: Label-Based Access Control (LBAC) for the Prometheus data source only works when the backend is Grafana Cloud Metrics (Mimir) or Grafana Enterprise Metrics (GEM). It doesn’t work with Google Managed Prometheus, self-hosted Prometheus, Thanos, or other Prometheus-compatible endpoints. The LBAC enforcement relies on Mimir-specific HTTP headers (X-Scope-OrgID and team-scoped label matchers) that other backends ignore.

Solution:

Verify your backend is Grafana Cloud Metrics or GEM. If you’re using a different Prometheus-compatible backend, LBAC isn’t supported.
For Google Managed Prometheus or other external endpoints, use data source permissions to control which teams can access the data source entirely, rather than per-label access control.
If you need per-label restrictions on a non-Mimir backend, consider proxying through a Mimir instance or using a separate data source per team with different credentials scoped to the appropriate data.

Azure AD or SigV4 authentication options not available

Symptom: The Azure AD or SigV4 authentication options don’t appear in the authentication drop-down when configuring the Prometheus data source.

Cause: These authentication methods require server-side feature flags that aren’t enabled by default, particularly on Grafana Cloud.

Solution:

Grafana Cloud: Contact Grafana Support to request the feature flag be enabled for your stack. This isn’t a self-service setting.
Self-managed Grafana: Add the required setting to your Grafana configuration file:
- For Azure AD: set azure_auth_enabled = true under [auth]
- For SigV4: set sigv4_auth_enabled = true under [auth]
For Amazon Managed Prometheus: Consider using the dedicated Amazon Managed Service for Prometheus data source instead, which handles AWS authentication natively without needing the SigV4 flag.

Forbidden (403)

Error message: “403 Forbidden” or “Access denied”

Cause: The authenticated user does not have permission to access the requested resource.

Solution:

Verify the user has read access to the Prometheus API.
Check Prometheus security settings and access control configuration.
If using a reverse proxy, verify the proxy is not blocking the request.
For AWS Managed Prometheus, verify the IAM policy grants the required permissions. Alternatively, consider using the Amazon Managed Service for Prometheus data source for simplified AWS authentication.

Query errors

The following errors occur when there are issues with PromQL syntax or query execution.

Query syntax error

Error message: “parse error: unexpected character” or “bad_data: 1:X: parse error”

Cause: The PromQL query contains invalid syntax.

Alternative cause: A proxy between Grafana and Prometheus requires authentication. When proxy authentication fails, the proxy redirects the request to an HTML authentication page. Grafana cannot parse the HTML response, which results in a parse error. This appears to be a query issue but is actually a proxy authentication issue.

Solution:

Check your query syntax for typos or invalid characters.
Verify that metric names and label names are valid identifiers.
Ensure string values in label matchers are enclosed in quotes.
Use the Prometheus expression browser to test your query directly.
Refer to the Prometheus querying documentation for syntax guidance.
If you have a proxy between Grafana and Prometheus, verify that proxy authentication is correctly configured. Check your proxy logs for authentication failures or redirects.

Query returns no data for a metric

Symptom: The query returns no data and the visualization is empty.

Cause: The specified metric does not exist in Prometheus, or there is no data for the selected time range.

Solution:

Verify the metric name is spelled correctly.
Check that the metric is being scraped by Prometheus.
Use the Prometheus API to browse available metrics at /api/v1/label/__name__/values.
Use the target metadata API to verify which metrics a target exposes.
Verify the time range includes data for the metric.

Query timeout limit exceeded

Error message: “query timed out in expression evaluation” or “query processing would load too many samples”

Cause: The query took longer than the configured timeout limit or would return too many samples.

Solution:

Reduce the time range of your query.
Add more specific label filters to limit the data scanned.
Increase the Query timeout setting in the data source configuration.
Use aggregation functions like sum(), avg(), or rate() to reduce the number of time series.
Increase the query.timeout or query.max-samples settings in Prometheus if you have admin access.

Too many time series

Error message: “exceeded maximum resolution of 11,000 points per timeseries” or “maximum number of series limit exceeded”

Cause: The query is returning more time series or data points than the configured limits allow.

Solution:

Reduce the time range of your query.
Add label filters to limit the number of time series returned.
Increase the Min interval or Resolution in the query options to reduce the number of data points.
Use aggregation functions to combine time series.
Adjust the Series limit setting in the data source configuration under Other settings.

Memory limit exceeded for high-cardinality queries

Error messages: “max-estimated-memory-consumption-per-query limit exceeded”, “query requires too much memory”, “max samples limit reached”

Symptom: Queries against high-cardinality metrics (thousands of unique label combinations) over long time ranges (days or weeks) fail with memory or sample limit errors. Short time ranges may work but expanding the range causes the failure.

Cause: Prometheus and Mimir enforce per-query memory and sample limits to protect the system from resource exhaustion. High-cardinality metrics (for example, metrics with a pod, request_id, or user_id label) multiplied by long time ranges produce result sets that exceed these limits.

Solutions:

Reduce the query scope:
- Shorten the time range.
- Add label matchers to select fewer series (for example, filter to a specific namespace or job).
- Increase the Min step or use a larger step interval to reduce the number of data points per series.
Use recording rules to pre-aggregate: Create recording rules that pre-compute the aggregation you need. For example, if you commonly query sum(rate(http_requests_total[5m])) by (service), create a recording rule for it and query the pre-aggregated metric instead.
YAML
```
groups:
  - name: aggregations
    rules:
      - record: service:http_requests_total:rate5m
        expr: sum(rate(http_requests_total[5m])) by (service)
```
Use Adaptive Metrics (Grafana Cloud): If you’re on Grafana Cloud, Adaptive Metrics automatically identifies and aggregates high-cardinality metrics that aren’t being queried at full resolution, reducing storage and query costs.
Restructure dashboards for high-cardinality data:
- Use template variables to scope queries to a subset of labels rather than querying all series at once.
- Split long-range overviews (weekly/monthly) into separate panels that use pre-aggregated recording rules.
- Use the Max data points setting in the query options to cap the resolution for overview panels.
Increase limits (self-managed only): If you have admin access to Prometheus or Mimir, you can increase limits in the server configuration:
- Prometheus: --query.max-samples flag
- Mimir: -querier.max-samples, -querier.max-estimated-memory-consumption-per-query
Note
Increasing limits allows larger queries to succeed but also increases the risk of resource exhaustion. Prefer reducing query scope or using recording rules over raising limits.

Invalid function or aggregation

Error message: “unknown function” or “parse error: unexpected aggregation”

Cause: The query uses an invalid or unsupported PromQL function.

Solution:

Verify the function name is spelled correctly and is a valid PromQL function.
Check that you are using the correct syntax for the function.
Ensure your Prometheus version supports the function you are using.
Refer to the PromQL functions documentation for available functions.

`rate()` or `increase()` returning unexpected values

Symptom: increase() returns fractional values on integer counters, rate() shows an ever-increasing value instead of a steady per-second rate, or counter resets cause large spikes in visualizations.

Possible causes and solutions:

Cause	Solution
`increase()` fractional values	Expected behavior — Prometheus uses linear interpolation. Use `ceil()` or `floor()` if you need integers.
`rate()` grows over time	Multiple instances write to the same series without unique labels. Ensure each target has unique `instance`/`pod` labels and aggregate with `sum by`.
Counter reset spikes after pod restarts	Use `$__rate_interval` or a longer range vector to smooth spikes. Investigate frequent restarts as the root cause.
Values differ between edit mode and dashboard	Panel width affects `$__interval` which affects `rate()` window calculations. Set a Min step on the query.

For detailed explanations of these behaviors, refer to Expected PromQL behaviors.

Aggregation by labels with dots

Symptom: Queries that aggregate by label names containing dots (for example, container.name) return incorrect or incomplete results.

Cause: Prior to Grafana 13, there was a bug where labels with dots in their names were not handled correctly during aggregation operations like sum by or avg by.

Solution:

Upgrade to Grafana 13 or later, which correctly handles labels with dots in aggregation queries.
Verify your query uses the correct label name with dots (for example, sum by (container.name) (metric_name)).

Configuration errors

The following errors occur when the data source is not configured correctly.

Invalid Prometheus type

Error message: Unexpected behavior when querying metrics or labels

Cause: The Prometheus type setting does not match your actual Prometheus-compatible database.

Solution:

Open the data source configuration in Grafana.
Under Performance, select the correct Prometheus type (Prometheus, Cortex, Mimir, or Thanos).
Different database types support different APIs, so setting this incorrectly may cause unexpected behavior.

Scrape interval mismatch

Symptom: Data appears sparse, or rate() queries return no data or incomplete results.

Cause: The Scrape interval setting in Grafana does not match the actual scrape interval in Prometheus. This especially affects rate() queries, which require at least two data points within the specified time window. For example, if your actual scrape interval is 5 minutes but Grafana uses the default (15 seconds for OSS, 1 minute for Grafana Cloud), a query like rate(http_requests_total[1m]) returns no data because there are no data points within that 1-minute window.

Solution:

Check your Prometheus configuration file for the scrape_interval setting.
Update the Scrape interval in the Grafana data source configuration under Interval behavior to match.
Use $__rate_interval instead of hard-coded time windows in rate() queries. This variable automatically adjusts based on your scrape interval.
For more information, refer to $__rate_interval for Prometheus rate queries that just work.

`$__rate_interval` returns no data or incorrect values

Symptom: Queries using $__rate_interval return no data, return different values in edit mode versus the dashboard, or produce unexpected gaps.

Cause: $__rate_interval is calculated as max($__interval + scrape_interval, 4 * scrape_interval). If any input to this formula is incorrect, the resulting window is wrong — either too small (no data) or inconsistent across contexts.

Common causes and solutions:

Cause	Solution
Data source scrape interval left at default `15s` while actual Prometheus scrape interval is longer (for example, `60s`)	Set the Scrape interval under Interval behavior in the data source configuration to match your Prometheus `scrape_interval`.
Query works in edit mode but shows gaps on the dashboard	Panel size affects `$__interval`. Smaller panels produce larger intervals. Set a Min step on the query to enforce a consistent floor.
LBAC-enabled data source doesn’t inherit scrape interval	Set the Min step explicitly on each query panel rather than relying on data source inheritance.
Using `$__rate_interval` in recording rules or alerting	Use a fixed interval (for example, `[5m]`) instead of `$__rate_interval` in contexts without a panel/dashboard.

To debug the current value:

Open the query inspector in a panel (click the panel title, then Inspect > Query).
Look at the expanded query sent to Prometheus — the actual interval value replacing $__rate_interval is visible in the request.
Compare this value against your actual scrape interval. If it’s smaller than your scrape interval, you need to configure the data source scrape interval or set a Min step.

For detailed documentation on how $__rate_interval works and how to configure it, refer to Use $__rate_interval.

Performance issues

The following issues affect query speed and data freshness.

Slow query performance

Symptom: Queries take a long time to execute, dashboards are slow to load, or the loading spinner persists.

Cause: Queries scan too much data, the Prometheus server is overloaded, or the network connection is slow.

Solution:

Reduce the time range of your query.
Add more specific label filters to limit the data scanned.
Increase the Min step in the query options to reduce the number of data points.
Check Prometheus server performance and resource utilization.
Enable Disable metrics lookup in the data source configuration for large Prometheus instances.
Enable Incremental querying (beta) to cache query results.
Use recording rules to pre-aggregate frequently queried data.
For high-cardinality metrics, refer to Memory limit exceeded for high-cardinality queries.

Data appears delayed or missing recent points

Symptom: The visualization doesn’t show the most recent data, even after refreshing.

Cause: Scrape timing, clock drift, or dashboard refresh settings.

Solution:

Check the dashboard time range and refresh settings.
Verify the Scrape interval is configured correctly in the data source settings.
Ensure Prometheus has finished scraping the target (there’s a delay between the scrape interval and data availability).
Check for clock synchronization issues between Grafana and Prometheus (use NTP).
For rate() and similar functions, the most recent partial scrape interval won’t have enough data points — this is expected.

Exemplars not showing

Symptom: Exemplar data doesn’t appear on graphs even though you expect it.

Cause: Exemplars require specific configuration in both the data source and the query editor.

Solution:

Verify that exemplars are configured in the data source settings under Exemplars (at least one exemplar link must be defined).
Check that your Prometheus version supports exemplars (2.26+).
Ensure your instrumented application is actually sending exemplar data with metrics.
Verify the tracing data source is correctly configured for the exemplar link.
Enable the Exemplars toggle in the query editor for the specific query.
Exemplars only appear with Range query type, not Instant.

Annotation errors

The following issues occur when using Prometheus as a data source for annotations.

Annotations not appearing

Symptom: You’ve configured a Prometheus annotation query, but no annotations appear on your dashboard.

Possible causes and solutions:

Cause	Solution
Query returns no data in the current time range	Verify the query returns results in Explore for the dashboard’s time range.
Query returns continuous data (too many annotations)	Every returned data point creates an annotation. If the query returns hundreds of points, annotations may render but are too dense to see. Increase the Min step or refine your query to only return data at event moments.
Wrong data source selected	Verify the correct Prometheus data source is selected in the annotation configuration.
Annotation is disabled	Check that the annotation toggle is enabled (eye icon) in the dashboard’s annotation settings.
Time range mismatch	Expand the dashboard time range to include the events you expect to see.

Note
Prometheus annotations create a marker for every data point returned by the query. There’s no automatic filtering of zero values. If you only want annotations at specific moments, your PromQL expression must return data only at those times (for example, using > 0, changes() > 0, or the ALERTS metric).

For more information on configuring annotations, refer to Prometheus annotations.

Alerting errors

The following issues occur when using Prometheus with Grafana Alerting.

Transient alert errors triggering false alarms

Error messages: sse.dependencyError, sse.dataQueryError, “context deadline exceeded”, “i/o timeout”

Symptom: Alert rules intermittently fire due to execution errors rather than genuine threshold breaches. On-call teams receive false positive notifications. Alert state history shows error states caused by transient backend issues (network blips, HTTP 502/500 responses, timeouts) rather than actual metric conditions being met.

Cause: By default, when an alert rule encounters an execution error or timeout, Grafana sets the alert state to Alerting — which fires the alert. Transient connectivity issues between Grafana and Prometheus (i/o timeouts, deadline exceeded, brief outages) trigger this behavior even though the underlying metric hasn’t crossed its threshold.

Solution:

Open each affected alert rule for editing.
In the alert conditions section, change Alert state if execution error or timeout from Alerting to Keep Last State.
Save the rule.

This ensures the alert retains its previous state during transient errors and only fires when a successful evaluation confirms the threshold is breached.

If errors are frequent, also investigate:

Network stability between Grafana and Prometheus.
Prometheus resource utilization (CPU, memory, disk I/O).
The Query timeout setting in the data source configuration — increase it if complex queries regularly exceed the limit.
Query complexity — simplify queries or use recording rules to pre-compute expensive expressions.

For configuration details, refer to Configure alert state for execution errors.

Alert rule fails to evaluate

Symptom: An alert rule using a Prometheus query shows evaluation errors or remains in a “No Data” state.

Possible causes and solutions:

Cause	Solution
Template variables in query	Alert queries don’t support template variables. Replace variables with hard-coded values.
Query timeout	Simplify the query or increase the evaluation timeout. Use recording rules for complex expressions.
Data source unreachable	Verify the Prometheus data source connection is working (test it in the data source settings).
No data in range	Ensure the metric has recent data. Check that Prometheus is actively scraping the target.

Data source-managed rules not visible

Symptom: Prometheus alerting rules don’t appear in the Grafana Alerting UI.

Solution:

Verify that Manage alerts via Alerting UI is enabled in the data source configuration.
Check that Prometheus has alerting rules configured in its rule files.
Ensure Grafana can access the Prometheus rules API endpoint (/api/v1/rules).
For Prometheus (unlike Mimir), the Alerting UI only supports viewing existing rules, not creating new ones.

For more information on alerting with Prometheus, refer to Prometheus alerting.

Get additional help

If you continue to experience issues after following this troubleshooting guide:

Check the Prometheus documentation for API and PromQL guidance.
Review the Grafana community forums for similar issues.
Contact Grafana Support if you are a Cloud Pro, Cloud Contracted, or Enterprise user.
When reporting issues, include:
- Grafana version
- Prometheus version and type (Prometheus, Mimir, Cortex, Thanos)
- Error messages (redact sensitive information)
- Steps to reproduce
- Relevant configuration such as data source settings, query timeout, and TLS settings (redact tokens, passwords, and other credentials)

Prometheus data source on Grafana Labs

Configure the Prometheus data source

Configure the Prometheus data source

Before you begin

Add the data source

Configuration options

Name and default

Connection

Authentication

TLS settings

HTTP headers

Advanced HTTP settings

Alerting

Interval behavior

Query editor

Performance

Other

Exemplars

Private data source connect

Save and test

Provision the data source

Provision with YAML

Provision with Terraform

Azure authentication settings (deprecated)

Troubleshooting

Prometheus query editor

Prometheus query editor

Query options

Legend

Min step

Format

Type

Exemplars

Builder mode

Select a metric

Add operations

Hints

Metrics explorer

Code mode

Metrics browser

Common query patterns

Request rate per service

Error rate percentage

Histogram quantile (p95 latency)

Aggregation by label (CPU usage per instance)

Multi-query expressions (available memory percentage)

Alert-compatible query (target down)

Use the query inspector

Query high-cardinality data

Expected PromQL behaviors

increase() returns fractional values on integer counters

rate() appears to grow over time

Counter reset spikes after pod restarts

Label cardinality causes “too many time series” errors

Related resources

Prometheus template variables

Prometheus template variables

Query variable types

Query type examples

Query options

Selection options

Use interval and range variables

Use $__rate_interval

How $__rate_interval is calculated

Configure $__rate_interval correctly

Common pitfalls

Variable syntax

Filters variable

Related resources

Prometheus annotations

Prometheus annotations

Before you begin

Create an annotation query

How Prometheus annotations work

Field mappings

Example annotation queries

Alert-based annotations using ALERTS

Service restart annotations

Deployment annotations

Threshold crossing annotations

`increase()` returns fractional values on integer counters

`rate()` appears to grow over time

Use `$__rate_interval`

How `$__rate_interval` is calculated

Configure `$__rate_interval` correctly

Alert-based annotations using `ALERTS`

`rate()` or `increase()` returning unexpected values

`$__rate_interval` returns no data or incorrect values