Prometheus data source on Grafana Labs

Configure the Prometheus data source

Fri, 07 Mar 2025 11:34:44 +0100

Configure Prometheus

Grafana ships with built-in support for Prometheus. If you are new to Prometheus the following documentation will help you get started working with Prometheus and Grafana:

Configure the data source

To add the Prometheus data source, complete the following steps:

Click Connections in the left-side menu.
Under Connections, click Add new connection.
Enter Prometheus in the search bar.
Select Prometheus data source.
Click Create a Prometheus data source in the upper right.

You will be taken to the Settings tab where you will set up your Prometheus configuration.

Configuration options

The following is a list of configuration options for Prometheus.

The first option to configure is the name of your connection:

Name - The data source name. This is how you refer to the data source in panels and queries. Examples: prometheus-1, prom-metrics.
Default - Toggle to select as the default name in dashboard panels. When you go to a dashboard panel this will be the default selected data source.

HTTP section

URL - The URL of your Prometheus server. If your Prometheus server is local, use <http://localhost:9090>. If it is on a server within a network, this is the URL with port where you are running Prometheus. Example: <http://prometheus.example.orgname:9090>.
Allowed cookies - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.
Timeout - The HTTP request timeout. This must be in seconds. There is no default, so this setting is up to you.

Auth section

There are several authentication methods you can choose in the Authentication section.

Note
Use TLS (Transport Layer Security) for an additional layer of security when working with Prometheus. For information on setting up TLS encryption with Prometheus see Securing Prometheus API and UI Endpoints Using TLS Encryption. You must add TLS settings to your Prometheus configuration file prior to setting these options in Grafana.

Basic authentication - The most common authentication method. Use your data source user name and data source password to connect.
With credentials - Toggle on to enable credentials such as cookies or auth headers to be sent with cross-site requests.
TLS client authentication - Toggle on to use client authentication. When enabled, add the Server name, Client cert and Client key. The client provides a certificate that is validated by the server to establish the client’s trusted identity. The client key encrypts the data between client and server.
With CA cert - Authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file.
Skip TLS verify - Toggle on to bypass TLS certificate validation.
Forward OAuth identity - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source.

Custom HTTP headers

Header - Add a custom header. This allows custom headers to be passed based on the needs of your Prometheus instance.
Value - The value of the header.

Additional settings

Following are additional configuration options.

Alerting

Manage alerts via Alerting UI - Toggle to enable Alertmanager integration for this data source.

Interval behavior

Scrape interval - Set this to the typical scrape and evaluation interval configured in Prometheus. The default is 15s.
Query timeout - The default is 60s.

Query editor

Default editor - Sets a default editor. Options are Builder or Code. For information on query editor types see Prometheus query editor.
Disable metrics lookup - Toggle on to disable the metrics chooser and metric/label support in the query field’s autocomplete. This helps if you have performance issues with large Prometheus instances.

Performance

Prometheus type - The type of your Prometheus server. There are four options: Prometheus, Cortex, Thanos, Mimir.
Version Select the version you are using. Once the Prometheus type has been selected, a list of versions auto-populates using the Prometheus buildinfo API. The Cortex Prometheus type does not support this API so you will need to manually add the version.
Incremental querying (beta) - Changes the default behavior of relative queries to always request fresh data from the Prometheus instance. Enable this option to decrease database and network load.

Other

Custom query parameters - Add custom parameters to the Prometheus query URL. For example timeout, partial_response, dedup, or max_source_resolution. Multiple parameters should be concatenated together with an ‘&’.
HTTP method - Use either POST or GET HTTP method to query your data source. POST is the recommended and pre-selected method as it allows bigger queries. Change to GET if you have a Prometheus version older than 2.1 or if POST requests are restricted in your network.

Exemplars

Support for exemplars is available only for the Prometheus data source. If this is your first time working with exemplars see Introduction to exemplars. An exemplar is a specific trace representative of measurement taken in a given time interval.

Internal link - Toggle on to enable an internal link. When enabled, reveals the data source selector. Select the backend tracing data store for your exemplar data.
URL - (Visible if you disable Internal link) Defines the external link’s URL trace backend. You can interpolate the value from the field by using the ${__value.raw} macro.
Data source - (Visible if you enable Internal link) The data source the exemplar will navigate to.
URL label - Adds a custom display label to override the value of the Label name field.
Label name - The name of the field in the labels object used to obtain the traceID property.
Remove exemplar link - Click to remove existing links.

Prometheus query editor

Thu, 12 Oct 2023 15:23:06 +0000

Prometheus query editor

Grafana provides a query editor for the Prometheus data source to create queries in PromQL. For more information about PromQL, see Querying Prometheus.

For general documentation on querying data sources in Grafana, see Query and transform data.

For options and functions common to all query editors, see Query editors.

Choose a query editing mode

The Prometheus query editor has two modes:

Each mode is explained in greater detail below.

Both modes are synchronized, so you can switch between them. However, if there is an issue with the query while switching modes, a warning message will appear.

The query editor toolbar contains the following elements:

Kick start your query - Click to see a list of operation patterns that help you quickly get started adding multiple operations to your query. These include:
- Rate query starters
- Histogram query starters
- Binary query starters

Click the arrow next to each to see available options to add to your query.

Explain - Toggle to display a step-by-step explanation of all query components and operations.

Builder/Code - Click the corresponding Builder or Code tab on the toolbar to select a editor mode.

Configure common options

You can configure Prometheus-specific options in the query editor by setting several options regardless of mode.

Legend

The Legend setting defines the time series’s name. You can use a predefined or custom format.

Auto - Displays unique labels. Also displays all overlapping labels if a series has multiple labels.
Verbose - Displays all label names.
Custom - Uses templating to select which labels will be included. For example, {{hostname}} is replaced by the label value for the label hostname. Clear the input and click outside of it to select another mode.

Min step

The Min step setting defines the lower bounds on the interval between data points. For example, set this to 1h to hint that measurements are taken hourly. This setting supports the $__interval and $__rate_interval macros.

Format

Switch between the following format options:

Time series - The default time series format. See Time series kind formats for information on time series data frames and how time and value fields are structured.
Table - This works only in a Table panel.
Heatmap - Displays metrics of the Histogram type on a Heatmap panel by converting cumulative histograms to regular ones and sorting the series by the bucket bound.

Type

The Type setting sets the query type. These include:

Both - The default option. Returns results for both a Range query and an Instant query.
Range - Returns a range vector consisting of a set of time series data containing a range of data points over time for each time series. You can choose lines, bars, points, stacked lines or stacked bars
Instant - Returns one data point per query and only the most recent point in the time range provided. The results can be shown in table format or as raw data. To depict instant query results in the time series panel, first add a field override, next add a property to the override named Transform, and finally select Constant from the Transform dropdown.

For more information, refer to the Time Series Transform option documentation.

Note
Grafana modifies the request dates for queries to align them with the dynamically calculated step. This ensures a consistent display of metrics data, but it can result in a small gap of data at the right edge of a graph.

Exemplars

Toggle Exemplars to run a query that includes exemplars in the graph. Exemplars are unique to Prometheus. For more information see Introduction to exemplars.

Note
There is no option to add exemplars with an Instant query type.

Inspector

Click Inspector to get detailed statistics regarding your query. Inspector functions as a kind of debugging tool that “inspects” your query. It provides query statistics under Stats, request response time under Query, data frame details under {} JSON, and the shape of your data under Data.

Builder mode

Builder mode helps you build queries using a visual interface. This option is best for users who have limited or no previous experience working with Prometheus and PromQL.

This video demonstrates how to use the visual Prometheus query builder available in Grafana v9.0:

Metrics

When you are ready to create a query, you can choose the specific metric name from the dropdown list under Metric. The data source requests the list of available metrics from the Prometheus server based on the selected time rage. You can also enter text into the selector when the dropdown is open to search and filter the list.

Metrics explorer

If you would like to explore your metrics in the query builder further, you can open the Metrics Explorer by clicking the first option in the metric select component of the query builder.

The metrics explorer is different than the metrics browser. The metrics explorer is only found in the query builder section. The metrics browser is only found in the code editor. The metrics explorer does not have the ability to browse labels yet, but the metrics browser can display all labels on a metric name.

The metrics explorer displays all metrics in a paginated table list. The list shows the total number of metrics, as well as the name, type and description for each metric. You can enter text into the search input to filter results. You can also filter by type.

There are also additional settings for the following items:

Include description in search. Search by name and description
Include results with no metadata. Many Prometheus metrics have no metadata. This allows users to include metrics with undefined type and description.
Disable text wrap.
Enable regex search. This uses the Prometheus API to enable regex search for the metric name.

Label filters

Select desired labels and their values from the dropdown list. When a metric is selected, the data source requests available labels and their values from the server. Use the + button to add a label, and the x button to remove a label.

Operations

Select the + Operations button to add operations to your query.

The query editor groups operations into the following sections:

Aggregations - for additional information see Aggregation operators.
Range functions - for additional information see Functions.
Functions - for additional information see Functions.
Binary operations - for additional information see Binary operators.
Trigonometric - for additional information see Trigonometric functions.
Time functions - for additional information see Functions.

All operations have function parameters under the operation header. Click the operator to see a full list of supported functions. Some operations allow you to apply specific labels to functions.

Some operations make sense only when used in a specific order. If adding an operation would result in nonsensical query, the query editor adds the operation to the correct place.

Hints

The query editor can detect which operations are most appropriate for some selected metrics. If it does, it displays a hint next to the + Operations button.

To add the operation to your query, click the Hint.

Once you are satisfied with your query, click Run query.

Code mode

Code mode is for the experienced Prometheus user with prior expertise in PromQL, Prometheus’ query language. The Code mode editor allows you to create queries just as you would in Prometheus. For more information about PromQL see Querying Prometheus.

The user interface (UI) also lets you select metrics, labels, filters and operations.

You can write complex queries using the text editor with autocompletion features and syntax highlighting. It also contains a Metrics browser to further help you write queries.

Use autocomplete

Code mode’s autocomplete feature works automatically while typing. The query editor can autocomplete static functions, aggregations, keywords, and also dynamic items like metrics and labels. The autocompletion dropdown includes documentation for the suggested items where available.

Metrics browser

The metrics browser locates metrics and selects relevant labels to help you build basic queries. When you click Metrics browser in Code mode, it displays all available metrics and labels. If supported by your Prometheus instance, each metric also displays its HELP and TYPE as a tooltip.

When you select a metric under Step 1, the browser narrows down the available labels to show only the ones applicable to the metric. You can then select one or more labels shown in Step 2. Select one or more values in Step 3 for each label to tighten your query scope. In Step 4, you can select Use query to run the query, Use as rate query to add the rate operation to your query ($__rate_interval), Validate selector to verify the selector is valid and show the number of series found, or Clear to clear your selections and start over.

Note
If you do not remember a metric name, you can also select a few labels to narrow down the list, then find relevant label values.

All lists in the metrics browser have a search field above them to quickly filter for metrics or labels that match a certain string. The values section has only one search field, and its filtering applies to all labels to help you find values across labels once selected.

For example, among your labels app, job, job_name only one might have the value you are looking for.

Once you are satisfied with your query, click Run query.

Prometheus template variables

Wed, 23 Aug 2023 20:27:59 +0000

Prometheus template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana refers to such variables as template variables. Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.

For an introduction to templating and template variables, see Templating and Add and manage variables.

Use query variables

You have the option to use several different variable types, but variables of the type Query will query Prometheus for a list of metrics, labels, label values, a query result or a series.

Select a Prometheus data source query type and enter the required inputs:

Query Type	Input(* required)	Description	Used API endpoints
`Label names`	`metric`	Returns a list of all label names matching the specified `metric` regex.	/api/v1/labels
`Label values`	`label`*, `metric`	Returns a list of label values for the `label` in all metrics or the optional metric.	/api/v1/label/`label`/values or /api/v1/series
`Metrics`	`metric`	Returns a list of metrics matching the specified `metric` regex.	/api/v1/label/__name__/values
`Query result`	`query`	Returns a list of Prometheus query result for the `query`.	/api/v1/query
`Series query`	`metric`, `label` or both	Returns a list of time series associated with the entered data.	/api/v1/series

For details on metric names, label names, and label values, refer to the Prometheus documentation.

Query options

Under the query variable type, you can set the following query options:

Option	Description
Data source	Select your data source from the dropdown list.
Select query type	Options are `default`, `value` and `metric name`. Each query type hits a different Prometheus endpoint.
Regex	Optional, if you want to extract part of a series name or metric node segment.
Sort	Default is `disabled`. Options include `alphabetical`, `numerical` and `alphabetical case-sensitive`.
Refresh	When to update the values for the variable. Options are `On dashboard load` and `On time range change`.

Selection options

The following selection options are available:

Multi-value - Check this option to enable multiple values to be selected at the same time.
Include All option - Check this option to include all variables.

Use interval and range variables

You can use some global built-in variables in query variables, for example, $__interval, $__interval_ms, $__range, $__range_s and $__range_ms. For details, see Global built-in variables. The label_values function doesn’t support queries, so you can use these variables in conjunction with the query_result function to filter variable queries.

Make sure to set the variable’s refresh trigger to be On Time Range Change to get the correct instances when changing the time range on the dashboard.

Example:

Populate a variable with the busiest 5 request instances based on average QPS over the time range shown in the dashboard:

Query: query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))
Regex: /"([^"]+)"/

Populate a variable with the instances having a certain state over the time range shown in the dashboard, using $__range_s:

Query: query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
Regex:

Use `$__rate_interval`

Note
Available in Grafana v7.2 and higher.

We recommend using $__rate_interval in the rate and increase functions instead of $__interval or a fixed interval value. Because $__rate_interval is always at least four times the value of the Scrape interval, it avoid problems specific to Prometheus.

For example, instead of using:

rate(http_requests_total[5m])

or:

rate(http_requests_total[$__interval])

We recommend that you use:

rate(http_requests_total[$__rate_interval])

The value of $__rate_interval is defined as max($__interval + Scrape interval, 4 * Scrape interval), where Scrape interval is the “Min step” setting (also known as query*interval, a setting per PromQL query) if any is set. Otherwise, Grafana uses the Prometheus data source’s “Scrape interval” setting.

The “Min interval” setting in the panel is modified by the resolution setting, and therefore doesn’t have any effect on Scrape interval.

For details, refer to the Grafana blog.

Choose a variable syntax

The Prometheus data source supports two variable syntaxes for use in the Query field:

$<varname>, for example rate(http_requests_total{job=~"$job"}[$_rate_interval]), which is easier to read and write but does not allow you to use a variable in the middle of a word.
[[varname]], for example rate(http_requests_total{job=~"[[job]]"}[$_rate_interval])

If you’ve enabled the Multi-value or Include all value options, Grafana converts the labels from plain text to a regex-compatible string, which requires you to use =~ instead of =.

Use the ad hoc filters variable type

Prometheus supports the special ad hoc filters variable type, which you can use to specify any number of label/value filters on the fly. These filters are automatically applied to all your Prometheus queries.

Prometheus data source on Grafana Labs

Configure the Prometheus data source

Configure Prometheus

Configure the data source

Configuration options

HTTP section

Auth section

Custom HTTP headers

Additional settings

Alerting

Interval behavior

Query editor

Performance

Other

Exemplars

Prometheus query editor

Prometheus query editor

Choose a query editing mode

Toolbar elements

Configure common options

Legend

Min step

Format

Type

Exemplars

Inspector

Builder mode

Metrics

Metrics explorer

Label filters

Operations

Hints

Code mode

Use autocomplete

Metrics browser

Prometheus template variables

Prometheus template variables

Use query variables

Query options

Selection options

Use interval and range variables

Use $__rate_interval

Choose a variable syntax

Use the ad hoc filters variable type

Use `$__rate_interval`