Prometheus data source

Data sourcesPrometheus

Grafana includes built-in support for Prometheus. This topic explains options, variables, querying, and other options specific to the Prometheus data source. Refer to Add a data source for instructions on how to add a data source to Grafana.

Prometheus settings

To access Prometheus settings, click the Configuration (gear) icon, then click Data Sources, and then click Prometheus.

Name Description
Name The data source name. This is how you refer to the data source in panels and queries.
Default Default data source means that it will be pre-selected for new panels.
Url The URL of your Prometheus server, e.g. http://prometheus.example.org:9090.
Access Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser.
Basic Auth Enable basic authentication to the Prometheus data source.
User User name for basic authentication.
Password Password for basic authentication.
Scrape interval Set this to the typical scrape and evaluation interval configured in Prometheus. Defaults to 15s.
Custom Query Parameters Add custom parameters to the Prometheus query URL. For example timeout, partial_response, dedup, or max_source_resolution. Multiple parameters should be concatenated together with an ‘&'.

Prometheus query editor

Open a graph in edit mode by clicking the title > Edit (or by pressing e key while hovering over panel).

Name Description
Query expression Prometheus query expression, check out the Prometheus documentation.
Legend format Controls the name of the time series, using name or pattern. For example {{hostname}} is replaced with the label value for the label hostname.
Min step An additional lower limit for the step parameter of Prometheus range queries and for the $__interval variable. The limit is absolute and not modified by the Resolution setting.
Resolution 1/1 sets both the $__interval variable and the step parameter of Prometheus range queries such that each pixel corresponds to one data point. For better performance, lower resolutions can be picked. 1/2 only retrieves a data point for every other pixel, and 1/10 retrieves one data point per 10 pixels. Note that both Min time interval and Min step limit the final value of $__interval and step.
Metric lookup Search for metric names in this input field.
Format as Switch between Table, Time series or Heatmap. Table will only work in the Table panel. Heatmap is suitable for displaying metrics of the Histogram type on a Heatmap panel. Under the hood, it converts cumulative histograms to regular ones and sorts series by the bucket bound.
Instant Perform an “instant” query, to return only the latest value that Prometheus has scraped for the requested time series. Instant queries return results much faster than normal range queries. Use them to look up label sets.
Min time interval This value multiplied by the denominator from the Resolution setting sets a lower limit to both the $__interval variable and the step parameter of Prometheus range queries. Defaults to Scrape interval as set in the data source options.

Note: Grafana modifies the request dates for queries to align them with the dynamically calculated step. This ensures consistent display of metrics data, but it can result in a small gap of data at the right edge of a graph.

Instant queries

The Prometheus data source allows you to run “instant” queries, which query only the latest value. You can visualize the results in a table panel to see all available labels of a timeseries.

Instant query results are made up only of one data point per series but can be shown in the graph panel with the help of series overrides. To show them in the graph as a latest value point, add a series override and select Points > true. To show a horizontal line across the whole graph, add a series override and select Transform > constant.

Support for constant series overrides is available from Grafana v6.4

Templating

Instead of hard-coding things like server, application and sensor name in your metric queries, you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data being displayed in your dashboard.

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

Query variable

Variable of the type Query allows you to query Prometheus for a list of metrics, labels or label values. The Prometheus data source plugin provides the following functions you can use in the Query input field.

Name Description
label_names() Returns a list of label names.
label_values(label) Returns a list of label values for the label in every metric.
label_values(metric, label) Returns a list of label values for the label in the specified metric.
metrics(metric) Returns a list of metrics matching the specified metric regex.
query_result(query) Returns a list of Prometheus query result for the query.

For details of what metric names, label names and label values are please refer to the Prometheus documentation.

Using interval and range variables

Support for $__range, $__range_s and $__range_ms only available from Grafana v5.3

You can use some global built-in variables in query variables; $__interval, $__interval_ms, $__range, $__range_s and $__range_ms, see Global built-in variables for more information. These can be convenient to use in conjunction with the query_result function when you need to filter variable queries since

label_values function doesn’t support queries.

Make sure to set the variable’s refresh trigger to be On Time Range Change to get the correct instances when changing the time range on the dashboard.

Example usage:

Populate a variable with the busiest 5 request instances based on average QPS over the time range shown in the dashboard:

Query: query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))
Regex: /"([^"]+)"/

Populate a variable with the instances having a certain state over the time range shown in the dashboard, using $__range_s:

Query: query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
Regex:

Using variables in queries

There are two syntaxes:

  • $<varname> Example: rate(http_requests_total{job=~”$job”}[5m])
  • [[varname]] Example: rate(http_requests_total{job=~”[[job]]"}[5m])

Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. When the Multi-value or Include all value options are enabled, Grafana converts the labels from plain text to a regex compatible string. Which means you have to use =~ instead of =.

Annotations

Annotations allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.

Prometheus supports two ways to query annotations.

The step option is useful to limit the number of events returned from your query.

Get Grafana metrics into Prometheus

Grafana exposes metrics for Prometheus on the /metrics endpoint. We also bundle a dashboard within Grafana so you can get started viewing your metrics faster. You can import the bundled dashboard by going to the data source edit page and click the dashboard tab. There you can find a dashboard for Grafana and one for Prometheus. Import and start viewing all the metrics!

For detailed instructions, refer to Internal Grafana metrics.

Provision the Prometheus data source

You can configure data sources using config files with Grafana’s provisioning system. Read more about how it works and all the settings you can set for data sources on the provisioning docs page

Here are some provisioning examples for this data source:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090