Documentation Grafana Cloud Connect to data sources Data sources Prometheus Configure the Prometheus data source

Configure the Prometheus data source

This document provides instructions for configuring the Prometheus data source and explains the available configuration options. Grafana includes built-in support for Prometheus, so you don’t need to install a plugin. For general information on adding a data source to Grafana, refer to Add a data source.

Before you begin

• You must have the Organization administrator role to add a data source. Administrators can also configure a data source via YAML files.

• Know which Prometheus-compatible database you are using.

• Familiarize yourself with your Prometheus security configuration and gather any necessary security certificates and client keys.

• Verify that data from Prometheus is being written to your Grafana instance.

Configure the data source using the UI

To add the Prometheus data source, complete the following steps:

1. Click Connections in the left-side menu.
2. Under Connections, click Add new connection.
3. Enter Prometheus in the search bar.
4. Click Prometheus data source.
5. Click Add new data source in the upper right.

Grafana takes you to the Settings tab where you will set up your Prometheus configuration.

Configuration options

Following is a list of configuration options for Prometheus.

• Name - The data source name. Sets the name you use to refer to the data source in panels and queries. Examples: prometheus-1, prom-metrics.
• Default - Toggle to select as the default name in dashboard panels. When you go to a dashboard panel this will be the default selected data source.

Connection:

• Prometheus server URL - The URL of your Prometheus server.

  If Prometheus is running locally, use http://localhost:9090. If it’s hosted on a networked server, provide the server’s URL along with the port where Prometheus is running. Example: http://prometheus.example.orgname:9090.

  Note

  When running Grafana and Prometheus in separate containers, localhost refers to each container’s own network namespace. This means that localhost:9090 points to port 9090 inside the Grafana container, not on the host machine.

  Use the IP address of the Prometheus container, or the hostname if you are using Docker Compose. Alternatively, you can use http://host.docker.internal:9090 to reference the host machine.

Authentication:

There are three authentication options for the Prometheus data source.

• Basic authentication - The most common authentication method.

  • User - The username you use to connect to the data source.
  • Password - The password you use to connect to the data source.

• Forward OAuth identity - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source.

• No authentication - Allows access to the data source without any authentication.

TLS settings:

• Add self-signed certificate - Check the box to authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file. Required for verifying self-signed TLS certificates.
  • CA certificate - Add your certificate.
• TLS client authentication - Check the box to enable TLS client authentication.
  • Server name - Add the server name, which is used to verify the hostname on the returned certificate.
  • Client certificate - The client certificate is generated from a Certificate Authority or its self-signed. Follow the instructions of the CA (Certificate Authority) to download the certificate file.
  • Client key - Add your client key, which can also be generated from a Certificate Authority (CA) or be self-signed. The client key encrypts data between the client and server.
• Skip TLS verify - Toggle on to bypass TLS certificate validation. Skipping TLS certificate validation is not recommended unless absolutely necessary or for testing purposes.

HTTP headers:

Pass along additional information and metadata about the request or response.

• Header - Add a custom header. This allows custom headers to be passed based on the needs of your Prometheus instance.
• Value - The value of the header.

Advanced settings:

Following are optional configuration settings you can configure for more control over your data source.

• Advanced HTTP settings:
  • Allowed cookies - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.
  • Timeout - The HTTP request timeout, must be in seconds.

Alerting:

• Manage alerts via Alerting UI -Toggled on by default. This enables data source-managed rules in Grafana Alerting for this data source. For Mimir, it enables managing data source-managed rules and alerts. For Prometheus, it only supports viewing existing rules and alerts, which are displayed as data source-managed. Change this by setting the default_manage_alerts_ui_toggle option in the grafana.ini configuration file.

Interval behavior:

• Scrape interval - Sets the standard scrape and evaluation interval in Prometheus. The default is 15s. This interval determines how often Prometheus scrapes targets. Set it to match the typical scrape and evaluation interval in your Prometheus configuration file. If you set a higher value than your Prometheus configuration, Grafana will evaluate data at this interval, resulting in fewer data points.
• Query timeout - Sets the Prometheus query timeout. The default is 60s. Without a timeout, complex or inefficient queries can run indefinitely, consuming CPU and memory resources.

Query editor:

• Default editor - Sets the default query editor. Options are Builder or Code. Builder mode helps you build queries using a visual interface. Code mode is geared for the experienced Prometheus user with prior expertise in PromQL. For more details on editor types refer to Prometheus query editor. You can switch easily code editors in the Query editor UI.
• Disable metrics lookup - Toggle on to disable the metrics chooser and metric and label support in the query field’s autocomplete. This can improve performance for large Prometheus instances.

Performance:

• Prometheus type - Select the type of your Prometheus-compatible database, such as Prometheus, Cortex, Mimir, or Thanos. Changing this setting will save your current configuration. Different database types support different APIs. For example, some allow regex matching for label queries to improve performance, while others provide a metadata API. Setting this incorrectly may cause unexpected behavior when querying metrics and labels. Refer to your Prometheus documentation to ensure you select the correct type.
• Cache level - Sets the browser caching level for editor queries. There are four options: Low, Medium, High, or None. Higher cache settings are recommended for high cardinality data sources.
• Incremental querying (beta) - Toggle on to enable incremental querying. Enabling this feature changes the default behavior of relative queries. Instead of always requesting fresh data from the Prometheus instance, Grafana will cache query results and only fetch new records. This helps reduce database and network load.
  • Query overlap window - If you are using incremental querying, specify a duration (e.g., 10m, 120s, or 0s). The default is 10m. This is a buffer of time added to incremental queries and this value is added to the duration of each incremental request.
• Disable recording rules (beta) - Toggle on to disable the recording rules. When recording rules are disabled, Grafana won’t fetch and parse recording rules from Prometheus, improving dashboard performance by reducing processing overhead..

Other settings:

• Custom query parameters - Add custom parameters to the Prometheus query URL, which allow for more control over how queries are executed. Examples: timeout, partial_response, dedup, or max_source_resolution. Multiple parameters should be joined using &.
• HTTP method - Select either the POST or GET HTTP method to query your data source. POSTis recommended and selected by default, as it supports larger queries. Select GET if you’re using Prometheus version 2.1 or older, or if your network restricts POST requests. Toggle on
• Use series endpoint - Enabling this option makes Grafana use the series endpoint (/api/v1/series) with the match[] parameter instead of the label values endpoint (/api/v1/label/<label_name>/values). While the label values endpoint is generally more performant, some users may prefer the series endpoint because it supports the POST method, whereas the label values endpoint only allows GET requests.

Exemplars:

Support for exemplars is available only for the Prometheus data source. For more information on exemplars refer to Introduction to exemplars. An exemplar is a trace that represents a specific measurement taken within a given time interval.

Click the + sign to add exemplars.

• Internal link - Toggle on to enable an internal link. This will display the data source selector, where you can choose the backend tracing data store for your exemplar data.
• URL - (Visible if you disable Internal link) Defines the external link’s URL trace backend. You can interpolate the value from the field by using the ${__value.raw} macro.
• Data source - (Visible whenInternal link is enabled.) Select the data source that the exemplar will link to from the drop-down.
• URL label - Adds a custom display label to override the value of the Label name field.
• Label name - The name of the field in the labels object used to obtain the traceID property.
• Remove exemplar link - Click the X to remove existing links.

You can add multiple exemplars.

• Private data source connect - Only for Grafana Cloud users. Private data source connect, or PDC, allows you to establish a private, secured connection between a Grafana Cloud instance, or stack, and data sources secured within a private network. Click the drop-down to locate the URL for PDC. For more information regarding Grafana PDC refer to Private data source connect (PDC) and Configure Grafana private data source connect (PDC) for steps on setting up a PDC connection.

Click Manage private data source connect to be taken to your PDC connection page, where you’ll find your PDC configuration details.

After you have configured your Prometheus data source options, click Save & test at the bottom to test out your data source connection.

You should see a confirmation dialog box that says:

Successfully queried the Prometheus API.

Next, you can start to visualize data by building a dashboard, or by querying data in the Explore view.

You can also remove a connection by clicking Delete.

Provision the Prometheus data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to Provision Grafana.

Example of a Prometheus data source configuration:

```yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    jsonData:
      httpMethod: POST
      manageAlerts: true
      prometheusType: Prometheus
      prometheusVersion: 3.3.0
      cacheLevel: 'High'
      disableRecordingRules: false
      timeInterval: 10s   # Prometheus scrape interval
      incrementalQueryOverlapWindow: 10m
      exemplarTraceIdDestinations:
        # Field with internal link pointing to data source in Grafana.
        # datasourceUid value can be anything, but it should be unique across all defined data source uids.
        - datasourceUid: my_jaeger_uid
          name: traceID

        # Field with external link.
        - name: traceID
          url: 'http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Jaeger%22,%7B%22query%22:%22$${__value.raw}%22%7D%5D'
```

Azure authentication settings

The Prometheus data source works with Azure authentication. To configure Azure authentication refer to Configure Azure Active Directory (AD) authentication.

In Grafana Enterprise, you need to update the .ini configuration file. Refer to Configuration file location to locate your .ini file.

Add the following setting in the [auth] section of the .ini configuration file:

bash Copy

[auth]
azure_auth_enabled = true

Recording rules (beta)

You can configure the Prometheus data source to disable recording rules in the data source configuration or provisioning file under disableRecordingRules in jsonData.

Troubleshooting configuration issues

Refer to the following troubleshooting information as needed.

Data doesn’t appear in Metrics Drilldown:

If you have successfully tested the connection to a Prometheus data source or are sending metrics to Grafana Cloud and there is no metric data appearing in Explore, make sure you’ve selected the correct data source from the data source drop-down menu. When using remote_write to send metrics to Grafana Cloud, the data source name follows the convention grafanacloud-stackname-prom.