Provision Grafana

This is documentation for the next version of Grafana. For the latest stable release, go to the latest version.

Enterprise Open source

Provision Grafana

Grafana has an active provisioning system that uses configuration files. This makes GitOps more natural since data sources and dashboards can be defined using files that can be version controlled.

Configuration file

Refer to Configuration for more information on what you can configure in grafana.ini.

Configuration file locations

Grafana reads its default configuration from <WORKING DIRECTORY>/conf/defaults.ini. By default, Grafana reads custom configuration from <WORKING DIRECTORY>/conf/custom.ini. You can override the custom configuration path with the --config option.

Note

The Deb and RPM packages install the configuration file at /etc/grafana/grafana.ini. The Grafana init.d script sets the --config option to that path.

Use environment variables

You can use environment variable lookups in all provisioning configuration. The syntax for an environment variable is $ENV_VAR_NAME or ${ENV_VAR_NAME}. If the environment variable value has a $ in it (for example, Pa$sw0rd), use the $ENV_VAR_NAME syntax to avoid double expansion. You can only use environment variables for configuration values and not for keys or bigger parts of the configuration file structure.

You can use environment variables in dashboard provisioning configuration but not the dashboard definition files themselves.

The following example looks up the data source URL port, user, and password using environment variables:

yaml
datasources:
  - name: Graphite
    url: http://localhost:$PORT
    user: $USER
    secureJsonData:
      password: $PASSWORD

To escape a literal $ in your provisioning file values, use $$.

Configuration management tools

The Grafana community maintains libraries for many popular configuration management tools.

Data sources

You can manage data sources in Grafana by adding YAML configuration files in the provisioning/datasources directory. Each configuration file contains a list of data sources, under the datasources key, to add or update during startup. If the data source already exists, Grafana reconfigures it to match the provisioned configuration file.

You can also list data sources to automatically delete, using the key deleteDatasources. Grafana deletes the data sources listed in deleteDatasources before adding or updating those in the datasources list.

You can configure Grafana to automatically delete provisioned data sources when they’re removed from the provisioning file. To do so, add prune: true to the root of your data source provisioning file. With this configuration, Grafana also removes the provisioned data sources if you remove the provisioning file entirely.

Run multiple Grafana instances

If you run multiple instances of Grafana, add a version number to each data source in the configuration and increase it when you update the configuration. Grafana only updates data sources with the same or lower version number than the one set in the configuration file. This prevents old configurations from overwriting newer ones if you have different versions of the datasource.yaml file that don’t define version numbers, and then restart instances at the same time.

Example data source configuration file

This example provisions a Graphite data source:

yaml
# Configuration file version
apiVersion: 1

# List of data sources to delete from the database.
deleteDatasources:
  - name: Graphite
    orgId: 1

# Mark provisioned data sources for deletion if they are no longer in a provisioning file.
# It takes no effect if data sources are already listed in the deleteDatasources section.
prune: true

# List of data sources to insert/update depending on what's
# available in the database.
datasources:
  # <string, required> Sets the name you use to refer to
  # the data source in panels and queries.
  - name: Graphite
    # <string, required> Sets the data source type.
    type: graphite
    # <string, required> Sets the access mode, either
    # proxy or direct (Server or Browser in the UI).
    # Some data sources are incompatible with any setting
    # but proxy (Server).
    access: proxy
    # <int> Sets the organization id. Defaults to orgId 1.
    orgId: 1
    # <string> Sets a custom UID to reference this
    # data source in other parts of the configuration.
    # If not specified, Grafana generates one.
    uid: my_unique_uid
    # <string> Sets the data source's URL, including the
    # port.
    url: http://localhost:8080
    # <string> Sets the database user, if necessary.
    user:
    # <string> Sets the database name, if necessary.
    database:
    # <bool> Enables basic authorization.
    basicAuth:
    # <string> Sets the basic authorization username.
    basicAuthUser:
    # <bool> Enables credential headers.
    withCredentials:
    # <bool> Toggles whether the data source is pre-selected
    # for new panels. You can set only one default
    # data source per organization.
    isDefault:
    # <map> Fields to convert to JSON and store in jsonData.
    jsonData:
      # <string> Defines the Graphite service's version.
      graphiteVersion: '1.1'
      # <bool> Enables TLS authentication using a client
      # certificate configured in secureJsonData.
      tlsAuth: true
      # <bool> Enables TLS authentication using a CA
      # certificate.
      tlsAuthWithCACert: true
    # <map> Fields to encrypt before storing in jsonData.
    secureJsonData:
      # <string> Defines the CA cert, client cert, and
      # client key for encrypted authentication.
      tlsCACert: '...'
      tlsClientCert: '...'
      tlsClientKey: '...'
      # <string> Sets the database password, if necessary.
      password:
      # <string> Sets the basic authorization password.
      basicAuthPassword:
    # <int> Sets the version. Used to compare versions when
    # updating. Ignored when creating a new data source.
    version: 1
    # <bool> Allows users to edit data sources from the
    # Grafana UI.
    editable: false

For provisioning examples of specific data sources, refer to that data source’s documentation.

JSON data

Not all data sources have the same configuration settings, only the most common ones are fields in the data source provisioning file. To provision the rest of a data source’s settings, include them as JSON in the jsonData field.

Common settings in the built-in core data sources include:

NameTypeData sourceDescription
tlsAuthbooleanHTTP*, MySQLEnable TLS authentication using client cert configured in secure JSON data
tlsAuthWithCACertbooleanHTTP*, MySQL, PostgreSQLEnable TLS authentication using CA cert
tlsSkipVerifybooleanHTTP*, MySQL, PostgreSQL, MSSQLControls whether a client verifies the server’s certificate chain and host name.
serverNamestringHTTP*, MSSQLOptional. Controls the server name used for certificate common name/subject alternative name verification. Defaults to using the data source URL.
timeoutstringHTTP*Request timeout in seconds. Overrides dataproxy.timeout option
graphiteVersionstringGraphiteGraphite version
timeIntervalstringPrometheus, Elasticsearch, InfluxDB, MySQL, PostgreSQL and MSSQLLowest interval/step value that should be used for this data source.
httpModestringInfluxDBHTTP Method. ‘GET’, ‘POST’, defaults to GET
maxSeriesnumberInfluxDBMax number of series/tables that Grafana processes
httpMethodstringPrometheusHTTP Method. ‘GET’, ‘POST’, defaults to POST
customQueryParametersstringPrometheusQuery parameters to add, as a URL-encoded string.
manageAlertsbooleanPrometheus and LokiManage alerts via Alerting UI
alertmanagerUidstringPrometheus and LokiUID of Alertmanager that manages Alert for this data source.
timeFieldstringElasticsearchWhich field that should be used as timestamp
intervalstringElasticsearchIndex date time format. nil(No Pattern), ‘Hourly’, ‘Daily’, ‘Weekly’, ‘Monthly’ or ‘Yearly’
logMessageFieldstringElasticsearchWhich field should be used as the log message
logLevelFieldstringElasticsearchWhich field should be used to indicate the priority of the log message
maxConcurrentShardRequestsnumberElasticsearchMaximum number of concurrent shard requests that each sub-search request executes per node
sigV4AuthbooleanElasticsearch and PrometheusEnable usage of SigV4
sigV4AuthTypestringElasticsearch and PrometheusSigV4 auth provider. default/credentials/keys
sigV4ExternalIdstringElasticsearch and PrometheusOptional SigV4 External ID
sigV4AssumeRoleArnstringElasticsearch and PrometheusOptional SigV4 ARN role to assume
sigV4RegionstringElasticsearch and PrometheusSigV4 AWS region
sigV4ProfilestringElasticsearch and PrometheusOptional SigV4 credentials profile
authTypestringAmazon CloudWatchAuth provider. default/credentials/keys
externalIdstringAmazon CloudWatchOptional External ID
assumeRoleArnstringAmazon CloudWatchOptional ARN role to assume
defaultRegionstringAmazon CloudWatchOptional default AWS region
customMetricsNamespacesstringAmazon CloudWatchNamespaces of Custom Metrics
profilestringAmazon CloudWatchOptional credentials profile
tsdbVersionstringOpenTSDBVersion
tsdbResolutionstringOpenTSDBResolution
sslmodestringPostgreSQLSSL mode. ‘disable’, ‘require’, ‘verify-ca’ or ‘verify-full’
tlsConfigurationMethodstringPostgreSQLSSL certificate configuration, either by ‘file-path’ or ‘file-content’
sslRootCertFilestringPostgreSQL, MSSQLSSL server root certificate file, must be readable by the Grafana user
sslCertFilestringPostgreSQLSSL client certificate file, must be readable by the Grafana user
sslKeyFilestringPostgreSQLSSL client key file, must be readable by only the Grafana user
encryptstringMSSQLDetermines SSL encryption handling. Options include: disable - data sent between client and server is not encrypted; false - data sent between client and server is not encrypted beyond the login packet; true - data sent between client and server is encrypted. Default is false.
postgresVersionnumberPostgreSQLPostgres version as a number (903/904/905/906/1000) meaning v9.3, v9.4, …, v10
timescaledbbooleanPostgreSQLEnable usage of TimescaleDB extension
maxOpenConnsnumberMySQL, PostgreSQL and MSSQLMaximum number of open connections to the database (Grafana v5.4+)
maxIdleConnsnumberMySQL, PostgreSQL and MSSQLMaximum number of connections in the idle connection pool (Grafana v5.4+)
connMaxLifetimenumberMySQL, PostgreSQL and MSSQLMaximum amount of time in seconds a connection may be reused (Grafana v5.4+)
keepCookiesarrayHTTP*Cookies that needs to be passed along while communicating with data sources
prometheusVersionstringPrometheusThe version of the Prometheus data source, such as 2.37.0, 2.24.0
prometheusTypestringPrometheusPrometheus database type. Options are Prometheus, Cortex, Mimir orThanos.
cacheLevelstringPrometheusDetermines the duration of the browser cache. Valid values include: Low, Medium, High, and None.
incrementalQueryingstringPrometheusExperimental: Turn on incremental querying to enhance dashboard reload performance with slow data sources
incrementalQueryOverlapWindowstringPrometheusExperimental: Configure incremental query overlap window. Requires a valid duration string, for example, 180s or 15m Default value is 10m (10 minutes).
disableRecordingRulesbooleanPrometheusExperimental: Turn off Prometheus recording rules
implementationstringAlertmanagerThe implementation of the Alertmanager data source, such as prometheus, cortex or mimir
handleGrafanaManagedAlertsbooleanAlertmanagerWhen enabled, Grafana-managed alerts are sent to this Alertmanager

Note

Data sources tagged with HTTP* communicate using the HTTP protocol, which includes all core data source plugins except MySQL, PostgreSQL, and MSSQL.

For examples of specific data sources’ JSON data, refer to that data source’s documentation.

Secure JSON data

Secure JSON data is a map of settings encrypted with a secret key. The encryption hides the JSON data from the users of Grafana. You should use secure JSON data to store TLS certificates and passwords for data source HTTP requests. All of these settings are optional.

NameTypeData sourceDescription
tlsCACertstringHTTP*, MySQL, PostgreSQLCA cert for out going requests
tlsClientCertstringHTTP*, MySQL, PostgreSQLTLS Client cert for outgoing requests
tlsClientKeystringHTTP*, MySQL, PostgreSQLTLS Client key for outgoing requests
passwordstringHTTP*, MySQL, PostgreSQL, MSSQLpassword
basicAuthPasswordstringHTTP*password for basic authentication
accessKeystringAmazon CloudWatchAccess key for connecting to Amazon CloudWatch
secretKeystringAmazon CloudWatchSecret key for connecting to Amazon CloudWatch
sigV4AccessKeystringElasticsearch and PrometheusSigV4 access key. Required when using keys auth provider
sigV4SecretKeystringElasticsearch and PrometheusSigV4 secret key. Required when using keys auth provider

Note

The HTTP* tag denotes data sources that communicate using the HTTP protocol, including all core data source plugins except MySQL, PostgreSQL, and MSSQL.

Custom HTTP headers for data sources

You can add HTTP headers to all requests sent to data sources managed by Grafana provisioning Configure the header name in the jsonData field and the header value in secureJsonData. The following example sets the HeaderName header to have the value HeaderValue and the Authorization header to have the value Bearer XXXXXXXXX:

yaml
apiVersion: 1

datasources:
  - name: Graphite
    jsonData:
      httpHeaderName1: 'HeaderName'
      httpHeaderName2: 'Authorization'
    secureJsonData:
      httpHeaderValue1: 'HeaderValue'
      httpHeaderValue2: 'Bearer XXXXXXXXX'

Plugins

You can manage plugin applications in Grafana by adding one or more YAML configuration files in the provisioning/plugins directory. Each configuration file contains a list of apps that Grafana configures during start up. Grafana configures each app to use the configuration in the file.

Note

This feature enables you to provision plugin configurations, not the plugins themselves. You must have already installed the plugin to use plugin configuration provisioning.

Example plugin configuration file

yaml
apiVersion: 1

apps:
  # <string> the type of app, plugin identifier. Required
  - type: raintank-worldping-app
    # <int> Org ID. Default to 1, unless org_name is specified
    org_id: 1
    # <string> Org name. Overrides org_id unless org_id not specified
    org_name: Main Org.
    # <bool> disable the app. Default to false.
    disabled: false
    # <map> fields that will be converted to json and stored in jsonData. Custom per app.
    jsonData:
      # key/value pairs of string to object
      key: value
    # <map> fields that will be converted to json, encrypted and stored in secureJsonData. Custom per app.
    secureJsonData:
      # key/value pairs of string to string
      key: value

Dashboards

You can manage dashboards in Grafana by adding one or more YAML configuration files in the provisioning/dashboards directory. Each configuration file contains a list of providers that Grafana uses to load dashboards from the local filesystem.

Example dashboard configuration file

yaml
apiVersion: 1

providers:
  # <string> an unique provider name. Required
  - name: 'a unique provider name'
    # <int> Org id. Default to 1
    orgId: 1
    # <string> name of the dashboard folder.
    folder: ''
    # <string> folder UID. will be automatically generated if not specified
    folderUid: ''
    # <string> provider type. Default to 'file'
    type: file
    # <bool> disable dashboard deletion
    disableDeletion: false
    # <int> how often Grafana will scan for changed dashboards
    updateIntervalSeconds: 10
    # <bool> allow updating provisioned dashboards from the UI
    allowUiUpdates: false
    options:
      # <string, required> path to dashboard files on disk. Required when using the 'file' type
      path: /var/lib/grafana/dashboards
      # <bool> use folder names from filesystem to create folders in Grafana
      foldersFromFilesStructure: true

When Grafana starts, it updates or creates all dashboards found in the configured path. It later polls that path every updateIntervalSeconds for updates to the dashboard files and updates its database.

Note

Grafana installs dashboards at the root level if you don’t set the folder field.

Make changes to a provisioned dashboard

You can make changes to a provisioned dashboard in the Grafana UI but its not possible to automatically save the changes back to the provisioning source. If allowUiUpdates is set to true and you make changes to a provisioned dashboard, when you save the dashboard, Grafana persists the changes to its database.

Caution

If you save a provisioned dashboard in the UI and then later update the provisioning source, Grafana always overwrites the database dashboard with the one from the provisioning file. Grafana ignores the version property in the JSON file, even if it’s lower than the dashboard in the database.

Caution

If you save a provisioned dashboard in the UI and remove the provisioning source, Grafana deletes the dashboard in the database unless you have set the option disableDeletion to true.

If you set allowUiUpdates to false, you can’t save changes to a provisioned dashboard. When you try to save changes to a provisioned dashboard, Grafana brings up a Cannot save provisioned dashboard dialog box.

Grafana offers options to export the JSON definition of a dashboard. Use either Copy JSON to Clipboard or Save JSON to file to sync your dashboard changes back to the provisioning source. Grafana removes the id field from the dashboard JSON to help the provisioning workflow.

The following screenshot illustrates this behavior.

Reusable dashboard URLs

If the dashboard in the JSON file contains an UID, Grafana updates that the dashboard with that UID in the database. This lets you migrate dashboards between Grafana instances and keep consistent dashboard URLs. When Grafana starts, it creates or updates all dashboards available in the configured folders.

Caution

You can overwrite existing dashboards with provisioning.

Be careful not to reuse the same title multiple times within a folder or uid within the same Grafana instance to avoid inconsistent behavior.

Provision folders structure from filesystem to Grafana

If you already store your dashboards using folders in a Git repository or on a filesystem, and want to have the same folder names in the Grafana menu, use foldersFromFilesStructure option.

For example, to replicate the following dashboards structure from the filesystem to Grafana:

/etc/dashboards
├── /server
│   ├── /common_dashboard.json
│   └── /network_dashboard.json
└── /application
    ├── /requests_dashboard.json
    └── /resources_dashboard.json

use the following provisioning configuration file:

yaml
apiVersion: 1

providers:
  - name: dashboards
    type: file
    updateIntervalSeconds: 30
    options:
      path: /etc/dashboards
      foldersFromFilesStructure: true

Grafana creates the server and application folders in the UI.

To use foldersFromFilesStructure, you must unset the folder and folderUid options.

To provision dashboards to the root level, store them in the root of your path.

Note

This feature doesn’t let you create nested folder structures, where you have folders within folders.

Alerting

For information on provisioning Grafana Alerting, refer to Provision Grafana Alerting resources.

Supported settings

The following sections detail the supported settings and secure settings for each alert notification type. In the provisioning YAML use settings for settings and secure_settings for secure settings. Grafana encrypts secure settings in the database.

Alert notification pushover

NameSecure setting
apiTokenyes
userKeyyes
device
priority
okPriority
retry
expire
sound
okSound

Alert notification discord

NameSecure setting
urlyes
avatar_url
content
use_discord_username

Alert notification slack

NameSecure setting
urlyes
recipient
username
icon_emoji
icon_url
uploadImage
mentionUsers
mentionGroups
mentionChannel
tokenyes
color

Alert notification victorops

Name
url
autoResolve

Alert notification kafka

Name
kafkaRestProxy
kafkaTopic

Alert notification LINE

NameSecure setting
tokenyes

Alert notification MQTT

NameSecure setting
brokerUrl
clientId
topic
messageFormat
username
passwordyes
retain
qos
tlsConfig
TLS configuration
NameSecure setting
insecureSkipVerify
clientCertificateyes
clientKeyyes
caCertificateyes

Alert notification pagerduty

NameSecure setting
integrationKeyyes
autoResolve

Alert notification sensu

NameSecure setting
url
source
handler
username
passwordyes

Alert notification sensugo

NameSecure setting
url
apikeyyes
entity
check
handler
namespace

Alert notification prometheus-alertmanager

NameSecure setting
url
basicAuthUser
basicAuthPasswordyes

Alert notification teams

NameSecure setting
url

Alert notification dingding

NameSecure setting
url

Alert notification email

NameSecure setting
singleEmail
addresses

Alert notification hipchat

NameSecure setting
url
apikey
roomid

Alert notification opsgenie

NameSecure setting
apiKeyyes
apiUrl
autoClose
overridePriority
sendTagsAs

Alert notification telegram

NameSecure setting
bottokenyes
chatid
uploadImage

Alert notification threema

NameSecure setting
gateway_id
recipient_id
api_secretyes

Alert notification webhook

NameSecure setting
url
http_method
username
passwordyes
tls_config
hmac_config
TLS configuration
NameSecure setting
insecureSkipVerify
clientCertificateyes
clientKeyyes
caCertificateyes
HMAC signature configuration
NameSecure setting
secretyes
header
timestampHeader

Alert notification googlechat

NameSecure setting
url

Alert notification Cisco Webex Teams

NameSecure setting
message
room_id
api_url
bot_tokenyes

Grafana Enterprise

Grafana Enterprise supports: