Grafana Enterprise Metrics configuration
Grafana Enterprise Metrics utilizes a configuration file that is a superset of the Cortex configuration file.
The following extensions are available in the Grafana Enterprise Metrics configuration:
You can enable the built-in, token based authentication mechanism in Grafana Enterprise Metrics by adding the following code to your configuration file:
auth: type: enterprise
Admin backend storage
You need to configure Grafana Enterprise Metrics with a bucket that stores administration objects such as clusters, access policies, tokens, or licenses. Ideally, this bucket is separate from the bucket that stores the TSDB blocks.
If you do not configure the object-storage bucket or do so incorrectly, Grafana Enterprise Metrics is unable to find its license and warnings will be logged. Failure to find a license is not a fatal error to minimise operational disruption in the case of misconfiguration or license expiry.
The client that is used is configured using the same configuration options as configuring a blocks storage backend. All of the clients supported for blocks storage are also supported for the admin API.
To verify that you have configured the object-storage bucket correctly, see either the S3 / Minio backend if you are using an S3 compatible API, or Google Cloud Storage (GCS) backend if you are using Google Cloud Storage.
Below are snippets that can be included in your configuration to set up each of the common object storage backends:
S3 / Minio backend
admin_client: storage: type: s3 s3: # The S3 bucket endpoint. It could be an AWS S3 endpoint listed at # https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an # S3-compatible service in hostname:port format. [endpoint: <string> | default = ""] # S3 bucket name [bucket_name: <string> | default = ""] # S3 secret access key [secret_access_key: <string> | default = ""] # S3 access key ID [access_key_id: <string> | default = ""] # If enabled, use http:// for the S3 endpoint instead of https://. This could # be useful in local dev/test environments while using an S3-compatible # backend storage, like Minio. [insecure: <boolean> | default = false]
To configure dynamic HTTP headers for the admin storage’s S3 / Minio backend, the following configuration can be used. Note that the poll interval controls how often the file is reloaded and there is a section on the format of the dynamic headers file.
admin_client: type: s3 storage: s3: header_map_file_path: <path to header file> header_map_poll_interval: <duration string>
Google Cloud Storage (GCS) backend
admin_client: storage: type: gcs gcs: # GCS bucket name [bucket_name: <string> | default = ""] # JSON representing either a Google Developers Console client_credentials.json # file or a Google Developers service account key file. If empty, fallback to # Google default logic. [service_account: <string> | default = ""]
Grafana Enterprise Metrics extends Cortex’s blocks storage configuration by those features:
Rate limiting of list calls
List calls to a storage bucket can be rate limited. The
blocks-storage.bucket-rate-limit.limit configuration option specifies how often per second a list call can be made before being rate limited. The
blocks-storage.bucket-rate-limit.burst configuration option specifies the burst size. If
blocks-storage.bucket-rate-limit.limit is smaller than or is equal to zero (the default) no rate limiting is applied. The snippet below shows how this can be set in your configuration file for an S3 backend:
blocks_storage: backend: s3 bucket_rate_limit: 100 bucket_rate_limit_burst: 1
Dynamic headers for S3 / Minio backend
To configure dynamic HTTP headers for the blocks storage’s S3 / Minio backend, the following configuration can be used. Note that the poll interval controls how often the file is reloaded and there is a section on the format of the dynamic headers file.
blocks_storage: backend: s3 s3: header_map_file_path: <path to header file> header_map_poll_interval: <duration string>
Grafana Enterprise Metrics extends Cortex’s
To enable time-based sharding in the compactor, use the following configuration option:
compactor: sharding_strategy: time-sharding compaction_concurrency: 4
When the compactor sharding strategy is
time-sharding, a single compactor instance can parallelize the compaction of multiple groups of blocks up to and including
compaction_concurrency concurrent compactions.
The workflow of a
time-sharding compaction strategy is as follows:
- For each tenant that belongs to the compactor’s shard, the following steps occur:
- Find groups of compactable blocks, where each group’s time range doesn’t overlap with other groups
- Concurrently compact the groups of blocks, up until
- Repeat until the tenant has no remaining compactable blocks.
- Repeat until the compactor has compacted all of the tenants that belong to its shard.
The Ruler is an optional service that executes PromQL queries in order to record rules and alerts. The ruler requires backend storage for the recording rules and alerts for each tenant.
Grafana Enterprise Metrics extends Cortex’s upstream ruler configuration by those features:
The following configuration options can be used to enable remote write rule groups and specify a desired directory to store the generated write-ahead log (WAL). To learn more see the remote-write rule forwarding documentation
ruler: remote_write: enabled: <bool. defaults to false> wal_dir: <directory path. defaults to ./wal>
Dynamic headers for S3 / Minio backend
To configure dynamic HTTP headers for the ruler’s S3 / Minio backend, the following configuration can be used. Note that the poll interval controls how often the file is reloaded and there is section on the format of the dynamic headers file.
ruler: storage: s3: header_map_file_path: <filename> header_map_poll_interval: <duration. defaults to 1m>
Dynamic headers configuration
A dynamic headers configuration file allows to configure HTTP headers, which can be dynamically changed during the runtime of Grafana Enterprise Metrics. It uses the YAML syntax and its format looks like this:
<header-name>: "header-value" <second-header-name>: "second-header-value"
An example of specifying a bearer token for authentication could look like this:
Authorization: "Bearer my-secret-bearer-token"
For information about the gateway and its configuration, refer to Gateway.
Related Metrics Enterprise resources
Running Prometheus-as-a-service with Grafana Enterprise Metrics
Introducing Grafana Enterprise Metrics (GME), a simple and scalable Prometheus service that is seamless to use, simple to maintain, and supported by Grafana Labs.
How Robinhood scaled from 100M to 700M time series with Grafana Enterprise Metrics
In this GrafanaCONline session, the Robinhood team tells how GME (GameStop) led to GEM (Grafana Enterprise Metrics).
Benchmarking Grafana Enterprise Metrics for horizontally scaling Prometheus up to 500 million active series
We stress-tested GEM to show how it horizontally scaled. One takeaway: Hardware usage scales linearly up to 500 million active series.