To set up a Grafana Metrics Enterprise (GME) cluster, use the following recommended practices:
- Single-binary deployments limit the complexity of the system while still allowing for horizontal scaling.
- Memberlist gossip to synchronize responsibility between processes in your cluster. Gossip performs better at scale for managing the state of the system and removes the need to deploy Consul or etcd to manage state.
- Blocks based storage using an object storage backend for persistence.
- Object storage: Amazon S3 or S3-compatible, Google Cloud Storage (GCS), or Azure Blob Storage
- A Grafana Metric Enterprise license file (license.jwt)
- Provisioned compute capacity
If you are setting up a cluster to familiarize yourself with GME, provisioning compute capacity might be optional depending on your use case. For production purposes, it is required. For capacity planning, see Cortex’s Capacity Planning.
Get a license
To get a license, contact a Grafana Labs representative and give them the cluster name that you want to associate with the license.
A cluster name must meet the following criteria:
- is 3 to 63 characters long
- contains lowercase letters, numbers, underscores (_), or hyphens (-)
- begins with a letter or number
- ends with a letter or number
Prepare configuration file
This guide will focus on using an unauthenticated S3 bucket. See the admin storage documentation for how to configure storage with different backends and authentication options.
The general configuration format is compatible with Cortex’s configuration file format and its documentation provides a more detailed explanation of all the configuration options.
When you start your Grafana Metrics Enterprise cluster, ensure that the license file (
license.jwt) is present on the local filesystem and the path to the license file is specified in the GME config YAML file.
Take the following config and store it locally under metrics-enterprise.yaml:
auth: type: enterprise target: all # TODO: Ensure the cluster name is set to match your Grafana Labs License File cluster_name: <my-cluster> license: path: /etc/metrics-enterprise/license.jwt # Update this path to match the location where you stored the license file. admin_client: storage: # TODO: Configure the admin bucket according to your environment type: s3 s3: endpoint: minio:9000 bucket_name: metrics-enterprise-admin access_key_id: metrics-enterprise secret_access_key: supersecret insecure: true distributor: shard_by_all_labels: true pool: health_check_ingesters: true ingester_client: grpc_client_config: grpc_compression: gzip frontend_worker: match_max_concurrent: true memberlist: abort_if_cluster_join_fails: false bind_port: 7946 ingester: lifecycler: num_tokens: 512 ring: kvstore: store: memberlist # replication_factor configures the number of ingesters each sample will be stored # on. We recommend that you use a replication factor of 3. However, # if you plan on deploying GME on less than three nodes, lower this # value to 1. replication_factor: 3 blocks_storage: tsdb: dir: /tmp/cortex/tsdb bucket_store: sync_dir: /tmp/cortex/tsdb-sync # TODO: Configure the tsdb bucket according to your environment. backend: s3 s3: endpoint: minio:9000 # set a valid s3 hostname bucket_name: metrics-enterprise-tsdb # set a value for an existing bucket at the provided s3 address. access_key_id: metrics-enterprise # set an access key id for the provided s3 bucket. secret_access_key: supersecret # set an access key secret for the provided s3 bucket. insecure: true # avoid validating TLS certificates of the s3 backend, this is useful for setting up an initial testing environment. storage: engine: blocks
cluster_nameconfiguration option to match the one used when requesting the Grafana Labs License File.
Update the bucket configuration options with the appropriate options for your object store. Two different buckets that are used in the above config:
- Admin bucket stores the configuration and the license information of your Metrics Enterprise cluster.
- TSDB bucket is where all time series data is persisted. Prometheus' TSDB format is used.
Bootstrap and deploy cluster
In order to use Metrics Enterprise, you will need to bootstrap your cluster first. This process will…
- … parse and validate your license file and persist the license into your object storage backend. Any
metrics-enterpriseprocess with access to this backend will be able to utilize this license during startup.
- … create a cluster config object in the provided object storage backend to store metadata about your cluster and to allow for instances to be provisioned on the cluster.
- … create an initial
adminscoped API token for accessing and configuring the cluster through the Admin API or the Metrics Enterprise Grafana plugin.
The generated token will be important to use the cluster later, so make sure to note this admin token down.
Following that you can start deploying your cluster’s replicas.
Both the bootstrap and the deployment require slightly different actions depending on your deployment target. Continue with the specific guides for your environment: