Deploy GEM on a Linux host
This guide provides a step by step process for installing Grafana Enterprise Metrics on a Linux machine. It assumes you have access to a Linux machine and the permissions required to deploy a service with network and filesystem access. At the end of this guide you will have deployed a single GEM instance on a single node.
In order to follow this guide you will the following:
- A valid Grafana Labs license with an associated GEM cluster name.
Setup an object storage bucket
To begin, you need access to an object storage backend. GEM uses object storage as a backend to store time series data as well as data related to the state of the system. This topic assumes that you are using Amazon S3 on the AWS
us-east-1 region. If you plan on using a different region or object storage service, update the storage fields in the configuration file to match below to configure your desired service. Currently, the supported services are object storage backends that support the S3 API or Google GCS.
After you have provisioned an object storage backend, be sure to pre-create two buckets:
grafana-metrics-tsdb. Those buckets will be referenced in the configuration file of this guide.
Prepare your system
Before running GEM, it is recommended you setup a system user and prepare the required directories on the filesystem. Run the following commands on every node as the root user. To login as the root user run:
sudo su -
Add dedicated user and group
groupadd --system ge-metrics useradd --system --home-dir /var/lib/ge-metrics -g ge-metrics ge-metrics
Create directories and assign ownership
mkdir -p /etc/ge-metrics /var/lib/ge-metrics chown ge-metrics:ge-metrics /etc/ge-metrics chown ge-metrics:ge-metrics /var/lib/ge-metrics chmod 0750 /etc/ge-metrics /var/lib/ge-metrics
Copy the license file
You will need to ensure you license token file is copied to the path
Create a GEM configuration file
Copy the following YAML config to a file called
auth: type: enterprise target: all # TODO: Ensure the cluster name is set to match your Grafana Labs License File cluster_name: # <name of you cluster cluster> license: path: /etc/ge-metrics/license.jwt admin_client: storage: type: s3 s3: endpoint: s3.amazonaws.com bucket_name: grafana-metrics-admin access_key_id: # TODO: insert access key id here secret_access_key: # TODO: insert secret access key here distributor: shard_by_all_labels: true pool: health_check_ingesters: true memberlist: abort_if_cluster_join_fails: false bind_port: 7946 ingester: lifecycler: num_tokens: 512 ring: kvstore: store: inmemory replication_factor: 1 blocks_storage: tsdb: dir: /tmp/cortex/tsdb bucket_store: sync_dir: /tmp/cortex/tsdb-sync backend: s3 s3: endpoint: s3.amazonaws.com bucket_name: grafana-metrics-tsdb access_key_id: # TODO: insert access key id here secret_access_key: # TODO: insert secret access key here storage: engine: blocks
Next you will need to update the config file to configure the following:
cluster_namefield must be updated to the cluster name associated with your license.
- Update the
admin_client.storage.s3section to include access credentials for your object storage backend.
Download and configure the GEM binary
curl -Lo /usr/local/bin/ge-metrics https://dl.grafana.com/gem/releases/metrics-enterprise-1.5.1-linux-amd64 echo "b167c2e5d1152f4229f2a35aa02dba21ec2151037c60ec69d994475b9e0b27f0 /usr/local/bin/ge-metrics" | sha256sum -c chmod 0755 /usr/local/bin/ge-metrics setcap 'cap_net_bind_service=+ep' /usr/local/bin/ge-metrics
Set up systemd unit
Copy the following file to the path
[Unit] After=network.target [Service] User=ge-metrics Group=ge-metrics WorkingDirectory=/var/lib/ge-metrics ExecStart=/usr/local/bin/ge-metrics \ -config.file=/etc/ge-metrics/config.yaml [Install] WantedBy=default.target
Enable startup at boot timeand start Grafana Enterprise Metrics
systemctl daemon-reload systemctl enable ge-metrics.service systemctl start ge-metrics.service
Generate an admin token
Generate an admin token by running the following on a single node in the cluster:
su ge-metrics -c "/usr/local/bin/ge-metrics \ --config.file=/etc/ge-metrics/config.yaml \ --target=tokengen"
The output of the above command should contain a token:
Token created: YWRtaW4tcG9saWN5LWJvb3RzdHJhcC10b2tlbjo8Ujc1IzQyfXBfMjd7fDIwMDRdYVxgeXw=
Verify your cluster is working
To verify your cluster is working you can run the following command using the token you generated in the previous step. For example:
curl -u :YWRtaW4tcG9saWN5LWJvb3RzdHJhcC10b2tlbjo8Ujc1IzQyfXBfMjd7fDIwMDRdYVxgeXw= localhost/ready
After running the above command you should see the following output:
Now that you have a working GEM deployment locally, refer to setup up the Grafana Enterprise Metrics plugin for instructions on how to integrate your metrics cluster with Grafana and give you a UI to interact with the Admin API.
Related Metrics Enterprise resources
Running Prometheus-as-a-service with Grafana Enterprise Metrics
Introducing Grafana Enterprise Metrics (GME), a simple and scalable Prometheus service that is seamless to use, simple to maintain, and supported by Grafana Labs.
How Robinhood scaled from 100M to 700M time series with Grafana Enterprise Metrics
In this GrafanaCONline session, the Robinhood team tells how GME (GameStop) led to GEM (Grafana Enterprise Metrics).
Benchmarking Grafana Enterprise Metrics for horizontally scaling Prometheus up to 500 million active series
We stress-tested GEM to show how it horizontally scaled. One takeaway: Hardware usage scales linearly up to 500 million active series.