Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Grafana Enterprise Metrics documentation

Set up GEM

Deploy on Kubernetes

Enterprise

Deploy GEM on Kubernetes

This guide will layout a step by step approach to deploying a Grafana Enterprise Metrics cluster on an existing Kubernetes namespace. This guide assumes you have a working Kubernetes cluster and the ability to deploy to that cluster using the kubectl tool. At the end of this guide you should have a functional GEM cluster running on Kubernetes using Minio as a storage backend. If you do not currently have access to a Kubernetes cluster consider following Linux deployment guide instead. See GEM hardware requirements for guidance on the type of hardware to use when running a GEM cluster.

Deploy Minio

For this guide we will be using Minio as our object storage backend. Minio is a open source S3 compatible object storage service that is freely available and easy to run on Kubernetes. If you want to follow this guide but use a different object storage backend, refer to Grafana Enterprise Metrics configuration.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # This name uniquely identifies the PVC. Will be used in deployment below.
  name: minio-pv-claim
  labels:
    app: minio-storage-claim
spec:
  # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    # This is the request for storage. Should be available in the cluster.
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        # Label is used as selector in the service.
        app: minio
    spec:
      # Refer to the PVC created earlier
      volumes:
        - name: storage
          persistentVolumeClaim:
            # Name of the PVC created earlier
            claimName: minio-pv-claim
      initContainers:
        - name: create-buckets
          image: busybox:1.28
          command:
            [
              "sh",
              "-c",
              "mkdir -p /storage/grafana-metrics-tsdb && mkdir -p /storage/grafana-metrics-admin",
            ]
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
      containers:
        - name: minio
          # Pulls the default Minio image from Docker Hub
          image: minio/minio:latest
          args:
            - server
            - /storage
          env:
            # Minio access key and secret key
            - name: MINIO_ACCESS_KEY
              value: "minio"
            - name: MINIO_SECRET_KEY
              value: "minio123"
          ports:
            - containerPort: 9000
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
---
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

Copy the above yaml into a file called minio.yaml and run the following command:

kubectl create -f minio.yaml

You can confirm you have setup Minio correctly by port-forwarding it and navigating to it in your browser:

kubectl port-forward service/minio 9000:9000

Then you can navigate to the minio admin console using your browser. The login credentials are the same as those set above, minio and minio123.

Create a license Secret

Next you will need to take the license.jwt file associated with you cluster and run the following command to load it as a Kubernetes Secret.

kubectl create secret generic ge-metrics-license --from-file license.jwt

Verify you have successfully created the secret by running the following command:

kubectl get secret ge-metrics-license -oyaml

The above command should print a kubernetes Secret object with a license.jwt field with a long base64 encoded value string.

Create a GEM config ConfigMap

Next you will create a configuration file for you cluster and deploy it as a Kubernetes ConfigMap. Copy the text and save it as config.yaml.

auth:
  type: enterprise

target: all

license:
  path: /etc/ge-metrics/license/license.jwt

admin_client:
  storage:
    type: s3
    s3:
      endpoint: minio:9000
      bucket_name: grafana-metrics-admin
      access_key_id: minio
      secret_access_key: minio123
      insecure: true

distributor:
  pool:
    health_check_ingesters: true

memberlist:
  abort_if_cluster_join_fails: false
  bind_port: 7946
  join_members:
    - ge-metrics-discovery

ingester:
  ring:
    num_tokens: 512
    kvstore:
      store: memberlist
    replication_factor: 3

compactor:
  data_dir: "/data"
  sharding_enabled: true
  sharding_ring:
    kvstore:
      store: memberlist

blocks_storage:
  tsdb:
    dir: /data/cortex/tsdb
  bucket_store:
    sync_dir: /data/cortex/tsdb-sync
  backend: s3
  s3:
    endpoint: minio:9000
    bucket_name: grafana-metrics-tsdb
    access_key_id: minio
    secret_access_key: minio123
    insecure: true
storage:
  engine: blocks

Next run the following command to create the ConfigMap.

kubectl create configmap ge-metrics-config --from-file=config.yaml

Create the services for Grafana Enterprise Metrics

Two Kubernetes Services are required to run Grafana Enterprise Metrics as a StatefulSet. First is a service to support GRPC requests between replicas. Second is a gossip service port to allow the replicas to join together and form a hash ring to coordinate work.

Copy the following content into services.yaml:

---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: ge-metrics-discovery
  name: ge-metrics-discovery
spec:
  clusterIP: None
  ports:
    - name: ge-metrics-grpc
      port: 9095
      targetPort: 9095
    - name: ge-metrics-gossip
      port: 7946
      targetPort: 7946
  publishNotReadyAddresses: true
  selector:
    name: ge-metrics
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: ge-metrics
  name: ge-metrics
spec:
  ports:
    - name: ge-metrics-http-metrics
      port: 80
      targetPort: 80
  selector:
    name: ge-metrics

Create the services:

kubectl apply -f services.yaml

Deploy the Grafana Enterprise Metrics StatefulSet

Copy the following content into the statefulset.yaml file:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    name: ge-metrics
  name: ge-metrics
spec:
  replicas: 3
  selector:
    matchLabels:
      name: ge-metrics
  serviceName: ge-metrics
  template:
    metadata:
      labels:
        name: ge-metrics
    spec:
      containers:
        - args:
            - -config.file=/etc/ge-metrics/config.yaml
          image: grafana/enterprise-metrics:v2.4.0
          imagePullPolicy: IfNotPresent
          name: enterprise-metrics
          ports:
            - containerPort: 80
              name: http-metrics
            - containerPort: 9095
              name: grpc
            - containerPort: 7946
              name: gossip
          readinessProbe:
            httpGet:
              path: /ready
              port: 80
            initialDelaySeconds: 15
            timeoutSeconds: 1
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
          volumeMounts:
            - mountPath: /data
              name: data
            - mountPath: /etc/ge-metrics
              name: ge-metrics-config
            - mountPath: /etc/ge-metrics/license
              name: ge-metrics-license
      securityContext:
        fsGroup: 10001
        runAsUser: 10001
        runAsGroup: 10001
        runAsNonRoot: true
      terminationGracePeriodSeconds: 300
      volumes:
        - name: ge-metrics-config
          configMap:
            name: ge-metrics-config
        - name: ge-metrics-license
          secret:
            secretName: ge-metrics-license
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi

Create the StatefulSet:

kubectl apply -f statefulset.yaml

Start the GEM compactor as a Kubernetes deployment

The single binary of GEM does not enable the compactor component by default. Therefore, you need to run the compactor separately as a Kubernetes StatefulSet. For more information about what the compactor does, refer to the Compactor section of the Grafana Mimir documentation.

Copy the following content into the compactor.yaml file:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    name: ge-metrics-compactor
  name: ge-metrics-compactor
spec:
  replicas: 1
  selector:
    matchLabels:
      name: ge-metrics-compactor
  serviceName: ge-metrics-compactor
  template:
    metadata:
      labels:
        name: ge-metrics-compactor
    spec:
      containers:
        - args:
            - -target=compactor
            - -config.file=/etc/ge-metrics/config.yaml
          image: grafana/enterprise-metrics:v2.4.0
          imagePullPolicy: IfNotPresent
          name: ge-metrics-compactor
          ports:
            - containerPort: 80
              name: http-metrics
            - containerPort: 9095
              name: grpc
            - containerPort: 7946
              name: gossip
          readinessProbe:
            httpGet:
              path: /ready
              port: 80
            initialDelaySeconds: 15
            timeoutSeconds: 1
          resources:
            limits:
              cpu: 1200m
              memory: 2Gi
            requests:
              cpu: 1
              memory: 1Gi
          volumeMounts:
            - mountPath: /data
              name: data
            - mountPath: /etc/ge-metrics
              name: ge-metrics-config
            - mountPath: /etc/ge-metrics/license
              name: ge-metrics-license
      securityContext:
        fsGroup: 10001
        runAsUser: 10001
        runAsGroup: 10001
        runAsNonRoot: true
      terminationGracePeriodSeconds: 300
      volumes:
        - name: ge-metrics-config
          configMap:
            name: ge-metrics-config
        - name: ge-metrics-license
          secret:
            secretName: ge-metrics-license
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi

Create the compactor StatefulSet:

kubectl apply -f compactor.yaml

Generate an admin token

To generate a token, use a Kubernetes Job.
Copy the following content it into the tokengen-job.yaml file:

apiVersion: batch/v1
kind: Job
metadata:
  name: ge-metrics-tokengen
spec:
  template:
    spec:
      containers:
        - name: ge-metrics-tokengen
          image: grafana/enterprise-metrics:v2.4.0
          args:
            - --config.file=/etc/ge-metrics/config.yaml
            - --target=tokengen
          volumeMounts:
            - mountPath: /etc/ge-metrics
              name: ge-metrics-config
            - mountPath: /etc/ge-metrics/license
              name: ge-metrics-license
          securityContext:
            runAsUser: 10001
            runAsGroup: 10001
            runAsNonRoot: true
      securityContext:
        fsGroup: 10001
      volumes:
        - name: ge-metrics-config
          configMap:
            name: ge-metrics-config
        - name: ge-metrics-license
          secret:
            secretName: ge-metrics-license
      restartPolicy: Never

Create the tokengen kubernetes Job:

$ kubectl apply -f tokengen-job.yaml

Check the status of the Pod, and after it displays ‘Completed’, check the logs for the new admin token:

kubectl logs job.batch/ge-metrics-tokengen

The output of the above command should contain a token in string in the logs:

Token created:  Ym9vdHN0cmFwLXRva2VuOmA3PzkxOF0zfVx0MTlzMVteTTcjczNAPQ==

Be sure to note down this token since it will be required later when setting up your cluster.

Verify your cluster is working

To verify your cluster is working you can run the following command using the token you generated in the previous step.

First, port-forward the ge-metrics Service to port 9000 on your machine:

kubectl port-forward service/ge-metrics 9000:80

Then run the following command:

curl -u :<your_token> localhost:9000/ready

After running the above command you should see the following output:

ready

Next steps

After you have a GEM deployment working locally, refer to Set up the GEM plugin for Grafana to integrate your metrics cluster into Grafana. Doing so gives you a UI via which you can interact with the Admin API.