This is archived documentation for v1.4.1. Go to the latest version.

Set up GEMDeploy on Kubernetes

Deploy GEM on Kubernetes

This guide will layout a step by step approach to deploying a Grafana Enterprise Metrics cluster on an existing Kubernetes namespace. This guide assumes you have a working Kubernetes cluster and the ability to deploy to that cluster using the kubectl tool. At the end of this guide you should have a functional GEM cluster running on Kubernetes using Minio as a storage backend. If you do not currently have access to a Kubernetes cluster consider following Linux deployment guide instead.

Deploy Minio

For this guide we will be using Minio as our object storage backend. Minio is a open source S3 compatible object storage service that is freely available and easy to run on Kubernetes. If you want to follow this guide but use a different object storage backend, refer to Grafana Enterprise Metrics configuration.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # This name uniquely identifies the PVC. Will be used in deployment below.
  name: minio-pv-claim
  labels:
    app: minio-storage-claim
spec:
  # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    # This is the request for storage. Should be available in the cluster.
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        # Label is used as selector in the service.
        app: minio
    spec:
      # Refer to the PVC created earlier
      volumes:
        - name: storage
          persistentVolumeClaim:
            # Name of the PVC created earlier
            claimName: minio-pv-claim
      initContainers:
        - name: create-buckets
          image: busybox:1.28
          command:
            [
              "sh",
              "-c",
              "mkdir -p /storage/grafana-metrics-tsdb && mkdir -p /storage/grafana-metrics-admin",
            ]
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
      containers:
        - name: minio
          # Pulls the default Minio image from Docker Hub
          image: minio/minio:latest
          args:
            - server
            - /storage
          env:
            # Minio access key and secret key
            - name: MINIO_ACCESS_KEY
              value: "minio"
            - name: MINIO_SECRET_KEY
              value: "minio123"
          ports:
            - containerPort: 9000
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
---
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

Copy the above yaml into a file called minio.yaml and run the following command:

kubectl create -f minio.yaml

You can confirm you have setup Minio correctly by port-forwarding it and navigating to it in your browser:

kubectl port-forward service/minio 9000:9000

Then you can navigate to the minio admin console using your browser. The login credentials are the same as those set above, minio and minio123.

minio-in-browser

Create a license Secret

Next you will need to take the license.jwt file associated with you cluster and run the following command to load it as a Kubernetes Secret.

kubectl create secret generic ge-metrics-license --from-file license.jwt

Verify you have successfully created the secret by running the following command:

kubectl get secret ge-metrics-license -oyaml

The above command should print a kubernetes Secret object with a license.jwt field with a long base64 encoded value string.

Create a GEM config ConfigMap

Next you will create a configuration file for you cluster and deploy it as a Kubernetes ConfigMap. Copy the text and save it as config.yaml. Then update the cluster_name field with the name of the cluster associated with your license.

auth:
  type: enterprise

target: all

# TODO: Ensure the cluster name is set to match your Grafana Labs License File
cluster_name: # <name of you cluster cluster>

license:
  path: /etc/ge-metrics/license/license.jwt 

admin_client:
  storage:
    type: s3
    s3:
      endpoint: minio:9000
      bucket_name: grafana-metrics-admin 
      access_key_id: minio
      secret_access_key: minio123
      insecure: true

distributor:
  shard_by_all_labels: true
  pool:
    health_check_ingesters: true

memberlist:
  abort_if_cluster_join_fails: false
  bind_port: 7946
  join_members:
  - ge-metrics-discovery

ingester:
  lifecycler:
    num_tokens: 512
    ring:
      kvstore:
        store: memberlist
      replication_factor: 3

blocks_storage:
  tsdb:
    dir: /data/cortex/tsdb
  bucket_store:
    sync_dir: /data/cortex/tsdb-sync
  backend: s3
  s3:
    endpoint: minio:9000
    bucket_name: grafana-metrics-tsdb
    access_key_id: minio
    secret_access_key: minio123 
    insecure: true 
storage:
  engine: blocks

Next run the following command to create the ConfigMap.

kubectl create configmap ge-metrics-config --from-file=config.yaml

Create the services for Grafana Enterprise Metrics

Two Kubernetes Services are required to run Grafana Enterprise Metrics as a StatefulSet. First is a service to support GRPC requests between replicas. Second is a gossip service port to allow the replicas to join together and form a hash ring to coordinate work.

Copy the following content into services.yaml:

---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: ge-metrics-discovery
  name: ge-metrics-discovery
spec:
  clusterIP: None
  ports:
    - name: ge-metrics-grpc
      port: 9095
      targetPort: 9095
    - name: ge-metrics-gossip
      port: 7946
      targetPort: 7946
  publishNotReadyAddresses: true
  selector:
    name: ge-metrics
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: ge-metrics
  name: ge-metrics
spec:
  ports:
    - name: ge-metrics-http-metrics
      port: 80
      targetPort: 80
  selector:
    name: ge-metrics 

Create the services:

kubectl apply -f services.yaml

Deploy the Grafana Enterprise Metrics StatefulSet

Copy the following content into statefulset.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    name: ge-metrics
  name: ge-metrics
spec:
  replicas: 3
  selector:
    matchLabels:
      name: ge-metrics
  serviceName: ge-metrics
  template:
    metadata:
      labels:
        name: ge-metrics
    spec:
      containers:
      - args:
        - -config.file=/etc/ge-metrics/config.yaml
        image: grafana/metrics-enterprise:v1.3.0
        imagePullPolicy: IfNotPresent
        name: metrics-enterprise
        ports:
        - containerPort: 80
          name: http-metrics
        - containerPort: 9095
          name: grpc
        - containerPort: 7946
          name: gossip
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 15
          timeoutSeconds: 1
        resources:
          requests:
            cpu: "2"
            memory: 4Gi
        volumeMounts:
        - mountPath: /data
          name: data
        - mountPath: /etc/ge-metrics
          name: ge-metrics-config
        - mountPath: /etc/ge-metrics/license
          name: ge-metrics-license
      securityContext:
        runAsUser: 0
      terminationGracePeriodSeconds: 300
      volumes:
      - name: ge-metrics-config
        configMap:
          name: ge-metrics-config
      - name: ge-metrics-license
        secret:
          secretName: ge-metrics-license
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 50Gi

Create the StatefulSet:

kubectl apply -f statefulset.yaml

Generate an admin token

A Kubernetes Job can be used to perform token generation.

Copy the following content it into tokengen-job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: ge-metrics-tokengen
spec:
  template:
    spec:
      containers:
      - name: ge-metrics-tokengen
        image: grafana/metrics-enterprise:v1.3.0
        args:
        - --config.file=/etc/ge-metrics/config.yaml
        - --target=tokengen
        volumeMounts:
        - mountPath: /etc/ge-metrics
          name: ge-metrics-config
        - mountPath: /etc/ge-metrics/license
          name: ge-metrics-license
      volumes:
      - name: ge-metrics-config
        configMap:
          name: ge-metrics-config
      - name: ge-metrics-license
        secret:
          secretName: ge-metrics-license
      restartPolicy: Never

Create the tokengen kubernetes Job:

$ kubectl apply -f tokengen-job.yaml

Check the status of the Pod and once it is ‘Completed’ check the logs for the new admin token

kubectl logs job.batch/ge-metrics-tokengen

The output of the above command should contain a token in string in the logs:

Token created:  Ym9vdHN0cmFwLXRva2VuOmA3PzkxOF0zfVx0MTlzMVteTTcjczNAPQ==

Be sure to note down this token since it will be required later when setting up your cluster.

Next steps

Now that you have a working GEM deployment locally you can refer to setup up the Grafana Enterprise Metrics plugin to integrate your metrics cluster with Grafana and give you a UI to interact with the Admin API.