---
title: "Deploy on Kubernetes | Grafana Enterprise Metrics documentation"
description: "Deploy GEM on Kubernetes This guide will layout a step by step approach to deploying a Grafana Enterprise Metrics cluster on an existing Kubernetes namespace. This guide assumes you have a working Kubernetes cluster and the ability to deploy to that cluster using the kubectl tool. At the end of this guide you should have a functional GEM cluster running on Kubernetes using Minio as a storage backend. If you do not currently have access to a Kubernetes cluster consider following Linux deployment guide instead."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Deploy GEM on Kubernetes

This guide will layout a step by step approach to deploying a Grafana Enterprise Metrics cluster on an existing Kubernetes namespace. This guide assumes you have a working Kubernetes cluster and the ability to deploy to that cluster using the `kubectl` tool. At the end of this guide you should have a functional GEM cluster running on Kubernetes using Minio as a storage backend. If you do not currently have access to a Kubernetes cluster consider following [Linux deployment guide instead](../linux).

## Deploy Minio

For this guide we will be using [Minio](https://minio.io/) as our object storage backend. Minio is a open source S3 compatible object storage service that is freely available and easy to run on Kubernetes. If you want to follow this guide but use a different object storage backend, refer to [Grafana Enterprise Metrics configuration](../../config/).

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # This name uniquely identifies the PVC. Will be used in deployment below.
  name: minio-pv-claim
  labels:
    app: minio-storage-claim
spec:
  # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    # This is the request for storage. Should be available in the cluster.
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        # Label is used as selector in the service.
        app: minio
    spec:
      # Refer to the PVC created earlier
      volumes:
        - name: storage
          persistentVolumeClaim:
            # Name of the PVC created earlier
            claimName: minio-pv-claim
      initContainers:
        - name: create-buckets
          image: busybox:1.28
          command:
            [
              "sh",
              "-c",
              "mkdir -p /storage/grafana-metrics-tsdb && mkdir -p /storage/grafana-metrics-admin",
            ]
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
      containers:
        - name: minio
          # Pulls the default Minio image from Docker Hub
          image: minio/minio:latest
          args:
            - server
            - /storage
          env:
            # Minio access key and secret key
            - name: MINIO_ACCESS_KEY
              value: "minio"
            - name: MINIO_SECRET_KEY
              value: "minio123"
          ports:
            - containerPort: 9000
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
---
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio
```

Copy the above yaml into a file called `minio.yaml` and run the following command:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl create -f minio.yaml
```

You can confirm you have setup Minio correctly by port-forwarding it and navigating to it in your browser:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl port-forward service/minio 9000:9000
```

Then you can navigate to the minio admin console using your browser. The login credentials are the same as those set above, `minio` and `minio123`.

## Create a license Secret

Next you will need to take the `license.jwt` file associated with you cluster and run the following command to load it as a Kubernetes Secret.

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl create secret generic ge-metrics-license --from-file license.jwt
```

Verify you have successfully created the secret by running the following command:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl get secret ge-metrics-license -oyaml
```

The above command should print a kubernetes Secret object with a `license.jwt` field with a long base64 encoded value string.

## Create a GEM config ConfigMap

Next you will create a configuration file for you cluster and deploy it as a Kubernetes ConfigMap. Copy the text and save it as `config.yaml`.

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
auth:
  type: enterprise

target: all

license:
  path: /etc/ge-metrics/license/license.jwt 

admin_client:
  storage:
    type: s3
    s3:
      endpoint: minio:9000
      bucket_name: grafana-metrics-admin 
      access_key_id: minio
      secret_access_key: minio123
      insecure: true

distributor:
  shard_by_all_labels: true
  pool:
    health_check_ingesters: true

memberlist:
  abort_if_cluster_join_fails: false
  bind_port: 7946
  join_members:
  - ge-metrics-discovery

ingester:
  lifecycler:
    num_tokens: 512
    ring:
      kvstore:
        store: memberlist
      replication_factor: 3

compactor:
  data_dir: "/data"
  sharding_enabled: true
  sharding_ring:
    kvstore:
      store: memberlist

blocks_storage:
  tsdb:
    dir: /data/cortex/tsdb
  bucket_store:
    sync_dir: /data/cortex/tsdb-sync
  backend: s3
  s3:
    endpoint: minio:9000
    bucket_name: grafana-metrics-tsdb
    access_key_id: minio
    secret_access_key: minio123 
    insecure: true 
storage:
  engine: blocks
```

Next run the following command to create the ConfigMap.

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl create configmap ge-metrics-config --from-file=config.yaml
```

## Create the services for Grafana Enterprise Metrics

Two Kubernetes Services are required to run Grafana Enterprise Metrics as a StatefulSet. First is a service to support GRPC requests between replicas. Second is a gossip service port to allow the replicas to join together and form a hash ring to coordinate work.

Copy the following content into `services.yaml`:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: ge-metrics-discovery
  name: ge-metrics-discovery
spec:
  clusterIP: None
  ports:
    - name: ge-metrics-grpc
      port: 9095
      targetPort: 9095
    - name: ge-metrics-gossip
      port: 7946
      targetPort: 7946
  publishNotReadyAddresses: true
  selector:
    name: ge-metrics
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: ge-metrics
  name: ge-metrics
spec:
  ports:
    - name: ge-metrics-http-metrics
      port: 80
      targetPort: 80
  selector:
    name: ge-metrics 
```

Create the services:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl apply -f services.yaml
```

## Deploy the Grafana Enterprise Metrics StatefulSet

Copy the following content into the `statefulset.yaml` file, and make sure to add the `<version_number>` of the GEM image that you would like to run:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    name: ge-metrics
  name: ge-metrics
spec:
  replicas: 3
  selector:
    matchLabels:
      name: ge-metrics
  serviceName: ge-metrics
  template:
    metadata:
      labels:
        name: ge-metrics
    spec:
      containers:
      - args:
        - -config.file=/etc/ge-metrics/config.yaml
        image: grafana/metrics-enterprise:<version_number>
        imagePullPolicy: IfNotPresent
        name: metrics-enterprise
        ports:
        - containerPort: 80
          name: http-metrics
        - containerPort: 9095
          name: grpc
        - containerPort: 7946
          name: gossip
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 15
          timeoutSeconds: 1
        resources:
          requests:
            cpu: "2"
            memory: 4Gi
        volumeMounts:
        - mountPath: /data
          name: data
        - mountPath: /etc/ge-metrics
          name: ge-metrics-config
        - mountPath: /etc/ge-metrics/license
          name: ge-metrics-license
      securityContext:
        fsGroup: 10001
        runAsUser: 10001
        runAsGroup: 10001
        runAsNonRoot: true
      terminationGracePeriodSeconds: 300
      volumes:
      - name: ge-metrics-config
        configMap:
          name: ge-metrics-config
      - name: ge-metrics-license
        secret:
          secretName: ge-metrics-license
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 50Gi
```

Create the StatefulSet:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl apply -f statefulset.yaml
```

## Start the GEM compactor as a Kubernetes deployment

The single binary of GEM does not enable the compactor component by default. Therefore, you need to run the compactor separately as a Kubernetes StatefulSet. For more information about what the compactor does, refer to the [Compactor](https://cortexmetrics.io/docs/blocks-storage/compactor/) section of the Cortex documentation.

1. Copy the following content into the `compactor.yaml` file, and make sure to update the `<version_number>` of the GEM image that you would like to run:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    labels:
      name: ge-metrics-compactor
    name: ge-metrics-compactor
  spec:
    replicas: 1
    selector:
      matchLabels:
        name: ge-metrics-compactor
    serviceName: ge-metrics-compactor
    template:
      metadata:
        labels:
          name: ge-metrics-compactor
      spec:
        containers:
        - args:
          - -target=compactor
          - -config.file=/etc/ge-metrics/config.yaml
          image: grafana/metrics-enterprise:<version_number>
          imagePullPolicy: IfNotPresent
          name: ge-metrics-compactor
          ports:
          - containerPort: 80
            name: http-metrics
          - containerPort: 9095
            name: grpc
          - containerPort: 7946
            name: gossip
          readinessProbe:
            httpGet:
              path: /ready
              port: 80
            initialDelaySeconds: 15
            timeoutSeconds: 1
          resources:
            limits:
              cpu: 1200m
              memory: 2Gi
            requests:
              cpu: 1
              memory: 1Gi
          volumeMounts:
          - mountPath: /data
            name: data
          - mountPath: /etc/ge-metrics
            name: ge-metrics-config
          - mountPath: /etc/ge-metrics/license
            name: ge-metrics-license
        securityContext:
          fsGroup: 10001
          runAsUser: 10001
          runAsGroup: 10001
          runAsNonRoot: true
        terminationGracePeriodSeconds: 300
        volumes:
        - name: ge-metrics-config
          configMap:
            name: ge-metrics-config
        - name: ge-metrics-license
          secret:
            secretName: ge-metrics-license
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
```

2. Create the compactor StatefulSet:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl apply -f compactor.yaml
```

## Generate an admin token

1. To generate a token, use a Kubernetes Job.
2. Copy the following content it into the `tokengen-job.yaml` file, and make sure to update the `<version_number>` of the GEM image that you would like to run:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ge-metrics-tokengen
spec:
  template:
    spec:
      containers:
      - name: ge-metrics-tokengen
        image: grafana/metrics-enterprise:<version_number>
        args:
        - --config.file=/etc/ge-metrics/config.yaml
        - --target=tokengen
        volumeMounts:
        - mountPath: /etc/ge-metrics
          name: ge-metrics-config
        - mountPath: /etc/ge-metrics/license
          name: ge-metrics-license
        securityContext:
          runAsUser: 10001
          runAsGroup: 10001
          runAsNonRoot: true
      securityContext:
        fsGroup: 10001
      volumes:
      - name: ge-metrics-config
        configMap:
          name: ge-metrics-config
      - name: ge-metrics-license
        secret:
          secretName: ge-metrics-license
      restartPolicy: Never
```

Create the tokengen kubernetes Job:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
$ kubectl apply -f tokengen-job.yaml
```

3. Check the status of the Pod, and after it displays ‘Completed’, check the logs for the new admin token:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl logs job.batch/ge-metrics-tokengen
```

The output of the above command should contain a token in string in the logs:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
Token created:  Ym9vdHN0cmFwLXRva2VuOmA3PzkxOF0zfVx0MTlzMVteTTcjczNAPQ==
```

Be sure to note down this token since it will be required later when setting up your cluster.

## Verify your cluster is working

To verify your cluster is working you can run the following command using the token you generated in the previous step.

First, port-forward the ge-metrics Service to port 9000 on your machine:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl port-forward service/ge-metrics 9000:80
```

Then run the following command:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
curl -u :<your_token> localhost:9000/ready
```

After running the above command you should see the following output:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
ready
```

## Next steps

After you have a GEM deployment working locally, refer to [Set up the GEM plugin for Grafana](../../setup-gem-plugin-grafana/) to integrate your metrics cluster into Grafana. Doing so gives you a UI via which you can interact with the [Admin API](../../admin-api/).