Set up a GET clusterDeploy on Kubernetes with Tanka

Deploy on Kubernetes with Tanka

To deploy Grafana Enterprise Traces to Kubernetes using a Jsonnet library and Grafana Tanka, you can use MinIO to provide object storage. That said, it is best to leverage your cloud provider’s object storage service to avoid the operational overhead of running object storage in production.

Prerequisites

  • Kubernetes cluster

The default configuration expects approximately 40CPUs and 46GiB of memory, although far smaller configurations would be appropriate for small ingest/query volumes.

  • kubectl
  • GET license

Procedure

  1. Create a new directory for the installation, and make it your current working directory:

    mkdir get
    cd get
    
  2. Create a Kubernetes namespace:

    kubectl create namespace enterprise-traces
    
  3. Create a Kubernetes Secret for your GET license:

    kubectl --namespace=enterprise-traces create secret generic get-license --from-file=/path/to/license.jwt
    
  4. Install Grafana Tanka; refer to Installing Tanka.

  5. Install jsonnet-bundler; refer to the jsonnet-bundler README.

  6. Set up the Tanka environment:

    1. Initialize Tanka

      tk init --k8s=false
      tk env add environments/enterprise-traces
      tk env set environments/enterprise-traces \
        --namespace=enterprise-traces \
        --server-from-context=<KUBECFG CONTEXT NAME>
      
    2. Install k.libsonnet for your version of Kubernetes:

      mkdir -p lib
      export K8S_VERSION=1.21
      jb install github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}@main
      cat <<EOF > lib/k.libsonnet
      import 'github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}/main.libsonnet'
      EOF
      
  7. Install the GET Jsonnet library and its dependencies.

    jb install github.com/grafana/tempo/operations/jsonnet/enterprise@main
    
  8. Install the Memcached library and its dependencies.

    jb install github.com/grafana/jsonnet-libs/memcached@master
    
  9. Deploy MinIO object storage.

    MinIO is an open source S3 compatible object storage service that is freely available and easy to run on Kubernetes.

    1. Create a file named minio.yaml, and copy the following YAML configuration into it. Note that you may need to remove/modify the storageClassName depending on your Kubernetes platform:

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        # This name uniquely identifies the PVC. Will be used in deployment below.
        name: minio-pv-claim
        labels:
          app: minio-storage-claim
      spec:
        # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
        accessModes:
          - ReadWriteOnce
        storageClassName: local-path
        resources:
          # This is the request for storage. Should be available in the cluster.
          requests:
            storage: 50Gi
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: minio
      spec:
        selector:
          matchLabels:
            app: minio
        strategy:
          type: Recreate
        template:
          metadata:
            labels:
              # Label is used as selector in the service.
              app: minio
          spec:
            # Refer to the PVC created earlier
            volumes:
              - name: storage
                persistentVolumeClaim:
                  # Name of the PVC created earlier
                  claimName: minio-pv-claim
            initContainers:
              - name: create-buckets
                image: busybox:1.28
                command:
                - "sh"
                - "-c"
                - "mkdir -p /storage/enterprise-traces-data && mkdir -p /storage/enterprise-traces-admin"
                volumeMounts:
                  - name: storage # must match the volume name, above
                    mountPath: "/storage"
            containers:
              - name: minio
                # Pulls the default Minio image from Docker Hub
                image: minio/minio:latest
                args:
                  - server
                  - /storage
                  - --console-address
                  - ":9001"
                env:
                  # Minio access key and secret key
                  - name: MINIO_ACCESS_KEY
                    value: "minio"
                  - name: MINIO_SECRET_KEY
                    value: "minio123"
                ports:
                  - containerPort: 9000
                  - containerPort: 9001
                volumeMounts:
                  - name: storage # must match the volume name, above
                    mountPath: "/storage"
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: minio
      spec:
        type: ClusterIP
        ports:
          - port: 9000
            targetPort: 9000
            protocol: TCP
            name: api
          - port: 9001
            targetPort: 9001
            protocol: TCP
            name: console
        selector:
          app: minio
      
    2. Run the following command.

      kubectl apply --namespace enterprise-traces -f minio.yaml
      
    3. Verify that you have set up MinIO correctly, by port-forwarding it and navigating to it in your browser:

      kubectl port-forward --namespace enterprise-traces service/minio 9001:9001
      
    4. Navigate to the MinIO admin bash using your browser. The sign-in credentials are username minio and password minio123.

  10. Configure the GET cluster using the MinIO object storage by replacing the contents of the environments/enterprise-traces/main.jsonnet file with the following configuration:

    cat <<EOF > environments/enterprise-traces/main.jsonnet
    local get = import 'github.com/grafana/tempo/operations/jsonnet/enterprise/main.libsonnet';
    
    get {
        _config+:: {
            namespace: 'enterprise-traces',
            bucket: 'enterprise-traces-data',
            backend: 's3',
    
            // Set to true the first time installing GET, this will create the tokengen job. Once this job
            // has run this settings should be deleted.
            create_tokengen_job: true,
        },
    
        tempo_config+:: {
            storage+: {
                trace+: {
                    s3: {
                        bucket: $._config.bucket,
                        access_key: 'minio',
                        secret_key: 'minio123',
                        endpoint: 'minio:9000',
                        insecure: true,
                    },
                },
            },
            admin_client+: {
                storage+: {
                    type: 's3',
                    s3: {
                        bucket_name: 'enterprise-traces-admin',
                        access_key_id: 'minio',
                        secret_access_key: 'minio123',
                        endpoint: 'minio:9000',
                        insecure: true,
                    },
                },
            },
        },
    
        tempo_ingester_container+:: {
            securityContext+: {
                runAsUser: 0,
            },
        },
    
        // Deploy tokengen Job available on a first run.
        tokengen_job+::: {},
    }
    EOF
    
  11. Finally, deploy GET using the Tanka command:

    tk apply environments/enterprise-traces/main.jsonnet