Grafana Cloud

Ship Kubernetes traces using Grafana Agent

In this guide, you’ll deploy the Grafana Agent into a Kubernetes cluster as a Deployment and configure it to collect traces for your Kubernetes workloads. You’ll then ship these traces to Grafana Cloud for storage and querying from your hosted Grafana instance.

You can instrument your Kubernetes workloads and apps to emit spans using client libraries from OpenTracing/Jaeger, Zipkin, and OpenTelemetry.

You can then use Grafana Agent to collect these spans from your app, buffer them, and forward them to Grafana Cloud for storage and querying. To learn more, please see Tracing with the Grafana Agent and Grafana Tempo from the Grafana Blog.

Before you begin

To complete the steps in this guide, you should have the following available:

  • A Kubernetes cluster with role-based access control (RBAC) enabled.
  • A Grafana Cloud account. To create an account, please see Grafana Cloud and click on Start for free.
  • The kubectl command-line tool installed on your local machine, configured to connect to your cluster. You can read more about installing kubectl in the official documentation.

Configure Grafana Agent

Paste the following script into your shell and run it to configure Grafana Agent:

cat <<'EOF' |

kind: ConfigMap
  name: grafana-agent-traces
apiVersion: v1
  agent.yaml: |
          - batch:
                send_batch_size: 1000
                timeout: 5s
            name: default
                        grpc: null
                        thrift_binary: null
                        thrift_compact: null
                        thrift_http: null
                        strategy_file: /etc/agent/strategies.json
                            insecure: true
                opencensus: null
                        grpc: null
                        http: null
                zipkin: null
              - basic_auth:
                    password: YOUR_TEMPO_PASSWORD
                    username: YOUR_TEMPO_USER
                endpoint: YOUR_TEMPO_ENDPOINT
                    enabled: false
              - bearer_token_file: /var/run/secrets/
                job_name: kubernetes-pods
                  - role: pod
                  - action: replace
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                    ca_file: /var/run/secrets/
                    insecure_skip_verify: false
  strategies.json: '{"default_strategy": {"param": 0.001, "type": "probabilistic"}}'

(export NAMESPACE=default && kubectl apply -n $NAMESPACE -f -)

Be sure to replace NAMESPACE=default with the Namespace into which you installed the Agent.

Also be sure to fill in the parameters in the remote_write stanza. You can find your Cloud Tempo credentials in the Cloud Portal. Your Tempo push endpoint should look something like:

This ConfigMap configures the Agent to accept traces from every supported receiver and set a default set of labels using relabel_configs. To learn more about the relabeling steps, please see the Tempo docs, and Grafana Agent traces_config docs.

This configuration also specifies a Jaeger trace sampling strategy. To learn more about Jaeger’s sampling strategies, plesae see Jaeger’s documentation.

Deploy Grafana Agent

In this step you’ll install the Grafana Agent and its required resources into your cluster.

To collect traces from your Kubernetes workloads, you can run the Agent as a Deployment or DaemonSet. In this quickstart we’ll roll out the Agent as a one-replica Deployment.

Run the following command from your shell to install the Grafana Agent into the default Namespace of your Kubernetes cluster:

MANIFEST_URL= NAMESPACE=default /bin/sh -c "$(curl -fsSL" | kubectl apply -f -

This installs a one-replica Grafana Agent Deployment into your cluster and configures RBAC permissions for the Agent. If you would like to deploy the Agent into a different Namespace, change the NAMESPACE=default variable, ensuring that this Namespace already exists. This also rolls out the Agent ClusterRole and ClusterRoleBinding, as well as a Service with the appropriate ports exposed. You can modify these defaults depending on the receivers you need. To learn more, see Distributor from the Tempo docs.