Grafana Cloud quickstart guidesGrafana Agent Kubernetes QuickstartsGrafana Agent Traces Kubernetes Quickstart

Grafana Agent Traces Kubernetes Quickstart

In this guide you’ll deploy the Grafana Agent into a Kubernetes cluster as a Deployment and configure it to collect traces for your Kubernetes workloads. You’ll then ship these traces to Grafana Cloud for storage and querying from your hosted Grafana instance.

Kubernetes is an open-source container orchestration system that automates software container deployment, scaling, and management. You can instrument your Kubernetes workloads and apps to emit spans using client libraries from OpenTracing/Jaeger, Zipkin, and OpenTelemetry. You can then use the Grafana Agent to collect these spans from your app, buffer them, and forward them to Grafana Cloud for storage and querying. To learn more, please see Tracing with the Grafana Agent and Grafana Tempo from the Grafana Blog.

Prerequisites

Before you begin, you should have the following available:

  • A Kubernetes cluster with role-based access control (RBAC) enabled.
  • A Grafana Cloud account. To create an account, please see Grafana Cloud and click on Start for free.
  • The kubectl command-line tool installed on your local machine, configured to connect to your cluster. You can read more about installing kubectl in the official documentation.

Step 1: Deploy Grafana Agent Deployment

In this step you’ll install the Grafana Agent and its required resources into your cluster.

To collect traces from your Kubernetes workloads, you can run the Agent as a Deployment or DaemonSet. In this quickstart we’ll roll out the Agent as a one-replica Deployment. A sample Deployment manifest can be found in the Agent GitHub repository.

Save the agent-tempo.yaml manifest on your local machine and replace YOUR_NAMESPACE values with the Namespace into which you’ll install the Agent.

When you’re done, roll out the Agent using kubectl apply -f:

kubectl apply -f agent-tempo.yaml

This will roll out the Agent ClusterRole and ClusterRoleBinding, as well as the Agent Deployment and Service with the appropriate ports exposed. You can modify these defaults depending on the receivers you need. To learn more, see Distributor from the Tempo docs.

In the next step, you’ll configure the Agent.

Step 2: Configure Grafana Agent

Copy the following ConfigMap manifest into a file called agent-tempo-configmap.yaml:

apiVersion: v1
data:
  agent.yaml: |
    server:
        http_listen_port: 8080
        log_level: debug
    tempo:
        configs:
          - batch:
                send_batch_size: 1000
                timeout: 5s
            name: default
            receivers:
                jaeger:
                    protocols:
                        grpc: null
                        thrift_binary: null
                        thrift_compact: null
                        thrift_http: null
                    remote_sampling:
                        insecure: true
                        strategy_file: /etc/agent/strategies.json
                opencensus: null
                otlp:
                    protocols:
                        grpc: null
                        http: null
                zipkin: null
            remote_write:
              - basic_auth:
                    password: YOUR_TEMPO_PASSWORD
                    username: YOUR_TEMPO_USER
                endpoint: tempo-us-central1.grafana.net:443
                retry_on_failure:
                    enabled: false
            scrape_configs:
              - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                job_name: kubernetes-pods
                kubernetes_sd_configs:
                  - role: pod
                relabel_configs:
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                tls_config:
                    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                    insecure_skip_verify: false
  strategies.json: '{"default_strategy": {"param": 0.001, "type": "probabilistic"}}'
kind: ConfigMap
metadata:
  name: grafana-agent-traces
  namespace: YOUR_NAMESPACE

Be sure to replace YOUR_NAMESPACE with the Namespace into which you installed the Agent.

Also be sure to fill in the parameters in the remote_write stanza. You can find your Cloud Tempo credentials in the Cloud Portal. Your Tempo push endpoint should look something like: tempo-us-central1.grafana.net:443.

When you’re done, roll out the ConfigMap using kubectl apply -f:

kubectl apply -f agent-tempo-configmap.yaml

This ConfigMap configures the Agent to accept traces from every supported receiver and set the appropriate labels using relabel_configs. To learn more about the relabeling steps, please see tempo.libsonnet, the Tempo Agent docs, and Grafana Agent tempo_config docs. The Tracing with the Grafana Agent and Grafana Tempo blog post can also help you get started with the Grafana Agent and Cloud Tempo.

This configuration also specifies a Jaeger trace sampling strategy. To learn more about Jaeger’s sampling strategies, plesae see Jaeger’s documentation.

After deploying the ConfigMap, the Grafana Agent should start.

Step 3: Restart the Grafana Agent

After modifying the Agent’s configuration, you will need to restart the Agent Pods to pick up configuration changes. Use kubectl rollout to restart the Agent:

kubectl rollout restart deployment/grafana-agent-traces

Conclusion

You’ve now deployed the Agent into your cluster, have configured it to collect traces, and are shippping these traces to Grafana Cloud.