Grafana Cloud Grafana Kubernetes Monitoring Other configuration methods Ship Kubernetes OTEL traces using Grafana Agent
Grafana Cloud

Ship Kubernetes OTEL traces using Grafana Agent

In this guide, you’ll deploy the Grafana Agent into a Kubernetes cluster as a Deployment and configure it to receive OpenTelemetry traces for your workloads. These traces are then buffered, and shipped to Grafana Cloud for storage and querying from your hosted Grafana instance.

You can instrument your Kubernetes workloads and apps to emit spans using client libraries from OpenTelemetry.

This guide only covers receiving traces, not metrics, logs, or processing. With the addition of OpenTelemetry components to the agent, more features and configurations for OTel are available. To learn more, please see the OpenTelemetry component release and the Grafana Agent Flow docs.

Before you begin

To complete the steps in this guide, you should have the following available:

  • A Kubernetes cluster with role-based access control (RBAC) enabled.
  • A Grafana Cloud account. To create an account, please see Grafana Cloud and click on Start for free.
  • The kubectl command-line tool installed on your local machine, configured to connect to your cluster. You can read more about installing kubectl in the official documentation.
  • An application that has the OpenTelemetry libraries instrumented and is deployed to the Kubernetes cluster ready to send traces.

Configure Grafana Agent

Paste the following script into your shell and run it to deploy a configuration map for Grafana Agent into your currently selected kubectl cluster:

cat <<'EOF' |

kind: ConfigMap
  name: grafana-agent-traces
apiVersion: v1
  agent.yaml: |
          - batch:
                send_batch_size: 1000
                timeout: 5s
            name: default
                        grpc: null
                        thrift_binary: null
                        thrift_compact: null
                        thrift_http: null
                        strategy_file: /etc/agent/strategies.json
                            insecure: true
                opencensus: null
                        grpc: null
                        http: null
                zipkin: null
              - basic_auth:
                    password: YOUR_TEMPO_PASSWORD
                    username: YOUR_TEMPO_USER
                endpoint: YOUR_TEMPO_ENDPOINT
                    enabled: false
              - bearer_token_file: /var/run/secrets/
                job_name: kubernetes-pods
                  - role: pod
                  - action: replace
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                    ca_file: /var/run/secrets/
                    insecure_skip_verify: false
  strategies.json: '{"default_strategy": {"param": 0.001, "type": "probabilistic"}}'

(export NAMESPACE=default && kubectl apply -n $NAMESPACE -f -)

Be sure to replace NAMESPACE=default with the namespace into which you would like to install the Agent.

Also be sure to fill in the Grafana Cloud parameters in the remote_write stanza.




You can find the Tempo credentials in your Cloud Portal. Your Tempo push endpoint should look similar to:

The deployed ConfigMap sets up the Agent to accept traces from the OpenTelemetry receiver and set a default set of labels using relabel_configs. To learn more about the relabeling steps, please see the Tempo agent docs, and Grafana Agent traces_config reference.

Deploy Grafana Agent

In this step you’ll install the Grafana Agent using the previous ConfigMap and deploy the required resources into your cluster.

Run the following command from your shell to install the Grafana Agent into the default Namespace of your Kubernetes cluster:

MANIFEST_URL= NAMESPACE=default /bin/sh -c "$(curl -fsSL" | kubectl apply -f -

This installs a one-replica Grafana Agent Deployment into your cluster and configures RBAC permissions for the Agent. If you would like to deploy the Agent into a different Namespace, change the NAMESPACE=default variable, ensuring that this Namespace already exists. This also rolls out the Agent ClusterRole and ClusterRoleBinding and a Kubernetes Service with the appropriate ports connected.

grafana-agent-traces Service

The deployed cluster Service is named grafana-agent-traces exposing otlp over grpc on port 4317, or http on port 4318. See the snippet below for reference.

apiVersion: v1
kind: Service
    name: grafana-agent-traces
  name: grafana-agent-traces
  namespace: default
  - name: grafana-agent-traces-otlp-grpc
    port: 4317
    protocol: TCP
    targetPort: 4317
  - name: grafana-agent-traces-otlp-http
    port: 4318
    protocol: TCP
    targetPort: 4318
    name: grafana-agent-traces

Workloads deployed to the Kubernetes cluster can now send otel traces to grafana-agent-traces.$NAMESPACE.svc.cluster.local:4317, or with the reduced grafana-agent-traces:4318 for workloads running within the same namespace.

Check grafana-agent logs

You can find Grafana Agent logs at /var/log/grafana-agent.err.log

If the deployment succeeded the following trace component line will be output.

ts=2023-04-01T21:23:10.949549748Z caller=zapadapter.go:78 level=info component=traces traces_config=grafana kind=receiver name=otlp pipeline=traces msg="Receiver started."

Search Traces

Once traces are delivered to the Grafana Cloud backend via the Agent deployment, you can search for traces using the Grafana Explore trace integration and querying with TraceQL.

For example, below is a simple traceQL snipit that would select all traces in the default namespace.