Send Kubernetes traces using Grafana Agent
You can deploy the Grafana Agent into a Kubernetes cluster as a Deployment and configure it to collect traces for your Kubernetes workloads. Then you send these traces to Grafana Cloud for storage and querying from your hosted Grafana instance. These steps deploy a single-replica Agent deployment that will receive traces, and remote_write them to Grafana Cloud.
You can instrument your Kubernetes workloads and apps to emit spans using client libraries from OpenTracing/Jaeger, Zipkin, and OpenTelemetry.
You can then use Grafana Agent to collect these spans from your app, buffer them, and forward them to Grafana Cloud for storage and querying. To learn more, refer to Tracing with the Grafana Agent and Grafana Tempo.
Before you begin
To complete all steps, have the following available:
- A Kubernetes cluster with role-based access control (RBAC) enabled
- A Grafana Cloud account. To create an account, navigate to Grafana Cloud, and click Create free account.
- The
kubectl
command-line tool installed on your local machine, configured to connect to your cluster. For more installation details, seekubectl
.
Configure Grafana Agent
Paste the following script into your terminal and run it to configure Grafana Agent:
cat <<'EOF' |
kind: ConfigMap
metadata:
name: grafana-agent-traces
apiVersion: v1
data:
agent.yaml: |
traces:
configs:
- batch:
send_batch_size: 1000
timeout: 5s
name: default
receivers:
jaeger:
protocols:
grpc: null
thrift_binary: null
thrift_compact: null
thrift_http: null
remote_sampling:
strategy_file: /etc/agent/strategies.json
tls:
insecure: true
opencensus: null
otlp:
protocols:
grpc: null
http: null
zipkin: null
remote_write:
- basic_auth:
password: YOUR_TEMPO_PASSWORD
username: YOUR_TEMPO_USER
endpoint: YOUR_TEMPO_ENDPOINT
retry_on_failure:
enabled: false
scrape_configs:
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
strategies.json: '{"default_strategy": {"param": 0.001, "type": "probabilistic"}}'
EOF
(export NAMESPACE=default && kubectl apply -n $NAMESPACE -f -)
Be sure to replace NAMESPACE=default
with the Namespace into which you installed the Agent.
Also be sure to fill in the parameters in the remote_write
stanza. You can find your Cloud Tempo credentials in the Cloud Portal. Your Tempo push endpoint should look something like: tempo-us-central1.grafana.net:443
.
This ConfigMap configures the Agent to accept traces from every supported receiver and set a default set of labels using relabel_configs
. To learn more about the relabeling steps, refer to the Grafana Agent, and Grafana Agent traces_config.
This configuration also specifies a Jaeger trace sampling strategy. To learn more about Jaeger’s sampling strategies, refer to Jaeger’s documentation.
Deploy Grafana Agent
Install the Grafana Agent and its required resources into your cluster. To collect traces from your Kubernetes workloads, you can run the Agent as a Deployment or DaemonSet. In these instructions, we deploy the Agent as a one-replica Deployment.
Run the following command from your terminal to install the Grafana Agent into the default
Namespace of your Kubernetes cluster:
MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/v0.27.0/production/kubernetes/agent-traces.yaml NAMESPACE=default /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/v0.27.0/production/kubernetes/install-bare.sh)" | kubectl apply -f -
This installs a one-replica Grafana Agent Deployment into your cluster, and configures RBAC permissions for the Agent.
If you want to deploy the Agent into a different Namespace, change the NAMESPACE=default
variable, ensuring that this Namespace already exists. This also deploys the Agent ClusterRole
and ClusterRoleBinding
, as well as a service with the appropriate ports exposed. You can modify these defaults depending on the receivers you need. To learn more, see Distributor.
Related resources from Grafana Labs


