Send Kubernetes OTel traces using Grafana Agent
You can deploy the Grafana Agent in a Kubernetes cluster as a deployment, and configure it to receive OpenTelemetry traces for your workloads. These traces are then buffered, and sent to Grafana Cloud for storage and querying from your hosted Grafana instance.
You can instrument your Kubernetes workloads and apps to emit spans by using client libraries from OpenTelemetry.
This content only covers receiving traces, not metrics, logs, or cluster events. With the addition of OpenTelemetry components to the Agent, more features and configurations for OTel are available. To learn more, refer to:
Before you begin
To complete the steps, have the following available:
- A Kubernetes cluster with role-based access control (RBAC) enabled
- A Grafana Cloud account. To create an account, navigate to Grafana Cloud, and click Create free account.
- The
kubectl
command-line tool installed on your local machine, configured to connect to your cluster. Refer tokubectl
installation. - An application that has the OpenTelemetry libraries instrumented and is deployed to the Kubernetes cluster ready to send traces
Configure Grafana Agent
Paste the following script into your terminal, and run it to deploy a configuration map for Grafana Agent into your currently selected kubectl
cluster:
cat <<'EOF' |
kind: ConfigMap
metadata:
name: grafana-agent-traces
apiVersion: v1
data:
agent.yaml: |
traces:
configs:
- batch:
send_batch_size: 1000
timeout: 5s
name: default
receivers:
jaeger:
protocols:
grpc: null
thrift_binary: null
thrift_compact: null
thrift_http: null
remote_sampling:
strategy_file: /etc/agent/strategies.json
tls:
insecure: true
opencensus: null
otlp:
protocols:
grpc: null
http: null
zipkin: null
remote_write:
- basic_auth:
password: YOUR_TEMPO_PASSWORD
username: YOUR_TEMPO_USER
endpoint: YOUR_TEMPO_ENDPOINT
retry_on_failure:
enabled: false
scrape_configs:
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
strategies.json: '{"default_strategy": {"param": 0.001, "type": "probabilistic"}}'
EOF
(export NAMESPACE=default && kubectl apply -n $NAMESPACE -f -)
Be sure to:
- Replace
NAMESPACE=default
with the namespace into which you would like to install the Agent. - Complete the Grafana Cloud parameters in the
remote_write
stanza.- YOUR_TEMPO_USER
- YOUR_TEMPO_PASSWORD
- YOUR_TEMPO_ENDPOINT
You can find the Tempo credentials in your Grafana Cloud Portal. Your Tempo push endpoint should look similar to this: tempo-us-central1.grafana.net:443
.
The deployed ConfigMap sets up the Agent to accept traces from the OpenTelemetry receiver, and sets a default set of labels using relabel_configs
. For more about the relabeling steps, refer to:
- Tempo agent docs
- Grafana Agent traces_config reference
Deploy Grafana Agent
Install the Grafana Agent using the previous deployed ConfigMap, and deploy the required resources into your cluster.
Run the following command from your shell to install the Grafana Agent into the default
Namespace of your Kubernetes cluster:
MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/v0.32.1/production/kubernetes/agent-traces.yaml NAMESPACE=default /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/v0.27.0/production/kubernetes/install-bare.sh)" | kubectl apply -f -
This installs a one-replica Grafana Agent Deployment into your cluster, and configures RBAC permissions for the Agent. If you want to deploy the Agent into a different Namespace, change the NAMESPACE=default
variable, ensuring that this Namespace already exists. This also applies the Agent ClusterRole
and ClusterRoleBinding
and a Kubernetes Service
with the appropriate ports connected.
grafana-agent-traces Service
The deployed cluster Service
is named grafana-agent-traces
, exposing otlp over grpc on port 4317 or http on port 4318. Refer to the snippet below for reference.
apiVersion: v1
kind: Service
metadata:
labels:
name: grafana-agent-traces
name: grafana-agent-traces
namespace: default
spec:
ports:
- name: grafana-agent-traces-otlp-grpc
port: 4317
protocol: TCP
targetPort: 4317
- name: grafana-agent-traces-otlp-http
port: 4318
protocol: TCP
targetPort: 4318
selector:
name: grafana-agent-traces
Workloads deployed to the Kubernetes cluster can now send OTel traces to grafana-agent-traces.$NAMESPACE.svc.cluster.local:4317
or with the reduced grafana-agent-traces:4318
for workloads running within the same namespace.
Check grafana-agent logs
The Grafana Agent logs are available at /var/log/grafana-agent.err.log
.
If the deployment succeeded, the following trace component line will be output.
ts=2023-04-01T21:23:10.949549748Z caller=zapadapter.go:78 level=info component=traces traces_config=grafana kind=receiver name=otlp pipeline=traces msg="Receiver started."
Search traces
After traces are delivered to the Grafana Cloud backend via the Agent Deployment, you can:
- Search for traces with the Grafana Explore trace integration.
- Query with TraceQL.
For example, below is a simple TraceQL snippet that selects all traces in the default namespace.
{.namespace="default"}
Related resources from Grafana Labs


