Ship Kubernetes metrics using Grafana Agent
Use the following instructions to deploy Grafana Agent directly into a Kubernetes cluster and configure it to scrape the kubelet
and cadvisor
endpoints on your cluster Nodes. You’ll also configure Grafana Agent to ship these scraped metrics to Grafana Cloud using its remote_write
feature.
Note: You can instead use Kubernetes Monitoring to guide you through the configuration process, allowing you to scrape kube-state-metrics, collect Kubernetes events, and use preconfigured dashboards and alerts. For comparisons of configuration methods, see Grafana Kubernetes Monitoring.
Before you begin
To complete the steps in this guide, you should have the following available:
- A Kubernetes cluster with role-based access control (RBAC) enabled.
- A Grafana Cloud account. To create an account, please see Grafana Cloud and click on Start for free.
- The
kubectl
command-line tool installed on your local machine, configured to connect to your cluster. To learn more aboutkubectl
, see the Kubernetes documentation.
Configure Grafana Agent
Paste the following script into your shell and run it to configure the Grafana Agent:
cat <<'EOF' |
kind: ConfigMap
metadata:
name: grafana-agent
apiVersion: v1
data:
agent.yaml: |
metrics:
wal_directory: /var/lib/agent/wal
global:
scrape_interval: 60s
external_labels:
cluster: YOUR_CLUSTER_NAME
configs:
- name: integrations
remote_write:
- url: YOUR_PROMETHEUS_REMOTE_WRITE_URL
basic_auth:
username: YOUR_PROMETHEUS_REMOTE_WRITE_USERNAME
password: YOUR_PROMETHEUS_REMOTE_WRITE_PASSWORD
scrape_configs:
- job_name: integrations/kubernetes/cadvisor
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
server_name: kubernetes
- job_name: integrations/kubernetes/kubelet
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
server_name: kubernetes
EOF
(export NAMESPACE=default && kubectl apply -n $NAMESPACE -f -)
If you deployed the Agent into a non-default Namespace in the previous step, replace NAMESPACE=default
in this command with the new Namespace. Be sure to also replace the capitalized variables beginning in YOUR_*
with the appropriate credentials.
This ConfigMap configures the Agent to scrape the cadvisor
and kubelet
endpoints in your cluster and ship these scraped metrics to Grafana Cloud.
To learn more about configuring the Agent, please see Configure Grafana Agent from the Agent docs.
Kubernetes ships a significant volume of metrics out-of-the-box. Kubernetes Monitoring generates a config that allowlists a tailored set of metrics to reduce your metrics usage and cost. To learn more about metric reduction techniques, please see Control Prometheus metrics usage.
Deploy Grafana Agent resources
In this step you’ll install the Grafana Agent and its required resources into your cluster.
Run the following command from your shell to install the Grafana Agent into the default
Namespace of your Kubernetes cluster:
MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/v0.24.2/production/kubernetes/agent-bare.yaml NAMESPACE=default /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/v0.24.2/production/kubernetes/install-bare.sh)" | kubectl apply -f -
This installs a single replica Grafana Agent StatefulSet into your cluster and configures RBAC permissions for the Agent. If you would like to deploy the Agent into a different Namespace, change the NAMESPACE=default
variable, ensuring that this Namespace already exists.