Menu
Grafana Cloud Grafana Kubernetes Monitoring Other configuration methods Ship Kubernetes metrics using Grafana Agent
Grafana Cloud

Ship Kubernetes metrics using Grafana Agent

Use the following instructions to deploy Grafana Agent directly into a Kubernetes cluster and configure it to scrape the kubelet and cadvisor endpoints on your cluster Nodes. You’ll also configure Grafana Agent to ship these scraped metrics to Grafana Cloud using its remote_write feature.

Note: You can instead use Kubernetes Monitoring to guide you through the configuration process, allowing you to scrape kube-state-metrics, collect Kubernetes events, and use preconfigured dashboards and alerts. For comparisons of configuration methods, see Grafana Kubernetes Monitoring.

Before you begin

To complete the steps in this guide, you should have the following available:

  • A Kubernetes cluster with role-based access control (RBAC) enabled.
  • A Grafana Cloud account. To create an account, please see Grafana Cloud and click on Start for free.
  • The kubectl command-line tool installed on your local machine, configured to connect to your cluster. To learn more about kubectl, see the Kubernetes documentation.

Configure Grafana Agent

Paste the following script into your shell and run it to configure the Grafana Agent:

cat <<'EOF' |

kind: ConfigMap
metadata:
  name: grafana-agent
apiVersion: v1
data:
  agent.yaml: |
    metrics:
      wal_directory: /var/lib/agent/wal
      global:
        scrape_interval: 60s
        external_labels:
          cluster: YOUR_CLUSTER_NAME
      configs:
      - name: integrations
        remote_write:
        - url: YOUR_PROMETHEUS_REMOTE_WRITE_URL
          basic_auth:
            username: YOUR_PROMETHEUS_REMOTE_WRITE_USERNAME
            password: YOUR_PROMETHEUS_REMOTE_WRITE_PASSWORD
        scrape_configs:
        - job_name: integrations/kubernetes/cadvisor
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
            - role: node
          relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
        - job_name: integrations/kubernetes/kubelet
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
            - role: node
          relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$1/proxy/metrics
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes

EOF
(export NAMESPACE=default && kubectl apply -n $NAMESPACE -f -)

If you deployed the Agent into a non-default Namespace in the previous step, replace NAMESPACE=default in this command with the new Namespace. Be sure to also replace the capitalized variables beginning in YOUR_* with the appropriate credentials.

This ConfigMap configures the Agent to scrape the cadvisor and kubelet endpoints in your cluster and ship these scraped metrics to Grafana Cloud.

To learn more about configuring the Agent, please see Configure Grafana Agent from the Agent docs.

Kubernetes ships a significant volume of metrics out-of-the-box. Kubernetes Monitoring generates a config that allowlists a tailored set of metrics to reduce your metrics usage and cost. To learn more about metric reduction techniques, please see Control Prometheus metrics usage.

Deploy Grafana Agent resources

In this step you’ll install the Grafana Agent and its required resources into your cluster.

Run the following command from your shell to install the Grafana Agent into the default Namespace of your Kubernetes cluster:

MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/v0.24.2/production/kubernetes/agent-bare.yaml NAMESPACE=default /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/v0.24.2/production/kubernetes/install-bare.sh)" | kubectl apply -f -

This installs a single replica Grafana Agent StatefulSet into your cluster and configures RBAC permissions for the Agent. If you would like to deploy the Agent into a different Namespace, change the NAMESPACE=default variable, ensuring that this Namespace already exists.