Grafana Cloud quickstart guidesGrafana Agent Kubernetes QuickstartsGrafana Agent Metrics Kubernetes Quickstart

Grafana Agent Metrics Kubernetes Quickstart

Introduction

In this guide you’ll learn how to deploy the Grafana Agent into a Kubernetes cluster and configure it to scrape the kubelet and cadvisor endpoints on your cluster Nodes. You’ll also configure the Agent to ship these scraped metrics to Grafana Cloud using its remote_write feature.

Note: The instructions in this quickstart guide mirror those from the Grafana Cloud Kubernetes Integration. To get started with Grafana Agent and the Kubernetes Integration, please click on the lighning bolt in the left-hand navigation panel of your hosted Grafana instance and follow the instructions for installing the Kubernetes Integration. The Kubernetes Integration provides preconfigured Kubernetes manifests for rolling out an Agent Deployment into your cluster to scrape metrics from cadvisor and kubelet endpoints, as well as prebuilt dashboards to visualize these metrics.

Kubernetes is an open-source container orchestration system that automates software container deployment, scaling, and management. By default, Kubernetes exposes several Prometheus-compatible metrics endpoints on both Nodes and the Control Plane.

You’ll collect Node cadvisor and kubelet metrics using the Grafana Agent, a lightweight telemetry collector based on Prometheus that only performs its scraping and remote_write functions. The Agent can also collect metrics, logs, and traces for storage in Grafana Cloud and Grafana Enterprise, as well as OSS deployments of Loki (logs), and Tempo (traces), Prometheus (metrics), and Cortex (metrics).

Note: The Agent Operator (beta) can also help you get up and running with Grafana Agent and Kubernetes.

Prerequisites

Before you begin, you should have the following available:

  • A Kubernetes cluster with role-based access control (RBAC) enabled.
  • A Grafana Cloud account. To create an account, please see Grafana Cloud and click on Start for free.
  • The kubectl command-line tool installed on your local machine, configured to connect to your cluster. You can read more about installing kubectl in the official documentation.

Step 1: Deploy Grafana Agent resources

In this step you’ll install the Grafana Agent and its required resources into your cluster.

Run the following command from your shell to install the Grafana Agent into the default Namespace of your Kubernetes cluster:

MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/main/production/kubernetes/agent-bare.yaml NAMESPACE=default /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install-bare.sh)" | kubectl apply -f -

This installs a single replica Grafana Agent Deployment into your cluster and configures RBAC permissions for the Agent. If you would like to deploy the Agent into a different Namespace, change the NAMESPACE=default variable, ensuring that this Namespace already exists.

In the next step, you’ll configure the Agent.

Step 2: Configure Grafana Agent

Paste the following script into your shell and run it to configure the Grafana Agent:

cat <<'EOF' |

kind: ConfigMap
metadata:
  name: grafana-agent
apiVersion: v1
data:
  agent.yaml: |
    server:
      http_listen_port: 12345
    prometheus:
      wal_directory: /tmp/grafana-agent-wal
      global:
        scrape_interval: 15s
        external_labels:
          cluster: YOUR_CLUSTER_NAME
      configs:
      - name: integrations
        remote_write:
        - url: YOUR_REMOTE_WRITE_URL
          basic_auth:
            username: YOUR_REMOTE_WRITE_USERNAME
            password: YOUR_REMOTE_WRITE_PASSWORD
        scrape_configs:
        - job_name: integrations/kubernetes/cadvisor
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
            - role: node
          metric_relabel_configs:
            - action: drop
              regex: container_([a-z_]+);
              source_labels:
                - __name__
                - image
            - action: drop
              regex: container_(network_tcp_usage_total|network_udp_usage_total|tasks_state|cpu_load_average_10s)
              source_labels:
                - __name__
          relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
        - job_name: integrations/kubernetes/kubelet
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
            - role: node
          relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$1/proxy/metrics
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes

EOF
(export NAMESPACE=default && kubectl apply -n $NAMESPACE -f -)

If you deployed the Agent into a non-default Namespace in the previous step, replace default in this ConfigMap with the new Namespace. Be sure to also replace the capitalized variables beginning in YOUR_*.

This ConfigMap configures the Agent to scrape cadvisor and kubelet endpoints in your cluster and ship scraped metrics to Grafana Cloud.

To learn more about configuring the Agent, please see Configure Grafana Agentfrom the Agent docs. After deploying the ConfigMap, the Grafana Agent should start.

Step 3: Restart the Grafana Agent

After modifying the Agent’s configuration, you will need to restart the Agent Pod to pick up configuration changes. Use kubectl rollout to restart the Agent:

kubectl rollout restart deployment/grafana-agent

Conclusion

You’ve now deployed the Agent into your cluster, have configured it to scrape the kubelet and cadvisor endpoints, and are shippping these scraped metrics to Grafana Cloud using its remote_write feature.

From here, you can deploy additional Agents to collect logs and traces in your Kubernetes clusters.