Grafana Cloud quickstart guidesGrafana Agent Kubernetes QuickstartsGrafana Agent Logs Kubernetes Quickstart

Grafana Agent Logs Kubernetes Quickstart

Introduction

In this guide you’ll deploy the Grafana Agent into a Kubernetes cluster as a DaemonSet and configure it to collect logs for your Kubernetes workloads. You’ll then ship these logs to Grafana Cloud for storage and querying from your hosted Grafana instance.

Kubernetes is an open-source container orchestration system that automates software container deployment, scaling, and management. With the Docker container engine, a logging driver (like json-file) writes the container stdout and stderr streams to a file on the node in JSON format. The Agent Pod tails this file, does some processing, and ships logs to Grafana Cloud.

Prerequisites

Before you begin, you should have the following available:

  • A Kubernetes cluster with role-based access control (RBAC) enabled.
  • A Grafana Cloud account. To create an account, please see Grafana Cloud and click on Start for free.
  • The kubectl command-line tool installed on your local machine, configured to connect to your cluster. You can read more about installing kubectl in the official documentation.

Step 1: Deploy Grafana Agent DaemonSet

In this step you’ll install the Grafana Agent and its required resources into your cluster.

To collect logs from your Kubernetes workloads, you can run the Agent as a DaemonSet, with an Agent on each cluster node. A sample deployment manifest can be found in the Agent GitHub repository.

Save the agent-loki.yaml manifest on your local machine and replace YOUR_NAMESPACE values with the Namespace into which you’ll install the Agent.

When you’re done, roll out the Agent using kubectl apply -f:

kubectl apply -f agent-loki.yaml

This will roll out the Agent ClusterRole and ClusterRoleBinding, as well as the Agent DaemonSet with the appropriate host volumes mounted into the Agent container.

In the next step, you’ll configure the Agent.

Step 2: Configure Grafana Agent

Copy the following ConfigMap manifest into a file:

apiVersion: v1
data:
  agent.yaml: |
    loki:
        configs:
          - clients:
              - url: YOUR_LOKI_ENDPOINT
                basic_auth:
                  username: YOUR_LOKI_USERNAME
                  password: YOUR_LOKI_PASSWORD
            name: default
            scrape_configs:
              - job_name: kubernetes-pods-name
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - source_labels:
                      - __meta_kubernetes_pod_label_name
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
              - job_name: kubernetes-pods-app
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - action: drop
                    regex: .+
                    source_labels:
                      - __meta_kubernetes_pod_label_name
                  - source_labels:
                      - __meta_kubernetes_pod_label_app
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
              - job_name: kubernetes-pods-direct-controllers
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - action: drop
                    regex: .+
                    separator: ""
                    source_labels:
                      - __meta_kubernetes_pod_label_name
                      - __meta_kubernetes_pod_label_app
                  - action: drop
                    regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
                    source_labels:
                      - __meta_kubernetes_pod_controller_name
                  - source_labels:
                      - __meta_kubernetes_pod_controller_name
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
              - job_name: kubernetes-pods-indirect-controller
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - action: drop
                    regex: .+
                    separator: ""
                    source_labels:
                      - __meta_kubernetes_pod_label_name
                      - __meta_kubernetes_pod_label_app
                  - action: keep
                    regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
                    source_labels:
                      - __meta_kubernetes_pod_controller_name
                  - action: replace
                    regex: ([0-9a-z-.]+)-[0-9a-f]{8,10}
                    source_labels:
                      - __meta_kubernetes_pod_controller_name
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
              - job_name: kubernetes-pods-static
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - action: drop
                    regex: ""
                    source_labels:
                      - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_label_component
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
        positions_directory: /tmp/positions
    server:
        http_listen_port: 8080
        log_level: info
kind: ConfigMap
metadata:
  name: grafana-agent-logs
  namespace: YOUR_NAMESPACE

Be sure to replace YOUR_NAMESPACE with the Namespace into which you installed the Agent.

Also be sure to fill in the parameters in the clients stanza. You can find your Cloud Logs credentials in the Cloud Portal. Your Logs push endpoint should look something like: https://logs-prod-us-central1.grafana.net/loki/api/v1/push.

When you’re done, roll out the ConfigMap using kubectl apply -f:

kubectl apply -f agent-loki-configmap.yaml

This ConfigMap configures the Agent to tail Pod logs in the /var/log/pods/ directory and set the appropriate labels. To learn more about the relabeling steps, please see kubernetes_logs.libsonnet, the Promtail docs, and Grafana Agent loki_config docs (the Agent uses an embedded Promtail).

Note: The above sample configuration assumes a Docker container runtime. If you’re using containerd or a runtime that publishes logs in a different format, you should change the pipeline_stages field to use the appropriate parsing stage. For containerd, replace - docker: {} with - cri: {}.

After deploying the ConfigMap, the Grafana Agent should start.

Step 3: Restart the Grafana Agent

After modifying the Agent’s configuration, you will need to restart the Agent Pod to pick up configuration changes. Use kubectl rollout to restart the Agent:

kubectl rollout restart ds/grafana-agent-logs

Conclusion

You’ve now deployed the Agent into your cluster, have configured it to collect Kubernetes workload logs, and are shippping these logs to Grafana Cloud.