Grafana Kubernetes MonitoringOther Kubernetes configuration methodsDeploy Grafana Agent directlyShip Kubernetes logs using Grafana Agent

Ship Kubernetes logs using Grafana Agent

In this guide you’ll deploy the Grafana Agent into a Kubernetes cluster as a DaemonSet and configure it to collect logs for your Kubernetes workloads. You’ll then ship these logs to Grafana Cloud for storage and querying from your hosted Grafana instance.

Note: To learn how to collect Kubernetes events, annotate dashboards with events, and much more, see how to get preconfigured dashboards and alerts.

Before you begin

To complete the steps in this guide, you should have the following available:

  • A Kubernetes cluster with role-based access control (RBAC) enabled.
  • A Grafana Cloud account. To create an account, please see Grafana Cloud and click on Start for free.
  • The kubectl command-line tool installed on your local machine, configured to connect to your cluster. To learn more about kubectl, see the Kubernetes documentation.

Configure Grafana Agent

Paste the following script into your shell and run it to configure the Grafana Agent:

cat <<'EOF' | NAMESPACE=default /bin/sh -c 'kubectl apply -n $NAMESPACE -f -'

kind: ConfigMap
  name: grafana-agent-logs
apiVersion: v1
  agent.yaml: |      
      - name: default
            username: YOUR_LOKI_USERNAME
            password: YOUR_LOKI_PASSWORD
            cluster: YOUR_CLUSTER_NAME
          filename: /tmp/positions.yaml
          sync_period: 10s
        - job_name: pod-logs
            - role: pod
            - docker: {}
            - source_labels:
                - __meta_kubernetes_pod_node_name
              target_label: __host__
            - action: labelmap
              regex: __meta_kubernetes_pod_label_(.+)
            - action: replace
              replacement: $1
              separator: /
                - __meta_kubernetes_namespace
                - __meta_kubernetes_pod_name
              target_label: job
            - action: replace
                - __meta_kubernetes_namespace
              target_label: namespace
            - action: replace
                - __meta_kubernetes_pod_name
              target_label: pod
            - action: replace
                - __meta_kubernetes_pod_container_name
              target_label: container
            - replacement: /var/log/pods/*$1/*.log
              separator: /
                - __meta_kubernetes_pod_uid
                - __meta_kubernetes_pod_container_name
              target_label: __path__


If you deployed the Agent into a non-default Namespace in the previous step, replace NAMESPACE=default in this command with the new Namespace. Be sure to also replace the capitalized variables beginning in YOUR_* with the appropriate credentials.

You can find your Cloud logs credentials in the Cloud Portal. Your logs push endpoint should look similar to the following:

Note: If your K8s cluster doesn’t use Docker as a container runtime, replace docker: {} with cri: {} in the pipeline_stages section.

This ConfigMap configures the agent to tail Pod logs in the /var/log/pods/ directory and set job, pod, namespace and container labels. . To learn more about the relabeling steps, please see the following documentation:

Deploy Grafana Agent resources

In this step you’ll install the Grafana Agent and its required resources into your cluster.

Run the following command from your shell to install the Grafana Agent into the default Namespace of your Kubernetes cluster:

MANIFEST_URL= NAMESPACE=default /bin/sh -c "$(curl -fsSL" | kubectl apply -f -

This installs a Grafana Agent DaemonSet into your cluster and configures RBAC permissions for the Agent. If you would like to deploy the Agent into a different Namespace, change the NAMESPACE=default variable, ensuring that this Namespace already exists.