Grafana Cloud Grafana Kubernetes Monitoring Other configuration methods Ship Kubernetes logs using Grafana Agent

Ship Kubernetes logs using Grafana Agent

In this guide you’ll deploy the Grafana Agent into a Kubernetes cluster as a DaemonSet and configure it to collect logs for your Kubernetes workloads. You’ll then ship these logs to Grafana Cloud for storage and querying from your hosted Grafana instance.

Before you begin

To complete the steps in this guide, you should have the following available:

• A Kubernetes cluster with role-based access control (RBAC) enabled.
• A Grafana Cloud account. To create an account, please see Grafana Cloud and click on Start for free.
• The kubectl command-line tool installed on your local machine, configured to connect to your cluster. To learn more about kubectl, see the Kubernetes documentation.

Configure Grafana Agent

Paste the following script into your shell and run it to configure the Grafana Agent:

cat <<'EOF' | NAMESPACE=default /bin/sh -c 'kubectl apply -n $NAMESPACE -f -'

kind: ConfigMap
metadata:
  name: grafana-agent-logs
apiVersion: v1
data:
  agent.yaml: |
    logs:
      configs:
      - name: default
        clients:
        - url: YOUR_LOKI_REMOTE_WRITE_URL
          basic_auth:
            username: YOUR_LOKI_USERNAME
            password: YOUR_LOKI_PASSWORD
          external_labels:
            cluster: YOUR_CLUSTER_NAME
        positions:
          filename: /tmp/positions.yaml
        target_config:
          sync_period: 10s
        scrape_configs:
        - job_name: pod-logs
          kubernetes_sd_configs:
            - role: pod
          pipeline_stages:
            - docker: {}
          relabel_configs:
            - source_labels:
                - __meta_kubernetes_pod_node_name
              target_label: __host__
            - action: labelmap
              regex: __meta_kubernetes_pod_label_(.+)
            - action: replace
              replacement: $1
              separator: /
              source_labels:
                - __meta_kubernetes_namespace
                - __meta_kubernetes_pod_name
              target_label: job
            - action: replace
              source_labels:
                - __meta_kubernetes_namespace
              target_label: namespace
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_name
              target_label: pod
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_container_name
              target_label: container
            - replacement: /var/log/pods/*$1/*.log
              separator: /
              source_labels:
                - __meta_kubernetes_pod_uid
                - __meta_kubernetes_pod_container_name
              target_label: __path__

EOF

If you deployed the Agent into a non-default Namespace in the previous step, replace NAMESPACE=default in this command with the new Namespace. Be sure to also replace the capitalized variables beginning in YOUR_* with the appropriate credentials.

You can find your Cloud logs credentials in the Cloud Portal. Your logs push endpoint should look similar to the following: https://logs-prod-us-central1.grafana.net/loki/api/v1/push.

This ConfigMap configures the agent to tail Pod logs in the /var/log/pods/ directory and set job, pod, namespace and container labels. . To learn more about the relabeling steps, please see the following documentation:

• Promtail
• Grafana Agent logs_config (Grafana Agent uses an embedded Promtail)

Deploy Grafana Agent resources

In this step you’ll install the Grafana Agent and its required resources into your cluster.

Run the following command from your shell to install the Grafana Agent into the default Namespace of your Kubernetes cluster:

MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/agent-loki.yaml NAMESPACE=default /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install-bare.sh)" | kubectl apply -f -

This installs a Grafana Agent DaemonSet into your cluster and configures RBAC permissions for the Agent. If you would like to deploy the Agent into a different Namespace, change the NAMESPACE=default variable, ensuring that this Namespace already exists.