Grafana Cloud

Set up Kubernetes Event monitoring (beta)

Grafana Agent bundles an eventhandler integration that watches for Kubernetes Events in your clusters and ships these to Grafana Cloud Loki. Kubernetes controllers emit Events as they perform operations in your cluster (like starting containers, scheduling Pods, etc.) and these can be a rich source of logging information to help you debug, monitor, and alert on your Kubernetes workloads. Generally, these Events can be queried using kubectl get event or kubectl describe; with the eventhandler integration enabled, you can query these directly from Grafana Cloud.

Before you start

To begin, you’ll need the following:

  • A Kubernetes cluster
  • The kubectl command-line tool installed and available on your machine
  • A Grafana Cloud account or Loki instance that will receive log entries

Deployment options

The eventhandler integration is one of several integrations embedded directly into Grafana Agent. You can run the integration in several ways:

  • A dedicated Grafana Agent StatefulSet running only the eventhandler integration
  • As part of an existing Agent StatefulSet

NOTE: Although you can run the integration without persistent storage, we recommend running it with dedicated disk storage (StatefulSet or Deployment with PersistentVolume & PersistentVolumeClaim) to take advantage of its caching feature. Kubernetes events have a lifespan of an hour; after an hour, they are deleted from the cluster’s internal key-value store. If you restart the integration within an hour of it going down, eventhandler will re-ship any Events present in the cluster’s internal store unless the cache file is provided.

Option 1: Run a dedicated eventhandler

To run a dedicated eventhandler StatefulSet and for full documentation and configuration instructions, see eventhandler_config from the Grafana Agent documentation. These docs provide sample manifests and configuration for an Agent StatefulSet running only the eventhandler integration.

You can also use a Deployment with a PersistentVolume and PersistentVolumeClaim or use node-local storage, but these methods are outside the scope of this guide and require modifying the provided manifests and instructions.

Option 2: Enable eventhandler in an existing Agent Deployment or StatefulSet

To enable the eventhandler integration in an existing Grafana Agent setup or to avoid running another Agent in your cluster, you can modify your existing Agent’s configuration to enable the integration.

Note: If you’re using a Deployment you should attach persistent disk storage and appropriately configure the integration’s cache_path to take advantage of eventhandler’s Event caching. This isn’t necessary but will prevent double-shipping cluster Events to Loki in the event of an Agent restart. To learn more about configuring a PersistentVolume for storage, please see Configure a Pod to Use a PersistentVolume for Storage from the K8s docs.

  1. Enable the integration

    Modify your existing Agent configuration by adding the following stanza to your Agent’s agent.yaml or ConfigMap:

      . . .
      . . .
        cache_path: "/etc/eventhandler/eventhandler.cache"
        logs_instance: "default"
      . . .

    This block enables the integration and instructs it to cache the last Event shipped at the path provided by cache_path. For a full configuration reference, please see eventhandler_config from the Agent documentation.

  2. Enable the logs instance.

    Add the following block of Agent logs config:

    server: . . .
    metrics: . . .
      ## see above
      . . .
        - name: default
            ## you may need to replace this with a different endpoint
            - url:
                username: YOUR_LOKI_USER
                password: YOUR_LOKI_API_KEY
                cluster: 'cloud'
                job: 'integrations/kubernetes/eventhandler'
            filename: /tmp/positions0.yaml

    This block enables an instance of Agent’s logs subsystem (embedded promtail) and configures it with the appropriate Loki credentials:

    • default determines where Events get shipped as Loki log lines. You can also set default labels on log lines using the external_labels parameter. The name must match logs_instance in the integrations config block.

    For full logs_config reference, please see logs_config from the Agent docs.

    You can find your Loki credentials in your org’s Grafana Cloud Portal.

  3. Run eventhandler

    To run eventhandler, you need to pass in the following flag when you run Agent:


    This enables the latest version of the Agent integration subsystem. To learn more, please see Integrations Revamp from the Agent documentation.

    A full Kubernetes container spec should be similar to this one:

      - name: agent
        image: grafana/agent:latest
        imagePullPolicy: IfNotPresent
          - -config.file=/etc/agent/agent.yaml
          - -enable-features=integrations-next
          - /bin/grafana-agent
          - name: HOSTNAME
                fieldPath: spec.nodeName
          - containerPort: 12345
            name: http-metrics
          ## Should use a ConfigMap volume, stores Agent config
          - name: grafana-agent
            mountPath: /etc/agent
          ## Optional, but should use a persistent volume, stores Event cache
          - name: eventhandler-cache
            mountPath: /etc/eventhandler

    You should modify these parameters depending on your architecture and configured PersistentVolumes and ConfigMaps.

  4. Add ClusterRole events permission

    You also need to allow Agent’s ClusterRole to access the events resource from K8s API:

    kind: ClusterRole
      name: grafana-agent
      - apiGroups:
          - ''
          - nodes
          - nodes/proxy
          - services
          - endpoints
          - pods
          ## added "events" here
          - events
          - get
          - list
          - watch
      - nonResourceURLs:
          - /metrics
          - get

    eventhandler only requires get list watch for the events resource, but for clarity we’ve appended the required permission to the default ClusterRole provided by the K8s integration (which also allows Prometheus service discovery).

Please surface any issues with this integration in the Grafana Agent GitHub Repo or on the Grafana Labs Community Slack (in #agent).

eventhandler is enabled by default in the latest version of the Kubernetes Monitoring agent manifests.