Set up Kubernetes Event monitoring (beta)
Grafana Agent bundles an
eventhandler integration that watches for Kubernetes Events in your clusters and ships these to Grafana Cloud Loki. Kubernetes controllers emit Events as they perform operations in your cluster (like starting containers, scheduling Pods, etc.) and these can be a rich source of logging information to help you debug, monitor, and alert on your Kubernetes workloads. Generally, these Events can be queried using
kubectl get event or
kubectl describe; with the eventhandler integration enabled, you can query these directly from Grafana Cloud.
Before you start
To begin, you’ll need the following:
- A Kubernetes cluster
kubectlcommand-line tool installed and available on your machine
- A Grafana Cloud account or Loki instance that will receive log entries
The eventhandler integration is one of several integrations embedded directly into Grafana Agent. You can run the integration in several ways:
- A dedicated Grafana Agent StatefulSet running only the eventhandler integration
- As part of an existing Agent StatefulSet
NOTE: Although you can run the integration without persistent storage, we recommend running it with dedicated disk storage (StatefulSet or Deployment with PersistentVolume & PersistentVolumeClaim) to take advantage of its caching feature. Kubernetes events have a lifespan of an hour; after an hour, they are deleted from the cluster’s internal key-value store. If you restart the integration within an hour of it going down, eventhandler will re-ship any Events present in the cluster’s internal store unless the cache file is provided.
Option 1: Run a dedicated eventhandler
To run a dedicated eventhandler StatefulSet and for full documentation and configuration instructions, see eventhandler_config from the Grafana Agent documentation. These docs provide sample manifests and configuration for an Agent StatefulSet running only the eventhandler integration.
You can also use a Deployment with a PersistentVolume and PersistentVolumeClaim or use node-local storage, but these methods are outside the scope of this guide and require modifying the provided manifests and instructions.
Option 2: Enable eventhandler in an existing Agent Deployment or StatefulSet
To enable the eventhandler integration in an existing Grafana Agent setup or to avoid running another Agent in your cluster, you can modify your existing Agent’s configuration to enable the integration.
Note: If you’re using a Deployment you should attach persistent disk storage and appropriately configure the integration’s
cache_pathto take advantage of eventhandler’s Event caching. This isn’t necessary but will prevent double-shipping cluster Events to Loki in the event of an Agent restart. To learn more about configuring a PersistentVolume for storage, please see Configure a Pod to Use a PersistentVolume for Storage from the K8s docs.
Enable the integration
Modify your existing Agent configuration by adding the following stanza to your Agent’s
server: . . . metrics: . . . integrations: eventhandler: cache_path: "/etc/eventhandler/eventhandler.cache" logs_instance: "default" . . .
This block enables the integration and instructs it to cache the last Event shipped at the path provided by
cache_path. For a full configuration reference, please see eventhandler_config from the Agent documentation.
Add the following block of Agent
server: . . . metrics: . . . integrations: ## see above . . . logs: configs: - name: default clients: ## you may need to replace this with a different endpoint - url: https://logs-prod-us-central1.grafana.net/api/prom/push basic_auth: username: YOUR_LOKI_USER password: YOUR_LOKI_API_KEY external_labels: cluster: 'cloud' job: 'integrations/kubernetes/eventhandler' positions: filename: /tmp/positions0.yaml
This block enables an instance of Agent’s logs subsystem (embedded promtail) and configures it with the appropriate Loki credentials:
defaultdetermines where Events get shipped as Loki log lines. You can also set default labels on log lines using the
logs_configreference, please see logs_config from the Agent docs.
You can find your Loki credentials in your org’s Grafana Cloud Portal.
eventhandler, you need to pass in the following flag when you run Agent:
This enables the latest version of the Agent integration subsystem. To learn more, please see Integrations Revamp from the Agent documentation.
A full Kubernetes
containerspec should be similar to this one:
containers: - name: agent image: grafana/agent:latest imagePullPolicy: IfNotPresent args: - -config.file=/etc/agent/agent.yaml - -enable-features=integrations-next command: - /bin/grafana-agent env: - name: HOSTNAME valueFrom: fieldRef: fieldPath: spec.nodeName ports: - containerPort: 12345 name: http-metrics volumeMounts: ## Should use a ConfigMap volume, stores Agent config - name: grafana-agent mountPath: /etc/agent ## Optional, but should use a persistent volume, stores Event cache - name: eventhandler-cache mountPath: /etc/eventhandler
You should modify these parameters depending on your architecture and configured PersistentVolumes and ConfigMaps.
You also need to allow Agent’s ClusterRole to access the
eventsresource from K8s API:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: grafana-agent rules: - apiGroups: - '' resources: - nodes - nodes/proxy - services - endpoints - pods ## added "events" here - events verbs: - get - list - watch - nonResourceURLs: - /metrics verbs: - get
eventhandler only requires
eventsresource, but for clarity we’ve appended the required permission to the default
ClusterRoleprovided by the K8s integration (which also allows Prometheus service discovery).
Please surface any issues with this integration in the Grafana Agent GitHub Repo or on the Grafana Labs Community Slack (in
eventhandler is enabled by default in the latest version of the Kubernetes Monitoring agent manifests.