--- title: "Cilium Enterprise integration | Grafana Cloud documentation" description: "Learn about Cilium Enterprise Grafana Cloud integration." --- # Cilium Enterprise integration for Grafana Cloud The Cilium Enterprise integration uses Grafana Alloy to collect metrics exposed by the Cilium Operator, Cilium Agent and its components, as well as Hubble. A series of dashboards have been provided, both for overviews and per-component basis. This integration includes 18 useful alerts and 20 pre-built dashboards to help monitor and visualize Cilium Enterprise metrics. ## Kubernetes instructions Instructions for Kubernetes ### Before you begin with Kubernetes **Please note**: These instructions assume the use of the [Kubernetes Monitoring Helm chart](https://github.com/grafana/k8s-monitoring-helm) This integration monitors a Cilium Enterprise & Hubble Enterprise deployment that has metrics exporters enabled. Please ensure you have completed the following setup steps: - Enabled the [embedded Prometheus exporter in your Cilium deployment](https://docs.cilium.io/en/stable/operations/metrics/#installation) to collect and expose metrics - Enabled the [embedded Prometheus exporter in Hubble](https://docs.cilium.io/en/stable/operations/metrics/#id1) if you want Hubble metrics to be included. Once the exporters have been enabled, the metrics will be automatically exposed and available for collection by either Prometheus or Grafana Alloy deployed to your cluster. This integration assumes Hubble metrics have been enabled for: - dns - drop - tcp - flow - icmp - http e.g. via a helm command similar to the following, adjusted for Cilium Enterprise: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none helm install --version 1.12.2 \ --namespace kube-system \ --set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" ``` Cilium version 1.12.2 and greater is supported. ### Configuration snippets for Kubernetes Helm chart The following snippets provide examples to guide you through the configuration process. To scrape your Cilium Enterprise instances, **manually** modify your Kubernetes Monitoring Helm chart with these configuration snippets. Replace any values between the angle brackets `<>` in the provided snippets with your desired configuration values. #### Metrics snippets Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```alloy # Replace any values between the angle brackets '<>', with your desired configuration alloy-metrics: extraConfig: |- // Cilium Agent discovery.kubernetes "cilium_agent" { role = "service" selectors { role = "service" label = "k8s-app=cilium" } } discovery.relabel "cilium_agent" { targets = discovery.kubernetes.cilium_agent.targets rule { source_labels = ["__meta_kubernetes_endpoint_port_name"] regex = "metrics" action = "keep" } rule { source_labels = ["__meta_kubernetes_service_label_k8s_app"] target_label = "k8s_app" } } prometheus.scrape "cilium_agent" { targets = discovery.relabel.cilium_agent.output job_name = "integrations/cilium-enterprise/cilium-agent" honor_labels = true forward_to = [prometheus.remote_write.grafana_cloud_metrics.receiver] } // Cilium Operator discovery.kubernetes "cilium_operator" { role = "service" selectors { role = "service" label = "name=cilium-operator,io.cilium/app=operator" } } discovery.relabel "cilium_operator" { targets = discovery.kubernetes.cilium_operator.targets rule { source_labels = ["__meta_kubernetes_endpoint_port_name"] regex = "metrics" action = "keep" } rule { source_labels = ["__meta_kubernetes_service_label_io_cilium_app_app"] target_label = "io_cilium_app" } } prometheus.scrape "cilium_operator" { targets = discovery.relabel.cilium_operator.output job_name = "integrations/cilium-enterprise/cilium-operator" honor_labels = true forward_to = [prometheus.remote_write.grafana_cloud_metrics.receiver] } // Hubble Relay discovery.kubernetes "hubble_relay" { role = "service" selectors { role = "service" label = "k8s-app=hubble-relay" } } discovery.relabel "hubble_relay" { targets = discovery.kubernetes.hubble_relay.targets rule { source_labels = ["__meta_kubernetes_endpoint_port_name"] regex = "metrics" action = "keep" } } prometheus.scrape "hubble_relay" { targets = discovery.relabel.hubble_relay.output job_name = "integrations/cilium-enterprise/hubble-relay" forward_to = [prometheus.remote_write.grafana_cloud_metrics.receiver] } // Hubble discovery.kubernetes "hubble" { role = "service" selectors { role = "service" label = "k8s-app=hubble" } } discovery.relabel "hubble" { targets = discovery.kubernetes.services.targets rule { source_labels = ["__meta_kubernetes_endpoint_port_name"] regex = "hubble-metrics" action = "keep" } } prometheus.scrape "hubble" { targets = discovery.relabel.hubble.output job_name = "integrations/cilium-enterprise/hubble" honor_labels = true forward_to = [prometheus.remote_write.grafana_cloud_metrics.receiver] } // Hubble Enterprise discovery.kubernetes "hubble_enterprise" { role = "service" selectors { role = "service" label = "app.kubernetes.io/name=hubble-enterprise" } } discovery.relabel "hubble_enterprise" { targets = discovery.kubernetes.hubble_enterprise.targets rule { source_labels = ["__meta_kubernetes_endpoint_port_name"] regex = "metrics" action = "keep" } } prometheus.scrape "hubble_enterprise" { targets = discovery.relabel.hubble_enterprise.output job_name = "integrations/cilium-enterprise/hubble-enterprise" honor_labels = true forward_to = [prometheus.remote_write.grafana_cloud_metrics.receiver] } // Hubble Timescape Ingester discovery.kubernetes "hubble_timescape_ingester" { role = "service" selectors { role = "service" label = "app.kubernetes.io/name=hubble-timescape-ingester,app.kubernetes.io/component=ingester" } } discovery.relabel "hubble_timescape_ingester" { targets = discovery.kubernetes.hubble_timescape_ingester.targets rule { source_labels = ["__meta_kubernetes_endpoint_port_name"] regex = "metrics" action = "keep" } } prometheus.scrape "hubble_timescape_ingester" { targets = discovery.relabel.hubble_timescape_ingester.output job_name = "integrations/cilium-enterprise/hubble-timescape-ingester" honor_labels = true forward_to = [prometheus.remote_write.grafana_cloud_metrics.receiver] } // Hubble Timescape Server discovery.kubernetes "hubble_timescape_server" { role = "service" selectors { role = "service" label = "app.kubernetes.io/name=hubble-timescape-server,app.kubernetes.io/component=server" } } discovery.relabel "hubble_timescape_server" { targets = discovery.kubernetes.hubble_timescape_server.targets rule { source_labels = ["__meta_kubernetes_endpoint_port_name"] regex = "metrics" action = "keep" } } prometheus.scrape "hubble_timescape_server" { targets = discovery.relabel.hubble_timescape_server.output job_name = "integrations/cilium-enterprise/hubble-timescape-server" honor_labels = true forward_to = [prometheus.remote_write.grafana_cloud_metrics.receiver] } ``` ## Dashboards The Cilium Enterprise integration installs the following dashboards in your Grafana Cloud instance to help monitor your system. - Cilium / Agent Overview - Cilium / Components / API - Cilium / Components / Agent - Cilium / Components / BPF - Cilium / Components / Conntrack - Cilium / Components / Datapath - Cilium / Components / External HA FQDN Proxy - Cilium / Components / FQDN Proxy - Cilium / Components / Identities - Cilium / Components / Kubernetes - Cilium / Components / L3 Policy - Cilium / Components / L7 Proxy - Cilium / Components / Network - Cilium / Components / Nodes - Cilium / Components / Policy - Cilium / Components / Resource Utilization - Cilium / Operator - Cilium / Overview - Hubble / Overview - Hubble / Timescape **Cilium Overview** **Cilium Overview (2)** **Cilium Agent Overview** ## Alerts The Cilium Enterprise integration includes the following useful alerts: **Cilium Endpoints** Expand table | Alert | Description | |------------------------------------------------------------|--------------------------------------------------------------------------------------------------| | CiliumAgentEndpointFailures | Warning: Cilium Agent endpoints in the invalid state. | | CiliumAgentEndpointUpdateFailure | Warning: API calls to Cilium Agent API to create or update Endpoints are failing. | | CiliumAgentContainerNetworkInterfaceApiErrorEndpointCreate | Info: Cilium Endpoint API endpoint rate limiter is reporting errors while doing endpoint create. | | CiliumAgentApiEndpointErrors | Warning: API calls to Cilium Endpoints API are failing due to server errors. | **Cilium IPAM** Expand table | Alert | Description | |-----------------------------------|--------------------------------------------------------------------------------------------| | CiliumOperatorExhaustedIpamIps | Critical: Cilium Operator has exhausted its IPAM IPs. | | CiliumOperatorLowAvailableIpamIps | Warning: Cilium Operator has used up over 90% of its available IPs. | | CiliumOperatorEniIpamErrors | Critical: Cilium Operator has high error rate while trying to create/attach ENIs for IPAM. | **Cilium Maps** Expand table | Alert | Description | |---------------------------------|-------------------------------------------------------------------------------| | CiliumAgentMapOperationFailures | Warning: Cilium Agent is experiencing errors updating BPF maps on Agent Pod. | | CiliumAgentBpfMapPressure | Warning: Map on Cilium Agent Pod is currently experiencing high map pressure. | **Cilium NAT** Expand table | Alert | Description | |-------------------------|-----------------------------------------------------------------------------------------------| | CiliumAgentNatTableFull | Critical: Cilium Agent Pod is dropping packets due to “No mapping for NAT masquerade” errors. | **Cilium API** Expand table | Alert | Description | |-----------------------------|------------------------------------------------------------------| | CiliumAgentApiHighErrorRate | Info: Cilium Agent API on Pod is experiencing a high error rate. | **Cilium Conntrack** Expand table | Alert | Description | |------------------------------------------------|----------------------------------------------------------------------------| | CiliumAgentConntrackTableFull | Critical: Ciliums conntrack map is failing on new insertions on Agent Pod. | | CiliumAgentConnTrackFailedGarbageCollectorRuns | Warning: Cilium Agent Conntrack GC runs are failing on Agent Pod. | **Cilium Drops** Expand table | Alert | Description | |---------------------------|--------------------------------------------------------------------------------| | CiliumAgentHighDeniedRate | Info: Cilium Agent is experiencing a high drop rate due to policy rule denies. | **Cilium Policy** Expand table | Alert | Description | |------------------------------|--------------------------------------------------------------| | CiliumAgentPolicyMapPressure | Warning: Cilium Agent is experiencing high BPF map pressure. | **Cilium Identity** Expand table | Alert | Description | |---------------------------------------|---------------------------------------------------------------------------------------------------------| | CiliumNodeLocalHighIdentityAllocation | Warning: Cilium is using a very high percent (over 80%) of its maximum per-node identity limit (65535). | | RunningOutOfCiliumClusterIdentities | Warning: Cilium is using a very high percent of its maximum cluster identity limit (65280). | **Cilium Nodes** Expand table | Alert | Description | |------------------------|-------------------------------------------------------------------| | CiliumUnreachableNodes | Info: Cilium Agent is reporting unreachable Nodes in the cluster. | ## Metrics The most important metrics provided by the Cilium Enterprise integration, which are used on the pre-built dashboards and Prometheus alerts, are as follows: - cilium\_agent\_api\_process\_time\_seconds\_count - cilium\_agent\_api\_process\_time\_seconds\_sum - cilium\_api\_limiter\_processed\_requests\_total - cilium\_bpf\_map\_ops\_total - cilium\_bpf\_map\_pressure - cilium\_controllers\_runs\_duration\_seconds\_count - cilium\_controllers\_runs\_duration\_seconds\_sum - cilium\_controllers\_runs\_total - cilium\_datapath\_conntrack\_gc\_duration\_seconds\_count - cilium\_datapath\_conntrack\_gc\_duration\_seconds\_sum - cilium\_datapath\_conntrack\_gc\_entries - cilium\_datapath\_conntrack\_gc\_key\_fallbacks\_total - cilium\_datapath\_conntrack\_gc\_runs\_total - cilium\_drop\_bytes\_total - cilium\_drop\_count\_total - cilium\_endpoint\_regeneration\_time\_stats\_seconds\_count - cilium\_endpoint\_regeneration\_time\_stats\_seconds\_sum - cilium\_endpoint\_regenerations\_total - cilium\_endpoint\_state - cilium\_errors\_warnings\_total - cilium\_forward\_bytes\_total - cilium\_forward\_count\_total - cilium\_identity - cilium\_ip\_addresses - cilium\_k8s\_client\_api\_calls\_total - cilium\_k8s\_client\_api\_latency\_time\_seconds\_count - cilium\_k8s\_client\_api\_latency\_time\_seconds\_sum - cilium\_kubernetes\_events\_received\_total - cilium\_kubernetes\_events\_total - cilium\_nodes\_all\_events\_received\_total - cilium\_nodes\_all\_num - cilium\_operator\_ces\_queueing\_delay\_seconds\_bucket - cilium\_operator\_ces\_sync\_errors\_total - cilium\_operator\_ec2\_api\_duration\_seconds\_bucket - cilium\_operator\_identity\_gc\_entries - cilium\_operator\_identity\_gc\_runs - cilium\_operator\_ipam\_allocation\_ops - cilium\_operator\_ipam\_deficit\_resolver\_duration\_seconds\_bucket - cilium\_operator\_ipam\_interface\_creation\_ops - cilium\_operator\_ipam\_ips - cilium\_operator\_ipam\_k8s\_sync\_queued\_total - cilium\_operator\_ipam\_nodes - cilium\_operator\_ipam\_resync\_queued\_total - cilium\_operator\_ipam\_resync\_total - cilium\_operator\_number\_of\_ceps\_per\_ces\_sum - cilium\_operator\_process\_cpu\_seconds\_total - cilium\_operator\_process\_open\_fds - cilium\_operator\_process\_resident\_memory\_bytes - cilium\_operator\_process\_virtual\_memory\_bytes - cilium\_policy - cilium\_policy\_endpoint\_enforcement\_status - cilium\_policy\_l7\_denied\_total - cilium\_policy\_l7\_forwarded\_total - cilium\_policy\_l7\_received\_total - cilium\_proxy\_redirects - cilium\_proxy\_upstream\_reply\_seconds\_count - cilium\_proxy\_upstream\_reply\_seconds\_sum - cilium\_services\_events\_total - cilium\_triggers\_policy\_update\_call\_duration\_seconds\_count - cilium\_triggers\_policy\_update\_call\_duration\_seconds\_sum - cilium\_unreachable\_nodes - cilium\_version - hubble\_dns\_queries\_total - hubble\_dns\_response\_types\_total - hubble\_dns\_responses\_total - hubble\_drop\_total - hubble\_flows\_processed\_total - hubble\_http\_request\_duration\_seconds\_bucket - hubble\_http\_requests\_total - hubble\_http\_responses\_total - hubble\_icmp\_total - hubble\_port\_distribution\_total - hubble\_tcp\_flags\_total - isovalent\_external\_dns\_proxy\_policy\_l7\_total - isovalent\_external\_dns\_proxy\_processing\_duration\_seconds - isovalent\_external\_dns\_proxy\_update\_errors\_total - isovalent\_external\_dns\_proxy\_update\_queue\_size - timescape\_clickhouse\_queries\_duration\_seconds\_bucket - timescape\_clickhouse\_queries\_results\_count - timescape\_clickhouse\_queries\_results\_sum - timescape\_ingestor\_flows\_ingested\_total - timescape\_ingestor\_ingest\_duration\_seconds\_bucket - timescape\_ingestor\_ingest\_running - timescape\_ingestor\_ingestfilter\_batch\_duration\_seconds\_bucket - timescape\_ingestor\_ingestfilter\_filtered\_errors\_total - timescape\_ingestor\_ingestfilter\_filtered\_skipped\_total - timescape\_ingestor\_ingestfilter\_filtered\_total - timescape\_ingestor\_ingestlog\_getinfo\_queries - up ## Changelog md ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```md # 1.0.0 - June 2024 * Update Mixin to latest version - Removed pod filter from alert rules - Added thresholds for alerts using rate() - Added aggregation label support # 0.0.4 - November 2023 * Replaced Angular dashboard panels with React panels # 0.0.3 - July 2023 * Added support for using the integration in the Grafana Cloud Kubernetes App * Update all scrape intervals to be 60s * Fix job name to correct value in static agent config # 0.0.2 - January 2023 * Update mixin to latest version: - Add new alert `CiliumOperatorEniIpamErrors` to alert on errors related to allocating new IPAM addresses and situations where nodes are experiencing IPAM exhaustion - Fix alert conditions to trigger correctly # 0.0.1 - October 2022 * Initial release ``` ## Cost By connecting your Cilium Enterprise instance to Grafana Cloud, you might incur charges. To view information on the number of active series that your Grafana Cloud account uses for metrics included in each Cloud tier, see [Active series and dpm usage](/docs/grafana-cloud/fundamentals/active-series-and-dpm/) and [Cloud tier pricing](/products/cloud/pricing/).