Istio integration for Grafana Cloud

Istio is an open-source service mesh platform designed to enhance the management, security, and observability of microservices in distributed applications. By providing a dedicated layer for handling communication between services, Istio facilitates features such as traffic management, security policies, and telemetry, making it easier for developers to implement and maintain complex microservices architectures typically across Kubernetes environments.

This integration supports Istio 1.20.0+

This integration includes 7 useful alerts and 4 pre-built dashboards to help monitor and visualize Istio metrics and logs.

Grafana Agent configuration

Before you begin

Metrics

Istio exposes a Prometheus metrics endpoint, /stats/prometheus, on all istio-proxy containers by default. You can verify that this endpoint is enabled by running the following command on an Istio pod’s istio-proxy container:

kubectl exec -it <pod-name> -n <namespace> -c istio-proxy -- sh -c 'curl http://localhost:15020/stats/prometheus'

Istio also exposes a Prometheus metrics endpoint, /metrics, on the discovery container of the istiod pod. You can verify that this endpoint is enabled by running the following command on this container:

kubectl exec -it istiod -n <namespace> -c discovery -- sh -c 'curl http://localhost:15014/metrics'

Logs

By default, Istio proxies send system logs to stdout.

In order to enable envoy proxy access logs as well, you can use an IstioOperator CR by adding the following field to your configuration:

spec:
  meshConfig:
    accessLogFile: /dev/stdout

Alternatively you can use the Telemetry API to enable access logs:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy

For up-to-date information on envoy proxy logging for Istio, refer to this documentation.

Configuration

The only way to actually configure this integration is to follow the instructions for Kubernetes Monitoring Configuration.

Find the Istio integration for Kubernetes here, and follow the configuration steps.

Install Istio integration for Grafana Cloud

In your Grafana Cloud stack, click Connections in the left-hand menu.
Find Istio and click its tile to open the integration.
Review the prerequisites in the Configuration Details tab and set up Grafana Agent to send Istio metrics and logs to your Grafana Cloud instance.
Click Install to add this integration’s pre-built dashboards and alerts to your Grafana Cloud instance, and you can start monitoring your Istio setup.

Post-install configuration for the Istio integration

To show logs and metrics signals correlated in your dashboards, as a single pane of glass, ensure the following:

job label must be set to integrations/istio (already configured in the snippets).

Configuration snippets for Grafana Agent

Below metrics.configs.scrape_configs, insert the following lines and change the URLs according to your environment:

    - job_name: integrations/istio
      static_configs:
        - targets: ["<hostname>:<port>"]

Below logs.configs.scrape_configs, insert the following lines according to your environment.

    - job_name: integrations/istio
      static_configs:
        - targets:
            - <hostname>
          labels:
            job: integrations/istio

Full example configuration for Grafana Agent

Refer to the following Grafana Agent configuration for a complete example that contains all the snippets used for the Istio integration. This example also includes metrics that are sent to monitor your Grafana Agent instance.

integrations:
  prometheus_remote_write:
  - basic_auth:
      password: <your_prom_pass>
      username: <your_prom_user>
    url: <your_prom_url>
  agent:
    enabled: true
    relabel_configs:
    - action: replace
      source_labels:
      - agent_hostname
      target_label: instance
    - action: replace
      target_label: job
      replacement: "integrations/agent-check"
    metric_relabel_configs:
    - action: keep
      regex: (prometheus_target_sync_length_seconds_sum|prometheus_target_scrapes_.*|prometheus_target_interval.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
      source_labels:
      - __name__
  # Add here any snippet that belongs to the `integrations` section.
  # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
logs:
  configs:
  - clients:
    - basic_auth:
        password: <your_loki_pass>
        username: <your_loki_user>
      url: <your_loki_url>
    name: integrations
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      # Add here any snippet that belongs to the `logs.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
    - job_name: integrations/istio
      static_configs:
        - targets:
            - <hostname>
          labels:
            job: integrations/istio
metrics:
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: <your_prom_pass>
        username: <your_prom_user>
      url: <your_prom_url>
    scrape_configs:
      # Add here any snippet that belongs to the `metrics.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
    - job_name: integrations/istio
      static_configs:
        - targets: ["<hostname>:<port>"]
  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal

Dashboards

The Istio integration installs the following dashboards in your Grafana Cloud instance to help monitor your system.

Istio logs
Istio overview
Istio services overview
Istio workloads overview

Istio overview (system)

Istio overview (control plane)

Istio services overview (client)

Alerts

The Istio integration includes the following useful alerts:

Alert	Description
IstioHighRequestLatencyWarning	Warning: High request latency between pods can indicate that there are performance issues within the k8s environment.
IstioGalleyValidationFailuresWarning	Warning: Istio Galley is reporting failures for a number of configurations.
IstioListenerConfigConflictsCritical	Critical: Istio Pilot is seeing a number of inbound and or outbound listener conflicts by envoy proxies.
IstioXDSConfigRejectionsWarning	Warning: Istio Pilot is seeing a number of xDS rejections from envoy proxies.
IstioHighHTTPRequestErrorsCritical	Critical: There are a high number of HTTP request errors in the Istio system.
IstioHighGRPCRequestErrorsCritical	Critical: There are a high number of GRPC request errors in the Istio system.
IstioMetricsDown	Critical: Istio metrics are down.

Metrics

The most important metrics provided by the Istio integration, which are used on the pre-built dashboards and Prometheus alerts, are as follows:

envoy_cluster_upstream_cx_rx_bytes_total
envoy_cluster_upstream_cx_tx_bytes_total
galley_validation_failed
galley_validation_passed
go_memstats_heap_alloc_bytes
go_memstats_heap_inuse_bytes
go_memstats_heap_sys_bytes
istio_agent_go_memstats_heap_alloc_bytes
istio_agent_go_memstats_heap_inuse_bytes
istio_agent_go_memstats_heap_sys_bytes
istio_agent_process_cpu_seconds_total
istio_agent_process_open_fds
istio_agent_process_resident_memory_bytes
istio_agent_process_virtual_memory_bytes
istio_build
istio_request_bytes_sum
istio_request_duration_milliseconds_count
istio_request_duration_milliseconds_sum
istio_requests_total
istio_response_bytes_sum
istio_tcp_received_bytes_total
istio_tcp_sent_bytes_total
istiod_uptime_seconds
pilot_conflict_inbound_listener
pilot_conflict_outbound_listener_tcp_over_current_tcp
pilot_info
pilot_k8s_cfg_events
pilot_total_xds_internal_errors
pilot_total_xds_rejects
pilot_xds_cds_reject
pilot_xds_eds_reject
pilot_xds_lds_reject
pilot_xds_pushes
pilot_xds_rds_reject
pilot_xds_write_timeout
process_cpu_seconds_total
process_open_fds
process_resident_memory_bytes
process_virtual_memory_bytes
sidecar_injection_failure_total
sidecar_injection_success_total
up

Changelog

# 1.0.0 - March 2023

- Initial release

Cost

By connecting your Istio instance to Grafana Cloud, you might incur charges. To view information on the number of active series that your Grafana Cloud account uses for metrics included in each Cloud tier, see Active series and dpm usage and Cloud tier pricing.