Hubble

Overview
Installation
Change log
Related content

Hubble datasource plugin for Grafana

Hubble datasource plugin allows querying data produced by Hubble - the observability layer of Cilium - from Grafana.

Some of the features work with Cilium OSS, others require Isovalent Cilium Enterprise - those are marked with an info box.

Configuration

There are three underlying data stores that can be configured in Hubble data source:

Timescape - an observability and analytics platform for eBPF collected Hubble flow data and Tetragon process events which allows querying historical data shipped as part of Isovalent Cilium Enterprise. It's used in Flows and Process Ancestry queries.
Prometheus - must store Hubble metrics, used in Service Map query
Tempo - used in Flows query

Queries

Hubble datasource plugin supports three query types: Service Map, Flows, and Process Ancestry. Queries support dashboard variables, with possible limitations.

Service Map

The Service Map query renders an interactive service map from Hubble HTTP metrics. It's designed to be visualized as a node graph. The graph nodes contain dropdowns with links to detailed Prometheus queries.

For Service Map to work correctly, Hubble HTTP metrics have to be enabled with correct labels: at least source and destination. Filters require additional labels, e.g. destination_namespace. Refer to the Hubble metrics documentation for configuration details. This also requires Hubble Layer 7 visibility to be configured for your HTTP services, in order to the HTTP metrics to be generated. Below is an example of Helm values for Cilium with Hubble metrics enabled:

hubble:
  metrics:
    enabled:
      - "httpV2:sourceContext=workload-name|pod-name|reserved-identity;destinationContext=workload-name|pod-name|reserved-identity;labelsContext=source_namespace,destination_namespace,traffic_direction"
      - "kafka:sourceContext=workload-name|pod-name|reserved-identity;destinationContext=workload-name|pod-name|reserved-identity;labelsContext=source_namespace,destination_namespace,traffic_direction"
    enableOpenMetrics: true
    serviceMonitor:
      enabled: true
    dashboards:
      enabled: true
      namespace: monitoring

Flows

The Flows query requires a connection to a Timescape server storing Hubble flows. It retrieves raw network flows for the selected time range and filters.

If applications propagate trace context following the W3C standard, then Hubble flows will contain trace ID. Providing a Tempo datasource is linked in Hubble datasource settings, trace IDs in Flows query results will link to Tempo queries.

Process Ancestry

The Process Ancestry query requires a connection to a Timescape server storing Tetragon process events. It renders a process ancestry tree for the selected pod. It's designed to be used with Process Ancestry panel plugin.

Limitations

At the moment Timescape connection doesn't support TLS and authentication. That means Timescape has to be deployed without TLS or Hubble RBAC enabled. Support for both is planned before the first stable (production-ready) release.
Some of the filters are multi-select and multi-value dashboard variables, but not all. Expect breaking changes to the filters before the first stable (production-ready) release.

Installing Hubble on Grafana Cloud:

For more information, visit the docs on plugin installation.

Changelog

1.0.0

Support Timescape v1.1.0
Process Ancestry: fetch full process ancestry

1.0.0-beta.5

Log Timescape client at debug level instead of info
Avoid using Grafana class-names directly

1.0.0-beta.4

Add support for TLS, OAuth2 token pass through, and basic auth
Convert latency dataframe field into float64 seconds with unit
Add Tetragon Pod Process Ancestry dashboard
Support filtering by namespace in pods variables
Improve error handling of URLs without schemes

1.0.0-beta.3

Update Flows by protocol dashboard
GetFlows: Support multi-select dashboard variables
GetFlows: Make protocol filter a MultiSelect
GetFlows: Make verdict filters a MultiSelect
GetFlows: Make namespace filter a MultiSelect
Fix Timescape healthcheck - use v1 API
Refactor service map prometheus queries
Support cilium to v1.13.2-cee.1-gen-tag
Sign plugin as commercial instead of private

1.0.0-beta.2

Service Map: Reduce the volume of Prometheus queries
Service Map: Fix statistics calculations
Add License
Add two dashboards using the plugin
Service Map: Display only active connections
backend: Add missing json tags on some struct members
backend: Reduce logs level from Info to Debug
Reuse Timescape gRPC client
QueryEditor: Don't display Enterprise InfoBox if Timescape URL is configured
Process Ancestry: Support pods dropdown

1.0.0-beta.1

Service Map: Add support for dashboard variables
Service Map: Fix displaying statistics
Service Map: Fix visualization in dashboards
Service Map: Query only metrics with non-empty source and destination
Process Ancestry: Fix namespace formatting
backend: Return error when getting the client fails
backend: Add unit tests for GetFlowsQuery
Add screenshots to plugin.json
Change logo to the Isovalent one
docs: Write user-facing documentation in README.md
docs: Move developer instructions to DEVELOPMENT.md
docs: Document release process in RELEASE.md

0.0.5-alpha.1 (Unreleased)

Renamed the plugin from Cilium to Hubble, updating the logo on the way
Timescape backend is now optional
Migrate to create-plugin instead of toolkit
Add alpo-1 and alpo-2 to plugin signed root urls
Bump vendored Hubble Timescape to v1.0.0-beta16

0.0.4-alpha.1 (Unreleased)

Add variable support to GetFlows query
GetFlows: Tidy flow table display
GetFlows: Link trace-id to Tempo
Config: Allow clearing the prometheus and tempo datasources

0.0.3-alpha.1 (Unreleased)

Display traceID in Flows table
Get Flows: Filter by workload and trace-id on the backend
Support filtering by source/destination namespace in ServiceMap

0.0.2-alpha.1 (Unreleased)

Fetch timescape pods for process ancestory
Fix request duration and request failure rate queries when linking to Prometheus
Get Flows: Only show drop reason when flow is dropped
Get Flows: Show workload information
Get Flows: Add inputs for filtering by workload, trace-id and protocol
Update to Hubble Timescape beta15