This is documentation for the next version of Grafana Beyla documentation. For the latest stable release, go to the latest version.

Documentation Beyla Configure Export data

Configure Beyla Prometheus and OpenTelemetry data export

Beyla can export OpenTelemetry metrics and traces to a OTLP endpoint.

To send metrics directly to the Grafana Cloud OpenTelemetry endpoint, see the Grafana Cloud OTLP endpoint configuration.

Grafana Cloud OTLP endpoint

YAML section: grafana.otlp

You can configure the component under the grafana.otlp section of your YAML configuration or via environment variables. You can configure Beyla to submit OpenTelemetry data to the Grafana Cloud OTEL endpoint using custom variables, allowing an easier setup of the endpoint and the authentication.

For more information on the Grafana Cloud OTLP endpoint, refer to the Send data to the Grafana Cloud OTLP endpoint documentation.

For example:

YAML Copy

grafana:
  otlp:
    cloud_submit: ["metrics", "traces"]
    cloud_zone: "eu-west-23"
    cloud_instance_id: "12345"
    cloud_api_key: "abcde"

You can also use the standard OpenTelemetry variables to submit the metrics and traces to any standard OpenTelemetry endpoint, including Grafana Cloud.

Cloud zone configuration

If any of the OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT variables are defined, they override the destination endpoint, and Beyla ignores the cloud_zone configuration option.

OpenTelemetry metrics exporter component

YAML section: otel_metrics_export

Enable the OpenTelemetry metrics export component by setting the endpoint attribute in your configuration file or via an environment variable, refer to metric export configuration options.

Configure the component under the otel_metrics_export section of your YAML configuration or via environment variables.

In addition to the configuration documented in this article, the component supports environment variables from the standard OpenTelemetry exporter configuration.

For example:

YAML Copy

otel_metrics_export:
  ttl: 5m
  endpoint: http://otelcol:4318
  protocol: grpc
  features: ["network", "network_inter_zone"]
  buckets:
    duration_histogram: [0, 1, 2]
  histogram_aggregation: base2_exponential_bucket_histogram

Metrics export protocol

If you don’t set a protocol Beyla sets the protocol as follows:

• grpc: if the port ends in 4317, for example 4317, 14317, or 24317.
• http/protobuf: if the port ends in 4318, for example 4318, 14318, or 24318.

Metrics export features

The Beyla metrics exporter can export the following metrics data groups for processes matching entries in the metrics discovery configuration.

• application: Application-level metrics
• application_span: Application-level trace span metrics using legacy naming conventions. Deprecated: use application_span_otel instead. The name keeps working, but Beyla logs a deprecation warning at startup and the feature is removed in a future release
• application_span_otel: Application-level trace span metrics using OpenTelemetry naming conventions. Refer to span metrics formats for differences between application_span and application_span_otel
• application_span_sizes: Application-level request and response body size metrics for trace spans. Deprecated: there is no direct replacement. The closest equivalents are the http.server.request.body.size and http.server.response.body.size metrics
• application_host: Application-level host metrics for host based pricing
• application_service_graph: Application-level service graph metrics. It’s recommended to use a DNS for service discovery and to ensure the DNS names match the OpenTelemetry service names Beyla uses. In Kubernetes environments, the OpenTelemetry service name set by the service name discovery is the best choice for service graph metrics.
• application_process: Low-level process metrics (for example, CPU, memory, disk metrics) for the selected services
• application_runtime: Go and JVM language runtime metrics. Refer to Application runtime metrics for the full list of exported metrics.
• network: Network-level metrics, refer to the network metrics configuration documentation to learn more
• network_inter_zone: Network inter-zone metrics, refer to the network metrics configuration documentation to learn more

Application runtime metrics

When application_runtime is enabled, Beyla collects language runtime metrics for instrumented Go and Java processes.

Go runtime metrics

JVM runtime metrics

JVM runtime metrics are collected via eBPF USDT probes on libjvm.so (hotspot:mem__pool__gc__begin and hotspot:mem__pool__gc__end) without requiring any bytecode agent.

All JVM metrics carry the attributes jvm.memory.type (for example, heap, non_heap) and jvm.memory.pool.name (for example, G1 Eden Space).

To control how frequently JVM GC events are sampled, refer to JVM runtime metrics.

Span metrics formats

Beyla provides two formats for span metrics:

• application_span: Legacy format using the metric names traces_spanmetrics_latency and traces_spanmetrics_calls_total. Deprecated: use application_span_otel instead
• application_span_otel: OpenTelemetry-compliant format using the metric names traces_span_metrics_duration_seconds and traces_span_metrics_calls_total

You can only enable one span metrics format at a time. If you specify both application_span and application_span_otel in the features list, Beyla returns a configuration error. The same applies when one format comes from the top-level metrics section and the other from a per-service discovery metrics.features section, because the exported metric names are selected from the combination of both.

If you select the formats implicitly with all or *, Beyla resolves the conflict automatically in favor of application_span_otel and logs a warning. This resolution also applies to per-service metrics.features sections, so a single service requesting all doesn’t move every service back to the legacy names.

Metrics instrumentation

The list of instrumentation areas Beyla can collection data from:

• *: all instrumentation, if * is present Beyla ignores other values
• aerospike: Aerospike database metrics
• amqp: AMQP message queue metrics
• couchbase: Couchbase database metrics
• dns: DNS request metrics
• genai: Generative AI metrics
• gpu: GPU operation metrics
• http: HTTP/HTTPS/HTTP2 application metrics
• grpc: gRPC application metrics
• kafka: Kafka client/server message queue metrics
• memcached: Memcached database metrics
• mongo: MongoDB client/server database metrics
• mqtt: MQTT message queue metrics
• nats: NATS message queue metrics
• redis: Redis client/server database metrics
• sql: SQL database client call metrics
• sunrpc: SunRPC call metrics

For example, setting the instrumentations option to: http,grpc enables the collection of HTTP/HTTPS/HTTP2 and gRPC application metrics, and disables other instrumentation.

OpenTelemetry traces exporter component

YAML section: otel_traces_export

You can configure the component under the otel_traces_export section of your YAML configuration or via environment variables.

In addition to the configuration documented in this article, the component supports the environment variables from the standard OpenTelemetry exporter configuration.

YAML Copy

otel_traces_export:
  endpoint: http://jaeger:4317
  protocol: grpc
  instrumentations: ["http", "sql"]

Traces export protocol

If you don’t set a protocol Beyla sets the protocol as follows:

• grpc: if the port ends in 4317, for example 4317, 14317, or 24317.
• http/protobuf: if the port ends in 4318, for example 4318, 14318, or 24318.

Traces instrumentation

The list of instrumentation areas Beyla can collection data from:

• *: all instrumentation, if * is present Beyla ignores other values
• aerospike: Aerospike database traces
• amqp: AMQP message queue traces
• couchbase: Couchbase database traces
• dns: DNS request traces
• genai: Generative AI traces
• gpu: GPU operation traces
• http: HTTP/HTTPS/HTTP2 application traces
• grpc: gRPC application traces
• kafka: Kafka client/server message queue traces
• memcached: Memcached database traces
• mongo: MongoDB client/server database traces
• mqtt: MQTT message queue traces
• nats: NATS message queue traces
• redis: Redis client/server database traces
• sql: SQL database client call traces
• sunrpc: SunRPC call traces

For example, setting the instrumentations option to: http,grpc enables the collection of HTTP/HTTPS/HTTP2 and gRPC application traces, and disables other instrumentation.

Note: By default, Beyla enables all supported trace instrumentations except dns, genai, and gpu. You can explicitly enable these instrumentations if needed.

Prometheus exporter component

YAML section: prometheus_export

You can configure the component under the prometheus_export section of your YAML configuration or via environment variables. This component opens an HTTP endpoint in the auto-instrumentation tool that allows any external scraper to pull metrics in Prometheus format. It is enabled if the port property is set.

YAML Copy

prometheus_export:
  port: 8999
  path: /metrics
  extra_resource_attributes: ["deployment_environment"]
  ttl: 1s
  buckets:
    request_size_histogram: [0, 10, 20, 22]
    response_size_histogram: [0, 10, 20, 22]
  features:
    - application
    - network
    - application_process
    - application_span
    - application_service_graph
  instrumentations: ["http, "sql"]

Prometheus extra resource attributes

Due to internal limitations of the Prometheus API client, Beyla needs to know beforehand which attributes are exposed for each metric. This would cause that some attributes that are discovered at runtime, during instrumentation, won’t be visible by default. For example, attributes defined on each application via Kubernetes annotations, or in the target application’s OTEL_RESOURCE_ATTRIBUTES environment variable.

For example, an application defining the OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production as environment variable, the target_info{deployment.environment="production"} attribute would be visible by default if the metrics are exported via OpenTelemetry but not if they are exported via Prometheus.

To make deployment_environment visible in Prometheus, you need to add it to the extra_resource_attributes list.

Prometheus export features

The Prometheus metrics exporter can export the following metrics data groups:

• application: Application-level metrics
• application_span: Application-level trace span metrics
• application_host Application-level host metrics for host based pricing
• application_service_graph: Application-level service graph metrics. It’s recommended to use a DNS for service discovery and to ensure the DNS names match the OpenTelemetry service names Beyla uses. In Kubernetes environments, the OpenTelemetry service name set by the service name discovery is the best choice for service graph metrics.
• application_process: Low-level process metrics (for example, CPU, memory, disk metrics) for the selected services
• application_runtime: Go and JVM language runtime metrics. Refer to Application runtime metrics for the full list of exported metrics.
• network: Network-level metrics, refer to the network metrics configuration documentation to learn more
• network_inter_zone: Network inter-zone metrics, refer to the network metrics configuration documentation to learn more

Prometheus instrumentation

The list of instrumentation areas Beyla can collection data from:

• *: all instrumentation, if * is present Beyla ignores other values
• aerospike: Aerospike database metrics
• amqp: AMQP message queue metrics
• couchbase: Couchbase database metrics
• dns: DNS request metrics
• genai: Generative AI metrics
• gpu: GPU operation metrics
• http: HTTP/HTTPS/HTTP2 application metrics
• grpc: gRPC application metrics
• kafka: Kafka client/server message queue metrics
• memcached: Memcached database metrics
• mongo: MongoDB client/server database metrics
• mqtt: MQTT message queue metrics
• nats: NATS message queue metrics
• redis: Redis client/server database metrics
• sql: SQL database client call metrics
• sunrpc: SunRPC call metrics

For example, setting the instrumentations option to: http,grpc enables the collection of HTTP/HTTPS/HTTP2 and gRPC application metrics, and disables other instrumentation.

Global metrics features

You can set the default features list for all exporters at once using the global metrics configuration block, rather than repeating it under each exporter. The global setting is overridden by any per-exporter features value.

The metrics.features option (environment variable: OTEL_EBPF_METRICS_FEATURES) accepts a list of feature group names. The default is ["application"].

To enable runtime metrics globally for all processes:

YAML Copy

metrics:
  features:
    - application
    - application_runtime

To enable application_runtime only for a specific process, use the per-service metrics.features override in service discovery:

YAML Copy

discovery:
  instrument:
    - name: my-java-service
      metrics:
        features:
          - application_runtime

JVM runtime metrics

When the application_runtime feature is enabled for Java services, Beyla collects JVM memory pool metrics by attaching eBPF USDT probes to hotspot:mem__pool__gc__begin and hotspot:mem__pool__gc__end in libjvm.so.

To prevent excessive data collection on JVMs with high GC frequency, Beyla throttles the collection with a configurable sampling interval.

Example:

YAML Copy

jvm_runtime_metrics:
  sampling_interval: 1s