This is documentation for the next version of Grafana Beyla documentation. For the latest stable release, go to the latest version.

Documentation Beyla Configure Service discovery

Configure Beyla service discovery

The BEYLA_AUTO_TARGET_EXE and BEYLA_OPEN_PORT are environment variables that make it easier to configure Beyla to instrument a single service or a group of related services.

In some scenarios, Beyla instruments many services. For example, as a Kubernetes DaemonSet that instruments all the services in a node. The discovery YAML section lets you specify more granular selection criteria for the services Beyla can instrument.

Discovery services

You can override the service name, namespace, and other configurations per service type.

Name

Defines a name for the matching instrumented service. Beyla uses it to populate the service.name OTEL property and the service_name Prometheus property in the exported metrics and traces.

This option is deprecated, as multiple matches for the same instrument entry mean multiple services share the same name. Refer to the override service name and namespace section to enable automatic configuration of service name and namespace from diverse metadata sources.

If you don’t set this property, Beyla uses the following properties, in order of precedence:

• If Kubernetes is enabled:
  1. The name of the Deployment that runs the instrumented process, if any
  2. The name of the ReplicaSet, DaemonSet, or StatefulSet that runs the instrumented process, if any
  3. The name of the Pod that runs the instrumented process
• If Kubernetes isn’t enabled:
  1. The name of the process executable file

If multiple processes match the service selection criteria, the metrics and traces for all the instances might share the same service name. For example, when multiple instrumented processes run under the same Deployment, or have the same executable name. In that case, the reported instance attribute lets you differentiate the different instances of the service.

Namespace

Defines a namespace for the matching instrumented service. If you don’t set this property, Beyla uses the Kubernetes namespace of the instrumented process, if available, or leaves it empty if Kubernetes isn’t available.

This option is deprecated. Refer to the overriding service name and namespace section to enable automatic configuration of service name and namespace from diverse metadata sources.

This namespace is not a selector for Kubernetes namespaces. Beyla uses its value to set the value of standard telemetry attributes. For example, the OpenTelemetry service.namespace attribute.

Open ports

Selects the process to instrument by the port it has open (listens to). This property accepts a comma-separated list of ports, for example 80, and port ranges, for example 8000-8999. If the executable matches only one of the ports in the list, Beyla considers it a match.

For example, specifying the following property:

YAML Copy

discovery:
  instrument:
    - open_ports: 80,443,8000-8999

Beyla selects any executable that opens port 80, 443, or any of the ports between 8000 and 8999 included.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

If an executable opens multiple ports, you only need to specify one of those ports for Beyla to instrument all the HTTP/S and GRPC requests on all application ports. Currently, you can’t restrict the instrumentation only to the methods exposed through a specific port.

Executable path

Selects the processes to instrument by their executable name path. This property accepts a glob to match against the full executable command line, including the directory where the executable resides on the file system.

Beyla tries to instrument all the processes with an executable path matching this property. For example, setting exe_path: * makes Beyla try to instrument all the executables in the host.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

Containers only

Selects processes to instrument which are running in an OCI container. To perform this check, Beyla inspects the process network namespace and matches it against its own network namespace. If Beyla doesn’t have enough permissions to perform the network namespace inspection, it ignores this option.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s namespace

This selector property limits the instrumentation to the applications running in the Kubernetes Namespaces with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s Pod name

This selector property limits the instrumentation to the applications running in the Kubernetes Pods with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s deployment name

This selector property limits the instrumentation to the applications running in the Kubernetes Deployments with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s replicaset name

This selector property limits the instrumentation to the applications running in the Kubernetes ReplicaSets with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s statefulset name

This selector property limits the instrumentation to the applications running in the Kubernetes StatefulSets with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s daemonset name

This selector property limits the instrumentation to the applications running in the Kubernetes DaemonSet with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s owner name

This selector property limits the instrumentation to the applications running in the Pods owned by a Deployment, ReplicaSet, DaemonSet, or StatefulSet with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s Pod labels

This selector property limits the instrumentation to the applications running in the Pods with labels matching the provided value as glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

For example:

YAML Copy

discovery:
  instrument:
    - k8s_namespace: frontend
      k8s_pod_labels:
        instrument: beyla

The preceding example discovers all Pods in the frontend namespace that have a label instrument with a value that matches the glob beyla.

K8s Pod annotations

This selector property limits the instrumentation to the applications running in the Pods with annotations matching the provided value as glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

For example:

YAML Copy

discovery:
  instrument:
    - k8s_namespace: backend
      k8s_pod_annotations:
        beyla.instrument: "true"

The preceding example discovers all Pods in the backend namespace that have an annotation beyla.instrument with a value that matches the glob true.

Exports

Controls which telemetry signals Beyla exports for the matching service. This property enables per-service feature enablement, allowing you to selectively enable specific observability capabilities for different services within a single Beyla instance.

The exports property accepts a list containing metrics, traces, or both. An empty list [] disables export for the service. If not specified, Beyla exports all configured telemetry signals.

For example:

YAML Copy

discovery:
  instrument:
    - k8s_deployment_name: "*"
      exports: [metrics]
    - k8s_deployment_name: backend
      exports: [traces]
    - k8s_deployment_name: worker
      exports: []

This example configures Beyla to export only metrics for all services. For the specific case of backend, only traces are enabled, and for worker, everything is disabled. The order of defined instrument selectors matters, as later entries can override earlier export rules.

For an export signal to function, you must configure the corresponding exporter in Beyla. For example, specifying traces in the exports list requires configuring the OTLP traces exporter via otel_traces_export. Specifying metrics requires configuring at least one metrics exporter, such as prometheus_export or otel_metrics_export. If you specify an export signal without configuring the corresponding exporter, Beyla ignores that signal.

Metrics export features

Additionally to configuring custom export modes per discovery instrumentation criteria, Beyla allows you to override the global metrics export features for each discovery criteria by adding metrics > features as a property to individual discovery > instrument entries.

For example:

YAML Copy

metrics:
  features: ['application_service_graph']
discovery:
  instrument:
    - open_ports: 3030,3040
      metrics:
        features:
          - 'application'
          - 'application_span'
          - 'application_service_graph'
    - name: pyserver
      open_ports: 7773
      metrics:
        features:
          - 'application'
    - name: apache
      open_ports: 8080
    - name: nginx
      open_ports: 8085
    - name: tomcat
      open_ports: 8090

This example configures Beyla to only export application service graph metrics by default, but it later overrides specific discovery criteria with different set of exported metrics. The outcome of the configuration is as follows:

• The apache, nginx, and tomcat service instances only export application_service_graph metrics (as defined in the top-level metrics > features configuration).

• The pyserver service only exports the application group of metrics.

• Services listening on ports 3030 or 3040 export the application, application_span, and application_service_graph metric groups.

Custom trace sampler

By using the sampler property for an instrument definition criteria, you can define individual trace sampling strategy for each instrument criteria. This option overrides the default specified trace sampling configuration if it is defined for all instrumented services. For more details on configuring sampling, refer to the sample traces documentation section.

For example:

YAML Copy

discovery:
  instrument:
    - k8s_deployment_name: backend
      sampler:
        name: "traceidratio"
        arg: "0.1"      
    - k8s_deployment_name: worker
otel_traces_export:
  sampler:
    name: "traceidratio"
    arg: "0.5"      

This example configures a global trace sampling configuration with the traceidratio sampling option of 50%, for all instrumented services. However, for the deployment name backend, this discovery criteria defines an override that uses the same traceidratio sampling option, but samples only 10% of the traces.

Custom route matching rules

By using the routes property for an instrument definition criteria, you can define individual incoming and outgoing route matching rules for each instrument criteria. This option appends to the default specified route matching patterns if it is defined for all instrumented services. For more details on configuring route matching patterns, refer to the routes decorator documentation section.

Unlike the default routes pattern matching configuration option, the instrument definition criteria routes option has separate section for defining route matching rules for incoming and outgoing requests. This allows for precise control of your HTTP route cardinality.

For example:

YAML Copy

discovery:
  instrument:
    - k8s_deployment_name: backend
      routes:
        incoming: ["/api/users/{user_id}", "/api/customers/{customer_id}"]
        outgoing: ["/*"]  
    - k8s_deployment_name: frontend
routes:
  patterns:
    - /user/{id}

In the example above, we have global route matching pattern definition for the HTTP route /user/{id}. However, the service backend has additional route matchers for incoming calls to /api/users/{user_id} and /api/customers/{customer_id}, and an additional outgoing request pattern for /*. If the backend service was making high-cardinality outgoing calls that we didn’t want to store in out metrics database, by specifying a catch all outgoing pattern of /*, we can selectively remove the problematic cardinality source without impacting all other services.

Survey mode

In survey mode, Beyla only performs service discovery and detects the programming language of each service, but doesn’t instrument any discovered services.

Beyla writes the discovered information from survey mode to a metric called survey_info, which uses the same attributes as the target_info metric. The Prometheus exporter creates this metric based on the OpenTelemetry metric resource attributes. You can use survey mode to build external automated instrumentation solutions. For example, you can use the survey_info metric to list available instrumentation targets and choose which ones to instrument.

Configure the survey section exactly like the instrument section. For more details, see the discovery services section of this document.

Exclude services from instrumentation

The exclude_instrument section lets you specify selection criteria for excluding services from being instrumented. It follows the same definition format as described in the discovery services section of this document.

This option helps you avoid instrumenting services typically found in observability environments. For example, use this option to exclude instrumenting Prometheus.

Default exclude services from instrumentation

The default_exclude_instrument section disables instrumentation of Beyla itself (self-instrumentation), as well as Grafana Alloy and the OpenTelemetry Collector. It also disables instrumentation of various Kubernetes system namespaces to reduce the overall cost of metric generation. The following section contains all excluded components:

• Excluded services by exe_path: *beyla, *alloy, *ebpf-instrument, *otelcol, *otelcol-contrib, *otelcol-contrib[!/]*.
• Excluded services by k8s_namespace: kube-system, kube-node-lease, local-path-storage, grafana-alloy, cert-manager, monitoring, gke-connect, gke-gmp-system, gke-managed-cim, gke-managed-filestorecsi, gke-managed-metrics-server, gke-managed-system, gke-system, gke-managed-volumepopulator, gatekeeper-system.

Change this option to allow Beyla to instrument itself or some of the other excluded components.

Note: to enable such self-instrumentation, you still need to include them in the instrument section, or these components need to be a part of a encompassing inclusion criteria.

Skip go specific tracers

The skip_go_specific_tracers option disables the detection of Go specifics when the ebpf tracer inspects executables to be instrumented. The tracer falls back to using generic instrumentation, which is generally less efficient. This option should only be used for Go services built with Go versions older than 1.17.

Exclude otel instrumented services

The exclude_otel_instrumented_services option disables Beyla instrumentation of services already instrumented with OpenTelemetry. Since Beyla is often deployed to monitor all services in a Kubernetes cluster, monitoring already instrumented services can lead to duplicate telemetry data, unless you carefully craft the instrumentation selection (or exclusion) criteria. To avoid unnecessary configuration overhead, Beyla monitors for OpenTelemetry SDK calls to publish metrics and traces, and automatically turns off instrumentation of services that publish their own telemetry data. Turn this option off if your application-generated telemetry data doesn’t conflict with the Beyla generated metrics and traces.

Override service name and namespace

If you export instrumentation data via OpenTelemetry or Prometheus, Beyla follows the service name conventions from the OpenTelemetry operator to improve interoperability with other instrumentation solutions.

Beyla uses the following criteria in this order to automatically set the service name and namespace:

1. Resource attributes set via OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME environment variables of the instrumented process or container.
2. In Kubernetes, resource attributes set via the following Pod annotations:
   • resource.opentelemetry.io/service.name
   • resource.opentelemetry.io/service.namespace
3. In Kubernetes, resource attributes set via the following Pod labels:
   • app.kubernetes.io/name sets the service name
   • app.kubernetes.io/part-of sets the service namespace
4. In Kubernetes, resource attributes calculated from the Pod owner’s metadata, in the following order (according to their availability):
   • k8s.deployment.name
   • k8s.replicaset.name
   • k8s.statefulset.name
   • k8s.daemonset.name
   • k8s.cronjob.name
   • k8s.job.name
   • k8s.pod.name
   • k8s.container.name
5. The executable name of the instrumented process.

You can override the Kubernetes labels from the previous bullet 3 via configuration.

In YAML:

YAML Copy

attributes:
  kubernetes:
    resource_labels:
      service.name:
        # gets service name from the first existing Pod label
        - override-svc-name
        - app.kubernetes.io/name
      service.namespace:
        # gets service namespace from the first existing Pod label
        - override-svc-ns
        - app.kubernetes.io/part-of

They accept a comma-separated list of annotation and label names.

Minimum process age

The min_process_age (environment variable BEYLA_MIN_PROCESS_AGE) option sets a requirement for a process to be alive for at least certain amount of time before it is considered for instrumentation. The default value is "5s" (five seconds). This option is a performance optimization related to cost of processing discovered binaries based on the chosen discovery criteria. It avoids instrumenting periodic short lived processes.

Route harvesting

Since Beyla instruments at the protocol level, for HTTP requests we see the actual URL path, while the OpenTelemetry specification requires that we provide a low-cardinality URL route. Beyla has purpose built route detector, which uses heuristics and cardinality reduction logic to automatically determine the low-cardinality route from the protocol provided URL path (for more information on this refer to Routes Decorator). However, for certain programming languages, Beyla can process the application symbols and extract the actual routes set in the application.

Currently the route harvesting is supported for Java, Go and NodeJS.

The route harvesting for Java applications works by communicating with the JVM at runtime. Java application typically load after a bit of time, which may result in incomplete route information, if Beyla harvests the Java application routes immediately as it instruments the process. Therefore, Beyla performs Java route harvesting on Java applications which have been running for at least 60 seconds. This value can be modified by setting the environment variable BEYLA_JAVA_ROUTE_HARVEST_DELAY or by setting the configuration file option:

Copy

discovery:
  route_harvester_advanced:
    java_harvest_delay: 30s