Menu
Open source

otelcol.processor.k8sattributes

otelcol.processor.k8sattributes accepts telemetry data from other otelcol components and adds Kubernetes metadata to the resource attributes of spans, logs, or metrics.

Note

otelcol.processor.k8sattributes is a wrapper over the upstream OpenTelemetry Collector k8sattributes processor. If necessary, bug reports or feature requests will be redirected to the upstream repository.

You can specify multiple otelcol.processor.k8sattributes components by giving them different labels.

Usage

river
otelcol.processor.k8sattributes "LABEL" {
  output {
    metrics = [...]
    logs    = [...]
    traces  = [...]
  }
}

Arguments

The following arguments are supported:

NameTypeDescriptionDefaultRequired
auth_typestringAuthentication method when connecting to the Kubernetes API.serviceAccountno
passthroughboolPassthrough signals as-is, only adding a k8s.pod.ip resource attribute.falseno

The supported values for auth_type are:

  • none: No authentication is required.
  • serviceAccount: Use the built-in service account that Kubernetes automatically provisions for each pod.
  • kubeConfig: Use local credentials like those used by kubectl.
  • tls: Use client TLS authentication.

Setting passthrough to true enables the “passthrough mode” of otelcol.processor.k8sattributes:

  • Only a k8s.pod.ip resource attribute will be added.
  • No other metadata will be added.
  • The Kubernetes API will not be accessed.
  • To correctly detect the pod IPs, Grafana Agent must receive spans directly from services.
  • The passthrough setting is useful when configuring the Agent as a Kubernetes Deployment. A Grafana Agent running as a Deployment cannot detect the IP addresses of pods generating telemetry data without any of the well-known IP attributes. If the Deployment Grafana Agent receives telemetry from Grafana Agents deployed as DaemonSet, then some of those attributes might be missing. As a workaround, you can configure the DaemonSet Grafana Agents with passthrough set to true.

Blocks

The following blocks are supported inside the definition of otelcol.processor.k8sattributes:

HierarchyBlockDescriptionRequired
outputoutputConfigures where to send received telemetry data.yes
extractextractRules for extracting data from Kubernetes.no
extract > annotationannotationCreating resource attributes from Kubernetes annotations.no
extract > labelextract_labelCreating resource attributes from Kubernetes labels.no
filterfilterFilters the data loaded from Kubernetes.no
filter > fieldfieldFilter pods by generic Kubernetes fields.no
filter > labelfilter_labelFilter pods by Kubernetes labels.no
pod_associationpod_associationRules to associate pod metadata with telemetry signals.no
pod_association > sourcesourceSource information to identify a pod.no
excludeexcludeExclude pods from being processed.no
exclude > podpodPod information.no

The > symbol indicates deeper levels of nesting. For example, extract > annotation refers to an annotation block defined inside an extract block.

extract block

The extract block configures which metadata, annotations, and labels to extract from the pod.

The following attributes are supported:

NameTypeDescriptionDefaultRequired
metadatalist(string)Pre-configured metadata keys to add.See belowno

The currently supported metadata keys are:

  • k8s.pod.name
  • k8s.pod.uid
  • k8s.deployment.name
  • k8s.node.name
  • k8s.namespace.name
  • k8s.pod.start_time
  • k8s.replicaset.name
  • k8s.replicaset.uid
  • k8s.daemonset.name
  • k8s.daemonset.uid
  • k8s.job.name
  • k8s.job.uid
  • k8s.cronjob.name
  • k8s.statefulset.name
  • k8s.statefulset.uid
  • k8s.container.name
  • container.image.name
  • container.image.tag
  • container.id

By default, if metadata is not specified, the following fields are extracted and added to spans, metrics, and logs as resource attributes:

  • k8s.pod.name
  • k8s.pod.uid
  • k8s.pod.start_time
  • k8s.namespace.name
  • k8s.node.name
  • k8s.deployment.name (if the pod is controlled by a deployment)
  • k8s.container.name (requires an additional attribute to be set: container.id)
  • container.image.name (requires one of the following additional attributes to be set: container.id or k8s.container.name)
  • container.image.tag (requires one of the following additional attributes to be set: container.id or k8s.container.name)

annotation block

The annotation block configures how to extract Kubernetes annotations.

The following attributes are supported:

NameTypeDescriptionDefaultRequired
fromstringThe source of the labels or annotations. Allowed values are pod and namespace.podno
key_regexstringA regular expression used to extract a key that matches the regular expression.""no
keystringThe annotation or label name. This key must exactly match an annotation or label name.""no
regexstringAn optional field used to extract a sub-string from a complex field value.""no
tag_namestringThe name of the resource attribute added to logs, metrics, or spans.""no

When you don’t specify the tag_name, a default tag name is used with the format:

  • k8s.pod.annotations.<annotation key>
  • k8s.pod.labels.<label key>

For example, if tag_name isn’t specified and the key is git_sha, the attribute name will be k8s.pod.annotations.git_sha.

You can set either the key attribute or the key_regex attribute, but not both. When key_regex is present, tag_name supports back reference to both named capturing and positioned capturing.

For example, assume your pod spec contains the following labels:

  • app.kubernetes.io/component: mysql
  • app.kubernetes.io/version: 5.7.21

If you’d like to add tags for all labels with the prefix app.kubernetes.io/ and trim the prefix, then you can specify the following extraction rules:

river
extract {
	label {
	    from = "pod"
		key_regex = "kubernetes.io/(.*)"
		tag_name  = "$1"
	}
}

These rules add the component and version tags to the spans or metrics.

You can set the from attribute to either "pod" or "namespace".

label block

The label block configures how to extract Kubernetes labels.

The following attributes are supported:

NameTypeDescriptionDefaultRequired
fromstringThe source of the labels or annotations. Allowed values are pod and namespace.podno
key_regexstringA regular expression used to extract a key that matches the regular expression.""no
keystringThe annotation or label name. This key must exactly match an annotation or label name.""no
regexstringAn optional field used to extract a sub-string from a complex field value.""no
tag_namestringThe name of the resource attribute added to logs, metrics, or spans.""no

When you don’t specify the tag_name, a default tag name is used with the format:

  • k8s.pod.annotations.<annotation key>
  • k8s.pod.labels.<label key>

For example, if tag_name isn’t specified and the key is git_sha, the attribute name will be k8s.pod.annotations.git_sha.

You can set either the key attribute or the key_regex attribute, but not both. When key_regex is present, tag_name supports back reference to both named capturing and positioned capturing.

For example, assume your pod spec contains the following labels:

  • app.kubernetes.io/component: mysql
  • app.kubernetes.io/version: 5.7.21

If you’d like to add tags for all labels with the prefix app.kubernetes.io/ and trim the prefix, then you can specify the following extraction rules:

river
extract {
	label {
	    from = "pod"
		key_regex = "kubernetes.io/(.*)"
		tag_name  = "$1"
	}
}

These rules add the component and version tags to the spans or metrics.

You can set the from attribute to either "pod" or "namespace".

filter block

The filter block configures which nodes to get data from and which fields and labels to fetch.

The following attributes are supported:

NameTypeDescriptionDefaultRequired
nodestringConfigures a Kubernetes node name or host name.""no
namespacestringFilters all pods by the provided namespace. All other pods are ignored.""no

If node is specified, then any pods not running on the specified node will be ignored by otelcol.processor.k8sattributes.

field block

The field block allows you to filter pods by generic Kubernetes fields.

The following attributes are supported:

NameTypeDescriptionDefaultRequired
keystringThe key or name of the field or labels that a filter can use.yes
valuestringThe value associated with the key that a filter can use.yes
opstringThe filter operation to apply on the given key: value pair.equalsno

For op, the following values are allowed:

  • equals: The field value must equal the provided value.
  • not-equals: The field value must not be equal to the provided value.
  • exists: The field value must exist. Only applicable to annotation fields.
  • does-not-exist: The field value must not exist. Only applicable to annotation fields.

label block

The label block allows you to filter pods by generic Kubernetes labels.

The following attributes are supported:

NameTypeDescriptionDefaultRequired
keystringThe key or name of the field or labels that a filter can use.yes
valuestringThe value associated with the key that a filter can use.yes
opstringThe filter operation to apply on the given key: value pair.equalsno

For op, the following values are allowed:

  • equals: The field value must equal the provided value.
  • not-equals: The field value must not be equal to the provided value.
  • exists: The field value must exist. Only applicable to annotation fields.
  • does-not-exist: The field value must not exist. Only applicable to annotation fields.

pod_association block

The pod_association block configures rules on how to associate logs/traces/metrics to pods.

The pod_association block does not support any arguments and is configured fully through child blocks.

The pod_association block can be repeated multiple times, to configure additional rules.

Example:

river
pod_association {
    source {
        from = "resource_attribute"
        name = "k8s.pod.ip"
    }
}

pod_association {
    source {
        from = "resource_attribute"
        name = "k8s.pod.uid"
    }
    source {
        from = "connection"
    }
}

source block

The source block configures a pod association rule. This is used by the k8sattributes processor to determine the pod associated with a telemetry signal.

When multiple source blocks are specified inside a pod_association block, both source blocks has to match for the pod to be associated with the telemetry signal.

The following attributes are supported:

NameTypeDescriptionDefaultRequired
fromstringThe association method. Currently supports resource_attribute and connectionyes
namestringName represents extracted key name. For example, ip, pod_uid, k8s.pod.ipno

exclude block

The exclude block configures which pods to exclude from the processor.

pod block

The pod block configures a pod to be excluded from the processor.

The following attributes are supported:

NameTypeDescriptionDefaultRequired
namestringThe name of the podyes

output block

The output block configures a set of components to forward resulting telemetry data to.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
logslist(otelcol.Consumer)List of consumers to send logs to.[]no
metricslist(otelcol.Consumer)List of consumers to send metrics to.[]no
traceslist(otelcol.Consumer)List of consumers to send traces to.[]no

You must specify the output block, but all its arguments are optional. By default, telemetry data is dropped. Configure the metrics, logs, and traces arguments accordingly to send telemetry data to other components.

Exported fields

The following fields are exported and can be referenced by other components:

NameTypeDescription
inputotelcol.ConsumerA value that other components can use to send telemetry data to.

input accepts otelcol.Consumer data for any telemetry signal (metrics, logs, or traces).

Component health

otelcol.processor.k8sattributes is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.processor.k8sattributes does not expose any component-specific debug information.

Examples

Basic usage

In most cases, this is enough to get started. It’ll add these resource attributes to all logs, metrics, and traces:

  • k8s.namespace.name
  • k8s.pod.name
  • k8s.pod.uid
  • k8s.pod.start_time
  • k8s.deployment.name
  • k8s.node.name

Example:

river
otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    metrics = [otelcol.processor.k8sattributes.default.input]
    logs    = [otelcol.processor.k8sattributes.default.input]
    traces  = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  output {
    metrics = [otelcol.exporter.otlp.default.input]
    logs    = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = env("OTLP_ENDPOINT")
  }
}

Add additional metadata and labels

river
otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    metrics = [otelcol.processor.k8sattributes.default.input]
    logs    = [otelcol.processor.k8sattributes.default.input]
    traces  = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  extract {
    label {
      from      = "pod"
      key_regex = "(.*)/(.*)"
      tag_name  = "$1.$2"
    }

    metadata = [
      "k8s.namespace.name",
      "k8s.deployment.name",
      "k8s.statefulset.name",
      "k8s.daemonset.name",
      "k8s.cronjob.name",
      "k8s.job.name",
      "k8s.node.name",
      "k8s.pod.name",
      "k8s.pod.uid",
      "k8s.pod.start_time",
    ]
  }

  output {
    metrics = [otelcol.exporter.otlp.default.input]
    logs    = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = env("OTLP_ENDPOINT")
  }
}

Adding Kubernetes metadata to Prometheus metrics

otelcol.processor.k8sattributes adds metadata to metrics signals in the form of resource attributes. To display the metadata as labels of Prometheus metrics, the OTLP attributes must be converted from resource attributes to datapoint attributes. One way to do this is by using an otelcol.processor.transform component.

river
otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    metrics = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  extract {
    label {
      from = "pod"
    }

    metadata = [
      "k8s.namespace.name",
      "k8s.pod.name",
    ]
  }

  output {
    metrics = [otelcol.processor.transform.add_kube_attrs.input]
  }
}

otelcol.processor.transform "add_kube_attrs" {
  error_mode = "ignore"

  metric_statements {
    context = "datapoint"
    statements = [
      "set(attributes[\"k8s.pod.name\"], resource.attributes[\"k8s.pod.name\"])",
      "set(attributes[\"k8s.namespace.name\"], resource.attributes[\"k8s.namespace.name\"])",
    ]
  }

  output {
    metrics = [otelcol.exporter.prometheus.default.input]
  }
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
  }
}

Compatible components

otelcol.processor.k8sattributes can accept arguments from the following components:

otelcol.processor.k8sattributes has exports that can be consumed by the following components:

Note

Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.