This is documentation for the next version of Alloy. For the latest stable release, go to the latest version.
otelcol.processor.k8sattributes
otelcol.processor.k8sattributes
accepts telemetry data from other otelcol
components and adds Kubernetes metadata to the resource attributes of spans, logs, or metrics.
Note
otelcol.processor.k8sattributes
is a wrapper over the upstream OpenTelemetry Collectork8sattributes
processor. If necessary, bug reports or feature requests will be redirected to the upstream repository.
You can specify multiple otelcol.processor.k8sattributes
components by giving them different labels.
Usage
otelcol.processor.k8sattributes "LABEL" {
output {
metrics = [...]
logs = [...]
traces = [...]
}
}
Arguments
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
auth_type | string | Authentication method when connecting to the Kubernetes API. | serviceAccount | no |
passthrough | bool | Passthrough signals as-is, only adding a k8s.pod.ip resource attribute. | false | no |
The supported values for auth_type
are:
none
: No authentication is required.serviceAccount
: Use the built-in service account that Kubernetes automatically provisions for each pod.kubeConfig
: Use local credentials like those used by kubectl.tls
: Use client TLS authentication.
Setting passthrough
to true
enables the “passthrough mode” of otelcol.processor.k8sattributes
:
- Only a
k8s.pod.ip
resource attribute will be added. - No other metadata will be added.
- The Kubernetes API will not be accessed.
- To correctly detect the pod IPs, Alloy must receive spans directly from services.
- The
passthrough
setting is useful when configuring Alloy as a Kubernetes Deployment. A Alloy running as a Deployment cannot detect the IP addresses of pods generating telemetry data without any of the well-known IP attributes. If the Deployment Alloy receives telemetry from Alloys deployed as DaemonSet, then some of those attributes might be missing. As a workaround, you can configure the DaemonSet Alloys withpassthrough
set totrue
.
Blocks
The following blocks are supported inside the definition of
otelcol.processor.k8sattributes
:
Hierarchy | Block | Description | Required |
---|---|---|---|
output | output | Configures where to send received telemetry data. | yes |
extract | extract | Rules for extracting data from Kubernetes. | no |
extract > annotation | annotation | Creating resource attributes from Kubernetes annotations. | no |
extract > label | extract_label | Creating resource attributes from Kubernetes labels. | no |
filter | filter | Filters the data loaded from Kubernetes. | no |
filter > field | field | Filter pods by generic Kubernetes fields. | no |
filter > label | filter_label | Filter pods by Kubernetes labels. | no |
pod_association | pod_association | Rules to associate pod metadata with telemetry signals. | no |
pod_association > source | source | Source information to identify a pod. | no |
exclude | exclude | Exclude pods from being processed. | no |
exclude > pod | pod | Pod information. | no |
debug_metrics | debug_metrics | Configures the metrics that this component generates to monitor its state. | no |
The >
symbol indicates deeper levels of nesting. For example, extract > annotation
refers to an annotation
block defined inside an extract
block.
extract block
The extract
block configures which metadata, annotations, and labels to extract from the pod.
The following attributes are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
metadata | list(string) | Pre-configured metadata keys to add. | See below | no |
The currently supported metadata
keys are:
k8s.pod.name
k8s.pod.uid
k8s.deployment.name
k8s.node.name
k8s.namespace.name
k8s.pod.start_time
k8s.replicaset.name
k8s.replicaset.uid
k8s.daemonset.name
k8s.daemonset.uid
k8s.job.name
k8s.job.uid
k8s.cronjob.name
k8s.statefulset.name
k8s.statefulset.uid
k8s.container.name
container.image.name
container.image.tag
container.id
By default, if metadata
is not specified, the following fields are extracted and added to spans, metrics, and logs as resource attributes:
k8s.pod.name
k8s.pod.uid
k8s.pod.start_time
k8s.namespace.name
k8s.node.name
k8s.deployment.name
(if the pod is controlled by a deployment)k8s.container.name
(requires an additional attribute to be set:container.id
)container.image.name
(requires one of the following additional attributes to be set:container.id
ork8s.container.name
)container.image.tag
(requires one of the following additional attributes to be set:container.id
ork8s.container.name
)
annotation block
The annotation
block configures how to extract Kubernetes annotations.
The following attributes are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
from | string | The source of the labels or annotations. Allowed values are pod , namespace , and node . | pod | no |
key_regex | string | A regular expression used to extract a key that matches the regular expression. | "" | no |
key | string | The annotation or label name. This key must exactly match an annotation or label name. | "" | no |
regex | string | An optional field used to extract a sub-string from a complex field value. | "" | no |
tag_name | string | The name of the resource attribute added to logs, metrics, or spans. | "" | no |
When you don’t specify the tag_name
, a default tag name is used with the format:
k8s.pod.annotations.<annotation key>
k8s.pod.labels.<label key>
For example, if tag_name
isn’t specified and the key is git_sha
, the attribute name will be k8s.pod.annotations.git_sha
.
You can set either the key
attribute or the key_regex
attribute, but not both.
When key_regex
is present, tag_name
supports back reference to both named capturing and positioned capturing.
For example, assume your pod spec contains the following labels:
app.kubernetes.io/component: mysql
app.kubernetes.io/version: 5.7.21
If you’d like to add tags for all labels with the prefix app.kubernetes.io/
and trim the prefix, then you can specify the following extraction rules:
extract {
label {
from = "pod"
key_regex = "kubernetes.io/(.*)"
tag_name = "$1"
}
}
These rules add the component
and version
tags to the spans or metrics.
You can set the from
attribute to either "pod"
or "namespace"
.
label block
The label
block configures how to extract Kubernetes labels.
The following attributes are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
from | string | The source of the labels or annotations. Allowed values are pod , namespace , and node . | pod | no |
key_regex | string | A regular expression used to extract a key that matches the regular expression. | "" | no |
key | string | The annotation or label name. This key must exactly match an annotation or label name. | "" | no |
regex | string | An optional field used to extract a sub-string from a complex field value. | "" | no |
tag_name | string | The name of the resource attribute added to logs, metrics, or spans. | "" | no |
When you don’t specify the tag_name
, a default tag name is used with the format:
k8s.pod.annotations.<annotation key>
k8s.pod.labels.<label key>
For example, if tag_name
isn’t specified and the key is git_sha
, the attribute name will be k8s.pod.annotations.git_sha
.
You can set either the key
attribute or the key_regex
attribute, but not both.
When key_regex
is present, tag_name
supports back reference to both named capturing and positioned capturing.
For example, assume your pod spec contains the following labels:
app.kubernetes.io/component: mysql
app.kubernetes.io/version: 5.7.21
If you’d like to add tags for all labels with the prefix app.kubernetes.io/
and trim the prefix, then you can specify the following extraction rules:
extract {
label {
from = "pod"
key_regex = "kubernetes.io/(.*)"
tag_name = "$1"
}
}
These rules add the component
and version
tags to the spans or metrics.
You can set the from
attribute to either "pod"
or "namespace"
.
filter block
The filter
block configures which nodes to get data from and which fields and labels to fetch.
The following attributes are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
node | string | Configures a Kubernetes node name or host name. | "" | no |
namespace | string | Filters all pods by the provided namespace. All other pods are ignored. | "" | no |
If node
is specified, then any pods not running on the specified node will be ignored by otelcol.processor.k8sattributes
.
field block
The field
block allows you to filter pods by generic Kubernetes fields.
The following attributes are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
key | string | The key or name of the field or labels that a filter can use. | yes | |
value | string | The value associated with the key that a filter can use. | yes | |
op | string | The filter operation to apply on the given key: value pair. | equals | no |
For op
, the following values are allowed:
equals
: The field value must equal the provided value.not-equals
: The field value must not be equal to the provided value.exists
: The field value must exist. Only applicable toannotation
fields.does-not-exist
: The field value must not exist. Only applicable toannotation
fields.
label block
The label
block allows you to filter pods by generic Kubernetes labels.
The following attributes are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
key | string | The key or name of the field or labels that a filter can use. | yes | |
value | string | The value associated with the key that a filter can use. | yes | |
op | string | The filter operation to apply on the given key: value pair. | equals | no |
For op
, the following values are allowed:
equals
: The field value must equal the provided value.not-equals
: The field value must not be equal to the provided value.exists
: The field value must exist. Only applicable toannotation
fields.does-not-exist
: The field value must not exist. Only applicable toannotation
fields.
pod_association block
The pod_association
block configures rules on how to associate logs/traces/metrics to pods.
The pod_association
block does not support any arguments and is configured
fully through child blocks.
The pod_association
block can be repeated multiple times, to configure additional rules.
Example:
pod_association {
source {
from = "resource_attribute"
name = "k8s.pod.ip"
}
}
pod_association {
source {
from = "resource_attribute"
name = "k8s.pod.uid"
}
source {
from = "connection"
}
}
source block
The source
block configures a pod association rule. This is used by the k8sattributes
processor to determine the
pod associated with a telemetry signal.
When multiple source
blocks are specified inside a pod_association
block, both source
blocks has to match for the
pod to be associated with the telemetry signal.
The following attributes are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
from | string | The association method. Currently supports resource_attribute and connection | yes | |
name | string | Name represents extracted key name. For example, ip , pod_uid , k8s.pod.ip | no |
exclude block
The exclude
block configures which pods to exclude from the processor.
Note
Pods with the namejaeger-agent
orjaeger-collector
are excluded by default.
pod block
The pod
block configures a pod to be excluded from the processor.
The following attributes are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
name | string | The name of the pod | yes |
output block
The output
block configures a set of components to forward resulting telemetry data to.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
logs | list(otelcol.Consumer) | List of consumers to send logs to. | [] | no |
metrics | list(otelcol.Consumer) | List of consumers to send metrics to. | [] | no |
traces | list(otelcol.Consumer) | List of consumers to send traces to. | [] | no |
You must specify the output
block, but all its arguments are optional.
By default, telemetry data is dropped.
Configure the metrics
, logs
, and traces
arguments accordingly to send telemetry data to other components.
debug_metrics block
The debug_metrics
block configures the metrics that this component generates to monitor its state.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
disable_high_cardinality_metrics | boolean | Whether to disable certain high cardinality metrics. | true | no |
level | string | Controls the level of detail for metrics emitted by the wrapped collector. | "detailed" | no |
disable_high_cardinality_metrics
is the Grafana Alloy equivalent to the telemetry.disableHighCardinalityMetrics
feature gate in the OpenTelemetry Collector.
It removes attributes that could cause high cardinality metrics.
For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.
Note
If configured,disable_high_cardinality_metrics
only applies tootelcol.exporter.*
andotelcol.receiver.*
components.
level
is the Alloy equivalent to the telemetry.metrics.level
feature gate in the OpenTelemetry Collector.
Possible values are "none"
, "basic"
, "normal"
and "detailed"
.
Exported fields
The following fields are exported and can be referenced by other components:
Name | Type | Description |
---|---|---|
input | otelcol.Consumer | A value that other components can use to send telemetry data to. |
input
accepts otelcol.Consumer
data for any telemetry signal (metrics, logs, or traces).
Component health
otelcol.processor.k8sattributes
is only reported as unhealthy if given an invalid
configuration.
Debug information
otelcol.processor.k8sattributes
does not expose any component-specific debug
information.
Examples
Basic usage
In most cases, this is enough to get started. It’ll add these resource attributes to all logs, metrics, and traces:
k8s.namespace.name
k8s.pod.name
k8s.pod.uid
k8s.pod.start_time
k8s.deployment.name
k8s.node.name
Example:
otelcol.receiver.otlp "default" {
http {}
grpc {}
output {
metrics = [otelcol.processor.k8sattributes.default.input]
logs = [otelcol.processor.k8sattributes.default.input]
traces = [otelcol.processor.k8sattributes.default.input]
}
}
otelcol.processor.k8sattributes "default" {
output {
metrics = [otelcol.exporter.otlp.default.input]
logs = [otelcol.exporter.otlp.default.input]
traces = [otelcol.exporter.otlp.default.input]
}
}
otelcol.exporter.otlp "default" {
client {
endpoint = sys.env("OTLP_ENDPOINT")
}
}
Add additional metadata and labels
otelcol.receiver.otlp "default" {
http {}
grpc {}
output {
metrics = [otelcol.processor.k8sattributes.default.input]
logs = [otelcol.processor.k8sattributes.default.input]
traces = [otelcol.processor.k8sattributes.default.input]
}
}
otelcol.processor.k8sattributes "default" {
extract {
label {
from = "pod"
key_regex = "(.*)/(.*)"
tag_name = "$1.$2"
}
metadata = [
"k8s.namespace.name",
"k8s.deployment.name",
"k8s.statefulset.name",
"k8s.daemonset.name",
"k8s.cronjob.name",
"k8s.job.name",
"k8s.node.name",
"k8s.pod.name",
"k8s.pod.uid",
"k8s.pod.start_time",
]
}
output {
metrics = [otelcol.exporter.otlp.default.input]
logs = [otelcol.exporter.otlp.default.input]
traces = [otelcol.exporter.otlp.default.input]
}
}
otelcol.exporter.otlp "default" {
client {
endpoint = sys.env("OTLP_ENDPOINT")
}
}
Adding Kubernetes metadata to Prometheus metrics
otelcol.processor.k8sattributes
adds metadata to metrics signals in the form of resource attributes.
To display the metadata as labels of Prometheus metrics, the OTLP attributes must be converted from
resource attributes to datapoint attributes. One way to do this is by using an otelcol.processor.transform
component.
otelcol.receiver.otlp "default" {
http {}
grpc {}
output {
metrics = [otelcol.processor.k8sattributes.default.input]
}
}
otelcol.processor.k8sattributes "default" {
extract {
label {
from = "pod"
}
metadata = [
"k8s.namespace.name",
"k8s.pod.name",
]
}
output {
metrics = [otelcol.processor.transform.add_kube_attrs.input]
}
}
otelcol.processor.transform "add_kube_attrs" {
error_mode = "ignore"
metric_statements {
context = "datapoint"
statements = [
"set(attributes[\"k8s.pod.name\"], resource.attributes[\"k8s.pod.name\"])",
"set(attributes[\"k8s.namespace.name\"], resource.attributes[\"k8s.namespace.name\"])",
]
}
output {
metrics = [otelcol.exporter.prometheus.default.input]
}
}
otelcol.exporter.prometheus "default" {
forward_to = [prometheus.remote_write.mimir.receiver]
}
prometheus.remote_write "mimir" {
endpoint {
url = "http://mimir:9009/api/v1/push"
}
}
Compatible components
otelcol.processor.k8sattributes
can accept arguments from the following components:
- Components that export OpenTelemetry
otelcol.Consumer
otelcol.processor.k8sattributes
has exports that can be consumed by the following components:
- Components that consume OpenTelemetry
otelcol.Consumer
Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.