Remove default OpenTelemetry labels
When Grafana Loki first started supporting OpenTelemetry, we selected a set of default resource attributes to promote to labels. We included some labels such as k8s.pod.name to be consistent with the scrape configs we have been suggesting for many years for Promtail and Grafana Alloy. However, Loki has evolved a lot over the years and with the introduction of Structured Metadata we no longer recommend indexing a few of the labels we included in the defaults as they often have very high cardinality and Structured Metadata is a much better place for them. We no longer recommend the following as default labels:
k8s.pod.name
service.instance.id
Because removing these resource attributes from labels would be a breaking change for existing users, they have not yet been deprecated. If you are a new user of Grafana Loki, we recommend that you modify your Grafana Alloy or OpenTelemetry Collector configuration to convert these resource attributes from index labels to structured metadata.
Alloy configuration file example
If you are using
Grafana Alloy to collect logs, the following example shows how to update your alloy.config
file to demote k8s.pod.name
from an index label to structured metadata.
Kubernetes Monitoring Helm chart configuration example
If you are using the
Kubernetes Monitoring Helm chart to collect Kubernetes logs to send to Loki, the following example shows how to update the values.yaml
file for your Helm chart to demote k8s.pod.name
and service.instance.id
from index labels to structured metadata.
# Enable pod log collection for the cluster. Will collect logs from all pods in both the meta and loki namespace.
podLogs:
enabled: true
collector: alloy-singleton
#List of attributes to use as index labels in oki
labelsToKeep:
- app
- app_kubernetes_io_name
- component
- container
- job
- level
- namespace
- service_name
- cluster
gatherMethod: kubernetesApi
namespaces:
- meta
- loki
#This is the section that specifies to send k8s.pod.name and service.instance.id to structured metadata
structuredMetadata:
instance_id:
pod:
Loki configuration example
If you are running Loki in a Docker container or on-premises, here is an example of how to modify your Loki values.yaml
file to demote k8s.pod.name
and service.instance.id
from index labels to structured metadata.
distributor:
otlp_config:
# List of default otlp resource attributes to be picked as index labels - EDIT TO REMOVE k8s.pod.name AND service.instance.id FROM THE LIST
# CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels
default_resource_attributes_as_index_labels: [service.name service.namespace deployment.environment deployment.environment.name cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name]
Kubernetes example values.yaml
:
loki:
distributor:
otlp_config:
# List of default otlp resource attributes to be picked as index labels - EDIT TO REMOVE k8s.pod.name AND service.instance.id FROM THE LIST
# CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels
default_resource_attributes_as_index_labels: [service.name service.namespace deployment.environment deployment.environment.name cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name]