Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Grafana Agent Flow mode Reference Components reference otelcol.processor.resourcedetection

otelcol.processor.resourcedetection

BETA: This is a beta component. Beta components are subject to breaking changes, and may be replaced with equivalent functionality that cover the same use case.

otelcol.processor.resourcedetection detects resource information from the host in a format that conforms to the OpenTelemetry resource semantic conventions, and appends or overrides the resource values in the telemetry data with this information.

You can specify multiple otelcol.processor.resourcedetection components by giving them different labels.

Usage

river Copy

otelcol.processor.resourcedetection "LABEL" {
  output {
    logs    = [...]
    metrics = [...]
    traces  = [...]
  }
}

Arguments

otelcol.processor.resourcedetection supports the following arguments:

detectors could contain the following values:

• env
• ec2
• ecs
• eks
• elasticbeanstalk
• lambda
• azure
• aks
• consul
• docker
• gcp
• heroku
• system
• openshift
• kubernetes_node

env is the only detector that is not configured through a River block. The env detector reads resource information from the OTEL_RESOURCE_ATTRIBUTES environment variable. This variable must be in the format <key1>=<value1>,<key2>=<value2>,..., the details of which are currently pending confirmation in the OpenTelemetry specification.

If a detector other than env is needed, you can customize it with the relevant River block. For example, you can customize the ec2 detector with the ec2 block. If you omit the ec2 block, the defaults specified in the ec2 block documentation are used.

If multiple detectors are inserting the same attribute name, the first detector to insert wins. For example, if you had detectors = ["eks", "ec2"] then cloud.platform will be aws_eks instead of ec2.

The following order is recommended for AWS:

1. lambda
2. elasticbeanstalk
3. eks
4. ecs
5. ec2

Blocks

The following blocks are supported inside the definition of otelcol.processor.resourcedetection:

output

The output block configures a set of components to forward resulting telemetry data to.

The following arguments are supported:

You must specify the output block, but all its arguments are optional. By default, telemetry data is dropped. Configure the metrics, logs, and traces arguments accordingly to send telemetry data to other components.

ec2

The ec2 block reads resource information from the EC2 instance metadata API using the AWS SDK for Go.

The ec2 block supports the following attributes:

If you are using a proxy server on your EC2 instance, it’s important that you exempt requests for instance metadata as described in the AWS cli user guide. Failing to do so can result in proxied or missing instance data.

If the instance is part of AWS ParallelCluster and the detector is failing to connect to the metadata server, check the iptable and make sure the chain PARALLELCLUSTER_IMDS contains a rule that allows the Grafana Agent user to access 169.254.169.254/32.

tags can be used to gather tags for the EC2 instance which Grafana Agent is running on. To fetch EC2 tags, the IAM role assigned to the EC2 instance must have a policy that includes the ec2:DescribeTags permission.

The ec2 block supports the following blocks:

ec2 > resource_attributes

The resource_attributes block supports the following blocks:

ecs

The ecs block queries the Task Metadata Endpoint (TMDE) to record information about the current ECS Task. Only TMDE V4 and V3 are supported.

The ecs block supports the following blocks:

ecs > resource_attributes

The resource_attributes block supports the following blocks:

eks

The eks block adds resource attributes for Amazon EKS.

The eks block supports the following blocks:

eks > resource_attributes

The resource_attributes block supports the following blocks:

Example values:

• cloud.provider: "aws"
• cloud.platform: "aws_eks"

elasticbeanstalk

The elasticbeanstalk block reads the AWS X-Ray configuration file available on all Beanstalk instances with X-Ray Enabled.

The elasticbeanstalk block supports the following blocks:

elasticbeanstalk > resource_attributes

The resource_attributes block supports the following blocks:

Example values:

• cloud.provider: "aws"
• cloud.platform: "aws_elastic_beanstalk"

lambda

The lambda block uses the AWS Lambda runtime environment variables to retrieve various resource attributes.

The lambda block supports the following blocks:

lambda > resource_attributes

The resource_attributes block supports the following blocks:

Cloud semantic conventions:

• cloud.provider: "aws"
• cloud.platform: "aws_lambda"
• cloud.region: $AWS_REGION

Function as a Service semantic conventions and AWS Lambda semantic conventions:

• faas.name: $AWS_LAMBDA_FUNCTION_NAME
• faas.version: $AWS_LAMBDA_FUNCTION_VERSION
• faas.instance: $AWS_LAMBDA_LOG_STREAM_NAME
• faas.max_memory: $AWS_LAMBDA_FUNCTION_MEMORY_SIZE

AWS Logs semantic conventions:

• aws.log.group.names: $AWS_LAMBDA_LOG_GROUP_NAME
• aws.log.stream.names: $AWS_LAMBDA_LOG_STREAM_NAME

azure

The azure block queries the Azure Instance Metadata Service to retrieve various resource attributes.

The azure block supports the following blocks:

azure > resource_attributes

The resource_attributes block supports the following blocks:

Example values:

• cloud.provider: "azure"
• cloud.platform: "azure_vm"

aks

The aks block adds resource attributes related to Azure AKS.

The aks block supports the following blocks:

aks > resource_attributes

The resource_attributes block supports the following blocks:

Example values:

• cloud.provider: "azure"
• cloud.platform: "azure_vm"

Azure AKS cluster name is derived from the Azure Instance Metadata Service’s (IMDS) infrastructure resource group field. This field contains the resource group and name of the cluster, separated by underscores. For example: MC_<resource group>_<cluster name>_<location>.

Example:

• Resource group: my-resource-group
• Cluster name: my-cluster
• Location: eastus
• Generated name: MC_my-resource-group_my-cluster_eastus

The cluster name is detected if it does not contain underscores and if a custom infrastructure resource group name was not used.

If accurate parsing cannot be performed, the infrastructure resource group value is returned. This value can be used to uniquely identify the cluster, because Azure will not allow users to create multiple clusters with the same infrastructure resource group name.

consul

The consul block queries a Consul agent and reads its configuration endpoint to retrieve values for resource attributes.

The consul block supports the following attributes:

token is only required if Consul’s ACL System is enabled.

The consul block supports the following blocks:

consul > resource_attributes

The resource_attributes block supports the following blocks:

docker

The docker block queries the Docker daemon to retrieve various resource attributes from the host machine.

You need to mount the Docker socket (/var/run/docker.sock on Linux) to contact the Docker daemon. Docker detection does not work on MacOS.

The docker block supports the following blocks:

docker > resource_attributes

The resource_attributes block supports the following blocks:

gcp

The gcp block detects resource attributes using the Google Cloud Client Libraries for Go, which reads resource information from the GCP metadata server. The detector also uses environment variables to identify which GCP platform the application is running on, and assigns appropriate resource attributes for that platform.

Use the gcp detector regardless of the GCP platform Grafana Agent is running on.

The gcp block supports the following blocks:

gcp > resource_attributes

The resource_attributes block supports the following blocks:

Google Compute Engine (GCE) metadata

• cloud.provider: "gcp"
• cloud.platform: "gcp_compute_engine"
• cloud.account.id: project id
• cloud.region: e.g. "us-central1"
• cloud.availability_zone: e.g. "us-central1-c"
• host.id: instance id
• host.name: instance name
• host.type: machine type
• (optional) gcp.gce.instance.hostname
• (optional) gcp.gce.instance.name

Google Kubernetes Engine (GKE) metadata

• cloud.provider: "gcp"
• cloud.platform: "gcp_kubernetes_engine"
• cloud.account.id: project id
• cloud.region: only for regional GKE clusters; e.g. "us-central1"
• cloud.availability_zone: only for zonal GKE clusters; e.g. "us-central1-c"
• k8s.cluster.name
• host.id: instance id
• host.name: instance name; only when workload identity is disabled

One known issue happens when GKE workload identity is enabled. The GCE metadata endpoints won’t be available, and the GKE resource detector won’t be able to determine host.name. If this happens, you can set host.name from one of the following resources:

• Get the node.name through the downward API with the env detector.
• Get the Kubernetes node name from the Kubernetes API (with k8s.io/client-go).

Google Cloud Run Services metadata

• cloud.provider: "gcp"
• cloud.platform: "gcp_cloud_run"
• cloud.account.id: project id
• cloud.region: e.g. "us-central1"
• faas.id: instance id
• faas.name: service name
• faas.version: service revision

Cloud Run Jobs metadata

• cloud.provider: "gcp"
• cloud.platform: "gcp_cloud_run"
• cloud.account.id: project id
• cloud.region: e.g. "us-central1"
• faas.id: instance id
• faas.name: service name
• gcp.cloud_run.job.execution: e.g. "my-service-ajg89"
• gcp.cloud_run.job.task_index: e.g. "0"

Google Cloud Functions metadata

• cloud.provider: "gcp"
• cloud.platform: "gcp_cloud_functions"
• cloud.account.id: project id
• cloud.region: e.g. "us-central1"
• faas.id: instance id
• faas.name: function name
• faas.version: function version

Google App Engine metadata

• cloud.provider: "gcp"
• cloud.platform: "gcp_app_engine"
• cloud.account.id: project id
• cloud.region: e.g. "us-central1"
• cloud.availability_zone: e.g. "us-central1-c"
• faas.id: instance id
• faas.name: service name
• faas.version: service version

heroku

The heroku block adds resource attributes derived from Heroku dyno metadata.

The heroku block supports the following blocks:

heroku > resource_attributes

The resource_attributes block supports the following blocks:

When Heroku dyno metadata is active, Heroku applications publish information through environment variables. We map these environment variables to resource attributes as follows:

For more information, see the Heroku cloud provider documentation under the OpenTelemetry specification semantic conventions.

system

The system block queries the host machine to retrieve various resource attributes.

The system block supports the following attributes:

The valid options for hostname_sources are:

• "dns": Uses multiple sources to get the fully qualified domain name. Firstly, it looks up the host name in the local machine’s hosts file. If that fails, it looks up the CNAME. Lastly, if that fails, it does a reverse DNS query. Note: this hostname source may produce unreliable results on Windows. To produce a FQDN, Windows hosts might have better results using the “lookup” hostname source, which is mentioned below.
• "os": Provides the hostname provided by the local machine’s kernel.
• "cname": Provides the canonical name, as provided by net.LookupCNAME in the Go standard library. Note: this hostname source may produce unreliable results on Windows.
• "lookup": Does a reverse DNS lookup of the current host’s IP address.

In case of an error in fetching a hostname from a source, the next source from the list of hostname_sources will be considered.

The system block supports the following blocks:

system > resource_attributes

The resource_attributes block supports the following blocks:

openshift

The openshift block queries the OpenShift and Kubernetes APIs to retrieve various resource attributes.

The openshift block supports the following attributes:

The “get”, “watch”, and “list” permissions are required:

yaml Copy

kind: ClusterRole
metadata:
  name: grafana-agent
rules:
- apiGroups: ["config.openshift.io"]
  resources: ["infrastructures", "infrastructures/status"]
  verbs: ["get", "watch", "list"]

By default, the API address is determined from the environment variables KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT and the service token is read from /var/run/secrets/kubernetes.io/serviceaccount/token. If TLS is not explicitly disabled and no ca_file is configured, /var/run/secrets/kubernetes.io/serviceaccount/ca.crt is used. The determination of the API address, ca_file, and the service token is skipped if they are set in the configuration.

The openshift block supports the following blocks:

openshift > tls

The tls block configures TLS settings used for the connection to the gRPC server.

The following arguments are supported:

If the server doesn’t support TLS, you must set the insecure argument to true.

To disable tls for connections to the server, set the insecure argument to true.

If reload_interval is set to "0s", the certificate never reloaded.

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

• ca_pem and ca_file
• cert_pem and cert_file
• key_pem and key_file

If cipher_suites is left blank, a safe default list is used. See the Go TLS documentation for a list of supported cipher suites.

openshift > resource_attributes

The resource_attributes block supports the following blocks:

kubernetes_node

The kubernetes_node block queries the Kubernetes API server to retrieve various node resource attributes.

The kubernetes_node block supports the following attributes:

The “get” and “list” permissions are required:

yaml Copy

kind: ClusterRole
metadata:
  name: grafana-agent
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list"]

auth_type can be set to one of the following:

• none: no authentication.
• serviceAccount: use the standard service account token provided to the Grafana Agent pod.
• kubeConfig: use credentials from ~/.kube/config.

The kubernetes_node block supports the following blocks:

kubernetes_node > resource_attributes

The resource_attributes block supports the following blocks:

Common configuration

Resource attribute config

This block describes how to configure resource attributes such as k8s.node.name and azure.vm.name. Every block is configured using the same set of attributes. Only the default values for those attributes might differ across resource attributes. For example, some resource attributes have enabled set to true by default, whereas others don’t.

The following attributes are supported:

To see the default value for enabled, refer to the tables in the sections above which list the resource attributes blocks. The “Description” column will state either…

Sets enabled to true by default.

… or:

Sets enabled to false by default.

Exported fields

The following fields are exported and can be referenced by other components:

input accepts otelcol.Consumer OTLP-formatted data for any telemetry signal of these types:

• logs
• metrics
• traces

Component health

otelcol.processor.resourcedetection is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.processor.resourcedetection doesn’t expose any component-specific debug information.

Examples

env detector

If you set up a OTEL_RESOURCE_ATTRIBUTES environment variable with value of TestKey=TestValue, then all logs, metrics, and traces have a resource attribute with a key TestKey and value of TestValue.

river Copy

otelcol.processor.resourcedetection "default" {
  detectors = ["env"]

  output {
    logs    = [otelcol.exporter.otlp.default.input]
    metrics = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

env and ec2

There is no need to put in an ec2 {} River block. The ec2 defaults are applied automatically, as specified in ec2.

river Copy

otelcol.processor.resourcedetection "default" {
  detectors = ["env", "ec2"]

  output {
    logs    = [otelcol.exporter.otlp.default.input]
    metrics = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

ec2 with default resource attributes

There is no need to put in a ec2 {} River block. The ec2 defaults are applied automatically, as specified in ec2.

river Copy

otelcol.processor.resourcedetection "default" {
  detectors = ["ec2"]

  output {
    logs    = [otelcol.exporter.otlp.default.input]
    metrics = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

ec2 with explicit resource attributes

river Copy

otelcol.processor.resourcedetection "default" {
  detectors = ["ec2"]
  ec2 {
    tags = ["^tag1$", "^tag2$", "^label.*$"]
    resource_attributes {
      cloud.account.id  { enabled = true }
      cloud.availability_zone  { enabled = true }
      cloud.platform  { enabled = true }
      cloud.provider  { enabled = true }
      cloud.region  { enabled = true }
      host.id  { enabled = true }
      host.image.id  { enabled = false }
      host.name  { enabled = false }
      host.type  { enabled = false }
    }
  }

  output {
    logs    = [otelcol.exporter.otlp.default.input]
    metrics = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

kubernetes_node

This example uses the default node_from_env_var option of K8S_NODE_NAME.

There is no need to put in a kubernetes_node {} River block. The kubernetes_node defaults are applied automatically, as specified in kubernetes_node.

river Copy

otelcol.processor.resourcedetection "default" {
  detectors = ["kubernetes_node"]

  output {
    logs    = [otelcol.exporter.otlp.default.input]
    metrics = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

You need to add this to your workload:

yaml Copy

        env:
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName

kubernetes_node with a custom environment variable

This example uses a custom node_from_env_var set to my_custom_var.

river Copy

otelcol.processor.resourcedetection "default" {
  detectors = ["kubernetes_node"]
  kubernetes_node {
    node_from_env_var = "my_custom_var"
  }

  output {
    logs    = [otelcol.exporter.otlp.default.input]
    metrics = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

You need to add this to your workload:

yaml Copy

        env:
          - name: my_custom_var
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName

Compatible components

otelcol.processor.resourcedetection can accept arguments from the following components:

• Components that export OpenTelemetry otelcol.Consumer

otelcol.processor.resourcedetection has exports that can be consumed by the following components:

• Components that consume OpenTelemetry otelcol.Consumer