Set up OpenTelemetry Collector for Application Observability
Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Monitor applicationsbreadcrumb arrow Application Observabilitybreadcrumb arrow Set up OpenTelemetry Collectorbreadcrumb arrow OpenTelemetry Collector
Grafana Cloud RSS

Set up OpenTelemetry Collector for Application Observability

The OpenTelemetry project maintainers and the Cloud Native Computing Foundation maintain the upstream OpenTelemetry Collector.

For production observability, Grafana Labs recommends Grafana Alloy, an OpenTelemetry Collector distribution, that packages various upstream OpenTelemetry Collector components and Prometheus exporters, to provide stability, support, and an integrated experience with Grafana Application Observability and other Grafana products.

For production, Application Observability requires an OpenTelemetry Collector on every host to seamlessly correlate data between Infrastructure and Application observability.

To install the OpenTelemetry Collector for Application Observability, install the contrib OpenTelemetry Collector distribution.

Grafana Labs recommends you use the OpenTelemetry Collector Grafana Cloud integration tile to configure the OpenTelemetry Collector.

If you have an existing OpenTelemetry Collector per host, refer to the advanced manual setup and configure your application sections of the documentation.

Advanced manual setup

For advanced use cases you can manually configure the OpenTelemetry config.yaml configuration file:

yaml 
# Tested with OpenTelemetry Collector Contrib v0.98.0
receivers:
  otlp:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver
    protocols:
      grpc:
      http:
  hostmetrics:
    # Optional. Host Metrics Receiver added as an example of Infra Monitoring capabilities of the OpenTelemetry Collector
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver
    scrapers:
      load:
      memory:

processors:
  batch:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor
  resourcedetection:
    # Enriches telemetry data with resource information from the host
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor
    detectors: ["env", "system"]
    override: false
  transform/drop_unneeded_resource_attributes:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor
    error_mode: ignore
    trace_statements:
      - context: resource
        statements:
          - delete_key(attributes, "k8s.pod.start_time")
          - delete_key(attributes, "os.description")
          - delete_key(attributes, "os.type")
          - delete_key(attributes, "process.command_args")
          - delete_key(attributes, "process.executable.path")
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")
          - delete_key(attributes, "process.runtime.name")
          - delete_key(attributes, "process.runtime.version")
    metric_statements:
      - context: resource
        statements:
          - delete_key(attributes, "k8s.pod.start_time")
          - delete_key(attributes, "os.description")
          - delete_key(attributes, "os.type")
          - delete_key(attributes, "process.command_args")
          - delete_key(attributes, "process.executable.path")
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")
          - delete_key(attributes, "process.runtime.name")
          - delete_key(attributes, "process.runtime.version")
    log_statements:
      - context: resource
        statements:
          - delete_key(attributes, "k8s.pod.start_time")
          - delete_key(attributes, "os.description")
          - delete_key(attributes, "os.type")
          - delete_key(attributes, "process.command_args")
          - delete_key(attributes, "process.executable.path")
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")
          - delete_key(attributes, "process.runtime.name")
          - delete_key(attributes, "process.runtime.version")
  transform/add_resource_attributes_as_metric_attributes:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["deployment.environment"], resource.attributes["deployment.environment"])
          - set(attributes["service.version"], resource.attributes["service.version"])

exporters:
  otlphttp/grafana_cloud:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter
    endpoint: "${env:GRAFANA_CLOUD_OTLP_ENDPOINT}"
    auth:
      authenticator: basicauth/grafana_cloud
    
extensions:
  basicauth/grafana_cloud:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/basicauthextension
    client_auth:
      username: "${env:GRAFANA_CLOUD_INSTANCE_ID}"
      password: "${env:GRAFANA_CLOUD_API_KEY}"

connectors:
  grafanacloud:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/grafanacloudconnector
    host_identifiers: ["host.name"]

service:
  extensions:
    [
      basicauth/grafana_cloud,
    ]
  pipelines:
    traces:
      receivers: [otlp]
      processors:
        [resourcedetection, transform/drop_unneeded_resource_attributes, batch]
      exporters: [otlphttp/grafana_cloud, grafanacloud]
    metrics:
      receivers: [otlp, hostmetrics]
      processors:
        [
          resourcedetection,
          transform/drop_unneeded_resource_attributes,
          transform/add_resource_attributes_as_metric_attributes,
          batch,
        ]
      exporters: [otlphttp/grafana_cloud]
    metrics/grafanacloud:
      receivers: [grafanacloud]
      processors: [batch]
      exporters: [otlphttp/grafana_cloud]
    logs:
      receivers: [otlp]
      processors:
        [
          resourcedetection,
          transform/drop_unneeded_resource_attributes,
          batch,
        ]
      exporters: [otlphttp/grafana_cloud]

Set the following environmental variables in the configuration file:

Environment VariableDescriptionExample
GRAFANA_CLOUD_API_KEYAPI key generated aboveeyJvSomeLongStringJ9fQ==
GRAFANA_CLOUD_OTLP_ENDPOINTOTLP endpoint from Grafana Cloud > OpenTelemetry > Configurehttps://otlp-endpoint-xyz.grafana.net/otlp
GRAFANA_CLOUD_INSTANCE_IDInstance ID from Grafana Cloud > OpenTelemetry > Configure

Data pipelines

OpenTelemetry Collector receives OTLP data and processes it with the following pipelines:

Traces:

The traces pipeline receives traces with the otlp receiver and exports them to the Grafana Cloud Tempo with the otlp exporter.

The traces pipeline uses the resourcedetection processor to enrich telemetry data with resource information from the host.

Consult the resource detection processor README.md for a list of configuration options.

Metrics:

The metrics pipeline receives traces from the otlp receiver and exports metrics to the Grafana Cloud Metrics with the prometheusremotewrite exporter.

The metrics pipeline uses the transform processor to add deployment.environment, and service.version labels to metrics.

Logs:

The logs pipeline receives logs with the otlp receiver and exports them to the Grafana Cloud Loki with the loki exporter.

Run OpenTelemetry Collector

Create the config.yaml file, set the necessary environment variables, and run the OpenTelemetry Collector.

Configure your application

Set the following environment variables to configure your application to use the OpenTelemetry Collector:

ConfigurationOptionsResult
export OTEL_EXPORTER_OTLP_ENDPOINT=<host>http://localhost:4318, remote host addressThe default local host address, or a remote host address.
export OTEL_EXPORTER_OTLP_PROTOCOL=<protocol>grpc, http/protobufThe default http/protobuf protocol or grpc

For example, for a local OpenTelemetry Collector set the following environment variables:

shell 
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Then restart your application.

Next steps

Observe your services in Application Observability