Monitor applications

Application Observability

Set up a production Collector

OpenTelemetry Collector

Grafana Cloud

Application Observability with OpenTelemetry Collector

Grafana Application Observability is compatible with the OpenTelemetry Collector, however the feature is experimental.

There are Grafana users that successfully use the upstream Collector with Grafana Application Observability, Grafana can provide the correct configuration to support the required features where possible.

Before you begin

To set up OpenTelemetry Collector as the data collector to send to Grafana Cloud:

Create and/or login to a Grafana Cloud account
Install an OpenTelemetry Collector

There are two distributions of the OpenTelemetry Collector: core and contrib. Application Observability requires the contrib distribution.

If you’ve already set up a Collector, configure your application to use it.

Configure the Collector

You need a config.yaml configuration file for the OpenTelemetry Collector to run. It’s recommended to use the OpenTelemetry Collector integration to generate a configuration file. To find this integration, navigate to OpenTelemetry > Application and click Add service. Choose the OpenTelemetry Collector integration from the list.

The OpenTelemetry Collector integration generates OpenTelemetry Collector configuration to use with Grafana Application Observability:

# Tested with OpenTelemetry Collector Contrib v0.98.0
receivers:
  otlp:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver
    protocols:
      grpc:
      http:
  hostmetrics:
    # Optional. Host Metrics Receiver added as an example of Infra Monitoring capabilities of the OpenTelemetry Collector
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver
    scrapers:
      load:
      memory:

processors:
  batch:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor
  resourcedetection:
    # Enriches telemetry data with resource information from the host
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor
    detectors: ["env", "system"]
    override: false
  transform/drop_unneeded_resource_attributes:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor
    error_mode: ignore
    trace_statements:
      - context: resource
        statements:
          - delete_key(attributes, "k8s.pod.start_time")
          - delete_key(attributes, "os.description")
          - delete_key(attributes, "os.type")
          - delete_key(attributes, "process.command_args")
          - delete_key(attributes, "process.executable.path")
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")
          - delete_key(attributes, "process.runtime.name")
          - delete_key(attributes, "process.runtime.version")
    metric_statements:
      - context: resource
        statements:
          - delete_key(attributes, "k8s.pod.start_time")
          - delete_key(attributes, "os.description")
          - delete_key(attributes, "os.type")
          - delete_key(attributes, "process.command_args")
          - delete_key(attributes, "process.executable.path")
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")
          - delete_key(attributes, "process.runtime.name")
          - delete_key(attributes, "process.runtime.version")
    log_statements:
      - context: resource
        statements:
          - delete_key(attributes, "k8s.pod.start_time")
          - delete_key(attributes, "os.description")
          - delete_key(attributes, "os.type")
          - delete_key(attributes, "process.command_args")
          - delete_key(attributes, "process.executable.path")
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")
          - delete_key(attributes, "process.runtime.name")
          - delete_key(attributes, "process.runtime.version")
  transform/add_resource_attributes_as_metric_attributes:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["deployment.environment"], resource.attributes["deployment.environment"])
          - set(attributes["service.version"], resource.attributes["service.version"])

exporters:
  otlphttp/grafana_cloud:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter
    endpoint: "${env:GRAFANA_CLOUD_OTLP_ENDPOINT}"
    auth:
      authenticator: basicauth/grafana_cloud
    
extensions:
  basicauth/grafana_cloud:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/basicauthextension
    client_auth:
      username: "${env:GRAFANA_CLOUD_INSTANCE_ID}"
      password: "${env:GRAFANA_CLOUD_API_KEY}"

connectors:
  grafanacloud:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/grafanacloudconnector
    host_identifiers: ["host.name"]

service:
  extensions:
    [
      basicauth/grafana_cloud,
    ]
  pipelines:
    traces:
      receivers: [otlp]
      processors:
        [resourcedetection, transform/drop_unneeded_resource_attributes, batch]
      exporters: [otlphttp/grafana_cloud, grafanacloud]
    metrics:
      receivers: [otlp, hostmetrics]
      processors:
        [
          resourcedetection,
          transform/drop_unneeded_resource_attributes,
          transform/add_resource_attributes_as_metric_attributes,
          batch,
        ]
      exporters: [otlphttp/grafana_cloud]
    metrics/grafanacloud:
      receivers: [grafanacloud]
      processors: [batch]
      exporters: [otlphttp/grafana_cloud]
    logs:
      receivers: [otlp]
      processors:
        [
          resourcedetection,
          transform/drop_unneeded_resource_attributes,
          batch,
        ]
      exporters: [otlphttp/grafana_cloud]

The instrumented application sends traces, metrics, and logs to the collector via OTLP. The collector receives data and processes it with defined pipelines.

The traces pipeline receives traces with the otlp receiver and exports them to the Grafana Cloud Tempo with the otlp exporter. The resourcedetection processor here and in the further pipelines is used to enrich telemetry data with resource information from the host.

Consult the resource detection processor README.md for a list of configuration options.

The metrics pipeline receives traces from the otlp receiver, and applies a transform processor to add deployment.environment, and service.version labels to metrics, and exports metrics to the Grafana Cloud Metrics with the prometheusremotewrite exporter. The hostmetrics receiver is optional. It’s added here as an example of Infra Monitoring capabilities of the OpenTelemetry Collector.

The logs pipeline receives logs with the otlp receiver and exports them to the Grafana Cloud Loki with the loki exporter.

The configuration file requires several environmental variable to be set. They are described in the following table.

Environment Variable	Description	Example
`GRAFANA_CLOUD_API_KEY`	API key generated above	`eyJvSomeLongStringJ9fQ==`
`GRAFANA_CLOUD_OTLP_ENDPOINT`	OTLP endpoint from Grafana Cloud > OpenTelemetry > Configure	`https://otlp-endpoint-***.grafana.net/otlp`
`GRAFANA_CLOUD_INSTANCE_ID`	Instance ID from Grafana Cloud > OpenTelemetry > Configure	`11111`

Run OpenTelemetry Collector

Create the config.yaml file, set the necessary environment variables, and start the OpenTelemetry Collector.

Configure your application

Set the following environment variables to configure the Collector for your application:

Configuration	Options	Result
`export OTEL_EXPORTER_OTLP_ENDPOINT=<host>`	`http://localhost:4318`, remote host address	The default local host address, or a remote host address.
`export OTEL_EXPORTER_OTLP_PROTOCOL=<protocol>`	`grpc`, `http/protobuf`	The default `http/protobuf` protocol or `grpc`

For example, for a local OpenTelemetry Collector set the following environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Then restart your application.

Next steps

Observe your services in Application Observability

Was this page helpful?

Email docs@grafana.com

Help and support

Community

Feedback

Relevant sources:

Feedback

Application Observability with OpenTelemetry Collector

Before you begin

Configure the Collector

Run OpenTelemetry Collector

Configure your application

Next steps

Was this page helpful?

Related documentation

Application Observability with OpenTelemetry Collector

Before you begin

Configure the Collector

Run OpenTelemetry Collector

Configure your application

Next steps

Was this page helpful?

Related documentation

Related resources from Grafana Labs