Connect OpenTelemetry Collector to Grafana Cloud databases

OpenTelemetry Collector is a middleware used to connect sources of telemetry data with backends that can store the data reliably, while also processing and cleaning data in between. This document guides you through configuring an OpenTelemetry Collector to send data to Grafana Cloud Metrics, Traces, and Logs.

Note: Grafana Labs also provides an OpenTelemetry Protocol (OTLP) endpoint, which sends OTLP data directly to Grafana Cloud databases on a single endpoint. We encourage you to try it out today especially if your telemetry data format is OTLP.

This guide uses the OpenTelemetry Demo application for logs, metrics and traces. At the end of this guide, you’ll have a good understanding of the flow of the telemetry data, as well as confidence in running the collector with production-quality configuration.

Even though this guide uses the OpenTelemetry Demo application, the same concepts apply to applications generating OTLP data, likely with the OpenTelemetry SDK. Note that the OpenTelemetry Demo generates OTLP metrics and traces natively, making it an excellent candidate to use Grafana Cloud’s OTLP endpoint. Logs are obtained from the stdout of the pods and are similar to what you’d see if you ran kubectl logs. They are then converted to OTLP by the Collector.

The demo contains several microservices and tools deployed via Docker Compose or Kubernetes. This guide has adapted the upstream demo so that logs from Kubernetes pods are also collected. This guide will be adapted in the future based on changes made to the upstream demo project.

Read the demo’s README first to get an idea of the demo and its scenarios.
Even though Docker Compose is listed as an option, follow the Kubernetes instructions, as that’s the one we’ll use for the remaining of this guide.
While the services are starting, sign up for a free (forever!) Grafana Cloud account if you don’t have one yet.
In your Grafana Cloud account, take note of the following details:
- Under the Prometheus section, note the Prometheus Remote Write endpoint, for example, https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push. We’ll refer to this as ${METRICS_URL}. Note the “Username / Instance ID” as well, for example, 123456. We’ll refer to this as ${METRICS_USER_ID}.
- Under the Tempo section, note down the URL, for example, https://tempo-us-central1.grafana.net/tempo. We’ll use the hostname and port from the URL as our ${TRACES_URL}, for example, tempo-us-central1.grafana.net:443. Note the “Username / Instance ID” as well, for example, 12345. We’ll refer to this as ${TRACES_USER_ID}
- Under the Loki section, note down the URL, for example, https://logs-prod-us-central1.grafana.net. For the Loki exporter, we specify the complete URL, which should include the path /loki/api/v1/push. Combine both (https://logs-prod-us-central1.grafana.net/loki/api/v1/push) and we’ll refer to this as ${LOGS_URL}. Note the “Username / Instance ID”, for example, 12345. We’ll refer to this as ${LOGS_USER_ID}
- You might need to create an API under the API Key tab with the MetricsPublisher role. The API key looks like this: eyJ...fQ==. We’ll refer to this as ${API_KEY}.

With the demo application started and an OpenTelemetry Collector running, create a custom Helm values file by saving the contents of the following snippet into a file named grafana.yaml, replacing the “${…}” values with the ones you obtained earlier.

opentelemetry-collector:
  mode: daemonset
  presets:
    logsCollection:
      enabled: true
      includeCollectorLogs: false
  config:
    extensions:
      basicauth/traces:
        client_auth:
          username: "${TRACES_USER_ID}"
          password: "${API_KEY}"
      basicauth/metrics:
        client_auth:
          username: "${METRICS_USER_ID}"
          password: "${API_KEY}"
      basicauth/logs:
        client_auth:
          username: "${LOGS_USER_ID}"
          password: "${API_KEY}"   

    exporters:
      otlp:
        endpoint: ${TRACES_URL}
        tls:
          insecure: false
        auth:
          authenticator: basicauth/traces

      prometheusremotewrite:
        endpoint: ${METRICS_URL}
        auth:
          authenticator: basicauth/metrics

      loki:
        endpoint: ${LOGS_URL}
        auth:
          authenticator: basicauth/logs

    processors:
      spanmetrics:
        metrics_exporter: prometheusremotewrite

    service:
      extensions: [ basicauth/traces, basicauth/metrics, basicauth/logs, health_check , memory_ballast ]
      pipelines:
        metrics:
          exporters: [prometheusremotewrite]
        logs:
          receivers: [filelog]
          exporters: [loki]

Note: On a production system, you’d add the credentials to a Kubernetes Secret, setting the credentials as environment variables in the Collector deployment instead of replacing the values directly in the configuration.

Install or upgrade the chart by running one of the following Helm commands.

helm upgrade my-otel-demo open-telemetry/opentelemetry-demo --values grafana.yaml

If you didn’t install the chart yet, use the following command instead:

helm install my-otel-demo open-telemetry/opentelemetry-demo --values grafana.yaml

Until the pull request open-telemetry/opentelemetry-helm-charts#623 is merged, create a service that will receive the spans from our services. If you changed the instance name from my-otel-demo to another value, make sure to apply the same change to the service definition:

apiVersion: v1
kind: Service
metadata:
  name: my-otel-demo-otelcol
  namespace: default
spec:
  ports:
  - name: jaeger-compact
    port: 6831
    protocol: UDP
    targetPort: 6831
  - name: jaeger-grpc
    port: 14250
    protocol: TCP
    targetPort: 14250
  - name: jaeger-thrift
    port: 14268
    protocol: TCP
    targetPort: 14268
  - name: metrics
    port: 8888
    protocol: TCP
    targetPort: 8888
  - name: otlp
    port: 4317
    protocol: TCP
    targetPort: 4317
  - name: otlp-http
    port: 4318
    protocol: TCP
    targetPort: 4318
  - name: prometheus
    port: 9464
    protocol: TCP
    targetPort: 9464
  - name: zipkin
    port: 9411
    protocol: TCP
    targetPort: 9411
  selector:
    app.kubernetes.io/instance: my-otel-demo
    app.kubernetes.io/name: otelcol
    component: agent-collector
  type: ClusterIP

Once the new Collector instance is running with the new configuration, open the Explore tab in your Grafana Cloud instance, select your traces data source, and click Search under Query type.
The list of services should contain names like frontend, adservice, cartservice, etc.
Select frontend and click Run query in the top-right corner.
You should now see a list of traces, each representing a specific HTTP request performed by the load generator:
Note: Spans from different services might arrive at the storage at different times, which explains the <root span not yet received> message. Eventually, the root span should be available.
Switch the data source to your metrics data source, and enter a query, for example:
rate(calls_total{service_name="frontend"}[5m])
You should start seeing lines in a graph, each one representing the rate of HTTP requests for specific HTTP return codes:
Switch the data source to your logs data source, and enter a query, for example:
```
{exporter="OTLP"} | json | resources_k8s_container_name=`frontend`
```
Note: Currently, the logging capabilities of the OpenTelemetry Collector are under development. That said, you can use a few components to capture log entries from sources like journald and local disk files, then convert them into OTLP data and make them available to exporters such as the OTLP and Loki exporters. We currently lack receivers that can grab the log entries from our containers or Kubernetes pods. Therefore, our example deploys an OpenTelemetry Collector as a DaemonSet, mounting the logs node path on the pod. We use the filelog receiver to read the pods’ logs.
You should see a few log entries for the frontend service:

Summary

The OpenTelemetry Demo contains multiple microservices, simulating a common distributed system pattern. Each service is instrumented with the OpenTelemetry SDK and exporting its telemetry using the OTLP Exporter for the individual SDKs. Each microservice sends the telemetry data to the Collector that we have deployed as a DaemonSet, which then sends the data to the final destination: the individual databases part of Grafana Cloud, such as Grafana Cloud Metrics and Grafana Cloud Traces. We also have the Collector to scrape the local log files pertaining the pods in the cluster, converting the raw log lines into OTLP, eventually exporting them to Grafana Cloud Logs.

Troubleshooting

Reduce the number of synthetic users and spawn rate from the load generator (http://localhost:8080/loadgen/) if you experience rate limiting errors, like:
the request has been rejected because the tenant exceeded the ingestion rate limit, set to 1500 items/s with a maximum allowed burst of 15000
The Grafana Cloud Traces URL should contain only the hostname and port. In our example, it would be tempo-us-central1.grafana.net:443. Otherwise, an error like the following might occur:
transport: Error while dialing dial tcp: address tempo-us-central1.grafana.net/tempo: missing port in address

Summary

In this article, you learned how to connect an application generating OTLP data with Grafana Cloud Metrics and Grafana Cloud Traces through the OpenTelemetry Collector. You also learned to get local logs and send them to Grafana Cloud Logs. While this is still the officially supported solution for which there is an SLA in Grafana Cloud, we encourage you to try our new OTLP endpoint, which simplifies much of the configuration used in this guide.

Connect OpenTelemetry Collector to Grafana Cloud databases

Summary

Troubleshooting

Summary

Related documentation