OpenTelemetry OpenTelemetry Collector Connect OpenTelemetry Collector to Grafana Cloud databases

Connect OpenTelemetry Collector to Grafana Cloud databases

OpenTelemetry Collector is a middleware used to connect sources of telemetry data with backends that can store the data reliably, while also processing and cleaning data in between. This document guides you through configuring an OpenTelemetry Collector to send data to Grafana Cloud Metrics, Traces, and Logs.

Note: Grafana Labs also provides an OpenTelemetry Protocol (OTLP) endpoint, which sends OTLP data directly to Grafana Cloud databases on a single endpoint. We encourage you to try it out today especially if your telemetry data format is OTLP.

This guide uses the OpenTelemetry Demo application for logs, metrics and traces. At the end of this guide, you’ll have a good understanding of the flow of the telemetry data, as well as confidence in running the collector with production-quality configuration.

Even though this guide uses the OpenTelemetry Demo application, the same concepts apply to applications generating OTLP data, likely with the OpenTelemetry SDK. Note that the OpenTelemetry Demo generates OTLP metrics and traces natively, making it an excellent candidate to use Grafana Cloud’s OTLP endpoint. Logs are obtained from the stdout of the pods and are similar to what you’d see if you ran kubectl logs. They are then converted to OTLP by the Collector.

The demo contains several microservices and tools deployed via Docker Compose or Kubernetes. This guide has adapted the upstream demo so that logs from Kubernetes pods are also collected. This guide will be adapted in the future based on changes made to the upstream demo project.

1. Read the demo’s README first to get an idea of the demo and its scenarios.

2. Even though Docker Compose is listed as an option, follow the Kubernetes instructions, as that’s the one we’ll use for the remaining of this guide.

3. While the services are starting, sign up for a free (forever!) Grafana Cloud account if you don’t have one yet.

4. In your Grafana Cloud account, take note of the following details:

   • Under the Prometheus section, note the Prometheus Remote Write endpoint, for example, https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push. We’ll refer to this as ${METRICS_URL}. Note the “Username / Instance ID” as well, for example, 123456. We’ll refer to this as ${METRICS_USER_ID}.

   • Under the Tempo section, note down the URL, for example, https://tempo-us-central1.grafana.net/tempo. We’ll use the hostname and port from the URL as our ${TRACES_URL}, for example, tempo-us-central1.grafana.net:443. Note the “Username / Instance ID” as well, for example, 12345. We’ll refer to this as ${TRACES_USER_ID}

   • Under the Loki section, note down the URL, for example, https://logs-prod-us-central1.grafana.net. For the Loki exporter, we specify the complete URL, which should include the path /loki/api/v1/push. Combine both (https://logs-prod-us-central1.grafana.net/loki/api/v1/push) and we’ll refer to this as ${LOGS_URL}. Note the “Username / Instance ID”, for example, 12345. We’ll refer to this as ${LOGS_USER_ID}

   • You might need to create an API under the API Key tab with the MetricsPublisher role. The API key looks like this: eyJ...fQ==. We’ll refer to this as ${API_KEY}.

5. With the demo application started and an OpenTelemetry Collector running, create a custom Helm values file by saving the contents of the following snippet into a file named grafana.yaml, replacing the “${…}” values with the ones you obtained earlier.

   yaml Copy

   default:
     envOverrides:
     - name: 'OTEL_COLLECTOR_NAME'
       value: $(OTEL_K8S_NODE_NAME)
   opentelemetry-collector:
     mode: daemonset
     presets:
       logsCollection:
         enabled: true
         includeCollectorLogs: false
     config:
       extensions:
         basicauth/traces:
           client_auth:
             username: "${TRACES_USER_ID}"
             password: "${API_KEY}"
         basicauth/metrics:
           client_auth:
             username: "${METRICS_USER_ID}"
             password: "${API_KEY}"
         basicauth/logs:
           client_auth:
             username: "${LOGS_USER_ID}"
             password: "${API_KEY}"   
   
       exporters:
         otlp:
           endpoint: ${TRACES_URL}
           tls:
             insecure: false
           auth:
             authenticator: basicauth/traces
   
         prometheusremotewrite:
           endpoint: ${METRICS_URL}
           auth:
             authenticator: basicauth/metrics
   
         loki:
           endpoint: ${LOGS_URL}
           auth:
             authenticator: basicauth/logs
   
       processors:
         spanmetrics:
           metrics_exporter: prometheusremotewrite
   
       service:
         extensions: [ basicauth/traces, basicauth/metrics, basicauth/logs, health_check , memory_ballast ]
         pipelines:
           metrics:
             exporters: [prometheusremotewrite]
           logs:
             receivers: [filelog]
             exporters: [loki]

   Note: On a production system, you’d add the credentials to a Kubernetes Secret, setting the credentials as environment variables in the Collector deployment instead of replacing the values directly in the configuration.

6. Install or upgrade the chart by running one of the following Helm commands.

   console Copy

   helm upgrade my-otel-demo open-telemetry/opentelemetry-demo --values grafana.yaml

   If you didn’t install the chart yet, use the following command instead:

   console Copy

   helm install my-otel-demo open-telemetry/opentelemetry-demo --values grafana.yaml

7. Once the new Collector instance is running with the new configuration, open the Explore tab in your Grafana Cloud instance, select your traces data source, and click Search under Query type.

   The list of services should contain names like frontend, adservice, cartservice, etc.

8. Select frontend and click Run query in the top-right corner.

   You should now see a list of traces, each representing a specific HTTP request performed by the load generator:

   Note: Spans from different services might arrive at the storage at different times, which explains the <root span not yet received> message. Eventually, the root span should be available.

9. Switch the data source to your metrics data source, and enter a query, for example:

   rate(calls_total{service_name="frontend"}[5m])

   You should start seeing lines in a graph, each one representing the rate of HTTP requests for specific HTTP return codes:

10. Switch the data source to your logs data source, and enter a query, for example:

    Copy

    {exporter="OTLP"} | json | resources_k8s_container_name=`frontend`

    Note: Currently, the logging capabilities of the OpenTelemetry Collector are under development. That said, you can use a few components to capture log entries from sources like journald and local disk files, then convert them into OTLP data and make them available to exporters such as the OTLP and Loki exporters. We currently lack receivers that can grab the log entries from our containers or Kubernetes pods. Therefore, our example deploys an OpenTelemetry Collector as a DaemonSet, mounting the logs node path on the pod. We use the filelog receiver to read the pods’ logs.

    You should see a few log entries for the frontend service:

Summary

The OpenTelemetry Demo contains multiple microservices, simulating a common distributed system pattern. Each service is instrumented with the OpenTelemetry SDK and exporting its telemetry using the OTLP Exporter for the individual SDKs. Each microservice sends the telemetry data to the Collector that we have deployed as a DaemonSet, which then sends the data to the final destination: the individual databases part of Grafana Cloud, such as Grafana Cloud Metrics and Grafana Cloud Traces. We also have the Collector to scrape the local log files pertaining the pods in the cluster, converting the raw log lines into OTLP, eventually exporting them to Grafana Cloud Logs.

Troubleshooting

1. Reduce the number of synthetic users and spawn rate from the load generator (http://localhost:8080/loadgen/) if you experience rate limiting errors, like:

   the request has been rejected because the tenant exceeded the ingestion rate limit, set to 1500 items/s with a maximum allowed burst of 15000

2. The Grafana Cloud Traces URL should contain only the hostname and port. In our example, it would be tempo-us-central1.grafana.net:443. Otherwise, an error like the following might occur:

   transport: Error while dialing dial tcp: address tempo-us-central1.grafana.net/tempo: missing port in address

Summary

In this article, you learned how to connect an application generating OTLP data with Grafana Cloud Metrics and Grafana Cloud Traces through the OpenTelemetry Collector. You also learned to get local logs and send them to Grafana Cloud Logs. While this is still the officially supported solution for which there is an SLA in Grafana Cloud, we encourage you to try our new OTLP endpoint, which simplifies much of the configuration used in this guide.