Documentation OpenTelemetry Send and ingest OTLP data Multi-Zone OTLP Ingestion

Multi-Zone OTLP Ingestion

Grafana Cloud is enhancing its cloud instances with multi-zone support to increase the resilience of your observability data ingestion. This feature is currently experimental and being rolled out incrementally, so your instance may or may not have support for it. It is worth noting that, when available, multi-zone support is exposed through a different OTLP endpoint than your regular Grafana Cloud OTLP endpoint.

What is multi-zone OTLP ingestion?

Multi-zone means that a Grafana Cloud instance is deployed across multiple zones in a cloud provider region. This architecture is designed to mitigate the potential impact of an issue scoped to one zone, allowing your data ingestion to continue through another healthy zone. Although the exact definition of a zone may vary by cloud provider, its purpose is the same, to provide a level of redundancy and fault tolerance.

How to use the multi-zone OTLP endpoint

To leverage the benefits of multi-zone resilience for your OTLP data, you need to use a separate OTLP endpoint than the one specified in your Grafana Cloud instance.

Currently, multi-zone OTLP endpoints do not support the ingestion of all telemetry signals. This means that, in terms of configuration, the multi-zone endpoint is not a drop-in replacement for the standard OTLP endpoint. Refer to Client Configuration for more details on how to configure your OpenTelemetry clients to use the multi-zone endpoint.

Aside from the partial support of telemetry signals, the multi-zone OTLP endpoint behaves the same way as the regular OTLP endpoint. Authentication methods remain unchanged, and data ingested through the multi-zone endpoint lands in the same instances as those ingested through the regular endpoint.

Below is a list of regions for which multi-zone OTLP endpoints are available, along with the telemetry signals they support:

Regions supporting multi-zone OTLP endpoints

Client configuration

Because the multi-zone OTLP endpoint does not support all telemetry signals, you need to configure your OpenTelemetry client to use the multi-zone endpoint for supported signals and the regular endpoint for the unsupported ones. This configuration varies depending on the client you are using.

Below are examples of how one would configure different clients for prod-eu-north-0, where only metrics are supported by the multi-zone endpoint.

Alloy

diff Copy

otelcol.exporter.otlphttp "grafana_cloud" {
+ metrics_endpoint = "https://otlp-gateway-prod-eu-north-0-multi-zone.grafana.net/otlp/v1/metrics"
  client {
    endpoint = "https://otlp-gateway-prod-eu-north-0.grafana.net/otlp"
    auth = otelcol.auth.basic.grafana_cloud.handler
  }
}

otelcol.auth.basic "grafana_cloud" {
  username = "123"
  password = "glc_..."
}

OpenTelemetry Collector

diff Copy

exporters:
  otlphttp:
    endpoint: https://otlp-gateway-prod-eu-north-0.grafana.net/otlp
    headers:
      authorization: ...
+  otlphttp/multiaz:
+    endpoint: https://otlp-gateway-prod-eu-north-0-multi-zone.grafana.net/otlp
+    headers:
+      authorization: ...

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
-      exporters: [otlphttp]
+      exporters: [otlphttp/multiaz]
    logs:
      receivers: [otlp]
      exporters: [otlphttp]

Application from OpenTelemetry SDKs

diff Copy

OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-eu-north-0.grafana.net/otlp"
+OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="https://otlp-gateway-prod-eu-north-0-multi-zone.grafana.net/otlp/v1/metrics"
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n $GRAFANA_INSTANCE_ID:$TOKEN | base64 -w 0)"