Documentation Grafana Cloud Instrument and send data OpenTelemetry OTLP format considerations

OTLP: OpenTelemetry Protocol format considerations

OpenTelemetry doesn’t specify a data storage approach and leaves it to the observability backend to implement. Grafana Cloud converts metrics, logs, and traces sent to the Grafana Cloud OTLP endpoint to be compatible with Grafana databases.

The Grafana Cloud OTLP endpoint supports the OTLP/HTTP with binary protocol buffer encoding with either no compression or gzip compression.

Metrics

Grafana Cloud stores metrics in Mimir, a Prometheus compatible database, and maps OpenTelemetry metrics following the official specification for OTLP Metric points to Prometheus.

Exponential histograms

Grafana Cloud accepts OpenTelemetry exponential histograms by default. Grafana Cloud converts OpenTelemetry exponential histograms into Prometheus native histograms following the official specifications for exponential histograms.

Metric and label name conversion

Prometheus metric and label names need to adhere to the regular expression [a-zA-Z_:][a-zA-Z0-9_:]* which contains alphanumeric, underscore, and colon characters. Grafana converts fullstops . and dashes - in OpenTelemetry metric or label names to underscores _ to store them in Prometheus.

Example:

If you sent the following OTLP data:

JSON Copy

requests.duration{http.response.status_code=500, http.route="/api/orders"}

Grafana converts it for Prometheus storage:

JSON Copy

requests_duration{http_response_status_code="500", http_route="/api/orders"}

Resource attributes added to target_info metric

Grafana Cloud follows the OpenTelemetry specification when it converts resource attributes to labels and into the target_info metric.

Grafana Cloud adds resource attributes as labels to the target_info metric, except for service.instance.id, service.name, and service.namespace resource attributes:

• service.namespace/service.name added to the job label. If service.namespace is empty, only service.name is added to the job label
• service.instance.id added to the instance label

To retrieve the resource attributes for a metric use the on(job, instance) operator to join the metric with the target_info metric, for example use the following Prometheus query to get the service.version attribute for a requests_total metric:

PromQL Copy

requests_total * on(job, instance) group_left(service_version) target_info

Work with default OpenTelemetry labels

OpenTelemetry metrics use resource attributes to describe the set of characteristics associated with a given resource, or entity, producing telemetry data. For example, a host resource might have multiple attributes, including an ID, an image, and a type.

Grafana Cloud automatically promotes the following OTel resource attributes to labels, with periods (.) replaced by underscores (_):

• service.instance.id
• service.name
• service.namespace
• service.version
• cloud.availability_zone
• cloud.region
• container.name
• deployment.environment
• deployment.environment.name
• k8s.cluster.name
• k8s.container.name
• k8s.cronjob.name
• k8s.daemonset.name
• k8s.deployment.name
• k8s.job.name
• k8s.namespace.name
• k8s.pod.name
• k8s.replicaset.name
• k8s.statefulset.name

Mimir stores additional OTel resource attributes in a separate series called target_info, which you can query using a join query or the Prometheus info() function. Refer to Functions in the Prometheus documentation for more information.

To learn more about OpenTelemetry resource attributes, refer to Resources in the OpenTelemetry documentation.

Promote OpenTelemetry labels

You can configure Mimir to promote metrics labels, for example service.instance.id, service.name, service.namespace, or any other resource attribute to labels.

To configure Mimir to promote metrics labels, refer to the Mimir documentation on how to work with default OpenTelemetry labels .

For example usage, refer to the practical guide to data collection with OpenTelemetry and Prometheus blog post.

Suffixes added to metric names

By default Prometheus adds suffixes to metric names to comply with the Prometheus metric types name conventions. Grafana Cloud follows the official specification for OTLP Metric points to Prometheus.

The suffix generation procedure is as follows:

1. Split the OTel unit into two parts, before and after any / character.
2. Add _<UNIT> to the suffix if the metric name doesn’t contain the first unit part.
3. Add _per_<UNIT> to the suffix if there’s a second unit part and the metric name doesn’t contain it.
4. Add _total to the suffix if the OTel metric type is a monotonic sum.
5. Add ratio to the suffix if the OTel metric type is gauge and the unit is 1.

Example:

Grafana Cloud converts the OpenTelemetry monotonic sum system.io with unit By into the Prometheus name system_io_bytes_total, and _bytes_total is the generated suffix.

Metrics ingestion limits

When you send batch metrics, Prometheus ingests only the metrics that don’t exceed limits and returns exceptions for the metrics that exceed limits.

Logs

Grafana Cloud converts OpenTelemetry logs and stores them in the Loki 3 format to leverage structured metadata and labels.

The conversion process is as follows:

Logs ingestion limits

When you send batch logs, Loki ingests only the logs that don’t exceed limits and returns exceptions for the logs that exceed limits.

Loki versions before v3

Before Loki 3 introduced structured metadata, Loki converted OpenTelemetry logs in the Alloy otelcol.exporter.loki component or in the OpenTelemetry Collector Loki Exporter and stored them as JSON in the message of log lines.

To promote more resource attributes and log attributes to Loki labels, use the hints loki.resource.labels and loki.attribute.labels as detailed in the OpenTelemetry Collector Loki Exporter documentation and the Alloy otelcol.exporter.loki component documentation.

Traces

Traces limits