OTLP: OpenTelemetry Protocol format considerations
OpenTelemetry doesn’t specify a data storage approach and leaves it to the observability backend to implement. Grafana Cloud converts metrics, logs, and traces sent to the Grafana Cloud OTLP endpoint to be compatible with Grafana databases.
The Grafana Cloud OTLP endpoint supports the OTLP/HTTP with binary protocol buffer encoding.
Metrics
Grafana Cloud stores metrics in Mimir, a Prometheus compatible database, and maps OpenTelemetry metrics following the official specification for OTLP Metric points to Prometheus.
Exponential histograms
By default, Grafana Cloud doesn’t accept OpenTelemetry exponential histograms.
To accept exponential histograms in Grafana Cloud, follow the send Prometheus native histograms documentation to enable Prometheus native histogram metrics for your environment.
When enabled, Grafana cloud converts OpenTelemetry exponential histograms into Prometheus native histograms following the official specifications for exponential histograms.
Metric and label name conversion
Prometheus metric and label names need to adhere to the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*
which contains alphanumeric, underscore, and colon characters. Grafana converts fullstops .
and dashes -
in OpenTelemetry metric or label names to underscores _
to store them in Prometheus.
Example
If you sent the following OTLP data:
requests.duration{http.response.status_code=500, http.route="/api/orders"}
Grafana converts it for Prometheus storage:
requests_duration{http_response_status_code="500", http_route="/api/orders"}
Resource attributes added to target_info
metric
Grafana Cloud follows the OpenTelemetry specification when it converts resource attributes to labels and into the target_info
metric.
Grafana Cloud adds resource attributes as labels to the target_info
metric, except for service.instance.id
, service.name
, and service.namespace
resource attributes:
service.namespace/service.name
added to thejob
label. Ifservice.namespace
is empty, onlyservice.name
is added to thejob
label.service.instance.id
added to theinstance
label
Note
Contact support if you would like to keepservice.instance.id
,service.name
, andservice.namespace
in thetarget_info
metric in additon to converting them intojob
andinstance
labels.
To retrieve the resource attributes for a metric:
Use the on(job, instance)
operator to join the metric with the target_info
metric, for example use the following Prometheus query to get the service.version
attribute for a requests_total
metric:
requests_total * on(job, instance) group_left(service_version) target_info
Alternatively you can promote (copy) service.instance.id
, service.name
, service.namespace
, or any other resource attribute to labels, for example as shown in A practical guide to data collection with OpenTelemetry and Prometheus.
Suffixes added to metric names
By default Prometheus adds suffixes to metric names to comply with the Prometheus metric types name conventions. Grafana Cloud follows the official specification for OTLP Metric points to Prometheus.
The suffix generation procedure is as follows:
- Split the OTel unit into two parts, before and after any
/
character. - Add
_<UNIT>
to the suffix if the metric name doesn’t contain the first unit part. - Add
_per_<UNIT>
to the suffix if there’s a second unit part and the metric name doesn’t contain it. - Add
_total
to the suffix if the OTel metric type is monotonic sum. - Add
ratio
to the suffix if the OTel metric type is gauge and the unit is1
.
Note
Grafana Labs recommends you keep the default Prometheus metric name suffix generation enabled. If you need it disabled contact support.
Example
Grafana Cloud converts the OpenTelemetry monotonic sum system.io
with unit By
into the Prometheus name system_io_bytes_total
, and _bytes_total
is the generated suffix.
Metrics ingestion limits
When you send batch metrics, Prometheus ingests only the metrics that don’t exceed limits and returns exceptions for the metrics that exceed limits.
Logs
Grafana Cloud converts OpenTelemetry logs and stores them in the Loki V3 format to leverage structured metadata and labels.
The conversion process is as follows:
Note
To configure the resource attributes that Loki promotes as labels contact support.
Logs ingestion limits
When you send batch logs, Loki ingests only the logs that don’t exceed limits and returns exceptions for the logs that exceed limits.
Loki versions before V3
Before Loki V3 introduced structured metadata, Loki converted OpenTelemetry logs in Alloy otelcol.exporter.loki
or in the OpenTelemetry Collector Loki Exporter and stored them as JSON in the message of log lines.
To promote more resource attributes and log attributes to Loki labels, use the hints loki.resource.labels
and loki.attribute.labels
as documented on OpenTelemetry Collector Loki Exporter and Alloy otelcol.exporter.loki
.