Why and How to Adopt the Native OTLP Log Format?

OpenTelemetry (OTel) is quickly becoming an industry standard with increasing adoption. Prior to the Loki 3.0 release, there was no native support for ingesting OTel logs to Loki, which led to creation of the LokiExporter. While there was an OTLP compliant log ingestion endpoint available for Grafana Cloud, internally it used LokiExporter to transform an OTel push payload into a Loki push payload, and ingested those logs using the non-OTel Loki endpoint.

Although the LokiExporter got the job done of ingesting OTel logs to Loki, it did not provide a native user experience, and the querying experience was not optimal. As part of an effort to improve user experience with OTel, native OTel log ingestion support was added to Loki with the 3.0 release.

Who is this guide relevant for?

Native OTLP ingestion support for Loki went live on Grafana Cloud on March 25, 2024. Due to the difference in formatting and querying experience between the old and new ingestion formats, routing OTel logs via the native endpoint for Loki tenants already ingesting OTel logs would have been a breaking change. To avoid breaking things for tenants already ingesting logs before the native ingestion endpoint went live, all the tenants who ingested any OTel logs 90 days before March 25, 2024 were pinned to old ingestion format. The guide is only relevant for Grafana Cloud Logs users whose Loki tenants have been pinned to the old OTel log ingestion format.

You can contact support if you are not getting the native OTel experience and want to find out if your Loki tenant is pinned to the old OTel json/logfmt ingestion format.

What has changed?

Support for indexing of Attributes

Logs ingested via LokiExporter support indexing Resource and Log Attributes. However, with native OTel log ingestion endpoint of Loki, indexing of only Resource Attributes is supported because Log Attributes are expected to be high cardinality, which is not a good fit for the Loki index and can affect performance.

Log formatting

LokiExporter’s README explains in detail the formatting of OTel logs ingested via LokiExporter, while native OTel log ingestion endpoint of Loki is described in detail in its documentation. Here is a summary of how the logs are formatted between the two:

LokiExporter

Index Labels:
- It supports label control via hints set by the OTel client/collector.
- Default labels:
  - job=LogRecord.Resource.Attributes["service.namespace"]/LogRecord.Resource.Attributes["service.name"]
  - instance=LogRecord.Resource.Attributes["service.instance.id"]
  - exporter=“OTLP”
  - level=LogRecord.SeverityText
Log Body: Encodes log records and attributes in json(default) or logfmt format.

Native OTel log ingestion endpoint of Loki

Index Labels:
- It supports label control via per-tenant OTLP configuration in Loki.
- By default, it picks some pre-configured resource attributes as index labels as explained here.
LogLine: LogRecord.Body as a string.
Structured Metadata: Anything not stored in Index labels and LogLine gets stored as Structured Metadata.

Sample Log

Given the following sample OTel log, here’s how the log record would appear in both formats after ingestion with the default configuration:

- Resource Attributes:
    - service.name: "shoppingcart"
    - service.namespace: "shop"
    - deployment.environment: "staging"
- InstrumentationScope:
    - Name: "login"
- LogRecord:
    - Timestamp: 1715247552000000000
    - Body: "user logged in"
    - SeverityText: "INFO"
    - SeverityNumber: 9
    - TraceId: "08040201000000000000000000000000"
    - SpanId: "0102040800000000"
    - Attributes:
        - thread.name: "main"

Ingested with LokiExporter:

- Index Labels:
    - job="shop/shoppingcart"
    - exporter="OTLP"
- Log Body: `{"body":"user logged in","severity":"INFO","attributes":{"traceid":"08040201000000000000000000000000", "spanid": "0102040800000000"}, "resources":{"deployment.environment": "staging","service.name":"shoppingcart","service.namespace":"shop"}, "instrumentation_scope": {"name":"login"}}`

Ingested with native OTel endpoint of Loki:

- Index Labels:
    - deployment_environment="staging"
    - service_name="shoppingcart"
    - service_namespace="shop"
- Log Body: "user logged in"
- Structured Metadata:
    - severity_text: "INFO"
    - severity_number: "9"
    - trace_id: "08040201000000000000000000000000"
    - span_id: "0102040800000000"
    - scope_name: "login"

Query experience

The LokiExporter encodes data to json or logfmt blob, which requires decoding it at query time to interact with OTel attributes. However, the native OTel endpoint doesn’t do any encoding of data before storing it. It leverages Structured Metadata, which makes it easier to interact with OTel attributes without having to use any parsers in the queries. Taking the above-ingested log line, here is how the querying experience would look between the two considering various scenarios:

Query all logs without any filters

Ingested with LokiExporter: {job="shop/shoppingcart"}
Ingested with native OTel endpoint of Loki: {service_name="shoppingcart", service_namespace="shop"}

Query logs with severity=INFO

Ingested with LokiExporter: {job="shop/shoppingcart"} | json | severity="INFO"
Ingested with native OTel endpoint of Loki: {service_name="shoppingcart", service_namespace="shop"} | severity_text="INFO"

Query logs with non-indexed Resource Attribute “deployment_environment” with value “staging”

Ingested with LokiExporter: {job="shop/shoppingcart"} | json | resources_deployment_environment="staging"
Ingested with native OTel endpoint of Loki: {service_name="shoppingcart", service_namespace="shop"} | deployment_environment="staging"

Query logs with TraceId “08040201000000000000000000000000”

Ingested with LokiExporter: {job="shop/shoppingcart"} | json | attributes_traceid="08040201000000000000000000000000"
Ingested with native OTel endpoint of Loki: {service_name="shoppingcart", service_namespace="shop"} | trace_id="08040201000000000000000000000000"

Query logs with InstrumentationScope.Name=“login”

Ingested with LokiExporter: {job="shop/shoppingcart"} | json | instrumentation_scope_name="login"
Ingested with native OTel endpoint of Loki: {service_name="shoppingcart", service_namespace="shop"} | scope_name="login"

Query logs with LogRecord.Attributes[“thread.name”]=“main”

Ingested with LokiExporter: {job="shop/shoppingcart"} | json | attributes_thread_name="name"
Ingested with native OTel endpoint of Loki: {service_name="shoppingcart", service_namespace="shop"} | thread_name="main"

Display log message as log line in logs query results

Ingested with LokiExporter: {job="shop/shoppingcart"} | json | line_format "{{.body}}"
Ingested with native OTel endpoint of Loki: {service_name="shoppingcart", service_namespace="shop"}

Count logs in the last hour by severity

Ingested with LokiExporter: sum(count_over_time({job="shop/shoppingcart"} | json[1h])) by (severity_text)
Ingested with native OTel endpoint of Loki: sum(count_over_time({service_name="shoppingcart", service_namespace="shop"}[1h])) by (severity_text)

Benefits of switching from LokiExporter to native OTel endpoint:

Future improvements: The native OTel endpoint of Loki represents the future of the product, and all future development and enhancements are going to be focused there. There is an ongoing discussion around deprecating the LokiExporter, which would stop receiving new enhancements. Upgrading to the native endpoint ensures that you benefit from the latest enhancements and have the best possible user experience.
Simplified client config: LokiExporter requires setting hints for managing stream labels, which defeats the purpose of choosing OTel in the first place since you can’t switch from one OTel-compatible storage to another without having to reconfigure your clients. With native OTel endpoint of Loki, there is no added complexity in setting hints for your client to manage the labels.
Simplified querying: Native OTel endpoint of Loki leverages Structured Metadata, which makes it easier to reference the attributes and other fields from the OTel LogRecord without having to use json or logfmt parsers for decoding the data at query time.
More modern high context data model: Better aligned with modern observability practices than the older JSON based model.

What do you need to do to switch from old OTel ingestion format to native OTel ingestion format?

Rewrite your LogQL queries in various places, including dashboards, alerts, starred queries in Grafana Explore, etc., to query OTel logs according to the new format.
Reach out to support if you’d like to update the OTLP configuration for your tenant. Per-tenant OTLP configuration in Loki lets you choose which Resource Attributes to use as index labels or remove any unwanted Attributes.
After you have ensured everything is in place to start using the new format, you can contact support to unpin your tenant from the old format. Unpinning the tenant from the old format should begin routing your OTel log ingestion requests via native OTel endpoint of Loki.
If you are using the Application Observability solution in Grafana, change the toggle under the Logs tab from Loki exporter query to OTLP gateway / native Loki otlp query for it to update the queries with the new format.
Perform visualization migration in Grafana to seamlessly navigate between logs and traces.