A guide to the native OTLP log format in Loki 3
Grafana Cloud

A guide to the native OTLP log format in Loki 3

Grafana Cloud introduced native OTLP ingestion support for Loki 3 on March 25, 2024. The new format aligns with the OTLP specification and offers an improved formatting and querying experience. All customers that were ingesting logs 90 days before March 25, 2024 were pinned to the previous format using LokiExporter.

This guide is for Grafana Cloud users who are pinned to the previous log ingestion format, and covers the benefits of the native OLTP format and how to switch.

Note

If you need help to switch to the native OTLP format or assess which format you are using, contact support.

Improved indexes

LokiExporter indexes both resource and log attributes. The native OTLP format supports indexing only resource attributes and not log attributes. Indexing log attributes can have high cardinality and affect Loki performance.

Format differences

The following summary describes the main differences between LokiExporter and native OTLP ingestion support for Loki 3:

LokiExporter:

  • index labels:
    • supports label control via hints set by the OpenTelemetry client/collector
    • default labels:
      • job=LogRecord.Resource.Attributes["service.namespace"]/LogRecord.Resource.Attributes["service.name"]
      • instance=LogRecord.Resource.Attributes["service.instance.id"]
      • exporter=“OTLP”
      • level=LogRecord.SeverityText
  • log body: encodes log records and attributes in json(default) or logfmt format

Native OTLP support:

  • index labels:
    • supports label control via per-tenant OTLP configuration in Loki
    • by default it picks pre-configured resource attributes as index labels, to learn more refer to the OTLP format considerations documentation
  • LogLine: LogRecord.Body as a string
  • structured metadata: anything not stored in index labels and LogLine gets stored as structured metadata.

Sample log

The following example demonstrates how the sample log record would appear in both formats after ingestion with the default configuration:

yaml
- Resource Attributes:
  - service.name: "shoppingcart"
  - service.namespace: "shop"
  - deployment.environment: "staging"
- InstrumentationScope:
  - Name: "login"
- LogRecord:
  - Timestamp: 1715247552000000000
  - Body: "user logged in"
  - SeverityText: "INFO"
  - SeverityNumber: 9
  - TraceId: "08040201000000000000000000000000"
  - SpanId: "0102040800000000"
  - Attributes:
    - thread.name: "main"

LokiExporter:

yaml
- Index Labels:
  - job="shop/shoppingcart"
  - exporter="OTLP"
- Log Body: `{"body":"user logged in","severity":"INFO","attributes":{"traceid":"08040201000000000000000000000000", "spanid": "0102040800000000"}, "resources":{"deployment.environment": "staging","service.name":"shoppingcart","service.namespace":"shop"}, "instrumentation_scope": {"name":"login"}}`

Native OTLP support:

yaml
- Index Labels:
  - deployment_environment="staging"
  - service_name="shoppingcart"
  - service_namespace="shop"
- Log Body: "user logged in"
- Structured Metadata:
  - severity_text: "INFO"
  - severity_number: "9"
  - trace_id: "08040201000000000000000000000000"
  - span_id: "0102040800000000"
  - scope_name: "login"

Query experience

The LokiExporter encodes data to json or logfmt blob, which requires decoding it at query time to interact with OpenTelemetry attributes. Loki 3 with native OTLP ingestion doesn’t encode data before storing it. It leverages structured metadata, allowing you to interact with attributes without having to use parsers in queries.

The following scenarios compare the querying experience of the two formats using the previous log line example:

Query all logs without any filters:

  • LokiExporter: {job="shop/shoppingcart"}
  • native OTLP support: {service_name="shoppingcart", service_namespace="shop"}

Query logs with severity=INFO:

  • LokiExporter: {job="shop/shoppingcart"} | json | severity="INFO"
  • native OTLP support: {service_name="shoppingcart", service_namespace="shop"} | severity_text="INFO"

Query logs with non-indexed resource attribute and deployment_environment="staging":

  • LokiExporter: {job="shop/shoppingcart"} | json | resources_deployment_environment="staging"
  • native OTLP support: {service_name="shoppingcart", service_namespace="shop"} | deployment_environment="staging"

Query logs with specific TraceId:

  • LokiExporter: {job="shop/shoppingcart"} | json | attributes_traceid="08040201000000000000000000000000"
  • native OTLP support: {service_name="shoppingcart", service_namespace="shop"} | trace_id="08040201000000000000000000000000"

Query logs with InstrumentationScope.Name="login":

  • LokiExporter: {job="shop/shoppingcart"} | json | instrumentation_scope_name="login"
  • native OTLP support: {service_name="shoppingcart", service_namespace="shop"} | scope_name="login"

Query logs with LogRecord.Attributes["thread.name"]="main":

  • LokiExporter: {job="shop/shoppingcart"} | json | attributes_thread_name="name"
  • native OTLP support: {service_name="shoppingcart", service_namespace="shop"} | thread_name="main"

Display log message as log line in logs query results:

  • LokiExporter: {job="shop/shoppingcart"} | json | line_format "{{.body}}"
  • native OTLP support: {service_name="shoppingcart", service_namespace="shop"}

Count logs in the last hour by severity:

  • LokiExporter: sum(count_over_time({job="shop/shoppingcart"} | json[1h])) by (severity_text)
  • native OTLP support: sum(count_over_time({service_name="shoppingcart", service_namespace="shop"}[1h])) by (severity_text)

Benefits of native OTLP support

  • Future improvements: The native OTLP ingestion support for Loki 3 represents the future of log ingestion. Upgrading to the native OTLP format ensures that you benefit from the latest enhancements and have the best possible user experience.
  • Simplified client config: LokiExporter requires setting hints for managing stream labels. With the native OTLP format there is no added complexity in setting hints for your client to manage labels.
  • Simplified querying: The native OTLP format uses structured metadata allowing you to interact with attributes without having to use parsers in queries.
  • Modern high context data model: More aligned with modern observability practices than the older JSON based model.

Switch to native OTLP ingestion support for Loki

  1. Rewrite all your LogQL queries to query logs according to the new format.
  2. Reach out to support if you’d like to update the OTLP configuration for your tenant. Refer to the Loki OTLP documentation for information on resource attributes and index labels.
  3. Contact support to unpin your tenant from the LokiExporter format and start using the native OTLP format.
  4. If you use Application Observability, update your logs configuration from Loki exporter query to OTLP gateway / native Loki otlp query.
  5. Perform a visualization migration in Grafana to be able to seamlessly navigate between logs and traces.