Configure the OpenTelemetry Collector to write metrics into Mimir

Note
To send OpenTelemetry data to Grafana Cloud, refer to Send data using OpenTelemetry Protocol (OTLP).

When using the OpenTelemetry Collector, you can use the OpenTelemetry protocol (OTLP) or the Prometheus remote write protocol to write metrics into Mimir. It’s recommended that you use the OpenTelemetry protocol.

Use the OpenTelemetry protocol

Mimir supports native OTLP over HTTP. To configure the collector to use the OTLP interface, use the otlphttp exporter and the native Mimir endpoint. For example:

yaml
exporters:
  otlphttp:
    endpoint: http://<mimir-endpoint>/otlp

Then, enable it in the service.pipelines block:

yaml
service:
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [..., otlphttp]

If you want to authenticate using basic auth, use the basicauth extension. For example:

yaml
extensions:
  basicauth/otlp:
    client_auth:
      username: username
      password: password

exporters:
  otlphttp:
    auth:
      authenticator: basicauth/otlp
    endpoint: http://<mimir-endpoint>/otlp

service:
  extensions: [basicauth/otlp]
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [..., otlphttp]

Use the Prometheus remote write protocol

To use the Prometheus remote write protocol to send metrics into Mimir, use the prometheusremotewrite exporter in the Collector and the native Mimir endpoint.

In the exporters section, add:

yaml
exporters:
  prometheusremotewrite:
    endpoint: http://<mimir-endpoint>/api/v1/push

Then, enable it in the service.pipelines block:

yaml
service:
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [..., prometheusremotewrite]

If you want to authenticate using basic auth, use the basicauth extension. For example:

yaml
extensions:
  basicauth/prw:
    client_auth:
      username: username
      password: password

exporters:
  prometheusremotewrite:
    auth:
      authenticator: basicauth/prw
    endpoint: http://<mimir-endpoint>/api/v1/push

service:
  extensions: [basicauth/prw]
  pipelines:
    metrics:
      receivers: [...]
      processors: [...]
      exporters: [..., prometheusremotewrite]

Work with default OpenTelemetry labels

OpenTelemetry metrics use resource attributes to describe the set of characteristics associated with a given resource, or entity, producing telemetry data. For example, a host resource might have multiple attributes, including an ID, an image, and a type.

To optimize the storage of and ability to query this data, you can configure Mimir to promote specified OTel resource attributes to labels at the time of ingestion, with periods (.) replaced by underscores (_).

Grafana Cloud automatically promotes the following OTel resource attributes to labels:

service.instance.id
service.name
service.namespace
service.version
cloud.availability_zone
cloud region
container.name
deployment.environment.name
k8s.cluster.name
k8s.container.name
k8s.cronjob.name
k8s.daemonset.name
k8s.deployment.name
k8s.job.name
k8s.namespace.name
k8s.pod.name
k8s.replicaset.name
k8s.statefulset.name

Note
To disable this setting or to update this list, contact Grafana Labs Support.

Mimir stores additional OTel resource attributes in a separate series called target_info, which you can query using a join query or the Prometheus info() function. Refer to Functions in the Prometheus documentation for more information.

To learn more about OpenTelemetry resource attributes, refer to Resources in the OpenTelemetry documentation.

Format considerations

We follow the official OTLP Metric points to Prometheus specification.

By default, Grafana Mimir does not accept OpenTelemetry Exponential Histogram metrics. For Grafana Mimir to accept them, ingestion of Prometheus Native Histogram metrics must first be enabled following the instructions in Configure native histogram ingestion. After this is done, Grafana Mimir will accept OpenTelemetry Exponential Histograms, and convert them into Prometheus Native Histograms following the conventions described in the Exponential Histograms specification.

You might experience the following common issues:

Dots (.) are converted to _
Prometheus metrics do not support . and - characters in metric or label names. Prometheus converts these characters to _.
For example:
requests.duration{http.status_code=500, cloud.region=us-central1} in OTLP
requests_duration{http_status_code=”500”, cloud_region=”us-central1”} in Prometheus
Resource attributes are added to the target_info metric.
However, <service.namespace>/<service.name> or <service.name> (if the namespace is empty), is added as the label job, and service.instance.id is added as the label instance to every metric.
For details, see the OpenTelemetry Resource Attributes specification.