Migrate to Alloy

Thu, 28 May 2026 17:50:33 +0100

Migrate to Alloy

Grafana Labs has provided tools and migration documentation to assist you in migrating to Grafana Alloy.

Read more about why we recommend migrating to Grafana Alloy.

This section provides links to documentation for how to migrate to Alloy.

Set up Alloy to remote-write to Tempo

Thu, 28 May 2026 17:50:33 +0100

Set up Alloy to remote-write to Tempo

This section uses a Grafana Alloy Helm chart deployment to send traces to Tempo.

To do this, you need to create a configuration that can be used by Alloy to receive and export traces in OTLP protobuf format.

Create a new values.yaml file which we’ll use as part of the Alloy install.

Edit the values.yaml file and add the following configuration to it:

alloy:
  extraPorts:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
      protocol: TCP
  configMap:
    create: true
    content: |-
      // Creates a receiver for OTLP gRPC.
      // You can easily add receivers for other protocols by using the correct component
      // from the reference list at: https://grafana.com/docs/alloy/latest/reference/components/
      otelcol.receiver.otlp "otlp_receiver" {
        // Listen on all available bindable addresses on port 4317 (which is the
        // default OTLP gRPC port) for the OTLP protocol.
        grpc {
          endpoint = "0.0.0.0:4317"
        }

        // Output straight to the OTLP gRPC exporter. We would usually do some processing
        // first, most likely batch processing, but for this example we pass it straight
        // through.
        output {
          traces = [
            otelcol.exporter.otlp.tempo.input,
          ]
        }
      }

      // Define an OTLP gRPC exporter to send all received traces to Tempo.
      // The unique label 'tempo' is added to uniquely identify this exporter.
      otelcol.exporter.otlp "tempo" {
          // Define the client for exporting.
          client {
             // Send to the Tempo distributor on port 4317 (OTLP gRPC).
             endpoint = "distributor.tempo.svc.cluster.local:4317"
              // Disable TLS for OTLP export.
              tls {
                  // The connection is insecure.
                  insecure = true
                  // Do not verify TLS certificates when connecting.
                  insecure_skip_verify = true
              }
          }
      }

Update the distributor endpoint to match your deployment. The service name and namespace depend on how you deployed Tempo:

Tanka: distributor.<NAMESPACE>.svc.cluster.local:4317
Helm: <RELEASE-NAME>-distributor.<NAMESPACE>.svc.cluster.local:4317

Deploy Alloy using Helm:
Bash
```
helm install -f values.yaml grafana-alloy grafana/alloy
```
If you deploy Alloy into a specific namespace, create the namespace first and specify it to Helm by appending --namespace=<grafana-alloy-namespace> to the end of the command.

Automatic logging: trace discovery through logs

Thu, 28 May 2026 17:50:33 +0100

Automatic logging: trace discovery through logs

When you run instrumented distributed systems, discovering which traces exist can be challenging. Grafana Alloy’s automatic logging solves this by writing well-formatted log lines for each span, root, or process that passes through the tracing pipeline.

Automatic logging lets you discover trace IDs through log messages. You can search for traces by key-value pairs in Loki and jump from a log message directly to the trace view in Grafana.

Automatic logging versus TraceQL

Tempo’s native search now provides the same trace discovery capabilities as automatic logging, without requiring a Loki instance or generating additional log volume. TraceQL can search traces by service name, span name, duration, status, and custom attributes. For a visual, query-free approach, use Traces Drilldown.

Automatic logging is still useful if your workflow centers on Loki: trace IDs and span metadata appear alongside your application logs, letting you discover traces while investigating log data rather than switching to a separate trace search.

Before you begin

To use automatic logging, you need:

Grafana Alloy installed and receiving traces from your application.
A Loki instance to store the generated logs.
A Tempo instance to store traces so you can navigate from logs to traces.
Grafana configured with both a Loki data source and a Tempo data source.

Configure automatic logging

Automatic logging uses the otelcol.connector.spanlogs connector to generate log lines from trace spans. The connector accepts traces and generates logs, but it doesn’t forward the original traces. You must send traces to both the spanlogs connector and your trace backend to avoid losing trace data.

For high-throughput systems, logging every span may generate too much volume. In those cases, log per root span or process instead.

The connector searches for a set of span or resource attributes in the spans and logs them as key-value pairs. You can then search by those key-value pairs in Loki.

For all available configuration options, refer to the otelcol.connector.spanlogs component reference.

Log roots to an OTLP endpoint

This example logs root spans and sends the generated logs to an OTLP endpoint. Traces are forwarded to the same endpoint so they aren’t lost:

otelcol.receiver.otlp "default" {
  grpc {}
  http {}

  output {
    traces = [
      otelcol.connector.spanlogs.default.input,
      otelcol.exporter.otlp.default.input,
    ]
  }
}

otelcol.connector.spanlogs "default" {
  roots = true

  output {
    logs = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = env("OTLP_ENDPOINT")
  }
}

Log roots with custom attributes to Loki

This example logs root spans with http.method and http.target attributes, then sends the generated logs to a Loki instance. Traces are forwarded separately to a Tempo instance.

Because otelcol.exporter.loki doesn’t promote log attributes to Loki labels by default, you must use otelcol.processor.attributes to add a loki.attribute.labels hint. This promotes the traces attribute to a Loki label so you can filter by log type:

otelcol.receiver.otlp "default" {
  grpc {}
  http {}

  output {
    traces = [
      otelcol.connector.spanlogs.default.input,
      otelcol.exporter.otlp.tempo.input,
    ]
  }
}

otelcol.connector.spanlogs "default" {
  roots           = true
  span_attributes = ["http.method", "http.target"]

  output {
    logs = [otelcol.processor.attributes.default.input]
  }
}

otelcol.processor.attributes "default" {
  action {
    key    = "loki.attribute.labels"
    action = "insert"
    value  = "traces"
  }

  output {
    logs = [otelcol.exporter.loki.default.input]
  }
}

otelcol.exporter.loki "default" {
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo:4317"
  }
}

Expected output

When you enable automatic logging, the connector generates a log line for each span, root, or process depending on which options you enable. Each log line uses a logfmt-style body with the following default keys:

Key	Description
`svc`	The service name from the span’s resource.
`span`	The name of the span.
`dur`	The duration of the span in nanoseconds, for example, `150200000ns`.
`tid`	The trace ID.
`status`	The status of the span. Only included when the status is explicitly set (not `STATUS_CODE_UNSET`). Values are `STATUS_CODE_OK` or `STATUS_CODE_ERROR`.

You can add more keys by configuring span_attributes, process_attributes, or event_attributes. You can customize all key names using the overrides block. For details, refer to the otelcol.connector.spanlogs component reference.

For example, a root span log line might look like this:

span="HTTP GET" dur=150200000ns http.method=GET http.target=/api/v1/query svc=my-service tid=7bba9f33312b3dbb8b2c2c62bb7abe2d

Each log line also has a traces attribute that indicates the log type: span, root, process, or event. If you configured the loki.attribute.labels hint as shown in the Loki example, this attribute becomes a Loki label that you can filter on.

Query automatic logging data in Loki

Use LogQL in Grafana Explore to query the generated logs.

To find all root span logs:

{traces="root"}

To filter for slow requests from a specific service:

{traces="root"} | logfmt | dur > 2s and svc="my-service"

TraceQL equivalents

The following TraceQL queries provide the same trace discovery as the LogQL queries above, without requiring automatic logging or a Loki instance.

To find all traces from a specific service:

{ resource.service.name = "my-service" }

To find slow traces (duration over 2 seconds) from a specific service:

{ resource.service.name = "my-service" && span:duration > 2s }

To find error traces:

{ status = error }

To search by a specific attribute:

{ span.http.method = "GET" && span.http.target = "/api/v1/query" }

Refer to Construct TraceQL queries for the full query syntax.

Navigate from logs to traces

To link from a log line directly to its trace in Tempo, configure derived fields on your Loki data source in Grafana:

Go to Connections > Data sources and select your Loki data source.
In Derived fields, add a new field with these settings:
- Name: TraceID
- Type: Label
- Match field name: tid
- Internal link enabled, pointing to your Tempo data source
Save the data source configuration.

After this configuration, log results in Explore display a link next to the tid field.

Grafana Alloy on Grafana Labs

Migrate to Alloy

Migrate to Alloy

Set up Alloy to remote-write to Tempo

Set up Alloy to remote-write to Tempo

Automatic logging: trace discovery through logs

Automatic logging: trace discovery through logs

Automatic logging versus TraceQL

Before you begin

Configure automatic logging

Log roots to an OTLP endpoint

Log roots with custom attributes to Loki

Expected output

Query automatic logging data in Loki

TraceQL equivalents

Navigate from logs to traces