Grafana Agent on Grafana Labs

Automatic logging: Trace discovery through logs

Wed, 11 Mar 2026 18:13:39 +0000

Automatic logging: Trace discovery through logs

Running instrumented distributed systems is a very powerful way to gain understanding over a system, but it brings its own challenges. One of them is discovering which traces exist.

In the beginning of Tempo, querying for a trace was only possible if you knew the ID of the trace you were looking for. One solution was automatic logging. Automatic logging provides an easy and fast way of discovering trace IDs through log messages. Well-formatted log lines are written to a Loki instance or to stdout for each span, root, or process that passes through the tracing pipeline. This allows for automatically building a mechanism for trace discovery. On top of that, we also get metrics from traces using Loki, and allow quickly jumping from a log message to the trace view in Grafana.

While this approach is useful, it isn’t as powerful as TraceQL. If you are here because you know you want to log the trace ID, to enable jumping from logs to traces, then read on!

If you want to query the system directly, read the TraceQL documentation. We doubt you’ll be sad.

Configuration

For high throughput systems, logging for every span may generate too much volume. In such cases, logging per root span or process is recommended.

Automatic logging searches for a given set of attributes in the spans and logs them as key-value pairs. This allows searching by those key-value pairs in Loki.

Before you begin

To configure automatic logging, you need to select your preferred backend and the trace data to log.

To see all the available config options, refer to the configuration reference.

This simple example logs trace roots to stdout and is a good way to get started using automatic logging:

traces:
  configs:
  - name: default
    ...
    automatic_logging:
      backend: stdout
      roots: true

This example pushes logs directly to a Loki instance also configured in the same Grafana Agent.

traces:
  configs:
  - name: default
    ...
    automatic_logging:
      backend: logs_instance
      logs_instance_name: default
      roots: true

Examples

Enable service graphs

Wed, 11 Mar 2026 18:13:39 +0000

Enable service graphs

A service graph is a visual representation of the interrelationships between various services. Service graphs help to understand the structure of a distributed system, and the connections and dependencies between its components.

The same service graph metrics can also be generated by Tempo. This is more efficient and recommended for larger installations. For a deep look into service graphs, visit this section.

Service graphs are also used in the application performance management dashboard. For more information, refer to the service graph view documentation.

Before you begin

Service graphs are generated in the Grafana Agent and pushed to a Prometheus-compatible backend. Once generated, they can be represented in Grafana as a graph. You will need these components to fully use service graphs.

Enable service graphs in Grafana Agent

To start using service graphs, enable the feature in the Grafana Agent config.

traces:
  configs:
    - name: default
      ...
      service_graphs:
        enabled: true

To see all the available config options, refer to the configuration reference.

Metrics are registered in the Agent’s default registerer. Therefore, they are exposed at /metrics in the Agent’s server port (default 12345). One option is to use the Agent self-scrape capabilities to export the metrics to a Prometheus-compatible backend.

metrics:
  configs:
    - name: default
      scrape_configs:
        - job_name: local_scrape
          static_configs:
            - targets: ['127.0.0.1:12345']
      remote_write:
        - url: <remote_write>

Grafana

The same service graph metrics can also be generated by Tempo. This is more efficient and recommended for larger installations.

For additional information about viewing service graph metrics in Grafana and calculating cardinality, refer to the server side documentation.

Generate metrics from spans

Wed, 11 Mar 2026 18:13:39 +0000

Generate metrics from spans

Span metrics allow you to generate metrics from your tracing data automatically. Span metrics aggregates request, error and duration (RED) metrics from span data. Metrics are exported in Prometheus format.

There are two options available for exporting metrics: using remote write to a Prometheus compatible backend or serving the metrics locally and scraping them.

Span metrics generate two metrics: a counter that computes requests, and a histogram that computes operation’s durations.

Span metrics are of particular interest if your system is not monitored with metrics, but it has distributed tracing implemented. You get out-of-the-box metrics from your tracing pipeline.

Even if you already have metrics, span metrics can provide in-depth monitoring of your system. The generated metrics show application-level insight into your monitoring, as far as tracing gets propagated through your applications.

Span metrics are also used in the service graph view. For more information, refer to the service graph view.

Server-side metrics

The same span metrics can also be generated by Tempo. This is more efficient and recommended for larger installations. For a deep look into span metrics, visit this section.

Example

Tail-based sampling

Wed, 11 Mar 2026 18:13:39 +0000

Tail-based sampling

Tempo aims to provide an inexpensive solution that makes 100% sampling possible. However, sometimes constraints make a lower sampling percentage necessary or desirable, such as runtime or egress traffic related costs. Probabilistic sampling strategies are easy to implement, but also run the risk of discarding relevant data that you’ll later want.

Tail-based sampling works with Grafana Agent in Flow or static modes. Flow mode configuration files are written in River. Static mode configuration files are written in YAML. Examples in this document are for Flow mode. You can also use the Static mode Kubernetes operator.

How tail-based sampling works

In tail-based sampling, sampling decisions are made at the end of the workflow allowing for a more accurate sampling decision. The Grafana Agent groups spans by trace ID and checks its data to see if it meets one of the defined policies (for example, latency or status_code). For instance, a policy can check if a trace contains an error or if it took longer than a certain duration.

A trace is sampled if it meets at least one policy.

To group spans by trace ID, the Agent buffers spans for a configurable amount of time, after which it considers the trace complete. Longer running traces are split into more than one. However, waiting longer times increases the memory overhead of buffering.

One particular challenge of grouping trace data is for multi-instance Agent deployments, where spans that belong to the same trace can arrive to different Agents. To solve that, you can configure the Agent to load balance traces across agent instances by exporting spans belonging to the same trace to the same instance.

This is achieved by redistributing spans by trace ID once they arrive from the application. The Agent must be able to discover and connect to other Agent instances where spans for the same trace can arrive. Kubernetes users should use a headless service.

Redistributing spans by trace ID means that spans are sent and received twice, which can cause a significant increase in CPU usage. This overhead increases with the number of Agent instances that share the same traces.

Quickstart

To start using tail-based sampling, define a sampling policy. If you’re using a multi-instance deployment of the agent, add load balancing and specify the resolving mechanism to find other Agents in the setup. To see all the available configuration options, refer to the configuration reference.

Example for Grafana Agent Flow

Grafana Agent Flow is a component-based revision of Grafana Agent with a focus on ease-of-use, debuggability, and ability to adapt to the needs of power users. Flow configuration files are written in River instead of YAML.

Grafana Agent Flow uses the otelcol.processor.tail_sampling component` for tail-based sampling.

otelcol.receiver.otlp "otlp_receiver" {
    grpc {
        endpoint = "0.0.0.0:4317"
    }

    output {
        traces = [
            otelcol.processor.tail_sampling.policies.input,
        ]
    }
}

otelcol.exporter.otlp "tempo" {
    client {
        endpoint = "tempo:4317"
    }
}

// The Tail Sampling processor will use a set of policies to determine which received
// traces to keep and send to Tempo.
otelcol.processor.tail_sampling "policies" {
    // Total wait time from the start of a trace before making a sampling decision.
    // Note that smaller time periods can potentially cause a decision to be made
    // before the end of a trace has occurred.
    decision_wait = "30s"

    // The following policies follow a logical OR pattern, meaning that if any of the
    // policies match, the trace will be kept. For logical AND, you can use the `and`
    // policy. Every span of a trace is examined by each policy in turn. A match will
    // cause a short-circuit.

    // This policy defines that traces that contain errors should be kept.
    policy {
        // The name of the policy can be used for logging purposes.
        name = "sample-erroring-traces"
        // The type must match the type of policy to be used, in this case examining
        // the status code of every span in the trace.
        type = "status_code"
        // This block determines the error codes that should match in order to keep
        // the trace, in this case the OpenTelemetry 'ERROR' code.
        status_code {
            status_codes = [ "ERROR" ]
        }
    }

    // This policy defines that only traces that are longer than 200ms in total
    // should be kept.
    policy {
        // The name of the policy can be used for logging purposes.
        name = "sample-long-traces"
        // The type must match the policy to be used, in this case the total latency
        // of the trace.
        type = "latency"
        // This block determines the total length of the trace in milliseconds.
        latency {
            threshold_ms = 200
        }
    }

    // The output block forwards the kept traces onto the batch processor, which
    // will marshall them for exporting to Tempo.
    output {
        traces = [otelcol.exporter.otlp.tempo.input]
    }
}

Examples for Grafana Agent static mode

For additional information, refer to the blog post, An introduction to trace sampling with Grafana Tempo and Grafana Agent.

Status code tail sampling policy

The following policy only samples traces where at least one span contains an OpenTelemetry Error status code.

traces:
  configs:
    - name: default
    ...
    tail_sampling:
      policies:
        - type: status_code
          status_code:
            status_codes:
              - ERROR

Span attribute tail sampling policy

The following policy only samples traces where the span attribute http.target does not contain the value /healthcheck or is prefixed with /metrics/.

traces:
   configs:
   - name: default
    tail_sampling:
      policies:
        - type: string_attribute
          string_attribute:
            key: http.target
            values:
              - ^\/(?:metrics\/.*|healthcheck)$
            enabled_regex_matching: true
            invert_match: true

And compound tail sampling policy

The following policy will only sample traces where all of the conditions for the sub-policies are met. In this case, it takes the prior two policies and will only sample traces where the span attribute http.target does not contain the value /healthcheck or is prefixed with /metrics/ and at least one of the spans of the trace contains an OpenTelemetry Error status code.

traces:
   configs:
   - name: default
    tail_sampling:
      policies:
       - type: and
            and_sub_policy:
            - name: and_tag_policy
              type: string_attribute
              string_attribute:
                key: http.target
                values:
                    - ^\/(?:metrics\/.*|healthcheck)$
                enabled_regex_matching: true
                invert_match: true
            - name: and_error_policy
              type: status_code
              status_code:
                status_codes:
                  - ERROR