This is documentation for the next version of Tempo. For the latest stable release, go to the latest version.
Automatic logging: Trace discovery through logs
Caution
Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.
Running instrumented distributed systems is a very powerful way to gain understanding over a system, but it brings its own challenges. One of them is discovering which traces exist.
In the beginning of Tempo, querying for a trace was only possible if you knew
the ID of the trace you were looking for. One solution was automatic logging.
Automatic logging provides an easy and fast way of discovering trace IDs
through log messages. Well-formatted log lines are written to a Loki instance
or to stdout
for each span, root, or process that passes through the tracing
pipeline. This allows for automatically building a mechanism for trace
discovery. On top of that, you can also get metrics from traces using Loki, and
allow quickly jumping from a log message to the trace view in Grafana.
While this approach is useful, it isn’t as powerful as TraceQL. If you are here because you know you want to log the trace ID, to enable jumping from logs to traces, then read on!
If you want to query the system directly, read the TraceQL documentation.
Configuration
For high throughput systems, logging for every span may generate too much volume. In such cases, logging per root span or process is recommended.
Automatic logging searches for a given set of attributes in the spans and logs them as key-value pairs. This allows searching by those key-value pairs in Loki.
Before you begin
Note
Grafana Alloy provides tooling to convert your Agent Static or Flow configuration files into a format that can be used by Alloy.
For more information, refer to Migrate to Alloy.
To configure automatic logging, you need to select your preferred backend and the trace data to log.
To see all the available configuration options, refer to the configuration reference.
This simple example logs trace roots to stdout
and is a good way to get started using automatic logging:
traces:
configs:
- name: default
...
automatic_logging:
backend: stdout
roots: true
This example pushes logs directly to a Loki instance also configured in the same Grafana Agent.
traces:
configs:
- name: default
...
automatic_logging:
backend: logs_instance
logs_instance_name: default
roots: true