Instrument for distributed tracing on Grafana Labs

About instrumentation

Fri, 03 Apr 2026 18:14:38 +0000

About instrumentation

Instrumentation is the act of modifying the source code of a service to emit span information tied to a common trace ID. Traces themselves are a metaobject, comprised of nothing but spans that hold the same ID.

Instrumentation methods

To add instrumentation, the code for a service uses a Software Development Kit (SDK) which supplies language-specific libraries that allow the:

Creation of a new trace, starting with a new root span.
Addition of new spans, that are siblings of children of pre-existing spans.
Addition of span attributes to add contextual information to each span, as well as span links and events.
Closure of spans when a unit of work is complete.

Adding code to carry out these operations is known as manual instrumentation, as it requires manual intervention by an engineer to write code to deal with traces, as well as to determine where in the code traces/spans should start, the attributes and other data that should be attached to spans, and where traces/spans should end.

There is an alternative/companion to manual instrumentation, auto-instrumentation. Auto-instrumentation is the act of allowing a tracing SDK to determine where traces/spans should start, what information should be added to spans, and where traces/spans should stop. Essentially, manual instrumentation is pre-packaged for a large number of popular frameworks and libraries which are used inside a service’s code, and it’s these libraries/frameworks that are actually emitting spans for a trace.

Zero-code instrumentation uses eBPF technology to instrument applications without code changes. Grafana Beyla is an example of a zero-code instrumentation tool.

Hybrid instrumentation combines auto and manual instrumentation, using automatic for most code and manual for custom tracing logic.

These libraries usually include those dealing with networking, so for example a request coming into a service might be via an auto-instrumented HTTP library, which would then start a trace until it sent a response back via HTTP to the requester. Along the course of the request, the service might use other libraries that process data, and if they’re also auto-instrumented then new spans are generated for the trace that include suitable attributes.

This produces a trace that can be very useful, but ultimately doesn’t include any spans that are specific to service code that isn’t executing library commands. Because of this, the best traced services are usually those that include a mixture of auto and manual instrumentation.

Traces usually exist in relation to a request made to an application, and spans are generated when new units of work occur whilst processing that request. A trace usually ends when a response to the request is sent.

The OpenTelemetry SDK is Grafana’s recommended solution for instrumenting, and it supports a myriad of modern languages, detailed in the linked page.

Additionally, instrumentation SDKs usually emit span information to a local receiver/collector. Examples of these are Grafana Alloy and the OpenTelemetry Collector. These receivers can then carry out additional processing on the traces (such as tail sampling, span batching, process filtering, etc.) to modify a trace and its spans before they’re sent to a final destination, usually a tracing backend like Grafana Tempo.

Propagation

Modern applications are not usually not the monolithic applications of old, where single binaries contained all of the code, and multiple instances of the monoliths were run to ‘scale’ to increased traffic. Modern applications are built using a large number of microservices, each one specifically designed to handle specific portions of a request/data flow.

Traces are (usually) tied to a request/response model, being initiated when a request is received by application and ending when a response is returned. Each request could flow through a number of different services (eg. an API service, a caching service, a processing service, a database service, etc.), and therefore to ensure that the trace includes every service that the data is routed through there needs to be some way of sending the details about a trace with the data that’s sent to each service.

This is called trace propagation, and the actual mechanics of it alter depending on the way that data is passed around services. The majority of services in the modern application world use REST (via HTTP) or protobufs (via gRPC, which is itself handled by HTTP/2). Because a trace is just a metaobject comprised of spans, all that needs to happen to continue a trace from one service to another is the addition of a trace ID and the last current span ID to the data send to a downstream service.

Many instrumentation SDKs, include propagators that can handle a number of different transport protocols, and when matched with appropriate auto-instrumentation mean that an engineer doesn’t even have to consider writing any code to inject the downstream data with the trace information, it gets handled automatically (for example in the case of HTTP, headers are included that specifically deal with trace information that gets extracted by the OpenTelemetry SDK in the downstream service to carry on a trace).

The W3C Trace Context is the default way of sending propagation information in OpenTelemetry SDKs. However, the SDK also allows you to provide custom propagators to it instead, which means that if an application uses a custom protocol or networking scheme, engineers can write propagators and extractors to handle these as well.

Best practices for traces

Fri, 03 Apr 2026 18:14:38 +0000

Best practices for traces

This page provides some general best practices for tracing.

Span and resource attributes

Traces are built from spans, which denote units of work such as a call to, or from, an upstream service. Spans are constructed primarily of span and resource attributes. Spans also have a hierarchy, where parent spans can have children or siblings.

In the screenshot below, the left side of the screen (1) shows the list of results for the query. The right side (2) lists each span that makes up the selected trace.

A span attribute is a key/value pair that provides context for its span. For example, if the span deals with calling another service via HTTP, an attribute could include the HTTP URL (maybe as the span attribute key http.url) and the HTTP status code returned (as the span attribute http.status_code). Span attributes can consist of varying, non-null types.

Unlike a span attribute, a resource attribute is a key/value pair that describes the context of how the span was collected. Generally, these attributes describe the process that created the span. This could be a set of resource attributes concerning a Kubernetes cluster, in which case you may see resource attributes, for example: k8s.namespace, k8s.container_name, and k8s.cluster. These can also include information on the libraries that were used to instrument the spans for a trace, or any other infrastructure information.

For more information, read the Attribute and Resource sections in the OpenTelemetry specification.

Naming conventions for span and resource attributes

The OpenTelemetry project defines a number of semantic conventions for attributes, which can help you to determine which attributes are most important to include in your spans. These conventions provide a common vocabulary for describing different types of entities, which can help to ensure that your data is consistent and meaningful.

When naming attributes, use consistent, nested namespaces to ensures that attribute keys are obvious to anyone observing the spans of a trace and that common attributes can be shared by spans. Using our example from above, the http prefix of the attribute is the namespace, and url and status_code are keys within that namespace. Attributes can also be nested, for example http.url.protocol might be HTTP or HTTPS, whereas http.url.path might be /api/v1/query.

For more details around semantic naming conventions, refer to the Recommendations for OpenTelemetry Authors and OpenTelemetry Semantic Conventions documentation.

Some third-party libraries provide auto-instrumentation that generate span and span attributes when included in a source base.

For more information about instrumenting your app for tracing, refer to the Instrument for distributed tracing documentation.

How many attributes should spans have?

The decision of how many attributes to include in your spans is up to you. There is no hard and fast rule. Keep the number of attributes to a minimum, as each attribute adds overhead to the tracing system. In Grafana Cloud, this results in higher tracing costs.

Only include attributes that are relevant to the operation that the span represents. For example, if you are tracing an HTTP request, you might include attributes such as the request method, the URL, and the response status code.

If you are unsure whether or not to include an attribute, it is always better to err on the side of caution and leave it out. You can always add additional attributes later if you need them.

In general, consider the following guidelines:

Don’t include metrics or logs as attributes in your spans.
Don’t use redundant attributes.
When determining which attributes to add, consider an application’s service flow, and execution in the context of only the current span.

The OpenTelemetry project doesn’t specify a maximum number of attributes that a span can have. However, the default limits for the number of attributes per span is 128 entries, so you will have to adjust that. There are also default limits on attribute value and name character lengths.

Determining where to add spans

When instrumenting, determine the smallest piece of work that you need to observe in a trace to be of value to ensure that you don’t over (or under) instrument.

Creating a new span for any work that has a relatively significant duration allows the observation of a trace to immediately show where significant amounts of time are spent during the processing of a request into your application or system.

For example, adding a span for a call to another services (either instrumented or not) may take an unknown amount of time to complete, and therefore being able to separate this work shows when services are taking longer than expected.

Adding a span for a piece of work that might call many other functions in a loop is a good signal of how long that loop is taking (you might add a span attribute that counts how many time the loop runs to determine if the duration is acceptable). However, adding a span for each method or function call in that loop might not, as it might produce hundreds or thousands of worthless spans.

Span length

While there are some (high) default limits to the length that a span (and by definition, the traces they belong to) can be, these can be adjusted by these configurations. Traces that include a large number of spans and/or long-running spans can have an impact on the time taken to query them once stored. Cloud Traces users should contact Grafana Support to modify overrides.

For long-running spans and traces, the best way to see this impact on requests is to send a few test cases and see what the performance looks like and to evaluate the trace size.

From there, you can modify the configuration for Tempo or determine ways to re-architect how the trace is produced.

You can consider breaking up the spans in several ways:

Decompose the query
- For example, if a complex SQL query involves multiple operations (for example, uses joins, subqueries, or unions), consider creating separate spans for each significant operation.
Improve granulation of long-running spans
- For long-running operations, you could create a new span for every predetermined interval of execution time.
  Note
  This requires time-based tracking in your application’s code and is more complex to implement.
Use span linking
- Should data flow hit bottlenecks where further operations on that data might be batched at a later time, the use of span links can help keep traces constrained to an acceptable time range, while sharing context with other traces that work on the same data. This can also improve the readability of traces.

Set up instrumentation

Fri, 03 Apr 2026 18:14:38 +0000

Set up instrumentation

Client instrumentation is the first building block to a functioning distributed tracing visualization pipeline. Client instrumentation is the process of adding instrumentation points in the application that create and offload spans.

When you set up instrumentation, you:

Choose an instrumentation method to use with your application
Instrument your application to generate traces

After you set up instrumentation, you can set up a collector to receive traces from your application. Refer to About instrumentation for more information about instrumentation and how it works.

Choose an instrumentation method

When sending traces to Tempo, you can choose between four methods:

Auto-instrumentation applies instrumentation automatically using agents or middleware, without code changes.
Zero-code instrumentation, which uses eBPF technology to instrument applications without code changes. Grafana Beyla is an example of a zero-code instrumentation tool.
Manual instrumentation involves adding code to create spans and traces, giving full control over collected data.
Hybrid instrumentation, which combines auto and manual instrumentation, using automatic for most code and manual for custom tracing logic.

Refer to About instrumentation for more information.

If you are using OTel or Alloy, refer to Instrument an application with OpenTelemetry for more information. These instructions are specific to Grafana Cloud, but can be adapted for self-hosted Tempo.

Instrument your app

Most of the popular client instrumentation frameworks have SDKs in the most commonly used programming languages. You should pick one according to your application needs.

OpenTelemetry has the most active development in the community and may be a better long-term choice.

Popular instrumentation frameworks include:

Instrument using OpenTelemetry

A collection of tools, APIs, and SDKs, OpenTelemetry helps engineers instrument, generate, collect, and export telemetry data such as metrics, logs, and traces, to analyze software performance and behavior. For more information refer to OpenTelemetry overview.

If you are using OTel with Grafana Cloud, refer to Instrument an application with OpenTelemetry for more information.

Use OpenTelemetry auto-instrumentation frameworks

OpenTelemetry provides auto-instrumentation agents and libraries of Java, .NET, Python, Go, and JavaScript applications, among others. For more information, refer for the OpenTelemetry Instrumentation documentation.

These libraries capture telemetry information from a client application with minimal manual instrumentation of the codebase.

Note
Jaeger client libraries have been deprecated. For more information, refer to the Deprecating Jaeger clients article. Jaeger recommends using OpenTelemetry SDKs.

Additional OTel resources

Instrument with Zipkin auto-instrumentation

Zipkin is a distributed tracing system that helps gather timing data needed to troubleshoot latency problems in microservice architectures.

Refer to the Zipkin Language Specific Instrumentation documentation for more information.

If you are using Zipkin with Alloy, refer to the Zipkin receiver, otelcol.receiver.zipkin documentation.

In addition, you can use Zipkin to instrument a library, refer to Instrumenting a library with Zipkin

Within Grafana, you can also use these Zipkin specific features:

Instrument with Grafana Beyla

Grafana Beyla is an eBPF-based application zero-code instrumentation tool to easily get started with Application Observability. Beyla uses eBPF to automatically inspect application executables and the OS networking layer, and capture trace spans related to web transactions and Rate Errors Duration (RED) metrics for Linux HTTP/S and gRPC services. All data capture occurs without any modifications to application code or configuration.

Refer to Set up Beyla for information about how to instrument using Beyla.

Note
While Beyla is a good choice for metrics, span metrics, and service graphs, if you choose to use it for tracing, check the Beyla documentation to verify that it meets your tracing needs.

Community resources

Next steps

After you set up instrumentation, you can set up a collector to receive traces from your application.

Set up your collector

Fri, 03 Apr 2026 18:14:38 +0000

Set up your collector

You can send data from your application using Grafana Alloy or OpenTelemetry Collector (OTel) collectors.

Grafana Alloy is a vendor-neutral distribution of the OpenTelemetry (OTel) Collector. Alloy uniquely combines the very best OSS observability signals in the community.

The OpenTelemetry Collector is a vendor-agnostic way to receive, process, and export telemetry data.

The collector is a component that runs alongside your application and periodically gathers tracing data from it. This method is suitable when you want to collect tracing from applications without modifying their source code.

Here’s how it works:

Install and configure the collector on the same machine or container where your application is running.
The collector periodically retrieves your application’s performance tracing data, regardless of the language or technology stack your application is using.
The captured traces are then sent to the Tempo server for storage and analysis.

Using a collector provides a hassle-free option, especially when dealing with multiple applications or microservices, allowing you to centralize the tracing process without changing your application’s codebase.

Use Alloy

Grafana Labs maintains and supports Alloy, which packages various upstream OpenTelemetry Collector components. Alloy provides stability, support, and integration with Grafana Labs products.

Refer to the Alloy documentation for information about Alloy and it’s tracing capabilities.

Refer to Collect and forward data with Alloy for examples of collecting data.

Use the OpenTelemetry Collector

The OpenTelemetry project maintainers and the Cloud Native Computing Foundation maintain the upstream OpenTelemetry Collector. This is a community-supported project.

Refer to the Install the Collector documentation for instructions on installation.

Refer to the OpenTelemetry Collector documentation to use the upstream OpenTelemetry Collector with Grafana Labs products.