Skip to main content

Add distributed tracing for backend plugins

note

This feature requires at least Grafana 9.5.0, and your plugin needs to be built at least with grafana-plugins-sdk-go v0.157.0. If you run a plugin with tracing features on an older version of Grafana, tracing is disabled.

Distributed tracing allows backend plugin developers to create custom spans in their plugins, and send them to the same endpoint and with the same propagation format as the main Grafana instance. The tracing context is also propagated from the Grafana instance to the plugin, so the plugin's spans will be correlated to the correct trace.

Plugin configuration​

Plugin tracing must be enabled manually on a per-plugin basis, by specifying tracing = true in the plugin's config section:

[plugin.myorg-myplugin-datasource]
tracing = true

OpenTelemetry configuration in Grafana​

Grafana supports OpenTelemetry for distributed tracing. If Grafana is configured to use a deprecated tracing system (Jaeger or OpenTracing), then tracing is disabled in the plugin provided by the SDK and configured when calling datasource.Manage | app.Manage.

OpenTelemetry must be enabled and configured for the Grafana instance. Refer to the Grafana configuration documentation for more information.

Refer to the OpenTelemetry Go SDK for in-depth documentation about all the features provided by OpenTelemetry.

note

If tracing is disabled in Grafana, backend.DefaultTracer() returns a no-op tracer.

Implement tracing in your plugin​

note

Make sure you are using at least grafana-plugin-sdk-go v0.157.0. You can update with go get -u github.com/grafana/grafana-plugin-sdk-go.

Configure a global tracer​

When OpenTelemetry tracing is enabled on the main Grafana instance and tracing is enabled for a plugin, the OpenTelemetry endpoint address and propagation format is passed to the plugin during startup. These parameters are used to configure a global tracer.

  1. Use datasource.Manage or app.Manage to run your plugin to automatically configure the global tracer. Specify any custom attributes for the default tracer using CustomAttributes:

    func main() {
        if err := datasource.Manage("MY_PLUGIN_ID", plugin.NewDatasource, datasource.ManageOpts{
            TracingOpts: tracing.Opts{
                // Optional custom attributes attached to the tracer's resource.
                // The tracer will already have some SDK and runtime ones pre-populated.
                CustomAttributes: []attribute.KeyValue{
                    attribute.String("my_plugin.my_attribute", "custom value"),
                },
            },
        }); err != nil {
            log.DefaultLogger.Error(err.Error())
            os.Exit(1)
        }
    }

  2. Once you have configured tracing, use the global tracer like this:

    tracing.DefaultTracer()

    This returns an OpenTelemetry trace.Tracer for creating spans.

    Example:

    func (d *Datasource) query(ctx context.Context, pCtx backend.PluginContext, query backend.DataQuery) (backend.DataResponse, error) {
        ctx, span := tracing.DefaultTracer().Start(
            ctx,
            "query processing",
            trace.WithAttributes(
                attribute.String("query.ref_id", query.RefID),
                attribute.String("query.type", query.QueryType),
                attribute.Int64("query.max_data_points", query.MaxDataPoints),
                attribute.Int64("query.interval_ms", query.Interval.Milliseconds()),
                attribute.Int64("query.time_range.from", query.TimeRange.From.Unix()),
                attribute.Int64("query.time_range.to", query.TimeRange.To.Unix()),
            ),
        )
        defer span.End()
        log.DefaultLogger.Debug("query", "traceID", trace.SpanContextFromContext(ctx).TraceID())

        // ...
    }

Tracing gRPC calls​

When tracing is enabled, a new span is created automatically for each gRPC call (QueryData, CheckHealth, etc.), both on Grafana's side and on the plugin's side. The plugin SDK also injects the trace context into the context.Context that is passed to those methods.

You can retrieve the trace.SpanContext with tracing.SpanContextFromContext by passing the original context.Context to it:

func (d *Datasource) query(ctx context.Context, pCtx backend.PluginContext, query backend.DataQuery) (backend.DataResponse, error) {
    spanCtx := trace.SpanContextFromContext(ctx)
    traceID := spanCtx.TraceID()

    // ...
}

Tracing HTTP requests​

When tracing is enabled, a TracingMiddleware is also added to the default middleware stack to all HTTP clients created using the httpclient.New or httpclient.NewProvider, unless you specify custom middleware. This middleware creates spans for each outgoing HTTP request and provides some useful attributes and events related to the request's lifecycle.

Plugin example​

Refer to the datasource-http-backend plugin example for a complete example of a plugin with full distributed tracing support.