Manual instrumentation of Go applications

Open source

Manual instrumentation of Go applications with OpenTelemetry Metrics SDK

To start recording and processing metrics from Go applications they have to be instrumented first. There are two types of instrumentation - automatic, and manual. To effectively instrument applications and produce high-quality metrics data you have to be familiar with both types of instrumentation. In this topic, we provide a high-level overview of how to manually instrument Go applications with the OpenTelemetry Go Metrics SDK.

Dependencies

To start instrumenting the Go application or library users should first install the OpenTelemetry Metrics API and OpenTelemetry Metrics SDK packages:

go get go.opentelemetry.io/otel/metric /
  go.opentelemetry.io/otel/sdk/metric

Go Metric SDK components

To start recording measurements for Go applications, users should configure all required SDK components, such as meter providers, meters, views, metrics readers, and exporters. Once the listed components are configured, users can start creating instruments that are used to record measurements. Let’s have a look at each component.

MeterProvider

MeterProvider handles the creation of Meters. All Meters created by an instance of MeterProvider will be associated with the same Resource, have the same Views, and pass produced metrics to configured Readers.

import (
    // ...
    "go.opentelemetry.io/otel/sdk/metric"
    // ...
)

//...

provider := metric.NewMeterProvider(
    metric.WithResource(...),
    metric.WithReader(...),
    metric.WithViews(...),
)

See the MeterProvider section in the OpenTelemetry specification for additional information.

Meter

Meter is responsible for creating Instruments. Meter instances are always created via MeterProvider. To create a Meter instance provide a name (use Go module name as the name of Meter), an optional version and schema URL arguments to the MeterProvider's Meter method.

import (
    // ...
    "go.opentelemetry.io/otel/metric"
    // ...
)

//...

meter := provider.Meter(
	"go.opentelemetry.io/otel/metric/example"
	metric.WithInstrumentationVersion(...),
	metric.WithSchemaURL(...), 
)

Once the Meter instance is initialized, it allows us to use Instruments factories that are used to create Instruments. The following factories are used to get synchronous or asynchronous Instruments of either integer or float data type:

Meter#AsyncInt64()
Meter#AsyncFloat64()
Meter#SyncInt64()
Meter#SyncFloat64()

The Meter instance is also used to register callback functions to record observations for asynchronous Instruments:

Meter#RegisterCallback([]instrument.Asynchronous{}, func(ctx context.Context) {})

See the Meter creation section for additional information.

Instruments

Instruments are used to report Measurements. Each Instrument has a name, an instrument kind, an optional description and a unit of measure. Each Instrument is associated with a Meter through which it was created.

Instruments can be synchronous or asynchronous. Synchronous instruments are supposed to be used in line with application logic. Measurements recorded by synchronous instruments can be associated with Context.

Asynchronous instruments allow users to register the callback function that will be invoked only on demand. Measurements recorded by asynchronous instruments cannot be associated with Context.

The Go Metrics SDK supports the following instrument kinds defined by the OpenTelemetry specification:

Instrument Kind	Sync	Async
Counter	✓	✓
UpDownCounter	✓	✓
Histogram	✓
Gauge		✓

To create an instance of Instrument choose the instrument factory that satisfies your requirements, then choose the instrument kind by invoking the corresponding factory method and provide required the arguments to create Instrument. For example, to create a synchronous integer Counter do the following:

import (
    // ...
    "go.opentelemetry.io/otel/metric/instrument"
    // ...
)

// ...

counter, err := meter.SyncInt64().Counter(
	"counter",
    instrument.WithDescription("..."),
    instrument.WithUnit("..."), 
)
if err != nil {
	// ...
}

For more detailed information on Instruments, please refer to the Instruments section of the OpenTelemetry specification.

Reporting Measurements

A Measurement represents a data point produced by instrumentation. Each Measurement contains a value, an optional set of attributes, and an associated Context if applicable.

Synchronous and asynchronous Instruments record measurements in different ways. As it was mentioned above, to record measurements with synchronous instruments we should inline it with application logic. Please see an example of the measurement recording by synchronous float UpDownCounter:

import (
    // ...
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/metric/instrument"
    // ...
)

// ...

ctx := context.Background()

counter, err := meter.SyncFloat64().UpDownCounter(
	"UpDownCounter",
    instrument.WithDescription("..."),
    instrument.WithUnit("..."), 
)
if err != nil {
	// ...
}

// ... application logic

attrs := []attribute.KeyValue{
    attribute.Key("...").String("..."),
}
counter.Add(ctx, 1, attrs...)

To observe measurements with asynchronous instruments it must be registered in a callback. Here is an example of using the integer Gauge to observe measurements of some rare or expensive operations:

import (
    // ...
    "go.opentelemetry.io/otel/metric/instrument"
    // ...
)

// ...

gauge, err := meter.AsyncInt64().Gauge(
    "gauge",
    instrument.WithDescription("..."),
    instrument.WithUnit("..."),
)
if err != nil {
    // ...
}

// ... application logic

err = meter.RegisterCallback([]instrument.Asynchronous{gauge}, func(ctx context.Context) {
	mesurement := // result of some rare operation
    gauge.Observe(ctx, mesurement)
})
if err != nil {
    // ...
}

Views

Views allow users to customize metrics that are produced by the SDK. It can be used to match metrics and modify their names, descriptions, aggregation types, and attributes. Also, it is used to filter out certain metrics produced by instrumentation. Views are usually most useful when working with existing instrumented libraries to modify or filter out produced metrics, therefore, we won’t cover Views in this manual.

For more information on Views, please refer to the View section of the OpenTelemetry specification.

MetricReader and MetricExporter

We talked about most of the SDK components and know how to record Measurements using Instruments. Now produced telemetry data has to be sent to some agent or to an observability platform. The SDK handles that with the help of MetricReader and MetricExporter components.

MetricReader is the interface between the Metrics SDK and an exporter. It collects recorded measurements from SDK and hands them over to the exporter either on demand or periodically. MetricReader is always associated with MetricExporter.

MetricExporter is a protocol specific component that supports sending or emitting produced by the SDK telemetry data.

OpenTelemetry Go Metrics SDK officially supports the following exporters:

See an example of the MeterProvided configuration that makes collected telemetry data available for pulling via the Prometheus exporter and sends it periodically to an upstream service that can receive data via HTTP in the OTLP format.

import (
    // ...
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
    "go.opentelemetry.io/otel/exporters/prometheus"
    "go.opentelemetry.io/otel/sdk/metric"
    // ...
)

// ...

// Push-based periodic OTLP exporter
otlpExporter, err := otlpmetrichttp.New(ctx)
if err != nil {
    // ...
}

// Pull-based Prometheus exporter
prometheusExporter, err := prometheus.New()
if err != nil {
    // ...
}

meterProvider := metric.NewMeterProvider(
    metric.WithResource(resources),
    metric.WithReader(metric.NewPeriodicReader(otlpExporter)),
    metric.WithReader(prometheusExporter),
)

Example

Let’s take a look at the example of a simple HTTP Go application with a single /health endpoint that is instrumented with Go Metrics SDK. We instrumented the application with two synchronous instruments: Counter and Histogram which record the number of incoming requests and their durations. Recorded observations will be periodically exported to the configured upstream service using the OTLP protocol via HTTP.

Instrumentation dependencies:

go get go.opentelemetry.io/otel/metric /
  go.opentelemetry.io/otel/sdk/metric /
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp

In this example telemetry data is exported in the OTLP format via HTTP, therefore, you can use anything that can receive the OTLP data format via HTTP as the
upstream service. Some options are:

OTLP endpoint in Grafana Cloud
OpenTelemetry Collector setup with the OTLP receiver (HTTP)
Grafana Agent

To configure the OTLP exporter, refer to Configure OpenTelemetry instrumentation via environment variables.

package main

import (
	"context"
	"log"
	"net/http"
	"time"

	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/metric"
	"go.opentelemetry.io/otel/metric/instrument"
	"go.opentelemetry.io/otel/metric/unit"
	sdk "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	semconv "go.opentelemetry.io/otel/semconv/v1.12.0"
)

func main() {
	ctx := context.Background()

	resources := resource.NewWithAttributes(
		semconv.SchemaURL,
		semconv.ServiceNameKey.String("service"),
		semconv.ServiceVersionKey.String("v0.0.0"),
	)

	// Instantiate the OTLP HTTP exporter
	exporter, err := otlpmetrichttp.New(ctx)
	if err != nil {
		log.Fatalln(err)
	}

	// Instantiate the OTLP HTTP exporter
	meterProvider := sdk.NewMeterProvider(
		sdk.WithResource(resources),
		sdk.WithReader(sdk.NewPeriodicReader(exporter)),
	)
	defer func() {
		err := meterProvider.Shutdown(context.Background())
		if err != nil {
			log.Fatalln(err)
		}
	}()

	// Create an instance on a meter for the given instrumentation scope
	meter := meterProvider.Meter(
		"github.com/.../example/manual-instrumentation",
		metric.WithInstrumentationVersion("v0.0.0"),
	)

	// Create two synchronous instruments: counter and histogram
	requestCount, err := meter.SyncInt64().Counter(
		"request_count",
		instrument.WithDescription("Incoming request count"),
		instrument.WithUnit("request"),
	)
	if err != nil {
		log.Fatalln(err)
	}
	requestDuration, err := meter.SyncFloat64().Histogram(
		"duration",
		instrument.WithDescription("Incoming end to end duration"),
		instrument.WithUnit(unit.Milliseconds),
	)
	if err != nil {
		log.Fatalln(err)
	}

	http.HandleFunc("/health", func(w http.ResponseWriter, req *http.Request) {
		requestStartTime := time.Now()

		_, _ = w.Write([]byte("UP"))

		elapsedTime := float64(time.Since(requestStartTime)) / float64(time.Millisecond)

		// Record measurements
		attrs := semconv.HTTPServerMetricAttributesFromHTTPRequest("", req)
		requestCount.Add(ctx, 1, attrs...)
		requestDuration.Record(ctx, elapsedTime, attrs...)
	})

	http.ListenAndServe(":8081", nil)
}