Documentation OpenTelemetry Instrument an application Grafana Java Agent

Instrument a JVM application

The Grafana OpenTelemetry distribution for Java provides a pre-configured and pre-packaged bundle of OpenTelemetry Java components, optimized for Grafana Cloud Application Observability.

Recommended setup for Grafana Cloud

Grafana Labs recommends that you set up OpenTelemetry components, including instrumentation and an OpenTelemetry Collector distribution, using one of the Grafana Cloud setup guides.

These opinionated guides make it easy to get started. They include all the binaries, configuration, and connection parameters you need to set up OpenTelemetry for Grafana Cloud.

Before you begin

Ensure you have a Java development environment and an application using JDK 8 or higher.

Download the distribution

Download the latest grafana-opentelemetry-java.jar from the GitHub repository release page.

Run your application

The distribution automatically instruments JVM applications. Start your application with the -javaagent flag and the path to the grafana-opentelemetry-java.jar, for example:

sh Copy

OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
java -javaagent:<path>/grafana-opentelemetry-java.jar -jar <my-app>.jar

Replace variables in <...> with values specific to your application and backend. For more details, see the OpenTelemetry service attributes documentation.

• <name>: your service name, for example, shoppingcart
• <namespace>: a string to group related services, such as ecommerce for a team that manages several services
• <environment>: the deployment environment, for example, the production deployment tier

Some backends, such as Grafana Cloud, also require you to set authentication headers in the OTEL_EXPORTER_OTLP_HEADERS environment variable.

Test your instrumentation

To verify successful instrumentation, follow these steps:

1. Configure your application to export to console.
2. Run your application.
3. Generate test traffic.
4. Check that metrics and logs appear in the console.
5. Disable console exporting.

Example application

The Grafana OpenTelemetry Java Spring PetClinic application demonstrates a complete example of a Java application instrumented with our distribution.

Configuration

You can configure the Grafana OpenTelemetry distribution for Java, which supports all upstream Java agent configuration.

Our distribution configures all exporters, including the logs exporter, to otlp by default.

Data saver

Application Observability uses only a selection of the generated OpenTelemetry metrics. To exclude unused metrics and reduce your Grafana Cloud cost, enable data saver mode by setting the following environment variable:

shell Copy

export GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true

Export to console

You can configure the exporters to output data to the console by setting environment variables. The following configuration examples export data to the Grafana Cloud otlp endpoint and the console:

To find trace spans, look for lines that include LoggingSpanExporter:

log Copy

INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'WebController.withSpan' : 2337335133908c9ce6e0dfc7bda74d7c 8bfef4eaac83e8cb INTERNAL [tracer: io.opentelemetry.opentelemetry-extension-annotations-1.0:1.32.0-alpha] AttributesMap{data={thread.id=32, code.namespace=io.opentelemetry.smoketest.springboot.controller.WebController, code.function=withSpan, thread.name=http-nio-8080-exec-1}, capacity=128, totalAddedValues=4}

To find metrics, look for lines that include LoggingMetricExporter:

log Copy

INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.21.0, attributes={container.id="048b9982e0b98cdc5579334bb1decc157ed1ebc23f391ebe306898898ec32fa4", host.arch="amd64", host.name="048b9982e0b9", os.description="Linux 6.2.0-39-generic", os.type="linux", process.command_line="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java -javaagent:/opentelemetry-javaagent.jar -Dgrafana.otel.use-tested-instrumentations=true io.opentelemetry.smoketest.springboot.SpringbootApplication", process.executable.path="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java", process.pid=1, process.runtime.description="BellSoft OpenJDK 64-Bit Server VM 25.312-b07", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_312-b07", service.instance.id="8231ca95-e9aa-474a-bd98-88349a9942ad", service.name="unknown_service:java", telemetry.auto.version="1.32.0", telemetry.distro.name="grafana-opentelemetry-java", telemetry.distro.version="0.32.0-beta.1", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.runtime-telemetry-java8, version=1.32.0-alpha, schemaUrl=null, attributes={}}, name=jvm.cpu.count, description=Number of processors available to the Java virtual machine., unit={cpu}, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1704964347622000000, epochNanos=1704964351627000000, attributes={}, value=12, exemplars=[]}], monotonic=false, aggregationTemporality=CUMULATIVE}}

To find logs, look for lines that include [scopeInfo: and the log body.

log Copy

10:12:34.031 [docker-java-stream-636643381] INFO  c.g.extensions.smoketest.SmokeTest - STDOUT: 2024-01-11T09:12:34.03Z INFO 'HTTP request received' : 2337335133908c9ce6e0dfc7bda74d7c 50d689015fd0a33c [scopeInfo: io.opentelemetry.smoketest.springboot.controller.WebController:] {thread.id=32, thread.name="http-nio-8080-exec-1"}

Troubleshoot your instrumentation

If Application Observability doesn’t list your services, check these common causes:

Generate traffic

Send a few requests to the service to generate data for Grafana Cloud.

Wait for data generation

It can take up to 5 minutes for your data to appear in Application Observability.

Inspect logs

First, look for errors from your application, or in your Docker or Kubernetes logs.

If you see errors when sending telemetry data, one of the configuration parameters might be incorrect.

A 5xx response code indicates an issue with the Grafana Cloud OTLP endpoint.

Log agent data:

If you don’t see errors in the logs, export data to the console to check if the instrumentation is sending data. If it isn’t, check for the -javaagent command line parameter to confirm you loaded the Java agent.

Log exporter data:

If the application sends data to an OpenTelemetry Collector, log the data from the collector and check for errors forwarding telemetry data to Grafana Cloud.

Enable internal debug logging for the agent:

To inspect the internals of the Java agent, enable the Java agent’s internal debug logging:

shell Copy

export OTEL_JAVAAGENT_DEBUG=true

Disable Java agent:

Finally, disable the Java agent to check if the application runs successfully without instrumentation:

shell Copy

export OTEL_JAVAAGENT_ENABLED=false

Migrate to the upstream

The Grafana OpenTelemetry Java distribution is fully compatible with the upstream OpenTelemetry auto-instrumentation SDK. To migrate to the upstream SDK, follow these steps:

1. Remove the grafana prefix from all configuration environment variables or system properties.
2. To replicate the data saver features, use the OpenTelemetry Collector to filter metrics.
3. The OpenTelemetry Java SDK does not bundle the resource detectors for Kubernetes. Use the OpenTelemetry Collector to add Kubernetes resource attributes.
4. Add the following environment variables required by Application Observability:

shell Copy

export OTEL_INSTRUMENTATION_MICROMETER_BASE_TIME_UNIT=s
export OTEL_INSTRUMENTATION_LOG4J_APPENDER_EXPERIMENTAL_LOG_ATTRIBUTES=true
export OTEL_INSTRUMENTATION_LOGBACK_APPENDER_EXPERIMENTAL_LOG_ATTRIBUTES=true

Resources

1. Grafana OpenTelemetry distribution for Java on GitHub
2. upstream OpenTelemetry SDK on GitHub
3. upstream OpenTelemetry SDK configuration docs