Documentation Grafana Cloud Monitor applications Application Observability Instrument an application Java (Scala, Kotlin)

Instrument a JVM application

The Grafana OpenTelemetry distribution for Java is our pre-configured and pre-packaged bundle of OpenTelemetry Java components, optimized for Grafana Cloud Application Observability.

Recommended set up

Grafana Labs recommends you set up instrumentation through the Grafana Cloud connection tiles. This opinionated approach includes all the binaries, configuration, and connection parameters you need to set up OpenTelemetry and Application Observability.

Advanced manual set up

For advanced use cases, follow the rest of this documentation to manually set up and configure OpenTelemetry instrumentation for JVM applications.

Before you begin

Ensure you have a Java development environment and application using using JDK 8 or higher.

Download the distribution

Download the latest grafana-opentelemetry-java.jar from the GitHub repository release page.

Instrument an application

The distribution automatically instruments JVM applications. Run your application with the -javaagent flag and path to the grafana-opentelemetry-java.jar, for example:

sh Copy

java -javaagent:/path/to/grafana-opentelemetry-java.jar -jar myapp.jar

Example application

The Grafana OpenTelemetry Java Spring PetClinic application is complete example of a Java application instrumented with our distribution.

Configuration

The Grafana OpenTelemetry distribution for Java is configurable supports all of the upstream Java agent configuration.

For our distribution, we’ve configured all exporters, including the logs exporter, to otlp.

Data saver

Application Observability only uses a selection of the generated OpenTelemetry metrics. To exclude unused metrics and reduce your Grafana Cloud cost, enable data saved mode by setting the following environment variable:

shell Copy

export GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true

Export to console

You can configure the exporters to output data to the console by setting environment variables. The following configuration examples export data to the Grafana Cloud otlp endpoint and the console:

To search for trace spans, look for lines that include LoggingSpanExporter:

log Copy

INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'WebController.withSpan' : 2337335133908c9ce6e0dfc7bda74d7c 8bfef4eaac83e8cb INTERNAL [tracer: io.opentelemetry.opentelemetry-extension-annotations-1.0:1.32.0-alpha] AttributesMap{data={thread.id=32, code.namespace=io.opentelemetry.smoketest.springboot.controller.WebController, code.function=withSpan, thread.name=http-nio-8080-exec-1}, capacity=128, totalAddedValues=4}

To search for metrics, look for lines that include LoggingMetricExporter:

log Copy

INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.21.0, attributes={container.id="048b9982e0b98cdc5579334bb1decc157ed1ebc23f391ebe306898898ec32fa4", host.arch="amd64", host.name="048b9982e0b9", os.description="Linux 6.2.0-39-generic", os.type="linux", process.command_line="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java -javaagent:/opentelemetry-javaagent.jar -Dgrafana.otel.use-tested-instrumentations=true io.opentelemetry.smoketest.springboot.SpringbootApplication", process.executable.path="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java", process.pid=1, process.runtime.description="BellSoft OpenJDK 64-Bit Server VM 25.312-b07", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_312-b07", service.instance.id="8231ca95-e9aa-474a-bd98-88349a9942ad", service.name="unknown_service:java", telemetry.auto.version="1.32.0", telemetry.distro.name="grafana-opentelemetry-java", telemetry.distro.version="0.32.0-beta.1", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.runtime-telemetry-java8, version=1.32.0-alpha, schemaUrl=null, attributes={}}, name=jvm.cpu.count, description=Number of processors available to the Java virtual machine., unit={cpu}, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1704964347622000000, epochNanos=1704964351627000000, attributes={}, value=12, exemplars=[]}], monotonic=false, aggregationTemporality=CUMULATIVE}}

To search for logs, look for lines that include [scopeInfo: and the log body.

log Copy

10:12:34.031 [docker-java-stream-636643381] INFO  c.g.extensions.smoketest.SmokeTest - STDOUT: 2024-01-11T09:12:34.03Z INFO 'HTTP request received' : 2337335133908c9ce6e0dfc7bda74d7c 50d689015fd0a33c [scopeInfo: io.opentelemetry.smoketest.springboot.controller.WebController:] {thread.id=32, thread.name="http-nio-8080-exec-1"}

Troubleshoot your instrumentation

If Application Observability doesn’t list your services, these are some of the most common causes:

Generate traffic

Make a few requests to the service to send data to Grafana Cloud.

Wait for data generation

It can take up to 5 minutes for your data to be visible in Application Observability.

Inspect logs

First, look for errors from your application, or in your Docker or Kubernetes logs.

If there are errors from sending telemetry data, one of the configuration parameters could be incorrect.

A 5xx response code means that there’s something wrong with the Grafana Cloud OTLP endpoint.

Log agent data:

If there are no errors in the logs, export data to the console to check the instrumentation is sending data. If it’s not sending data, look for the -javaagent command line parameter to check if you’ve loaded the Java agent.

Log exporter data:

If the application is sending data to an OpenTelemetry Collector, log the data from the collector and check if there errors forwarding the telemetry data to Grafana Cloud.

Enable internal debug logging for the agent:

To inspect the internals of the Java agent, enable the Java agent’s internal debug logging:

shell Copy

export OTEL_JAVAAGENT_DEBUG=true

Disable Java agent:

Lastly, disable the Java agent to check the application runs successfully without instrumentation:

shell Copy

export OTEL_JAVAAGENT_ENABLED=false

Next steps

1. Create a free Grafana Cloud account.
2. For a local development and testing, send data to the Grafana Cloud OTLP endpoint.
3. For production, set up an OpenTelemetry Collector.
4. Observe your services in Application Observability.

Resources

1. Grafana OpenTelemetry distribution for Java on GitHub
2. upstream OpenTelemetry SDK on GitHub
3. upstream OpenTelemetry SDK configuration docs