Menu
Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Monitor applicationsbreadcrumb arrow Application Observabilitybreadcrumb arrow Instrument your applicationbreadcrumb arrow JVM (Java, Scala, or Kotlin)
Grafana Cloud

Instrument a JVM application

The recommended way to instrument your JVM application (Java, Scala, or Kotlin) and send data to Grafana Cloud is to use the Grafana Cloud connection tiles.

If you have advanced setup and configuration needs that the connection tiles don’t cater for you can manually set up instrumentation and ingestion.

For most use cases, Grafana Labs recommends to use the Grafana Cloud integration tiles to set up and send OpenTelemetry data to Grafana Cloud. This opinionated approach includes all the binaries, connection parameters, and configuration snippets you need to set up OpenTelemetry and Application Observability for JVM applications.

Integration tiles are available to guide you to set up a production architecture with:

  • Grafana Labs distribution of the OpenTelemetry SDK for Java (JVM) with Grafana Alloy
  • Grafana Beyla eBPF auto-instrumentation and Grafana Alloy
  • OpenTelemetry (OTLP) with Grafana Alloy, for an application already instrumented with OpenTelemetry

To use the recommended set up for Java, refer to the Grafana Cloud, Instrument and send data documentation.

Advanced manual setup

For advanced use cases you can manual set up the packaged release of the Grafana OpenTelemetry distribution for Java and instrument your JVM application (Java, Scala, or Kotlin) for Grafana Cloud Application Observability. The distribution is a pre-configured bundle of open source OpenTelemetry Java components, optimized for Grafana Cloud Application Observability.

Before you begin

The is the first step to get telemetry data into Application Observability, you need:

  1. A Java development environment.
  2. A Java, Scala, or Kotlin application to instrument using JDK 8 or higher.

Install the SDK

Download the latest JAR release of the instrumentation agent from the GitHub repository release page.

Instrumentation configuration

Grafana OpenTelemetry distribution for Java is configurable with all the system properties or environment variables from the upstream OpenTelemetry SDK auto-configuration.

The distribution sets all exporters to otlp by default, including the logs exporter.

Test your instrumentation

The SDK automatically instruments a JVM based application. Run your application by supplying the -javaagent flag with a path to the SDK jar. For example, to instrument your application run:

sh 
java -javaagent:/path/to/grafana-opentelemetry-java.jar -jar myapp.jar

To test if you successfully instrumented your application and are producing telemetry data, consult the Troubleshoot your instrumentation section of the documentation to find out how to log data.

Example application

See the Grafana OpenTelemetry Java Spring PetClinic configuration for a complete example setup.

Configuration

Grafana OpenTelemetry distribution for Java is configurable with all the system properties or environment variables from the upstream OpenTelemetry SDK auto-configuration.

The distribution sets all exporters to otlp by default, including the logs exporter.

Data saver

Application Observability uses a specific selection of metrics and ingesting all OpenTelemetry metrics can increase cost.

If you want the distribution to only send metrics that Application Observability uses and reduce Metrics costs in Grafana Cloud, set the following environment variable:

shell 
export GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true

Note

If you have enabled data saver and want to send manually created metrics, set the metric name to application.

Troubleshoot your instrumentation

If Application Observability doesn’t list your services, these are some of the most common causes:

No traffic

Make a few requests to the service to send data to Grafana Cloud.

Be patient

It can take up to 5 minutes until the data is visible in Application Observability.

Look for errors

Look for errors - either on the console or in Docker or Kubernetes logs, using Application Observability logs doesn’t make sense in this case.

If there are errors sending telemetry data, one of the parameters is usually wrong.

A 5xx response code means that there’s something wrong with the Grafana Cloud OTLP endpoint.

Log agent data

If there are no errors in the logs, enable logging to check the application is sending data.

If the application isn’t sending data, look for the -javaagent command line parameter to check if you loaded the Java agent.

Log exporter data

If the application is sending data to an data collector, Grafana Alloy or OpenTelemetry Collector, log the data from the data collector and check there are no errors forwarding the telemetry data to Grafana Cloud.

Java agent logging

To inspect the internals of the Java agent, enable the Java agent’s internal debug logging:

shell 
export OTEL_JAVAAGENT_DEBUG=true

Note

Only enable the Java agent debug logging when needed. The logs are extremely verbose and impact application performance.

Disable Java agent

Lastly, disable the Java agent to check the application runs successfully without instrumentation:

shell 
export OTEL_JAVAAGENT_ENABLED=false

OTLP debug logging

You can configure the OTLP debug logging can via environment variables, previous versions use logging instead of console:

ConfigurationResult
OTEL_TRACES_EXPORTER=otlp,consoleLog all traces and send them to Tempo.
OTEL_METRICS_EXPORTER=otlp,consoleLog all metrics and send them to Mimir.
OTEL_LOGS_EXPORTER=otlp,consoleLog all logs and send them to Loki.

Note

You should disable debug logging when it isn’t needed as it produces many logs.

This produce log output like this:

Spans - look for LoggingSpanExporter:

log 
INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'WebController.withSpan' : 2337335133908c9ce6e0dfc7bda74d7c 8bfef4eaac83e8cb INTERNAL [tracer: io.opentelemetry.opentelemetry-extension-annotations-1.0:1.32.0-alpha] AttributesMap{data={thread.id=32, code.namespace=io.opentelemetry.smoketest.springboot.controller.WebController, code.function=withSpan, thread.name=http-nio-8080-exec-1}, capacity=128, totalAddedValues=4}

Metrics - look for LoggingMetricExporter:

log 
INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.21.0, attributes={container.id="048b9982e0b98cdc5579334bb1decc157ed1ebc23f391ebe306898898ec32fa4", host.arch="amd64", host.name="048b9982e0b9", os.description="Linux 6.2.0-39-generic", os.type="linux", process.command_line="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java -javaagent:/opentelemetry-javaagent.jar -Dgrafana.otel.use-tested-instrumentations=true io.opentelemetry.smoketest.springboot.SpringbootApplication", process.executable.path="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java", process.pid=1, process.runtime.description="BellSoft OpenJDK 64-Bit Server VM 25.312-b07", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_312-b07", service.instance.id="8231ca95-e9aa-474a-bd98-88349a9942ad", service.name="unknown_service:java", telemetry.auto.version="1.32.0", telemetry.distro.name="grafana-opentelemetry-java", telemetry.distro.version="0.32.0-beta.1", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.runtime-telemetry-java8, version=1.32.0-alpha, schemaUrl=null, attributes={}}, name=jvm.cpu.count, description=Number of processors available to the Java virtual machine., unit={cpu}, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1704964347622000000, epochNanos=1704964351627000000, attributes={}, value=12, exemplars=[]}], monotonic=false, aggregationTemporality=CUMULATIVE}}

Logs - look for [scopeInfo: and duplicated log body. The second line is for reference to see that it also contains HTTP request received:

log 
10:12:34.031 [docker-java-stream-636643381] INFO  c.g.extensions.smoketest.SmokeTest - STDOUT: 2024-01-11T09:12:34.03Z INFO 'HTTP request received' : 2337335133908c9ce6e0dfc7bda74d7c 50d689015fd0a33c [scopeInfo: io.opentelemetry.smoketest.springboot.controller.WebController:] {thread.id=32, thread.name="http-nio-8080-exec-1"}
10:12:34.031 [docker-java-stream-636643381] INFO  c.g.extensions.smoketest.SmokeTest - STDOUT: INFO  [http-nio-8080-exec-1] io.opentelemetry.smoketest.springboot.controller.WebController: HTTP request received trace_id=

Resources

  1. Grafana OpenTelemetry distribution for Java on GitHub
  2. upstream OpenTelemetry SDK on GitHub
  3. upstream OpenTelemetry SDK configuration docs