Menu
Grafana Cloud

Set up production instrumentation for Scala

Note

This guide is identical to the Java guide, because the instrumentation agent also supports Scala libraries and frameworks, such as Play.

Follow these steps set up production Application Observability with a telemetry collector:

  1. Install the instrumentation library
  2. Configure an application
  3. Set up a telemetry data collector
  4. Run the application
  5. Observe the service in Application Observability

Note

For a quick and easy local development setup, consult the quickstart documentation.

Application Observability - Architecture

Install the instrumentation library

First, download the latest JAR release of the instrumentation agent, the Grafana OpenTelemetry auto-instrumentation Distribution for Java.

Configure an application

Next, customize the following shell script to configure an application:

shell

export OTEL_SERVICE_NAME=<Service Name> export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=<Environment>,service.namespace=<Namespace>,service.version=<Version>,service.instance.id=<Instance>

java -javaagent:path/to/grafana-opentelemetry-java.jar -jar myapp.jar

  1. Choose a Service Name to identify the service, for example cart
  2. Add attributes to filter data:
    • deployment.environment: Name of the deployment environment (staging or production)
    • service.namespace: A namespace to group similar services (e.g. shop would create shop/cart in Application Observability)
    • service.version: The application version, to see if a new version has introduced a bug
    • service.instance.id: The unique instance, for example the Pod name (a UUID is generated by default)
  1. Replace myapp.jar with the path to the application to instrument

Set up a telemetry data collector

In production environments, a robust and flexible observability setup needs to process telemetry data before ingestion into databases. Follow the collector setup documentation to set up a collector to process and send telemetry data to Application Observability.

Finally, set the following environment variables from the exporter configuration:

ConfigurationOptionsResult
export OTEL_EXPORTER_OTLP_ENDPOINT=<host>http://localhost:4317, remote host addressThe default local host address, or a remote host address.
export OTEL_EXPORTER_OTLP_PROTOCOL=<protocol>grpc, http/protobufThe default grpc protocol or http/protobuf

For example, for a local Grafana Agent:

shell
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Set the OpenTelemetry Service name and resource attributes and run the application with the Grafana OpenTelemetry Java agent:

System properties can be used instead of environment variables, for example -Dotel.service.name=<OTEL_SERVICE_NAME> instead of export OTEL_SERVICE_NAME=<OTEL_SERVICE_NAME>

Run the application

Finally, run the application with the shell script and make some requests to the service to send data to Grafana Cloud.

Observe the service in Application Observability

Open Application observability:

  1. Navigate to a stack https://<your-stack-name.>grafana.net
  2. Expand the top left menu below the Grafana logo
  3. Click on Application

Activate metrics generation

Application Observability relies on metrics generated from traces already sent to Grafana Cloud Traces.

Metrics generation is self-serve, and can be enabled during onboarding and disabled from Application Observability configuration

To complete the setup, click Activate Application Observability to enable metrics generation.

Enable metrics generation

Note

After activating Application Observability and enabling metrics generation, it might take up to five minutes to generate metrics and show graphs.

Visualize and discover

Discover more about Application Observability in the documentation:

  • Service Inventory: filter, and search services and view RED metrics.
  • Service Overview: traces, logs, RED metrics, operations, and runtime information.
  • Service Map: graph of connected services, service dependencies, and data flow.

Example application

See the Grafana OpenTelemetry Java Spring PetClinic configuration for a complete example production setup.

Configuration options

The Grafana OpenTelemetry Java distribution can be configured with all the system properties or environment variables from the upstream OpenTelemetry SDK auto-configuration.

All exporters are set to otlp by default (even the logs exporter).

OTLP debug logging

OTLP debug logging can be configured via environment variables:

ConfigurationResult
OTEL_SPAN_EXPORTER=otlp,loggingLog all spans. This will also send spans to Loki.
OTEL_METRICS_EXPORTER=otlp,loggingLog all metrics. This will also send metrics to Loki.
OTEL_LOGS_EXPORTER=otlp,loggingLog all logs (again, just to see they are sent).

Note

Debug logging will produce many logs and should be disabled when not needed.

This will produce log output like this:

Spans - look for LoggingSpanExporter:

INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'WebController.withSpan' : 2337335133908c9ce6e0dfc7bda74d7c 8bfef4eaac83e8cb INTERNAL [tracer: io.opentelemetry.opentelemetry-extension-annotations-1.0:1.32.0-alpha] AttributesMap{data={thread.id=32, code.namespace=io.opentelemetry.smoketest.springboot.controller.WebController, code.function=withSpan, thread.name=http-nio-8080-exec-1}, capacity=128, totalAddedValues=4}

Metrics - look for LoggingMetricExporter:

INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.21.0, attributes={container.id="048b9982e0b98cdc5579334bb1decc157ed1ebc23f391ebe306898898ec32fa4", host.arch="amd64", host.name="048b9982e0b9", os.description="Linux 6.2.0-39-generic", os.type="linux", process.command_line="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java -javaagent:/opentelemetry-javaagent.jar -Dgrafana.otel.use-tested-instrumentations=true io.opentelemetry.smoketest.springboot.SpringbootApplication", process.executable.path="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java", process.pid=1, process.runtime.description="BellSoft OpenJDK 64-Bit Server VM 25.312-b07", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_312-b07", service.instance.id="8231ca95-e9aa-474a-bd98-88349a9942ad", service.name="unknown_service:java", telemetry.auto.version="1.32.0", telemetry.distro.name="grafana-opentelemetry-java", telemetry.distro.version="0.32.0-beta.1", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.runtime-telemetry-java8, version=1.32.0-alpha, schemaUrl=null, attributes={}}, name=jvm.cpu.count, description=Number of processors available to the Java virtual machine., unit={cpu}, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1704964347622000000, epochNanos=1704964351627000000, attributes={}, value=12, exemplars=[]}], monotonic=false, aggregationTemporality=CUMULATIVE}}

Logs - look for [scopeInfo: and duplicated log body (the second line is just for reference to see that is also contains HTTP request received):

10:12:34.031 [docker-java-stream-636643381] INFO  c.g.extensions.smoketest.SmokeTest - STDOUT: 2024-01-11T09:12:34.03Z INFO 'HTTP request received' : 2337335133908c9ce6e0dfc7bda74d7c 50d689015fd0a33c [scopeInfo: io.opentelemetry.smoketest.springboot.controller.WebController:] {thread.id=32, thread.name="http-nio-8080-exec-1"}
10:12:34.031 [docker-java-stream-636643381] INFO  c.g.extensions.smoketest.SmokeTest - STDOUT: INFO  [http-nio-8080-exec-1] io.opentelemetry.smoketest.springboot.controller.WebController: HTTP request received trace_id=

Data saver

Application Observability uses a specific selection of metrics and ingesting all OpenTelemetry metrics can increase cost.

Set the following environment variable to only send metrics that are used by Application Observability and reduce Metrics costs in Grafana Cloud.

shell
export GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true

Note

To still send manually created metrics when data saver is enabled, set the metric name to application.

The following metrics are used by Application Observability:

MetricUsage
process.runtime.jvm.system.cpu.utilizationCPU utilization on the JVM Runtime page
process.runtime.jvm.memory.usageMemory usage on the JVM Runtime page
process.runtime.jvm.memory.limitMemory limit on the JVM Runtime page
process.runtime.jvm.gc.durationGarbage collection duration on the JVM Runtime page
process.runtime.jvm.classes.current_loadedCurrent load on the JVM Runtime page
process.runtime.jvm.threads.countThread count on the JVM Runtime page
db.client.connections.usageJDBC dashboard
db.client.connections.maxJDBC dashboard
db.client.connections.pending_requestsJDBC dashboard
r2dbc.pool.acquiredReactive Database example
r2dbc.pool.max.allocatedReactive Database example
r2dbc.pool.pendingReactive Database example
kafka.producer.record_error_totalKafka example
mongodb.driver.pool.waitqueuesizeMongoDB example
mongodb.driver.pool.checkedoutMongoDB example

Troubleshooting

If services are not visible in Application Observability, these are the most common causes:

No traffic

Make a few requests to the service to send data to Grafana Cloud.

Be patient

It can take up to 5 minutes until the data is visible in Application Observability.

Look for errors

Look for errors - either on the console or in docker or Kubernetes logs, using Application Observability logs doesn’t make sense in this case.

If there are errors sending telemetry data, one of the parameters is usually wrong.

A 5xx response code means that there’s something wrong with the Grafana Cloud OTLP Gateway.

Log agent data

If there are no errors in the logs, enable logging to check the application is sending data.

If the application is not sending data, check that the java agent was loaded - look for the -javaagent command line parameter.

Log exporter data

If the application is sending data to an exporter (Grafana Agent or OpenTelemetry Collector), log the data there and check there are no errors forwarding the telemetry data to Grafana Cloud.

javaagent logging

To inspect the internals of the javaagent, enable the javaagent’s internal debug logging:

shell
export OTEL_JAVAAGENT_DEBUG=true

Note

Only enable the javaagent debug logging when needed. The logs are extremely verbose and impact application performance.

Disable javaagent

Lastly, disable the javaagent to check the application runs successfully without instrumentation:

shell
export OTEL_JAVAAGENT_ENABLED=false

Migrating to upstream

The Grafana OpenTelemtry Java distribution is compatible with the upstream OpenTelemetry auto-instrumentation SDK. Follow these steps to migrate to the upstream SDK:

  1. Remove the grafana prefix from all configuration environment variables or system properties.

  2. Add the service.instance.id to the OTEL_RESOURCE_ATTRIBUTES.

    For example OTEL_RESOURCE_ATTRIBUTES=service.instance.id=<pod>,...

    <pod> is the Kubernetes pod or another unique identifier for the service.

  3. To replicate the data saver features, use the OpenTelemetry Collector to filter metrics.

  4. The resource detectors for Kubernetes (for EKS and GKE) are not bundled in OpenTelemetry Java SDK, use the OpenTelemetry Collector to add Kubernetes resource attributes.

  5. Add the following environment variables required by Application Observability:

    shell
    export OTEL_SEMCONV_STABILITY_OPT_IN=http
    export OTEL_INSTRUMENTATION_MICROMETER_BASE_TIME_UNIT=s
    export OTEL_INSTRUMENTATION_LOG4J_APPENDER_EXPERIMENTAL_LOG_ATTRIBUTES=true
    export OTEL_INSTRUMENTATION_LOGBACK_APPENDER_EXPERIMENTAL_LOG_ATTRIBUTES=true