Documentation Grafana Cloud Observe and act Monitor applications Application Observability User manual Configuration

Configure Application Observability

Admin users can configure the Application Observability plugin.

Data source

You can configure the default data source for each signal type, when you select an option the changes are automatically saved.

On the Data source tab, you can configure the default data source for each signal type: metrics, logs, and traces.

By default, Application Observability uses the automatically provisioned data sources for the stack. This gives users simple getting started workflow that only needs them to send traces, and Application Observability generated the necessary metrics.

If you are using a custom data source for metrics you don’t need the automatic metrics generation provided Application Observability and can disable it to reduce your Grafana Cloud usage and bill.

Logs query

On the Logs query tab, you can configure the default log queries and formatting Application Observability uses to query and format logs.

Default log query format

Select a log query format that matches your logs ingestion path, and customize your default log query:

• Use the “Loki exporter” query if you are exporting logs directly to Loki using the OpenTelemetry Collector Loki Exporter or the Alloy otelcol.exporter.loki
• Use the “OTLP gateway / native Loki OTLP query” mode if you are exporting logs via the Grafana OTLP endpoint

The log query format needs to match the log storage and query format used by the ingestion method. If your log queries use a previous log format version, Application Observability prompts you to update the query.

You can switch between log query formats on the Logs tab, changes won’t persist beyond the current session. If you would like to permanently change the default log query format, update the setting from the Configuration page.

Logs query, without namespace

You can configure the logs query to execute in the logs tab if service.namespace resource attribute isn’t provided. Available variables: $job, and $serviceName.

Logs query, with namespace

You can configure the logs query to execute in the logs tab when you supply the service.namespace resource attribute. Available variables: $job, $serviceName, and $serviceNamespace.

Logs query formatting

You can specify the formatting section of a log query to automatically appended to the end of logs queries to improve readability.

For example, the default formatting appended to log queries is:

LogQL Copy

line_format "\u001b[1m{{if .severity}}{{alignRight 5 .severity}}{{end}}\u001b[0m \u001b[90m[{{alignRight 10 .resources_service_instance_id}}{{if .attributes_thread_name}}/{{alignRight 20 .attributes_thread_name}}{{end}}]\u001b[0m \u001b[36m{{if .instrumentation_scope_name }}{{alignRight 40 .instrumentation_scope_name}}{{end}}\u001b[0m{{if .traceid}} \u001b[37m\u001b[3m[traceid={{.traceid}}]{{end}}: {{.body}}"

Important: the formatting is automatically applied at the end of the actual query. Messing this up can lead to invalid queries.

Filtering logs by trace and span ID

Application Observability provides options to automatically filter logs by trace ID and span ID when navigating from a trace view to logs. These options are available in the Configuration page as:

• Filter by trace id: When enabled, navigating to logs from a trace view automatically add a filter for the trace ID. This allows you to see only the logs associated with the selected trace, making it easier to correlate trace and log data.
• Filter by span id: When enabled, navigating to logs from a trace view automatically add a filter for the span ID. This further narrows the log results to those associated with a specific span within the trace.

Both options are enabled by default. You can enable or disable them in the Configuration page under the logs query sections.

Requirements:

• Your logs must include the trace ID and/or span ID fields, typically as traceid and spanid attributes, for these filters to work. Ensure that your log ingestion pipeline is configured to include these fields if you want to use this feature.
• Most logging auto-instrumentation libraries automatically add traceid and spanid attributes to your logs, so these filters work out of the box for most users.

These filters help you quickly correlate logs and traces, improving troubleshooting and root cause analysis workflows.

Settings

You can use the settings tab to configure the behavior of different Application Observability features.

UI

By default, Application Observability does not allow you to sort services on the service inventory page. This prevents performance issues for very large lists of services.

If the amount of services is not too large, you can select the toggle to enable service inventory sorting and gain the ability to sort by any of the Rate, Errors or Duration columns, as well as the always available Name and Namespace columns.

Deployment environment attribute and value

By default Application Observability uses the OpenTelemetry recommended attribute deployment.environment to distinguish environments. You can select a custom attribute from a list of possible attributes from the data.

Group and filter attributes

To enable data grouping and filtering in Application Observability, first add the necessary attributes to traces that you would like to group and filter by, and then activate them as dimensions on the generated metrics.

Dimensions are managed in the Knowledge Graph configuration: navigate to Observability > Configuration > Traces metrics generation > Dimensions. For more information, refer to the Configure Traces metrics generation documentation.

By default you can group and filter by the deployment environment.

Include web applications and mobile devices

Application Observability can display web applications, including services like load generators, and mobile devices when metrics are generated for spans with a CLIENT and PRODUCER spanKind.

If you use Grafana Cloud metrics generation, a filter rule controls which span kinds generate metrics: navigate to Observability > Configuration > Traces metrics generation > Filter rules and add Client and Producer to the span kind rule. Keep Server and Consumer selected, otherwise metrics are no longer generated for your backend services. For more information, refer to the Configure Traces metrics generation documentation.

If you generate metrics with span metrics connector, you need to update your configuration to not ignore CLIENT and PRODUCER spans:

To view the Grafana Alloy configuration, select the river tab below. To view the OpenTelemetry Collector configuration, select the yaml tab below.

Alloy YAML

Copy

Alloy Copy

otelcol.processor.filter "drop_unneeded_span_metrics" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.filter/
	error_mode = "ignore"

	metrics {
		datapoint = [
			"IsMatch(metric.name, \"calls|duration\") and IsMatch(attributes[\"span.kind\"], \"SPAN_KIND_INTERNAL\")",
		]
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

YAML Copy

filter/drop_unneeded_span_metrics:
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor
  error_mode: ignore
  metrics:
    datapoint:
      - 'IsMatch(metric.name, "calls|duration") and IsMatch(attributes["span.kind"], "SPAN_KIND_INTERNAL")'

System

Use the system configuration tab to configure system-wide settings.

Span metrics source

Application Observability uses metrics generated from traces for most of the visualizations in the application. This data can come from different places depending on your setup. You can change which source you use from the System tab, under the Span metrics source section:

• Tempo (Grafana Cloud): the default span metrics source, for metrics generation through the Tempo metrics generator
• OpenTelemetry Collector, Grafana Alloy, Grafana Agent: for advanced setups using the OpenTelemetry collector, Grafana Alloy or Grafana Agent, with the span metrics connector and the service graph metrics connector
• Legacy: for customers who use the OpenTelemetry Collector with metric names that match those used by the Tempo metrics generator.

The following metric names are used by each of the options:

Metrics generation

Enable or disable metrics generation for the default provisioned Traces data source for the stack.

The Tempo metrics generation settings, such as dimensions, filter rules, histogram buckets, service graph metrics, instance label, and span multiplier key, are managed in the Knowledge Graph configuration under Observability > Configuration > Traces metrics generation. For more information, refer to the Configure Traces metrics generation documentation.

Service graph metrics

Service graph metrics generation is managed in the Knowledge Graph configuration: navigate to Observability > Configuration > Traces metrics generation > General and toggle Service graph metrics. For more information, refer to the Configure Traces metrics generation documentation.

Histogram type

Histogram type and bucket settings are managed in the Knowledge Graph configuration: navigate to Observability > Configuration > Traces metrics generation > Histogram buckets. For more information, refer to the Configure Traces metrics generation documentation.

Beyla

Beyla can generate its own span metrics and service graph metrics. When used together with Grafana Cloud metrics generation, this can lead to duplicate series, which can display erroneous data in Application Observability.

To exclude Beyla traces from Grafana Cloud metrics generation, navigate to Observability > Configuration > Traces metrics generation > Filter rules and add the recommended rule from the catalog that excludes Beyla traces. For more information, refer to the Configure Traces metrics generation documentation.

Sampling

To configure head or tail sampling to reduce the cost of processing and storing large volumes of data, consult the sampling documentation.