DocumentationGrafana CloudMonitor applicationsApplication ObservabilityUser manualConfiguration

Configure Application Observability

Admin users can configure the Application Observability plugin.

Data source

Configure default data source for each signal type. Changes are saved on selection.

On the Data source tab, the default data source for each signal type can be configured including: metrics, logs, and traces.

By default, the default provisioned data sources for the stack will be used. This enables easier onboarding, allowing only traces to be sent, and necessary metrics to be generated from traces.

If you are using a custom data source, metrics generation is not needed and can be disabled to reduce usage and billing.

Logs query

On the Logs query tab, the default log queries and formatting can be configured.

Default log query format

Select a log query format that matches your logs ingestion path, and customize your default log query:

• Use the Loki exporter query if you are exporting logs directly to Loki using loki-exporter
• Use the OTLP gateway native Loki OTLP query if you are exporting logs via Grafana OTLP gateway or Loki native OTLP endpoint

If necessary, you will be prompted to update any log queries that are in an older log query format. This is required because each ingestion methids uses a different log storage and query format.

You can switch between log query formats on the Logs tab, changes will not persist beyond the current session. If you would like to permantly change the default log query fomat, do so from the Configuration page.

Logs query, without namespace

Configure the logs query to execute in the logs tab if service.namespace resource attribute is not provided. Available variables: $job, and $serviceName.

Logs query, with namespace

Configure the logs query to execute in the logs tab if service.namespace resource attribute is provided. Available variables: $job, $serviceName, and $serviceNamespace.

Logs query formatting

Specify the formatting section of a log query that is automatically appended to the end of logs queries to improve readability.

For example, the default formatting appended to log queries is:

LogQL Copy

line_format "\u001b[1m{{if .severity}}{{alignRight 5 .severity}}{{end}}\u001b[0m \u001b[90m[{{alignRight 10 .resources_service_instance_id}}{{if .attributes_thread_name}}/{{alignRight 20 .attributes_thread_name}}{{end}}]\u001b[0m \u001b[36m{{if .instrumentation_scope_name }}{{alignRight 40 .instrumentation_scope_name}}{{end}}\u001b[0m{{if .traceid}} \u001b[37m\u001b[3m[traceid={{.traceid}}]{{end}}: {{.body}}"

Important: the formatting is automatically applied at the end of the actual query. Messing this up can lead to invalid queries.

Semantics

Use the semantics configuration tab to configure which attributes are used for grouping and filtering data in Application Observability.

Deployment environment attribute and value

By default Application Observability uses the OpenTelemetry recommended attribute deployment.environment to distinguish environments. A custom attribute can be selected from a list of possible attributes from the data.

Application Observability will select a default value that resembles the production environment. A custom value can be set in the Default environment section.

Group and filter attributes

To enable data to be grouped and filtered in Application Observability, first add the necessary attributes to traces and then select them from a list of available attributes in Application Observability.

By default you can group and filter by the deployment environment.

System

Use the system configuration tab to configure system-wide settings.

Span metrics source

Application Observability uses metrics generated from traces for most of the visualizations in the application. This data can come from different places depending on your setup. You can change which source you use from the System tab, under the Span metrics source section:

• Tempo (Grafana Cloud): (default) for metrics generation through the Tempo metrics generator
• OpenTelemetry Collector: for advanced setups using the OpenTelemetry collector with the span metrics connector and the service graph metrics connector – Legacy: for customers who use the OpenTelemetry Collector with metric names that match those used by the Tempo metrics generator.

Metrics generation

Enable or disable metrics generation for the default provisioned Traces data source for the stack.