Application Observability

Service overview

Grafana Cloud

Application Observability service overview

The overview page provides a health overview for a service with RED metrics (request rates, errors, and duration) for:

The service as a whole
Related upstream and downstream services
Operations performed on the service

Navigate to various child tabs and pages to uncover further details:

View Traces and Logs
View the service in the Service map
View RED metrics for each Operation
View Runtime specific performance metrics
On most panels, click Explore to view the queries used

Uninstrumented services display limited metrics, from what the plugin is able to discover.

Like other Application Observability pages, the service overview page has a header with a title, sub-tabs to navigate to, an action bar to change the data source and time-related filters and refresh intervals, and an environment selector.

The header on the service overview page also includes technology icons, alongside the service name, with contextual menus.

Technology icons

The following icons are displayed alongside a service name:

Technology icon, inferred from telemetry.sdk.language of the trace span
Kubernetes icon, if the service is using Kubernetes
Cloud icon, if the service is deployed on a cloud environment

If the service uses Kubernetes, hover over the Kubernetes icon to view the environment labels and navigate to Kubernetes monitoring.

If the service is deployed on a cloud environment, hover over the Cloud icon to view the environment labels.

Action bar

Use the action bar to change the data source(s) and time-related filters and refresh intervals:

The default time range for all panels is 30 mins
If no data is shown, change the time range
Set a time frame to compare metrics
Set the environment to show metrics for
Use the refresh interval dropdown to select an automatic refresh interval
Click the refresh icon to toggle a manual refresh
On the Logs tab, click Live to toggle log streaming

Time frame comparison

Use the time frame comparison to compare RED metrics and duration distribution for the current time frame to the previous period or a selection of days, weeks, months or a year before.

Service health

The three panels in the top row show the RED method trends for monitoring service health:

Rate: the number of requests per second
Errors: the number of errors per second
Duration: the 95th percentile request duration

And with a time frame comparison the RED metrics panels show the current period with a solid line and the time period being compared with a dotted line.

Note the Duration metric for time frame comparisons only shows the p95 values.

The Distribution duration panel shows a histogram for request duration.

And with a time frame comparison the distribution duration panel shows the current period in a dark blue and the time period being compared with in a light blue.

Connected services

The Downstream & databases and Upstream panels show a table of connected services and databases.

Each row shows the RED metrics trends for request rate, errors, and duration (p95th).

Click a service row to navigate to the service overview page for further details.

Operations

The Operations panel shows a table of operations within the service, derived from the span_name.

Each row shows the RED metrics trends for request rate, errors, and duration (p95th).

Click an operation row to navigate to the Operation overview page for detailed RED metrics.

Operation Overview

The Operation overview page shows the RED metrics trends for a specific operation.

Navigate to the Traces tab to search and view traces for the operation.

Traces

Navigate to the Traces tab to search and view traces for the service.

Search

Search for traces with the Trace query language (TraceQL).

By default, traces are filtered by the service and service namespace. There are two ways to customize search queries:

Click TraceQL to use the Trace query language (TraceQL)
Click Search to use the visual TraceQL query builder

View

Click a Trace ID in the search results to inspect a trace using the trace view. Consult the trace view documentation to learn how to navigate the trace view and filter spans.

Logs

Navigate to the Logs tab to search and view logs for the service.

Note: works with Agent v0.35.0 and above.

Search

Search for logs with the Log Query Language (LogQL).

By default, logs are filtered by the service and service namespace. There are two ways to customize search queries:

Click Code to use the Log Query Language (LogQL)
Click Builder to use the the visual query builder

View

Log search results are displayed in the logs Volume and a Logs panels. Consult the Logs visualization documentation to learn more.

Runtime

Service that use the OpenTelemetry auto-instrumentation SDKs for .NET, Go, or Java have access to a Runtime tab with various CPU and memory utilization graphs.