Documentation Grafana Cloud Monitor applications Database Observability Monitor Find application queries

Find your application’s queries

When you need to understand how your application uses the database, Database Observability helps you identify specific queries, filter by application components, and correlate database performance with application traces.

Overview

This guide covers three approaches to finding your application’s queries:

1. Filtering: Use the Queries Overview dashboard filters to narrow down by instance, schema, or custom labels
2. Labels: Organize queries by application, team, or environment using Alloy labels
3. Trace correlation: Link database queries to application traces for end-to-end visibility

Filter queries in the dashboard

The Queries Overview dashboard provides several filtering options to find specific queries.

Use the search box

The search box at the top of the dashboard filters queries by SQL text:

1. Navigate to Database Observability in Grafana Cloud.
2. In the search box, enter part of your query text.
3. The table filters to show only matching queries.

Search examples:

• SELECT * FROM users - Find queries that select from the users table
• INSERT INTO orders - Find insert operations on orders
• JOIN products - Find queries that join with the products table

Filter by instance

To view queries for a specific database server:

1. Click the Filters dropdown and select the instance label.
2. Select one database instance.
3. The dashboard updates to show only queries from the selected instance.

Filter by schema or database

To narrow down to a specific schema:

1. Click the Filters dropdown and select the schema label (MySQL) or datname label (PostgreSQL).
2. Select the schema or database name.
3. View queries that execute against that schema.

Group queries

Use the Group by control to organize queries:

Use labels to identify applications

Configure Alloy to add custom labels that identify which application or service generates each query.

Add application labels in Alloy

In your Alloy configuration, add labels in the loki.relabel component:

Alloy Copy

loki.relabel "db_o11y" {
  forward_to = [loki.write.default.receiver]

  rule {
    target_label = "app"
    replacement  = "order-service"
  }

  rule {
    target_label = "team"
    replacement  = "checkout"
  }

  rule {
    target_label = "environment"
    replacement  = "production"
  }
}

Ensure the same labels are added to your discovery.relabel component for metrics:

Alloy Copy

discovery.relabel "db_o11y" {
  targets = prometheus.exporter.postgres.default.targets

  rule {
    target_label = "app"
    replacement  = "order-service"
  }

  rule {
    target_label = "team"
    replacement  = "checkout"
  }

  rule {
    target_label = "environment"
    replacement  = "production"
  }
}

Filter by custom labels

After adding labels, use them in the dashboard:

1. Click Add filter in the Queries Overview dashboard.
2. Select your custom label (for example, app).
3. Choose the value to filter by (for example, order-service).

Label strategy for multi-application environments

When multiple applications share a database, use a consistent labeling strategy:

Correlate with application traces

You can link a database span in a trace to the query executions in Database Observability. When you click the query text on the span, the app opens the matching query. This requires trace context to be present in the SQL that reaches the database.

A database span, for example in Drilldown Traces or Explore Traces, shows the query text from db.query.text or db.statement. For supported engines, a link appears on that text. Clicking it opens the Database Observability app and searches for a query that matches that trace and span ID.

• Trace → Database Observability: Click the link on the DB span’s query text in the trace UI to open Database Observability and, when we have a matching sample, land on that query.

  • If a match is found, then you are taken directly to the Query Samples tab for that query.
  • If no sample with that trace/span ID exists, you stay on the Queries Overview page. You can still find the query by using the search bar on the overview page to filter the list.
  • To improve the chance of a match, see Query samples collect interval.

• Database Observability → Trace:

  • If the sample has an appended traceparent, you can take the trace ID and search for it in Explore or Drilldown Traces.
  • To improve retention of traces with database spans, see setting up an Adaptive Traces policy.

Requirements

• Your app must be instrumented so the span has db.query.text (or db.statement) and trace/span IDs. That comes from your tracing setup (OpenTelemetry or similar), auto or manual.
• SQL text must have a W3C traceparent appended to it, see using sqlcommenter to append trace context to queries.
• Enable query_samples in your Alloy component and set disable_query_redaction = true in the query_samples block. Disabling redaction is required so that the full SQL text including the appended traceparent comment is preserved and can be matched against the trace span.

Use SQLCommenter to inject trace context

SQLCommenter is an open specification part of the OpenTelemetry project for embedding the traceparent in SQL comments. Middleware like SQLCommenter hooks into the driver or ORM and adds a comment containing traceparent to each statement. For example:

SQL Copy

SELECT * FROM users WHERE id = 42
/*traceparent='00-5bd66ef5095369c7b0d1f8f4bd33716a-c532cb4098ac3dd2-01'*/

Use the OpenTelemetry SQLCommenter integration for your language and stack, for example opentelemetry-sqlcommenter (Python, Node, Java, etc.). Enable it once in your app; it will then annotate SQL for all queries.

Adaptive Traces policy

Leverage Adaptive Traces with a policy that keeps any trace where a span includes query text for our currently supported engines. This facilitates searching for a query sample in Database Observability to traces.

Configure a policy of type and with the following sub-policies:

Copy

{
  "and_sub_policy": [
    {
      "name": "db-system-check",
      "type": "string_attribute",
      "string_attribute": {
        "key": "db.system",
        "values": ["mysql", "postgres"]
      }
    },
    {
      "name": "has-query-text",
      "type": "ottl_condition",
      "ottl_condition": {
        "error_mode": "ignore",
        "span": [
          "attributes[\"db.query.text\"] != nil or attributes[\"db.statement\"] != nil"
        ]
      }
    }
  ]
}

Query samples collect_interval and trace linkage match rate

When clicking from a trace span to Database Observability, the app searches for a query sample that matches that trace and span ID. Whether we find a match depends on whether Alloy had already collected a sample for that execution.

The query_samples block has a collect_interval option. It controls how often the collector scrapes query samples from the database. A shorter interval means more frequent collection, so more executions are sampled and the chance that the exact execution clicked on in the trace is present in Database Observability increases.

If confronted frequently with “no matching sample,” consider lowering collect_interval for query_samples.