Link traces and database queries

Note
Trace linkage is in active development. Match rates can vary while the feature evolves, and we’re actively improving it.

Trace linkage connects your application traces to the database queries they run, so you can move between the two in either direction:

From a trace to a query. In Explore Traces or Traces Drilldown, a database span can show a Database Observability link. Selecting it takes you to that query in Database Observability, where you can see its samples, latency, and history.
From a query back to a trace. When a query sample carries trace context, you can jump from the query back to the trace of the request that ran it.

Why this is useful

When an application request is slow, a trace usually shows that a database call was slow, but not why. Trace linkage lets you cross from that span into the query’s full picture in Database Observability: whether it’s habitually slow, what it runs against, and how often it executes. Going the other way, when you find a problem query, you can trace it back to the user-facing request that triggered it, instead of guessing which part of the application is responsible.

What to expect

Trace linkage is best-effort correlation, not a guaranteed link on every span. It works best for queries that Database Observability has already collected.

At minimum, the database spans you want to link need:

Query text in db.query.text or db.statement
A supported database system in db.system.name or db.system: mysql or postgresql

Database Observability matches in two ways:

Query text match is the default and needs no application changes. It matches the span’s query text against the queries Database Observability has collected.
Exact trace match is optional. When enabled, it can link to the single query execution behind a specific span using its trace ID and span ID.

If Database Observability can’t resolve a span to a single query, it sends you to Queries Overview so you can search from there.

Improve precision with exact trace matching

Exact trace matching only covers executions that Database Observability sampled, so it improves the precision of the matches it makes but doesn’t increase how often a link resolves. Most links still come from query text matching.

For exact trace matching, at minimum the query samples collected by the Alloy Database Observability component need:

a W3C traceparent comment appended to your SQL text
Alloy 1.17.0 or later. If you use an earlier version, set disable_query_redaction = true in the query_samples block of your Alloy configuration.

To append a W3C traceparent to your SQL text, refer to SQLCommenter, an open specification that’s part of the OpenTelemetry project. Enable it once in your application and it will annotate SQL for all queries.

You don’t need SQLCommenter for query text matching. Add SQLCommenter only when you want exact trace matching.

Troubleshoot trace matching

If selecting a link doesn’t take you to a specific query, use the checks below for the type of matching you expect.

If query text matching doesn’t find a query

Query text matching relies on the queries Database Observability has already collected. If a link sends you to Queries Overview instead of a specific query, check the following:

Confirm the query text is collected. Database Observability refreshes the queries based on the sampling rate configured in Alloy, so a query it hasn’t collected yet can’t be matched. Wait for the next collection and try again. To check whether a query is available for matching, query the logs under op="query_association" in your logs data source and confirm your query text appears.
The text must match after normalization. Matching uses normalized query text, so unusual literals or formatting can prevent a match.

If exact matching doesn’t find a sample

Exact matching only succeeds if Alloy sampled the specific execution behind the span and captured its traceparent. Alloy samples running queries periodically rather than capturing every execution, so most executions are never sampled and exact matching remains best-effort even when SQLCommenter is configured correctly.

Confirm the traceparent is logged. In your logs data source, query the logs under op="query_sample" and confirm the log lines include a traceparent field. If it’s missing, SQLCommenter isn’t appending the traceparent. A matching log line looks like this:

level="info" datname="shop" pid="1089" leader_pid="" user="shop-api" app="" client="203.0.113.10:54321" backend_type="client backend" state="idle" xid="0" xmin="0" xact_time="" query_time="4.294ms" queryid="5810693783667443671" traceparent="00-5bd66ef5095369c7b0d1f8f4bd33716a-c532cb4098ac3dd2-01"

Lower collect_interval with care. The query_samples block has a collect_interval option that controls how often Alloy samples queries from the database. A shorter interval samples more executions and raises the chance that the one you selected was captured, but it cannot ensure every execution is captured from the database. Shorter intervals also increase telemetry volume and add load to your database, so lower collect_interval gradually and watch the impact. For broader coverage, rely on query text matching.

Link a query sample back to a trace

When a query sample includes a traceparent, you can trace it back to the application request that ran it. Copy the trace ID from the sample and search for it in Explore or Traces Drilldown.

Use Adaptive Traces to keep query spans

If you use Adaptive Traces, you can choose to retain traces that contain database query spans. Retaining those traces keeps the trace available, so you can reliably match from a query sample back to its trace.

Configure an Adaptive Traces policy of type and. The policy retains traces only when a span has both a supported database system and query text:

{
  "and_sub_policy": [
    {
      "name": "db-system-check",
      "type": "ottl_condition",
      "ottl_condition": {
        "span": [
          "attributes[\"db.system.name\"] == \"mysql\" or attributes[\"db.system.name\"] == \"postgresql\" or attributes[\"db.system\"] == \"mysql\" or attributes[\"db.system\"] == \"postgresql\""
        ],
        "error_mode": "ignore"
      }
    },
    {
      "name": "has-query-text",
      "type": "ottl_condition",
      "ottl_condition": {
        "span": [
          "attributes[\"db.query.text\"] != nil or attributes[\"db.statement\"] != nil"
        ],
        "error_mode": "ignore"
      }
    }
  ]
}