Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Knowledge graphbreadcrumb arrow Configurebreadcrumb arrow Telemetry correlation
Grafana Cloud

Configure telemetry correlation

The knowledge graph automatically correlates entities with their telemetry data across logs, traces, and profiles. Telemetry correlation configurations control how entity properties map to labels in your observability data sources, enabling seamless exploration without writing custom queries.

How telemetry correlation works

When you click on an entity in the entity catalog or RCA workbench, the knowledge graph opens pre-filtered views of logs, traces, and profiles specific to that entity. Correlation configurations define the mapping between entity properties and data source labels.

For example, a service entity has properties like namespace, cluster, and service_name. Your Loki logs use labels like service_namespace and k8s_cluster_name. A correlation configuration maps these properties to the correct labels, allowing the knowledge graph to automatically generate queries.

Configuration types

You can configure correlation for three telemetry types:

  • Logs - Map entity properties to Loki labels
  • Traces - Map entity properties to Tempo labels
  • Profiles - Map entity properties to Pyroscope labels

Each configuration type uses the same flexible matching system with priority-based selection.

Priority-based matching

You can create multiple configurations for each telemetry type. The knowledge graph evaluates configurations from highest to lowest priority and applies the first matching configuration.

Priority-based matching enables:

  • Environment-specific mappings - Different label schemas for production versus development
  • Entity type variations - Services use different labels than infrastructure entities
  • Gradual migrations - Maintain old and new label schemas simultaneously

How priority works

  1. Configurations are ordered by priority (drag and drop in the UI or set explicit priority values)
  2. For each entity, the knowledge graph evaluates configurations from top to bottom
  3. The first configuration where all property matchers succeed is applied
  4. If no custom configuration matches, the default configuration applies

Entity property matchers

Property matchers define which entities use a specific configuration. You can match based on any entity property using these operations:

  • equals - Property value exactly matches one of the specified values
  • not equals - Property value doesn’t match any specified values
  • contains - Property value contains one of the specified substrings
  • does not contain - Property value doesn’t contain any specified substrings
  • is null - Property doesn’t exist on the entity
  • is not null - Property exists on the entity

Example matchers

Match production services:

  • Property: asserts_env
  • Operation: equals
  • Values: prod, production

Match all Kubernetes entities:

  • Property: cluster
  • Operation: is not null

Exclude test namespaces:

  • Property: namespace
  • Operation: does not contain
  • Values: test, dev, staging

Entity property to label mapping

After matching an entity to a configuration, the knowledge graph uses property-to-label mappings to generate queries.

Each mapping connects:

  • Entity property - A property from the entity (for example, namespace, service_name)
  • Data source label - The corresponding label in Loki, Tempo, or Pyroscope (for example, service_namespace, resource.service.name)

The knowledge graph automatically substitutes entity property values into the generated queries.

Common mappings

For OpenTelemetry instrumented services:

Entity propertyLoki/Tempo labelPyroscope label
otel_namespaceresource.service.namespacenamespace
otel_serviceresource.service.name
clusterresource.k8s.cluster.namecluster

Mappings depend on your instrumentation and label schema. Refer to your data source configuration to determine the correct labels.

Default configurations

Each telemetry type includes a default configuration that applies when no custom configurations match. Default configurations:

  • Apply to all environments and cloud regions
  • Can’t be deleted
  • Use standard OpenTelemetry semantic conventions
  • Provide sensible mappings for common use cases

You can edit default configurations to adjust mappings, but the default always remains as a fallback.

Manage configurations

You manage telemetry correlation configurations in the Grafana Cloud UI:

  1. Navigate to Observability > Configuration
  2. Select the Logs, Traces, or Profiles tab
  3. Click Add logs config, Add trace config, or Add profile config
  4. Configure property matchers and label mappings
  5. Save the configuration

To change priority, click Change priority and drag configurations into the desired order.

Troubleshoot missing data

If telemetry data doesn’t appear when viewing an entity, the knowledge graph displays which configuration was evaluated and why it didn’t match.

Check the entity’s Logs, Traces, or Profiles tab for:

  • Configuration match status - Shows which property matchers succeeded or failed
  • Entity property values - Displays actual values from the entity
  • Suggested configurations - Recommends fixes or new configurations

Use this diagnostic information to:

  • Verify entity properties match your expectations
  • Adjust property matcher operations or values
  • Create new configurations for edge cases

Next steps