Understand labels on Grafana Labs

Label best practices

Thu, 09 Apr 2026 02:28:18 +0000

Label best practices

Grafana Loki is under active development, and we are constantly working to improve performance. But here are some of the most current best practices for labels that will give you the best experience with Loki.

Static labels are good

Use labels for things like regions, clusters, servers, applications, namespaces, and environments. They will be fixed for a given system/app and have bounded values. Use static labels to make it easier to query your logs in a logical sense (for example, show me all the logs for a given application and specific environment, or show me all the logs for all the apps on a specific host).

Use dynamic labels sparingly

Too many label value combinations leads to too many streams. The penalties for that in Loki are a large index and small chunks in the store, which in turn can actually reduce performance.

To avoid those issues, don’t add a label for something until you know you need it! Use filter expressions (|= "text", |~ "regex", …) and brute force those logs. It works – and it’s fast.

If you often parse a label from a log line at query time, the label has a high cardinality, and extracting that label is expensive in terms of performance; consider extracting the label on the client side attaching it as structured metadata to log lines .

From early on, we have set a label dynamically using client pipelines for level. This seemed intuitive for us as we often wanted to only show logs for level="error"; however, we are re-evaluating this now as writing a query. {app="loki"} |= "level=error" is proving to be just as fast for many of our applications as {app="loki",level="error"}.

This may seem surprising, but if applications have medium to low volume, that label causes one application’s logs to be split into up to five streams, which means 5x chunks being stored. And loading chunks has an overhead associated with it. Imagine now if that query were {app="loki",level!="debug"}. That would have to load way more chunks than {app="loki"} != "level=debug".

Above, we mentioned not to add labels until you need them, so when would you need labels?? A little farther down is a section on chunk_target_size. If you set this to 1MB (which is reasonable), this will try to cut chunks at 1MB compressed size, which is about 5MB-ish of uncompressed logs (might be as much as 10MB depending on compression). If your logs have sufficient volume to write 5MB in less time than max_chunk_age, or many chunks in that timeframe, you might want to consider splitting it into separate streams with a dynamic label.

What you want to avoid is splitting a log file into streams, which result in chunks getting flushed because the stream is idle or hits the max age before being full. As of Loki 1.4.0, there is a metric which can help you understand why chunks are flushed sum by (reason) (rate(loki_ingester_chunks_flushed_total{cluster="dev"}[1m])).

It’s not critical that every chunk be full when flushed, but it will improve many aspects of operation. As such, our current guidance here is to avoid dynamic labels as much as possible and instead favor filter expressions. For example, don’t add a level dynamic label, just |= "level=debug" instead.

Here are some best practices for using dynamic labels with Loki:

Ensure the labels have low cardinality, ideally limited to tens of values.
Use labels with long-lived values, such as the initial segment of an HTTP path: /load, /save, /update.
- Do not extract ephemeral values like a trace ID or an order ID into a label; the values should be static, not dynamic.
Only add labels that users will frequently use in their queries.
- Don’t increase the size of the index and fragment your log streams if nobody is actually using these labels. This will degrade performance.

Label values must always be bounded

If you are dynamically setting labels, never use a label which can have unbounded or infinite values. This will always result in big problems for Loki.

Try to keep values bounded to as small a set as possible. We don’t have perfect guidance as to what Loki can handle, but think single digits, or maybe 10’s of values for a dynamic label. This is less critical for static labels. For example, if you have 1,000 hosts in your environment it’s going to be just fine to have a host label with 1,000 values.

As a general rule, you should try to keep any single tenant in Loki to less than 100,000 active streams, and less than a million streams in a 24-hour period. These values are for HUGE tenants, sending more than 10 TB a day. If your tenant is 10x smaller, you should have at least 10x fewer labels.

Be aware of dynamic labels applied by clients

Loki has several client options: Grafana Alloy (which also supports systemd journal ingestion and TCP-based syslog ingestion), Fluentd, Fluent Bit, a Docker plugin, and more.

Each of these come with ways to configure what labels are applied to create log streams. But be aware of what dynamic labels might be applied. Use the Loki series API to get an idea of what your log streams look like and see if there might be ways to reduce streams and cardinality. Series information can be queried through the Series API, or you can use logcli.

In Loki 1.6.0 and newer the logcli series command added the --analyze-labels flag specifically for debugging high cardinality labels:

Total Streams:  25017
Unique Labels:  8

Label Name  Unique Values  Found In Streams
requestId   24653          24979
logStream   1194           25016
logGroup    140            25016
accountId   13             25016
logger      1              25017
source      1              25016
transport   1              25017
format      1              25017

In this example you can see the requestId label had a 24653 different values out of 24979 streams it was found in, this is bad!!

This is a perfect example of something which should not be a label, requestId should be removed as a label and instead filter expressions should be used to query logs for a specific requestId. For example if requestId is found in the log line as a key=value pair you could write a query like this: {logGroup="group1"} |= "requestId=32422355"

Cardinality

Thu, 09 Apr 2026 02:28:18 +0000

Cardinality

The cardinality of a data attribute is the number of distinct values that the attribute can have. For example, a boolean column in a database, which can only have a value of either true or false has a cardinality of 2.

High cardinality refers to a column or row in a database that can have many possible values. For an online shopping system, fields like userId, shoppingCartId, and orderId are often high-cardinality columns that can have hundreds of thousands of distinct values.

Other examples of high cardinality attributes include the following:

Timestamp
IP addresses
Kubernetes pod names
User ID
Customer ID
Trace ID

When we talk about cardinality in Loki we are referring to the combination of labels and values and the number of log streams they create. In Loki, the fewer labels you use, the better. This is why Loki has a default limit of 15 index labels.

High cardinality can result from using labels with a large range of possible values, or combining many labels, even if they have a small and finite set of values, such as combining status_code and action. A typical set of status codes (200, 404, 500) and actions (GET, POST, PUT, PATCH, DELETE) would create 15 unique streams. But, adding just one more label like endpoint (/cart, /products, /customers) would triple this to 45 unique streams.

To see an example of series labels and cardinality, refer to the LogCLI tutorial. As you can see, the cardinality for individual labels can be quite high, even before you begin combining labels for a particular log stream, which increases the cardinality even further.

To view the cardinality of your current labels, you can use logcli.

logcli series '{}' --since=1h --analyze-labels

Impact of high cardinality in Loki

High cardinality causes Loki to create many streams, especially when labels have many unique values, and when those values are short-lived (for example, active for seconds or minutes). This causes Loki to build a huge index, and to flush thousands of tiny chunks to the object store.

Loki was not designed or built to support high cardinality label values. In fact, it was built for exactly the opposite. It was built for very long-lived streams and very low cardinality in the labels. In Loki, the fewer labels you use, the better.

High cardinality can lead to significant performance degradation.

Avoiding high cardinality

To avoid high cardinality in Loki, you should:

Avoid assigning labels with unbounded values, for example timestamp, trace ID, order ID.
Prefer static labels that describe the origin or context of the log message, for example, application, namespace, environment.
Don’t assign “dynamic” labels, which are values from the log message itself, unless it is low-cardinality, or a long-lived value.
Use structured metadata to store frequently-searched, high-cardinality metadata fields, such as customer IDs or transaction IDs, without impacting Loki’s index.

Note
Structured metadata is a feature in Loki and Cloud Logs that allows customers to store metadata that is too high cardinality for log lines, without needing to embed that information in log lines themselves.
It is a great home for metadata which is not easily embeddable in a log line, but is too high cardinality to be used effectively as a label. Query acceleration with Blooms also utilizes structured metadata.

What is structured metadata

Thu, 09 Apr 2026 02:28:18 +0000

What is structured metadata

Warning
Structured metadata was added to chunk format V4 which is used if the schema version is greater or equal to 13. See Schema Config for more details about schema versions.

Selecting proper, low cardinality labels is critical to operating and querying Loki effectively. Some metadata, especially infrastructure related metadata, can be difficult to embed in log lines, and is too high cardinality to effectively store as indexed labels (and therefore reducing performance of the index).

Structured metadata is a way to attach metadata to logs without indexing them or including them in the log line content itself. Examples of useful metadata are kubernetes pod names, process ID’s, or any other label that is often used in queries but has high cardinality and is expensive to extract at query time.

Structured metadata can also be used to query commonly needed metadata from log lines without needing to apply a parser at query time. Large json blobs or a poorly written query using complex regex patterns, for example, come with a high performance cost. Examples of useful metadata include container_IDs or user IDs.

When to use structured metadata

You should only use structured metadata in the following situations:

If you are ingesting data in OpenTelemetry format, using Grafana Alloy or an OpenTelemetry Collector. Structured metadata was designed to support native ingestion of OpenTelemetry data.
If you have high cardinality metadata that should not be used as a label and does not exist in the log line. Some examples might include process_id or thread_id or Kubernetes pod names.
If you are using Logs Drilldown to visualize and explore your Loki logs. You must set discover_log_levels and allow_structured_metadata to true in your Loki configuration.
If you are a large-scale customer, who is ingesting more than 75TB of logs a month and are using Bloom filters (Experimental), starting in Loki 3.3 Bloom filters now utilize structured metadata.

Enable or disable structured metadata

You enable structured metadata in the Loki config.yaml file.

limits_config:
    allow_structured_metadata: true
    volume_enabled: true
    retention_period: 672h # 28 days retention

You can disable Structured Metadata by setting allow_structured_metadata: false in the limits_config section or set the command line argument -validation.allow-structured-metadata=false. Note structured metadata is required to support ingesting OTLP data.

Attaching structured metadata to log lines

You have the option to attach structured metadata to log lines in the push payload along with each log line and the timestamp. For more information on how to push logs to Loki via the HTTP endpoint, refer to the HTTP API documentation.

Alternatively, you can use Grafana Alloy to extract and attach structured metadata to your log lines. See the Alloy stage.structured_metadata block for more information.

With Loki version 1.2.0, support for structured metadata has been added to the Logstash output plugin. For more information, see logstash.

Warning
Structured metadata size is taken into account while asserting ingestion rate limiting. Along with that, there are separate limits on how much structured metadata can be attached per log line.
YAML
# Maximum size accepted for structured metadata per log line.
# CLI flag: -limits.max-structured-metadata-size
[max_structured_metadata_size: <int> | default = 64KB]

# Maximum number of structured metadata entries per log line.
# CLI flag: -limits.max-structured-metadata-entries-count
[max_structured_metadata_entries_count: <int> | default = 128]

Querying structured metadata

Structured metadata is extracted automatically for each returned log line and added to the labels returned for the query. You can use labels of structured metadata to filter log line using a label filter expression.

For example, if you have a label pod attached to some of your log lines as structured metadata, you can filter log lines using:

{job="example"} | pod="myservice-abc1234-56789"

Of course, you can filter by multiple labels of structured metadata at the same time:

{job="example"} | pod="myservice-abc1234-56789" | trace_id="0242ac120002"

Note that since structured metadata is extracted automatically to the results labels, some metric queries might return an error like maximum of series (50000) reached for a single query. You can use the Keep and Drop stages to filter out labels that you don’t need. For example:

count_over_time({job="example"} | trace_id="0242ac120002" | keep job  [5m])

Modify default OpenTelemetry labels

Thu, 09 Apr 2026 02:28:18 +0000

Modify default OpenTelemetry labels

When Grafana Loki first started supporting OpenTelemetry, we selected a set of default resource attributes to promote to labels. We included some labels such as k8s.pod.name to be consistent with the scrape configs we have been suggesting for many years for Grafana Alloy. However, Loki has evolved a lot over the years and with the introduction of Structured Metadata we no longer recommend indexing a few of the labels we included in the defaults as they often have very high cardinality and Structured Metadata is a much better place for them. We no longer recommend the following as default labels:

k8s.pod.name
service.instance.id

Because removing these resource attributes from labels would be a breaking change for existing users, they have not yet been deprecated. If you are a new user of Grafana Loki, we recommend that you modify your Grafana Alloy or OpenTelemetry Collector configuration to convert these resource attributes from index labels to structured metadata.

Alloy configuration file example

If you are using Grafana Alloy to collect logs, the following example shows how to update your alloy.config file to demote k8s.pod.name from an index label to structured metadata.

// discovery.kubernetes allows you to find scrape targets from Kubernetes resources.
// It watches cluster state and ensures targets are continually synced with what is currently running in your cluster.
discovery.kubernetes "pod" {
  role = "pod"
}

// discovery.relabel rewrites the label set of the input targets by applying one or more relabeling rules.
// If no rules are defined, then the input targets are exported as-is.
discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pod.targets

  // Label creation - "namespace" field from "__meta_kubernetes_namespace"
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    action = "replace"
    target_label = "namespace"
  }

  // Label creation - "pod" field from "__meta_kubernetes_pod_name"
  //rule {
  //  source_labels = ["__meta_kubernetes_pod_name"]
  //  action = "replace"
  //  target_label = "pod"
  //}


  // Label creation - "container" field from "__meta_kubernetes_pod_container_name"
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    action = "replace"
    target_label = "container"
  }

  // Label creation -  "app" field from "__meta_kubernetes_pod_label_app_kubernetes_io_name"
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
    action = "replace"
    target_label = "app"
  }

  // Label creation -  "job" field from "__meta_kubernetes_namespace" and "__meta_kubernetes_pod_container_name"
  // Concatenate values __meta_kubernetes_namespace/__meta_kubernetes_pod_container_name
  rule {
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
    action = "replace"
    target_label = "job"
    separator = "/"
    replacement = "$1"
  }

  // Label creation - "container" field from "__meta_kubernetes_pod_uid" and "__meta_kubernetes_pod_container_name"
  // Concatenate values __meta_kubernetes_pod_uid/__meta_kubernetes_pod_container_name.log
  rule {
    source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
    action = "replace"
    target_label = "__path__"
    separator = "/"
    replacement = "/var/log/pods/*$1/*.log"
  }

  // Label creation -  "container_runtime" field from "__meta_kubernetes_pod_container_id"
  rule {
    source_labels = ["__meta_kubernetes_pod_container_id"]
    action = "replace"
    target_label = "container_runtime"
    regex = "^(\\S+):\\/\\/.+$"
    replacement = "$1"
  }
}

// loki.source.kubernetes tails logs from Kubernetes containers using the Kubernetes API.
loki.source.kubernetes "pod_logs" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.process.pod_logs.receiver]
}

// loki.process receives log entries from other Loki components, applies one or more processing stages,
// and forwards the results to the list of receivers in the component's arguments.
loki.process "pod_logs" {
  stage.static_labels {
      values = {
        cluster = "<CLUSTER_NAME>",
      }
  }

  forward_to = [loki.write.<WRITE_COMPONENT_NAME>.receiver]
}

Kubernetes Monitoring Helm chart configuration example

If you are using the Kubernetes Monitoring Helm chart to collect Kubernetes logs to send to Loki, the following example shows how to update the values.yaml file for your Helm chart to demote k8s.pod.name and service.instance.id from index labels to structured metadata.

# Enable pod log collection for the cluster. Will collect logs from all pods in both the meta and loki namespace. 

podLogs:
  enabled: true
  collector: alloy-singleton
  #List of attributes to use as index labels in loki
  labelsToKeep:
    - app
    - app_kubernetes_io_name
    - component
    - container
    - job
    - level
    - namespace
    - service_name
    - cluster
  gatherMethod: kubernetesApi
  namespaces:
    - meta
    - loki
  #This is the section that specifies to send k8s.pod.name and service.instance.id to structured metadata
  structuredMetadata:
    instance_id:
    pod:

Loki configuration example

If you are running Loki in a Docker container or on-premises, here is an example of how to modify your Loki values.yaml file to demote k8s.pod.name and service.instance.id from index labels to structured metadata.


distributor:
  otlp_config:
    # List of default otlp resource attributes to be picked as index labels - EDIT TO REMOVE k8s.pod.name AND service.instance.id FROM THE LIST
    # CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels
    default_resource_attributes_as_index_labels: [service.name service.namespace deployment.environment deployment.environment.name cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name]

Kubernetes example values.yaml:

loki:
  distributor:
    otlp_config:
    # List of default otlp resource attributes to be picked as index labels - EDIT TO REMOVE k8s.pod.name AND service.instance.id FROM THE LIST
    # CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels
      default_resource_attributes_as_index_labels: [service.name service.namespace deployment.environment deployment.environment.name cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name]