How to redact secrets from logs with Grafana Alloy and Loki

In any observability stack, logs are essential for uncovering insights, troubleshooting issues, and ensuring system health. However, managing the security of logged data presents its own challenges, especially when it comes to preventing sensitive information, like API keys and credentials, from slipping into logs. 

Secrets can originate from a variety of sources, and it’s often challenging to predict which applications or services might inadvertently expose sensitive information. Configuring each application individually to avoid logging these secrets can be complex, time-consuming, and error-prone, especially when you consider the range of applications and services running across modern infrastructures. 

Redacting secrets as the logs are received and processed can therefore be a convenient and powerful solution to this problem, as well as an extra line of defense. That’s why the Security Operations team at Grafana Labs is excited to be working on a new experimental component (loki.secretfilter) for  Grafana Alloy, our distribution of the OpenTelemetry Collector with built-in Prometheus pipelines.

How the secret filtering component works 

While the component is currently designated experimental (more information about experimental features and their intended audience and risks can be found in our release life cycle docs) and at an early development stage, it can already enhance your log security within Grafana Loki, our open source log aggregation system. It works by scanning logs for predefined patterns that typically indicate sensitive information, such as API keys, passwords, or tokens, and redacts them before they reach Loki.

As an Alloy component, loki.secretfilter is designed to be flexible in how it’s integrated into your infrastructure. Alloy typically collects logs from various sources, processes them, and sends them to your log management system. When working with Loki, this component can be seamlessly added to this flow to filter log lines, regardless of their origin or content.

To achieve this, it leverages the gitleaks.toml configuration file from the popular open source project Gitleaks. This file includes a wide range of predefined patterns for detecting secrets across various formats and frameworks. 

This approach offers a lot of flexibility. The component embeds a default Gitleaks configuration file, which allows users to quickly start detecting secrets in their logs, but it can also be used with customized configuration files that have already been tailored to meet the specific needs of an ecosystem and products in use. While the component doesn’t currently support the full range of features offered by the Gitleaks configuration file format, we have detailed documentation covering the features that it does. 

The component scans each log line and searches for potential secrets using the regexes defined in the configuration file. Once a secret is detected, and unless it has been specifically added to the allowlist, it will be redacted with a customizable string. This string can include the secret type (determined by the regex rule that identifies it), and if desirable, a hash of the secret. 

The redacted logs can then be forwarded into Loki, which you can query to look for the redaction string. This way, you’ll be able to detect secrets that would have leaked before the component was enabled, and potentially address the issue at the source (for instance, by disabling the debug mode of a given tool).

Including a hash of the secret in the redaction string helps identify which secret is leaking without exposing the actual value in the logs. Engineers with access to the secrets store can compare the hash in the logs with the hashes stored there to pinpoint the impacted secret.

Note: The default Gitleaks configuration file contains a large number of regexes (close to a couple of hundred). For performance reasons, we advise you to focus on the secret types that are known to be part of the ecosystem. This can be done via the configuration of the Alloy component itself (specifying the secret types it should be looking for), or by using a custom Gitleaks configuration file containing the regexes that apply to the ecosystem. 

It is also possible to run the component with everything enabled to catch potentially unexpected secret types and then fine-tune the configuration to improve performance. You can then review the configuration file regularly to account for secret types or products that weren’t on the radar initially.

The component may generate false positives, but these can typically be managed through the allowlist or by refining the regexes for your use case. While Personally Identifiable Information (PII) isn’t currently within scope, custom regexes can be written to identify certain PII formats, such as phone numbers.

How to test it

In order to test the component, you need to run Alloy 1.5.0 or later. As it is still experimental, Alloy must be started with a specific argument. You can find more information about it and the potential side effects on other items in the Alloy documentation about stability levels. The argument to use is:

Copy

--stability.level experimental

In your test environment, the component can then be added to your Alloy configuration like this: 

Copy

loki.secretfilter "<LABEL>" {
    forward_to = <RECEIVER_LIST>
}

The component documentation describes all the available configuration settings, but here are some common ones:

• gitleaks_config: This is the path to a gitleaks.toml configuration file. If you don’t specify your configuration file, the embedded one is used instead. We encourage using a custom configuration file containing the specific regexes relevant to the ecosystem. This also ensures better control over future updates of this file. As we mentioned previously, not all features of the Gitleaks configuration file are available currently, so please refer to the component documentation for more information.
• types: This is a list of secret types to look for. The secret types correspond to the various rules present in the configuration file, and the values in the array will be used as prefixes to match the corresponding rules. For instance, the type grafana will match rules grafana-api-key, grafana-cloud-api-token, and grafana-service-account-token. If no type is specified, all the rules in the configuration file are used, which can be resource-intensive, especially with the default configuration file.
• redact_with: This is the string used to redact identified secrets. Two variables can be used in the string: $SECRET_NAME, which will be replaced by the id of the rule used to identify the secret, and $SECRET_HASH, which will be replaced with a hash of the secret. 
• include_generic: This boolean indicates whether the rule generic-api-key should be used, if present in the configuration file. It is disabled by default as it generates a lot of false positives since the secret filtering component doesn’t currently support entropy.
• allowlist: This array of regexes can be used to allowlist specific secret patterns. It is used in addition to the allowlists present in the gitleaks.toml configuration file.

Here’s an example of an Alloy configuration file (with placeholders) that can be used to redact some secrets:

Copy

local.file_match "local_logs" {
	path_targets = <PATH_TARGETS>
}

loki.source.file "local_logs" {
	targets    = local.file_match.local_logs.targets
	forward_to = [loki.secretfilter.secret_filter.receiver]
}

loki.secretfilter "secret_filter" {
	forward_to  = [loki.write.local_loki.receiver]
	redact_with = "<ALLOY-REDACTED-SECRET:$SECRET_NAME>"
	types = ["grafana", "slack", "github"]
}

And there you have it! With this Alloy configuration in place, Grafana, Slack, and GitHub secrets that match the defined regexes will be redacted from log lines and replaced with a known string that can then be queried in Grafana. It’s even possible to create an alert to look for this redaction string in order to be informed when a leaking secret has been detected and redacted. This can be useful to detect secrets that were previously leaking without being redacted, so that they can be rotated.

What’s next

We’ll be working on the secret filtering component to add more exciting features to empower you with real-time visibility into secret-related events and trends. We’re aiming to help teams detect issues earlier, respond faster, and ensure that sensitive data stays protected. Here’s a sneak peek: one feature on our roadmap is the collection of secret-related metrics so that dashboards can be used to give an overview of what is happening in your logs and see potential issues at a glance. 

Your feedback is also very important to us! If there are features you would like to see implemented, or if you face bugs, please let us know by creating an issue on the Grafana Alloy repository. 

We hope secret redaction from logs is something you’re as excited about as we are and stay tuned for more information regarding secret filtering with Alloy! 

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!