Grafana security update: Grafana Loki and unintended data write attempts to Amazon S3 buckets
Editor’s note: We have updated the copy to reflect changes from our partners at Google. (June 27 19:00 UTC)
Out of an abundance of caution, we are publishing this as a security advisory.
You may have read or heard about the Medium blog post, “How an empty S3 bucket can make your AWS bill explode,” which described how unauthorized write attempts to an Amazon S3 storage bucket led to a $1,300 bill, and the news that AWS subsequently made a change to S3 in which unauthorized requests that customers did not initiate are free of charge. In the initial report, Maciej Pocwierz said that the unauthorized writes were the result of an unnamed open source project’s default configuration for storing backups in S3 buckets.
The open source project in question is Grafana Loki, our log aggregation tool. Part of the root cause was the use of the default value chunks in our Loki Helm chart for convenience when using MinIO. When Maciej created an AWS bucket named chunks, it appeared as a valid target for other AWS customers. This bucket was created without world-writeable permissions, but the rejected write attempts were still billed.
In response to the issue, we have updated the Loki Helm chart to remove default bucket names except when using MinIO.
Now that our investigation has concluded, we are publishing this blog post. We’d like to thank Maciej for responsibly disclosing this issue to us via our bug bounty program.
Who is impacted
AWS supports bucket namespaces that go beyond organizational boundaries; therefore, third parties can create a bucket name that appears as a valid target to others and you. We have worked with AWS to confirm that these default bucket names are currently claimed by legitimate entities and have been configured in a way that does not allow for data collection. Out of an abundance of caution, we advise that users with Loki or Grafana Enterprise Logs (GEL) deployments on AWS upgrade their Helm charts or change the names of their buckets, as outlined in the solutions and mitigations section below.
Google Cloud supports globally unique bucket namespaces. We have worked with Google to confirm that Google Cloud users have not been impacted. Out of an abundance of caution, we advise that users with Loki or GEL deployments on Google Cloud upgrade their Helm charts or change the names of their buckets, as outlined in the solutions and mitigations section.
Microsoft Azure does not support bucket namespaces that go beyond organizational boundaries, so users who have Loki or GEL deployments on Azure are not affected.
Grafana Cloud customers are not affected.
Background
Loki stores its data in an S3-compatible object storage. For convenience in local installations, the Loki Helm chart contained default values for a local MinIO installation.
Maciej Pocwierz, a Senior Software Engineer at Semantive and the author of “How an empty S3 bucket can make your AWS bill explode,” reported that he incurred $1,300 in charges for unauthorized write attempts. In configuring his S3 storage, he used the same name as the Loki Helm chart default. This created a data path within AWS that Loki clusters could attempt to write to if those clusters were installed via Helm using the default bucket configuration. According to Maciej, he later made his bucket world-writeable for 30 seconds and collected roughly 10GB of data. He reached out to companies that were the source of this data, but at the time he published his blog post, he had not heard back.
AWS has since announced a billing change for Amazon S3, and no longer charges for any unauthorized PUT requests from outside their AWS account or AWS organization.
Prior to this point, any Loki or GEL deployments running with these default bucket names on AWS didn’t have a valid write target. These clusters could query locally cached data only, which would have made the cluster largely useless. As such, we have relatively high confidence that no one would have run an unconfigured cluster for long, not even as a trial.
Impacted versions
Loki Helm charts for Grafana Loki and Grafana Enterprise Logs from 3.0.0 until 5.47.2, and additionally 6.0.0
As a functional check: The default buckets in the Helm chart are not configured for global writes, so clusters in this misconfigured state can only return data from the last two hours. As such, clusters that return more than two hours of data are unlikely to be impacted.
Patched versions
Loki Helm charts for Grafana Loki and Grafana Enterprise Logs version 5.48.0 and 6.1.0 have been patched.
Solutions and mitigations
There are several solutions:
Upgrade the Loki Helm chart to version 5.48.0 or to 6.1.0 or higher.
Configure your storage: In your
values.yaml, change all that apply to your instance to the below:loki.storage.bucketNames.chunksaway fromchunksloki.storage.bucketNames.ruleraway fromrulerloki.storage.bucketNames.adminaway fromadmin
The upstream fix was to no longer provide default values in the Loki Helm chart. The Loki Helm chart will only configure MinIO default values when the user spins up a MinIO instance.
For contracted customers, we have reached out to help mitigate any issues.
Timeline
All times in UTC
2024-04-09 15:34: Maciej Pocwierz reported to Grafana Labs via our bug bounty program.
2024-04-09 17:04: Report triaged.
2024-04-09 17:04: Escalated to the Loki team to investigate and confirm.
2024-04-09 19:27: Incident declared.
2024-04-09 20:32: Partial replication by Loki team.
2024-04-09 22:35: PR merged to Loki Helm chart version 6.1.0 fixing the issue.
2024-04-09 22:35: Loki Helm chart version 6.1.0 released.
2024-04-10 19:01: AWS and Google Cloud contacted.
2024-04-11 06:24: AWS confirmed all unauthorized third party requests other than 5xx are billed.
2024-05-03 01:28: Google Cloud confirmed failed requests are not billed.
2024-05-06 15:43: PR merged to Loki Helm chart version 5.48.0 fixing the issue.
2024-05-06 15:43: Loki Helm chart version 5.48.0 released.
2024-05-08 14:20: Google Cloud confirmed that the buckets are claimed by legitimate entities and have been configured in a way that prevents data collection.
2024-05-13 17:30: AWS made a change to Amazon S3 in which unauthorized requests that customers did not initiate are free of charge.
2024-05-31 17:43: AWS confirmed that the buckets are claimed by legitimate entities and have been configured in a way that does not allow for data collection.
2024-06-27 05:30: All cloud partner reviews of this blog post finished.
2024-06-27 17:00: Blog published.
2024-06-27 19:00: Blog updated to reflect changes from our partners at Google.
Reporting security issues
If you think you have found a security vulnerability, please go to our Report a security issue page to learn how to send a security report.
Grafana Labs will send you a response indicating the next steps in handling your report. After the initial reply to your report, the security team will keep you informed of the progress towards a fix and full announcement, and may ask for additional information or guidance.
Important: We ask you to not disclose the vulnerability before it has been fixed and announced, unless you received a response from the Grafana Labs security team that you can do so.
Security announcements
We maintain a security category on our blog, where we will always post a summary, remediation, and mitigation details for any patch containing security fixes. You can also subscribe to our RSS feed.
