Post-incident review for TanStack npm supply chain ransom incident: No unauthorized access to customer production systems

On May 27, we completed our internal investigation of the recent TanStack supply chain ransom incident and confirmed our initial findings: The incident was strictly limited to Grafana Labs' GitHub environment. There was no unauthorized access to customer production systems, and the Grafana Cloud platform was not affected. 

For an additional, independent audit, we engaged Mandiant, a leader in cybersecurity and incident response. We provided them with API access to Grafana Labs' log environment to conduct queries across our systems for their investigation, which started on June 1. Mandiant confirmed that there was “no evidence of code tampering or repository poisoning within public organizations or production repositories delivered to end users.” 

Since we discovered the incident, the Grafana Labs security teams have been running two parallel workstreams: completing the investigation and hardening our security operations. We are publishing this blog in the spirit of transparency to share more details about our incident response and remediation efforts.

Summary and impact 

If you’re looking for the short version instead of reading our previous updates, here is the TL;DR: The TanStack supply chain attack hit us on May 11 via the Mini Shai-Hulud campaign. At the time, we believed we had successfully rotated every credential involved in this incident. We missed one. I won’t blame this oversight on hubris; the data we had at the time simply led us to believe our rotation was exhaustive. We were mistaken.

A bad actor utilized that overlooked credential to clone our entire repository collection. They then reached out on May 16, demanding a ransom to prevent a code leak.

Since Grafana Labs is an open source company, you might wonder why this is a concern. While most of our source code is public, we do maintain private repos for things like internal tools and specific Grafana Cloud features. It was a heavy decision, but we stuck to our principles and the FBI’s documented guidance: We did not pay. 

We launched our mitigation efforts immediately, and we confirmed that there was no unauthorized access to customer production systems, and the Grafana Cloud platform was not affected. We also confirmed that while our codebase was downloaded, it was not altered. Our customers and open source users do not need to take any action.

Grafana Labs’ response 

We were alerted to the incident on a Saturday, and teams across the entire company took action quickly and decisively. (Or to borrow a phrase from one my favorite rappers Big Daddy Kane, ain't no half-stepping at Grafana Labs.)

In response, Grafana Labs suspended all GitHub applications on May 17, initiated a global code freeze on May 18, and conducted a cross-platform audit of Vault, GitHub, Okta, Kubernetes, AWS, GCP, and host logs to verify that no production customer data was compromised.

In the weeks following, our engineering teams contributed to a comprehensive audit that included but was not limited to:

• Completing 1,500 security-focused PR reviews
• Auditing 280 GitHub applications, stripping permissions and removing several
• Scanning 1,200 repositories for any signs of tampering
• Executing 2,300 PR reviews looking for unauthorized changes in a single critical repo
• Finishing infrastructure audits and retiring legacy systems
• Performing wide-ranging new access audits

It was a massive undertaking, but each team stepped up in an extraordinary way to do their part. Engineering, security, and cross-functional partners worked tirelessly to respond, demonstrating the collaboration and the shared commitment we have to our community and our customers that I have always valued here at Grafana Labs. 

After the initial assessment, we found that in addition to source code, the downloaded content included GitHub repositories that some Grafana Labs teams use to collaborate on and store internal operational information and other details about our business. This includes, for example, business contact names and email addresses that would be exchanged in a professional setting and email addresses that were used in some past marketing campaigns. This was not information pulled from or processed through the use of production systems or the Grafana Cloud platform. 

If you wish to know if email addresses with your domain were identified, please reach out to Grafana Labs support. 

Incident timeline

All times are in UTC 

• 19:21 11 May - First malicious code executed on self-hosted runners by Shai Hulud threat actors, leaking credentials. Rotated credentials.
• 07:21 14 May - First malicious commit made by the threat actor using grafana-delivery-bot, leaked from Shai Hulud attackers.
• 13:28 14 May - Data exfiltration of repos begins.
• 20:57 15 May - Data extortion threat actor publishes their extortion demand.
• 08:30 16 May - Grafana Labs security team becomes aware of the claimed ransom and begins seeking confirmation.
• 17:39 16 May - Compromise confirmed; incident declared.
• 19:33 16 May - All known affected credentials and GitHub applications suspended/rotated. Suspension and rotation of all other GitHub applications and accessible credentials begins.
• 21:10 16 May - Suspension of all GitHub applications completed.
• 16:40 17 May - All code changes made by GitHub application accounts associated with the threat actor identified and reverted.
• 16:52 17 May - Root cause, attack chain of compromise identified.
• 17:21 17 May - DockerHub credentials determined not compromised.
• 17:51 17 May - All malicious workflow runs identified. Final list of affected secrets compiled and rotated. Rotation of all other ci/common secrets from affected repos continues. 
• 23:23 17 May - Last of the potentially accessible credentials confirmed rotated or suspended.
• 03:08 18 May - Begin freeze of all non-critical code and deployment changes.
• 08:00 25 May - All-engineering security hardening week commences.
• 10:58 26 May - Commit review completed, service thawing begins. A repository needs to have been fully reviewed and transitioned to use a GitHub application token broker for short-term, finely-scoped credentials before being thawed. 
• 10:54 27 May - Transition from repos directly pushing images to DockerHub to pushing to Google Cloud Artifact Registry occurs.
• 27 May - Internal investigation complete. No additional attack activity or compromised credentials were discovered. 
• 08:00 2 June - All-engineering security hardening week concludes. 
• 20:43 3 June - Review of repositories for data loss completed. 
• 18 June - Mandiant investigation completed, corroborating internal investigation. 

What’s next 

The investigation is now closed, but our work to improve security operations at Grafana Labs will continue. Dostoyevsky once noted that "when reason fails, the devil helps!" I’m quoting “Crime and Punishment” to underscore our philosophy: We only wanted to implement changes that actually moved the needle on security. 

We’ve spent the past month executing high-impact controls, including a token broker, fine-grained access controls, additional alerting, and static analysis. In addition, we have moved off of certain GitHub Actions and now use more tightly scoped actions with short-lived tokens. 

We have also started the process of compartmentalizing our GitHub organizations and isolating all archived repos into a dedicated organization with actions disabled.

We will share an overview of our response efforts and the technical details of how we improved our security posture from our post-incident review in the coming weeks.