Lessons from that security incident when everything went wrong (but ended up right)

April 26, 2025, is a date the Grafana Labs security team won’t forget. Internally, it needs no explanation: “The Incident” is enough.

In this talk, David Andersson and Nick Moore walk through a real security incident response, from first alert to resolution, and how open source tooling and open collaboration shaped every step. It started with a Saturday morning alert from canary tokens, turning a quiet weekend into an immediate investigation.

They'll explain how a misconfigured GitHub Actions workflow led to unauthorized access to CI credentials, and how the team used open source security and observability tools to understand what happened and how far it went. Logs in Loki, incident coordination with Grafana Cloud IRM, credential scanning using Trufflehog, and workflow auditing using both Gato-X and Zizmor allowed them to trace activity, coordinate response, rotate tokens, and verify that no customer data or systems were impacted.

A key aspect of the incident response was the fact that the team was in fact working in the open. Instead of waiting to communicate with the open source community after the fact, they collaborated directly with maintainers and contributors during triage and validation. Open tools, shared context, and public artifacts helped the Grafana Labs security team move faster and with more confidence.

This is a candid look at what happens when things go wrong in complex, open systems. It’s also the story of how preparation, openness, and trust in open source tooling meant that the team got to write the “no customer impact” post.