Tiger teams: How we tackle urgent, cross-functional challenges at Grafana Labs

A year ago, we hit a wall. 

Our Grafana OSS releases were excruciating to execute. The process was confusing and hard to follow, security patches were non-trivial, and many engineering hours were lost to an overly manual process. We needed to move fast, cut through ambiguity, and pull in just the right people without waiting on roadmaps or org charts. That’s when I pitched the idea of forming the first ever “tiger team” at Grafana Labs, a small and focused group designed to tackle high-priority, cross-functional problems with speed and clarity.

A year later, tiger teams have greatly improved the release cycle process, and they’ve quickly become our go-to model for tackling urgent, high-impact issues that don’t fit neatly into a single team’s backlog. In this post, I’ll share what tiger teams are, examples of how we’ve used this model at Grafana Labs, hard lessons learned along the way, and my recommended best practices.

Whether you’re facing a deep technical challenge, landing a major deal, or just need a way to unblock cross-functional work, I hope our approach gives you a few ideas to take back to your own organization.

What is a tiger team?

The term “tiger team” originated at NASA in the 1960s during the Apollo 13 mission. In Walter C. Williams’ paper, “Program Management in Design and Development,” the NASA Manned Spacecraft Center engineer defined a tiger team as a “team of undomesticated and uninhibited technical specialists, selected for their experience, energy, and imagination, and assigned to track down relentlessly every possible source of failure in a spacecraft subsystem or simulation.“​

This approach was put to the test when an explosion in the spacecraft’s service module put the lives of the Apollo 13 astronauts in jeopardy. NASA’s flight director Gene Kranz assembled a group of specialists, later dubbed the tiger team, to find a way to safely return the astronauts to Earth. The success of this effort proved how powerful a temporary cross-functional team of experts can be when united by urgency and a singular focus.

At Grafana Labs, we use the tiger team model to tackle complex, time-sensitive challenges that our org structure is not currently set up to handle. Unlike traditional engineering teams, which are often organized around long-term product areas or services, tiger teams are temporary, goal-oriented groups formed to address specific issues. They are typically composed of three to five specialized engineers, and they last for one to two quarters, or until the targeted goal is reached. 

A Grafana Labs example: The Release Tiger Team

Looking back at that wall we hit a year ago with releasing Grafana, the formation of the Release Tiger Team marked a turning point. At the time the challenges we faced didn’t fall neatly within the scope of any single team. Ownership was ambiguous, and the expertise needed to fix our build and release pipelines was scattered across the org. 

We brought together engineers from different teams who had the right mix of experience in CI/CD, and deep familiarity with our systems. These were the engineers that had been vocal about the problems, already hacking at them on the side, and were best positioned to drive real change. We banded together, blatantly ignoring our current organizational structures to focus solely on the one unified goal of fixing our build and release pipelines. 

Our company had put trust in us that we could deliver on this goal in a single quarter. We were a team of just three senior engineers, with myself as lead. I soon realized we needed laser focus to achieve this on such a short timescale. I drafted a roadmap with three major milestones: simplify the release process, add the ability to patch our Grafana Cloud release channels directly, and fully automate our builds. With this plan in place, we got to work.

Tiger team, take 2

Of course, the problem with the best laid plans is that they never go exactly as expected. 

We tried our best to stay singularly focused on our goals, but there were still releases that needed our immediate attention. This included unexpected security releases that quarter, and those releases had their own unique issues that needed our expertise to fix. This took valuable time away from our project, but also served as a real-world user test that highlighted our most critical issues, and what was most urgent for us to address. 

We reorganized and rallied behind our first milestone to simplify our release process by reducing the amount of manual, and thus error prone, steps involved. Originally responsibilities were too fragmented, and no single team had clear ownership of the release process. 

Taking a critical look at who was actually doing the work, who understood the system best, and who had the drive to fix it, we restructured engineers from various teams into this singularly focused tiger team, cutting across squads and reporting lines while doing so. 

These engineers knew how the current release process worked, where steps could be combined, what documentation was missing, how we built and packaged the release artifacts, and where the pipelines most commonly failed. With this, we had the momentum we needed to move quickly and make meaningful progress.

As the quarter came to a close we had drastically reduced the manual steps involved in our release process. The engineers running the releases did so effectively and independently, and suddenly security releases were not so intimidating. We even made good progress on our other milestones, such as Grafana Cloud version patching. 

But then what? Our tiger team was set to dissolve at the end of the quarter, so what did that mean for the work still outstanding, or the tooling we had built? Who would own this moving forward? Thankfully, due to the success of the Release Tiger Team, we were able to advocate for a full-time dedicated team that would continue to iterate on our release issues and take full ownership in this area.

However, this is not always possible—or even advisable. Our team fit a specific, long-term need, so it made sense to keep it around. But if your tiger team has achieved its goal and no longer serves a purpose, then it can be dissolved and everyone can return to their normal responsibilities.

How can you run effective tiger teams?

Running effective tiger teams is a delicate act, and it requires some forethought. I can only speak from my experience with the Release Tiger Team, and other tiger teams we have since created. However, I did learn some valuable lessons in the process.

• Be strategic. Be more strategic than you even think you need to be. Prioritize, and reprioritize, and reduce your ambitions down to a single laser-focused goal. With this goal you should be able to say: If this is all we achieved, our mission was still successful.
• Get the right people. The power of tiger teams is that you can pull people to your cause regardless of the constraints of existing organizational and team structures. You have high priority needs, ambitious goals, and a short amount of time to execute on them. None of this is possible without the right people by your side.
• Define the end state at the beginning. When is the tiger team complete? When can the people involved go back to their original teams? What happens to the code, tooling, features, or processes that were created? Who will own these long-term? Don’t wait for the project to end to find this out. The sooner these can be laid out in the process the better. No one wants their project suddenly dropped or thrown over the wall at someone else.

In closing, I hope my own journey in this space has offered you some insight. Tiger teams are a powerful tool for rallying to solve critical failures, or land big deals, or address systemic issues that otherwise cannot be solved within the existing organizational structure of the company. 

We at Grafana have continued to lean into the power of tiger teams. Another of our tiger teams successfully landed a big Fortune 500 technology partner, and yet another is trailblazing greenfield opportunities that could expand our offerings. What can a tiger team accomplish at your company?

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!