Milliseconds Matter: How Stack Overflow uses Grafana to Optimize their Systems.
Stack Overflow's custom monitoring tool Opserver provided purpose-built dashboards, but lacked a way to easily create custom, self-service dashboards.They needed a tool to allow their developers and SRE teams the freedom to quickly create custom-tailored dashboards to visualize data from OpenTSDB, Elasticsearch and their alerting tools, all in the same experience. They wanted to empower the teams that knew the data the best.
Stack Overflow's Ad-server team was the first to discover Grafana. They were searching for a tool to create custom server latency dashboards. Displaying ads on a website is latency sensitive, in which a millisecond delay can have a huge impact on revenue. Server latency also affects which ads are displayed to whom on the site. The quicker the ad is served, the more targeted it can be for the user. Grafana's real-time dashboards were critical in discovering where the Ad-Server team could optimize to have the best server performance possible. Grafana quickly spread from the Ad-Server team to other teams at Stack Overflow since it can visualize data from many different data sources, both open source and commercial. For Stack Overflow, this meant OpenTSDB data could be visualized alongside Elasticsearch data, which could be viewed alongside their custom alerting data from Bosun.
The Bosun Alerting System and a New Grafana Plugin
Bosun is an open source alerting system Stack Overflow created which has an expressive domain-specific language for evaluating alerts and creating detailed notifications. It also tests alerts against history for a faster development experience. Bosun is robust, but comes with a complex user interface and an often steep learning curve. Its visualization options are also limited. In Kyle's recent GrafanaCon talk: "The Culture and Realities of Monitoring at Stack Overflow," he described monitoring as "a medium for humans to communicate with other humans through machines". The Bosun project reflects a deep understanding of the impact of alerting on culture, which Kyle is extremely sensitive to. An intuitive UI and consistent user experience is key to making complex systems easier to understand – something Grafana has always prioritized. So the team decided to build a plugin to bring the power of Bosun into Grafana's user friendly interface.
"Grafana's plugin architecture allowed the Stack Overflow team to create a data source plugin for Bosun to visualize Bosun alerting data directly in Grafana." Kyle noted, "Since Grafana is more user friendly, people tend to pick it up more naturally. We have turned Grafana users into Bosun consumers, and soon hope to turn Bosun consumers into Bosun authors."
The plugin also allows teams to use the Bosun expression language inside Grafana to achieve visualizations not previously possible. Grafana can also display annotations created from Bosun, adding valuable context to various metrics behavior. Showing relevant alerts directly on Grafana dashboards also provides the benefit of having to look in fewer places for information. This consolidation provides actionable insight at a critical moment – right when they're viewing the data. The Bosun team didn't keep this new plugin for themselves, it is freely available for download at Grafana Labs' Plugin Repository (grafana.com/plugins).
"Stack Overflow is focused on the audience and placing yourself in their shoes. Think of what they know and what they might not know. Don't tell them what they need to know, show them what they need to know on a Grafana dashboard."
Grafana allows teams across Stack Overflow to quickly and easily build custom self-service dashboards for what's important to them, no matter where the data lives or which database it's stored in. Because Grafana is open source and has a robust plugin architecture, the Bosun team was able to create a plugin to leverage their powerful alerting system, and can now visualize the data in new ways. The new plugin empowers users new to Bosun to write queries and set alerts directly from Grafana's UI, as well as the flexibility to leverage Bosun's native expression language. With the popularity of the Bosun plugin internally, the team shared their plugin with the entire Grafana community, which has been installed thousands of times. by users of both projects.