Hacking for better observability at scale: How Geotab quickly improved their monitoring with Grafana

Hackathons are a fantastic way to experiment with new ideas and build projects quickly. In fact, we run several hackathons every year inside of Grafana Labs, and many of our coolest features started out as hackathon projects. 

I always love seeing what people create. And it’s even more fun when I get to see what other organizations are hacking on!

In June of this year I had a fantastic opportunity to assist a Grafana-focused hackathon run by the engineers from Geotab, an AI-based platform that tracks and manages location and telemetry data for fleets of vehicles. (Think car rental companies, construction companies, delivery services—anyone that has a large number of vehicles to keep track of and make sure they’re in top condition at all times.)

As part of the Developer Advocacy team here at Grafana Labs, I often speak with people who use our software. But it’s rare that I get a front row seat to watch a team build their newest iteration of an observability platform at this scale!

Read on to learn more about how Geotab ran the event and for some ideas you can apply in your own hackathons!

The idea

Geotab has been a Grafana user for several years, and they reached out to us for some guidance on the project. It was structured as a three-day intensive effort with six teams of engineers focused on improving their existing in-house monitoring and finding new ways to work with their data.

My colleague Colin Wood on the Solutions Engineering team joined for the first day, which was all about learning. He presented to a team of around 30 engineers on topics like efficiently structuring and querying Prometheus data, as well as my personal favorite topic: taking your dashboards from blah to beautiful.

Days Two and Three were where the Geotab engineers got down to work. The teams came up with ideas and a plan to implement them in just 16 working hours—the pressure was intense!

Colin and I went from team to team, answering questions and providing some suggestions on visualization types and efficient queries. I was impressed by the seriousness and level of effort put in by the teams! 

Around half of the hackathon participants focused on existing dashboards and alerts, finding ways to make dashboards easier to read and alerts less noisy. The other half picked new data sources that weren’t being displayed in their existing monitoring, which gave them an opportunity to really experiment with dashboard structure and data representation. 

I loved seeing teams start with defaults like the time series panel only to realize that there were other ways to present the data that gave a more intuitive understanding of what they were looking at.

By lunchtime on Day Two it was clear that the competition would be intense. The judging would happen in the afternoon, so teams sat together and continued discussions while eating. They only had a couple of hours to finish up their projects and prepare to present.

Judging

Finally, it was time to see what everyone had built. The teams all gathered in a conference room to present their projects. In order to heighten the tension a bit (and give everyone’s fingers a few minutes of rest from all the hacking) I gave an overview of the Grafana plugin system, talking through how plugins format data from any source on one side and display it however you choose on the other.

Watching the groups present was a treat! I was constantly reminded of how seriously everyone there took both the hackathon itself, but also data integrity and security. It was clear that everyone involved cared about their work and their customers.

Teams each took their own individual approaches to presenting. In the short time that they had, some had managed to put together slides with problem statements and architectural overviews. Others dove right into Grafana to show how dashboards would be used in production. One team even used AI to code a couple of small HTML5 games that could be embedded in a dashboard (so you have something to do while you’re watching the graphs on a long-running process, of course).

In the end, there were two winning teams. One was selected by Geotab managers with an eye toward the biggest improvement to existing workflows. I was also privileged to select the other winner. And while I didn’t know much about Geotab’s inner workings, the most beautiful and functional dashboard design got the prize from me.

The outcome

It’s always fun to see how other organizations handle hackathons like this.  Geotab did a great job structuring the event in ways that can be applied almost anywhere:

• Keep teams small (4-6 engineers).  Having too many people on one project means that you end up with too many ideas, which makes it really difficult to focus on any one idea in a short timeframe.
• Give the participants the resources they need.  Let teams fully focus on the hackathon for a few days. If they’re distracted by production issues or an on-call shift, they won’t have the space to really let loose and engage with a problem.
• Have clear guidelines. In Geotab’s case, focusing on improving monitoring and incorporating Grafana gave everyone a direction and focus, enabling them to solve real problems while still giving them some creativity.

I’m grateful for the experience and the welcome shown by the Geotab team. (If they ever need more Grafana stickers, all they have to do is ask.) But the best part of all was getting to see the results of the hackathon on the big screens in their operations center!

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!