Blog

This summer I had the opportunity to present my practical fault detection concepts and hands-on approach as conference presentations. First at Velocity and then at SRECon16 Europe. The latter page also contains the recorded video. If you’re interested at all in tackling non-trivial timeseries alerting use cases (e.g. working with seasonal or trending data) this video should be useful to you. It’s basically me trying to convey in a concrete way why I think the big-data and math-centered algorithmic approaches come with a variety of problems making them unrealistic and unfit, whereas the real breakthroughs happen when tools recognize the symbiotic relationship between operators and software, and focus on supporting a collaborative, iterative process to managing alerting over time.

Collectd is a program that you can run on your systems to gather statistics on performance, processes, and overall status of the system in question. When you send these statistics to a time series database like Graphite, you’ll need some way to access and visualize all that data – after all, if you collect all that data but don’t have any way to use or access it, it’s not going to do you a whole lot of good.

A while back I read coders at work, which is a book of interviews with some great computer scientists who earned their stripes, the questions just as thoughtful as the answers. For one thing, it re-ignited my interest in functional programming, for another I got interested in literate programming but most of all, it struck me how common of a recommendation it was to read other people’s code as a means to become a better programmer.

Go, while not the only part of our toolkit here at raintank, is an integral part of many of our projects. Let’s spend a little bit of time exploring where and why we use it. Where we use Go grafana - While a lot of Grafana’s code is javascript, Go powers Grafana’s backend. raintank-metric - This worker, which processes metrics received by Grafana and puts them into the backend TSDB, is written in Go.

The raintank software stack is comprised of a number of different components that all work together to deliver our platform. Some of our components include: front-end visualization and dashboarding (based on Grafana) network performance data “collectors” event and metric processing and persisting core RESTful API To support these services we have a number of shared services including: Elasticsearch InfluxDB MySQL RabbitMQ Redis As the stack has grown it has become increasingly more difficult to manage our development environments.