Blog

As the Grafana Labs team pushes the Loki project toward general availability, we will be doing deep dives on its key features. Today, we’re focusing on the loki-canary project.

Meta tags are coming soon for Metrictank. Here's how to use them.

As the Grafana Labs team pushes the Loki project toward general availability, we will be doing deep dives on its key features. Here are the first two: the Docker logging driver plugin and support for systemd.

Coming Soon: Seamless and Cost-Effective Meta Tags for Metrictank One of the major projects we’re working on for Metrictank – our large scale Graphite solution – is the meta tags feature, which we started last year and are targeting to release in a few months. A lot of people don’t realize this, but Graphite has had tag support for more than a year. Our mission with Metrictank is to provide a more scalable version of Graphite, so introducing meta tags was a logical next step.

This summer I had the opportunity to present my practical fault detection concepts and hands-on approach as conference presentations. First at Velocity and then at SRECon16 Europe. The latter page also contains the recorded video. If you’re interested at all in tackling non-trivial timeseries alerting use cases (e.g. working with seasonal or trending data) this video should be useful to you. It’s basically me trying to convey in a concrete way why I think the big-data and math-centered algorithmic approaches come with a variety of problems making them unrealistic and unfit, whereas the real breakthroughs happen when tools recognize the symbiotic relationship between operators and software, and focus on supporting a collaborative, iterative process to managing alerting over time.

Collectd is a program that you can run on your systems to gather statistics on performance, processes, and overall status of the system in question. When you send these statistics to a time series database like Graphite, you’ll need some way to access and visualize all that data – after all, if you collect all that data but don’t have any way to use or access it, it’s not going to do you a whole lot of good.

A while back I read coders at work, which is a book of interviews with some great computer scientists who earned their stripes, the questions just as thoughtful as the answers. For one thing, it re-ignited my interest in functional programming, for another I got interested in literate programming but most of all, it struck me how common of a recommendation it was to read other people’s code as a means to become a better programmer.

Go, while not the only part of our toolkit here at raintank, is an integral part of many of our projects. Let’s spend a little bit of time exploring where and why we use it. Where we use Go grafana - While a lot of Grafana’s code is javascript, Go powers Grafana’s backend. raintank-metric - This worker, which processes metrics received by Grafana and puts them into the backend TSDB, is written in Go.

The raintank software stack is comprised of a number of different components that all work together to deliver our platform. Some of our components include: front-end visualization and dashboarding (based on Grafana) network performance data “collectors” event and metric processing and persisting core RESTful API To support these services we have a number of shared services including: Elasticsearch InfluxDB MySQL RabbitMQ Redis As the stack has grown it has become increasingly more difficult to manage our development environments.