How the Jsonnet-based project Tanka improves Kubernetes usage

At FOSDEM 2020, Grafana Labs software engineers Tom Braack and Malcolm Holmes explained how and why the team developed Tanka, a scalable Jsonnet-based tool for deploying and managing Kubernetes infrastructure.

They also shared how Grafana Labs leverages the project to manage and monitor its own infrastructure as well as showcased how Tanka makes deploying a Grafana instance faster and more efficient.

Why YAML was never the answer

YAML is not every developer’s favorite language. (Or in the words of Braack: “YAML sucks.")

YAML is basically laid down like data; therefore, “it’s not much more clever than a sheet of paper. You can’t use any cool tricks that fancy programming languages can do,” said Braack. “It did not completely fit our needs.”

Case in point, Braack broke down a Kubernetes deployment for Grafana with YAML for the crowd gathered at FOSDEM, where Grafana Labs engineers also presented talks on configuring Grafana as code and the secret history of Prometheus histograms.

Braack began with walking through the equivalent of a Docker run command. “Most of these lines aren’t actually expressive at all,” said Braack. “We have the word Grafana four times on the screen. Why?!”

To add to that, an organization rarely deploys objects only once. Instead there’s a development environment and a production environment. YAML is limiting because it doesn’t allow engineers to utilize the same code for development and for production, even though they are virtually the same. The only thing that can be done is to duplicate the entire directory.

“Then as soon as you want to add something, you need to propagate these changes to all of the directories,” said Braack. If changes slip through the cracks, that could lead to a severe drift in your configurations.

“Imagine being on call at 3 a.m. and needing to fix something. First you would need to propagate all the fixes you’ve applied previously to other environments that have not been done yet,” pointed out Braack.

So that raises the question: What can be done about it?

Adopting Jsonnet

Grafana Labs looked around for previous attempts at adding abstraction to Kubernetes. Helm did it with templating but that didn’t work for the team because with string substitution on YAML – which is already a fragile syntax – it proved hard to read.

Overall, Helm was “too distant from what actually matters to us,” said Braack.

Enter Jsonnet from Google, which is basically JSON but with enhancements such as comments, variables, hidden fields, and a sizable set of common language features.

Some of the most useful aspects of Jsonnet include:

Functions

Functions work by taking arguments and returning any type of JSON object such as a dictionary, an array, an integer, or a boolean.

Patches

A feature that’s natively integrated into the language is patching.

If there were missing values or labels in an existing YAML file, users cannot add them in after the fact. But in Jsonnet, patches can be formatted to solve that problem.

“We don’t want to overwrite metadata and labels completely but just inject a single change in there. So you can modify down the tree without affecting others, which really speaks to the flexibility of Jsonnet,” said Braack. “So if something is missing from the library, the library does not let you down completely. Instead you just format these patches.”

Imports

One of the biggest issues when using YAML is that as soon as a file gets big, it’s hard to keep up with it. Where is something defined? Why do we need to define things multiple times? Why do we need to copy files all the time for different environments?

On the other hand, Jsonnet supports imports. Consider a file called common.libsonnet with the content shown in the slide below, another file main.jsonnet, and another one that imports it and extracts the labels subkey. The Jsonnet runtime then just copy-pastes the content into the correct location on the object.

“This can be used to reduce a big wall of code so that it’s nearly as concise as if it was Docker run again. You specify the name, specify the image, and you get all of these fields filled in and it works out,” said Braack. “I don’t need to copy my entire YAML file anymore only for prod. I could format a library lib/grafana and import it. And I just need to add my prod rules to the Ingress and I’m covered.”

Packages

“Another cool feature is packages,” said Braack. “In Jsonnet, it is not only about having it all locally on your computer. There’s also a package manager available so you can actually share your Jsonnet files on the Internet so other people can make use of them.”

“You can build proper libraries, there can be shared projects, and others can build their own libraries, which can just be installed and used,” Braack explained.

Which is exactly what Grafana Labs has done. “We have Jsonnet libs that contain, for example, memcached. There’s also a library for Loki available,” said Braack. “We have modules for certain applications. We also have mixins that include Grafana dashboards, bots, and alerts. All of this allows you to install an application to Kubernetes that is monitored out of the box, which I think is quite cool.”

How Tanka works

Malcolm Holmes then put together a demo of how to use Tanka to deploy a Grafana instance.

The demo started with an empty k3d cluster as a simple Kubernetes infrastructure and a default namespace that doesn’t have any pods. With these parameters, Holmes demonstrated how to easily deploy Grafana with a sample dashboard using Tanka.

“A Grafana instance with a dashboard can be deployed using 30 lines of Jsonnet instead of a bunch of YAML files – and the whole thing can be done in about seven minutes,” said Holmes, who has also outlined how to connect the ksonnet-prometheus library to the Grafana Cloud to manage multiple clusters and long-term storage needs.

How you can get started

For anyone interested in trying out Tanka, go to https://tanka.dev and follow a tutorial. “It really walks you through and shows how to use it in depth,” said Braack.

Once you are up and running with Tanka, share your experience with the Grafana community on Twitter or give the project a star on Github.