Grafana Mimir: 3 reasons to run the TSDB for Prometheus on bare metal

Wilfried Roset is an engineering manager who leads an SRE team and he is a Grafana Champion. Wilfried currently works at OVHcloud where he focuses on prioritizing sustainability, resilience, and industrialization to guarantee customers satisfaction.

Whether it’s for efficient resource allocation, flexibility, high availability, or scalability, it makes a lot of sense to run Grafana Mimir on Kubernetes—but it’s not the only way to deploy Mimir.

Mimir is Grafana Labs’ horizontally scalable, highly available, multi-tenant TSDB for long-term storage for Prometheus metrics, and in this blog post I’ll cover three reasons why you might want to run it on virtual machines and/or bare metal instead.

You don’t have Kubernetes

The most obvious reason you wouldn’t deploy Mimir on Kubernetes is that you don’t have a Kubernetes cluster at hand, so you cannot deploy Mimir—or anything else—on it. If you are in that camp, you might want to use Puppet or Ansible to deploy your TSDB. Similar to the guidance in the official Grafana documentation, you could install Mimir in a few commands on a Debian-based system:

Copy

sudo apt-get update
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install mimir

For a RHEL-based system, run this command (the full instruction can be found in the corresponding documentation):

Copy

sudo dnf install wget
wget -q -O gpg.key https://rpm.grafana.com/gpg.key
sudo rpm --import gpg.key
sudo bash -c 'cat << EOF > /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF'
sudo dnf install mimir

You want to break circular dependencies

Depending on your information system and organization, you could be in a position where you’re the service provider to many teams, and you provide an internal metrics service to the Kubernetes/platform team. 

In this scenario, you probably have at least one critical circular dependency—the ops team running the Kubernetes cluster rely on their metrics, which are stored in the service you provide. An outage on the Kubernetes cluster, even a small one, can have a severe impact on your deployment, which then impacts access to the metrics. 

One way to break this circular dependency is to run at least two Kubernetes clusters and two Mimir clusters:

• The Kubernetes cluster-A hosts Mimir cluster-1.
• The Kubernetes cluster-B hosts Mimir cluster-2.
• Mimir cluster-1 hosts the metrics of Kubernetes cluster-B.
• Mimir cluster-2 hosts the metrics of Kubernetes cluster-A.

With this deployment approach, you have broken the circular dependencies, though you have increased the number of Kubernetes clusters run by your ops team. Be mindful that cross-Kubernetes outages exist and you should take that into account when designing your information system.

Another way to break these circular dependencies is to not run Mimir on Kubernetes at all. In that case, I’d recommend following the process outlined in the “You don’t have Kubernetes” section above. 

You want to learn the hard way

Finally, you might want to run Mimir outside of Kubernetes, just to learn how it works the hard way. 

Kubernetes offers so many features, integrations, and automation that it can sometimes be mistaken for magic. But with a bit of free time you can learn a lot about software without the Kubernetes magic; Mimir is not different. 

If you are looking for your next metrics TSDB, I recommend spending some time looking at all aspects of your software. You could begin by following the getting started guide, then evaluate your critical dependencies. Once you are ready, you can deploy Mimir on Kubernetes or on bare metal. If you choose to deploy Mimir on bare metal and you manage your configuration with Puppet like I discussed earlier, we have the module for you.

Running Mimir on Kubernetes vs. bare metal

While running Mimir on Kubernetes can bring many benefits, there are also some potential drawbacks and considerations to be aware of. Ultimately, the decision depends on your specific needs, resources, and expertise. 

Weigh the benefits against the potential drawbacks and consider a proof-of-concept deployment to determine the best approach for your use case. More specifically, make sure to have a closer look at the pros and cons of bare metal deployments:

Pros:

• Can be deployed without a Kubernetes cluster
• Allows for breaking of circular dependencies
• Opportunity to learn about the software without relying on Kubernetes features and automation

Cons:

• May require more manual configuration and management
• Could increase the complexity of deployment and maintenance

Want to share your Grafana story and dashboards with the community? Drop us a note at stories@grafana.com!