Now GA: Cortex blocks storage for running Prometheus at scale with reduced operational complexity

Published: 6 Oct 2020

We’ve just launched Cortex 1.4.0, one of the most significant releases of 2020. The big headline: The new blocks storage engine has exited the experimental phase and is now marked as Generally Available.

Blocks storage aims to reduce the operational complexity and costs of running a Cortex cluster at scale. In particular, it removes the dependency from a NoSQL database to store series indexes, and only leverages object storage (e.g., S3, GCS, or Azure blob storage) to store all the ingested series, without compromising reliability, scalability, or performance. Removing this dependency should significantly reduce the cost to run Cortex, as NoSQL databases tend to be quite expensive to run compared to object storage – whether you are paying a cloud provider to run it for you, or running on premise.

In blocks storage mode, the Cortex architecture looks like this:

If you’re familiar with Cortex, you’ll notice that the general architecture remains unchanged, with only the addition of two services: store-gateway and compactor.

After successfully running blocks storage in production for three months, Grafana Labs is currently completing the migration of all production clusters from chunks to blocks storage. You can read more about our experience with blocks storage at Grafana Labs in this blog.

Other notable developments in Cortex 1.4.0 include:

  • Cassandra marked as GA.
  • Support for Redis sentinel and Redis cluster to the Redis caching backend introduced.
  • Major improvements to ruler and alertmanager.
  • Shuffle-sharding support introduced to store-gateways blocks sharding. This allows tenant blocks to be spread across a subset of store-gateway instances, which limits the potential impact that can be introduced by any single tenant’s workload. A lot more about shuffle-sharding will come in the next release.

For more details, please read the Cortex 1.4.0 release notes.

Finally, our recently released commercial offering Grafana Metrics Enterprise, which adds some great enterprise-specific features on top of Cortex, will of course include all of these new changes in a future release, as it brings along with it the full upstream Cortex.

Related Posts

The Cortex community spent almost a year working on removing the need to run an index store and instead storing all the time series data as blocks in the object store.
The Technical Oversight Committee of CNCF voted to move Cortex up from sandbox to incubation, an indicator of project maturity.
At KubeCon + CloudNativeCon EU, Grafana Labs software engineer Marco Pracucci discussed the new experimental Cortex blocks storage, how it can reduce the Cortex operational cost without compromising scalability and performance, and lessons learned from running Cortex at scale.