Grafana Tempo is now GA with the release of v1.0

It’s exciting to see a project that you’ve poured so much time into progress at the rate Tempo has. Tempo is not the first piece of software I have shepherded from the very first line of code to a production release, but it is the first large-scale open source project I have led. Working with a community that is able to use and improve your software as a community is a powerful thing. For this reason, I am very proud to announce that Tempo has been promoted from beta to GA and Tempo 1.0.0 is released!

Performance at scale

We announced Tempo at ObservabilityCON in October 2020. At the time, I was quite excited to announce our new distributed tracing project was consuming ~150k spans/second. It felt like such an accomplishment from our previous ~10k spans a second, but I suppose everything is relative.

Currently we are seeing sustained ingestion of greater than 2M spans and nearly 350 MB per second. All of this data is batched, compressed, and shoved into GCS for retrieval later. If you search Tempo at Grafana for a traceID, your search will cover 136TB of compressed trace data and over 79 billion traces, and your response will be returned with a p50 of 1.75s.

Impact and future

The past 8 months of Tempo have been all about stability and scale. Compression, WAL replay, ingester performance, exhaustive search, and on and on. As a result, Tempo has really connected with power users that deal with huge trace volumes as well as those who appreciate its operational simplicity. The desire to store more trace data at a lower cost resonates with those whose trace volumes reach into the tens or hundreds of thousands of spans/second. The outreach has been incredible, and some days I am inundated with requests to talk about Tempo and the impact it can have.

Next we intend to start adding the features that make Tempo more attractive to the medium and small users. Currently Tempo requires discovering traces through logs or exemplars, but soon we intend to add native search to make the case to switch to Tempo even more compelling.

Keep an eye open for future announcements! And please tune into our GrafanaCONline session, Open source distributed tracing with Grafana Tempo, on June 10 for a deeper dive into Tempo.

Try out Tempo on Grafana Cloud

Along with the 1.0 release, we’re also announcing the general availability of Grafana Cloud Traces, which integrates traces with metrics, logs, and dashboards in our fully managed Grafana Cloud observability stack. Users can upload up to 50GB of traces per month at no cost in the free tier of Grafana Cloud. Sign up for a Grafana Cloud account to quickly get started with Grafana Cloud Traces.

Team

Let’s wrap this up by recognizing those who have put so much into this project. As the great coach Norman Dale once said:

Five players on the floor functioning as one single unit: team, team, team — no one more important than the other.

Annanay Agarwal (annanay25), Daniel Gonzalez (dgzlopes), Koenraad Verheyden (kvrhdn), Mario Rodriguez (mapno), and Marty Disibio (mdisibio) should all be proud of the work they’ve put in helping the community, responding to issues, and helping to build Grafana Tempo, an intensely high scale cost effective distributed tracing backend. There have been countless others providing PRs, issues, and just some chatter in our #tempo slack. All contributions are welcome!

Please join our community and say hello on Slack! Lower your tracing costs, increase your volume, and let’s all work to improve Tempo together!