Inside Loki’s new architecture for faster logging at petabyte scale

The next generation of Loki is coming, with big architectural changes to make it the most performant open source logging database available and a new storage engine for faster insights. 

In this session, Loki engineers Poyzan Taneli and Trevor Whitney explain the new architecture designed for petabyte-scale logging. They discuss the benefits of the key changes, which include: a new ingestion path backed by Kafka that effectively achieves replication factor 1 (RF-1) durability, a redesigned query engine, a new scheduler, and a new data format. All of these developments are aimed at enabling significantly higher performance and scalability at large log volumes.

Plus, as structured logging and OpenTelemetry adoption grow, users are running more analytical queries over massive volumes of increasingly high cardinality data. To help with that, Loki team members Ben Clive and Jason Nochlin will dive into the open source database's new storage engine and data format, and the design decisions behind it that enable faster large-scale scans, minimize the impact of stream cardinality, and improve performance for analytical workloads. They'll share results from benchmarks to show where Loki is already faster, where performance is comparable, and where the team is still pushing forward.