This is documentation for the next version of Grafana Loki documentation. For the latest stable release, go to the latest version.
TSDB index format v2
Version 2 is Loki’s initial TSDB index format. It is written for periods with schema versions v9 through v12 and is the closest to the Prometheus TSDB index it was derived from. See
TSDB index format for the encoding notation used on this page.
File layout
+-----------------------------------------------------------+
| magic(0xBAAAD700) <4b> | version(2) <1b> |
+-----------------------------------------------------------+
| symbol table (TOC.symbols) |
+-----------------------------------------------------------+
| series (TOC.series) |
+-----------------------------------------------------------+
| label indices (TOC.label_indices) |
+-----------------------------------------------------------+
| postings (TOC.postings) |
+-----------------------------------------------------------+
| label offset table (TOC.label_offset_table) |
+-----------------------------------------------------------+
| postings offset table (TOC.postings_offset_table) |
+-----------------------------------------------------------+
| fingerprint offsets table (TOC.fingerprint_offsets) |
+-----------------------------------------------------------+
| TOC (last 76 bytes of the file) |
+-----------------------------------------------------------+Symbol table
All label names and label values are deduplicated into a lexicographically sorted symbol table. Series reference a symbol by its ordinal position in this table, which keeps series entries small. The index decoder keeps the file offset of every 32nd symbol in memory and scans forward from there.
+-----------------------------------------------------------+
| len <4b> |
+-----------------------------------------------------------+
| #symbols <4b> |
+-----------------------------------------------------------+
| symbol_0 <uvarint_str> |
+-----------------------------------------------------------+
| ... |
+-----------------------------------------------------------+
| symbol_n-1 <uvarint_str> |
+-----------------------------------------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+Series
Series are ordered by fingerprint, then by label set. Each entry starts on a 16-byte boundary, and the series reference used by the postings is the entry offset divided by 16.
+-----------------------------------------------------------+
| len(entry) <uvarint> |
+-----------------------------------------------------------+
| fingerprint <8b> |
+-----------------------------------------------------------+
| #labels <uvarint> |
+-----------------------------+-----------------------------+
| ref(name_0) <uvarint> | ref(value_0) <uvarint> |
+-----------------------------+-----------------------------+
| ... | ... |
+-----------------------------+-----------------------------+
| ref(name_k-1) <uvarint> | ref(value_k-1) <uvarint> |
+-----------------------------+-----------------------------+
| chunks |
+-----------------------------------------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+The stored fingerprint is not necessarily hash(labels). Multi-tenant indices carry a synthetic __loki_tenant__ label whose fingerprint must match the single-tenant fingerprint of the same stream, so the writer takes the fingerprint as an explicit argument.
Chunks
Chunk metadata entries are sorted by (min_time, max_time, checksum) and stored inline in the series entry. min_time is delta encoded against the max_time of the previous chunk, so it is written as a signed varint because chunks may overlap. The first chunk’s delta is relative to 0.
+-----------------------------------------------------------+
| #chunks <uvarint> |
+-----------------------------------------------------------+
| chunk_0: |
| min_time delta <varint> (vs. previous max_time) |
| max_time - min_time <uvarint> |
| KB <uvarint> (size, rounded to KB) |
| #entries <uvarint> |
| checksum <4b> (chunk identity) |
+-----------------------------------------------------------+
| ... |
+-----------------------------------------------------------+
| chunk_n-1: (same fields) |
+-----------------------------------------------------------+KB and #entries are Loki additions used by the query frontend to plan
dynamic query sharding. Because the metas form a single delta chain, computing statistics for a time range requires decoding chunks from the start of the chain.
Label indices
One entry per label name, holding the symbol references of all values seen for that name. Entries are 4-byte aligned.
+-----------------------------------------------------------+
| len <4b> |
+-----------------------------------------------------------+
| #names <4b> (always 1) |
+-----------------------------------------------------------+
| #entries <4b> |
+-----------------------------------------------------------+
| ref(value_0) <4b> |
+-----------------------------------------------------------+
| ... |
+-----------------------------------------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+Postings
One sorted list of series references per label name/value pair, 4-byte aligned. The pair ("", "") holds all series in the index.
+-----------------------------------------------------------+
| len <4b> |
+-----------------------------------------------------------+
| #entries <4b> |
+-----------------------------------------------------------+
| series_ref_0 <4b> |
+-----------------------------------------------------------+
| ... |
+-----------------------------------------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+Label offset table
Maps a label name to the file offset of its label index entry.
+-----------------------------------------------------------+
| len <4b> |
+-----------------------------------------------------------+
| #entries <4b> |
+-----------------------------------------------------------+
| entry_0: |
| #names <uvarint> (always 1) |
| name <uvarint_str> |
| offset <uvarint> |
+-----------------------------------------------------------+
| ... |
+-----------------------------------------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+Postings offset table
Maps a label name/value pair to the file offset of its postings list.
+-----------------------------------------------------------+
| len <4b> |
+-----------------------------------------------------------+
| #entries <4b> |
+-----------------------------------------------------------+
| entry_0: |
| #keys <uvarint> (always 2) |
| name <uvarint_str> |
| value <uvarint_str> |
| offset <uvarint> |
+-----------------------------------------------------------+
| ... |
+-----------------------------------------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+Fingerprint offsets table
A Loki-specific sampled index from fingerprint to series reference, recorded for every 1024th series. It lets a querier seek directly into the series section for a fingerprint range instead of walking the postings, which is what makes fingerprint-bounded shard queries cheap.
+-----------------------------------------------------------+
| len <4b> |
+-----------------------------------------------------------+
| #entries <4b> |
+-----------------------------+-----------------------------+
| series_ref_0 <8b> | fingerprint_0 <8b> |
+-----------------------------+-----------------------------+
| ... | ... |
+-----------------------------+-----------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+TOC
The table of contents occupies the last 76 bytes of the file. Readers parse it first and then address every other section by offset. An offset of 0 means the section is empty.
+-----------------------------------------------------------+
| symbols <8b> |
+-----------------------------------------------------------+
| series <8b> |
+-----------------------------------------------------------+
| label_indices <8b> |
+-----------------------------------------------------------+
| label_offset_table <8b> |
+-----------------------------------------------------------+
| postings <8b> |
+-----------------------------------------------------------+
| postings_offset_table <8b> |
+-----------------------------------------------------------+
| fingerprint_offsets <8b> |
+-----------------------------------------------------------+
| metadata.from <8b> (oldest chunk min_time) |
+-----------------------------------------------------------+
| metadata.through <8b> (newest chunk max_time) |
+-----------------------------------------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+metadata.from and metadata.through are Loki additions. They expose the time range covered by the index file, so a querier can discard an entire index without reading any of its sections.
Differences from the Prometheus index
- Series entries store an explicit 8-byte fingerprint.
- Chunk metas store
KB,#entries, and the chunk checksum instead of a chunk file reference, because Loki chunks are addressed in object storage by series fingerprint and checksum. - The index adds a fingerprint offsets table and the
from/throughmetadata in the TOC.
