This is documentation for the next version of Grafana Loki documentation. For the latest stable release, go to the latest version.
TSDB index format v3
Version 3 is written for periods with schema version v13. It is identical to
version 2 except for the chunks part of a series entry, which gains page markers so that readers can skip over chunks that fall outside the queried time range. See
TSDB index format for the encoding notation used on this page.
Differences from v2
- The chunks part of a series entry is prefixed with a markers section:
len(markers),#markers, and one marker per page of 16 chunks. - Chunk statistics for a time range can be answered from the markers, without decoding individual chunk metas.
File layout
+-----------------------------------------------------------+
| magic(0xBAAAD700) <4b> | version(3) <1b> |
+-----------------------------------------------------------+
| symbol table (TOC.symbols) |
+-----------------------------------------------------------+
| series (TOC.series) |
+-----------------------------------------------------------+
| label indices (TOC.label_indices) |
+-----------------------------------------------------------+
| postings (TOC.postings) |
+-----------------------------------------------------------+
| label offset table (TOC.label_offset_table) |
+-----------------------------------------------------------+
| postings offset table (TOC.postings_offset_table) |
+-----------------------------------------------------------+
| fingerprint offsets table (TOC.fingerprint_offsets) |
+-----------------------------------------------------------+
| TOC (last 76 bytes of the file) |
+-----------------------------------------------------------+Symbol table
Unchanged from v2: a lexicographically sorted, deduplicated table of all label names and values, referenced by ordinal position.
+-----------------------------------------------------------+
| len <4b> |
+-----------------------------------------------------------+
| #symbols <4b> |
+-----------------------------------------------------------+
| symbol_0 <uvarint_str> |
+-----------------------------------------------------------+
| ... |
+-----------------------------------------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+Series
The series header is unchanged from v2. Entries start on a 16-byte boundary and the series reference is the entry offset divided by 16.
+-----------------------------------------------------------+
| len(entry) <uvarint> |
+-----------------------------------------------------------+
| fingerprint <8b> |
+-----------------------------------------------------------+
| #labels <uvarint> |
+-----------------------------+-----------------------------+
| ref(name_0) <uvarint> | ref(value_0) <uvarint> |
+-----------------------------+-----------------------------+
| ... | ... |
+-----------------------------+-----------------------------+
| chunks (see below, v3 layout) |
+-----------------------------------------------------------+
| CRC32 <4b> |
+-----------------------------------------------------------+Chunks
The chunks part is split into a markers section and the chunk metas. Each marker summarizes a fixed-size page of chunks: its aggregated size and entry count, its time bounds, and the byte offset at which the page starts.
+-----------------------------------------------------------+
| #chunks <uvarint> |
+-----------------------------------------------------------+
| len(markers) <4b> (skip to chunk metas) |
+-----------------------------------------------------------+
| #markers <uvarint> |
+-----------------------------------------------------------+
| marker_0: |
| #chunks in page <uvarint> |
| KB <4b> (sum over page) |
| #entries <4b> (sum over page) |
| offset <uvarint> (page start, relative to |
| the first chunk meta) |
| min_time <varint> (page bounds) |
| max_time - min_time <varint> |
+-----------------------------------------------------------+
| ... |
+-----------------------------------------------------------+
| marker_m-1: (same fields, may cover a partial page) |
+-----------------------------------------------------------+
| chunk_0: |
| min_time delta <varint> (vs. previous max_time) |
| max_time - min_time <uvarint> |
| KB <uvarint> |
| #entries <uvarint> |
| checksum <4b> |
+-----------------------------------------------------------+
| ... |
+-----------------------------------------------------------+
| chunk_n-1: (same fields) |
+-----------------------------------------------------------+The chunk meta encoding itself is byte-for-byte the same as in v2.
How readers use the markers:
- A page holds 16 chunks. The page size is recoverable from each marker’s
#chunks in pagefield, so it can be changed without a new index version. - Series with fewer than 64 chunks skip the markers entirely, using
len(markers), because a linear scan is faster at low chunk counts. - If a page is fully contained in the queried time range, the aggregated
KBand#entriesof its marker are used directly and no chunk meta is decoded. Partially overlapping pages are decoded chunk by chunk, and pages that start after the query end terminate the scan. - The
min_timedelta chain of the chunk metas does not reset at page boundaries. When a reader jumps to a page via itsoffset, it discards the first chunk’s delta and uses the page’smin_timeinstead.
Remaining sections
The label indices, postings, label offset table, postings offset table, fingerprint offsets table, and TOC are identical to v2. See:
