This is documentation for the next version of Grafana Loki documentation. For the latest stable release, go to the latest version.

Reference

TSDB index format

TSDB v4

Open source

TSDB index format v4

Version 4 is written for periods with schema version v14. This version is experimental. It keeps the paging layout of version 3 and appends an ingestion timestamp to every chunk meta, which allows retention to be measured from the time a chunk was ingested rather than from the timestamps of the log lines it contains. See TSDB index format for the encoding notation used on this page.

Differences from v3

Each chunk meta ends with an ingested_at delta field, encoded at day precision relative to max_time.
The field is 0 for all chunks that were not backfilled.

File layout

+-----------------------------------------------------------+
| magic(0xBAAAD700) <4b> | version(4) <1b>                  |
+-----------------------------------------------------------+
| symbol table              (TOC.symbols)                   |
+-----------------------------------------------------------+
| series                    (TOC.series)                    |
+-----------------------------------------------------------+
| label indices             (TOC.label_indices)             |
+-----------------------------------------------------------+
| postings                  (TOC.postings)                  |
+-----------------------------------------------------------+
| label offset table        (TOC.label_offset_table)        |
+-----------------------------------------------------------+
| postings offset table     (TOC.postings_offset_table)     |
+-----------------------------------------------------------+
| fingerprint offsets table (TOC.fingerprint_offsets)       |
+-----------------------------------------------------------+
| TOC                       (last 76 bytes of the file)     |
+-----------------------------------------------------------+

Symbol table

Unchanged from v2 and v3: a lexicographically sorted, deduplicated table of all label names and values, referenced by ordinal position.

+-----------------------------------------------------------+
| len <4b>                                                  |
+-----------------------------------------------------------+
| #symbols <4b>                                             |
+-----------------------------------------------------------+
| symbol_0 <uvarint_str>                                    |
+-----------------------------------------------------------+
| ...                                                       |
+-----------------------------------------------------------+
| CRC32 <4b>                                                |
+-----------------------------------------------------------+

Series

The series header is unchanged. Entries start on a 16-byte boundary and the series reference is the entry offset divided by 16.

+-----------------------------------------------------------+
| len(entry) <uvarint>                                      |
+-----------------------------------------------------------+
| fingerprint <8b>                                          |
+-----------------------------------------------------------+
| #labels <uvarint>                                         |
+-----------------------------+-----------------------------+
| ref(name_0) <uvarint>       | ref(value_0) <uvarint>      |
+-----------------------------+-----------------------------+
| ...                         | ...                         |
+-----------------------------+-----------------------------+
| chunks                          (see below, v4 layout)    |
+-----------------------------------------------------------+
| CRC32 <4b>                                                |
+-----------------------------------------------------------+

Chunks

The markers section is identical to v3. Each chunk meta carries one additional trailing field, ingested_at delta.

+-----------------------------------------------------------+
| #chunks <uvarint>                                         |
+-----------------------------------------------------------+
| len(markers) <4b>               (skip to chunk metas)     |
+-----------------------------------------------------------+
| #markers <uvarint>                                        |
+-----------------------------------------------------------+
| marker_0:                                                 |
|   #chunks in page <uvarint>                               |
|   KB <4b>                       (sum over page)           |
|   #entries <4b>                 (sum over page)           |
|   offset <uvarint>              (page start, relative to  |
|                                  the first chunk meta)    |
|   min_time <varint>             (page bounds)             |
|   max_time - min_time <varint>                            |
+-----------------------------------------------------------+
| ...                                                       |
+-----------------------------------------------------------+
| marker_m-1: (same fields, may cover a partial page)       |
+-----------------------------------------------------------+
| chunk_0:                                                  |
|   min_time delta <varint>       (vs. previous max_time)   |
|   max_time - min_time <uvarint>                           |
|   KB <uvarint>                                            |
|   #entries <uvarint>                                      |
|   checksum <4b>                                           |
|   ingested_at delta <uvarint>   (v4 only, see below)      |
+-----------------------------------------------------------+
| ...                                                       |
+-----------------------------------------------------------+
| chunk_n-1: (same fields)                                  |
+-----------------------------------------------------------+

Because the field is appended to the chunk meta, a v4 chunk meta cannot be read by a v3 reader. The index version in the header selects the decoder, so a cluster only needs to know that indices written under schema v14 cannot be read by a version of Loki that predates v4 support.

Ingestion timestamp encoding

ingested_at records when a chunk was ingested, which for backfilled data differs from the timestamps of its log lines. It is the highest timestamp of the Kafka records that fed the chunk. It is only recorded for backfilled chunks ingested from Kafka, that is, chunks of a stream carrying the __backfill__="true" label, at day precision relative to the chunk’s max_time:

day               = 86400000 ms
delta_days        = ceil_day(ingested_at) - floor_day(max_time)
ingested_at delta = zigzag(delta_days) + 1

The value 0 is a sentinel meaning “no ingestion timestamp”. Every non-backfilled chunk stores 0, so the field costs a single byte for the common case.
ingested_at is rounded up to the next UTC day boundary. Rounding must never move the timestamp earlier, otherwise retention measured from ingestion could expire a chunk before its window elapsed. The cost is retaining a chunk for up to one day longer.
The zigzag mapping keeps small negative deltas cheap: 0 days encodes as 0, -1 day as 1, +1 day as 2. Adding 1 keeps the encoded 0 free for the sentinel.
Rounding is stable. Re-encoding an already rounded ingested_at yields the same delta, so compaction does not drift the value.

For example, if max_time is 2026-01-10 14:30 UTC and the chunk was ingested at 2026-01-12 08:00 UTC, ingestion rounds up to 2026-01-13 00:00 UTC, the day of max_time is 2026-01-10, and the stored delta is +3 days.