Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Operations

Storage

Open source

Loki Storage

High level storage overview here

Loki needs to store two different types of data: chunks and indexes.

Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are compressed as “chunks” and saved in the chunks store. See chunk format for how chunks are stored internally.

The index stores each stream’s label set and links them to the individual chunks.

Refer to Loki’s configuration for details on how to configure the storage and the index.

For more information:

Supported Stores

The following are supported for the index:

Single Store (boltdb-shipper) - Recommended for 2.0 and newer index store which stores boltdb index files in the object store
Amazon DynamoDB
Google Bigtable
Apache Cassandra
BoltDB (doesn’t work when clustering Loki)

The following are supported for the chunks:

Amazon DynamoDB
Google Bigtable
Apache Cassandra
Amazon S3
Google Cloud Storage
Filesystem (please read more about the filesystem to understand the pros/cons before using with production data)

Cloud Storage Permissions

S3

When using S3 as object storage, the following permissions are needed:

s3:ListObjects
s3:ListBucket
s3:PutObject
s3:GetObject
s3:DeleteObject (if running the Single Store (boltdb-shipper) compactor)

Resources: arn:aws:s3:::<bucket_name>, arn:aws:s3:::<bucket_name>/*

DynamoDB

When using DynamoDB for the index, the following permissions are needed:

dynamodb:BatchGetItem
dynamodb:BatchWriteItem
dynamodb:DeleteItem
dynamodb:DescribeTable
dynamodb:GetItem
dynamodb:ListTagsOfResource
dynamodb:PutItem
dynamodb:Query
dynamodb:TagResource
dynamodb:UntagResource
dynamodb:UpdateItem
dynamodb:UpdateTable
dynamodb:CreateTable
dynamodb:DeleteTable (if table_manager.retention_period is more than 0s)

Resources: arn:aws:dynamodb:<aws_region>:<aws_account_id>:table/<prefix>*

dynamodb:ListTables

Resources: *

AutoScaling

If you enable autoscaling from table manager, the following permissions are needed:

Application Autoscaling

application-autoscaling:DescribeScalableTargets
application-autoscaling:DescribeScalingPolicies
application-autoscaling:RegisterScalableTarget
application-autoscaling:DeregisterScalableTarget
application-autoscaling:PutScalingPolicy
application-autoscaling:DeleteScalingPolicy

Resources: *

IAM

iam:GetRole
iam:PassRole

Resources: arn:aws:iam::<aws_account_id>:role/<role_name>

Chunk Format

  -------------------------------------------------------------------
  |                               |                                 |
  |        MagicNumber(4b)        |           version(1b)           |
  |                               |                                 |
  -------------------------------------------------------------------
  |         block-1 bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |         block-2 bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |         block-n bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |                        #blocks (uvarint)                        |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  |                      checksum(from #blocks)                     |
  -------------------------------------------------------------------
  |           metasOffset - offset to the point with #blocks        |
  -------------------------------------------------------------------