This is documentation for the next version of Loki. For the latest stable release, go to the latest version.
Manage storage
You can read a high level overview of Loki storage here
Grafana Loki needs to store two different types of data: chunks and indexes.
Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are compressed as “chunks” and saved in the chunks store. See chunk format for how chunks are stored internally.
The index stores each stream’s label set and links them to the individual chunks.
Refer to Loki’s configuration for details on how to configure the storage and the index.
For more information:
Supported Stores
The following are supported for the index:
- TSDB index store which stores TSDB index files in the object store. This is the recommended index store for Loki 2.8 and newer.
- Single Store (boltdb-shipper) index store which stores boltdb index files in the object store.
- Amazon DynamoDB
- Google Bigtable
- Apache Cassandra
- BoltDB (doesn’t work when clustering Loki)
The following are deprecated for the index and will be removed in a future release:
- Amazon DynamoDB. Support for this is deprecated and will be removed in a future release.
- Google Bigtable. Support for this is deprecated and will be removed in a future release.
- Apache Cassandra. Support for this is deprecated and will be removed in a future release.
The following are supported and recommended for the chunks:
- Amazon S3
- Google Cloud Storage
- Filesystem (please read more about the filesystem to understand the pros/cons before using with production data)
- Baidu Object Storage
- IBM Cloud Object Storage
The following are supported for the chunks, but not typically recommended for production use:
- Filesystem (please read more about the filesystem to understand the pros/cons before using with production data)
The following are deprecated for the chunks and will be removed in a future release:
- Amazon DynamoDB. Support for this is deprecated and will be removed in a future release.
- Google Bigtable. Support for this is deprecated and will be removed in a future release.
- Apache Cassandra. Support for this is deprecated and will be removed in a future release.
Cloud Storage Permissions
S3
When using S3 as object storage, the following permissions are needed:
s3:ListBucket
s3:PutObject
s3:GetObject
s3:DeleteObject
(if running the Single Store (boltdb-shipper) compactor)
Resources: arn:aws:s3:::<bucket_name>
, arn:aws:s3:::<bucket_name>/*
See the AWS deployment section on the storage page for a detailed setup guide.
DynamoDB
Note: DynamoDB support is deprecated and will be removed in a future release.
When using DynamoDB for the index, the following permissions are needed:
dynamodb:BatchGetItem
dynamodb:BatchWriteItem
dynamodb:DeleteItem
dynamodb:DescribeTable
dynamodb:GetItem
dynamodb:ListTagsOfResource
dynamodb:PutItem
dynamodb:Query
dynamodb:TagResource
dynamodb:UntagResource
dynamodb:UpdateItem
dynamodb:UpdateTable
dynamodb:CreateTable
dynamodb:DeleteTable
(iftable_manager.retention_period
is more than 0s)
Resources: arn:aws:dynamodb:<aws_region>:<aws_account_id>:table/<prefix>*
dynamodb:ListTables
Resources: *
AutoScaling
If you enable autoscaling from table manager, the following permissions are needed:
Application Autoscaling
application-autoscaling:DescribeScalableTargets
application-autoscaling:DescribeScalingPolicies
application-autoscaling:RegisterScalableTarget
application-autoscaling:DeregisterScalableTarget
application-autoscaling:PutScalingPolicy
application-autoscaling:DeleteScalingPolicy
Resources: *
IAM
iam:GetRole
iam:PassRole
Resources: arn:aws:iam::<aws_account_id>:role/<role_name>
IBM Cloud Object Storage
When using IBM Cloud Object Storage (COS) as object storage, IAM Writer
role is needed.
See the IBM Cloud Object Storage section on the storage page for a detailed setup guide.
Chunk Format
-------------------------------------------------------------------
| | |
| MagicNumber(4b) | version(1b) |
| | |
-------------------------------------------------------------------
| block-1 bytes | checksum (4b) |
-------------------------------------------------------------------
| block-2 bytes | checksum (4b) |
-------------------------------------------------------------------
| block-n bytes | checksum (4b) |
-------------------------------------------------------------------
| #blocks (uvarint) |
-------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
-------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
-------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
-------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
-------------------------------------------------------------------
| checksum(from #blocks) |
-------------------------------------------------------------------
| metasOffset - offset to the point with #blocks |
-------------------------------------------------------------------
Was this page helpful?
Related resources from Grafana Labs


