Grafana Loki configuration parameters on Grafana Labs

Configuration best practices

Mon, 29 Apr 2024 15:52:20 +0000

Configuration best practices

Grafana Loki is under active development, and we are constantly working to improve performance. But here are some of the most current best practices for configuration that will give you the best experience with Loki.

Configure caching

Loki can cache data at many levels, which can drastically improve performance. Details of this will be in a future post.

Time ordering of logs

Loki accepts out-of-order writes by default. This section identifies best practices when Loki is not configured to accept out-of-order writes.

One issue many people have with Loki is their client receiving errors for out of order log entries. This happens because of this hard and fast rule within Loki:

For any single log stream, logs must always be sent in increasing time order. If a log is received with a timestamp older than the most recent log received for that stream, that log will be dropped.

There are a few things to dissect from that statement. The first is this restriction is per stream. Let’s look at an example:

{job="syslog"} 00:00:00 i'm a syslog!
{job="syslog"} 00:00:01 i'm a syslog!

If Loki received these two lines which are for the same stream, everything would be fine. But what about this case:

{job="syslog"} 00:00:00 i'm a syslog!
{job="syslog"} 00:00:02 i'm a syslog!
{job="syslog"} 00:00:01 i'm a syslog!  <- Rejected out of order!

What can we do about this? What if this was because the sources of these logs were different systems? We can solve this with an additional label which is unique per system:

{job="syslog", instance="host1"} 00:00:00 i'm a syslog!
{job="syslog", instance="host1"} 00:00:02 i'm a syslog!
{job="syslog", instance="host2"} 00:00:01 i'm a syslog!  <- Accepted, this is a new stream!
{job="syslog", instance="host1"} 00:00:03 i'm a syslog!  <- Accepted, still in order for stream 1
{job="syslog", instance="host2"} 00:00:02 i'm a syslog!  <- Accepted, still in order for stream 2

But what if the application itself generated logs that were out of order? Well, I’m afraid this is a problem. If you are extracting the timestamp from the log line with something like the Promtail pipeline stage, you could instead not do this and let Promtail assign a timestamp to the log lines. Or you can hopefully fix it in the application itself.

It’s also worth noting that the batching nature of the Loki push API can lead to some instances of out of order errors being received which are really false positives. (Perhaps a batch partially succeeded and was present; or anything that previously succeeded would return an out of order entry; or anything new would be accepted.)

Use `chunk_target_size`

Using chunk_target_size instructs Loki to try to fill all chunks to a target compressed size of 1.5MB. These larger chunks are more efficient for Loki to process.

Other configuration variables affect how full a chunk can get. Loki has a default max_chunk_age of 2h and chunk_idle_period of 30m to limit the amount of memory used as well as the exposure of lost logs if the process crashes.

Depending on the compression used (we have been using snappy which has less compressibility but faster performance), you need 5-10x or 7.5-10MB of raw log data to fill a 1.5MB chunk. Remembering that a chunk is per stream, the more streams you break up your log files into, the more chunks that sit in memory, and the higher likelihood they get flushed by hitting one of those timeouts mentioned above before they are filled.

Lots of small, unfilled chunks negatively affect Loki. We are always working to improve this and may consider a compactor to improve this in some situations. But, in general, the guidance should stay about the same: try your best to fill chunks.

If you have an application that can log fast enough to fill these chunks quickly (much less than max_chunk_age), then it becomes more reasonable to use dynamic labels to break that up into separate streams.

Use `-print-config-stderr` or `-log-config-reverse-order`

Loki and Promtail have flags which will dump the entire config object to stderr or the log file when they start.

-print-config-stderr works well when invoking Loki from the command line, as you can get a quick output of the entire Loki configuration.

-log-config-reverse-order is the flag we run Loki with in all our environments. The configuration entries are reversed, so that the order of the configuration reads correctly top to bottom when viewed in Grafana’s Explore.

Query frontend example

Wed, 06 Sep 2023 12:47:18 +0000

Query frontend example

Disclaimer

This aims to be a general purpose example; there are a number of substitutions to make for it to work correctly. These variables take the form of <variable_name>. You should override them with specifics to your environment.

Use case

It’s a common occurrence to start running Grafana Loki as a single binary while trying it out in order to simplify deployments and defer learning the (initially unnecessary) nitty gritty details. As we become more comfortable with its paradigms and begin migrating towards a more production ready deployment there are a number of things to be aware of. A common bottleneck is on the read path: queries that executed effortlessly on small data sets may churn to a halt on larger ones. Sometimes we can solve this with more queriers. However, that doesn’t help when our queries are too large for a single querier to execute. Then we need the query frontend.

Parallelization

One of the most important functions of the query frontend is the ability to split larger queries into smaller ones, execute them in parallel, and stitch the results back together. How often it splits them is determined by the querier.split-queries-by-interval flag or the yaml config query_range.split_queriers_by_interval. With this set to 1h, the frontend will dissect a day long query into 24 one hour queries, distribute them to the queriers, and collect the results. This is immensely helpful in production environments as it not only allows us to perform larger queries via aggregation, but also evens the work distribution across queriers so that one or two are not stuck with impossibly large queries while others are left idle.

Kubernetes Deployment

ConfigMap

Use this ConfigMap to get the benefits of query parallelisation and caching with the query-frontend component.

apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-frontend
  namespace: <namespace>
data:
  config.yaml: |
    # Disable the requirement that every request to Loki has a
    # X-Scope-OrgID header. `fake` will be substituted in instead.
    auth_enabled: false

    # We don't want the usual /api/prom prefix.
    http_prefix:

    server:
      http_listen_port: 3100

    query_range:
      # make queries more cache-able by aligning them with their step intervals
      align_queries_with_step: true
      max_retries: 5
      cache_results: true

      results_cache:
        cache:
          # We're going to use the in-process "FIFO" cache
          enable_fifocache: true
          fifocache:
            size: 1024
            validity: 24h

    limits_config:
      max_cache_freshness_per_query: '10m'
      # parallelize queries in 15min intervals
      split_queries_by_interval: 15m

    frontend:
      log_queries_longer_than: 5s
      downstream_url: http://querier.<namespace>.svc.cluster.local:3100
      compress_responses: true

Frontend Service

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    name: query-frontend
  name: query-frontend
  namespace: <namespace>
spec:
  ports:
  - name: query-frontend-http
    port: 3100
    protocol: TCP
    targetPort: 3100
  selector:
    name: query-frontend
  sessionAffinity: None
  type: ClusterIP
  ClusterIP: None
  publishNotReadyAddresses: true

Frontend Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    name: query-frontend
  name: query-frontend
  namespace: <namespace>
spec:
  minReadySeconds: 10
  replicas: 2
  selector:
    matchLabels:
      name: query-frontend
  template:
    metadata:
      labels:
        name: query-frontend
    spec:
      containers:
      - args:
        - -config.file=/etc/loki/config.yaml
        - -log.level=debug
        - -target=query-frontend
        image: grafana/loki:latest
        imagePullPolicy: Always
        name: query-frontend
        ports:
        - containerPort: 3100
          name: http
          protocol: TCP
        resources:
          limits:
            memory: 1200Mi
          requests:
            cpu: "2"
            memory: 600Mi
        volumeMounts:
        - mountPath: /etc/loki
          name: loki-frontend
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: loki-frontend
        name: loki-frontend

Grafana

Once you’ve deployed these, point your Grafana datasource to the new frontend service. The service is available within the cluster at http://query-frontend.<namespace>.svc.cluster.local:3100.

GRPC Mode (Pull model)

The query frontend operates in one of two ways:

Specify --frontend.downstream-url or its YAML equivalent, frontend.downstream_url. This proxies requests over HTTP to the specified URL.
Without --frontend.downstream-url or its yaml equivalent frontend.downstream_url, the query frontend defaults to a pull service. As a pull service, the frontend instantiates per-tenant queues that downstream queriers pull queries from via GRPC. To act as a pull service, queriers need to specify -querier.frontend-address or its YAML equivalent frontend_worker.frontend_address.

Set ClusterIP=None for the query frontend pull service. This causes DNS resolution of each query frontend pod IP address. It avoids wrongly resolving to the service IP.

Enable publishNotReadyAddresses=true on the query frontend pull service. Doing so eliminates a race condition in which the query frontend address is needed before the query frontend becomes ready when at least one querier connects.

Examples

Mon, 29 Apr 2024 15:52:20 +0000

Examples

alibaba-cloud-storage-config.yaml


# This partial configuration uses Alibaba for chunk storage

schema_config:
  configs:
  - from: 2020-05-15
    object_store: alibabacloud
    schema: v11
    index:
      prefix: loki_index_
      period: 168h

storage_config:
  alibabacloud:
    bucket: <bucket>
    endpoint: <endpoint>
    access_key_id: <access_key_id>
    secret_access_key: <secret_access_key>

1-Local-Configuration-Example.yaml


auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /tmp/loki

schema_config:
  configs:
  - from: 2020-05-15
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h

2-S3-Cluster-Example.yaml


# This is a complete configuration to deploy Loki backed by a s3-Comaptible API
# like MinIO for storage. Loki components will use memberlist ring to shard and
# the index will be shipped to storage via boltdb-shipper.

auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
  replication_factor: 1
  path_prefix: /loki # Update this accordingly, data will be stored here.

memberlist:
  join_members:
  # You can use a headless k8s service for all distributor, ingester and querier components.
  - loki-gossip-ring.loki.svc.cluster.local:7946 # :7946 is the default memberlist port.

schema_config:
  configs:
  - from: 2020-05-15
    store: boltdb-shipper
    object_store: s3
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
 boltdb_shipper:
   active_index_directory: /loki/index
   cache_location: /loki/index_cache
   shared_store: s3
 aws:
   s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name
   s3forcepathstyle: true

compactor:
  working_directory: /loki/compactor
  shared_store: s3
  compaction_interval: 5m

3-S3-Without-Credentials-Snippet.yaml


# If you don't wish to hard-code S3 credentials you can also configure an EC2
# instance role by changing the `storage_config` section

schema_config:
  configs:
  - from: 2020-05-15
    store: aws
    object_store: s3
    schema: v11
    index:
      prefix: loki_
storage_config:
  aws:
    s3: s3://region/bucket_name
    dynamodb:
      dynamodb_url: dynamodb://region

4-BOS-Example.yaml


schema_config:
  configs:
    - from: 2020-05-15
      store: boltdb-shipper
      object_store: bos
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: bos

  bos:
    bucket_name: bucket_name_1
    endpoint: bj.bcebos.com
    access_key_id: access_key_id
    secret_access_key: secret_access_key

compactor:
  working_directory: /tmp/loki/compactor
  shared_store: bos

5-S3-And-DynamoDB-Snippet.yaml


# This partial configuration uses S3 for chunk storage and uses DynamoDB for index storage

schema_config:
  configs:
  - from: 2020-05-15
    store: aws
    object_store: s3
    schema: v11
    index:
      prefix: loki_
storage_config:
  aws:
    s3: s3://access_key:secret_access_key@region/bucket_name
    dynamodb:
      dynamodb_url: dynamodb://access_key:secret_access_key@region

6-Cassandra-Snippet.yaml


# This is a partial config that uses the local filesystem for chunk storage and Cassandra for index storage

schema_config:
  configs:
  - from: 2020-05-15
    store: cassandra
    object_store: filesystem
    schema: v11
    index:
      prefix: cassandra_table
      period: 168h

storage_config:
  cassandra:
    username: cassandra
    password: cassandra
    addresses: 127.0.0.1
    auth: true
    keyspace: lokiindex

  filesystem:
    directory: /tmp/loki/chunks

7-Schema-Migration-Snippet.yaml


schema_config:
  configs:
    # Starting from 2018-04-15 Loki should store indexes on Cassandra
    # using weekly periodic tables and chunks on filesystem.
    # The index tables will be prefixed with "index_".
  - from: "2018-04-15"
    store: cassandra
    object_store: filesystem
    schema: v11
    index:
        period: 168h
        prefix: index_

  # Starting from 2020-6-15 we moved from filesystem to AWS S3 for storing the chunks.
  - from: "2020-06-15"
    store: cassandra
    object_store: s3
    schema: v11
    index:
        period: 168h
        prefix: index_

8-GCS-Snippet.yaml


# This partial configuration uses GCS for chunk storage and uses BigTable for index storage

schema_config:
  configs:
  - from: 2020-05-15
    store: bigtable
    object_store: gcs
    schema: v11
    index:
      prefix: loki_index_
      period: 168h

storage_config:
  bigtable:
    instance: BIGTABLE_INSTANCE
    project: BIGTABLE_PROJECT
  gcs:
    bucket_name: GCS_BUCKET_NAME

9-Expanded-S3-Snippet.yaml


# S3 configuration supports an expanded configuration. 
# Either an `s3` endpoint URL can be used, or an expanded configuration can be used.

schema_config:
  configs:
  - from: 2020-05-15
    store: aws
    object_store: s3
    schema: v11
    index:
      prefix: loki_
storage_config:
  aws:
    bucketnames: bucket_name1, bucket_name2
    endpoint: s3.endpoint.com
    region: s3_region
    access_key_id: s3_access_key_id
    secret_access_key: s3_secret_access_key
    insecure: false
    sse_encryption: false
    http_config:
      idle_conn_timeout: 90s
      response_header_timeout: 0s
      insecure_skip_verify: false
    s3forcepathstyle: true

10-S3-And-DynamoDB-With-KMS-Snippet.yaml


# This partial configuration uses S3 for chunk storage and uses DynamoDB for index storage and a KMS CMK for encryption

schema_config:
  configs:
  - from: 2020-05-15
    store: aws
    object_store: s3
    schema: v11
    index:
      prefix: loki_
storage_config:
  aws:
    s3: s3://access_key:secret_access_key@region/bucket_name
    sse_encryption: true
    sse:
      type: SSE-KMS
      kms_key_id: 1234abcd-12ab-34cd-56ef-1234567890ab
    dynamodb:
      dynamodb_url: dynamodb://access_key:secret_access_key@region
      kms_key_id: 0987dcba-09fe-87dc-65ba-ab0987654321

11-COS-HMAC-Example.yaml


# This partial configuration uses IBM Cloud Object Storage (COS) for chunk storage. HMAC will be used for authenticating with COS.

schema_config:
  configs:
    - from: 2020-10-01
      index:
        period: 24h
        prefix: loki_index_
      object_store: "cos"
      schema: v11
      store: "boltdb-shipper"

storage_config:
  cos:
    bucketnames: <bucket1, bucket2>
    endpoint: <endpoint>
    region: <region>
    access_key_id: <access_key_id>
    secret_access_key: <secret_access_key>

12-COS-APIKey-Example.yaml


# This partial configuration uses IBM Cloud Object Storage (COS) for chunk storage. APIKey will be used for authenticating with COS.

schema_config:
  configs:
    - from: 2020-10-01
      index:
        period: 24h
        prefix: loki_index_
      object_store: "cos"
      schema: v11
      store: "boltdb-shipper"

storage_config:
  cos:
    bucketnames: <bucket1, bucket2>
    endpoint: <endpoint>
    region: <region>
    api_key: <api_key_to_authenticate_with_cos>
    service_instance_id: <cos_service_instance_id>
    auth_endpoint: <iam_endpoint_for_authentication>

13-COS-Trusted-Profile-Example.yaml


# This partial configuration uses IBM Cloud Object Storage (COS) for chunk storage. 
# A trusted profile will be used for authenticating with COS. We can either pass
# the trusted profile name or trusted profile ID along with the compute resource token file.
# If we pass both trusted profile name and trusted profile ID it should be of 
# the same trusted profile.
# In order to use trusted profile authentication we need to follow an additional step to create a trusted profile.
# For more details about creating a trusted profile, see https://cloud.ibm.com/docs/account?topic=account-create-trusted-profile&interface=ui.

schema_config:
  configs:
    - from: 2020-10-01
      index:
        period: 24h
        prefix: loki_index_
      object_store: "cos"
      schema: v11
      store: "boltdb-shipper"

storage_config:
  cos:
    bucketnames: <bucket1, bucket2>
    endpoint: <endpoint>
    region: <region>
    auth_endpoint: <iam_endpoint_for_authentication>
    cr_token_file_path: <path_to_compute_resource_token>
    trusted_profile_name: <name_of_the_trusted_profile> # You can also use trusted_profile_id instead of trusted_profile_name

15-Memberlist-Ring-Snippet.yaml


# This partial configuration uses memberlist for the ring.

common:
  ring:
    kvstore:
      store: memberlist
  replication_factor: 1
  path_prefix: /loki

memberlist:
  join_members:
    # You can use a headless k8s service for all distributor, ingester and querier components.
    - loki-gossip-ring.loki.svc.cluster.local:7946 # :7946 is the default memberlist port.

16-Azure-Account-Name-Example.yaml


# This partial configuration uses Azure for chunk storage
storage_config:
  azure:
    # For the account-key, see docs: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal
    account_key: <azure_blob_access_key>
    account_name: <azure_account_name>
    container_name: <azure_storage_bucket_name>
    use_managed_identity: false
    # Providing a user assigned ID will override use_managed_identity
    #user_assigned_id: <user-assigned-identity-id>
    request_timeout: 0
    # Configure `endpoint_suffix` if you are using private azure cloud like azure stack hub and will use this endpoint suffix to compose container and blob storage URL. Ex: https://account_name.endpoint_suffix/container_name/blob_name
    #endpoint_suffix: <endpoint-suffix>
    # If `connection_string` is set, the `account_name` and `endpoint_suffix` values will not be used. Use this method over `account_key` if you need to authenticate via an SAS token. Or if you use the Azurite emulator.
    #connection_string: <connection-string>

17-Azure-Service-Principal-Example.yaml


# This partial configuration uses Azure for chunk storage and a service principal for authentication
storage_config:
  azure:
    use_service_principal: true
    # Azure tenant ID used to authenticate through Azure OAuth
    tenant_id: <tenant-id>
    # Azure Service Principal ID
    client_id: <client-id>
    # Azure Service Principal secret key
    client_secret: <client-secret>
    # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers
    container_name: <azure_storage_bucket_name>
    request_timeout: 0

Grafana Loki configuration parameters on Grafana Labs

Configuration best practices

Configuration best practices

Configure caching

Time ordering of logs

Use chunk_target_size

Use -print-config-stderr or -log-config-reverse-order

Query frontend example

Query frontend example

Disclaimer

Use case

Parallelization

Kubernetes Deployment

ConfigMap

Frontend Service

Frontend Deployment

Grafana

GRPC Mode (Pull model)

Examples

Examples

alibaba-cloud-storage-config.yaml

1-Local-Configuration-Example.yaml

2-S3-Cluster-Example.yaml

3-S3-Without-Credentials-Snippet.yaml

4-BOS-Example.yaml

5-S3-And-DynamoDB-Snippet.yaml

6-Cassandra-Snippet.yaml

7-Schema-Migration-Snippet.yaml

8-GCS-Snippet.yaml

9-Expanded-S3-Snippet.yaml

10-S3-And-DynamoDB-With-KMS-Snippet.yaml

11-COS-HMAC-Example.yaml

12-COS-APIKey-Example.yaml

13-COS-Trusted-Profile-Example.yaml

15-Memberlist-Ring-Snippet.yaml

16-Azure-Account-Name-Example.yaml

17-Azure-Service-Principal-Example.yaml

Use `chunk_target_size`

Use `-print-config-stderr` or `-log-config-reverse-order`