Migrate on Grafana Labs

Migrate to TSDB

Thu, 10 Apr 2025 12:15:54 +0000

Migrate to TSDB

TSDB is the recommended index type for Loki and is where the current development lies. If you are running Loki with boltb-shipper or any of the legacy index types that have been deprecated, we strongly recommend migrating to TSDB.

Configure TSDB index for an upcoming period

To begin the migration, add a new period_config entry in your schema_config. You can read more about schema config here.

Note
You must roll out the new period_config change to all Loki components in order for it to take effect.

This example adds a new period_config which configures Loki to start using the TSDB index for the data ingested starting from 2023-10-20.

schema_config:
  configs:
    - from: 2023-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: 2023-10-20 ①
      store: tsdb ②
      object_store: filesystem ③
      schema: v13 ④
      index:
        prefix: index_
        period: 24h

You must set the new period from to a date in the future.
Update the new period to use TSDB as the index type by setting store: tsdb.
This sample configuration uses filesystem as the storage in both the periods. If you want to use a different storage for the TSDB index and chunks, you can specify a different object_store in the new period.
Update the schema to v13 which is the recommended version at the time of writing. Please refer to the configure page for the current recommend version.

Configure TSDB shipper

It’s also important that you configure the tsdb_shipper block in storage_config. Specifically the following options:

active_index_directory: directory where ingesters would write index files which will then be uploaded by shipper to configured storage.
cache_location: cache location for downloading index files from the storage for use in query path.

storage_config:
  tsdb_shipper:
    active_index_directory: /data/tsdb-index
    cache_location: /data/tsdb-cache

Run compactor

We strongly recommended running the compactor when using TSDB index. It is responsible for running compaction and retention on TSDB index. Not running index compaction will result in sub-optimal query performance.

Please refer to the compactor section for more information and configuration examples.

Migrate from `loki-distributed` Helm chart

Wed, 06 Sep 2023 12:47:18 +0000

Migrate from `loki-distributed` Helm chart

This guide will walk you through migrating to the loki Helm Chart, v3.0 or higher, from the loki-distributed Helm Chart (v0.63.2 at time of writing). The process consists of deploying the new loki Helm Chart alongside the existing loki-distributed installation. By joining the new cluster to the exsiting cluster’s ring, you will create one large cluster. This will allow you to manually bring down the loki-distributed components in a safe way to avoid any data loss.

Before you begin:

We recommend having a Grafana instance available to monitor both the existing and new clusters, to make sure there is no data loss during the migration process. The loki chart ships with self-monitoring features, including dashboards. These are useful for monitoring the health of the new cluster as it spins up.

Start by updating your existing Grafana Agent or Promtail config (whatever is scraping logs from your environment) to exclude the new deployment. The new loki chart ships with its own self-monitoring mechanisms, and we want to make sure it’s not scraped twice, which would produce duplicate logs. The best way to do this is via a relabel config that will drop logs from the new deployment, for example something like:

- source_labels:
    - "__meta_kubernetes_pod_label_app_kubernetes_io_component"
  regex: "(canary|read|write)"
  action: "drop"

This leverages the fact that the new deployment adds a app.kubernetes.io/component label of either read for the Read pods, write for the Write pods, and canary for the Loki Canary pods. These annotations are not present in the loki-distributed deployment, so this should only match logs from the new deployment.

To Migrate from loki-distributed to loki

Deploy new Loki Cluster

Next you will deploy the new loki chart in the same namespace as your existing loki-distributed installation. Make sure to use the same buckets and storage configuration as your existing installation. You will need to set some special migrate values as well:
YAML
```
migrate:
  fromDistributed:
    enabled: true
    memberlistService: loki-loki-distributed-memberlist
```
The value of memberlistService must be the kubernetes service created for the purpose of Memberlist DNS in the loki-distributed deployment. It should match the value of memberlist.join_members in the config of the loki-distributed deployment. This is what will make the new cluster join the existing clusters ring. It is important to join all existing rings, if you are using different memberlist DNS for different rings, you must manually set the value of each applicable join_members configuration for each ring. If using the same memberlist DNS for all rings, as is the default in the loki-distributed chart, setting migrate.memberlistService should be enough.

Once the new cluster is up, add the appropriate data source in Grafana for the new cluster. Check that the following queries return results:
- Confirm new and old logs are in the new deployment. Using the new deployment’s Loki data source in Grafana, look for:
  - Logs with a job that is unqiue to your existing Promtail or Grafana Agent, the one we adjusted above to exclude logs from the new deployment which is not yet pushing logs to the new deployment. If you can query those via the new deployment in shows we have not lost historical logs.
  - Logs with the label job="loki/loki-read". The read component does not exist in loki-distributed, so this show the new Loki cluster’s self monitoring is working correctly.
- Confirm new logs are in the old deployment. Using the old deployment’s Loki data source in Grafana, look for:
  - Logs with the label job="loki/loki-read". Since you have excluded logs from the new deployment from going to the loki-distributed deployment, if you can query them through the loki-distributed Loki data source that show the ingesters have joined the same ring, and are queryable from the loki-distributed queriers.
Convert all Clients to Push Logs to New loki Deployment

Assuming everything is working as expected, you can now modify the clients section of your Grafana Agent or Promtail configuration to push logs to the new deployment. After this change is made, the loki-distributed cluster will no longer recieve new logs and can be carefully scaled down.

Once this has deployed, query the new loki cluster’s Loki data source for new logs to make sure they’re still being ingested.
Tear Down the Old Loki Canary

If you had previously deployed the canary via the loki-canary Helm Chart, you can now tear it down. The new chart ships the canary by default and is automatically configured to scrape it.
Update Flush Config On loki-distributed Deployment

You are almost ready to start scaling down the old loki-distributed cluster. Before you start, however, it is important to make sure the loki-distributed ingesters are configured to flush on shutdown. Since these ingesters will not be coming back, there will be no opportunity for them to replay their WALs, so they need to flush all in-memory logs before shutting down to prevent data loss.

The easiest way to do this is via the extraArgs argument in the ingester section of the loki-distributed chart. You may also want to set the ingester’s log level to debug to see messages about the flushing process.
YAML
```
ingester:
  replicas: 3
  extraArgs:
    - '-ingester.flush-on-shutdown=true'
    - '-log.level=debug'
    ```

Deploy this change, and make sure all ingesters restart and are running the latest configuration.
```
Scale Down Ingesters One at a Time

It is now time to start scaling down loki-distributed. Scale down the ingester StatefulSet or Deployment (depending on how your loki-distributed chart is deployed) 1 replica at a time. If debug logs were enabled, you can monitor the logs of each ingester as it’s terminating to make sure the flushing process was successful. Once the ingester pod is fully terminated, continue decrementing by another 1 replica. Continue until there are 0 instances of the ingester running.

You can use the following command to edit the ingester StatefulSet in order to change the number of replicas (making sure to replace <NAMESPACE> with the correct namespace):
Bash
```
kubectl -n <NAMESPACE> edit statefulsets.apps loki-distributed-ingester
```
Remove loki-distributed cluster

Double check that logs are still coming in to the new cluster. If something is wrong, it will be much easier to quickly scale back up loki-distributed ingesters before tearing down the whole cluster so you can investigate. If everything looks good, you can tear down loki-distributed using helm uninstall. For example:
Bash
```
helm uninstall -n loki loki-distributed
```
Now edit the new loki cluster to remove the migrate options you added when first installing. Remove all of the following from your values.yaml:
YAML
```
migrate:
  fromDistributed:
    enabled: true
    memberlistService: loki-loki-distributed-memberlist
```
To apply the new configuration (assuming you installed the new chart as loki in the loki namespace):
Bash
```
helm upgrade -n loki loki grafana/loki --values values.yaml
```
The migrate.fromDistributed.memberlistService was added as an additional memberlist join member, so once this new config is pushed, the components should roll without interruption.
Check the Dashboards

Now that the migration is finished, you can check the dashboards included with the new loki chart to make sure everything is working as expected. You can also check the loki canary metrics to make sure nothing was lost during the migration. Assuming everything was in the loki namespace, the following query, if run over a time period that starts before the migration, and ends after it was complete, should clearly illustrate when metrics started coming from the new canary, and if and when there was data detected by either during the process:
logql
```
(
  sum(increase(loki_canary_missing_entries_total{namespace="loki"}[$__range])) by (job)
  /
  sum(increase(loki_canary_entries_total{namespace="loki"}[$__range])) by (job)
)*100
```

Migrate to three scalable targets

Wed, 06 Sep 2023 12:47:18 +0000

Migrate to three scalable targets

This guide will walk you through migrating from the old, two target, scalable configuration to the new, three target, scalable configuration. This new configuration introduces a backend component, and reduces the read component to running just a Querier and QueryFrontend, allowing it to be run as a kubernetes Deployment rather than a StatefulSet.

Before you begin:

To Migrate from a “read & write” to a “backend, read & write” deployment

Make sure your deployment is using a new enough version of loki

This feature landed as an option in the helm chart while still in the main branch of Loki. As a result, depending on when you run this migration, you may neeed to manually override the Loki or GEL image being used to one that has the third, backend target available. For loki, add the following to your values.yaml.

loki:
  image:
    repository: "grafana/loki"
    tag: "main-f5fbfab-amd64"

For GEL, you’ll need to add:

enterprise:
  image:
    repository: "grafana/enterprise-logs"
    tag: "main-96f32b9f"

Set the legacyReadTarget flag to false

Set the value read.legacyReadTarget to false. In your values.yaml, add:

read:
  legacyReadTarget: false

Upgrade the helm installation

Run helm upgrade on your installation with your updated values.yaml file.

Migrate on Grafana Labs

Migrate to TSDB

Migrate to TSDB

Configure TSDB index for an upcoming period

Configure TSDB shipper

Run compactor

Migrate from `loki-distributed` Helm chart

Migrate from loki-distributed Helm chart

Migrate to three scalable targets

Migrate to three scalable targets

Migrate from `loki-distributed` Helm chart