This is documentation for the next version of Grafana Loki documentation. For the latest stable release, go to the latest version.

Documentation Grafana Loki Set up Migrate Migrate from SSD to HA monolithic

Migrate from SSD to highly available monolithic deployment

This guide provides instructions for migrating from a simple scalable deployment (SSD) to a highly available (HA) monolithic deployment of Loki. Before starting the migration, make sure you have read the planning section.

Planning your migration

Migrating from a simple scalable deployment to a HA monolithic deployment without sustained downtime is possible but requires careful planning. The following considerations should be taken into account:

1. Data: No changes are required to your underlying data storage. Both SSD and HA monolithic require object storage for chunks and indexes (TSDB Shipper). Cloud Service Providers usually have guarantees around data availability. Although data loss or corruption is unlikely, it is always recommended to back up your data before starting the migration process. If you are using a cloud provider you can take a snapshot/backup.
2. Compute resources: While SSD has 3 different components and each has their own shape in term of CPU and memory consumption, HA monolithic only has a single component that processes both writes, reads, and auxiliary operations. Therefore the shape of compute nodes running Loki instances differ from SSD. Make sure you have sufficient headroom to scale both horizontally and vertically if needed. Each instance also needs its own persistent volume for the write-ahead log (WAL). Instances must not share a volume.
3. Configuration: This guide only covers the parameters that need to be changed. Other parameters can remain the same. Loki 4.0 removes or renames several settings that SSD configurations commonly contain. Stage 2 lists them, and the upgrade guide describes all breaking changes. The deprecated-config-checker tool described there finds removed settings in your configuration file.
4. Failed requests: A restart of an instance can fail individual requests that were in flight. Make sure that your clients retry failed requests. The procedures in this guide avoid sustained failures but not individual failed requests.
5. Recent logs after the traffic switch: Between routing traffic to the new instances and stopping the old write instances, queries don’t return the most recent logs. Stage 3 describes how to keep this period short.

Does HA monolithic deployment fit my purpose?

Stage 1: Preparing existing deployment

Before beginning the actual migration you want to check your existing deployment. All changes in this stage are made to the configuration of the existing SSD instances and take effect after a restart.

1. Set ingester.wal.flush_on_shutdown: true to make sure that chunks are written and uploaded to object storage when ingesters shut down. In Stage 3, this setting makes the logs that are still in memory of the old ingesters available to the new instances.

2. Make sure that ingester.lifecycler.unregister_on_shutdown isn’t set to false, so that after final shutdown ingesters leave the ring immediately. The default is true.

3. Set a memberlist cluster label, as described in the next section. If your existing deployment already has the cluster label, you can skip this step.

Set a memberlist cluster label

The new instances run in a separate memberlist cluster with their own rings. A cluster label makes sure that the old and the new instances don’t join the same cluster if they can reach each other, for example because they use the same network or the same DNS names to find peers.

Instances only talk to instances with the same label when verification is enabled. An instance with a label discards messages from instances without a label and vice versa.

To avoid longer downtime while restarting the instances with a new cluster label, roll out the label in three steps. Restart the instances one at a time and wait until each instance is ready before you restart the next one. Include the ingester settings from the previous section in the first step.

1. Disable verification of the label, without setting a label:

   YAML Copy

   memberlist:
     cluster_label_verification_disabled: true

   With verification disabled, an instance accepts messages with and without a label. After this step every instance can receive messages with a label.

2. Set the label. Keep verification disabled:

   YAML Copy

   memberlist:
     cluster_label: <your-unique-cluster-name>
     cluster_label_verification_disabled: true

3. Enable verification:

   YAML Copy

   memberlist:
     cluster_label: <your-unique-cluster-name>
     cluster_label_verification_disabled: false

If you accept a short downtime, you can set the label and restart all instances at the same time.

Stage 2: Deploying the monolithic component

In this stage, you deploy the new instances alongside the existing SSD instances. The new instances use the same object storage as the SSD instances. They don’t receive any traffic yet.

Since there is only a single type of Loki instance in the HA monolithic deployment, all instances are configured equally, except for the compactor mode, and started with the -target=all target.

To achieve high availability you need at least three Loki instances and a replication factor (common.replication_factor) of 3. This ensures that you still have high availability - a quorum of two - during restarts where one instance is not available.

The following example shows the settings that differ from an SSD configuration. The sections after the example explain them.

YAML Copy

common:
  replication_factor: 3
  compactor_grpc_address: <main-instance-address>:9095
  ring:
    kvstore:
      store: memberlist

storage_config:
  use_thanos_objstore: true
  object_store:
    s3:
      # Same bucket and credentials as the existing deployment.
      endpoint: <endpoint>
      bucket_name: <bucket-name>

memberlist:
  join_members:
    - <address-of-instance-1>
    - <address-of-instance-2>
    - <address-of-instance-3>
  cluster_label: <your-new-unique-cluster-name>
  cluster_label_verification_disabled: false

ingester:
  lifecycler:
    unregister_on_shutdown: true
  wal:
    enabled: true
    dir: <wal-directory-on-the-persistent-volume>
    flush_on_shutdown: true

compactor:
  working_directory: <directory-on-the-persistent-volume>
  horizontal_scaling_mode: worker
  retention_enabled: true
  delete_request_store: s3

# Only if you use the ruler.
ruler_storage:
  backend: <backend>

Configure the compactor

A HA monolithic deployment runs one main compactor and one or more workers. The main compactor must run on exactly one instance at any time.

1. Set compactor.horizontal_scaling_mode to worker in the configuration that all instances share. The default is disabled.
2. Override the mode on exactly one instance with -compactor.horizontal-scaling-mode=main. You can also use an environment variable, if you reference it in the configuration file and start Loki with -config.expand-env=true. Your deployment tool must be able to configure one instance differently from the others.
3. Set common.compactor_grpc_address to the gRPC address of the main instance. Worker instances use it to connect to the main compactor. The address must stay valid when the main instance restarts, for example a stable DNS name that resolves only to that instance.
4. Set compactor.retention_enabled: true and compactor.delete_request_store to the name of your object store, for example s3.

Worker instances only start when retention is enabled, because deletion is the only job that workers process. Without it, the instance exits at startup with error="starting module compactor: ... no job runners registered". Enabling retention doesn’t delete data on its own. limits_config.retention_period defaults to 0s, which disables retention.

The main compactor waits for one compactor.compaction_interval (default 10m) after it starts before it compacts. Refer to Horizontal scaling of Compactor for details.

Configure storage

To achieve a transition without downtime, configure the storage of the HA monolithic instances the same as the existing deployment: same bucket, same prefix, and same schema_config.

Use the Thanos object client (storage_config.use_thanos_objstore: true and the storage_config.object_store configuration block), because the legacy object store clients are deprecated. Loki 4.0 uses the Thanos clients by default. If your configuration still contains a legacy storage block, such as storage_config.aws, and you don’t set use_thanos_objstore: false, Loki fails to start with when use_thanos_objstore is true, config must be specified in the object_store section. Refer to Migrate to Thanos storage clients to convert your configuration. The Thanos clients read the data that the legacy clients wrote.

Update removed and renamed settings

The following settings of a typical SSD configuration need to change.

Configure the ruler

If you use the ruler, make the rules available to all HA monolithic instances, either through object storage in ruler_storage or through the same rule files on every instance. Set ruler.evaluation.query_frontend.address to an address that resolves to all HA monolithic instances.

While both deployments run, both rulers evaluate your rules. Each ruler evaluates them against the data that its own deployment can query.

Configure memberlist

Set memberlist.join_members to the addresses of the HA monolithic instances only. Use a cluster label that is different from the label of the existing deployment. This keeps the rings of both deployments separate.

Verify the deployment

1. When all instances are started and running, open the ring page (/ring) of one instance. Verify that all instances are registered and in an ACTIVE state. The ring page of the existing SSD deployment still shows only the SSD instances.
2. Push a log line to one instance and query it from another instance. If multi-tenancy is enabled, use a test tenant.
3. Check the logs of all instances for errors.

Stage 3: Transitioning to monolithic components

The final stage of the migration is routing the traffic to the new downstream instances in the reverse proxy or load balancer and shutting down the old SSD components.

Route the traffic to the new instances

The reverse proxy or load balancer must do the following:

• Send all requests to all HA monolithic instances. Any instance can handle any request.
• Send requests to /loki/api/v1/delete only to the main compactor instance. This is only needed when deletes are enabled and you want to be able to access the API from outside. Other instances return 404 for this path.
• Forward the WebSocket upgrade for /loki/api/v1/tail. Without it, tailing logs fails with 400 Bad Request.
• Track address changes of instances. An instance that restarts can get a new IP address. A proxy that resolves names only when it starts or reloads keeps sending requests to the old addresses, and all requests fail when every instance has restarted.

When you run Loki on Kubernetes, send the traffic to a Service with a cluster IP that selects all HA monolithic Pods, and to a second Service with a cluster IP that selects only the main instance. The addresses of these Services don’t change when Pods restart.

The following NGINX configuration re-resolves the names every 5 seconds. The name loki resolves to all HA monolithic instances, and loki-main resolves to the main instance.

nginx Copy

http {
    # DNS server of your platform, for example 127.0.0.11 in Docker.
    resolver 127.0.0.11 valid=5s;

    server {
        listen 3100;

        proxy_connect_timeout 2s;

        location / {
            set $loki loki;
            proxy_pass http://$loki:3100;
        }

        location = /loki/api/v1/tail {
            set $loki loki;
            proxy_pass http://$loki:3100;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }

        location ~ ^/loki/api/v1/delete {
            set $compactor loki-main;
            proxy_pass http://$compactor:3100;
        }
    }
}

The resolver directive of NGINX ignores the search domains of the host. Use fully qualified names, for example loki.<namespace>.svc.cluster.local on Kubernetes.

Reload the configuration without restarting the proxy, for example with nginx -s reload. After the reload, the new instances receive the traffic for both writes (push) and reads (queries).

Stop the old write instances

Immediately after the reload, shut down the write instances of the old SSD. The new instances can’t query the logs that are still in the memory of the old ingesters. Until the old instances flush these logs and the new instances load the uploaded index, queries don’t return the most recent logs.

With flush_on_shutdown: true, the old ingesters flush their chunks to object storage and upload their index when they receive the termination signal. Make sure that your deployment tool waits until the shutdown completes. The time depends on the amount of data in memory. Set the termination grace period accordingly.

After the instances stop, the new instances load the uploaded index files with their next index sync (storage_config.tsdb_shipper.resync_interval, default 5m). In a test with default settings, the flushed logs were queryable about 70 seconds after the old write instances stopped.

Verify the migration

1. Query a time range that starts before the traffic switch and compare the number of log lines with a count you have from before the migration or from your log shipper.
2. Check the logs of the new instances for errors.
3. If you use the ruler, list your rules with the ruler API of one instance and verify that the rules evaluate without errors.

Clean up

Terminate all old read and backend instances, and remove the old write instances. Keep the persistent volumes of the old write instances until you have verified that all logs are queryable. Then delete them.

Operate the HA monolithic deployment

• Restart instances one at a time and wait until an instance is ready before you restart the next one. Individual in-flight requests can fail.
• When the main instance restarts, the compactor starts again with the startup delay of one compactor.compaction_interval.

Troubleshooting