Migrate to Grafana Mimir on Grafana Labs

Migrate from Cortex to Grafana Mimir

Fri, 06 Mar 2026 14:53:36 -0800

Migrate from Cortex to Grafana Mimir

As an operator, you can migrate a Jsonnet deployment of Cortex to Grafana Mimir. The overview includes the steps required for any environment. To migrate deployment environments with Jsonnet, see Migrate to Grafana Mimir using Jsonnent.

Note: This document was tested with Cortex versions 1.10 and 1.11. It might work with more recent versions of Cortex, but that is not guaranteed.

To migrate a Helm deployment of Cortex refer to Migrate from Cortex.

Grafana Mimir includes significant changes that simplify the deployment and continued operation of a horizontally scalable, multi-tenant time series database with long-term storage.

The changes make Grafana Mimir easier to run out of the box:

Removed configuration parameters that don’t require tuning
Renamed some parameters so that they’re more easily understood
Updated the default values of some existing parameters

The mimirtool automates configuration conversion. It provides a simple migration by generating Mimir configuration from Cortex configuration.

Before you begin:

Ensure that you are running either Cortex 1.10.X or Cortex 1.11.X.

If you are running an older version of Cortex, upgrade to Cortex 1.11.1 before proceeding with the migration.
Ensure you have installed Cortex alerting and recording rules as well as Cortex dashboards.

Using the monitoring mixin, you need to install both alerting and recording rules in either Prometheus or Cortex. You also need to install dashboards in Grafana. To download a prebuilt ZIP file that contains the alerting and recording rules, refer to Release Cortex-jsonnet 1.11.0.

To upload rules to the ruler using mimirtool, refer to mimirtool rules. To import the dashboards into Grafana, refer to Import dashboard.

Notable changes

Note: For the full list of changes, refer to Mimir’s CHANGELOG.

The Grafana Mimir HTTP server defaults to listening on port 8080; Cortex defaults to listening on port 80. To maintain port 80 as the listening port, set -server.http-listen-port=80.
Grafana Mimir uses anonymous as the default tenant ID when -auth.multitenancy=false. Cortex uses fake as the default tenant ID when -auth.enabled=false. Use -auth.no-auth-tenant=fake when -auth.multitenancy=false to match the Cortex default tenant ID.

Grafana Mimir removes the legacy HTTP prefixes deprecated in Cortex.

Query endpoints

Legacy	Current
`/<legacy-http-prefix>/api/v1/query`	`<prometheus-http-prefix>/api/v1/query`
`/<legacy-http-prefix>/api/v1/query_range`	`<prometheus-http-prefix>/api/v1/query_range`
`/<legacy-http-prefix>/api/v1/query_exemplars`	`<prometheus-http-prefix>/api/v1/query_exemplars`
`/<legacy-http-prefix>/api/v1/series`	`<prometheus-http-prefix>/api/v1/series`
`/<legacy-http-prefix>/api/v1/labels`	`<prometheus-http-prefix>/api/v1/labels`
`/<legacy-http-prefix>/api/v1/label/{name}/values`	`<prometheus-http-prefix>/api/v1/label/{name}/values`
`/<legacy-http-prefix>/api/v1/metadata`	`<prometheus-http-prefix>/api/v1/metadata`
`/<legacy-http-prefix>/api/v1/read`	`<prometheus-http-prefix>/api/v1/read`
`/<legacy-http-prefix>/api/v1/cardinality/label_names`	`<prometheus-http-prefix>/api/v1/cardinality/label_names`
`/<legacy-http-prefix>/api/v1/cardinality/label_values`	`<prometheus-http-prefix>/api/v1/cardinality/label_values`
`/api/prom/user_stats`	`/api/v1/user_stats`

Distributor endpoints

Legacy endpoint	Current
`/<legacy-http-prefix>/push`	`/api/v1/push`
`/all_user_stats`	`/distributor/all_user_stats`
`/ha-tracker`	`/distributor/ha_tracker`

Ingester endpoints

Legacy	Current
`/ring`	`/ingester/ring`
`/shutdown`	`/ingester/shutdown`
`/flush`	`/ingester/flush`
`/push`	`/ingester/push`

Ruler endpoints

Legacy	Current
`/<legacy-http-prefix>/api/v1/rules`	`<prometheus-http-prefix>/api/v1/rules`
`/<legacy-http-prefix>/api/v1/alerts`	`<prometheus-http-prefix>/api/v1/alerts`
`/<legacy-http-prefix>/rules`	`<prometheus-http-prefix>/config/v1/rules`
`/<legacy-http-prefix>/rules/{namespace}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}`
`/<legacy-http-prefix>/rules/{namespace}/{groupName}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}`
`/<legacy-http-prefix>/rules/{namespace}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}`
`/<legacy-http-prefix>/rules/{namespace}/{groupName}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}`
`/<legacy-http-prefix>/rules/{namespace}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}`
`/ruler_ring`	`/ruler/ring`

Alertmanager endpoints

Legacy	Current
`/<legacy-http-prefix>`	`/alertmanager`
`/status`	`/multitenant_alertmanager/status`

Generate the configuration for Grafana Mimir

The mimirtool config convert command converts Cortex configuration to Mimir configuration. You can use it to update both flags and configuration files.

Install mimirtool

To install Mimirtool, download the appropriate binary from the latest release for your operating system and architecture and make it executable.

Alternatively, use a command line tool such as curl to download mimirtool. For example, for Linux with the AMD64 architecture, use the following command:

curl -fLo mimirtool https://github.com/grafana/mimir/releases/latest/download/mimirtool-linux-amd64
chmod +x mimirtool

Use mimirtool

The mimirtool config convert command converts Cortex 1.11 configuration files to Grafana Mimir configuration files. It removes any configuration parameters that are no longer available in Grafana Mimir, and it renames configuration parameters that have a new name. If you have explicitly set configuration parameters to a value matching the Cortex default, by default, mimirtool config convert doesn’t update the value. To have mimirtool config convert update explicitly set values from the Cortex defaults to the new Grafana Mimir defaults, provide the --update-defaults flag. Refer to convert for more information on using mimirtool for configuration conversion.

Migrate to Grafana Mimir using Jsonnet

Grafana Mimir has a Jsonnet library that replaces the existing Cortex Jsonnet library and updated monitoring mixin.

Migrate to Grafana Mimir video

The following video shows you how to migrate to Grafana Mimir using Jsonnet.

{ responded = true; response && response.status === 200 ? vimeo_is_up = true : vimeo_is_up = false; }) .catch(error => { responded = true; })" >

Migrate to Grafana Mimir instructions

The following instructions describe how to migrate to Grafana Mimir using Jsonnet.

To install the updated libraries using jsonnet-bundler, run the following commands:

jb install github.com/grafana/mimir/operations/mimir@main
jb install github.com/grafana/mimir/operations/mimir-mixin@main

To deploy the updated Jsonnet:

Install the updated monitoring mixin.

a. Add the dashboards to Grafana. The dashboards replace your Cortex dashboards and continue to work for monitoring Cortex deployments.

Note: Resource dashboards are now enabled by default and require additional metrics sources. To understand the required metrics sources, refer to Additional resources metrics.

b. Install the recording and alerting rules into the ruler or a Prometheus server.
Replace the import of the Cortex Jsonnet library with the Mimir Jsonnet library. For example:
jsonnet
```
import 'github.com/grafana/mimir/operations/mimir/mimir.libsonnet'
```
Remove the cortex_ prefix from any member keys of the <MIMIR>._config object. For example, cortex_compactor_disk_data_size becomes compactor_disk_data_size.
If you are using the Cortex defaults, set the server HTTP port to 80. The new Mimir default is 8080. For example:
jsonnet
```
(import 'github.com/grafana/mimir/operations/mimir/mimir.libsonnet') {
  _config+: {
    server_http_port: 80,
  },
}
```
For each component, use mimirtool to update the configured arguments. To extract the flags for each component, refer to Extracting flags from Jsonnet.
Apply the updated Jsonnet

To verify that the cluster is operating correctly, use the monitoring mixin dashboards.

Migrate from Thanos or Prometheus to Grafana Mimir

Fri, 06 Mar 2026 14:53:36 -0800

Migrate from Thanos or Prometheus to Grafana Mimir

This document guides an operator through the process of migrating a deployment of Thanos or Prometheus to Grafana Mimir.

Overview

Grafana Mimir stores series in TSDB blocks uploaded in an object storage bucket. These blocks are the same as those used by Prometheus and Thanos. Each project stores blocks in different places and uses slightly different block metadata files.

Configuring remote write to Grafana Mimir

For configuration of remote write to Grafana Mimir, refer to Configuring Prometheus remote write.

Uploading historic TSDB blocks to Grafana Mimir

Grafana Mimir supports uploading of historic TSDB blocks, notably from Prometheus. In order to enable this functionality, either for all tenants or a specific one, refer to Configure TSDB block upload.

Prometheus stores TSDB blocks in the path specified in the --storage.tsdb.path flag.

To find all block directories in the TSDB <STORAGE TSDB PATH>, run the following command:

find <STORAGE TSDB PATH> -name chunks -exec dirname {} \;

Grafana Mimir supports multiple tenants and stores blocks per tenant. With multi-tenancy disabled, there is a single tenant called anonymous.

Use Grafana mimirtool to upload each block, such as those identified by the previous command, to Grafana Mimir:

mimirtool backfill --address=http://<mimir-hostname> --id=<tenant> <block1> <block2>...

Note: If you need to authenticate against Grafana Mimir, you can provide an API key via the --key flag, for example --key=$(cat token.txt).

Grafana Mimir performs some sanitization and validation of each block’s metadata. As a result, it rejects Thanos blocks due to unsupported labels. As a workaround, if you need to upload Thanos blocks, upload the blocks directly to the Grafana Mimir blocks bucket, prefixed by <tenant>/<block ID>/.

Block metadata

Each block has a meta.json metadata file that is used by Grafana Mimir, Prometheus, and Thanos to identify the block contents. Each project has its own metadata conventions.

In the Grafana Mimir 2.1 (or earlier) release, the ingesters added an external label to the meta.json file to identify the tenant that owns the block.

In the Grafana Mimir 2.2 (or later) release, blocks no longer have a label that identifies the tenant.

Note: Blocks from Prometheus do not have any external labels stored in them; only blocks from Thanos use labels.

Considerations on Thanos specific features

Thanos requires that Prometheus is configured with external labels. When the Thanos sidecar uploads blocks, it includes the external labels from Prometheus in the meta.json file inside the block. When you query the block, Thanos injects Prometheus’ external labels in the series returned in the query result. Thanos also uses labels for the deduplication of replicated data.

If you want to use existing blocks from Thanos by Grafana Mimir, there are some considerations:

Grafana Mimir doesn’t inject external labels into query results. This means that blocks that were originally created by Thanos will not include their external labels in the results when queried by Grafana Mimir. If you need to have external labels in your query results, this is currently not possible to achieve in Grafana Mimir.

Grafana Mimir will not respect deduplication labels configured in Thanos when querying the blocks. For best query performance, only upload Thanos blocks from a single Prometheus replica from each HA pair. If you upload blocks from both replicas, the query results returned by Mimir will include samples from both replicas.

Grafana Mimir does not support Thanos’ downsampling feature. To guarantee query results correctness please only upload original (raw) Thanos blocks into Mimir’s storage. If you also upload blocks with downsampled data (ie. blocks with non-zero Resolution field in meta.json file), Grafana Mimir will merge raw samples and downsampled samples together at the query time. This may cause that incorrect results are returned for the query.

Migrate historic TSDB blocks from Thanos to Grafana Mimir

Copy the blocks from Thanos’s bucket to an Intermediate bucket.

Create an intermediate object storage bucket (such as Amazon S3 or GCS) within your cloud provider, where you can copy the historical blocks and work on them before uploading them to the Mimir bucket.

Tip: Run the commands within a screen or tmux session, to avoid any interruptions because the steps might take time depending on the amount of data being processed.

For Amazon S3, use the aws tool:
Bash
```
aws s3 cp -r s3://<THANOS-BUCKET> s3://<INTERMEDIATE-MIMIR-BUCKET>/
```
For Google Cloud Storage (GCS), use the gsutil tool:
Bash
```
gsutil -m cp -r gs://<THANOS-BUCKET>/* gs://<INTERMEDIATE-MIMIR-BUCKET>/
```
After the copy process completes, inspect the blocks in the bucket to make sure that they are valid from a Thanos perspective.
Bash
```
thanos tools bucket inspect \
    --objstore.config-file bucket.yaml
```
Remove the downsampled blocks.

Mimir doesn’t understand the downsampled blocks from Thanos, such as blocks with a non-zero Resolution field in the meta.json file. Therefore, you need to remove the 5m and 1h downsampled blocks from this bucket.

Mark the downsampled blocks for deletion:
Bash
```
thanos tools bucket retention \
    --objstore.config-file bucket.yaml \
    --retention.resolution-1h=1s \
    --retention.resolution-5m=1s \
    --retention.resolution-raw=0s
```
Cleanup the blocks marked for deletion.
Bash
```
thanos tools bucket cleanup \
    --objstore.config-file bucket.yaml \
    --delete-delay=0
```

Remove the duplicated blocks.

If two replicas of Prometheus instances are deployed for high-availability, then only upload the blocks from one of the replicas and drop from the other replica.

# Get list of all blocks in the bucket
thanos tools bucket inspect \
    --objstore.config-file bucket.yaml \
    --output=tsv > blocks.tsv

# Find blocks from replica that we will drop
cat blocks.tsv| grep prometheus_replica=<PROMETHEUS-REPLICA-TO-DROP> \
    | awk '{print $1}' > blocks_to_drop.tsv

# Mark found blocks for deletion
for ID in $(cat blocks_to_drop.tsv)
do
    thanos tools bucket mark \
       --marker="deletion-mark.json" \
       --objstore.config-file bucket.yaml \
       --details="Removed as duplicate" \
       --id $ID
done

Note: Replace prometheus_replica with the unique label that would differentiate Prometheus replicas in your setup.

Clean up the duplicate blocks marked for deletion again:

thanos tools bucket cleanup \
    --objstore.config-file bucket.yaml \
    --delete-delay=0

Tip: If you want to visualize exactly what is happening in the blocks, with respect to the source of blocks, external labels, compaction levels, and more, you can use the following command to get the output as CSV and import it into a Google spreadsheet:
Bash
thanos tools bucket inspect \
    --objstore.config-file bucket-prod.yaml \
    --output=csv > thanos-blocks.csv

Relabel the blocks with external labels.

Mimir doesn’t inject external labels from the meta.json file into query results. Therefore, you need to relabel the blocks with the required external labels in the meta.json file.

Note: You can get the external labels in the meta.json file of each block from the CSV file that is imported (as mentioned in the preceding tip), and build the rewrite configuration accordingly.

Create a rewrite configuration that is similar to this:

# relabel-config.yaml
- action: replace
  target_label: "<LABEL-KEY>"
  replacement: "<LABEL-VALUE>"

Perform the rewrite dry run to confirm all works well.

# Get list of all blocks in the bucket after removing the depuplicate and downsampled blocks.
thanos tools bucket inspect \
    --objstore.config-file bucket.yaml \
    --output=tsv > blocks-to-rewrite.tsv

# Check if rewrite of the blocks with external labels is working as expected.
for ID in $(cat blocks-to-rewrite.tsv)
do
    thanos tools bucket rewrite \
        --objstore.config-file bucket.yaml \
        --rewrite.to-relabel-config-file relabel-config.yaml \
        --dry-run \
        --id $ID
done

After you confirm that the rewrite is working as expected via --dry-run, apply the changes with the --no-dry-run flag. Remember to include --delete-blocks, otherwise the original blocks will not be marked for deletion.

# Rewrite the blocks with external labels and mark the original blocks for deletion.
for ID in $(cat blocks-to-rewrite.tsv)
do
    thanos tools bucket rewrite  \
        --objstore.config-file bucket.yaml \
        --rewrite.to-relabel-config-file relabel-config.yaml \
        --no-dry-run \
        --delete-blocks \
        --id $ID
done

The output of relabelling of every block would look like something below.

level=info ts=2022-10-10T13:03:32.032820262Z caller=factory.go:50 msg="loading bucket configuration"
level=info ts=2022-10-10T13:03:32.516953867Z caller=tools_bucket.go:1160 msg="downloading block" source=01GEGWPME2187SVFH63G8DH7KH
level=info ts=2022-10-10T13:03:35.825009556Z caller=tools_bucket.go:1197 msg="changelog will be available" file=/tmp/thanos-  rewrite/01GF0ZWPWGEPHG5NV79NH9KMPV/change.log
level=info ts=2022-10-10T13:03:35.836953593Z caller=tools_bucket.go:1212 msg="starting rewrite for block" source=01GEGWPME2187SVFH63G8DH7KH  new=01GF0ZWPWGEPHG5NV79NH9KMPV toDelete= toRelabel="- action: replace\n  target_label: \"cluster\"\n  replacement: \"prod-cluster\"\n"
level=info ts=2022-10-10T13:04:47.57624244Z caller=compactor.go:42 msg="processed 10.00% of 701243 series"
level=info ts=2022-10-10T13:04:53.4046885Z caller=compactor.go:42 msg="processed 20.00% of 701243 series"
level=info ts=2022-10-10T13:04:59.649337602Z caller=compactor.go:42 msg="processed 30.00% of 701243 series"
level=info ts=2022-10-10T13:05:02.986219042Z caller=compactor.go:42 msg="processed 40.00% of 701243 series"
level=info ts=2022-10-10T13:05:05.990498497Z caller=compactor.go:42 msg="processed 50.00% of 701243 series"
level=info ts=2022-10-10T13:05:09.349918024Z caller=compactor.go:42 msg="processed 60.00% of 701243 series"
level=info ts=2022-10-10T13:05:12.040895624Z caller=compactor.go:42 msg="processed 70.00% of 701243 series"
level=info ts=2022-10-10T13:05:15.253899238Z caller=compactor.go:42 msg="processed 80.00% of 701243 series"
level=info ts=2022-10-10T13:05:18.471471014Z caller=compactor.go:42 msg="processed 90.00% of 701243 series"
level=info ts=2022-10-10T13:05:21.536267363Z caller=compactor.go:42 msg="processed 100.00% of 701243 series"
level=info ts=2022-10-10T13:05:21.536466158Z caller=tools_bucket.go:1222 msg="wrote new block after modifications; flushing" source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV
level=info ts=2022-10-10T13:05:28.675240198Z caller=tools_bucket.go:1231 msg="uploading new block" source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV
level=info ts=2022-10-10T13:05:38.922348564Z caller=tools_bucket.go:1241 msg=uploaded source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV
level=info ts=2022-10-10T13:05:38.979696873Z caller=block.go:203 msg="block has been marked for deletion" block=01GEGWPME2187SVFH63G8DH7KH
level=info ts=2022-10-10T13:05:38.979832767Z caller=tools_bucket.go:1249 msg="rewrite done" IDs=01GEGWPME2187SVFH63G8DH7KH
level=info ts=2022-10-10T13:05:38.980197796Z caller=main.go:161 msg=exiting

Cleanup the original blocks which are marked for deletion.

thanos tools bucket cleanup \
    --objstore.config-file bucket.yaml \
    --delete-delay=0

Note: If there are multiple prometheus clusters, then relabelling each of them in parallel may speed up the entire process. Get the list of blocks that for each cluster, and process it separately.
Bash
thanos tools bucket inspect \
    --objstore.config-file bucket.yaml \
    --output=tsv \
    | grep <PROMETHEUS-CLUSTER-NAME> \
    | awk '{print $1}' > prod-blocks.tsv

for ID in `cat prod-blocks.tsv`
do
    thanos tools bucket rewrite  \
       --objstore.config-file bucket.yaml \
       --rewrite.to-relabel-config-file relabel-config.yaml \
       --delete-blocks \
       --no-dry-run \
       --id $ID
done

Remove external labels from meta.json.

Mimir compactor will not be able to compact the blocks having external labels with Mimir’s own blocks that don’t have any such labels in their meta.json. Therefore these external labels have to be removed before copying them to the Mimir bucket.

Use the below script to remove the labels from the meta.json.

#!/bin/bash

BUCKET="GCS Bucket name"

echo "Fetching list of meta.json files (this can take a while if there are many blocks)"
gsutil ls "gs://$BUCKET/*/meta.json" > meta-files.txt

echo "Processing meta.json files"
for FILE in $(cat meta-files.txt); do
   echo "Removing Thanos labels from $FILE"
   ORIG_META_JSON=$(gsutil cat "$FILE")
   UPDATED_META_JSON=$(echo "$ORIG_META_JSON" | jq "del(.thanos.labels)")

   if ! diff -u <( echo "$ORIG_META_JSON" | jq . ) <( echo "$UPDATED_META_JSON" | jq .) > /dev/null; then
      echo "Backing up $FILE to $FILE.orig"
      gsutil cp "$FILE" "$FILE.orig"
      echo "Uploading modified $FILE"
      echo "$UPDATED_META_JSON" | gsutil cp - "$FILE"
   else
      echo "No diff for $FILE"
   fi
done

Copy the blocks from the intermediate bucket to the Mimir bucket.

For Amazon S3, use the aws tool:
Bash
```
aws s3 cp -r s3://<INTERMEDIATE-MIMIR-BUCKET> s3://<MIMIR-GCS-BUCKET>/<TENANT>/
```
For Google Cloud Storage (GCS), use the gsutil tool:
Bash
```
gsutil -m cp -r gs://<INTERMEDIATE-MIMIR-BUCKET> gs://<MIMIR-GCS-BUCKET>/<TENANT>/
```
Historical blocks are not available for querying immediately after they are uploaded because the bucket index with the list of all available blocks first needs to be updated by the compactor. The compactor typically perform such an update every 15 minutes. After an update completes, other components such as the querier or store-gateway are able to work with the historical blocks, and the blocks are available for querying through Grafana.
Check the store-gateway HTTP endpoint at http://<STORE-GATEWAY-ENDPOINT>/store-gateway/tenant/<TENANT-NAME>/blocks to verify that the uploaded blocks are there.

Migrate from Thanos to Mimir using Thanos sidecar

Fri, 06 Mar 2026 14:53:36 -0800

Migrate from Thanos to Mimir using Thanos sidecar

As an operator, you can migrate a deployment of Thanos to Grafana Mimir by using Thanos sidecar to move metrics over with a stepped workflow.

Overview

An option when migrating is to allow Thanos to query Mimir. This way you retain historical data via your existing Thanos deployment while pointing all of your Prometheus servers (or grafana agents and other metric sources) to Mimir.

The overall setup consists of setting up Thanos Sidecar alongside Mimir and then pointing Thanos Query to the sidecar as if it was just a normal sidecar.

This is not so much of a guide as it is a collection of configurations that have been used with success prior.

Warning: This setup is unsupported, experimental and has not been battle tested. Your mileage may vary, ensure you have tested this for your use case and have made appropriate backups (and tested your procedures for recovery)

Technical details

There are very few obstacles to get this working properly. Thanos Sidecar is designed to work with the Prometheus API which Mimir (almost) implements. There are only 2 endpoints that are not implemented that we need to “spoof” to get Thanos sidecar to believe it is connecting to a Prometheus server: /api/v1/status/buildinfo and /api/v1/status/config. We spoof these endpoints using NGINX (more details later).

The only other roadblock is the requirement for requests to Mimir to contain the X-Scope-Org-Id header to identify the Tenant. We inject this header using another NGINX container sitting in between Thanos Sidecar and Mimir.

In our case, everything was being deployed in Kubernetes. Mimir was deployed using the mimir-distributed helm chart. Therefore configurations shown will be Kubernetes manifests that will need to be modified for your environment. If you are not using Kubernetes then you will need to set up the appropriate configuration for NGINX to implement the technical setup described above. You may use the configurations below for inspiration but they will not work out of the box.

Deploy Thanos sidecar

We deployed the sidecar using Kubernetes using the below manifest:

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-sidecar
  namespace: mimir
  labels:
    app: thanos-sidecar
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thanos-sidecar
  template:
    metadata:
      labels:
        app: thanos-sidecar
    spec:
      volumes:
        - name: config
          configMap:
            name: sidecar-nginx
            defaultMode: 420
      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:1.19-alpine
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          volumeMounts:
            - name: config
              mountPath: /etc/nginx
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
        - name: thanos-sidecar
          image: quay.io/thanos/thanos:v0.26.0
          args:
            - sidecar
            - "--prometheus.url=http://localhost:8080/prometheus"
            - "--grpc-address=:10901"
            - "--http-address=:10902"
            - "--log.level=info"
            - "--log.format=logfmt"
          ports:
            - name: http
              containerPort: 10902
              protocol: TCP
            - name: grpc
              containerPort: 10901
              protocol: TCP
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

Service

apiVersion: v1
kind: Service
metadata:
  name: thanos-sidecar
  namespace: mimir
  labels:
    app: thanos-sidecar
spec:
  ports:
    - name: 10901-10901
      protocol: TCP
      port: 10901
      targetPort: 10901
  selector:
    app: thanos-sidecar
  type: ClusterIP

NGINX configuration for Thanos sidecar

Notice that there is an NGINX container within the sidecar deployment definition. This NGINX instance sits between the sidecar and Mimir (ie. sidecar --> nginx --> mimir) it is responsible for injecting the tenant ID into the requests from the sidecar to Mimir. The configuration that this NGINX instance uses is as follows:

worker_processes  5;  ## Default: 1
error_log  /dev/stderr;
pid        /tmp/nginx.pid;
worker_rlimit_nofile 8192;

events {
  worker_connections  4096;  ## Default: 1024
}

http {
  client_body_temp_path /tmp/client_temp;
  proxy_temp_path       /tmp/proxy_temp_path;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;

  default_type application/octet-stream;
  log_format   main '$remote_addr - $remote_user [$time_local]  $status '
        '"$request" $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for"';
  access_log   /dev/stderr  main;

  sendfile     on;
  tcp_nopush   on;
  resolver kube-dns.kube-system.svc.cluster.local;

  server {
    listen 8080;

    # Distributor endpoints
    location / {
      proxy_set_header X-Scope-OrgID 1;
      proxy_pass      http://mimir-distributed-nginx.mimir.svc.cluster.local:80$request_uri;
    }
  }
}

NGINX configuration for Mimir

Now we modified the Mimir Distributed NGINX configuration to allow the sidecar to fetch the “external labels” and the “version” of our “Prometheus” server. You should be careful with the external labels to make sure that you don’t override any existing labels (in our case we used “source=‘mimir’”). You should also consider existing dashboards and make sure that they will continue to work as expected. Make sure to test your setup first.

In particular, these location blocks were added (full config further down):

location /prometheus/api/v1/status/config {
        add_header Content-Type application/json;
        return 200 "{\"status\":\"success\",\"data\":{\"yaml\": \"global:\\n  external_labels:\\n    source: mimir\"}}";
    }

location /prometheus/api/v1/status/buildinfo {
        add_header Content-Type application/json;
        return 200 "{\"status\":\"success\",\"data\":{\"version\":\"2.35.0\",\"revision\":\"6656cd29fe6ac92bab91ecec0fe162ef0f187654\",\"branch\":\"HEAD\",\"buildUser\":\"root@cf6852b14d68\",\"buildDate\":\"20220421-09:53:42\",\"goVersion\":\"go1.18.1\"}}";
    }

You need to modify the first one to set your external labels. The second one just allows Thanos to detect the version of Prometheus running.

To modify the external labels alter this string: "{\"status\":\"success\",\"data\":{\"yaml\": \"global:\\n external_labels:\\n source: mimir\"}}". You will notice it is escaped JSON. Simply add elements to the data.external_labels field to add more external labels as required or alter the existing key to modify them.

Full Mimir distributed NGINX configuration

worker_processes  5;  ## Default: 1
error_log  /dev/stderr;
pid        /tmp/nginx.pid;
worker_rlimit_nofile 8192;

events {
  worker_connections  4096;  ## Default: 1024
}

http {
  client_body_temp_path /tmp/client_temp;
  proxy_temp_path       /tmp/proxy_temp_path;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;

  default_type application/octet-stream;
  log_format   main '$remote_addr - $remote_user [$time_local]  $status '
        '"$request" $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for"';
  access_log   /dev/stderr  main;

  sendfile     on;
  tcp_nopush   on;
  resolver kube-dns.kube-system.svc.cluster.local;

  server {
    listen 8080;

    location = / {
      return 200 'OK';
      auth_basic off;
    }

    # Distributor endpoints
    location /distributor {
      proxy_pass      http://mimir-distributed-distributor.mimir.svc.cluster.local:8080$request_uri;
    }
    location = /api/v1/push {
      proxy_pass      http://mimir-distributed-distributor.mimir.svc.cluster.local:8080$request_uri;
    }

    # Alertmanager endpoints
    location /alertmanager {
      proxy_pass      http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$request_uri;
    }
    location = /multitenant_alertmanager/status {
      proxy_pass      http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$request_uri;
    }
    location = /api/v1/alerts {
      proxy_pass      http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$request_uri;
    }

    # Ruler endpoints
    location /prometheus/config/v1/rules {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }
    location /prometheus/api/v1/rules {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }

    location /api/v1/rules {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }
    location /prometheus/api/v1/alerts {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }
    location /prometheus/rules {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }
    location = /ruler/ring {
      proxy_pass      http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$request_uri;
    }

    location /prometheus/api/v1/status/config {
      add_header Content-Type application/json;
      return 200 "{\"status\":\"success\",\"data\":{\"yaml\": \"global:\\n  external_labels:\\n    source: mimir\"}}";
    }

    location /prometheus/api/v1/status/buildinfo {
    add_header Content-Type application/json;
    return 200 "{\"status\":\"success\",\"data\":{\"version\":\"2.35.0\",\"revision\":\"6656cd29fe6ac92bab91ecec0fe162ef0f187654\",\"branch\":\"HEAD\",\"buildUser\":\"root@cf6852b14d68\",\"buildDate\":\"20220421-09:53:42\",\"goVersion\":\"go1.18.1\"}}";
    }



    # Rest of /prometheus goes to the query frontend
    location /prometheus {
      proxy_pass      http://mimir-distributed-query-frontend.mimir.svc.cluster.local:8080$request_uri;
    }

    # Buildinfo endpoint can go to any component
    location = /api/v1/status/buildinfo {
      proxy_pass      http://mimir-distributed-query-frontend.mimir.svc.cluster.local:8080$request_uri;
    }
  }
}