Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
Migrating from Grafana Enterprise Metrics 1.7
As an operator, you can migrate a deployment of Grafana Enterprise Metrics 1.7 to Grafana Enterprise Metrics 2.0. The steps to do so apply to any deployment environment, and there are specific instructions for environments deployed with Helm.
Grafana Enterprise Metrics 2.0 extends Grafana Mimir. Grafana Enterprise Metrics 1.7 extends Cortex. Grafana Mimir includes significant changes that simplify the deployment and continuous operation of a horizontally scalable, multi-tenant time series database with long-term storage.
The following changes make Grafana Mimir easy to run:
- Removed configuration parameters that don’t require tuning
- Renamed some parameters so that they’re more easy to understand
- Updated the default values of some existing parameters to work out of the box
The Grafana Mimirtool (mimirtool
) automates configuration conversion for you.
It provides a simple migration by generating a Grafana Mimir configuration from a Cortex configuration.
It can also generate a Grafana Enterprise Metrics 2.0 configuration from a Grafana Enterprise Metrics 1.7 configuration.
Before you begin
Ensure that you are running Grafana Enterprise Metrics 1.7.X.
If you are running an older version of Grafana Enterprise Metrics, upgrade to 1.7.0 before proceeding with the migration.
Ensure that you have installed the Grafana Mimir monitoring mixin rules.
The monitoring mixin has both alerting and recording rules to install in either Prometheus or Grafana Enterprise Metrics. The rules replace the Cortex monitoring mixin rules and work for monitoring both Grafana Enterprise Metrics 1.7 and Grafana Enterprise Metrics 2.0. To upload rules to the ruler using Grafana Mimirtool, refer to Grafana Mimirtool rules.
Ensure that you have installed the Grafana Mimir monitoring mixin dashboards.
The dashboards replace the Cortex monitoring mixin dashboards and work for monitoring both Grafana Enterprise Metrics 1.7 and Grafana Enterprise Metrics 2.0. Alternatively, update the Grafana Enterprise Metrics plugin to automatically install the latest self monitoring dashboards.
Notable changes
Note: For a full list of changes, refer to the Grafana Mimir CHANGELOG.
By default, Grafana Enterprise Metrics 2.0 enables
enterprise
authentication. To continue using the Grafana Enterprise Metrics 1.7 default namedtrust
, use-auth.type=trust
.The Grafana Enterprise Metrics 2.0 HTTP server defaults to listening on port 8080; previous versions defaulted to listening on port 80. The
enterprise-metrics
Helm chart already used port 8080 and is unaffected by this change.Grafana Enterprise Metrics 2.0 uses
anonymous
as the default tenant ID where-auth.multitenancy=false
. Previous versions usedfake
as the default tenant ID where-auth.enabled=false
. Use-auth.no-auth-tenant=fake
where-auth.multitenancy=false
, to match the default tenant ID of previous versions.Grafana Enterprise Metrics 2.0 removes the legacy HTTP prefixes.
Query endpoints
Legacy Current /<legacy-http-prefix>/api/v1/query
<prometheus-http-prefix>/api/v1/query
/<legacy-http-prefix>/api/v1/query_range
<prometheus-http-prefix>/api/v1/query_range
/<legacy-http-prefix>/api/v1/query_exemplars
<prometheus-http-prefix>/api/v1/query_exemplars
/<legacy-http-prefix>/api/v1/series
<prometheus-http-prefix>/api/v1/series
/<legacy-http-prefix>/api/v1/labels
<prometheus-http-prefix>/api/v1/labels
/<legacy-http-prefix>/api/v1/label/{name}/values
<prometheus-http-prefix>/api/v1/label/{name}/values
/<legacy-http-prefix>/api/v1/metadata
<prometheus-http-prefix>/api/v1/metadata
/<legacy-http-prefix>/api/v1/read
<prometheus-http-prefix>/api/v1/read
/<legacy-http-prefix>/api/v1/cardinality/label_names
<prometheus-http-prefix>/api/v1/cardinality/label_names
/<legacy-http-prefix>/api/v1/cardinality/label_values
<prometheus-http-prefix>/api/v1/cardinality/label_values
/api/prom/user_stats
/api/v1/user_stats
Distributor endpoints
Legacy endpoint Current /<legacy-http-prefix>/push
/api/v1/push
/all_user_stats
/distributor/all_user_stats
/ha-tracker
/distributor/ha_tracker
Ingester endpoints
Legacy Current /ring
/ingester/ring
/shutdown
/ingester/shutdown
/flush
/ingester/flush
/push
/ingester/push
Ruler endpoints
Legacy Current /<legacy-http-prefix>/api/v1/rules
<prometheus-http-prefix>/api/v1/rules
/<legacy-http-prefix>/api/v1/alerts
<prometheus-http-prefix>/api/v1/alerts
/<legacy-http-prefix>/rules
/api/v1/rules
(see below)/<legacy-http-prefix>/rules/{namespace}
/api/v1/rules/{namespace}
(see below)/<legacy-http-prefix>/rules/{namespace}/{groupName}
/api/v1/rules/{namespace}/{groupName}
(see below)/<legacy-http-prefix>/rules/{namespace}
/api/v1/rules/{namespace}
(see below)/<legacy-http-prefix>/rules/{namespace}/{groupName}
/api/v1/rules/{namespace}/{groupName}
(see below)/<legacy-http-prefix>/rules/{namespace}
/api/v1/rules/{namespace}
(see below)/ruler_ring
/ruler/ring
Note: The
/api/v1/rules/**
endpoints are considered deprecated with Grafana Enterprise Metrics 2.0.0 and will be removed in Grafana Enterprise Metrics 2.2.0. After upgrading to 2.0.0, we recommend switching to the equivalent/<prometheus-http-prefix>/config/v1/**
endpoints that Grafana Enterprise Metrics 2.0.0 introduces.Alertmanager endpoints
Legacy Current /<legacy-http-prefix>
/alertmanager
/status
/multitenant_alertmanager/status
Generating configuration for Grafana Enterprise Metrics 2.0
Grafana Mimirtool provides the mimirtool config convert --gem
command for converting a Grafana Enterprise Metrics 1.7 configuration into a Grafana Enterprise Metrics 2.0 configuration.
It can convert both flags and configuration files.
Downloading Grafana Mimirtool
Download the appropriate release asset for your operating system and architecture, and make it executable. For Linux with the AMD64 architecture:
curl -fLo mimirtool https://github.com/grafana/mimir/releases/latest/download/mimirtool-linux-amd64
chmod +x mimirtool
Using Grafana Mimirtool
Note: To ensure that the Grafana Enterprise Metrics specific configuration isn’t removed from the output, you must use the
--gem
flag.
The tool removes any configuration parameters that are no longer available in Grafana Enterprise Metrics 2.0, and it renames any configuration parameter that has a new name.
If you have explicitly set configuration parameters to a value that matches the Grafana Enterprise Metrics 1.7 default, by default, mimirtool config convert
doesn’t update the value.
To have mimirtool config convert --gem
update explicitly set values from the Grafana Enterprise Metrics 1.7 defaults to the new Grafana Enterprise Metrics 2.0 defaults, provide the --update-defaults
flag.
For more information about using mimirtool
for configuration conversion, refer to the Grafana Mimirtool configuration command named convert
.
Migrating to Grafana Enterprise Metrics 2.0 with Helm
Grafana Enterprise Metrics 2.0 uses the Grafana Mimir mimir-distributed
Helm chart.
You can update to the mimir-distributed
Helm chart from the enterprise-metrics
Helm chart.
Before you begin
- Ensure that you are running the v1.8.1 release of the
enterprise-metrics
Helm chart.
To migrate to the mimir-distributed
Helm chart:
Update your Helm repositories.
helm repo update
Convert your existing Grafana Enterprise Metrics configuration in your Helm values file.
a. Remove the
config
member at the top level of your Helm values file. Themimir-distributed
chart uses a string configuration at the keymimir.config
instead.b. Enable enterprise configuration by setting
enterprise.enabled
totrue
andenterprise.legacyLabels
totrue
. The valueenterprise.legacyLabels
installs resources using the same labels as theenterprise-metrics
Helm chart.In your Helm values file:
enterprise: enabled: true legacyLabels: true
c. If you’re using the
enterprise-metrics
chart MinIO for object storage, set theminio.accesskey
toenterprise-metrics
In your Helm values file:
minio: accesskey: "enterprise-metrics"
d. Extract the updated default configuration from the
mimir-distributed
Helm chart into themimir-distributed.yaml
file. The following command requires the Goyq
tool.helm template <RELEASE NAME> -f <VALUES FILE> grafana/mimir-distributed | yq 'select(.metadata.name == "<RELEASE NAME>-enterprise-metrics-config") | .data."mimir.yaml"' | base64 -d > mimir-distributed.yaml
e. Extract the Grafana Enterprise Metrics configuration from the deployed Grafana Enterprise Metrics Secret and write the output to the
enterprise-metrics.yaml
file.kubectl get secret <RELEASE>-enterprise-metrics -o json | jq -r '.data."enterprise-metrics.yaml"' | base64 -d > enterprise-metrics-1.7.yaml
If using an external configuration Secret, replace
<RELEASE>-enterprise-metrics
with the name of your Secret.f. Use
mimirtool
to convert the configuration, writing the output YAML to theenterprise-metrics-2.0.yaml
file.mimirtool config convert --gem --yaml-file enterprise-metrics-1.7.yaml --yaml-out enterprise-metrics-2.0.yaml
g. Merge the converted
enterprise-metrics-2.0.yaml
into themimir-distributed.yaml
defaults into theconfig.yaml
file. The following command requires the Goyq
tool.yq '. *= load("enterprise-metrics-2.0.yaml")' mimir-distributed.yaml > config.yaml
h. In
config.yaml
, setactivity_tracker.filepath
to/data/metrics-activity.log
. Themimir-distributed
chart uses a different writable data directory to theenterprise-metrics
chart.i. In
config.yaml
, setfrontend.cache_results
tofalse
andfrontend.results_cache.backend
to''
. The query-frontend results cache isn’t yet available in themimir-distributed
Helm chart.j. In your Helm values file, add the contents of the
config.yaml
file as the value for themimir.config
key.Note: The Grafana Mimir Helm chart expects the configuration as a string value. You can provide a literal block string with the
|
symbol.In your Helm values file:
mimir: config: | <CONFIG.YAML CONTENTS>
Run the Helm upgrade with the
mimir-distributed
chart.During the upgrade, requests to the gateway fail for a short period of time. Prometheus and Grafana Agent retry failed write requests to avoid data loss.
Note: The name of the release must match your
enterprise-metrics
Helm chart release.helm upgrade <RELEASE> grafana/mimir-distributed -f <VALUES FILE> --set-file 'license.contents=<LICENSE FILE>' [-n <NAMESPACE>]
To verify that the cluster is operating correctly, use the self monitoring or monitoring mixin dashboards installed in Grafana.