The admin API allows you to create and manage resources in your Grafana Enterprise Metrics (GEM) cluster such as tenants, access policies, and tokens. When performing operations that modify resources such as updates or deletes, the admin API requires the use of resource version numbers to make sure the most recent version of resources are being operated on. An additional safety feature the admin API makes use of to ensure concurrent modifications don’t overwrite each other is leadership election. Admin API components elect a single component to perform all modification requests, making sure that simultaneous requests don’t overwrite each other. Details about how to configure or disable this feature are provided below.
Leadership election is a recommended feature of the GEM admin API, and is therefore enabled by default. It can be
configured or disabled under the
admin_api section of the GEM
Leadership election uses the same mechanism as other GEM components to coordinate among themselves, the
hash ring. This means you’ll need to configure
this instance of the ring the same way you configured the ring used by distributors and ingesters when
setting up your GEM cluster.
An example of this configuration is provided below using the memberlist gossip ring.
admin_api: leader_election: ring: kvstore: store: memberlist tokens_observe_period: 0
Note that any supported backend (Etcd, Consul, or memberlist) can be used for leader election with the admin API, memberlist is only used as an example here. For simplicity, it is recommended to use whatever ring backend you are already making use of for other GEM components.
Admin API leadership election does not require a quorum to operate like other uses of the ring in GEM components. This means it is not required to run an odd number of admin API components. Instead, it is sufficient to run two for redundancy.
Disabling Leadership Election
Leadership election can be disabled under the
admin_api section of the GEM
configuration file. This is not recommended and
leadership election will be enabled permanently in a future release.
admin_api: leader_election: enabled: false
Related Enterprise Metrics resources
Running Prometheus-as-a-service with Grafana Enterprise Metrics
Introducing Grafana Enterprise Metrics (GEM), a simple and scalable Prometheus service that is seamless to use, simple to maintain, and supported by Grafana Labs.
How Robinhood scaled from 100M to 700M time series with Grafana Enterprise Metrics
In this GrafanaCONline session, the Robinhood team tells how GME (GameStop) led to GEM (Grafana Enterprise Metrics).
Benchmarking Grafana Enterprise Metrics for horizontally scaling Prometheus up to 500 million active series
We stress-tested GEM to show how it horizontally scaled. One takeaway: Hardware usage scales linearly up to 500 million active series.