Grafana security update: GPG signing key rotation

Update 2023-09-06 19:30 UTC: The post-incident review and timeline have been posted on the blog.

Update 2023-08-31 16:00 UTC: The post-incident review and full timeline were not yet finalized when we originally published this blog post. However, we decided to publish the below as soon as possible because we strongly believe in transparency. We will be sharing the incident timeline when it’s ready.

On Aug. 24, the GPG private key and passphrase with the ID of 0E22EB88E39E12277A7760AE9E439B102CF3C0C6 was unintentionally shared. As a best practice, we have revoked the exposed certificate on our full GPG public keychain and issued a new public key. 

As a result, users of our apt and yum repositories will need to update their trusted certificate to safely continue using our repositories.

GPG key rotation and trust relationship

Who is affected:

• Users who have installed Grafana via our package repositories (packages.grafana.com, apt.grafana.com, rpm.grafana.com)
• Users who use our binary .deb or .rpm releases and validate the GPG signatures with a local copy of the impacted GPG key

Who is not affected:

• Users who have installed Grafana via their OS package repository
• Users who have installed Grafana via a Docker image
• Grafana Cloud/hosted users

If you are affected, you should follow the instructions below to remove the impacted GPG key from your keyrings and install our new key.

How to update your GPG trust relationship

Debian/Ubuntu

Copy

# Remove the old key

apt-key del 0E22EB88E39E12277A7760AE9E439B102CF3C0C6

# Verify that the GPG key was removed. This should return 0.

apt-key list | grep -i grafana | wc -l

# Download the new key and place it in the right location

sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

rpm (RHEL, CentOS, Fedora, etc.)

Our rpm-based installation instructions do not add the identified GPG key to the package manager, but rather suggest downloading the .rpm and gpg key and performing a manual verification. If you have saved a local copy of the Grafana gpg key (fingerprint: 0E22EB88E39E12277A7760AE9E439B102CF3C0C6), you will need to replace it with the rotated key, which can be downloaded at https://rpm.grafana.com/gpg.key.

yum and dnf (RHEL, CentOS, Fedora, etc.)

yum and dnf cache repository gpg keys. In order to force yum or dnf to fetch and use the rotated gpg key, you will need to flush this cache. Instructions are as follows:

1. Locate your yum cache directory. This will default to /var/cache/dnf for systems running dnf, but may be at /var/lib/yum/repos/ or /etc/yum/repos.d on systems running original yum. If you don’t see either of these, check your yum.conf file for a persistdir parameter. This parameter overrides the default and specifies a cache directory manually.
2. Locate the cache for the Grafana repository inside your yum cache directory. On dnf systems, this should be something like /var/cache/dnf/grafana-369737ffb0934a7b/. On yum systems this will likely be an architecture-specific subdirectory under /var/lib/yum/repos/.
3. Remove the identified GPG key from the cached keyring. Inside the Grafana repository cache directory, there will either be a pubring folder or a gpgdir and gpgdir-ro folders. These are where the keyrings are actually cached. To remove the identified GPG key from the cached keyring, use the following gpg command:

Copy

gpg --batch --homedir YOUR_PATH_GOES_HERE --delete-key 0E22EB88E39E12277A7760AE9E439B102CF3C0C6

If you have both a gpgdir and a gpgdir-ro directory, run the gpg command on both of them.

4. Run dnf update or yum update (depending on which you have installed). You should be prompted more than once to accept a new GPG key for the Grafana repository with the following fingerprint:

Copy

B53A E77B ADB6 30A6 8304 6005 963F A277 1045 8545

GPG revocation

We have issued a revocation certificate for the exposed GPG key. If you would prefer to load the full set of keys that includes the revocation certificate — which would prevent the leaked public key from being used unintentionally — a file containing all three (the old key, followed by a revocation certificate, followed by the new key) can be found at https://apt.grafana.com/gpg-full.key. As mentioned above, many package managers have idiosyncratic behavior when loading key files containing revocations, so please consult your package manager documentation for specific guidance.

Reporting security issues

If you think you have found a security vulnerability, please send a report to security@grafana.com. This address can be used for all of Grafana Labs’ open source and commercial products (including but not limited to Grafana, Grafana Cloud, Grafana Enterprise, and grafana.com). We can accept only vulnerability reports at this address. We would prefer that you encrypt your message to us by using our PGP key. The key fingerprint is

225E 6A9B BB15 A37E 95EB 6312 C66A 51CC B44C 27E0

The key is available from keyserver.ubuntu.com.

Security announcements

We maintain a security category on our blog, where we will always post a summary, remediation, and mitigation details for any patch containing security fixes.

You can also subscribe to our RSS feed.