Update as of January 13, 2023: Instructions updated to avoid errors on some OSes/platforms. GPG key file updated to contain only the current key (the old file contained both a revocation certificate for the old key, and the current key). A more detailed description of these changes can be found at the end of this post.
On January 4, CircleCI published a security update on their blog asking all customers on their platform to rotate secrets. While CircleCI is no longer a part of our CI/CD pipeline, we have made use of their services in the past.
As a best practice, we have rotated or invalidated any secrets that may have been present on CircleCI. Following a thorough review, we have not identified any signs of suspicious activity and we believe, with a high level of confidence, that none of our systems or code are or have been at risk of compromise.
In our review, we identified two GPG keys used to sign a subset of binaries and Helm charts. Best practice recommendations for these follow below.
GPG key rotation and trust relationship
Who is affected:
- Users who have installed Grafana via our package repositories (packages.grafana.com, apt.grafana.com, rpm.grafana.com)
- Users who use our binary .deb or .rpm releases and validate the GPG signatures with a local copy of the impacted GPG key
Who is not affected:
- Users who have installed Grafana via their OS package repository
- Users who have installed Grafana via a Docker image
- Grafana Cloud / Hosted users
If you are affected, you should follow the instructions below to remove the impacted GPG key from your keyrings and install our new key.
How to confirm if your GPG trust relationship is configured correctly
The following command should return ‘0’:
apt-key list | grep -i grafana | wc -l
If the above command returned anything except ‘0’, remove the old key:
# Remove the old key apt-key del 4E40DDF6D76E284A4A6780E48C8C34C524098CB6 # Verify that the GPG key was removed. This should return 0. apt-key list | grep -i grafana | wc -l
Regardless of whether the above command returned ‘0’ or not, you will need to configure the new key as well:
# Download the new key and place it in the right location sudo wget -q -O /etc/apt/keyrings/grafana.key https://apt.grafana.com/gpg.key # Replace the apt source # This command will replace the contents of the grafana.list file. echo "deb [signed-by=/etc/apt/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list # Update your repositories list sudo apt update
rpm (RHEL, CentOS, Fedora, etc.)
Our rpm-based installation instructions do not add the identified GPG key to the package manager, but rather suggest downloading the .rpm and gpg key and performing a manual verification. If you have saved a local copy of the Grafana gpg key (fingerprint: 4E40 DDF6 D76E 284A 4A67 80E4 8C8C 34C5 2409 8CB6), you will need to replace it with the rotated key, which can be downloaded at
dnf (RHEL, CentOS, Fedora, etc)
dnf cache repository gpg keys. In order to force
dnf to fetch and use the rotated gpg key, you will need to flush this cache. Instructions are as follows:
- Locate your yum cache directory. This will default to
/var/cache/dnffor systems running
dnf, but may be at
/etc/yum/repos.don systems running original
yum. If you don’t see either of these, check your
yum.conffile for a
persistdirparameter. This parameter overrides the default and specifies a cache directory manually.
- Locate the cache for the Grafana repository inside your yum cache directory. On
dnfsystems this should be something like
yumsystems this will likely be an architecture-specific subdirectory under
- Remove the identified GPG key from the cached keyring. Inside the Grafana repository cache directory, there will either be a
pubringfolder or a
gpgdir-rofolders. These are where the keyrings are actually cached. To remove the identified GPG key from the cached keyring, use the following
gpg --batch --homedir YOUR_PATH_GOES_HERE --delete-key 4E40DDF6D76E284A4A6780E48C8C34C524098CB6
If you have both a
gpgdir and a
gpgdir-ro directory, run this command on both of them.
yum update(depending on which you have installed). You should be prompted more than once to accept a new GPG key for the Grafana repository with fingerprint
0E22 EB88 E39E 1227 7A77 60AE 9E43 9B10 2CF3 C0C6
GPG revocation - Helm chart signing
A GPG signing key used for signing Helm charts was also identified in our CircleCI account. After some discussion, rather than replacing this with a new key we have instead decided to no longer sign Helm charts. Our data indicates Helm chart provenance is not commonly used and doesn’t provide sufficient value to offset the risks and complications of managing a GPG key used for such purpose.
If you were previously using Helm chart provenance, the below GPG key can no longer be trusted and must be removed from any GPG keyrings:
6737 B033 4836 C334 9DFF 2DD4 7054 A955 9D3C FB0B
Changes from original post
Some users were reporting issues with the instructions as originally posted not working on their systems– most of these issues centered around one of two things: the format of the GPG key fingerprint provided to
gpg commands (the original instructions had a fingerprint with spaces in it, which caused problems on some systems), and the format of the
gpg.key file served by our package repositores (it contained both a revocation certificate for the old key and the new key, which some package managers did not process correctly). We have updated the
gpg.key file served to only contain the new public key.
If you would prefer to load the full set of keys (the old key, followed by a revocation certificate for it, followed by the new key) a file containing all three can be found at
https://apt.grafana.com/gpg-full.key. As mentioned above, many package managers have idiosyncratic behavior when loading key files containing revocations, so please consult your package manager documentation for specific guidance.
Reporting security issues
If you think you have found a security vulnerability, please send a report to firstname.lastname@example.org. This address can be used for all of Grafana Labs’ open source and commercial products (including but not limited to Grafana, Grafana Cloud, Grafana Enterprise, and grafana.com). We can accept only vulnerability reports at this address. We would prefer that you encrypt your message to us by using our PGP key. The key fingerprint is
225E 6A9B BB15 A37E 95EB 6312 C66A 51CC B44C 27E0
The key is available from keyserver.ubuntu.com.
We maintain a security category on our blog, where we will always post a summary, remediation, and mitigation details for any patch containing security fixes.
You can also subscribe to our RSS feed.