Releasing Grafana Loki on Grafana Labs

Backport commits

Thu, 09 Apr 2026 02:28:18 +0000

Backport commits

Any PRs or commits not on the release branch that you want to include in the release must be backported to the release branch.

Before you begin

Determine the VERSION_PREFIX.
If the release branch already has all the code changes on it, skip this step.

Steps

Pick a PR that you want to backport to release-VERSION_PREFIX branch.
Add two labels to the PR. First, one of the product-approved, type/doc or type/bug appropriately. This is to make sure the PRs that are backported are done with right intention. Second backport release-VERSION_PREFIX label. Now CI should automatically create backport PR to the correct release branch. Example PR

Note
CI automation can fail sometimes if there are some merge conflicts in cherry picking the commits. In those cases, the original PR where you added the label should have additional comment explaining how to backport it manually.

Note
The CI job that helps with backporting PR is .github/workflows/backport.yml. Useful for debugging purposes.
Repeat the above steps for any PRs that need to be backported.

Backporting Release PRs

If backporting a release PR, make sure you remove any autorelease: pending or autorelease: tagged labels before merging the backport PR. By default our backport action brings over all labels, but these labels are reserved for the release workflow and will cause future pipelines to fail if left of backport PRs.

Create Release Branch

Thu, 09 Apr 2026 02:28:18 +0000

Create Release Branch

A single release branch is created for every major or minor release (not for patch releases). That release branch is then used for all the Stable Releases, and all Patch Releases for that major and minor versions of the Grafana Loki.

Before you begin

Determine the VERSION_PREFIX.
Announce about the upcoming release in #loki-releases internal slack channel.
Skip this announcement for a patch release. Create an issue to communicate beginning of the release process with the community. Example issue here.

Steps

Determine which commit should be used as a base for the release branch. Usually this is a kxx weekly release branch.
Create and push the release branch from the selected base commit:

The name of the release branch should be release-VERSION_PREFIX, such as release-2.9.x.

Note
Branches are only made for VERSION_PREFIX; do not create branches for the full VERSION such as release-2.9.1.

Note
Don’t create any other branches that are prefixed with release when creating PRs or those branches will collide with our automated release build publish rules.
Create a label to make backporting PRs to this branch easy.

The name of the label should be backport release-VERSION_PREFIX, such as backport release-2.9.x.

Note
Note there is space in the label name. The label name must follow this naming convention to trigger CI related jobs.

Document metrics and configurations changes

Thu, 09 Apr 2026 02:28:18 +0000

Document metrics and configurations changes

Any metrics and configurations that are removed or modified need to be documented in the upgrade guide. Configurations whose default values changed should also be documented in the upgrade guide.

Before you begin

All the steps are performed on release-VERSION_PREFIX branch.

Steps

Check which configs changed, including whose default values changed.
```
$ OLD_VERSION=X.Y.Z ./tools/diff-config.sh
```
Record configurations that are modified (either renamed or had its default value changed) in the upgrade guide.

Check if any metrics have changed.

$ OLD_VERSION=X.Y.Z ./tools/diff-metrics.sh

Record metrics whose names have been modified in the upgrade guide.

Merge Release PR

Thu, 09 Apr 2026 02:28:18 +0000

Merge Release PR

To release Loki, merge the release PR. This PR will have the title chore(<BRANCH>): release <VERSION>. Here’s what the 3.0 release PR looked like.

3.0 was the first major release using this new process, and enforcing conventional commits was relatively new. Going forward the release notes in the release PR will be much more thorough, as we’re now enforcing that every PR has a conventional commit message.

To test the artifacts before releasing, you can download the artifacts from the link in the footer of the PR description. Merging the PR will:

Fetch the built artifacts.
Create a draft GitHub release with the release notes in the PR description.
Upload fetched binareis to the draft release.
Publish fetched images to Docker Hub as multi-arch images.
Publish the draft release and create the GitHub tag.
(Optionally) Mark the release as the latest if it represents the newest version of Loki.

Troubleshooting / Retrying Release

If something goes wrong with the release automation, you may need to rerun the job. This may require updating the release code in the grafana/loki repo via a PR, or it may include updating the code in grafana/loki-release that the release pipeline fetches. In either case, if you need to re-release a merge release PR, you’ll need to remove the autorelease: tagged label from that PR and add the autorelease: pending label. The automation relies on these labels to determine which merged PRs have and have not yet been released. After fixing the labels, re-run the release workflow, and it should correctly get passed the should-release step. The release process is idempotent, so if a draft release has already been created via a previously failed release, the process will continue, re-upload the binaries, and re-publish the images.

Once the release is finished, you may need to manually flip the label back to autorelease: tagged.

Patch Go version

Thu, 09 Apr 2026 02:28:18 +0000

Patch Go version

Update vulnerable Go version to non-vulnerable Go version to build Grafana Loki binaries.

Before you begin.

Determine the VERSION_PREFIX.
Need to sign-in to Docker hub to be able to push Loki build image.

Steps

Find Go version to which you need to update. Example 1.20.5 to 1.20.6
Update Go version in the Grafana Loki build image (loki-build-image/Dockerfile) on the main branch.
Release a new Loki Build Image
Backport the Dockerfile change to release-VERSION_PREFIX branch.
Backport the Loki Build Image version change from main to release-VERSION_PREFIX branch.

Patch vulnerabilities

Thu, 09 Apr 2026 02:28:18 +0000

Patch vulnerabilities

This step patches vulnerabilities in Grafana Loki binaries and Docker images.

Before you begin

Determine the VERSION_PREFIX.

Vulnerabilities can be from two main sources.

Grafana Loki source code.
Grafana Loki dependencies.

Grafana Loki dependencies can be

Go dependencies in go.mod
The Go compiler itself
Grafana Loki Docker dependencies, for example, the base images

Before start patching vulnerabilities, know what are you patching. It can be one or more from sources mentioned above. Use #security-go, #security slack channels to clarify.

Steps

Patch Grafana Loki source code.

Means, there are vulnerabilities in Grafana Loki source code itself.
1. Patch it on main branch
2. Backport to release-$VERSION_PREFIX branch.
Patch Go dependencies.
1. Pick all the Go dependencies that need to be patched.
2. Check if dependabot already patched the dependency or have a PR opened to patch . If not, manually upgrade the package on the main branch as follows.
  shell
```
go get -u -v <PACKAGE_PATH>@<PATCHED_VERSION>
go mod tidy
go mod vendor
```
3. Backport it to release-$VERSION_PREFIX branch.
4. Repeat for each Go dependency
Patch Go compiler.
Patch Grafana Loki Docker dependencies, for example: Alphine Linux base images).
1. Update Docker image version. Example PR.
2. Backport to release-$VERSION_PREFIX branch

Prepare Major Release

Thu, 09 Apr 2026 02:28:18 +0000

Prepare Major Release

A major release follows the same process as minor and patch releases, but requires a custom workflow to be created to run on the branch we want to release from. The reason for this is that we don’t do major releases very often, so it is not economical to keep those workflows running all the time.

To create a major release workflow, follow the steps below.

Edit ./github/release-workflows.jsonnet
Add a new workflow for the major release. For example, the 3.0 release looked like the following:

  'three-zero-release.yml': std.manifestYamlDoc(
    lokiRelease.releasePRWorkflow(
      branches=['release-3.0.0'],
      buildImage=buildImage,
      checkTemplate=checkTemplate,
      golangCiLintVersion=golangCiLintVersion,
      imageBuildTimeoutMin=imageBuildTimeoutMin,
      imageJobs=imageJobs,
      imagePrefix=imagePrefix,
      releaseLibRef=releaseLibRef,
      releaseRepo='grafana/loki',
      skipArm=false,
      skipValidation=false,
      useGitHubAppToken=true,
      releaseAs='3.0.0',
    ) + {
      name: 'Prepare Loki 3.0 release',
    }, false, false
  ),

Make sure the branches field is set to the release branch you want to release from.
Make sure the releaseAs field is set to the version you want to release.
Run make release-workflows to generate the new workflow. Merge this change to both the main and release branch. You may also want to disable the patch release workflow as part of this PR, see below.

Race Condition with Patch Release

Once this new workflow is created there is a bit of a race condition with the patch workflow. This can be resolved in 1 of 2 ways:

Disable the patch release workflow until after the major release is complete.
Watch all actions for the release branch (ie. https://github.com/grafana/loki/actions?query=branch%3Arelease-3.0.x) and cancel and patch release runs.

Prepare Release

Thu, 09 Apr 2026 02:28:18 +0000

Prepare Release

Releasing Grafana Loki consists of merging a long-running release PR. Two workflows keep these PRs up to date, one for patch releases (which runs on commits to branches matching release-[0-9]+.[0-9]+.x) and one for minor releases (which runs on commits to branches matching k[0-9]+). These pipelines use release please to do the following:

Run tests and linting
Build binaries and images for the proposed release version
Generate release notes based on conventional commits since the last release
Create or update the long-running release PR for that release, indicating which commit will be released if the PR is merged with a link to the built artifacts that will be published.

Major releases

Major releases follow the same process as minor and patch releases, but require a custom workflow to be created to run on the branch we want to release from. The reason for this is that we don’t do major releases very often, so it is not economical to keep those workflows running all the time.To create a major release workflow, follow the steps in the major release workflow document.

Prepare Upgrade guide

Thu, 09 Apr 2026 02:28:18 +0000

Prepare Upgrade guide

The upgrade guide records changes that require user attention or interaction to upgrade to specific Loki version from previous versions.

Before you begin

The upgrade guide in Grafana Loki works as follows.

We have a setup/upgrade/_index.md file that records upgrade information for all Loki releases.

Steps

Make sure the upgrade guide is up to date on the release-VERSION_PREFIX branch under the Main/Unreleased section.
On the release-VERSION_PREFIX branch promote Main/Unreleased to VERSION. Example PR.
On the main branch remove entries from Main/Unreleased that are already part of VERSION (YYY-MM-DD).

Update version numbers

Thu, 09 Apr 2026 02:28:18 +0000

Update version numbers

Upgrade the Loki version to the new release version in documents, examples, jsonnets, etc.

Before you begin

Determine the VERSION_PREFIX.
Skip this step if you are doing a patch release on old release branch.

Example: Latest release branch is release-2.9.x but you are releasing patch version in release-2.8.x (or older)

Steps

Upgrade the versions in the release-VERSION_PREFIX branch.

Example commands:
```
LOKI_NEW_VERSION=$VERSION ./tools/release_update_tags.sh
```
Note
Do not upgrade the version numbers in the operator/ directory as @periklis and team have a different process to upgrade the Operator version.

Version

Thu, 09 Apr 2026 02:28:18 +0000

Version

Grafana Loki uses Semantic Versioning. The next version can be determined by looking at the current version and incrementing it. When the release is a new major version (for example, after 2.9.1 we decide to go 3.0.0 instead of 2.9.2) this should not be done.

You need to set two environmental values VERSION and VERSION_PREFIX to do the release process.

Version

To determine the VERSION for a Stable Release or Patch Release, use the Semantic Version A.B.C.

Examples
- For example, 2.9.0 is the Stable Release VERSION for the v2.9.0 release.
- For example, 2.9.1 is the first Patch Release VERSION for the v2.9.0 release.

Version Prefix

To determine the VERSION PREFIX, use only the major and minor version A.B.x.

Examples
- 2.9.x