Skip to main content

CI workflow

This article walks through the process of running end-to-end tests against a matrix of Grafana versions.

Why run end-to-end tests against a matrix of Grafana versions​

Due to Grafana’s dependency sharing mechanism, many plugin-related issues only emerge at runtime. For example, if a plugin invokes a function, component or class that is unavailable in the Grafana runtime environment, any page loading that part of the plugin will crash. These runtime-specific issues are beyond the scope of unit tests but can be effectively identified through end-to-end testing.

To maintain reliability and compatibility, plugin developers must regularly perform end-to-end tests across all supported Grafana versions. The e2e-versions GitHub Action simplifies this process by automatically resolving supported Grafana versions based on your plugin's grafanaDependency, while also including Grafana's main development branch. Integrating this Action into your CI workflows ensures your plugin remains stable and compatible with both older and newer versions of Grafana, giving you confidence in its functionality across versions."

The e2e-versions Action​

The e2e-versions GitHub Action generates a matrix of Grafana image names and versions for use in end-to-end testing a Grafana plugin within a GitHub workflow. The Action supports two modes:

  • plugin-grafana-dependency: This mode resolves the most recent grafana-dev image and returns all the latest patch releases of Grafana Enterprise since the version specified as grafanaDependency in the plugin.json file. To prevent the initiation of too many jobs, the output is capped at 6 versions. This is the default mode.
  • version-support-policy: In this mode, the action resolves versions based on Grafana's plugin compatibility support policy. It retrieves the latest patch release for each minor version within the current major Grafana version. Additionally, it includes the most recent release for the latest minor version of the previous major Grafana version.

For detailed information on configuring the inputs, visit the e2e-versions GitHub page.

Example workflows​

All Grafana plugins created with grafana/create-plugin version 4.7.0 or later automatically include end-to-end tests against a Grafana version matrix as part of their ci.yml workflow. If your plugin was created with an earlier version, you can use the following example workflows to set up and run end-to-end tests with a Grafana version matrix in a separate GitHub workflow:

note

The following examples are generic and based on frontend and backend plugins. Depending on the specifics of your plugin, you may need to modify or remove certain steps in the playwright-tests job before integrating them into your plugin’s workflow.

Backend plugin workflow

.github/workflows/e2e.yml
name: E2E tests
on:
pull_request:
schedule:
- cron: '0 11 * * *' #Run e2e tests once a day at 11 UTC

permissions:
contents: read
id-token: write

jobs:
resolve-versions:
name: Resolve Grafana images
runs-on: ubuntu-latest
timeout-minutes: 3
outputs:
matrix: ${{ steps.resolve-versions.outputs.matrix }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Resolve Grafana E2E versions
id: resolve-versions
uses: grafana/plugin-actions/e2e-version@main

playwright-tests:
needs: resolve-versions
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
GRAFANA_IMAGE: ${{fromJson(needs.resolve-versions.outputs.matrix)}}
name: e2e ${{ matrix.GRAFANA_IMAGE.name }}@${{ matrix.GRAFANA_IMAGE.VERSION }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Setup Node.js environment
uses: actions/setup-node@v4
with:
node-version-file: .nvmrc

- name: Install npm dependencies
run: npm ci

- name: Install Mage
uses: magefile/mage-action@v3
with:
install-only: true

- name: Build binaries
run: mage -v build:linux

- name: Build frontend
run: npm run build

- name: Install Playwright Browsers
run: npx playwright install --with-deps

- name: Start Grafana
run: |
docker compose pull
GRAFANA_VERSION=${{ matrix.GRAFANA_IMAGE.VERSION }} GRAFANA_IMAGE=${{ matrix.GRAFANA_IMAGE.NAME }} docker compose up -d

- name: Wait for grafana server
uses: grafana/plugin-actions/wait-for-grafana@main

- name: Run Playwright tests
id: run-tests
run: npx playwright test

Frontend plugin workflow

.github/workflows/e2e.yml
name: E2E tests
on:
pull_request:
schedule:
- cron: '0 11 * * *' #Run e2e tests once a day at 11 UTC

permissions:
contents: read
id-token: write

jobs:
resolve-versions:
name: Resolve Grafana images
runs-on: ubuntu-latest
timeout-minutes: 3
outputs:
matrix: ${{ steps.resolve-versions.outputs.matrix }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Resolve Grafana E2E versions
id: resolve-versions
uses: grafana/plugin-actions/e2e-version@main

playwright-tests:
needs: resolve-versions
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
GRAFANA_IMAGE: ${{fromJson(needs.resolve-versions.outputs.matrix)}}
name: e2e ${{ matrix.GRAFANA_IMAGE.name }}@${{ matrix.GRAFANA_IMAGE.VERSION }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Setup Node.js environment
uses: actions/setup-node@v4
with:
node-version-file: .nvmrc

- name: Install npm dependencies
run: npm ci

- name: Build frontend
run: npm run build

- name: Install Playwright Browsers
run: npx playwright install --with-deps

- name: Start Grafana
run: |
docker compose pull
GRAFANA_VERSION=${{ matrix.GRAFANA_IMAGE.VERSION }} GRAFANA_IMAGE=${{ matrix.GRAFANA_IMAGE.NAME }} docker compose up -d

- name: Wait for grafana server
uses: grafana/plugin-actions/wait-for-grafana@main

- name: Run Playwright tests
id: run-tests
run: npx playwright test

Publish Playwright reports to GitHub Pages​

The Playwright HTML report, along with the Trace Viewer, provides powerful tools for troubleshooting issues found during the execution of end-to-end test. This section explains how to deploy these reports to GitHub's static site hosting service GitHub Pages, making them immediately accessible for review after tests complete.

This guide is based on the example workflow provided earlier in this document.

Steps to enable report publishing​

  1. Immediately following the step that executes the tests, add a step that uses the upload-report-artifacts Action to upload the report and a test summary as an to GitHub artifacts.
- name: Run Playwright tests
id: run-tests
run: npx playwright test

- name: Upload e2e test summary
uses: grafana/plugin-actions/playwright-gh-pages/upload-report-artifacts@main
if: ${{ (always() && !cancelled()) }}
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
test-outcome: ${{ steps.run-tests.outcome }}
  1. After the playwright-tests job, add a new job to download the report artifacts, deploy them to GitHub Pages, and publish a PR comment summarizing the test results, including links to the reports.
publish-report:
if: ${{ (always() && !cancelled()) }}
needs: [playwright-tests]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Publish report
uses: grafana/plugin-actions/playwright-gh-pages/deploy-report-pages@main
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
  1. Modify the workflow permissions to allow it to push changes, query the GitHub API and update PR comments.
permissions:
contents: write
id-token: write
pull-requests: write
  1. If GitHub Pages is not yet enabled for your repository, configure a source branch for deployment. Follow the detailed instructions here to set it up.

For additional configuration options and examples, refer to the playwright-gh-pages documentation.

Important considerations​

  • Public visibility: By default, GitHub Pages sites are publicly accessible on the Internet. If your end-to-end tests include sensitive data or secrets, be aware of potential exposure risks.
  • Enterprise access control: If you have a GitHub Enterprise account, you can configure access controls to restrict visibility. For details, refer to the GitHub documentation.

Report summary​

The publish-report job adds a PR comment summarizing all the tests executed as part of the matrix. For tests that failed, the comment includes links to the GitHub Pages website, where the detailed reports can be browsed.