Set up distributed k6 on Grafana Labs

Install k6 Operator

Mon, 21 Sep 2026 15:16:28 +0000

Install k6 Operator

This guide provides step-by-step instructions on how to install k6 Operator.

Before you begin

To install k6 Operator, you’ll need:

A Kubernetes cluster, along with access to it.
kubectl.

Deploy the operator

There are three different methods to deploy the k6 Operator. The first two methods, with bundle and Helm, install the latest official release of the k6 Operator by default. The third method installs from the branch of the k6-operator repository and is intended for development purposes or for users who have their own kustomize pipeline.

Deploy with bundle

The easiest way to install the operator is with bundle:

curl https://raw.githubusercontent.com/grafana/k6-operator/main/bundle.yaml | kubectl apply -f -

Bundle includes default manifests for k6 Operator, including a k6-operator-system namespace and k6 Operator deployment with the latest tagged Docker image. Customizations can be made on top of this manifest as needed, for example, with kustomize.

Deploy with Helm

Helm releases of k6 Operator are published together with other Grafana Helm charts. You can install it with the following commands:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install k6-operator grafana/k6-operator

You can also pass additional configuration options with a values.yaml file:

helm install k6-operator grafana/k6-operator -f values.yaml

Refer to the k6 Operator samples folder for an example file.

You can find a complete list of Helm options in the k6 Operator charts folder.

The Helm chart configures only the k6-operator application (the controller-manager and related deployment). It does not configure PrivateLoadZone (PLZ) or TestRun resources—those are configured via their CRD definitions. For details, see Configure TestRun CRD and PLZ configuration options.

Deploy with Makefile

This method installs the k6 Operator from the GitHub repository, using the latest files in a branch. By default, it uses the main branch, which is not guaranteed to be stable at all times. This method is helpful for:

Development purposes.
Users who have a kustomize pipeline and want to adjust manifests with it.

To use this method for production deployments, it’s recommended to do that from the tagged commits:

git clone https://github.com/grafana/k6-operator && cd k6-operator
git checkout v0.0.22

The tagged commits correspond to the official releases and are expected to be stable.

In order to install the operator with the Makefile, you need:

go
kustomize

Then, you can install the k6 Operator by running the command below from the root of the cloned repository:

make deploy

Install the CRDs

The k6 Operator includes custom resources called TestRun and PrivateLoadZone. They’re automatically installed when you use one of the three installation methods above. But you can also manually install the CRDs by running:

make install

Uninstall k6 Operator

Removal of the k6 Operator depends on the installation method. If you use the bundle installation method, you can remove all of the resources created by the k6 Operator with the following:

curl https://raw.githubusercontent.com/grafana/k6-operator/main/bundle.yaml | kubectl delete -f -

If you use Helm installation, then removal should be done with the helm command:

helm uninstall k6-operator

Finally, if you use Makefile installation, use the make command to uninstall:

make delete

Watch namespaces

By default, the k6 Operator watches the TestRun and PrivateLoadZone custom resources in all namespaces. You can also configure the k6 Operator to watch specific namespaces by setting either of the following environment variables on the controller’s deployment:

WATCH_NAMESPACE: expects the name of a single namespace.
WATCH_NAMESPACES: expects a comma-separated list of namespaces.

Don’t set both variables. If WATCH_NAMESPACES is set, WATCH_NAMESPACE will be ignored.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k6-operator-controller-manager
  namespace: k6-operator-system
spec:
  template:
    spec:
      containers:
        - name: manager
          image: ghcr.io/grafana/k6-operator:controller-v0.0.22
          env:
            - name: WATCH_NAMESPACE
              value: "some-ns"
            # Only use one option, WATCH_NAMESPACE or WATCH_NAMESPACES
            # - name: WATCH_NAMESPACES
            #   value: "some-ns,some-other-namespace"
# ...

Upgrade the k6 Operator

Mon, 21 Sep 2026 15:16:28 +0000

Upgrade the k6 Operator

This guide explains how to upgrade the k6 Operator, depending on your installation method. Before upgrading, make sure to read the release notes to know what changed and if any breaking changes happened.

k6 Operator installed with bundle

If you installed the k6 Operator with bundle, you can usually upgrade by re-running the kubectl apply command with the latest bundle file. However, this won’t remove resources that are no longer present in the new bundle. To avoid leaving outdated resources in your cluster, follow these steps:

Download the version of bundle.yaml you originally used and save it as old.bundle.yaml.
Delete the old installation:
sh
```
kubectl -f old.bundle.yaml delete
```

Download the latest bundle and install it:

curl https://raw.githubusercontent.com/grafana/k6-operator/main/bundle.yaml > bundle.yaml
kubectl apply -f bundle.yaml

Save the latest bundle as your new old.bundle.yaml for future upgrades:
sh
```
cp bundle.yaml old.bundle.yaml
```

Note
Always review the release notes and compare the new bundle.yaml with your previous version before upgrading.

Split the bundle for long-term maintenance

Bundle installation is quick, but not always flexible for long-term maintenance. You can split the bundle into separate manifest files and manage them with Kustomize or a GitOps workflow. For inspiration, refer to the run-tests.sh script in the k6 Operator repository.

k6 Operator installed with Helm

If you installed the k6 Operator with Helm, upgrade with these commands:

helm repo update
helm upgrade k6-operator grafana/k6-operator

k6 Operator installed with Makefile

If you installed the k6 Operator with a Makefile, upgrade with these commands:

git clone https://github.com/grafana/k6-operator && cd k6-operator
git checkout <LATEST_RELEASE_TAG>

make deploy

Replace <LATEST_RELEASE_TAG> with the latest release tag from the k6 Operator releases.

Caution
Make sure to switch to the tagged commit before running make deploy. If you run make deploy from the latest commit of the main branch, you may end up with a broken installation, because the main branch isn’t guaranteed to be stable.

This method is similar to the split bundle approach, but it’s closer to development mode and requires Golang and Kustomize to be installed.

Run browser tests in Kubernetes

Mon, 21 Sep 2026 15:16:28 +0000

Run browser tests in Kubernetes

Running browser tests in Kubernetes provides several benefits especially in locked down environments. Some reasons you might want to run browser tests in Kubernetes include staying consistent with other testing tools and infrastructure components in your Kubernetes environment, and the ability to run tests within your private network for security and privacy reasons.

This guide will help you get setup with k6 browser testing using the official grafana/k6:latest-with-browser image. We first explain how to make a simple setup for browser test, and then introduce requirements and how to integrate it with k6 Operator.

Before you begin

This list may expand in the future, as we learn about new use cases of browser testing. Pay attention to the following:

If you use securityContext or OpenShift SCCs/PSPs, verify the policy allows Chromium to start. Capture the exact error string in Troubleshooting so it is searchable.
Chromium is sensitive to restrictive security policies. In particular, avoid overly aggressive capability drops without testing browser startup.

Create and run a browser test

Start with the simplest Pod flow to verify that Chromium can launch in your cluster.

Create a browser test and name it script.js.
Create a ConfigMap with your script:

kubectl create configmap script --from-file script.js

Create a Pod with kubectl apply -f pod.yaml, where pod.yaml looks like

apiVersion: v1
kind: Pod
metadata:
  name: k6-browser-test
spec:
  containers:
    - name: k6
      image: grafana/k6:latest-with-browser
      command:
        - k6
      args:
        - run
        - /script/script.js
      resources:
        requests:
          cpu: "4"
          memory: "8G"
        limits:
          cpu: "4"
          memory: "8G"
      volumeMounts:
        - name: script
          mountPath: /script/script.js
          subPath: script.js
  volumes:
    - name: script
      configMap:
        name: script

Now wait and view the logs to see if it ran successfuly or not. We have some troubleshooting steps below that might help.

Architecture and performance considerations

Browser runner model

grafana/k6:latest-with-browser bundles the k6 binary and Chromium in the same pod. Each k6 runner starts its own browser process, which is simple but CPU and memory-intensive.

There is an open proposal to split browser and k6 runners, enabling an M:N model (M k6 runners to N browser instances) and potentially improving resource efficiency. This is relevant for capacity planning and for teams running larger browser workloads.

CPU rendering

grafana/k6:latest-with-browser includes both the k6 binary and a Chromium build that uses SwiftShader (CPU rendering). This keeps setup simple but can be heavy on CPU and may not work well for apps that require GPU-backed rendering. You can find more details on this GitHub issue.

Performance and sizing recommendations

Browser tests are CPU and memory-heavy. Start with higher limits to confirm stability, then scale down.

Set CPU and memory to match, or exceed, a modern desktop machine for initial runs.
Try to keep CPU/memory utilization under ~80% during steady-state.
Refer to Running large tests for general guidance on running larger tests.

Troubleshooting

If Chromium crashes immediately:
- It’s often due to security policies preventing Chromium from starting.
- In rare cases, we have seen it occur due to trying to run the incorrect arch image on an incompatible arch, for example, running an image built for AMD64 on an ARM-based machine.
Headless: true is not supported in this environment. A common error looks like: Missing X server or $DISPLAY\n[13:13:1211/091921.199737:ERROR:ui/aura/env.cc:257] The platform failed to initialize. Exiting.
capabilities.drop: all is generally a good security posture, but can break Chromium. It requires more capabilities than most CLI workloads.
Either add larger instances or reduce the CPU allocation if you see something like: 0/1 nodes are available: 1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.. Keep in mind that reducing CPU might worsen the browser test result.
error building browser on IterStart: making browser data directory "/tmp/k6browser-data-...": read-only file system
Fix: mount a writable emptyDir and set TMPDIR to that path (or mount emptyDir at /tmp). Example:
YAML
```
securityContext:
  readOnlyRootFilesystem: true
env:
  - name: TMPDIR
    value: /var/tmp
volumeMounts:
  - name: tmp
    mountPath: /var/tmp
volumes:
  - name: tmp
    emptyDir: {}
```
Error from server (BadRequest): error when creating "plz.yaml": PrivateLoadZone in version "v1alpha1" cannot be handled as a PrivateLoadZone: strict decoding error: unknown field "spec.*.securityContext" Fix: This is a known issue with the PLZ CRD. Refer to this GitHub issue for more details.

If you can’t find the answer you are looking for, open a new issue in the k6 Operator GitHub repository.

References

Usage

Mon, 21 Sep 2026 15:16:28 +0000

Usage

This section includes the following topics:

Troubleshoot

Mon, 21 Sep 2026 15:16:28 +0000

Troubleshoot

This topic includes instructions to help you troubleshoot common issues with the k6 Operator.

If you’re using Private Load Zones in Grafana Cloud k6, refer to Troubleshoot Private Load Zones.

How to troubleshoot

Test your script locally

Always run your script locally before trying to run it with the k6 Operator:

k6 run script.js

If you’re using environment variables or CLI options, pass them in as well:

MY_ENV_VAR=foo k6 run script.js --tag my_tag=bar

That ensures that the script has correct syntax and can be parsed with k6 in the first place. Additionally, running locally can help you check if the configured options are doing what you expect. If there are any errors or unexpected results in the output of k6 run, make sure to fix those prior to deploying the script elsewhere.

`TestRun` deployment

The Jobs and Pods

In case of one TestRun Custom Resource (CR) creation with parallelism: n, there are certain repeating patterns:

There will be n + 2 Jobs (with corresponding Pods) created: initializer, starter, and n runners.
If any of these Jobs didn’t result in a Pod being deployed, there must be an issue with that Job. Some commands that can help here:
Bash
```
kubectl get jobs -A
kubectl describe job mytest-initializer
```
If one of the Pods was deployed but finished with Error, you can check its logs with the following command:
Bash
```
kubectl logs mytest-initializer-xxxxx
```

`TestRun` with `cleanup` option

If a TestRun has the spec.cleanup option set, as PrivateLoadZone tests always do, for example, it may be harder to locate and analyze the Pod before it’s deleted.

In that case, we recommend using observability solutions, like Prometheus and Loki, to store metrics and logs for later analysis.

As an alternative, it’s also possible to watch for the resources manually with the following commands:

kubectl get jobs -n my-namespace -w
kubectl get pods -n my-namespace -w

# To get detailed information (this one is quite verbose so use with caution):
kubectl get pods -n my-namespace -w -o yaml

k6 Operator

Another source of info is the k6 Operator itself. It’s deployed as a Kubernetes Deployment, with replicas: 1 by default, and its logs together with observations about the Pods from the previous section usually contain enough information to help you diagnose any issues. With the standard deployment, the logs of the k6 Operator can be checked with:

kubectl -n k6-operator-system -c manager logs k6-operator-controller-manager-xxxxxxxx-xxxxx

Inspect `TestRun` resource

After you or PrivateLoadZone deployed a TestRun CR, you can inspect it the same way as any other resource:

kubectl describe testrun my-testrun

Firstly, check if the spec is as expected. Then, see the current status:

Status:
  Conditions:
    Last Transition Time:  2024-01-17T10:30:01Z
    Message:
    Reason:                CloudTestRunFalse
    Status:                False
    Type:                  CloudTestRun
    Last Transition Time:  2024-01-17T10:29:58Z
    Message:
    Reason:                TestRunPreparation
    Status:                Unknown
    Type:                  TestRunRunning
    Last Transition Time:  2024-01-17T10:29:58Z
    Message:
    Reason:                CloudTestRunAbortedFalse
    Status:                False
    Type:                  CloudTestRunAborted
    Last Transition Time:  2024-01-17T10:29:58Z
    Message:
    Reason:                CloudPLZTestRunFalse
    Status:                False
    Type:                  CloudPLZTestRun
  Stage:                   error

If Stage is equal to error, you can check the logs of k6 Operator.

Conditions can be used as a source of info as well, but it’s a more advanced troubleshooting option that should be used if the previous steps weren’t enough to diagnose the issue. Note that conditions that start with the Cloud prefix only matter in the setting of k6 Cloud test runs, for example, for cloud output and PLZ test runs.

Debugging k6 process

If the script is working locally as expected, and the previous steps show no errors as well, yet you don’t see an expected result of a test and suspect k6 process is at fault, you can use the k6 verbose option in the TestRun spec:

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: k6-sample
spec:
  parallelism: 2
  script:
    configMap:
      name: 'test'
      file: 'test.js'
  arguments: --verbose

Common errors

Issues with environment variables

Refer to Environment variables for details on how to pass environment variables to the k6 Operator.

Tags not working

Tags are a rather common source of errors when using the k6 Operator. For example, the following tags would lead to parsing errors:

  arguments: --tag product_id="Test A"
  # or
  arguments: --tag foo=\"bar\"

You can see those errors in the logs of either the initializer or the runner Pod, for example:

time="2024-01-11T11:11:27Z" level=error msg="invalid argument \"product_id=\\\"Test\" for \"--tag\" flag: parse error on line 1, column 12: bare \" in non-quoted-field"

This is a common problem with escaping the characters. You can find an issue in the k6 Operator repository that can be upvoted.

An error on reading output of the initializer Pod

The k6 runners fail to start, and in the k6 Operator logs, you see the unable to marshal error. This can happen for several reasons:

Your Kubernetes setup includes some tool that is implicitly adding symbols to the log output of Pods. You can verify this case by checking the logs of the initializer Pod: they should contain valid JSON, generated by k6. Currently, to fix this, the tool adding symbols must be switched off for the k6 Operator workloads.
Multi-file script includes many files which all must be fully accessible from the runner Pod. You can verify this case by checking the logs of the initializer Pod: there will be an error about some file not being found. To fix this, refer to Multi-file tests on how to configure multi-file tests in TestRun.
There are problems with environment variables or with importing an extension. Following the steps found in testing locally and in troubleshooting extensions can help debug this issue. One additional command that you can use to help diagnose issues with your script is the following:

k6 inspect --execution-requirements script.js

That command is a shortened version of what the initializer Pod is executing. If the command produces an error, there’s a problem with the script itself and it should be solved outside of the k6 Operator. The error itself may contain a hint to what’s wrong, such as a syntax error.

If the standalone k6 inspect --execution-requirements executes successfully, then it’s likely a problem with TestRun deployment specific to your Kubernetes setup.

An issue with `volumeClaim`

Storing k6 scripts on a persistent volume is one approach to multi-file tests. However, errors can occur due to misconfiguration of the volume. These errors are not within the purview of the k6 Operator; they are inherent to the Kubernetes setup itself, as the k6 Operator only mounts volumes to the Pods. However, here are some general recommendations to help debug such errors.

The volumeClaim option is expecting a persistent volume claim, so first, check the Kubernetes documentation and your infrastructure provider documentation to confirm if the volume is indeed set up correctly and can be mounted by Kubernetes pods.

Then, if the volume appears to be correct and is mounted to the k6 Pods without an issue, yet the TestRun fails with an error like the following:

The moduleSpecifier \"/test/utils.js\" couldn't be found on local disk.

This error implies that either the file was not written successfully to the Volume or there is a misconfiguration with a path. So it makes sense to create a separate debug Pod, for example, with the busybox image to confirm that the Volume contains the script and all its dependencies. Such a Pod should have a configuration similar to this one:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  volumes:
  - name: test-volume
    volumeSource:
      persistentVolumeClaim:
        claimName: test-pvc
        readOnly: false
  containers:
  - image: busybox
    name: busybox
    imagePullPolicy: IfNotPresent
    command:
      - sleep
      - "3600"
    volumeMounts:
    - mountPath: /test
      name: test-volume
  restartPolicy: Always

Then execute ls /test on this debug Pod to see which files are present.

k6 runners do not start

The k6 runners fail to start, and in the k6 Operator logs, you see the error Waiting for initializing pod to finish.

In this case, it’s most likely that an initializer Pod was not able to start for some reason.

How to fix

Refer to The Jobs and Pods section to see how to:

Check if the initializer Pod has started and finished.
See an issue in the initializer Job’s description that prevents a Pod from being scheduled.

Once the error preventing the initializer Pod from starting and completing is resolved, redeploy the TestRun or, in case of a PrivateLoadZone test, restart the k6 process.

Non-existent ServiceAccount

A ServiceAccount can be defined as serviceAccountName in a PrivateLoadZone, and as runner.serviceAccountName in a TestRun CRD. If the specified ServiceAccount doesn’t exist, k6 Operator will successfully create Jobs but corresponding Pods will fail to be deployed, and the k6 Operator will wait indefinitely for Pods to be Ready. This error can be best seen in the events of the Job:

kubectl describe job plz-test-xxxxxx-initializer
...
Events:
  Warning  FailedCreate  57s (x4 over 2m7s)  job-controller  Error creating: pods "plz-test-xxxxxx-initializer-" is forbidden: error looking up service account plz-ns/plz-sa: serviceaccount "plz-sa" not found

k6 Operator doesn’t try to analyze such scenarios on its own, but you can refer to the following issue for improvements.

How to fix

To fix this issue, the incorrect serviceAccountName must be corrected, and the TestRun or PrivateLoadZone resource must be re-deployed.

Non-existent `nodeSelector`

nodeSelector can be defined as nodeSelector in a PrivateLoadZone, and as runner.nodeSelector in the TestRun CRD.

This case is very similar to the ServiceAccount: the Pod creation will fail, but the error is slightly different:

kubectl describe pod plz-test-xxxxxx-initializer-xxxxx
...
Events:
  Warning  FailedScheduling  48s (x5 over 4m6s)  default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector.

How to fix

To fix this issue, the incorrect nodeSelector must be corrected and the TestRun or PrivateLoadZone resource must be re-deployed.

Insufficient resources

A related problem can happen when the cluster does not have sufficient resources to deploy the runners. There’s a higher probability of hitting this issue when setting small CPU and memory limits for runners or using options like nodeSelector, runner.affinity or runner.topologySpreadConstraints, and not having a set of nodes matching the spec. Alternatively, it can happen if there is a high number of runners required for the test (via parallelism in TestRun or during PLZ test run) and autoscaling of the cluster has limits on the maximum number of nodes, and can’t provide the required resources on time or at all.

This case is somewhat similar to the previous two: the k6 Operator will wait indefinitely and can be monitored with events in Jobs and Pods. If it’s possible to fix the issue with insufficient resources on-the-fly, for example, by adding more nodes, k6 Operator will attempt to continue executing a test run.

OOM of a runner Pod

If there’s at least one runner Pod that OOM-ed, the whole test will be stuck and will have to be deleted manually:

kubectl delete testrun my-test

A PrivateLoadZone test or a TestRun with cloud output will be aborted by Grafana Cloud k6 after its expected duration is up.

How to fix

In the case of OOM, review your k6 script to understand what kind of resource usage the script requires. It may be that the k6 script can be improved to be more performant. Then, set the spec.runner.resources in the TestRun CRD, or spec.resources in the PrivateLoadZone CRD accordingly.

Disruption of the k6 runners

A k6 test can be executed for a long time. But depending on the Kubernetes setup, it’s possible that the Pods running k6 are disrupted and moved elsewhere during execution. This will skew the test results. In the case of a PrivateLoadZone test or a TestRun with cloud output, the test run may additionally be aborted by Grafana Cloud k6 once its expected duration is up, regardless of the exact state of k6 processes.

How to fix

Ensure that k6 Pods can’t be disrupted by the Kubernetes setup, for example, with PodDisruptionBudget and a less aggressive configuration of the autoscaler.

Set up distributed k6 on Grafana Labs

Install k6 Operator

Install k6 Operator

Before you begin

Deploy the operator

Deploy with bundle

Deploy with Helm

Deploy with Makefile

Install the CRDs

Uninstall k6 Operator

Watch namespaces

Upgrade the k6 Operator

Upgrade the k6 Operator

k6 Operator installed with bundle

Split the bundle for long-term maintenance

k6 Operator installed with Helm

k6 Operator installed with Makefile

Run browser tests in Kubernetes

Run browser tests in Kubernetes

Before you begin

Create and run a browser test

Architecture and performance considerations

Browser runner model

CPU rendering

Performance and sizing recommendations

Troubleshooting

References

Usage

Usage

Troubleshoot

Troubleshoot

How to troubleshoot

Test your script locally

TestRun deployment

The Jobs and Pods

TestRun with cleanup option

k6 Operator

Inspect TestRun resource

Debugging k6 process

Common errors

Issues with environment variables

Tags not working

An error on reading output of the initializer Pod

An issue with volumeClaim

k6 runners do not start

How to fix

Non-existent ServiceAccount

How to fix

Non-existent nodeSelector

How to fix

Insufficient resources

OOM of a runner Pod

How to fix

Disruption of the k6 runners

How to fix

`TestRun` deployment

`TestRun` with `cleanup` option

Inspect `TestRun` resource

An issue with `volumeClaim`

Non-existent `nodeSelector`