--- title: "Set up Private Load Zones | Grafana Cloud documentation" description: "How to deploy and configure Private Load Zone for k6 tests." --- # Set up Private Load Zones In Grafana Cloud k6, there are 20+ [load zones](/docs/grafana-cloud/testing/k6/author-run/use-load-zones/) that you can use to run your tests from different locations. But what happens if you: - Want to run your tests from a location that’s not included in the default load zones? - Want to test an internal service that isn’t exposed to the internet? Private Load Zones (PLZ) are load zones that you can host inside your network. They can run on top of any Kubernetes cluster you have. These PLZs are built on top of the existing [k6 Operator project](https://github.com/grafana/k6-operator) as an additional [custom resource definition](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CRD). You can start a cloud test in a PLZ by referencing it by name from your script, and the test executes on the nodes of your Kubernetes cluster. > Note > > The Private Load Zone feature is generally available. If you have any feedback, open an issue on the [grafana/k6-operator GitHub repository](https://github.com/grafana/k6-operator). ## Before you begin To set up and use a Private Load Zone, you’ll need: 1. A [Grafana Cloud account](/auth/sign-up/create-user). 2. A Kubernetes cluster (version 1.23 or higher): - That can access the APIs and services you wish to test. - That can access [“https://ingest.k6.io”](https://ingest.k6.io) and [“https://api.k6.io”](https://api.k6.io) > Note > > PLZs don’t work in air-gapped environments that can’t make outbound calls to Grafana Cloud k6 APIs. ### Multiple PLZs As of [v0.0.21](https://github.com/grafana/k6-operator/releases/tag/v0.0.21), the k6 Operator supports multiple Private Load Zones per cluster. The `name` of each PLZ must be unique because GCk6 recognizes PLZs by names. This allows k6 Operator to remain backwards compatible with the older versions. Each organization has a limit of 5 PLZs. [Contact support](mailto:support@k6.io) if you need to increase that limit. ## Set up a Private Load Zone First, you need to set up a Private Load Zone in your Kubernetes cluster. > Note > > You must have write access to your Kubernetes cluster to complete this step. This may require help from a system administrator (DevOps, SRE, etc.), depending on your organization. ### Install k6 Operator There are two ways to install k6 Operator in your cluster. #### With bundle Install k6 Operator by using the k6 Operator YAML bundle generated on every release: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f https://raw.githubusercontent.com/grafana/k6-operator/main/bundle.yaml ``` By default, k6 Operator is installed into the `k6-operator-system` namespace. You can check that it’s running there with: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl get pods -n k6-operator-system ``` And the output should look similar to: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash NAME READY STATUS RESTARTS AGE k6-operator-controller-manager-6f88ddf8df-7vvgx 2/2 Running 0 12s ``` #### With Helm Install k6 Operator by using the Helm chart: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install k6-operator grafana/k6-operator ``` With the command above, the operator will be installed into the `k6-operator-system` namespace. You can check that it’s running there with: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl get pods -n k6-operator-system ``` And the output should look similar to: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash NAME READY STATUS RESTARTS AGE k6-operator-controller-manager-6f88ddf8df-7vvgx 2/2 Running 0 12s ``` ### Create the PLZ CRD 1. Next, you must pick the namespace where you want to create your Private Load Zone and where k6 PLZ tests are to be executed by creating a [custom resource definition](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CRD). Namespace can be different from where the k6 Operator executes. Create a new namespace (replace `plz-ns` with the name you would like to use): Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl create namespace plz-ns ``` 2. Generate a new Grafana Stack API Token: 1. Log in to your Grafana Cloud account. 2. Go to **Testing & synthetics > Performance > Settings**. 3. Under **Access**, click **Stack token**. 4. Click **Create token**. 5. Give your token a name and select **Create new token**. 6. Copy and save the token. > Note > > This must be a Grafana Stack API Token and not a personal token. 3. Then, create a Kubernetes secret with the token by using the following command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl create secret generic grafana-k6-token -n plz-ns \ --from-literal=token=GRAFANA_STACK_API_TOKEN ``` > Note > > You can also use [external-secrets](https://github.com/external-secrets/external-secrets#external-secrets) or [other tools](https://kubernetes.io/docs/concepts/configuration/secret/#alternatives-to-secrets) to create a secret. 4. Now, create a definition of your Private Load Zone as a `plz.yaml` file. Replace *`PLZ_NAME`* with the name you’d like to use in your tests: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash apiVersion: k6.io/v1alpha1 kind: PrivateLoadZone metadata: name: PLZ_NAME namespace: plz-ns spec: token: grafana-k6-token resources: limits: cpu: 256m memory: 1024Mi ``` The `token` and `resources.limits` parameters are required. Resources must indicate how much CPU and memory a pod executing a k6 test can take up on a node. You can only configure `resources` while creating a Private Load Zone. To change your resource values, you have to delete and re-create the Private Load Zone. 5. Now that the Private Load Zone is defined, create it in your cluster: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f plz.yaml ``` 6. Check that your Private Load Zone was created: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl -n plz-ns get privateloadzones.k6.io ``` ## Run a test in the PLZ To run a performance test in a PLZ, you must first assign the project that contains the test to the PLZ. After that, any user in your organization with the ability to run a test in Grafana Cloud k6 can run the tests in that project using a PLZ. Newly created PLZs are automatically assigned to the organization’s default project. Running a test in a PLZ has certain limitations: - You can only specify one load zone, and the `percent` property value should be `100`. - You can’t specify multiple PLZs in a single script. - You can’t use a PLZ with [public load zones](/docs/grafana-cloud/testing/k6/author-run/use-load-zones/). - You can’t pass CLI arguments to k6. - You can’t pass more complex node configurations, such as affinity rules. ### Assign a project to a PLZ > Note > > Only users with `Admin` or `Performance Testing Admin` roles can assign projects to a PLZ. To assign a project to a PLZ: 1. Log in to your Grafana Cloud account. 2. Go to **Testing & synthetics > Performance > Settings**. 3. Under **Execution resources**, click **Load zones**. 4. Click the **Edit** icon in the **Projects** column. 5. Select any projects you want to allow to run tests in the PLZ. 6. Click **Save**. You can also assign projects to a PLZ by using the [Load zones REST API](/docs/grafana-cloud/testing/k6/reference//cloud-rest-api/load-zones/). ### From the CLI Make sure that you have [authenticated](/docs/grafana-cloud/testing/k6/author-run/tokens-and-cli-authentication/) with the k6 CLI. Then, go to a project or create a new project, and copy the ProjectID by selecting the **Copy to clipboard** icon. Create a new file and name it `plz-test.js`. You can copy the example script below, and make sure to change: - *`PLZ_NAME`* to the Private Load Zone name you defined in the `metadata` property of your CRD file. - *`PROJECT_ID`* to the projectID you copied in the previous step js ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```js import http from 'k6/http'; import { check } from 'k6'; export let options = { stages: [ { target: 200, duration: '3m30s' }, { target: 0, duration: '30s' }, ], cloud: { projectID: PROJECT_ID, distribution: { private: { loadZone: 'PLZ_NAME', percent: 100 }, }, }, }; export default function () { const result = http.get('https://quickpizza.grafana.com'); check(result, { 'http response status code is 200': result.status === 200, }); } ``` Now run your test with: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash k6 cloud run plz-test.js ``` ### From the UI Go to a project, or create a new one. In the top right corner of the project page, click on **Create new test**. You can use the Test Builder or Script Editor to create a new test. #### Script Editor By default, the Script Editor will create an example test script. Replace the load zone section from: js ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```js 'amazon:us:ashburn': { loadZone: 'amazon:us:ashburn', percent: 100 }, ``` To: js ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```js 'private': { loadZone: PLZ_NAME, percent: 100 }, ``` Make sure to replace *`PLZ_NAME`* with the Private Load Zone name you defined in the `metadata` property of your CRD file. Make other changes to the script as necessary, and select **Create and Run** to run the test. #### Test Builder By default, your test will be empty. Under **Options**, select **Load zones**, and then select the load zone drop-down and pick your PLZ. Add all the requests that your test needs, and select **Create and Run** to run your test. ## Additional information ### PLZ configuration options There are a few additional options you can use in your Private Load Zone definition: Expand table | Option | Description | Default value | |---------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------| | `resources.requests` | Values of requested [resources](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for each runner Pod. | Equal to `resources.limits` | | `serviceAccountName` | The name of the [service account](https://kubernetes.io/docs/concepts/security/service-accounts/#assign-to-pod) that’s assigned to each Pod. It must be present in the same namespace as the `PrivateLoadZone` itself. | \- | | `nodeSelector` | Map of key-value pairs to assign pods to targeted nodes. | \- | | `image` | The name of the k6 image for the k6 Operator load runners to use. You can use this option to run a custom k6 build that contains [extensions](/docs/k6/latest/extensions/). This option only impacts the [k6 CLI tests](#from-the-cli). [UI tests](#from-the-ui) don’t support a custom image. Refer to [k6 Operator with extensions](/docs/k6/latest/set-up/set-up-distributed-k6/usage/extensions/) for a sample `Dockerfile`. | `grafana/k6:latest` | | `imagePullSecrets` | Reference to the Secrets required to pull k6 images from private registries. Refer to the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-pod-that-uses-your-secret) for details on the field properties. | \- | | `config.secrets` | A list of `secretRef` or `configMapRef` objects pointing to a Secret or a ConfigMap with environment variables. These are passed as [`envFrom`](https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#define-an-environment-variable-for-a-container) to all pods running k6. | \- | | `podTemplate.metadata` | Labels and annotations attached to the k6 pods. Refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#attaching-metadata-to-objects) for details. | \- | | `podTemplate.spec.securityContext` | Pod-level security attributes and common container settings for all k6 pods. Refer to the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) for details. | \- | | `podTemplate.spec.tolerations` | Tolerations attached to all pods running k6. Refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) for details. | \- | | `podTemplate.spec.containers[k6].securityContext` | Security configuration applied to a k6 container. Refer to the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container) for details. | \- | Example of an extended definition: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml apiVersion: k6.io/v1alpha1 kind: PrivateLoadZone metadata: name: PLZ_NAME namespace: plz-ns spec: token: grafana-k6-token resources: limits: cpu: 256m memory: 1024Mi requests: cpu: 256m memory: 512Mi serviceAccountName: plz-sa nodeSelector: foo: bar image: / podTemplate: metadata: annotations: foo: 'bar' spec: securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 tolerations: - key: 'app' operator: 'Equal' value: 'blue' effect: 'NoSchedule' containers: - name: k6 securityContext: allowPrivilegeEscalation: false ``` When a test is created, `serviceAccountName` and `nodeSelector` are passed to the spec of all pods as [`spec.serviceAccountName`](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) and [`spec.nodeSelector`](https://kubernetes.io/docs/concepts/security/service-accounts/#assign-to-pod) respectively. Resources are passed only to [runner](https://github.com/grafana/k6-operator/blob/main/README.md#runner) pods. > Note > > If you pass a service account or reference nodes with specific labels, both must be present in your Kubernetes cluster. Otherwise, the creation of pods for the test run will fail, and the test will be stuck and will have to be deleted by an administrator manually. In the UI, such a test will time out after 10 min. ### Environment variables You can set [cloud environment variables](/docs/grafana-cloud/testing/k6/author-run/cloud-scripting-extras/cloud-environment-variables/) and reference them in your PLZ tests to make your tests more reusable and configurable. For example, they can be used to switch between test environments or to store credentials. These environment variables are passed as [`env`](https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#define-an-environment-variable-for-a-container) to all pods running k6. They also override any values passed in `config.secrets`. ### Private Load Zone lifecycle For each `PrivateLoadZone` object, The k6 Operator reads the PLZ definition, registers it against Grafana Cloud, and starts the [polling loop](https://github.com/grafana/k6-operator/blob/main/docs/plz.md) for PLZ test runs. Private Load Zones do not support mutability yet. If you need to modify the already registered PLZ, delete it, edit the definition and create it anew with `kubectl apply -f plz.yaml`. #### Connectivity loss If your k6 Operator deployment with the registered PLZ stops polling Grafana Cloud, the PLZ is marked as “Offline” after 10 minutes. The PLZ can then be deleted from the UI. An “Offline” PLZ is considered an “invalid load zone”, and a test cannot be started with it. There are two common cases where the PLZ polling can suddenly stop: - Your cluster has a connectivity issue that doesn’t impact the PLZ object directly. In this case, after the cluster comes back online, the PLZ should come back “Online” in the UI, allowing you to start tests with it again. - Your cluster has an issue that damaged the PLZ object. In this case, restoring the cluster is insufficient: the “Offline” PLZ must be removed from the UI first, then a new PLZ can be registered. ### Test lifecycle When you run a test in a PLZ, during its creation Grafana Cloud k6 estimates how many pods are required to run such a test. Then, k6 Operator starts the test by creating a `TestRun` CRD with all the parameters GCk6 provides. If your Kubernetes setup doesn’t have enough resources to run the test, GCk6 will wait for 10 minutes (to wait for auto-scaling or some other event that frees / creates resources). If enough resources are still not available after that time, the test will fail to be scheduled. After the test is finished, k6 Operator deletes the `TestRun` CRD. You can find more details about the PLZ logic and implementation in [Private Load Zone under the hood](https://github.com/grafana/k6-operator/blob/main/docs/plz.md).