Documentation Grafana k6 Testing guides Injecting faults with xk6-disruptor xk6-disruptor examples Inject HTTP faults into Pod

Inject HTTP faults into Pod

This example shows how PodDisruptor can be used for testing the effect of faults injected in the HTTP requests served by a pod.

You will find the complete source code at the end of this document. Next sections examine the code in detail.

The example uses httpbin, a simple request/response application that offers endpoints for testing different HTTP requests.

The test requires httpbin to be deployed in a cluster in the namespace httpbin and exposed with an external IP. the IP address is expected in the environment variable SVC_IP.

You will find the Kubernetes manifests and the instructions of how to deploy it to a cluster in the test setup section at the end of this document. You can learn more about how to get the external IP address in the expose your application section.

Initialization

The initialization code imports the external dependencies required by the test. The PodDisruptor class imported from the xk6-disruptor extension provides functions for injecting faults in pods. The k6/http module provides functions for executing HTTP requests.

JavaScript Copy

import { PodDisruptor } from 'k6/x/disruptor';
import http from 'k6/http';

Test Load

The test load is generated by the default function, which executes a request to the httpbin pod using the IP obtained from the environment variable SVC_IP. The test makes requests to the endpoint delay/0.1 which will return after 0.1 seconds (100ms).

JavaScript Copy

export default function (data) {
  http.get(`http://${data.SVC_IP}/delay/0.1`);
}

Fault injection

The disrupt function creates a PodDisruptor using a selector that matches pods in the namespace httpbin with the label app: httpbin.

The http faults are injected by calling the PodDisruptor.injectHTTPFaults method using a fault definition that introduces a delay of 50ms on each request and an error code 500 in 10% of the requests.

JavaScript Copy

export function disrupt(data) {
  if (__ENV.SKIP_FAULTS == '1') {
    return;
  }
  const selector = {
    namespace: namespace,
    select: {
      labels: {
        app: 'httpbin',
      },
    },
  };
  const podDisruptor = new PodDisruptor(selector);
  // delay traffic from one random replica of the deployment
  const fault = {
    averageDelay: '50ms',
    errorCode: 500,
    errorRate: 0.1,
  };
  podDisruptor.injectHTTPFaults(fault, '30s');
}

Notice the following code snippet in the injectFaults function above:

JavaScript Copy

export function disrupt(data) {
  if (__ENV.SKIP_FAULTS == '1') {
    return;
  }
  //...
}

This code makes the function return without injecting faults if the SKIP_FAULTS environment variable is passed to the execution of the test with a value of “1”. We will use this option to obtain a baseline execution without faults.

Scenarios

This test defines two scenarios to be executed. The load scenario applies the test load to the httpbin application for 30s invoking the default function. The disrupt scenario invokes the disrupt function to inject a fault in the HTTP requests of the target application.

JavaScript Copy

export const options = {
  scenarios: {
    load: {
      executor: 'constant-arrival-rate',
      rate: 100,
      preAllocatedVUs: 10,
      maxVUs: 100,
      exec: 'default',
      startTime: '0s',
      duration: '30s',
    },
    disrupt: {
      executor: 'shared-iterations',
      iterations: 1,
      vus: 1,
      exec: 'disrupt',
      startTime: '0s',
    },
  },
};

Executions

Baseline execution

We will first execute the test without introducing faults to have an baseline using the following command:

bashwindows-powershell

Copy

bash Copy

xk6-disruptor run --env SKIP_FAULTS=1 --env SVC_IP=$SVC_IP disrupt-pod.js

windows-powershell Copy

xk6-disruptor run --env SKIP_FAULTS=1 --env "SVC_IP=$Env:SVC_IP" disrupt-pod.js

Notice the argument --env SKIP_FAULT=1, which makes the disrupt function return without injecting any fault as explained in the fault injection section. Also notice the --env SVC_IP argument, which passes the external IP used to access the httpbin application.

You should get an output similar to the one shown below (click Expand button to see all output).

Fault injection

We repeat the execution injecting the faults. Notice we have removed the --env SKIP_FAULTS=1 argument.

bashwindows-powershell

Copy

bash Copy

xk6-disruptor run --env SVC_IP=$SVC_IP disrupt-pod.js

windows-powershell Copy

xk6-disruptor run --env "SVC_IP=$Env:SVC_IP" disrupt-pod.js

Comparison

Let’s take a closer look at the results for the requests on each scenario. We can observe that he base scenario has an average of 103ms and an error rate of 0% while the faults scenario has a median around 151.9ms and an error rate of nearly 10%, matching the definition of the faults defined in the disruptor.

Source Code

disrupt-pod.js

Test setup

The tests requires the deployment of the httpbin application. The application must also be accessible using an external IP available in the SVC_IP environment variable.

The manifests below define the resources required for deploying the application and exposing it as a LoadBalancer service.

You can deploy the application using the following commands:

bash Copy

# Create Namespace
kubectl apply -f namespace.yaml
namespace/httpbin created

# Deploy Pod
kubectl apply -f pod.yaml
pod/httpbin created

# Expose Pod as a Service
kubectl apply -f service.yaml
service/httpbin created

You can retrieve the resources using the following command:

bash Copy

kubectl -n httpbin get all
NAME          READY   STATUS    RESTARTS   AGE
pod/httpbin   1/1     Running   0          1m

NAME              TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
service/httpbin   LoadBalancer   10.96.169.78   172.18.255.200   80:31224/TCP   1m

You must set the environment variable SVC_IP with the external IP address and port used to access the httpbin service from the test script.

You can learn more about how to get the external IP address in the expose your application section.

Manifests

namespace.yaml

pod.yaml

service.yaml