Documentation Grafana k6 Testing guides Injecting faults with xk6-disruptor xk6-disruptor examples Inject gRPC faults into Service

Inject gRPC faults into Service

This example shows a way to use the ServiceDisruptor to test the effect of faults injected in the gRPC requests served by a service.

The complete source code is at the end of this document. The next sections examine the code in detail.

The example uses grpcpbin, a simple request/response application that offers endpoints for testing different gRPC requests.

The test requires grpcpbin to be deployed in a cluster in the namespace grpcbin and exposed with an external IP. The IP address is expected in the environment variable GRPC_HOST.

For the Kubernetes manifests and the instructions on how to deploy it to a cluster, refer to the test setup section at the end of this document. To learn more about how to get the external IP address, refer to Expose your application.

Initialization

The initialization code imports the external dependencies required by the test. The ServiceDisruptor class imported from the xk6-disruptor extension provides functions for injecting faults in services. The k6/net/grpc module provides functions for executing gRPC requests. The check function verifies the results from the requests.

JavaScript Copy

import { ServiceDisruptor } from 'k6/x/disruptor';
import grpc from 'k6/net/grpc';
import { check } from 'k6';

We also create a grpc client and load the protobufs definitions for the HelloService service.

JavaScript Copy

const client = new grpc.Client();
client.load(['pb'], 'hello.proto');

Test Load

The test load is generated by the default function, which connects to the grpcbin service using the IP and port obtained from the environment variable GRPC_HOST and invokes the SayHello method of the hello.HelloService service. Finally, The status code of the response is checked. When faults are injected, this check should fail.

JavaScript Copy

export default function () {
  client.connect(__ENV.GRPC_HOST, {
    plaintext: true,
    timeout: '1s',
  });

  const data = { greeting: 'Bert' };
  const response = client.invoke('hello.HelloService/SayHello', data, {
    timeout: '1s',
  });
  client.close();

  check(response, {
    'status is OK': (r) => r && r.status === grpc.StatusOK,
  });
}

Fault injection

The disrupt function creates a ServiceDisruptor for the grpcbin service in the namespace grpcbin.

The gRPC faults are injected by calling the ServiceDisruptor.injectGrpcFaults method using a fault definition that introduces a delay of 300ms on each request and an error with status code 13 (“Internal error”) in 10% of the requests.

JavaScript Copy

export function disrupt() {
  if (__ENV.SKIP_FAULTS == '1') {
    return;
  }

  const fault = {
    averageDelay: '300ms',
    statusCode: grpc.StatusInternal,
    errorRate: 0.1,
    port: 9000,
  };
  const disruptor = new ServiceDisruptor('grpcbin', 'grpcbin');
  disruptor.injectGrpcFaults(fault, '30s');
}

Notice the following code snippet in the injectFaults function above:

JavaScript Copy

export function disrupt() {
  if (__ENV.SKIP_FAULTS == '1') {
    return;
  }
  //...
}

This code makes the function return without injecting faults if the SKIP_FAULTS environment variable is passed to the execution of the test with a value of “1”. We will use this option to obtain a baseline execution without faults.

Scenarios

This test defines two scenarios to be executed. The load scenario applies the test load to the grpcpbin application for 30s invoking the default function. The disrupt scenario invokes the disrupt function to inject a fault in the gRPC requests to the target application.

JavaScript Copy

export const options = {
  scenarios: {
    load: {
      executor: 'constant-arrival-rate',
      rate: 100,
      preAllocatedVUs: 10,
      maxVUs: 100,
      exec: 'default',
      startTime: '0s',
      duration: '30s',
    },
    disrupt: {
      executor: 'shared-iterations',
      iterations: 1,
      vus: 1,
      exec: 'disrupt',
      startTime: '0s',
    },
  },
};

Executions

Baseline execution

We will first execute the test without introducing faults to have an baseline using the following command:

Bashwindows-powershell

Copy

Bash Copy

xk6-disruptor --env SKIP_FAULTS=1 --env GRPC_HOST=$GRPC_HOST run grpc-faults.js

windows-powershell Copy

xk6-disruptor --env SKIP_FAULTS=1 --env "GRPC_HOST=$Env:GRPC_HOST" run grpc-faults.js

Notice the argument --env SKIP_FAULT=1, which makes the disrupt function return without injecting any fault as explained in the fault injection section. Also notice the --env GRPC_HOST argument, which passes the external IP used to access the grpcbin application.

You should get an output similar to the following (use the Expand button to see all output).

Fault injection

We repeat the execution injecting the faults. Notice we have removed the --env SKIP_FAULTS=1 argument.

Bashwindows-powershell

Copy

Bash Copy

xk6-disruptor --env GRPC_HOST=$GRPC_HOST run grpc-faults.js

windows-powershell Copy

xk6-disruptor --env "GRPC_HOST=$Env:GRPC_HOST" run grpc-faults.js

Comparison

Let’s take a closer look at the results for the requests on each scenario. We can observe that in the base scenario requests duration has an percentile 95 of 2.09ms and 100% of requests pass the check. The faults scenario has a percentile 96 of 303.57ms and only 89% of requests pass the check (or put in another way, 11% of requests failed), closely matching the fault definition.

Source Code

grpc-faults.js

pb/hello.proto

Test setup

The tests requires the deployment of the grpcbin application. The application must also be accessible using an external IP available in the GRPC_HOST environment variable.

The following manifests define the resources required for deploying the application and exposing it as a LoadBalancer service.

You can deploy the application using the following commands:

Bash Copy

# Create Namespace
kubectl apply -f namespace.yaml
namespace/grpcbin created

# Deploy Pod
kubectl apply -f pod.yaml
pod/grpcbin created

# Expose Pod as a Service
kubectl apply -f service.yaml
service/grpcbin created

You must set the environment variable GRPC_HOST with the external IP address and port used to access the grpcbin service from the test script.

Learn more about how to get the external IP address in the Expose your application.

Manifests

namespace.yaml

pod.yaml

service.yaml