Performance Testing in Kubernetes using Kangal
Tags: kubernetes, performance, testing, webperf
Published: 2021-04-22 15:00:54

<!-- wp:paragraph -->
<p>From past ten days, I have spent my time on the latest open source solution to run performance tests on Kubernetes cluster using Kangal. In this blog post, we are going to deep-dive into how you can spin up Kubernetes cluster and run Locusts and JMeter tests using Kangal. Let us get started.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"80ceeb67"} -->
<h2 class="gb-headline gb-headline-80ceeb67 gb-headline-text">What is Kangal?</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>The full form of Kangal is <strong><span style="text-decoration: underline;">K</span></strong>ubernetes&nbsp;<strong><span style="text-decoration: underline;">an</span></strong>d&nbsp;<strong><span style="text-decoration: underline;">G</span></strong>o&nbsp;<strong><span style="text-decoration: underline;">A</span></strong>utomatic&nbsp;<strong><span style="text-decoration: underline;">L</span></strong>oader. But the actual meaning of <a href="https://en.wikipedia.org/wiki/Kangal_Shepherd_Dog" target="_blank" rel="noreferrer noopener">Kangal is a breed of Shepherd dog</a>. In Tamil language, Kangal means eyes.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Kangal is an open source project from Hello Fresh to run performance tests on Kubernetes cluster using multiple load generators. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>It supports Apache JMeter and Locust at this time of writing.</p>
<!-- /wp:paragraph -->

<!-- wp:buttons {"contentJustification":"center"} -->
<div class="wp-block-buttons is-content-justification-center"><!-- wp:button {"gradient":"luminous-dusk"} -->
<div class="wp-block-button"><a class="wp-block-button__link has-luminous-dusk-gradient-background has-background" href="https://github.com/hellofresh/kangal" target="_blank" rel="noreferrer noopener">Kangal GitHub Repo</a></div>
<!-- /wp:button --></div>
<!-- /wp:buttons -->

<!-- wp:buttons {"contentJustification":"center"} -->
<div class="wp-block-buttons is-content-justification-center"><!-- wp:button {"gradient":"vivid-cyan-blue-to-vivid-purple"} -->
<div class="wp-block-button"><a class="wp-block-button__link has-vivid-cyan-blue-to-vivid-purple-gradient-background has-background" href="https://github.com/QAInsights/Kangal-Demo" target="_blank" rel="noreferrer noopener">Kangal Demo GitHub Repo</a></div>
<!-- /wp:button --></div>
<!-- /wp:buttons -->

<!-- wp:generateblocks/headline {"uniqueId":"3a86f636"} -->
<h2 class="gb-headline gb-headline-3a86f636 gb-headline-text">Prerequisites</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>It is time to get our hands dirty. We need the following as prerequisites before you begin spinning up the pods.</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>JMeter test plan or Locust Script and its dependencies</li><li>Kubernetes cluster</li><li>Helm installation</li><li>S3 bucket to store the result (optional)</li></ol>
<!-- /wp:list -->

<!-- wp:generateblocks/headline {"uniqueId":"5bcc5ad9"} -->
<h2 class="gb-headline gb-headline-5bcc5ad9 gb-headline-text">Kangal Setup</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>First, let us add the repository to Helm using the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>helm repo add kangal https://hellofresh.github.io/kangal</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>This will add the repo to your Helm. Next, we are going to install Kangal using the below command. Below command configures multiple environment variables such as S3 bucket, S3 URL, AWS Secrets and Kangal Proxy.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>helm install \
  --set environment=dev \
  --set secrets.AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXXXXXX" \
  --set secrets.AWS_SECRET_ACCESS_KEY="XXXXXXXXXXXXXXXX" \
  --set controller.env.KANGAL_PROXY_URL="http://kangal-proxy-k8s.default.svc.cluster.local" \
  --set configMap.AWS_BUCKET_NAME="kangal-report" \
  --set configMap.AWS_ENDPOINT_URL="s3.us-east-2.amazonaws.com" \
  --set configMap.AWS_DEFAULT_REGION="us-east-2" \
  kangal kangal/kangal</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>You will have to replace the masked values with your secrets and hit enter. This will install relevant Kubernetes resources and below will be the output.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>NAME: kangal
LAST DEPLOYED: Wed Apr 21 22:22:17 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To view the helm releases, send <code>helm list</code> command which will list the releases.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To check the Kangal pods, issue the below command which will list the resources from the default namespace.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>kubectl get po</code></pre>
<!-- /wp:code -->

<!-- wp:image {"id":8300,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/04/image.png"><img src="https://qainsights.com/wp-content/uploads/2021/04/image-1024x205.png" alt="Performance Testing in Kubernetes using Kangal - Get Pods" class="wp-image-8300"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To view the service list, issue <code>kubectl get svc</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8301,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/04/image-1.png"><img src="https://qainsights.com/wp-content/uploads/2021/04/image-1-1024x174.png" alt="Get Service" class="wp-image-8301"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Kangal uses Custom Resources which extends the Kubernetes API. To view the custom resource  definitions, issue <code>kubectl get crd</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8302,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/04/image-2.png"><img src="https://qainsights.com/wp-content/uploads/2021/04/image-2.png" alt="Get CRD" class="wp-image-8302"/></a></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Before starting the load test, you have to make sure that the ingress controller are working fine.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In my case, I do not have proper Ingress Controller set up. I am going to port forward the deployment using the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>kubectl port-forward deployment/kangal-proxy 8081:8080</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Now we are good to start our first load test in Kangal.</p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"00897e62"} -->
<h2 class="gb-headline gb-headline-00897e62 gb-headline-text">Locust Load Test</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>Open a new terminal session and issue the curl command to make sure the localhost is working fine.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>curl localhost:8081</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Output</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&lt;a href="https://kangal-openapi-ui.local"&gt;Found&lt;/a&gt;</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>You can use the below Locust file for your testing purpose. Save below Python code as <code>hello.py</code> in the current directory. Replace https://example.com with your application under test URL.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>import glob
import os
import requests
import tarfile
from locust import HttpUser, task, between
from locust import events
from locust.runners import MasterRunner


class QuickstartUser(HttpUser):
    wait_time = between(1, 2.5)
    host = "https://example.com"

    @task
    def hello_world(self):
        self.client.get("/")

    @events.quitting.add_listener
    def hook_quit(environment):
        presigned_url = os.environ.get('REPORT_PRESIGNED_URL')
        if None == presigned_url:
            return
        if False == isinstance(environment.runner, MasterRunner):
            return
        report = '/home/locust/report.tar.gz'
        tar = tarfile.open(report, 'w:gz')

        for item in glob.glob('/tmp/*.csv'):
            print('Adding %s...' % item)
            tar.add(item, arcname=os.path.basename(item))
        tar.close()
        request_headers = {'content-type': 'application/gzip'}
        r = requests.put(presigned_url, data=open(report, 'rb'), headers=request_headers)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To begin the load test, we have to send the below POST request to the Kangal Proxy. Below test spins up two pods for Locust (one for master and one for worker) and run the test for 1 minute hitting example.com. After the test is done, results will be stored in S3.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>curl -X POST http://localhost:8081/load-test \
  -H 'Content-Type: multipart/form-data' \
  -F distributedPods=1 \
  -F testFile=@hello.py \
  -F type=Locust \
  -F duration=1m \
  -F targetURL=http://example.com</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Output of the above command will provide the unique load test name. This test name is <code>loadtest-guiding-mouse</code></p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>{
	"type": "Locust",
	"distributedPods": 1,
	"loadtestName": "loadtest-guiding-mouse",
	"phase": "creating",
	"tags": {},
	"hasEnvVars": false,
	"hasTestData": false
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To monitor the status, you can issue the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>curl localhost:8081/load-test/loadtest-guiding-mouse/logs</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>For every test, Kangal creates a namespace. To view all the namespaces, issue the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>kubectl get namespaces</code></pre>
<!-- /wp:code -->

<!-- wp:image {"align":"center","id":8303,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/04/image-3.png"><img src="https://qainsights.com/wp-content/uploads/2021/04/image-3.png" alt="" class="wp-image-8303"/></a></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To change the context in Kubernetes, you can use the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>kubectl config set-context --current --namespace=loadtest-guiding-mouse</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Then issue <code>kubectl get po</code> to view the Locust pods. As you see, there are two pods, one is for master and another one is worker.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8305,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/04/image-5.png"><img src="https://qainsights.com/wp-content/uploads/2021/04/image-5.png" alt="" class="wp-image-8305"/></a></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>You can view the logs of these pods by issuing the pod name.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>kubectl logs loadtest-guiding-mouse-master-v47gs</code></pre>
<!-- /wp:code -->

<!-- wp:code -->
<pre class="wp-block-code"><code>kubectl logs loadtest-guiding-mouse-worker-qkgs7</code></pre>
<!-- /wp:code -->

<!-- wp:generateblocks/headline {"uniqueId":"709186f1"} -->
<h2 class="gb-headline gb-headline-709186f1 gb-headline-text">S3 Locust Report</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>If the S3 properly configured, all the reports will be stored in the S3 bucket. Below is the screenshot from my S3 bucket.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":8307,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2021/04/image-7.png"><img src="https://qainsights.com/wp-content/uploads/2021/04/image-7-1024x326.png" alt="" class="wp-image-8307"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Below is the output stored in S3.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8308,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/04/image-8.png"><img src="https://qainsights.com/wp-content/uploads/2021/04/image-8.png" alt="" class="wp-image-8308"/></a></figure></div>
<!-- /wp:image -->

<!-- wp:generateblocks/headline {"uniqueId":"b04b1c1f"} -->
<h2 class="gb-headline gb-headline-b04b1c1f gb-headline-text">JMeter Test</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>To run JMeter test, you can issue the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>curl -X POST http://localhost:8081/load-test \
  -F "distributedPods=1" \
  -F "testFile=@T00_Example.jmx" \
  -F "type=JMeter"</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>HTML report will be stored in S3.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8309,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/04/image-9.png"><img src="https://qainsights.com/wp-content/uploads/2021/04/image-9.png" alt="" class="wp-image-8309"/></a></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To view all the tests, you can issue the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>curl localhost:8081/load-test</code></pre>
<!-- /wp:code -->

<!-- wp:code -->
<pre class="wp-block-code"><code>{
  "limit": 50,
  "continue": "",
  "remain": null,
  "items": &#91;
    {
      "type": "JMeter",
      "distributedPods": 1,
      "loadtestName": "loadtest-foppish-albatross",
      "phase": "finished",
      "tags": {},
      "hasEnvVars": false,
      "hasTestData": false
    },
    {
      "type": "Locust",
      "distributedPods": 1,
      "loadtestName": "loadtest-guiding-mouse",
      "phase": "finished",
      "tags": {},
      "hasEnvVars": false,
      "hasTestData": false
    }
  ]
}</code></pre>
<!-- /wp:code -->

<!-- wp:generateblocks/headline {"uniqueId":"c212fdf2"} -->
<h2 class="gb-headline gb-headline-c212fdf2 gb-headline-text">Housekeeping</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>After the test is done, you can delete the namespaces by issuing <code>kubectl delete ns &lt;namespace_name&gt;</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To delete Kangal, you can issue the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>helm delete kangal</code></pre>
<!-- /wp:code -->

<!-- wp:code -->
<pre class="wp-block-code"><code>kubectl delete crd loadtests.kangal.hellofresh.com</code></pre>
<!-- /wp:code -->

<!-- wp:generateblocks/headline {"uniqueId":"9c76e4ef","element":"h3"} -->
<h3 class="gb-headline gb-headline-9c76e4ef gb-headline-text">Important Links</h3>
<!-- /wp:generateblocks/headline -->

<!-- wp:list -->
<ul><li>For the detailed configuration, please check this <a href="https://github.com/hellofresh/kangal/tree/master/charts/kangal" target="_blank" rel="noreferrer noopener">page</a>.</li><li><a href="https://artifacthub.io/packages/helm/kangal/kangal" target="_blank" rel="noreferrer noopener">Artifact Hub</a></li><li><a href="https://github.com/hellofresh/kangal/blob/master/docs/README.md" target="_blank" rel="noreferrer noopener">Kangal Docs</a></li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2 id="h-kangal-features">Kangal Features</h2>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li>Supports two major open source load testing tools (JMeter and Locust)</li><li>Supports distributed load testing by default</li><li>Persistent storage using AWS S3</li><li>Grafana integration for live monitoring</li><li>Easy housekeeping</li></ul>
<!-- /wp:list -->

<!-- wp:generateblocks/headline {"uniqueId":"8d0de5cf"} -->
<h2 class="gb-headline gb-headline-8d0de5cf gb-headline-text">Kangal Architecture</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:image {"align":"center"} -->
<div class="wp-block-image"><figure class="aligncenter"><img src="https://github.com/hellofresh/kangal/blob/master/diagram.png?raw=true" alt="Kangal Architecture"/><figcaption>Kangal Architecture</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:generateblocks/headline {"uniqueId":"2bdb8a5c"} -->
<h2 class="gb-headline gb-headline-2bdb8a5c gb-headline-text">Conclusion</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>If you are looking to run your load tests in Kubernetes Cluster, here is the free and reliable infrastructure setup. All you need is to know about Kubernetes and Helm to steer this. I have tested this from past 10 days and raised an issue which got fixed immediately. Thanks to <a href="https://github.com/diegomarangoni">Diego Marangoni</a>. Go for it. If you face any issues, please let me know.</p>
<!-- /wp:paragraph -->