Performance Testing using Iter8
Tags: kubernetes, performance, testing, tutorial
Published: 2022-02-15 15:46:53

<!-- wp:paragraph -->
<p>There are umpteen <a href="https://github.com/QAInsights/Performance-Testing-Tools" target="_blank" rel="noreferrer noopener">performance testing tools</a> available in the commercial market as well as in the open source repositories. Based on our requirements, we can choose the best tool from the arsenal. Recently, Go based performance testing tools are exploding in the open source world. Go runtime provides very light-weight goroutines which execute the tasks quickly and efficiently. In this blog post, we are going to see about Iter8 - a simple Go based performance testing tool which validates the SLOs, performs Chaos testing, and more. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="iter8">Iter8</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>There are a couple of things you should be aware. Iter8's (pronunciation - Iter-eight) primary target audience are DevOps, MLOps, Developers, Performance Engineers and Testers in this order. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a href="https://github.com/iter8-tools/iter8/" target="_blank" rel="noreferrer noopener">Iter8</a> is a command line interface tool which sends loads of requests to the target URL and then evaluates the SLOs and, optionally, you can generate an HTML report.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Iter8 gels with the CI/CD and GitOps pipelines and is apt for the Kubernetes ecosystem. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="getting-started-with-iter8">Getting Started with Iter8</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Installing Iter8 is simple and easy. There are multiple methods you can follow to install Iter8: via Binaries, Brew, from the Source or using Go. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I am in the Windows ecosystem. I am going to use Go to install Iter8. The only prerequisite in this method is to have the latest version of Go installed. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Download Go from https://go.dev/dl/ and install it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To  validate the installation, open Windows Terminal or Command Prompt and enter <code>go version</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To install Iter8, enter <code>go install github.com/iter8-tools/iter8@latest</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To validate Iter8 installation, enter <code>iter8 version</code></p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9082,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/02/image-11.png"><img src="https://qainsights.com/wp-content/uploads/2022/02/image-11.png" alt="Validating Iter8 - Performance Testing using Iter8" class="wp-image-9082"/></a><figcaption>Validating Iter8 - Performance Testing using Iter8</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="run-an-experiment">Run an Experiment</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The next step is to run a performance test against a URL. In Iter8, an experiment consists of an <code><strong>Experiment Chart</strong></code> (similar to Helm charts) and user inputs via command line.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Experiment Charts</strong> are available in the Iter8 GitHub repo e.g. load test experiment is located at https://github.com/iter8-tools/iter8/tree/master/hub/load-test-http which is called <strong>Iter Hub</strong>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Right now, Iter8 has the following experiments: load-test-http and load-test-grpc.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="hello-world-experiment">Hello World Experiment</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Let us begin with a simple load test using <code>load-test-http</code> chart. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Enter the below command in your terminal and hit enter. This command will create a directory and download the experiment from Iter8 Hub.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>iter8 hub -e load-test-http</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>cd into <code>load-test-http</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Below is the content in <code>load-test-http</code> folder.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9084,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/02/image-12.png"><img src="https://qainsights.com/wp-content/uploads/2022/02/image-12.png" alt="Iter8 Experiment" class="wp-image-9084"/></a><figcaption>Iter8 Experiment</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>In <code>values.yaml</code>, the default values for the HTTP load test are available. Most of the values would be <code>null</code>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In <code>Chart.yaml</code>, the source of the experiment and the API definition can be seen.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To run an experiment against <a href="https://reqres.in/api/users/1" target="_blank" rel="noreferrer noopener">https://reqres.in</a> (please do not put more load), enter the below command and hit enter.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>iter8 run --set url=https://reqres.in/api/users/1</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This command will generate an <code>experiment.yaml</code> in the current directory. Below is the content of <code>experiment.yaml</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code># task 1: generate HTTP requests for application URL
# collect Iter8's built-in HTTP latency and error-related metrics
- task: gen-load-and-collect-metrics-http
  with:
    qps: 8
    connections: 4
    errorRanges:
    - lower: 400
    versionInfo:
    - url: https://reqres.in/api/users/1
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>By default, Iter8 will send 8 queries per second with 4 parallel connections for the duration will be automatically calculated.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Below is the output of the above experiment.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9087,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/02/image-13.png"><img src="https://qainsights.com/wp-content/uploads/2022/02/image-13-1024x181.png" alt="Experiment Output" class="wp-image-9087"/></a><figcaption>Experiment Output</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Once the experiment is done, Iter8 will generate <code>result.yaml</code> which contains the performance metrics such as latency, error count, error rate and more with the aggregation of min, max, percentiles, mean, standard deviation and more.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To generate the visualization, enter the below command which will generate a simple HTML report<code>.html</code> in the current directory. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>iter8 report -o html &gt; report.html</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Open <code>report.html</code> in your favorite browser.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9088,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/02/image-14.png"><img src="https://qainsights.com/wp-content/uploads/2022/02/image-14-684x1024.png" alt="Iter8 Report" class="wp-image-9088"/></a><figcaption>Iter8 Report</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>The HTML report consists of an Experiment Report, Histogram analysis, and metrics as shown above.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Following are the default metrics and SLOs collected by <code>load-test-http</code> experiment.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><code>request-count</code>: total number of requests sent</li><li><code>error-count</code>: number of error responses</li><li><code>error-rate</code>: fraction of error responses</li><li><code>latency-mean</code>: mean of observed latency values</li><li><code>latency-stddev</code>: standard deviation of observed latency values</li><li><code>latency-min</code>: min of observed latency values</li><li><code>latency-max</code>: max of observed latency values</li><li><code>latency-pX</code>: X<sup>th</sup>&nbsp;percentile of observed latency values, for&nbsp;<code>X</code>&nbsp;in&nbsp;<code>[50.0, 75.0, 90.0, 95.0, 99.0, 99.9]</code></li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2 id="sending-post-request">Sending POST request</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Iter8 supports the POST method as well. Below is an example of a POST request. </p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>iter8 run --set url=https://reqres.in/api/users `
--set payloadStr="{"name":"morpheus","job":"leader"}" `
--set contentType="application/json"</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2 id="assertions">Assertions</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Iter8 supports assertions to validate the output of each experiment. Send the below command which checks for completion, failures, and SLOs. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>iter8 assert -c completed -c nofailure -c slos</code></p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="slos">SLOs</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Below command sets the various SLOs.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>iter8 run --set url=https://reqres.in/api/users/1 `
--set SLOs.error-rate=0 `
--set SLOs.latency-mean=50 `
--set SLOs.latency-p90=100 `
--set SLOs.latency-p'97\.5'=200</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>For the above experiment, SLOs might fail, as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":9089,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2022/02/image-15.png"><img src="https://qainsights.com/wp-content/uploads/2022/02/image-15-1024x590.png" alt="Iter8 SLOs" class="wp-image-9089"/></a><figcaption>Iter8 SLOs</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:image {"align":"center","id":9090,"sizeSlug":"full","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-full"><a href="https://qainsights.com/wp-content/uploads/2022/02/image-16.png"><img src="https://qainsights.com/wp-content/uploads/2022/02/image-16.png" alt="Iter8 validation" class="wp-image-9090"/></a><figcaption>Iter8 validation</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="iter8-features">Iter8 Features</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Load testing is one of the features of Iter8. Apart from load testing, it supports A/B testing, K8s ecosystem, serverless, ML frameworks, and more.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>There are experiments available with Prometheus, Chaos testing, mirroring, traffic split, session affinity, gradual traffic shifting, and more.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I hope there will be more experiments available in the repo in the near future. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="what-is-missing">What is missing?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As a performance engineer, my expectations are high. Iter8 supports only a minimal subset of features for performance testing. E.g. distributed load testing and aggregation, data parameterization, easy debugging, UI performance, other protocols, and more are missing in this release. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="conclusion">Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The pitch of Iter8 is <code>Kubernetes Release Optimizer</code> where the users will conduct experiments either locally or in the Kubernetes ecosystem or via CI/CD/GitOps pipelines, and try out various versions, configurations, and modelling to find the best deployment for production.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Iter8 is one of the easiest frameworks to learn. Definitely you should check it out and see whether it can be used by your team. </p>
<!-- /wp:paragraph -->