Load Testing with Locust
Tags: testing, loadtesting, azure, webdev
Published: 2020-12-09 18:38:16

In this post, I will walk through how my team mapped performance characteristics for a web app we created for a customer, and how we picked between using just an App Service, a CDN (Content Delivery Network) and Front Door (all Azure Services). I will use a simple [voting app](https://github.com/Azure-Samples/azure-voting-app-redis) in this blog as an example to compare the results. For our load/performance testing framework we chose to use [Locust](https://locust.io/), an open source load testing tool, as it is scriptable and distributed, and very easy to deploy on Azure. Feel free to follow along for testing your own app!

###Goal
Through load/performance testing, our goal was to determine the number of concurrent users and response times that our customer's application could handle. This would help us understand the bottlenecks and capabilities of the page speed, scalability and stability of the web app. We tested the time-to-first-byte speed for the web app as the key performance metric.

###Definitions
Before I move into the how and why of our top pick, I'm going to define the 3 solutions so you can better understand what we are comparing here.
1. [Azure App Service](https://azure.microsoft.com/en-us/services/app-service/): A fully managed platform for building, deploying and scaling your web apps.
2. [Content Delivery Network](https://www.cloudflare.com/learning/cdn/what-is-a-cdn/): A content delivery network (CDN) refers to a geographically distributed group of servers which work together to provide fast delivery of Internet content. A CDN allows for the quick transfer of assets needed for loading Internet content including HTML pages, JavaScript files, stylesheets, images, and videos.
3. [Azure Front Door](https://docs.microsoft.com/en-us/azure/frontdoor/front-door-overview): Front Door is a global, scalable entry-point that uses the Microsoft global edge network to create fast, secure, and widely scalable web applications.

---

###How to Set up and Use Locust
You can use [this](https://github.com/tavishigupta/locust-blog/tree/master/deployment) deployment script to deploy the locust structure in Azure. If you already know how set up Locust and what the different metrics mean, feel free to skip this section!

####Start a Test
Go to the Locust dashboard and follow the instructions. The Locust dashboard can be reached by going to the IP of the master node and port 8089 (Example: http://7xttd3hv5jaac-master.eastus.azurecontainer.io:8089).  
Enter the desired numbers and start swarming.
![Dashboard](https://dev-to-uploads.s3.amazonaws.com/i/cfjcqquvhkim6letjhui.png)  

#####How to pick values:
* _Total users to simulate_: It's recommended that you start with a number of simulated users that are greater than `number of user classes * number of workers` when running Locust distributed. In our current case, we have 1 user class and 3 worker nodes.
* _Hatch rate_: If the hatch rate is lower than the number of worker nodes, the hatching would occur in "bursts" where all worker node would hatch a single user and then sleep for multiple seconds, hatch another user, sleep and repeat.
* _Host_: The host attribute is a URL prefix (e.g., "http://google.com") to the host that is to be loaded.  

>Note: If number of workers on the dashboard is more that the worker nodes available, [redeploy](https://github.com/tavishigupta/locust-blog/tree/master/deployment) the dashboard with the required number of worker nodes/instances.  

####View and Analyze Results
After swarming for a while, your dashboard will look something like this:
![Analysis Dashboard](https://dev-to-uploads.s3.amazonaws.com/i/unw28myqp76czpse66xe.png)

#####Definitions:
* _Requests_: Total number of requests made so far
* _Fails_: Number of requests that have failed
* _Median_: Response speed for 50 percentile in ms
* _90%ile_: Response speed for 90 percentile in ms
* _Average_: Average response speed in ms  
* _Min_: Minimum response speed in ms  
* _Max_: Maximum response speed in ms  
* _Average size (bytes)_: Average response size in bytes  
* _Current RPS_: Current requests per second
* _Current Failures/s_: Total number of failures per second  

Your graphs will look something like this:
![graph1](https://dev-to-uploads.s3.amazonaws.com/i/gfd1p1eyi04r20kyqrza.png)
![graph2](https://dev-to-uploads.s3.amazonaws.com/i/68tn4dju6bp2jy4dftr8.png)

These graphs can be downloaded using the download icon next to them.
![downloads button](https://dev-to-uploads.s3.amazonaws.com/i/uhv5wotr3wdkn82gs4k8.png)

And/or you can download the data under the download data tab.
![downloads tab](https://dev-to-uploads.s3.amazonaws.com/i/uhv5wotr3wdkn82gs4k8.png)

You can analyze the graphs based on **response** and **volume** metrics.

#####Response Metrics
* _Average response time_ measures the average amount of time that passes between a client's initial request and the last byte of a server's response, including the delivery of HTML, images, CSS, JavaScript, and any other resources. It's the most accurate standard measurement of the actual user experience.
* _Peak response time_ measures the roundtrip of a request/response cycle (RTT) but focuses on the longest cycle rather than taking an average. High peak response times help identify problematic anomalies.
* _Error rates_ measure the percentage of problematic requests compared to total requests. It's not uncommon to have some errors with a high load, but obviously, error rates should be minimized to optimize the user experience.

#####Volume Metrics
* _Concurrent users_ measure how many virtual users are active at a given point in time. While similar to requests per second (see below), the difference is that each concurrent user can generate a high number of requests.
* _Requests per second_ measures the raw number of requests that are being sent to the server each second, including requests for HTML pages, CSS stylesheets, XML documents, JavaScript files, images, and other resources.
* _Throughput_ measures the amount of bandwidth, in kilobytes per second, consumed during the test. Low throughput could suggest the need to compress resources.


---

###Deploy the App Service, CDN and Front Door
Next, I deployed my app service in conjunction with front door. I used a [terraform script](https://github.com/Azure-Samples/frontdoor-appservice-vnet-terraform) written by my colleague to do so. Next, I added a CDN to my app service using [this](https://docs.microsoft.com/en-us/azure/cdn/cdn-add-to-web-app) tutorial. For the purpose of this test, I removed all access restrictions for the app service, so that I could avoid the forbidden access (403) issues. I did this by navigating to the **networking** tab in my app service resource, and then removing all the access restrictions. You can follow along though these screenshots:
![networking](https://dev-to-uploads.s3.amazonaws.com/i/1x5n8a6vxfpmr2uyazto.png)
![access restriction](https://dev-to-uploads.s3.amazonaws.com/i/sfx0dnq52y08hsquhocb.png)

Once this is done, you should have _3 endpoints_: an App Service endpoint, a CDN endpoint and a Front Door endpoint.


---

###Running the Locust Tests
####Assumptions
* The goal of load/performance testing was **not** to check the correctness of the code or data. Those would require integration or acceptance tests.
* The responsiveness of the different elements of the page were **not** being tested in the load/performance test.
* The loading time of each components on the map (graphs, content, logos) was **not** measured since the data points were not being rendered on the initial load.
* The data was embedded as a static geojson in the web app.

####Constraints
For our customer, we only tested 2 routes (/ and /about) based on how the web app was set up. For our purposes, to keep things simple, I am only testing 1 route (/). If you would like to test more than 1 route, feel free to edit the python script however you'd like by reading the [locust documentation](https://docs.locust.io/en/stable/writing-a-locustfile.html).

####Inputs & Controlled Variables
* **Number of concurrent users**: This value was kept constant at 5000 concurrent users.
* **Hatching rate**: This value was kept constant at 10 users being spawned every second.
* **Time**: Each test was run for about 20 minutes.
* **Host**: This variable was specified based on which infrastructure was being tested. So there were 3 hosts that were tested.

---

###Comparing and Analyzing the Results
Some things to note here:
* In the **Total Requested** chart, the green line shows the successful requests, and the red line shows the failures.
* In the **Response Time** chart, the green line shows the median response time, and the yellow line shows the 90th percentile.

####Case 1: Just the App Service
![case1 dashboard](https://dev-to-uploads.s3.amazonaws.com/i/654mklmg64qcshmemdfr.png)
![case1 charts](https://dev-to-uploads.s3.amazonaws.com/i/xrizb3jhlvk4mj84y5ka.png)

####Case 2: App Service with a CDN
![case2 dashboard](https://dev-to-uploads.s3.amazonaws.com/i/3uyp0e1x82o8f6o6ulwj.png)
![case2 charts](https://dev-to-uploads.s3.amazonaws.com/i/2zysvm4fbvcs759zm9g1.png)

####Case 3: App Service with Front Door
![case3 dashboard](https://dev-to-uploads.s3.amazonaws.com/i/eod9ddhsb47cmfwsl27l.png)
![case3 charts](https://dev-to-uploads.s3.amazonaws.com/i/dpvpflegin2epdo1u6j1.png)

####Analysis
If you have been following along with the set up, deployment and running test, the next step is to analyze our data and understand the metrics we use to measure performance. For my particular example, I will be looking at the response time, error rate, requests per second and peak response time. Before we dive into the comparisons, here are some benchmarks and explanations for comparison:
* In 2020, [the average Time-To-First-Byte](https://backlinko.com/page-speed-stats) (TTFB) speed was found to be 1.28 seconds (1280ms) on desktop and 2.59 seconds (1590ms) on mobile. However, Google's best practice is to achieve a time under 200ms.
* On average, larger scale applications can reach ~2000 requests per second. Since the application I am testing is lightweight, these numbers might not be very useful. I will still do an analysis so you can do it for yourself.
* Similar to the average response time, the peak response time (PRT) is the measurement of the longest responses for all requests coming through the server. This is a good indicator of performance pain points in the application.
* According to [HTTPArchive](https://httparchive.org/) and their [page weight report](https://httparchive.org/reports/page-weight), the average size of a website is 1.966 Mb for desktop and 1.778 Mb for mobile at the time of writing. Google's best practice is to be below 0.5 Mb.
    * The web app we are testing is very lightweight, so it's only 949 bytes. However, our customer's app size was about 4.5 Mb, which was larger to load.  

![table](https://dev-to-uploads.s3.amazonaws.com/i/ciuu3vsuzikt4yiqm5tk.PNG)

#####What do the errors mean?
* _104 Connection Reset by Peer_: Connection Reset error indicates that a TCP RST was received, and the connection is now closed. This occurs when a packet is sent from the user's end of the connection, but the other end does not recognize the connection; it will send back a packet with the RST bit set in order to forcibly close the connection. This usually happens when there is too much load on the server.
* _502 Bad-Gateway_: Bad Gateway server error response code indicates that the server, while acting as a gateway or proxy, received an invalid response from the upstream server.
* _503 Service unavailable_: Service Unavailable server error response code indicates that the server is not ready to handle the request. This can also be a result of an uncaught error in your code.
* _504 Gateway-Timeout_: Gateway Timeout server error response code indicates that the server, while acting as a gateway or proxy, did not get a response in time from the upstream server that it needed in order to complete the request.

---

###Our Final Choice
As we can see in the analysis section, the app service performs quite poorly by itself with poor average TTFB, many failures, high peak response time and (for a lightweight application) low requests per seconds. On the other hand, the CDN and Front Door solutions perform pretty much on par given most of the metrics. For our purposes, we picked Front Door because we needed a WAF (Web Application Firewall), which is still in preview for CDN.


---

I hope reading about how we went about performance/load testing is helpful in setting up your own testing practices! Please feel free to comment or DM me on Twitter ([@itstavishi](https://twitter.com/itstavishi)) with any questions, updates or comments you would like me to address!