Magento Cloud Performance testing of sugarfina.com using K6.io
Tags: magento, aws, magento2, adobe
Published: 2020-10-27 23:26:56

Sugarfina was migrated from M1 to M2 Cloud by Corra Magento Agency and today we will check the real performance of the Magento Expansive Cloud.

Sugarfina candy shop has traffic around 100-140K visitors per month.

Pretty small, however for our test it is a good candidate.

We will test Page generation time By Magento in different throughput/concurrency load. 1, 5, 10, 20, 40 concurrent requests to ensure that multiple users can harmoniously use a Magento eCommerce website without increasing TTFB (Time to First Bite) latency, which badly affecting user experience. 

**TTFB** is a measurement of how long it takes the first bits to reach the browser from the initial request to the Magento Cloud server.

TTFB is a very key metric because the browser cannot begin rendering Magento Page content until it has the initial markup. Google recommending to have TTFB around 300 ms.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/26xrep6k1xhz1u7airst.png)

For the test, I will use the NodeJS scripting and GO **K6**:

```
          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

```

k6 is a developer-centric, free and open-source load testing tool built for making Magento 2 performance testing a productive and enjoyable experience.

## Install Magento k6 load test globally as root:

```
wget https://bintray.com/loadimpact/rpm/rpm -O bintray-loadimpact-rpm.repo
sudo mv bintray-loadimpact-rpm.repo /etc/yum.repos.d/
sudo yum install k6 

or Docker 

docker pull loadimpact/k6
```

# Magento load test usage 

The set of basic options are not compatible with Apache ab toll which I have used before in my previous posts. But while ab can only set a concurrency level and lets the server adjust to it, K6 allows you to make scripting.


K6 is also quite extensible. Using the provided API it is very easy to integrate K6 with your package and run programmatic Magento 2 load tests. K6 makes it very easy to run load tests as part of systems tests, before deploying a new version of your software. The results include mean response times and percentiles so that you can abort deployment e.g. if 99% of the requests don't finish in 10 ms or less.

## Calculating Magento 2 Requests Per Second with k6
With k6, you can test Magento 2 in terms of * requests per second (RPS) * by limiting how many requests each VU (virtual user) is able to make per unit of time. The formula to calculate the RPS is as follows:
```
Request Rate = (VU * R) / T
```
Request Rate: measured by the number of requests per second (RPS)
**VU**: the number of virtual users
**R**: the number of requests per VU iteration
**T**: a value larger than the time needed to complete a VU iteration

The above formula has can be thought of as a way to calculate the required number of virtual users you need for your script. The number of requests R are already known to you, since you have already defined them in your script. In or case it is 1. We are testing just a single Magento page per user. Then you need to calculate the time T needed to complete a VU iteration. Suppose you have 10 requests per virtual user, and you expect each to take 10 s to complete under the load (http_req_duration). The overall time T to complete the whole VU iteration is the number of requests multiplied by the supposed round-trip of requests/responses. It is also good practice to add one or more seconds to T to account for delays.
```
T = (R * http_req_duration) + (delay/sleep time) →

T = (1 * 10s) + 0s = 10s
```
Server response time is important.

Now that you know the values for R and T, you need to decide on the RPS you want to achieve. Suppose you want to achieve 1000 RPS, i.e. you want your system to handle 1000 requests per second. Given the previous formula:
```
Request Rate = (VU * R) / T
```
The VU calculation formula is:

```
VU = (Request Rate * T) / R →

VU = (1000 * 10) / 1 = 10000

```

10000 virtual users (threads) we need to generate 1000 requests per second load. 

However, using my I7 CPU I can achieve only 100 Request per second using 1000 VUs Also with a higher RPS rate Response time will increase and you will need more VUs. basically, you need to find a point where response time is constant and this number is your optimal throughput. For the Magento Cloud, it is 1 request per second. See the results below.    

You can check this in the results: 
```
 http_reqs..................: 1000   61.712989/s
 iteration_duration.........: min=200.27µs med=8.73s    avg=8.38s    max=16.05s  p(95)=13.24s 
vus_max....................: 1000
```
where 1000 is VUs and 61 requests per second 

max iteration duration = 16 sec. 

RPS = 1000 / 16 = 62.5

We will test only two main use cases of the Magento eCommerce functionality.

1. Catalog Category Page Generation Performance 
2. Catalog Product Page Generation Performance 

We will test the same scenarios multiple times to calculate an average and 95 percentile of the Magento 2 page generation performance. 

# 1. Magento cloud Product Page Generation performance 

let's start by running a simple local script. Copy the code below, paste it into your favourite editor, and save it as "magento.js":

```
import http from 'k6/http';
import { sleep } from 'k6';
export default function () {
  http.get('http://test.k6.io');
  sleep(1);
}
```

Then run k6 using this command:

```
# k6 run magen.js -e url=https://www.sugarfina.com/sugar-skull-3-piece-candy-bento-box -u 1 -i 1 --include-system-env-vars=false
```

We will use 95% percentile -  latency for 95% of the users without maximum extremums.

A percentile is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall. 

*** Why to use 95% percentile with Magento performance test***

The 95th percentile encompasses the experience of almost all of your eCommerce users, with only the most severe outliers excluded. This makes it perfect for spotting short-term trends and anomalies. 

```
export function setup(data) {

 console.log(JSON.stringify(__ENV));
 return {'url': __ENV.url}

}

export default function (data) {

// The order of precedence is: defaults < config file < exported script options < environment variables < command-line flags. 
// Options from each subsequent level can be used to overwrite the options from the previous levels, with the CLI flags having the highest priority.

  var res = http.get(data.url);
  console.log('Response time : ' + String(res.timings.duration) + ' ms');
  console.log('Response status : ' + String(res.status) + ' ms');
  console.log('Response TTFB : ' + String(res.timings.waiting) + ' ms');
}
```

Ok, it is our K6 Magento test results:

## Concurrency 1 

```
data_received..............: 2.7 MB 141 kB/s
    data_sent..................: 16 kB  814 B/s
    http_req_blocked...........: min=1.33µs   med=1.44µs   avg=43.76ms  max=875.23ms p(95)=43.76ms 
    http_req_connecting........: min=0s       med=0s       avg=421.72µs max=8.43ms   p(95)=421.72µs
    http_req_duration..........: min=763.87ms med=924.78ms avg=917.88ms max=1.13s    p(95)=1.05s   
    http_req_receiving.........: min=5.64ms   med=8.58ms   avg=10.68ms  max=38.65ms  p(95)=25.34ms 
    http_req_sending...........: min=63.09µs  med=100.17µs avg=121.9µs  max=533.39µs p(95)=151.09µs
    http_req_tls_handshaking...: min=0s       med=0s       avg=2.13ms   max=42.78ms  p(95)=2.13ms  
    http_req_waiting...........: min=754.7ms  med=915.74ms avg=907.07ms max=1.12s    p(95)=1.05s   
    http_reqs..................: 20     1.033262/s
    iteration_duration.........: min=249.36µs med=925.82ms avg=875.22ms max=1.66s    p(95)=1.13s   
    iterations.................: 20     1.033262/s
    vus........................: 1      min=1 max=1
    vus_max....................: 1      min=1 max=1
```

Average TTFB is: 907 ms 
95 Percentile: 1.05 ms

Let's check Magento 2 self-hosted on AWS Graviton2 medium (cheapest one) processor TTFB to feel the difference. 

```
data_received..............: 1.3 MB 198 kB/s
    data_sent..................: 2.2 kB 322 B/s
    http_req_blocked...........: min=8.07µs   med=8.98µs   avg=3.69ms   max=73.64ms  p(95)=3.69ms  
    http_req_connecting........: min=0s       med=0s       avg=3.67ms   max=73.42ms  p(95)=3.67ms  
    http_req_duration..........: min=317.29ms med=320.63ms avg=323.99ms max=388.13ms p(95)=329.85ms
    http_req_receiving.........: min=36.75ms  med=37.33ms  avg=39.35ms  max=76.4ms   p(95)=41.19ms 
    http_req_sending...........: min=34.94µs  med=49.95µs  avg=58.19µs  max=207.18µs p(95)=73.37µs 
    http_req_tls_handshaking...: min=0s       med=0s       avg=0s       max=0s       p(95)=0s      
    http_req_waiting...........: min=279.81ms med=283.13ms avg=284.58ms max=311.51ms p(95)=288.75ms
    http_reqs..................: 20     2.994795/s
    iteration_duration.........: min=229.61µs med=321.17ms avg=298.58ms max=462.92ms p(95)=327.35ms
    iterations.................: 20     2.994795/s
    vus........................: 1      min=1 max=1
    vus_max....................: 1      min=1 max=1

```

Average TTFB hosted on AWS without silly Magento Clouds is - 284ms with 95 percentile - 288.75 ms vs 1.05s on the Adobes Commerce Cloud.

**Magento 2 Open Source is not so bad as its Commerce Cloud.**

## Concurrency 5

```
    data_received..............: 1.4 MB 491 kB/s
    data_sent..................: 10 kB  3.6 kB/s
    http_req_blocked...........: min=1.53µs   med=26.98ms  avg=28.69ms  max=60.71ms  p(95)=60.69ms 
    http_req_connecting........: min=0s       med=3.51ms   avg=3.55ms   max=7.19ms   p(95)=7.17ms  
    http_req_duration..........: min=777.66ms med=1.08s    avg=1.16s    max=1.58s    p(95)=1.53s   
    http_req_receiving.........: min=3.25ms   med=16.62ms  avg=17.04ms  max=31.84ms  p(95)=31.5ms  
    http_req_sending...........: min=96.09µs  med=213.98µs avg=219.29µs max=344.63µs p(95)=337.84µs
    http_req_tls_handshaking...: min=0s       med=12.36ms  avg=14.08ms  max=31.48ms  p(95)=31.46ms 
    http_req_waiting...........: min=751.53ms med=1.06s    avg=1.14s    max=1.57s    p(95)=1.52s   
    http_reqs..................: 10     3.513193/s
    iteration_duration.........: min=209.78µs med=1.11s    avg=993.06ms max=1.58s    p(95)=1.52s   
    iterations.................: 10     3.513193/s
    vus........................: 4      min=4 max=5
    vus_max....................: 5      min=5 max=5
```

Average: 993.06 ms however 
95 percentile: 1.52 s 

#Concurrency 10

```
 data_received..............: 2.8 MB 658 kB/s
    data_sent..................: 21 kB  4.9 kB/s
    http_req_blocked...........: min=1.15µs   med=430.42ms avg=431.66ms max=865.56ms p(95)=864.4ms 
    http_req_connecting........: min=0s       med=3.68ms   avg=4.02ms   max=8.85ms   p(95)=8.84ms  
    http_req_duration..........: min=743.42ms med=1.01s    avg=1.15s    max=2s       p(95)=1.93s   
    http_req_receiving.........: min=3.5ms    med=19.7ms   avg=20.08ms  max=52.83ms  p(95)=36.34ms 
    http_req_sending...........: min=64.98µs  med=201.84µs avg=239.58µs max=571.64µs p(95)=510.48µs
    http_req_tls_handshaking...: min=0s       med=16.26ms  avg=18.19ms  max=38.56ms  p(95)=37.81ms 
    http_req_waiting...........: min=732.85ms med=990.26ms avg=1.13s    max=1.95s    p(95)=1.9s    
    http_reqs..................: 20     4.711487/s
    iteration_duration.........: min=236.87µs med=1.64s    avg=1.44s    max=2.86s    p(95)=2.79s   
    iterations.................: 20     4.711487/s
    vus........................: 2      min=2  max=10
    vus_max....................: 10     min=10 max=10
```
Average: 1.13 s
95 Percentile: 1.9 s 

## Concurrency 20

```
 data_received..............: 5.6 MB 1.1 MB/s
    data_sent..................: 42 kB  8.4 kB/s
    http_req_blocked...........: min=1.15µs   med=27.47ms avg=30.02ms  max=66.41ms  p(95)=63.32ms
    http_req_connecting........: min=0s       med=3.62ms  avg=4.17ms   max=9.68ms   p(95)=9.61ms 
    http_req_duration..........: min=771.83ms med=1.65s   avg=1.72s    max=3.28s    p(95)=2.7s   
    http_req_receiving.........: min=2.27ms   med=20.44ms avg=17.4ms   max=32.78ms  p(95)=30.48ms
    http_req_sending...........: min=95.26µs  med=188.4µs avg=235.17µs max=511.37µs p(95)=447.4µs
    http_req_tls_handshaking...: min=0s       med=18.08ms avg=20.18ms  max=44.44ms  p(95)=43.85ms
    http_req_waiting...........: min=748.61ms med=1.63s   avg=1.71s    max=3.26s    p(95)=2.68s  
    http_reqs..................: 40     8.063447/s
    iteration_duration.........: min=272.16µs med=1.64s   avg=1.67s    max=3.28s    p(95)=2.69s  
    iterations.................: 40     8.063447/s
    vus........................: 5      min=5  max=20
    vus_max....................: 20     min=20 max=20
```

Average: 1.71 s
95 Percentile: 2.69 s

## Concurrency 40 

```
 data_received..............: 11 MB 1.9 MB/s
    data_sent..................: 83 kB 14 kB/s
    http_req_blocked...........: min=1.23µs   med=40.84ms avg=43.5ms   max=99.45ms  p(95)=89.57ms 
    http_req_connecting........: min=0s       med=3.48ms  avg=4.24ms   max=10.41ms  p(95)=9.64ms  
    http_req_duration..........: min=789.82ms med=2.35s   avg=2.2s     max=3.17s    p(95)=2.95s   
    http_req_receiving.........: min=2.87ms   med=24.22ms avg=24ms     max=61.64ms  p(95)=41.88ms 
    http_req_sending...........: min=57.8µs   med=208.3µs avg=215.18µs max=538.94µs p(95)=420.11µs
    http_req_tls_handshaking...: min=0s       med=32.07ms avg=33.33ms  max=76.46ms  p(95)=67.97ms 
    http_req_waiting...........: min=760.18ms med=2.32s   avg=2.18s    max=3.14s    p(95)=2.93s   
    http_reqs..................: 80    13.485888/s
    iteration_duration.........: min=223.3µs  med=2.36s   avg=2.19s    max=3.27s    p(95)=3.04s   
    iterations.................: 80    13.485888/s
    vus........................: 13    min=13 max=40
    vus_max....................: 40    min=40 max=40
```

Avarage: 2.18 s
Percentile 95: 3.05s

And this results in measuring only initial page rendering performance because of Magento 2 bad architecture your website also needs several other slow Ajax calls to fully rendered pages (Cart, Banners, Customer Data, Customisations requests). Order and checkout pages need 10-15 HTTP requests to be fully functional. Nothing bad to have additional requests however they should be fast and don't affect user experience.
In my next post, I will add that HTTP requests to the test scenarios to show AJAX functionality impact on Magento 2 performance. Thanks to the K6 performance tool we can achieve that without using legacy Jmeter scripts.  

# Magento 2 Product Page Performance result Pivot Table:
```
+----+------+---------------+
|    | AVG  | Percentile 95 |
+----+------+---------------+
|  1 |  907 |          1005 |
+----+------+---------------+
|  5 |  993 |          1520 |
+----+------+---------------+
| 10 | 1130 |          1900 |
+----+------+---------------+
| 20 | 1710 |          2690 |
+----+------+---------------+
| 40 | 2180 |          3050 |
+----+------+---------------+
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/7xq8gju9i9vut2yxsgk0.png)

# Magento 2 Cloud Category Page Performance


## Concurrency 1

```
 data_received..............: 3.8 MB 356 kB/s
    data_sent..................: 20 kB  1.9 kB/s
    http_req_blocked...........: min=1.37µs   med=1.6µs    avg=4.26ms   max=42.59ms  p(95)=23.43ms
    http_req_connecting........: min=0s       med=0s       avg=818.34µs max=8.18ms   p(95)=4.5ms  
    http_req_duration..........: min=780.04ms med=928.68ms avg=1.05s    max=2.01s    p(95)=1.62s  
    http_req_receiving.........: min=18.25ms  med=22.32ms  avg=25.97ms  max=50.4ms   p(95)=41.3ms 
    http_req_sending...........: min=88.63µs  med=99.24µs  avg=143.06µs max=480.58µs p(95)=333.4µs
    http_req_tls_handshaking...: min=0s       med=0s       avg=2.31ms   max=23.14ms  p(95)=12.73ms
    http_req_waiting...........: min=750.57ms med=905.09ms avg=1.02s    max=1.99s    p(95)=1.6s   
    http_reqs..................: 10     0.936201/s
    iteration_duration.........: min=202.77µs med=880.62ms avg=883.05ms max=2.01s    p(95)=1.54s 
```

Avarage: 1.02 s
95 Percentile: 1.6 s

# Cuncurancy 5 

```
data_received..............: 5.7 MB 1.2 MB/s
    data_sent..................: 32 kB  6.5 kB/s
    http_req_blocked...........: min=1.38µs   med=2.03µs   avg=14.44ms max=46.7ms   p(95)=45.25ms 
    http_req_connecting........: min=0s       med=0s       avg=2.78ms  max=9.59ms   p(95)=9.55ms  
    http_req_duration..........: min=913.24ms med=1.32s    avg=1.42s   max=2.03s    p(95)=1.99s   
    http_req_receiving.........: min=8.16ms   med=23.72ms  avg=29.28ms max=76.4ms   p(95)=57.03ms 
    http_req_sending...........: min=93.49µs  med=129.68µs avg=177.4µs max=401.33µs p(95)=361.66µs
    http_req_tls_handshaking...: min=0s       med=0s       avg=7.77ms  max=25.51ms  p(95)=24.12ms 
    http_req_waiting...........: min=868.41ms med=1.31s    avg=1.39s   max=1.96s    p(95)=1.95s   
    http_reqs..................: 15     3.088035/s
    iteration_duration.........: min=226.43µs med=1.24s    avg=1.27s   max=2.07s    p(95)=2s      

```
Avarage: 1.39 s
95 percentile: 1.95s

## Concurrency 10 

```
http_req_blocked...........: min=1.42µs   med=426.01ms avg=427.6ms  max=860.05ms p(95)=856.9ms 
    http_req_connecting........: min=0s       med=3.52ms   avg=3.68ms   max=8.55ms   p(95)=8.08ms  
    http_req_duration..........: min=864.14ms med=1.19s    avg=1.57s    max=3.69s    p(95)=3.14s   
    http_req_receiving.........: min=12.09ms  med=36.62ms  avg=43.11ms  max=128.41ms p(95)=78.15ms 
    http_req_sending...........: min=68.59µs  med=202.28µs avg=205.02µs max=398.16µs p(95)=335.29µs
    http_req_tls_handshaking...: min=0s       med=11.38ms  avg=12.67ms  max=28.67ms  p(95)=27.09ms 
    http_req_waiting...........: min=835.87ms med=1.17s    avg=1.53s    max=3.65s    p(95)=3.13s   
    http_reqs..................: 20     3.825479/s
    iteration_duration.........: min=232.35µs med=1.8s     avg=1.82s    max=4.55s    p(95)=3.34s   
  
``` 
Average: 1.53 ms
95 percentile: 3.13 s

## Concurrency 20 

```
    http_req_blocked...........: min=55.08ms  med=59.98ms  avg=60ms    max=64.07ms  p(95)=63.9ms  
    http_req_connecting........: min=6.84ms   med=7.97ms   avg=8.08ms  max=9.16ms   p(95)=9.16ms  
    http_req_duration..........: min=1.04s    med=2.28s    avg=2.17s   max=3.36s    p(95)=3.23s   
    http_req_receiving.........: min=34.65ms  med=52.98ms  avg=68.49ms max=296.89ms p(95)=109.76ms
    http_req_sending...........: min=208.84µs med=311.28µs avg=328.6µs max=473.67µs p(95)=470.09µs
    http_req_tls_handshaking...: min=35.95ms  med=40.02ms  avg=39.95ms max=42.3ms   p(95)=42.19ms 
    http_req_waiting...........: min=1s       med=2.23s    avg=2.1s    max=3.29s    p(95)=3.17s   
    http_reqs..................: 20     5.661847/s
    iteration_duration.........: min=259.99µs med=2.18s    avg=2.02s   max=3.42s    p(95)=3.27s   

```

Average: 2.1 s
95 Percentile: 3.17 s 

## Concurrency 40 

```
     http_req_blocked...........: min=1.27µs   med=445.03ms avg=448.82ms max=901.91ms p(95)=900.54ms
    http_req_connecting........: min=0s       med=3.59ms   avg=4.83ms   max=15.95ms  p(95)=11.08ms 
    http_req_duration..........: min=810.1ms  med=2.22s    avg=2.43s    max=5.91s    p(95)=4.06s   
    http_req_receiving.........: min=11.92ms  med=55.08ms  avg=64.52ms  max=500.94ms p(95)=106.56ms
    http_req_sending...........: min=60.61µs  med=179.17µs avg=194.95µs max=575.51µs p(95)=366.95µs
    http_req_tls_handshaking...: min=0s       med=31.87ms  avg=33.64ms  max=69.3ms   p(95)=68.84ms 
    http_req_waiting...........: min=765.03ms med=2.13s    avg=2.37s    max=5.41s    p(95)=4s      
    http_reqs..................: 80     9.819241/s
    iteration_duration.........: min=231.72µs med=2.79s    avg=2.81s    max=6.82s    p(95)=4.94s   
    iterations.................: 80     9.819241/s
    vus........................: 1      min=1  max=40
    vus_max....................: 40     min=40 max=40

```

Average: 2.37 s
95 Percentile: 4 s

# Magento 2 Category Page Performance result Pivot Table:

```
+----+------+---------------+
|    | AVG  | Percentile 95 |
+----+------+---------------+
|  1 | 1020 |          1600 |
+----+------+---------------+
|  5 | 1390 |          1950 |
+----+------+---------------+
| 10 | 1530 |          3130 |
+----+------+---------------+
| 20 | 2100 |          2690 |
+----+------+---------------+
| 40 | 2370 |          4000 |
+----+------+---------------+
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/o3fsi8p7kz2ubtib77e1.png)

How about Corra eCommerce implementation? 
The backend performance is pretty awful. However, it can be a Magento Cloud mistake, not Corra's implementation.

Letch check theme frontend performance

# Corra Magento theme Google Page Speed Frontend performance

## Mobile:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/wombg1rh2hlrt4bv7bp0.jpg)

You can check it there: https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fwww.sugarfina.com%2F3-piece-haunted-house-candy-bento-box

First Contentful Paint: 5.3 s
Speed Index: 19.6 s
Largest Contentful Paint: 55.9 s
Time to Interactive: 58.3 s
Total Blocking Time: 2,440 ms
Cumulative Layout Shift: 0.534

58 seconds to fully load the page on a mobile device is Absolutely not acceptable...

How about the desktop?

## Desktop:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/7p9ernasgtz1ne0keo8e.jpg)

Pretty decent result.

So, we can see that Magento 2 is not the best for eCommerce websites and Magento Agencies should spend 100000 hours and merchants should pay for that 1 00000 000 000$$$ for the Magento and Adobe architectural and design failures. 

As we can see, Magento Commerce Cloud and Magento 2 itself (no big difference hosting can't improve results so much even if TTFB will be 2 times faster) can generate 1-2 simultaneous requests without affecting user experience.