Black Friday Performance Testing Experiment Results
Tags: webperf, performance, testing, tooling
Published: 2020-12-14 16:42:38

<!-- wp:paragraph -->
<p><strong>Black Friday Performance Testing Experiment Results </strong>I conduct some sort of performance testing around holidays season every year. Couple of years ago, I was doing it manually and keep it myself. But this year, I did a semi-automated performance testing experiment on US retailers. This blog post will slice and dice the performance data of the experiment; I hope it reveals some interesting facts.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-black-friday-performance-testing-experiment-setup">Black Friday Performance Testing Experiment Setup</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you would like to conduct this experiment on your own it will costs you nothing. The only setup you have to do is API keys. Below is the GitHub repository which will help you get started.</p>
<!-- /wp:paragraph -->

{% github QAInsights/Black-Friday-Performance-Testing %}

<!-- wp:heading -->
<h2>Objective</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The objective of this exercise is to measure the response time trend of popular US retail websites during <strong>Thanksgiving and Black Friday 2020</strong>. This will help us to reveal the performance and its effect of US retail giants.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":7945,"sizeSlug":"large","linkDestination":"media"} -->
<figure class="wp-block-image size-large"><a href="https://qainsights.com/wp-content/uploads/2020/12/Black-Friday-Experiment.png"><img src="https://qainsights.com/wp-content/uploads/2020/12/Black-Friday-Experiment-1024x512.png" alt="Black Friday Performance Testing Experiment Results" class="wp-image-7945"/></a><figcaption>Black Friday Performance Testing Experiment Results</figcaption></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Strategy</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This experiment <strong>will not inject load</strong> to the websites under test. The script will <strong>send only one HTTP(S) request every 30 minutes</strong> to measure the performance. The actual workload will be from the real users.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>HTTP request timeout is set to 30 seconds. If the script doesn't receive any response from Page Speed, then that request will be ignored.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Only the desktop performance will be captured.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>List of Websites under test</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here are the <a href="https://github.com/QAInsights/Black-Friday-Performance-Testing/blob/master/urls.yaml" target="_blank" rel="noreferrer noopener">URLs </a>which will be tested. If you would like to add any other websites, please submit a PR.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Tools</h2>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li><a href="https://developers.google.com/speed/docs/insights/v5/about">Page Speed Insights</a></li><li>Python 3.8</li><li>GitHub Actions<ul><li>Chrome browser</li><li>Ubuntu 18.04 LTS</li></ul></li><li>Influx DB Cloud (Free Plan)</li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>Test Window</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Test will start at 12.00 AM EST on Nov 25 and ends at 12.01 AM EST Nov 30.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Performance Metrics</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Following metrics will be captured:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>First Contentful Paint (FCP)</li><li>First Input Delay (FID)</li><li>Largest Contentful Paint (LCP)</li><li>Cumulative Layout Shift (CLS) </li>
</ul>
# How to setup this experiment on your own?
<ul>
<li>Clone this repo</li><li>Install the requirements using <code>pip install -r requirements.txt</code></li><li>Grab a Page Speed Insights API from <a href="https://developers.google.com/speed/docs/insights/v5/get-started">here</a>.</li><li>Create a GitHub Secret <code>PAGE_SPEED_API_KEY</code> in your repo. For instructions, please check <a href="https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets">here</a>. Never ever expose your API key in your repo.</li><li>In your <code>.github/workflows/python-app.yml</code> file, configure your details in the line 41 and 42.</li><li>By default, GitHub Action will get triggered on every push.</li><li>To schedule the run for every 15 minutes, use the below config in the yaml file.</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code> on:
  schedule:
    # * is a special character in YAML so you have to quote this string
    - cron:  '*/15 * * * *'</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2 id="h-faqs">FAQs</h2>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li>Can I view the performance realtime?<ul><li>No. The setup doesn't include that. But you can download the raw results anytime once the experiment started and analyze it yourself.</li></ul></li><li>What is the cost involved in this experiement?<ul><li>Nothing. All are using free resources.?</li></ul></li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2 id="h-result-analysis">Result Analysis</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here comes the interesting part of this experiment. The raw data is present in <a href="https://github.com/QAInsights/Black-Friday-Performance-Testing/blob/master/results.csv" target="_blank" rel="noreferrer noopener">here</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Before we see the numbers, let us see what is FCP, LCP, FID and CLS is.</p>
<!-- /wp:paragraph -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>“First Contentful Paint reports the time when the browser first rendered any text, image (including background images), non-white canvas or SVG. This includes text with pending webfonts. This is the first time users could start consuming page content.”</p><cite>https://developers.google.com/web/tools/chrome-user-experience-report?hl=en</cite></blockquote>
<!-- /wp:quote -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>“First Input Delay (FID) is an important, user-centric metric for measuring load responsiveness because it quantifies the experience users feel when trying to interact with unresponsive pages—a low FID helps ensure that the page is usable.” </p><cite><a href="https://web.dev/fid/">web.dev/fid/</a></cite></blockquote>
<!-- /wp:quote -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>“Largest Contentful Paint (LCP) is an important, user-centric metric for measuring perceived load speed because it marks the point in the page load timeline when the page's main content has likely loaded—a fast LCP helps reassure the user that the page is useful.” - </p><cite><a href="https://web.dev/lcp/">web.dev/lcp/</a></cite></blockquote>
<!-- /wp:quote -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>“Cumulative Layout Shift (CLS) is an important, user-centric metric for measuring visual stability because it helps quantify how often users experience unexpected layout shifts—a low CLS helps ensure that the page is delightful.” -</p><cite><a href="https://web.dev/cls/">web.dev/cls/</a></cite></blockquote>
<!-- /wp:quote -->

<!-- wp:heading {"level":3} -->
<h3 id="h-field-data">Field Data</h3>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li>Highest FCP is sprint.com <strong>3548 ms</strong></li><li>Lowest FCP is lenovo.com and lowes.com - <strong>905 ms and 908 ms</strong></li><li>Highest LCP is sprint.com <strong>9271 ms</strong></li><li>Lowest LCP is lenovo.com and lowes.com - <strong>977 ms and 1402 ms</strong></li><li>Highest First Input Delay is bjs.com <strong>182 ms</strong></li><li>Lowest First Input Delay is wholefoods.com <strong>3 ms</strong></li><li>Highest Cumulative Layout Shift Score is harborfrieght.com <strong>181 ms</strong></li><li>Lowest Cumulative Layout Shift Score is ofcourse <strong>google.com</strong></li></ul>
<!-- /wp:list -->

<!-- wp:image {"align":"center","id":7940,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2020/12/image-25.png"><img src="https://qainsights.com/wp-content/uploads/2020/12/image-25-1024x664.png" alt="" class="wp-image-7940"/></a><figcaption>FCP Max - Sprint.com - Lab data</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:image {"align":"center","id":7941,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2020/12/image-26.png"><img src="https://qainsights.com/wp-content/uploads/2020/12/image-26-1024x599.png" alt="" class="wp-image-7941"/></a><figcaption>FCP Low - Lowes.com - Lab data</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>It will be time consuming to go thru all the 30 URLs data, so I have analyzed only the highest and lowest performance number. If you are interested to deep-dive, you can download the raw results and check it out.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>But here is the big 3 (apple, amazon, and google) performance graph.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":7943,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2020/12/image-28.png"><img src="https://qainsights.com/wp-content/uploads/2020/12/image-28-1024x486.png" alt="Big 3 Performance Stats" class="wp-image-7943"/></a><figcaption>Big 3 Performance Stats</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Below is the Page Speed Insights classification.</p>
<!-- /wp:paragraph -->
<!-- wp:table -->
<figure class="wp-block-table"><table><tbody><tr><th></th><th>Good</th><th>Needs Improvement</th><th>Poor</th></tr><tr><td>FCP</td><td>[0, 1000ms]</td><td>(1000ms, 3000ms]</td><td>over 3000ms</td></tr><tr><td>FID</td><td>[0, 100ms]</td><td>(100ms, 300ms]</td><td>over 300ms</td></tr><tr><td>LCP</td><td>[0, 2500ms]</td><td>(2500ms, 4000ms]</td><td>over 4000ms</td></tr><tr><td>CLS</td><td>[0, 0.1]</td><td>(0.1, 0.25]</td><td>over 0.25</td></tr></tbody></table><figcaption>PSI Classification</figcaption></figure>
<!-- /wp:table -->

<!-- wp:paragraph -->
<p>Below URLs clocked &lt; 1000 ms FCP:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Google Store</li><li>Amazon</li><li>BJs</li><li>Lenovo</li><li>Lowes</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Below URls clocked &lt; 2500 ms LCP:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Amazon</li><li>Apple</li><li>BestBuy</li><li>Dell</li><li>Kohls</li><li>Lenovo</li><li>Lowes</li><li>Overstock</li><li>Verizon</li><li>Walgreens</li><li>Walmart</li><li>Wholefoods</li></ul>
<!-- /wp:list -->

<!-- wp:generateblocks/headline {"uniqueId":"cb2de07f"} -->
<h2 class="gb-headline gb-headline-cb2de07f gb-headline-text">Final Words</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>As you know this experiment conducted only the subset of retailers, not every website out there. Also, it doesn't conclude which website is better performance or compare with each other. The purpose of this study is to understand the importance of performance and how website performs under heavy load.</p>
<!-- /wp:paragraph -->