Load Testing Shiny Apps
Tags: performance, webperf, testing, tutorial
Published: 2021-03-01 15:33:03

<!-- wp:paragraph -->
<p>Data is ubiquitous. Shiny from R Studio helps to build interactive web applications from R. But performance testing is little tricky. If you sniff the traffic, you don't get any network data. If you are working in Shiny apps performance testing, then this blog article is for you. This blog article explains about load testing shiny apps using <code>shinyloadtest</code> package. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-what-is-shiny">What is Shiny?</h2>
<!-- /wp:heading -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p>Shiny is an R package that makes it easy to build interactive web apps straight from R. You can host standalone apps on a webpage or embed them in <a href="http://rmarkdown.rstudio.com/">R Markdown</a> documents or build <a href="http://rstudio.github.io/shinydashboard/">dashboards</a>. You can also extend your Shiny apps with <a href="http://rstudio.github.io/shinythemes/">CSS themes</a>, <a href="http://www.htmlwidgets.org/">htmlwidgets</a>, and JavaScript <a href="https://github.com/daattali/shinyjs/blob/master/README.md">actions</a>.</p></blockquote>
<!-- /wp:quote -->

<!-- wp:paragraph -->
<p>You can develop shiny app without any advanced development skills. Once the development is done, you can leverage shinyapps.io to deploy the app in the cloud or use open-source or commercial version of Shiny Server or RStudio Connect respectively.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-sample-demo-app">Sample Demo App</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here is one of the sample demo app from the shiny package.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8229,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/02/image-10.png"><img src="https://qainsights.com/wp-content/uploads/2021/02/image-10-901x1024.png" alt="Sample Demo App - Load Testing Shiny Apps" class="wp-image-8229"/></a><figcaption>Sample Demo App - Load Testing Shiny Apps</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:generateblocks/headline {"uniqueId":"4cc12e12"} -->
<h2 class="gb-headline gb-headline-4cc12e12 gb-headline-text">Network Sniffing Shiny App</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>If we leverage dev tools, it doesn't capture any network calls. Here is the screencast of the sample demo app network capture. We cannot traditional tools like LoadRunner, JMeter or NeoLoad etc. Enter <code>shinyloadtest</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<iframe src="https://share.qainsights.com/xQunEnYd?embed=true" allowtransparency="true" allowfullscreen="allowfullscreen" style="border: medium none;" data-frame-src="https://share.qainsights.com/xQunEnYd?embed=true" width="575" height="400" frameborder="0"></iframe>
<!-- /wp:html -->

<!-- wp:generateblocks/headline {"uniqueId":"f218f756"} -->
<h2 class="gb-headline gb-headline-f218f756 gb-headline-text">What is shinyloadtest?</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p><code>shinyloadtest</code> is a package which works along with <code>shinycannon</code> (command line tool) which enables load testing Shiny apps. </p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-load-testing-shiny-apps">Load Testing Shiny Apps</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To load test shiny apps, following are the prerequisites:</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>RStudio</li><li>shinyloadtest package</li><li>shinycannon</li></ol>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3 id="h-rstudio">RStudio</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>RStudio is an IDE for R. It comes with free and commercial version. You can download the open source edition from <a href="https://rstudio.com/products/rstudio/" target="_blank" rel="noreferrer noopener nofollow">here</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-shinyloaddtest">shinyloaddtest</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>You can install shinyloadtest package from here. shinyloadtest helps to record and analyze the results. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Launch RStudio, create a new project. In the console enter <code>install.packages("shinyloadtest")</code> and hit enter.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This will install <code>shinyloadtest</code> package as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8231,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/02/image-11.png"><img src="https://qainsights.com/wp-content/uploads/2021/02/image-11.png" alt="Install shinyloadtest" class="wp-image-8231"/></a><figcaption>Install shinyloadtest</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To run the sample demo app, enter the following commands.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>library('shiny')</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>runExample("07_widgets")</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This will launch the demo app locally with random port as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8232,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/02/image-12.png"><img src="https://qainsights.com/wp-content/uploads/2021/02/image-12.png" alt="Sample Demo App" class="wp-image-8232"/></a><figcaption>Sample Demo App</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Do not close this console. We are going to make use of this to record and load testing using <code>shinyloadtest</code> and <code>shinycannon</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-shinycannon">shinycannon</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>shinycannon helps to inject the load by replaying the recorded log. To install it in Windows, download the jar file from <a href="https://rstudio.github.io/shinyloadtest/articles/shinycannon.html#installation" target="_blank" rel="noreferrer noopener">here</a>. </p>
<!-- /wp:paragraph -->

<!-- wp:generateblocks/headline {"uniqueId":"28a7ef40"} -->
<h2 class="gb-headline gb-headline-28a7ef40 gb-headline-text">Let's Start Recording</h2>
<!-- /wp:generateblocks/headline -->

<!-- wp:paragraph -->
<p>As you know, the first step is to record the business flow. To begin recording, we are going to leverage shinyloadtest package.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Create a new project in RStudio (do not close the previous project where we are running the demo app).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In the Console, enter <code>library("shinyloadtest")</code> to load the library.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Then, enter <code>record_session("http://127.0.0.1:5049/")</code>, make sure that you are entering the correct port number where the demo app is running.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This will launch a new tab in your default browser with the different port number. That is how it works.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8234,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/02/image-13.png"><img src="https://qainsights.com/wp-content/uploads/2021/02/image-13.png" alt="Recording Shiny App" class="wp-image-8234"/></a><figcaption>Recording Shiny App</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>In the browser, just record your business actions and once you are done, close the browser to stop the recording.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Back the console, you could see the message as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8235,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/02/image-14.png"><img src="https://qainsights.com/wp-content/uploads/2021/02/image-14.png" alt="Recording Done" class="wp-image-8235"/></a><figcaption>Recording Done</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To view the recording log, open the folder of your RStudio project, you could see the recording.log file. Here is how it looks.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8236,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/02/image-15.png"><img src="https://qainsights.com/wp-content/uploads/2021/02/image-15-1024x414.png" alt="Recording Log" class="wp-image-8236"/></a><figcaption>Recording Log</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="h-let-s-smoke-test">Let's Smoke Test</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>After the recording, it is always the best practice to run a smoke test to debug the script. To replay the log, we need <code>shinycannon</code>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Open the command prompt or PowerShell (or Terminal) and navigate to the folder where you have downloaded the shinycannon jar file.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Copy the <code>recording.log</code> file in this location as well.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To start the smoke test, enter the below command. This will run the smoke test with 1 thread and store the results to <strong>SmokeTesting </strong>folder.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>java -jar ..\shinycannon-1.1.0-45731f0.jar .\recording.log http://127.0.0.1:5049 --workers 1 --loaded-duration-minutes 1 --output-dir SmokeTesting</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here is the output of smoke testing.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>2021-02-28 22:51:59.648 INFO &#91;thread00] - Detected target application type: R/Shiny
2021-02-28 22:51:59.673 INFO &#91;thread00] - Waiting for warmup to complete
2021-02-28 22:51:59.879 INFO &#91;thread01] - Warming up
2021-02-28 22:51:59.881 INFO &#91;progress] - Running: 0, Failed: 0, Done: 0
2021-02-28 22:51:59.932 INFO &#91;thread00] - Maintaining for 1 minutes (60000 ms)
2021-02-28 22:52:04.941 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:09.943 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:14.943 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:19.955 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:24.956 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:29.960 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:34.962 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:39.963 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:44.966 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:49.966 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:54.967 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:52:59.940 INFO &#91;thread00] - Stopped maintaining, waiting for workers to stop
2021-02-28 22:52:59.967 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:53:04.990 INFO &#91;progress] - Running: 1, Failed: 0, Done: 0
2021-02-28 22:53:07.549 INFO &#91;thread01] - Stopped
2021-02-28 22:53:07.553 INFO &#91;thread00] - Complete. Failed: 0, Done: 1</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>We are good as we are not seeing any failures in our smoke testing. Now, let us inject more load to the demo app.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-let-s-start-load-testing">Let's Start Load Testing</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As we are running the app locally, it is not a good idea to inject high load. Just to demonstrate, I am going with 2 workers for 2 minutes. Issue the below command and monitor the output for any issues.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After the testing is completed, the next step is to generate the test results.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Back to RStudio project to issue few commands to generate the HTML report. Below command store the run to <code>df</code>. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>df &lt;- load_runs('Run1' = 'C:/Tools/Shinyapp/ShinyAppLoadTest/LoadTestRun1')</code></p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8237,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/02/image-16.png"><img src="https://qainsights.com/wp-content/uploads/2021/02/image-16-1024x278.png" alt="Load Test Raw Results" class="wp-image-8237"/></a><figcaption>Load Test Raw Results</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>To plot the results in RStudio, you can enter the below commands.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>slt_http_latency(df)
slt_hist_loadtimes(df, max_load_time = 5)
slt_session(df)
slt_websocket_latency(df, cutoff = 5)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To generate the HTML report, enter the below command.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><code>shinyloadtest_report(df)</code></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This will generate all kinds of graphs, plots, table etc. You can find the HTML report in the project folder. Here is the Waterfall chart.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8238,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/02/image-17.png"><img src="https://qainsights.com/wp-content/uploads/2021/02/image-17-1024x473.png" alt="Load Test Report" class="wp-image-8238"/></a><figcaption>Load Test Report</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Here is the event duration chart.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":8239,"sizeSlug":"large","linkDestination":"media"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><a href="https://qainsights.com/wp-content/uploads/2021/02/image-18.png"><img src="https://qainsights.com/wp-content/uploads/2021/02/image-18-1024x530.png" alt="Event Duration" class="wp-image-8239"/></a><figcaption>Event Duration</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Congratulations! Now you have successfully completed a load test of Shiny app.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-limitations">Limitations</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>shinyload test comes with its own limitations in terms of deployment and application types.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You cannot record or load test the shiny app which are deploed at shinyapps.io.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You need WebSockets and recordings are server dependent.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>R markdown documents with <code>runtime: shiny</code> are unsupported. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Apps with high latency may cause failures.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>I suggest you to look out for the issues in the <a href="https://github.com/rstudio/shinyloadtest/issues" target="_blank" rel="noreferrer noopener">GitHub repo</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-conclusion">Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you are working in Shiny apps, definitely you need to check out shinyloadtest. This package is king of fresh out of the oven, so it may not fulfill all of your requirements. But this is a good start for load testing shiny apps. Also, you can check out out my another article which uses <a href="https://qainsights.com/performance-testing-in-r-using-apache-jmeter/" target="_blank" rel="noreferrer noopener">JMeter under the hood</a>.</p>
<!-- /wp:paragraph -->