In Memory Automated UI Testing ASP.NET Core
Tags: webdev, csharp, testing
Published: 2021-11-05 15:53:04

<h2>Introduction</h2>
<p>In this article we look at how to run in-memory automated UI tests for an ASP.NET Core web app using Playwright and NUnit.</p>
<p>The article provides demo code and solutions to issues found along the way.</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b8h42milcnnqqfbnomk8.jpg" alt="Wet tennis ball rotating quickly" width="492" height="328" /></p>
<h2>Automated UI Testing</h2>
<p>Automated testing of any web application is essential to ensure it functions correctly.&nbsp; On the top of the &ldquo;testing pyramid&rdquo; proudly sits UI testing or end-to-end testing, above integration and unit testing.&nbsp; Automated UI testing involves launching and controlling a browser to drive through a set of interactions that a user would perform.&nbsp; Assertions are made to see if the web app and browser behave as expected.&nbsp; Browsers are controlled by testing tools such as Selenium,&nbsp; Puppeteer or the new kid on the block, Playwright.</p>
<p>Normally, tests are run against a web application deployed to a test environment which has been configured with the same infrastructure as the production environment.&nbsp; However, in this post we will be looking at running our UI tests against an <em>in-memory</em> instance of a web application.&nbsp; By running the test in this way, we can:</p>
<ul>
<li>Develop tests faster.</li>
<li>Reconfigure the app with test configuration.</li>
<li>Catch some errors earlier in the dev/test process.</li>
<li>Still run the tests against a deployed environment later.</li>
</ul>
<p>Before we get started, we need to take a small step back.</p>
<h2>Integration testing with WebApplicationFactory</h2>
<p>Microsoft provide a neat way to run <a title="ASP.NET Core integration testing" href="https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests" target="_blank" rel="noopener">integration tests</a> using an in-memory web server called <em>TestServer</em>.</p>
<p>Using the <code>WebApplicationFactory</code> class will provide you with a running TestServer. &nbsp;It requires a type in the entry point assembly of the application. Typically the Startup or Program classes can be used. &nbsp;The <code>WebApplicationFactory</code> class also provides a virtual <code>ConfigureWebHost()</code> method, which allows for the web host to be manipulated.&nbsp; This is powerful and means services can be removed, added, or swapped.&nbsp; For example, a SQL Server database can be removed and replaced with an in-memory database.</p>
<p>A unit test framework such as xUnit or NUnit can be used to write and run the tests targeting the app hosted in the in-memory TestServer.&nbsp; When tests run they use an <code>HttpClient</code> connection given by the TestServer.&nbsp; The <code>HttpClient</code> is a C# class and is not a browser, which means its a bit limited and inconvenient to use.</p>
<p>Using the TestServer approach, we have an in-memory web server at our disposal.&nbsp; Not only that, it provides a way to override the way the webhost is built so that we can customise our application for our testing needs.&nbsp; That sounds great! Unfortunately, when using a browser automation tool, such as Selenium or Playwright, the TestServer approach does not quite work.</p>
<h2>Modifying Test Server to Work with Selenium and Playwright</h2>
<p>After reading some really useful articles by <a href="https://www.benday.com/2021/07/19/asp-net-core-integration-tests-with-selenium-webapplicationfactory/" target="_blank" rel="noopener">Ben Day</a>, <a href="https://blog-bertrand-thomas.devpro.fr/2020/01/27/fix-breaking-change-asp-net-core-3-integration-tests-selenium/" target="_blank" rel="noopener">Bertrand Thomas</a>&nbsp; and an oldie from <a href="https://www.hanselman.com/blog/real-browser-integration-testing-with-selenium-standalone-chrome-and-aspnet-core-21" target="_blank" rel="noopener">Scott Hanselman</a> (thanks guys!), I&rsquo;ve derived a class from <code>WebApplicationFactory</code> which works with browser testing tools including Selenium and Playwright. In <a title="Demo project on GitHub" href="https://github.com/JD-Innovensa/InMemoryPlaywrightDemo" target="_blank" rel="noopener">the demo project</a>, this is the <code>AutomatedTestServerFactory</code> class.</p>
<p>One of the main issues when using a TestServer with a browser testing tool is that <code>ConfigureWebHost()</code> is not called.&nbsp; This means the app cannot be changed at start up to use a test configuration.&nbsp; In the &ldquo;<em>traditional</em>&rdquo; way of using TestServer, we are given an <code>HttpClient </code>instance for our request/response interaction with the server. &nbsp;The <code>HttpClient </code>instance is created using the <code>CreateClient()</code> method in <code>WebApplicationFactory</code>, which in turn runs the <code>ConfigureWebHost()</code> method.&nbsp; So, the really useful re-configuration we need is only done when an <code>HttpClient </code>is created! Browser testing tools effectively replace the client and are controlled independently of the TestServer.</p>
<p>To get round this issue, <code>AutomatedTestServerFactory</code>&nbsp; sets the configuration when the webhost is built. &nbsp;The work is mainly done in two methods <code>CreateServer()</code> and <code>CreateWebHostBuilder()</code>, which are called by the constructor.</p>
<p><code>CreateWebHostBuilder()</code> is called first and sets the web host configuration, which is passed through as a constructor parameter. &nbsp;Also optionally set is the environment, otherwise the default environment is Production, which may not be ideal.</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mo9ha1rsauha73yy8n2q.png" alt="Code from CreateWebHostBuilder method" width="640" height="312" /></p>
<p><code>CreateServer()</code> builds and starts the web application and begins listening for requests. This method returns a null TestServer.&nbsp; I didn&rsquo;t see any impact of this and it avoids a new and different instance of the web app from being created.</p>
<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6qs3xisoljqyaxu9mpjd.png" alt="Code from CreateServer method" width="759" height="168" /></p>
<p>With these changes in place, we have a WebApplicationFactory that creates an in-memory TestServer hosting our re-configured web application and is suitable for use with Selenium or Playwright.&nbsp;&nbsp; Super!</p>
<h2>Using Selenium or Playwright</h2>
<p>When developing this project, I started using Selenium as the browser testing tool, however I noticed in October 2021, that Microsoft updated the integration test article (see above) to recommend Playwright instead of Selenium.&nbsp; So I took a look, I liked it, and I have used Playwright for the remainder of the project.&nbsp; <a href="https://playwright.dev/">Playwright</a> is a relatively new end-to-end testing tool developed and maintained by Microsoft.</p>
<p>The reasons I liked Playwright include its simplification of browser driver management, that it works with Chromium, Firefox and Webkit browsers and the ease of use of the API compared to Selenium&rsquo;s.&nbsp; Out of interest, I found migrating tests from Selenium to Playwright to be straightforward too!</p>
<p>The decision to use Playwright also led to selecting NUnit as the test runner.&nbsp; Typically, I use xUnit for testing, but it doesn't support <a title="xUnit support for Playwright" href="https://playwright.dev/dotnet/docs/test-runners#xunit-support" target="_blank" rel="noopener">running Playwright tests in parallel</a>.</p>
<h3>Configuring Playwright on local PC</h3>
<p>To get Playwright to work, there are two steps required using Package Manager Console:</p>
<ol>
<li>Install Playwright CLI as a global tool.
<ul>
<li>Run the command: <code>install -g Microsoft.Playwright.CLI</code></li>
</ul>
</li>
<li>Install browsers.
<ul>
<li>Navigate to the path of the test project and run the command: <code>playwright install</code></li>
</ul>
</li>
</ol>
<h2>Code Time!</h2>
<p>In order to demonstrate this working, I&rsquo;ve created a <a href="https://github.com/JD-Innovensa/InMemoryPlaywrightDemo" target="_blank" rel="noopener">demo application available on GitHub</a>.&nbsp; The codebase has three parts as described below.</p>
<h3>Employee System</h3>
<p>This app is a very simple ASP.NET Core MVC application using EntityFrameworkCore with SqlClient. The only functionality is a homepage and a page returning a list of employees.&nbsp; This is enough for our purposes though!&nbsp; It has a migration script, so remember to run <code>update-database</code> before launching it.</p>
<p>This is what the Employee List page looks like:</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yx4e73n5ni9nx8tvquaj.png" alt="Table of employees in demo app." width="475" height="169" /></p>
<h3>Test code</h3>
<p>The test code is comprised of two libraries, AutomatedTesting and Employees.UITests. &nbsp;The AutomatedTesting library contains the custom <code>AutomatedTestServerFactory </code>class to enable in-memory testing with Selenium or Playwright.&nbsp;</p>
<p>Employees.UITests is an NUnit test library.&nbsp; It contains a base class, <code>BaseTestClass</code>, which is responsible for creating the web server and passing in the required configuration, in this case changing the database to in-memory and adding test data. &nbsp;The base class is decorated with the attribute <code>[Parallelizable(ParallelScope.Children)]</code> to allow child tests to run in parallel.</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9415z8crk8xyfhjtq1fg.png" alt="Code of BaseTestClass" width="676" height="445" /></p>
<p>The <code>EmployeeListTests </code>class inherits from <code>BaseTestClass </code>and focusses on the tests, so the code is quite clean. There are three tests, each one is the same, but using different browsers.&nbsp; Running the test project will cause all three tests to run in parallel.&nbsp; Finally, I&rsquo;ve used FluentAssertions (<a title="10 ten tips on Fluent Assertions" href="../fluent-assertions-10-top-tips" target="_blank" rel="noopener">as I like it</a>!) for evaluating the outcomes of tests.</p>
<p>In the excerpt below, the Employee menu option is clicked and assertions are made on the output.</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n4lkdfgxmraptcdkymwm.png" alt="Fragment of test code using Playwright" width="680" height="200" /></p>
<p>I have started to pull out some common tasks in <code>TestHelper</code>, such as taking a screenshot and attaching it to the test report.</p>
<p>After running the tests, you can see the results in the test log by right-clicking on a test and opening the log.</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dkchj2qxzccv1vkdcopd.png" alt="" width="271" height="268" /></p>
<p>In Visual Studio 2019, the results include both the log messages and the screenshots.</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9wifxtcq5g929us39rjr.png" alt="Test log" width="307" height="168" /></p>
<h3>Devops</h3>
<p>I have included a yaml build pipeline for Azure DevOps.&nbsp; This configuration deals with the Playwright requirements, builds the code and executes the tests in a cloud-hosted Windows server.&nbsp; It can target a self-hosted Windows target too.</p>
<p>&nbsp;</p>
<h2>Resolved Issues</h2>
<p>Developing this project was not without its issues.&nbsp; Apart from getting the AutomatedTestServerFactory to work, here are the issues I found and overcame.</p>
<h3>Issues found on Developer PC</h3>
<h4>Issue 1:</h4>
<p>The web application launches, but is missing styling, JavaScript and images. This was because the test server would launch in the bin directory of the test project and not have a reference to the wwwroot folder.&nbsp;</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5w7v6470nraahqm5gi6d.png" alt="" width="291" height="140" /></p>
<p><strong>Solution: </strong>This was resolved by adding a post-build target to the Employees.UITests project file.&nbsp; The target copies the wwwroot folder recursively from the Employees.UI project to the Employees.UITests\bin folder.&nbsp;</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/swageep6cu0g5hn5o3x2.png" alt="" width="349" height="134" /></p>
<p><a href="https://docs.microsoft.com/en-us/visualstudio/msbuild/copy-task?view=vs-2019#example-2">Microsoft have an example of how to copy files recursively during after build.</a></p>
<p>&nbsp;</p>
<h4>Issue 2:</h4>
<p><strong><span style="color: #e03e2d;">Microsoft.Playwright.PlaywrightException : Executable doesn't exist</span></strong></p>
<p><strong>Solution</strong>: One or more of the browsers is missing.&nbsp; Run <code><span style="font-family: 'courier new', courier, monospace; font-size: 10pt;">playwright install</span></code> command in the Package Manager Console, but significantly make sure this is done in the directory of the test project.</p>
<h3>Issues found in Azure Devops</h3>
<h4>Issue 3:</h4>
<p>Install step of Playwright Tool causes error when re-run.&nbsp; The following two lines in the log file indicate the issue.</p>
<pre>Tool 'microsoft.playwright.cli' is already installed.<br />Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1</pre>
<p><img style="display: block; margin-left: auto; margin-right: auto;" src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mo7kfy3rtksq470kxer7.png" alt="Playwright install error in Azure Devops" width="637" height="61" /></p>
<p><strong>Solution</strong>: Instead of calling <code>dotnet install</code>, run the <code>dotnet update</code> command as this removes and clean installs the tool.</p>
<h4>Issue 4:</h4>
<p>Error: <strong><span style="color: #e03e2d;">Cannot find a tool in the manifest file that has a command named 'playwright'.</span></strong></p>
<p><strong>Solution:</strong> Make sure the playwright install command runs in the directory of the test project.</p>
<h4>Issue 5:</h4>
<p>Cloud-hosted Chrome fails with exception: <span style="color: #e03e2d;"><strong>Microsoft.Playwright.PlaywrightException : net::ERR_CERT_AUTHORITY_INVALID at <a style="color: #e03e2d;" href="https://localhost:5001/">https://localhost:5001/</a></strong></span></p>
<p><strong>Solution:</strong> Get a new browser instance to get round this issue using the code below.&nbsp; This effectively runs the tests in incognito mode.</p>
<p><code>await browser.NewContextAsync(new BrowserNewContextOptions { IgnoreHTTPSErrors = true });</code></p>
<h4>Issue 6:</h4>
<p>Firefox does not navigate to test site. &nbsp;Instead a warning is displayed in the browser</p>
<p><span style="background-color: #f1c40f;">&ldquo;Warning: Potential Security Risk Ahead&rdquo;</span>.</p>
<p>Solution: Firefox prevents navigation to another port on localhost.&nbsp; The warning requires manual intervention, but the settings can be overridden so that the automated test can run.&nbsp; To do so, set the following capabilities (<a href="https://medium.com/volosoft/how-to-disable-firefox-warning-potential-security-risk-ahead-f081fbf81a4f">these settings are briefly covered here</a>):</p>
<p><code>caps.Add("security.insecure_field_warning.contextual.enabled", false);</code></p>
<p><code>caps.Add("security.certerrors.permanentOverride", false);</code></p>
<p><code>caps.Add("network.stricttransportsecurity.preloadlist", false);</code></p>
<p><code>caps.Add("security.enterprise_roots.enabled", true);</code></p>
<h3>Other points:</h3>
<ol>
<li>Firefox not waiting for AJAX request to complete.</li>
</ol>
<p style="padding-left: 40px;">When waiting for a page to load that fires off an AJAX request, for example fetching data on page load, Firefox does not seem wait.&nbsp; To resolve add <code>await WaitForLoadStateAsync()</code> after the command to load the page.</p>
<ol start="2">
<li>Azure DevOps only includes NUnit log on test failure.</li>
</ol>
<p style="padding-left: 40px;">NUnit provides the convenient <code>TestContext.Out</code> to log messages during testing.&nbsp; These will always appear using the test runner in Visual Studio 2019, but these messages are only present in Azure Devops when a test fails.</p>
<h2>Conclusion</h2>
<p>In this article we have explored through the code, configuration and gotchas of running automated UI tests of an ASP.NET Core web app using Microsoft&rsquo;s TestServer.&nbsp; We have been able to perform UI testing using Playwright with multiple browsers running in parallel using NUnit.</p>
<p>We have seen a number of issues, mainly arising in Azure DevOps, and how to overcome them.</p>
<p>In-memory automated UI testing could be great for smaller systems and to help develop test code. It's not an approach for all cases, but certainly has its merits.</p>
<p>Finally, if you&rsquo;ve not used Playwright for automated testing, perhaps it is a good opportunity to try it out!</p>
<p>&nbsp;</p>