Run and parse MSTest with multiple containers
Asked by Alfredo Hidalgo:

Hello, I'm trying to achive the following:

Run MSTest with multiple containers and get the test result in Build result. (ex: 11 Tests, 3 failed)

The point is the current MSTest task only accept one container, I want to run all ?y test assemblies in one row to generate only one testresult file.

Also, I dont want to explicit specify all containers because I want to avoid to reconfigure Bamboo task each time a new test assembly is added to solution.

So, my current approach is to execute MSTest from a custom MSBuild task that will look for \*.Test.dll assemblies and generate a command line with multiple "/container" parameters. This is running ok but when a test fail, the MSBuild task fail (exit code = 1, the MSTest exit code actually) and Bamboo assume build is failing and dont show test results information (ex 11 Test executed, 3 failed)

So, any one can give a hint on it ?


Answers:
author: Scott Guymer
body: I came up with a solution to this in the end

I created an MSBuild task to get all the names of the tests, run mstest and save the output to a file

Then use the test parser to pick up the file and parse the tests

here is the build XML which i saved as a .proj file in the troot of the repository (but coudl go anywhere althought you might have to change the include mask in the build file)

```
&lt;Project ToolsVersion="4.0" DefaultTarget="RunTests" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"&gt;  &lt;PropertyGroup&gt;    &lt;results_file&gt;./TestResults/results.trx&lt;/results_file&gt;  &lt;/PropertyGroup&gt;  &lt;PropertyGroup&gt;    &lt;MsTestExePath Condition="'$(MsTestExePath)'==''"&gt;C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe&lt;/MsTestExePath&gt;  &lt;/PropertyGroup&gt;  &lt;ItemGroup&gt;    &lt;TestAssemblies Include=".\**\bin\**\*.Tests.dll" /&gt;  &lt;/ItemGroup&gt;  &lt;ItemGroup&gt;    &lt;ResultsFile Include="$(results_file)" /&gt;  &lt;/ItemGroup&gt;  &lt;Target Name="RunTests"&gt;    &lt;Message Text="Found test assemblies: @(TestAssemblies)" /&gt;    &lt;MakeDir Directories="./TestResults" /&gt;    &lt;CallTarget Targets="RunMsTest" /&gt;  &lt;/Target&gt;  &lt;Target Name="RunMsTest" &gt;    &lt;Message Text="Running to results file: %(ResultsFile.Identity)" /&gt;    &lt;PropertyGroup&gt;      &lt;MsTestCommand&gt;"$(MsTestExePath)" @(TestAssemblies -&gt;'/testcontainer:&amp;quot;%(RecursiveDir)%(Filename)%(Extension)&amp;quot; ', ' ') /resultsfile:TestResults\%(ResultsFile.Identity)&lt;/MsTestCommand&gt;    &lt;/PropertyGroup&gt;    &lt;Message Text="Running command $(MsTestCommand)" /&gt;    &lt;Exec Condition=" '@(TestAssemblies)' != ''" Command="$(MsTestCommand)" ContinueOnError="true"&gt;      &lt;Output TaskParameter="ExitCode" ItemName="ErrorCode"/&gt;    &lt;/Exec&gt;    &lt;Message Text="Tests complete" /&gt;    &lt;Message Text="The exit code is $(ErrorCode)"/&gt;    &lt;Error Text="Error while executing MSTest" Condition="'$(ErrorCode)' != ''" /&gt;    &lt;OnError ExecuteTargets="MessageErrorHandler"/&gt;         &lt;/Target&gt;  &lt;Target Name="MessageErrorHandler"&gt;    &lt;Message Text="An error has occurred while executing MSTest"/&gt;  &lt;/Target&gt;&lt;/Project&gt;
```

what i then did was create an msbuild task in bamboo and set it to build the proj file. then set the following in the options line

/p:results_file="testresults-${bamboo.buildKey}-${bamboo.buildNumber}.trx"

this tells it which file name to use.

I then created an mstest parser task to pick up the file

testresults-${bamboo.buildKey}-${bamboo.buildNumber}.trx

took a little trial and error but works great. Even spits out the test results into the log..

comments: author: Nimesh Shah
body: I tried this and it runs all tests successfully however Ms test Parser task is not picking up the file.

as you mentioned i did little trial and error too however none of my tricks worked. Can you elaborate on what did you do to make it work finally.


author: Florian Hoerschlaeger
body: I currently have the problem that MSTest parser does not seem to work correctly. Of an expected number of around 1300 tests only 650 tests are shown in the bamboo status page of the build. The approach for generating the TRX file is similar to the one mentioned above.

The genenerated TRX file contains all expected tests as well as the correct results, hence the test runner does his jobs. I've looked into the implementation of MSTestXmlTestResultsParser, which should be responsible for parsing trx files. MSTestXmlTestResultsParser does not perform any filtering, hence the problem must be somewhere else.

First I suspected that duplicate testnames are filtered, but that does not seem to be the case. The trx contains indeed duplicate test names, but after filtering duplicates there are only 617 left and not 650. Does anybody have an idea?


author: Alfredo Hidalgo
body: Hello, almost the same solution I applied in my side. Custom MSBuild task, like Scott describe.

Its working ok for me.


author: Scott Guymer
body: Im trying to overcome the same issue. Did you get anywhere?

The only thing i can think of is to runa script to identify all \*.test.dll and put them into a bamboo run config variable separated by /testcontainer

Then run a custom task to start mstest with the previous string and output to a path using variables

Then run the test parser...


Tags: ms-testrunner