Beyond Sentiment Analysis: Object Detection with ML.NET
Tags: machinelearning, azure, dotnet
Published: 2020-09-21 12:13:31

Despite my diverse engagements with different Microsoft stacks ranging from Power Platform to Azure AI, I have always been a .NET (C#) guy and because of that I have a special place in my heart for all things XAML and ML.NET ❤

As we've seen a lot of improvements and contributions in AI / ML spectrum from Microsoft and the community, I thought to share something super exciting and <strong>new </strong>in ML.NET. Nowadays, tools &amp; frameworks have improved so much that I thought to explore and learn new ways to solve more business problems rather than just focusing on Sentiment Analysis. Those fantastic developers and content creators, who are still working to address this problem (Sentiment Analysis) in many ways are awesome and I totally respect that. Anyway, today we're going to talk about the latest offering by ML.NET Model Builder <strong><sup>Preview </sup></strong>and that is, <strong>Object Detection.</strong>
<blockquote>Don't get me wrong. Even I love Sentiment Analysis use-case and that's the reason I built <a href="https://www.nuget.org/packages/SentimentAnalyzer/">SentimentAnalyzer</a> last year but yeah, let's not stick there!</blockquote>
<h3>Know '<strong>What' </strong>Before '<strong>How'</strong></h3>
Object Detection is a common Computer Vision problem. It's different than a simple image classification because it goes a little deeper, tries to locate and categorizes objects within the images.
<table align="center">
<tbody>
<tr>
<td><a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/Park-Cropped.jpeg"><img class="aligncenter wp-image-31011" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/Park-Cropped.jpeg" alt="Person" width="443" height="429" /></a></td>
<td><a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/objects.jpg"><img class="aligncenter wp-image-31012 size-full" title="Image taken from Azure Cognitive Services website" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/objects.jpg" alt="Objects" width="566" height="437" /></a></td>
</tr>
<tr>
<th>{Person} - <em>Image Classification</em></th>
<th>{Footwear, Person, Laptop, Plant} - <em>Object Detection<strong> </strong></em></th>
</tr>
</tbody>
</table>
There were numerous scenarios / cases available in ML.NET to solve many problems from Image Classification to Anomaly Detection to Text Classification but until last week, Object Detection wasn't available and there's a fair reason behind it.

Training of an object detection model requires A LOT of resources ⛽ in terms of compute (several hundreds of GPU hours) which may have more than million parameters with the labeled data. Therefore, the awesome team at ML.NET had to come with an option that should work fine for everyone. So, what they did is, instead of using the power of your machines, they gave you an ability to train your model using Azure ML, right from the Visual Studio experience!
<h3>But How...?</h3>
So for that, you first need to install the new <a href="https://marketplace.visualstudio.com/items?itemName=MLNET.07">Model Builder (Preview) extension</a> from Visual Studio Marketplace. Once it's downloaded, you just need to create a .NET Core (<em>Console) </em>app. Right click the project, go to Add and Click <strong>Machine Learning.</strong>

&nbsp;

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/ML-Experience.gif"><img class="aligncenter wp-image-31017 size-full" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/ML-Experience.gif" alt="Machine Learning Experience from Visual Studio" width="1920" height="724" /></a>

Once you're there, you need to choose <strong>Object Detection </strong>scenario where you will be asked to setup your <a href="https://docs.microsoft.com/en-us/dotnet/machine-learning/resources/azure-training-concepts-model-builder?WT.mc_id=AI-MVP-5003464#what-is-an-azure-machine-learning-workspace">Azure Machine Learning workspace</a>. Please note that you must have an Enterprise workspace. <em>Basic did not work for me so I had to upgrade.</em>

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/Azure-Workspace.png"><img class="aligncenter size-large wp-image-31016" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/Azure-Workspace-1024x557.png" alt="Azure Workspace setup from Visual Studio" width="1024" height="557" /></a>

&nbsp;

You can always create the Workspace, <a href="https://docs.microsoft.com/en-us/dotnet/machine-learning/resources/azure-training-concepts-model-builder?WT.mc_id=AI-MVP-5003464#what-is-an-azure-machine-learning-compute">Compute</a> and Experiment directly from Visual Studio experience and I can tell you that it has no issues at all. Once you're done with creation, click on *Create* (it won't create if it's the experiment is already there, so don't worry).
<h3>It's all about your Data</h3>
Next step is to add Data. You will be asked to add a JSON file with the image labeling. <em>Before you wonder what's that and how will you get that, just take a deep breath </em>💡

Microsoft Commercial Software Engineering team has done a GREAT job 💪 in building a tool that'd help you label your images with an easy effort. That tool is known as <a href="https://github.com/Microsoft/VoTT">Visual Object Tagging Tool (VoTT).</a> In this post, I am going to keep my scope very limited to Object Detection scenario so I will request you to go through <a href="https://github.com/Microsoft/VoTT#getting-started">this doc</a> and hopefully you will be able to get enough knowledge to label your images. In case you find any issues, don't forget to connect with me on <a href="https://twitter.com/arafattehsin">Twitter</a> or comment here with your specific question and I'll be more than happy to help you!

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/VOTT.jpg"><img class="aligncenter size-large wp-image-31019" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/VOTT-1024x507.jpg" alt="Visual Object Tagging Tool" width="1024" height="507" /></a>

&nbsp;

Once you're done with labeling your data (in my case it was just the images of Flowers, which is a very famous <a href="https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz">TensorFlow flower dataset</a>) then you can export the project. The export will be a JSON file which you need to locate in your Data section.

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/Data.jpg"><img class="aligncenter size-full wp-image-31018" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/Data.jpg" alt="Add Data in Visual Studio Machine Learning" width="809" height="646" /></a>

&nbsp;

Then comes your training part which will be done on Azure and you can monitor it in real time. Since it will take some good amount of time so instead of sitting back and relax, do something productive 👨‍💻

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/Flowers-Azure.jpg"><img class="aligncenter size-full wp-image-31021" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/Flowers-Azure.jpg" alt="Training in Azure" width="843" height="627" /></a>
<blockquote>&nbsp;

When I initially started exploring this, I got an <a href="https://github.com/dotnet/machinelearning-modelbuilder/issues/997">issue</a> which even <a href="https://github.com/dotnet/machinelearning-modelbuilder/issues/991#issuecomment-692916457">amazed ML.NET Product team</a>. Later on, I <a href="https://github.com/dotnet/machinelearning-modelbuilder/issues/997#issuecomment-693650673">got to know</a> that my Compute always need to be <strong>dedicated</strong> rather than of something Low priority or shared. So the team has raised another bug internally to <a href="https://github.com/dotnet/machinelearning-modelbuilder/issues/1004">correct the error message</a>.</blockquote>
After it's done, you will see an option to Evaluate your model. You can simply upload an image which was not a part of your training set to test the accuracy of your model. I just provided limited set of pictures that's why my model's accuracy is not as good as it could be.

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/Evaluate.jpg"><img class="aligncenter wp-image-31022" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/Evaluate-1024x607.jpg" alt="Evaluation for Model Builder in Visual Studio" width="719" height="426" /></a>

<em>Since this scenario (Object Detection) is in Preview that's why I always get this error after completion of my training set, right before it downloads the ONNX model to my machine.</em>

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/Error.jpg"><img class="aligncenter size-full wp-image-31020" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/Error.jpg" alt="Error while downloading the ONNX Model from Azure" width="502" height="239" /></a>

Then if you're satisfied with the result, you can always generate the code to be used in your applications later on. Furthermore, you can also check how it's providing you the results (similar to other ML.NET Model Builder scenarios).

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/Add-Code.jpg"><img class="aligncenter size-large wp-image-31023" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/Add-Code-1024x317.jpg" alt="Add Code from Model Builder Visual Studio" width="1024" height="317" /></a>

&nbsp;

&nbsp;

You can always test the code by providing a different image too!

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/Code-Test.jpg"><img class="aligncenter size-large wp-image-31025" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/Code-Test-1024x407.jpg" alt="Validate Code" width="1024" height="407" /></a>

&nbsp;

Just a glimpse for friends who want to see the complexity of this model;

<a href="https://www.arafattehsin.com/wp-content/uploads/2020/09/ONNX.gif"><img class="aligncenter wp-image-31024 size-full" src="https://www.arafattehsin.com/wp-content/uploads/2020/09/ONNX.gif" alt="ONNX Model from Birds eye view" width="1590" height="998" /></a>

Lastly, I was planning to make a better use-case than this to show you through my UWP app but I figured out that the ONNX model which is generated is not compatible with the Windows.AI.MachineLearning library hence I've opened up a <a href="https://github.com/dotnet/machinelearning/issues/5402">GitHub issue</a> and will see how it will go.

Until next time