Object Detection Simplified with AI Builder
Tags: machinelearning, powerfuldevs, beginners
Published: 2022-02-08 12:13:05

Few months back, I posted a <a href="https://www.linkedin.com/posts/arafattehsin_from-custom-machine-learning-model-to-its-activity-6818494208110821376--3aF/">small video clip</a> on my LinkedIn to show how I used ML.NET (w/ Azure), WinML &amp; UWP to build a custom real-time object detection application. I received a few messages of folks on how I actually did that therefore, I decided to submit my talk &amp; spoke to <a href="https://www.arafattehsin.com/speaking/real-time-object-detection-with-your-custom-models/">NDC Sydney, 2021</a> to actually showcase the working of it.

After its implementation, one of former colleagues asked if I could achieve this using Power Platform i.e. AI Builder (<em>because of its object detection capability</em>) and Canvas Apps. I responded in negative because I remember this limitation when I wrote about using <a href="https://www.arafattehsin.com/beginners-lets-build-covid-safe-solutions-with-ai-power-platform/">AI Builder with Camera Stream</a> in one of my previous posts.

The limitation was that we could not pass the image (or encoded base 64) to the <strong>AI Model</strong> in Canvas Apps because you had just 4-5 built-in controls which were actually not super helpful. Even if you had a requirement to pass an image (via camera stream) to your object detection model, you had to send it to Power Automate and process its response. So whenever I got a chance, I spoke about this limitation with the awesome team at Microsoft and to my fortune (with many others), we didn't have to wait long enough. 🙂

On November 8, at Microsoft Ignite 2021, Lobe &amp; AI Builder team <a href="https://powerapps.microsoft.com/en-us/blog/new-ways-to-add-intelligence-to-your-business-with-ai-builder/">announced</a> fantastic features to add intelligence to your Power Apps. Now, not only you can easily add your model as a data source to your Canvas Apps but also you can use them with PowerFx. This became the reason for me to re-visit my problem which I couldn't solve because of the discussed limitation.
<blockquote>The purpose of this post is to show how you can achieve near real-time object detection using AI Builder models with Power Apps. In case you're interested in building the AI Builder Object Detection model, you may want to check out this <a href="https://www.arafattehsin.com/beginners-lets-build-covid-safe-solutions-with-ai-power-platform/">post</a> where I have discussed its process step by step.</blockquote>
<h1>Object detection using Camera Stream</h1>
The capability of using the AI into your apps with Power Fx allows you to actually access your AI models by any control, be it a Gallery, DataTable and so on. Therefore, we will try to build (or mimic) the same experience as I did for my .NET app. Not because I couldn't think of any other idea but because I want to show the limitless capabilities of Power Platform. <em>Yes I know one works offline while other does not but I think you got my point, didn't you?</em>

So I already have got a template app ready which I will be using to add necessary controls to achieve above. 👆
<h2>Camera</h2>
Let's start by adding a <strong>Camera</strong>, set its <code>StreamRate</code> and <code>OnStream</code> properties with a variable that is going to capture the image from the stream. The property we call is <code>captureTrigger</code>

[Video Walkthrough](https://www.arafattehsin.com/wp-content/uploads/2022/01/add-camera-to-canvasapps.webm)

<h2>Timer</h2>
Add the <strong>Timer</strong> control to the screen and hide it. Set its <code>Duration</code> to <code>2000</code> and <code>OnTimerEnd</code> should have <code>Set(captureTrigger,100);</code>which would set the trigger's value to 100 after every 2 seconds to take a picture from the camera.

[Video Walkthrough](https://www.arafattehsin.com/wp-content/uploads/2022/01/timer.webm)

Note: Do not forget to add toggle <code>Repeat</code> and <code>Auto start</code>. I missed this in the video.
<h2>Gallery</h2>
Now, at the same position (X and Y axes), add the <strong>Vertical Gallery </strong>and align its <code>height</code> and <code>width</code> similar to the Camera. The reason for this is that Gallery will be used to create <code>Bounding Boxes</code> on top of the Camera control. Change the <code>TemplateSize</code> to <code>0</code> otherwise it is going stack the items vertically one by one (if there are more than one object identified in an image). Delete all the items inside your Gallery as we're going to add some new controls here.

Lastly, add go to Add Data and add your AI Model, set the <code>Items</code> property of a Gallery to <code>'Your Model Name'.Predict(YourCameraControl.Stream).results</code> - that's the beauty of using your AI Model along with PowerFx in any control.

[Video Walkthrough](https://www.arafattehsin.com/wp-content/uploads/2022/01/add-gallery-and-data-source.webm)

Add a <code>Container</code> with <code>Text Label</code> and <code>Rectangle</code> inside your gallery control.
<h4><strong>Container</strong></h4>
<ul>
 	<li>Change its <code>X</code> value to <code>ThisItem.boundingBox.left * YourCameraControl.Width</code></li>
 	<li>Change its <code>Y</code> value to <code>ThisItem.boundingBox.top * YourCameraControl.Height</code></li>
 	<li>Change its <code>Width</code> value to <code>ThisItem.boundingBox.width * YourCameraControl.Width</code></li>
 	<li>Change its <code>Height</code> value to <code>ThisItem.boundingBox.height * YourCameraControl.Height</code></li>
</ul>
<h4><strong>Text Label</strong></h4>
<ul>
 	<li>Choose your choice of <code>Color</code> with the <code>Border</code> and set its value to <code>1</code></li>
 	<li>Change its <code>PaddingTop</code> to <code>1</code></li>
 	<li>Change its <code>Height</code> to something of your own choice, I kept it as <code>28</code></li>
 	<li>Keep the <code>Width</code> as <code>Parent.Width</code></li>
 	<li>Also, please ensure you set <code>X</code> and <code>Y</code> both to <code>0</code></li>
 	<li>Set its <code>Text</code> property to <code>ThisItem.tagName</code></li>
</ul>
<h4><strong>Rectangle</strong></h4>
<ul>
 	<li>Choose your choice of <code>BorderColor</code> with the <code>Border</code> and set its value to <code>8</code></li>
 	<li>Keep the <code>Height</code> as <code>Parent.Height</code></li>
 	<li>Keep the <code>Width</code> as <code>Parent.Width</code></li>
</ul>
Note: Please do not set the <code>Color</code> property, just set the color of the <code>Border</code> instead.
<blockquote>You may notice a bit of a lag it is because unlike .NET (or any other framework which runs offline), this is being run online and it takes a second or two to get the response back.</blockquote>
<h3>Bounding Boxes (Bonus Control)</h3>
There's a bonus custom control for you to use instead of the above discussed Gallery and that is <code>Bounding Boxes</code>.
<h4><strong>Motivation</strong></h4>
Few days back I had a chat with Jake Cohen and Bill Barnes from AI Builder &amp; Lobe Team at Microsoft on the above topic and we discussed a couple of options to achieve that. During our discussion, Bill went ahead and instead of Gallery approach, he came up with a smart idea of using a <code>HtmlText</code> instead. He then converted <code>HtmlText</code> into a PowerApps Component called <strong>Bounding Boxes</strong>. As the name suggests, this custom component exposes a property called Boxes which actually takes a Bounding Box of an AI Model as an argument and shows the bounding box on the screen. The best part of this control is its weight which is not heavy as the Gallery and simple for anyone to use.

[Video Walkthrough](https://www.arafattehsin.com/wp-content/uploads/2022/01/bounding-box.webm)

The good news is, Bill has just shared his <a href="https://github.com/billba/PowerApps-AI-Components">repo</a> with me so you can have this (and several other controls) added to your Canvas Apps easily. Here's how you add the Bounding Boxes to your application. 👆

Add the Component to your app and then set the Boxes property of the component to <span style="color: #222222; font-family: monospace;"><span style="font-size: 11.7px; background-color: #e9ebec;">'Your Model Name'.Predict(StreamCamera.Stream).results</span></span> so after every 2 seconds, it will start showing the bounding box as they are processed by the model.

As I discussed above that the purpose of this post is to show you the limitless capabilities of AI Builder. It is now up to your business needs where you really want to use this, for example, as a business owner of a store, you'd like to keep a periodic (~10 minutes) check on people wearing masks within a location, counting the number of allowed people in a café and so on. In case you get stuck with whatever I have shown you here - please feel free to comment down and I'll be happy to assist you.

Until next time.