Android Image Classification with TensorFlow Lite & Azure Custom Vision Service
Tags: xamarin, azure, aiapril, android
Published: 2020-04-09 20:59:09

_This is a guest blog by [Daniel Hindrikes](https://twitter.com/hindrikes). Daniel is a Microsoft MVP, Speaker, and Xamarin + Cloud Solution Architect at [tretton37](https://tretton37.com/)_

Image Classification allows our Xamarin apps to recognize objects in a photo.

Being able to take a photo and recognize its contents is becoming more and more common. We experience it in our banking apps when making a mobile deposit, in our photo apps when adding filters, and in our HotDog apps to determine whether or not our meal is a hotdog.

Thanks to Azure Custom Vision Service, we don't have to learn complex machine learning algorithms to implement image classification.

This article will walkthrough how to implement it using Azure's [Custom Vision Service](https://customvision.ai/?WT.mc_id=aiapril-devto-bramin), [TensorFlow Lite](https://www.tensorflow.org/lite) (an open source machine learning platform) and [Xamarin.Android](https://docs.microsoft.com/xamarin/android?WT.mc_id=aiapril-devto-bramin).

> **Note**: For Xamarin.iOS, we can also use Azure's Custom Vision Service with CoreML, but we'll save that for another blog post.

![](https://devblogs.microsoft.com/xamarin/wp-content/uploads/sites/44/2020/03/mushrooms.gif)

## Image Classification Libraries

We'll be leveraging Azure [Custom Vision Service](https://customvision.ai/?WT.mc_id=aiapril-devto-bramin) and [TensorFlow Lite](https://www.tensorflow.org/lite) to implement our image classification.

### 1. Azure Custom Vision Service

Azure's [Custom Vision Service](https://customvision.ai/?WT.mc_id=aiapril-devto-bramin) makes it easy to create and train machine learning models - no previous Artificial Intelligence (AI) or Machine Learning (ML) experience is necessary.

Using the [Custom Vision Service Web Portal](https://customvision.ai/?WT.mc_id=aiapril-devto-bramin), we can do the following without writing any AI/ML code:

1. Upload a training image
2. Tag / Label the object(s) in the image
3. Repeat (the model gets better with more training data)
4. That's it - Custom Vision Service takes care of the rest!

### 2. TensorFlow Lite

[TensorFlow Lite](https://www.tensorflow.org/lite) is an open source machine learning platform that allows us to use [TensorFlow](https://www.tensorflow.org/) on IoT and Mobile devices.

Both TensoryFlow Lite and TensorFlow are completely [open-source on GitHub](https://github.com/tensorflow/tensorflow).

## Implementing Image Classification with Azure + Xamarin.Android
The completed Image Classification sample app is available here on GitHub: https://github.com/dhindrik/MushroomDetector/

### 1. Training the Model

Using the [Custom Vision Service Web Portal](https://customvision.ai/?WT.mc_id=aiapril-devto-bramin), we will first train models for image classification.

1) In the [Custom Vision Service Web Portal](https://customvision.ai/?WT.mc_id=aiapril-devto-bramin), click **New Project**

![New Project](https://devblogs.microsoft.com/xamarin/wp-content/uploads/sites/44/2020/03/New-Project-300x300.png)


2) In the **Create new project window**, make the following selections:

- **Name**: XamarinImageClassification
- **Description**: Identify Objects in Images
- **Resource**: [Create a new resource]
- **Project Type**: Classification
- **Classification Types**: Multilabel (Multiple tags per image)
- **Domains**: General (compact)
- **Export Capabilities**: Basic platforms

3) In the **Create new project window**, click **Create project**

4) In the **XamarinImageClassification** window, click **Add images**

5) Select images that contain the object to identify

6) In the **Image Upload** window, add a tag

> **Note**: In this example, we are tagging images of mushrooms

7) In the Image upload window, click Upload

> **Note**: Continue uploading images until you have at least 5 images for each tag


8) In the **XamarinImageClassification** window, in the top-right of the window, click the **Train Model** button (the green image of gears)

9) In the **Choose Training Type** window, select **Quick Training**

10) In the **Choose Training Type** window, select **Train**


![Choose Training Type](https://devblogs.microsoft.com/xamarin/wp-content/uploads/sites/44/2020/03/Train-300x207.png)

### 2. Export Trained Models from Azure Custom Vision Service

Now that we've trained our model, let's [export it for use in our mobile app](https://docs.microsoft.com/azure/cognitive-services/custom-vision-service/export-your-model?WT.mc_id=aiapril-devto-bramin).

This allows us to use the model without an internet connection which improves the user's privacy because their photos will never leave their mobile device.

To export our model, lets do the following:

1) On the **XamarinImageClassifcation** window, on the top of the page, select the **Performance** tab

2) On The **Performance** tab, tap the **Export** button (the downward-pointing arrow)

3) On the **Choose your platform** window, select **TensorFlow**

![Choose Your Platform](https://devblogs.microsoft.com/xamarin/wp-content/uploads/sites/44/2020/03/Export-to-TensorFlow-300x288.png)

4) In the **Choose your platform** dropdown, select **TensorFlow Lite**

5) In the **Choose your platform** window, select **Download**


![Choose your platform](https://devblogs.microsoft.com/xamarin/wp-content/uploads/sites/44/2020/03/Choose-Your-Platform-300x149.png)

### 3. Import TensorFlowLite into our Xamarin.Android App

1) In our Xamarin.Android project, install the [TensorFlow Lite NuGet Package](https://www.nuget.org/packages/Xamarin.TensorFlow.Lite)

> **Note**: This NuGet Package is an [open-source project](https://github.com/xamarin/XamarinComponents/tree/master/XPlat/TensorFlow.Lite) created by the Xamarin team at Microsoft. It contains C# bindings for the [orginal TensorFlow Lite library](https://github.com/tensorflow/tensorflow) which makes it possible to use in our Xamarin.Android app

![Xamarin.TensorFlow.Lite NuGet Package](https://user-images.githubusercontent.com/13558917/76364288-b24f5880-631c-11ea-9d0a-d817efab4233.png)

2) Unzip the exported model that we downloaded from the [Custom Vision Service Web Portal](https://customvision.ai/?WT.mc_id=aiapril-devto-bramin)

> **Note**: Inside the zip-file, are `labels.txt` and `model.tflite`:
>
> - `labels.txt`  contains the image tags created during the training preparation at the Custom Vision website
> - `models.tflite`  is the machine learning model that we use to make our predictions

3) In Visual Studio, in the Xamarin.Android project, right-click on the **Assets** folder

4) On the right-click menu, select **Add** > **Existing Item...**

5) In the **Add Existing Item** menu, select both of the recently unzipped files:

- models.tflite
- labels.txt

6) In Visual Studio, in the **Xamarin.Android** > **Assets** folder, right-click on **labels.txt**

7) In the right-click menu, select **Properties**

8) In the **Properties** window, select **Build Action** > **Android Asset**

9) In Visual Studio, in the **Xamarin.Android** > **Assets** folder, right-click on **models.tflite**

10) In the right-click menu, select **Properties**

11) In the **Properties** window, select **Build Action** > **Android Asset**

![AndroidAsset](https://user-images.githubusercontent.com/13558917/76364295-b4b1b280-631c-11ea-845f-fe0ff2e0a415.png)

### 4. Implement Image Classification Code for Xamarin.Android

Now that we've imported the model, it is time to write code.

The completed Image Classification sample app is available here on GitHub: [https://github.com/dhindrik/MushroomDetector/](https://github.com/dhindrik/MushroomDetector/)

In the Xamarin.Android project, add both `ImageClassifcationModel.cs`  and `TensorflowClassifier.cs:`

### ImageClassificationModel.cs

```csharp
public class ImageClassificationModel 
{ 
    public ImageClassificationModel(string tagName, float probability) 
    { 
        TagName = tagName; 
        Probability = probability; 
    } 
   
    public float Probability { get; } 
    public string TagName { get; } 
}
```

### TensorflowClassifier.cs

```csharp
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Android.App;
using Android.Graphics;
using Java.IO;
using Java.Nio;
using Java.Nio.Channels;

public class TensorflowClassifier
{
    //FloatSize is a constant with the value of 4 because a float value is 4 bytes
    const int FloatSize = 4;
    //PixelSize is a constant with the value of 3 because a pixel has three color channels: Red Green and Blue
    const int PixelSize = 3;

    public List<ImageClassificationModel> Classify(byte[] image)
    {
        var mappedByteBuffer = GetModelAsMappedByteBuffer();
        var interpreter = new Xamarin.TensorFlow.Lite.Interpreter(mappedByteBuffer);

        //To resize the image, we first need to get its required width and height
        var tensor = interpreter.GetInputTensor(0);
        var shape = tensor.Shape();

        var width = shape[1];
        var height = shape[2];

        var byteBuffer = GetPhotoAsByteBuffer(image, width, height);

        //use StreamReader to import the labels from labels.txt
        var streamReader = new StreamReader(Application.Context.Assets.Open("labels.txt"));

        //Transform labels.txt into List<string>
        var labels = streamReader.ReadToEnd().Split('\n').Select(s => s.Trim()).Where(s => !string.IsNullOrEmpty(s)).ToList();

        //Convert our two-dimensional array into a Java.Lang.Object, the required input for Xamarin.TensorFlow.List.Interpreter
        var outputLocations = new float[1][] { new float[labels.Count] };
        var outputs = Java.Lang.Object.FromArray(outputLocations);

        interpreter.Run(byteBuffer, outputs);
        var classificationResult = outputs.ToArray<float[]>();

        //Map the classificationResult to the labels and sort the result to find which label has the highest probability
        var classificationModelList = new List<ImageClassificationModel>();

        for (var i = 0; i < labels.Count; i++)
        {
            var label = labels[i]; classificationModelList.Add(new ImageClassificationModel(label, classificationResult[0][i]));
        }

        return classificationModelList;
    }

    //Convert model.tflite to Java.Nio.MappedByteBuffer , the require type for Xamarin.TensorFlow.Lite.Interpreter
    private MappedByteBuffer GetModelAsMappedByteBuffer()
    {
        var assetDescriptor = Application.Context.Assets.OpenFd("model.tflite");
        var inputStream = new FileInputStream(assetDescriptor.FileDescriptor);

        var mappedByteBuffer = inputStream.Channel.Map(FileChannel.MapMode.ReadOnly, assetDescriptor.StartOffset, assetDescriptor.DeclaredLength);

        return mappedByteBuffer;
    }

    //Resize the image for the TensorFlow interpreter
    private ByteBuffer GetPhotoAsByteBuffer(byte[] image, int width, int height)
    {
        var bitmap = BitmapFactory.DecodeByteArray(image, 0, image.Length);
        var resizedBitmap = Bitmap.CreateScaledBitmap(bitmap, width, height, true);

        var modelInputSize = FloatSize * height * width * PixelSize;
        var byteBuffer = ByteBuffer.AllocateDirect(modelInputSize);
        byteBuffer.Order(ByteOrder.NativeOrder());

        var pixels = new int[width * height];
        resizedBitmap.GetPixels(pixels, 0, resizedBitmap.Width, 0, 0, resizedBitmap.Width, resizedBitmap.Height);

        var pixel = 0;

        //Loop through each pixels to create a Java.Nio.ByteBuffer
        for (var i = 0; i < width; i++)
        {
            for (var j = 0; j < height; j++)
            {
                var pixelVal = pixels[pixel++];

                byteBuffer.PutFloat(pixelVal >> 16 & 0xFF);
                byteBuffer.PutFloat(pixelVal >> 8 & 0xFF);
                byteBuffer.PutFloat(pixelVal & 0xFF);
            }
        }

        bitmap.Recycle();

        return byteBuffer;
    }
}
```

That's it! Now we can pass an image into `TensorflowClassifier.Classify` to retrieve our `ImageClassificationModel`.

## Learn More

Check out the links below:

- [Completed Code on GitHub](https://github.com/dhindrik/MushroomDetector/blob/master/MushroomDetector.Android/TensorflowClassifier.cs)
- [Azure Custom Vision Service](https://customvision.ai/?WT.mc_id=aiapril-devto-bramin)
- [Azure Cognitive Services](https://docs.microsoft.com/azure/cognitive-services?WT.mc_id=aiapril-devto-bramin)
- [The Community AI Show: Custom Vision, Xamarin and Simpsons](https://www.youtube.com/watch?v=pRDL5BtV_Fg&feature=youtu.be)
- [Microsoft's AI April](https://medium.com/microsoftazure/ai-april-content-dashboar-8e8cd6e8aa98)
- [App In The Cloud - Build an app powered by Custom Vision in Azure Cognitive Services](https://www.youtube.com/watch?v=yDqvOsMRceo&feature=youtu.be)