[Huawei Developers] ProManojKumar:
Overview

In this article, I will create a demo app along with the integration of HMS ML Kit which based on Cross-platform Technology Xamarin. User can easily scan any items from this application with camera Product Vision Search ML Kit technique and choose best price and details of product.

Service Introduction

HMS ML Kit allows your apps to easily leverage Huawei's long-term proven expertise in machine learning to support diverse artificial intelligence (AI) applications throughout a wide range of industries.

A user can take a photo of a product. Then the Product Visual Search service searches for the same or similar products in the pre-established product image library and returns the IDs of those products and related information. In addition, to better manage products in real-time, this service supports offline product import, online product addition, deletion, modification, query, and product distribution.

We can capture any kind of image for products to buy or check the price of a product using Machine Learning. It will give the other options so that you can improve your buying skills.

Prerequisite

1. Xamarin Framework

2. Huawei phone

3. Visual Studio 2019

App Gallery Integration process

1. Sign In and Create or Choose a project on AppGallery Connect portal.

2. Add SHA-256 key.

3. Navigate to Project settings and download the configuration file.

4. Navigate to General Information, and then provide Data Storage location.

5. Navigate to Manage APIs and enable APIs which require by application.

Xamarin ML Kit Setup Process

1. Download Xamarin Plugin all the aar and zip files from below url:

https://developer.huawei.com/consum...Library-V1/xamarin-plugin-0000001053510381-V1

2. Open the XHms-ML-Kit-Library-Project.sln solution in Visual Studio.

3. Navigate to Solution Explore and right-click on jar Add > Exsiting Item and choose aar file which download in Step 1.

4. Right click on added aar file then choose Properties > Build Action > LibraryProjectZip

Note: Repeat Step 3 & 4 for all aar file.

5. Build the Library and make dll files.

Xamarin App Development

1. Open Visual Studio 2019 and Create A New Project.

2. Navigate to Solution Explore > Project > Assets > Add Json file.

3. Navigate to Solution Explore > Project > Add > Add New Folder.

4. Navigate to Folder(created) > Add > Add Existing and add all DLL files.

5. Select all DLL files.

6. Right-click on Properties, choose Build Action > None.

7. Navigate to Solution Explore > Project > Reference > Right Click > Add References, then navigate to Browse and add all DLL files from recently added folder.

8. Added reference, then click OK.

ML Product Visual Search API Integration

1. Create an analyzer for product visual search. You can create the analyzer using the MLRemoteProductVisionSearchAnalyzerSetting class.

// Method 1: Use default parameter settings.
MLRemoteProductVisionSearchAnalyzer analyzer = MLAnalyzerFactory.Instance.RemoteProductVisionSearchAnalyzer;

// Method 2: Use customized parameter settings.
MLRemoteProductVisionSearchAnalyzerSetting settings = new MLRemoteProductVisionSearchAnalyzerSetting.Factory()

// Set the maximum number of products that can be returned.
.SetLargestNumOfReturns(16)
.Create();

MLRemoteProductVisionSearchAnalyzer analyzer = MLAnalyzerFactory.Instance.GetRemoteProductVisionSearchAnalyzer(settings);

2. Create an MLFrame object by using Android.Graphics.Bitmap. JPG, JPEG, PNG, and BMP images are supported.

// Create an MLFrame object using the bitmap, which is the image data in bitmap format.
MLFrame frame = MLFrame.FromBitmap(bitmap);

3. Implement image detection.

Task<IList<MLProductVisionSearch>> task = this.analyzer.AnalyseFrameAsync(frame);
await task;
if (task.IsCompleted && task.Result != null)
{
// Analyze success.
var productVisionSearchList = task.Result;
if (productVisionSearchList.Count != 0)
{
//Product detected successfully
}
else
{
//Product not found
}
}

4. After the recognition is complete, stop the analyzer to release recognition resources.

if (analyzer != null) {

analyzer.Stop();

}

ProductVisionSearchAnalyseActivity.cs

This activity performs all the operation regarding product search with camera.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Android;
using Android.App;
using Android.Content;
using Android.Content.PM;
using Android.Graphics;
using Android.OS;
using Android.Runtime;
using Android.Util;
using Android.Views;
using Android.Widget;
using AndroidX.AppCompat.App;
using AndroidX.Core.App;
using AndroidX.Core.Content;
using Com.Huawei.Hms.Mlplugin.Productvisionsearch;
using Com.Huawei.Hms.Mlsdk;
using Com.Huawei.Hms.Mlsdk.Common;
using Com.Huawei.Hms.Mlsdk.Productvisionsearch;
using Com.Huawei.Hms.Mlsdk.Productvisionsearch.Cloud;
using Java.Lang;

namespace HmsXamarinMLDemo.MLKitActivities.ImageRelated.ProductVisionSearch
{
[Activity(Label = "ProductVisionSearchAnalyseActivity")]
public class ProductVisionSearchAnalyseActivity : AppCompatActivity, View.IOnClickListener
{
private const string Tag = "ProductVisionSearchTestActivity";

private static readonly int PermissionRequest = 0x1000;
private int CameraPermissionCode = 1;
private static readonly int MaxResults = 1;

private TextView mTextView;
private ImageView productResult;
private Bitmap bitmap;
private MLRemoteProductVisionSearchAnalyzer analyzer;

protected override void OnCreate(Bundle savedInstanceState)
{
base.OnCreate(savedInstanceState);

this.SetContentView(Resource.Layout.activity_image_product_vision_search_analyse);
this.mTextView = (TextView)this.FindViewById(Resource.Id.result);
this.productResult = (ImageView)this.FindViewById(Resource.Id.image_product);
this.bitmap = BitmapFactory.DecodeResource(this.Resources, Resource.Drawable.custom_model_image);
this.productResult.SetImageResource(Resource.Drawable.custom_model_image);
this.FindViewById(Resource.Id.product_detect_plugin).SetOnClickListener(this);
this.FindViewById(Resource.Id.product_detect).SetOnClickListener(this);
// Checking Camera Permissions
if (!(ActivityCompat.CheckSelfPermission(this, Manifest.Permission.Camera) == Permission.Granted))
{
this.RequestCameraPermission();
}
}

private void RequestCameraPermission()
{
string[] permissions = new string[] { Manifest.Permission.Camera };

if (!ActivityCompat.ShouldShowRequestPermissionRationale(this, Manifest.Permission.Camera))
{
ActivityCompat.RequestPermissions(this, permissions, this.CameraPermissionCode);
return;
}
}

private void CheckPermissions(string[] permissions)
{
bool shouldRequestPermission = false;
foreach (string permission in permissions)
{
if (ContextCompat.CheckSelfPermission(this, permission) != Permission.Granted)
{
shouldRequestPermission = true;
}
}

if (shouldRequestPermission)
{
ActivityCompat.RequestPermissions(this, permissions, PermissionRequest);
return;
}
StartVisionSearchPluginCapture();
}

private async void RemoteAnalyze()
{
// Use customized parameter settings for cloud-based recognition.
MLRemoteProductVisionSearchAnalyzerSetting setting =
new MLRemoteProductVisionSearchAnalyzerSetting.Factory()
// Set the maximum number of products that can be returned.
.SetLargestNumOfReturns(MaxResults)
.SetProductSetId("vmall")
.SetRegion(MLRemoteProductVisionSearchAnalyzerSetting.RegionDrChina)
.Create();
this.analyzer = MLAnalyzerFactory.Instance.GetRemoteProductVisionSearchAnalyzer(setting);
// Create an MLFrame by using the bitmap.
MLFrame frame = MLFrame.FromBitmap(bitmap);
Task<IList<MLProductVisionSearch>> task = this.analyzer.AnalyseFrameAsync(frame);
try
{
await task;

if (task.IsCompleted && task.Result != null)
{
// Analyze success.
var productVisionSearchList = task.Result;
if(productVisionSearchList.Count != 0)
{
Toast.MakeText(this, "Product detected successfully", ToastLength.Long).Show();
this.DisplaySuccess(productVisionSearchList);
}
else
{
Toast.MakeText(this, "Product not found", ToastLength.Long);
}

}
else
{
// Analyze failure.
Log.Debug(Tag, " remote analyze failed");
}
}
catch (System.Exception e)
{
// Operation failure.
this.DisplayFailure(e);
}
}

private void StartVisionSearchPluginCapture()
{
// Set the config params.
MLProductVisionSearchCaptureConfig config = new MLProductVisionSearchCaptureConfig.Factory()
//Set the largest OM detect Result,default is 20，values in 1-100
.SetLargestNumOfReturns(16)
//Set the fragment you created (the fragment should implement AbstractUIExtendProxy)
.SetProductFragment(new ProductFragment())
//Set region，current values：RegionDrChina，RegionDrSiangapore，RegionDrGerman，RegionDrRussia
.SetRegion(MLProductVisionSearchCaptureConfig.RegionDrChina)
//设set product id，you can get the value by AGC
//.SetProductSetId("xxxxx")
.Create();
MLProductVisionSearchCapture capture = MLProductVisionSearchCaptureFactory.Instance.Create(config);
//Start plugin
capture.StartCapture(this);
}

private void DisplayFailure(System.Exception exception)
{
string error = "Failure. ";
try
{
MLException mlException = (MLException)exception;
error += "error code: " + mlException.ErrCode + "\n" + "error message: " + mlException.Message;
}
catch (System.Exception e)
{
error += e.Message;
}
this.mTextView.Text = error;
}

private void DrawBitmap(ImageView imageView, Rect rect, string product)
{
Paint boxPaint = new Paint();
boxPaint.Color = Color.White;
boxPaint.SetStyle(Paint.Style.Stroke);
boxPaint.StrokeWidth = (4.0f);
Paint textPaint = new Paint();
textPaint = new Paint();
textPaint.Color = Color.White;
textPaint.TextSize = 100.0f;

imageView.DrawingCacheEnabled = true;
Bitmap bitmapDraw = Bitmap.CreateBitmap(this.bitmap.Copy(Bitmap.Config.Argb8888, true));
Canvas canvas = new Canvas(bitmapDraw);
canvas.DrawRect(rect, boxPaint);
canvas.DrawText("product type: " + product, rect.Left, rect.Top, textPaint);
this.productResult.SetImageBitmap(bitmapDraw);
}

private void DisplaySuccess(IList<MLProductVisionSearch> productVisionSearchList)
{
List<MLVisionSearchProductImage> productImageList = new List<MLVisionSearchProductImage>();
foreach (MLProductVisionSearch productVisionSearch in productVisionSearchList)
{
this.DrawBitmap(this.productResult, productVisionSearch.Border, productVisionSearch.Type);
foreach (MLVisionSearchProduct product in productVisionSearch.ProductList)
{
productImageList.AddRange(product.ImageList);
}
}
StringBuffer buffer = new StringBuffer();
foreach (MLVisionSearchProductImage productImage in productImageList)
{
string str = "ProductID: " + productImage.ProductId + "\nImageID: " + productImage.ImageId + "\nPossibility: " + productImage.Possibility;
buffer.Append(str);
buffer.Append("\n");
}

this.mTextView.Text = buffer.ToString();

this.bitmap = BitmapFactory.DecodeResource(this.Resources, Resource.Drawable.custom_model_image);
this.productResult.SetImageResource(Resource.Drawable.custom_model_image);
}

public void OnClick(View v)
{
switch (v.Id)
{
case Resource.Id.product_detect:
this.RemoteAnalyze();
break;
case Resource.Id.product_detect_plugin:
CheckPermissions(new string[]{Manifest.Permission.Camera, Manifest.Permission.ReadExternalStorage,
Manifest.Permission.WriteExternalStorage, Manifest.Permission.AccessNetworkState});
break;
default:
break;
}
}

protected override void OnDestroy()
{
base.OnDestroy();
if (this.analyzer == null)
{
return;
}
this.analyzer.Stop();
}
}

}

Xamarin App Build

1. Navigate to Solution Explore > Project > Right Click > Archive/View Archive to generate SHA-256 for build release and Click on Distribute.

2. Choose Distribution Channel > Ad Hoc to sign apk.

3. Choose Demo Keystore to release apk.

4. Finally here is the Result.

Tips and Tricks

1. HUAWEI ML Kit complies with GDPR requirements for data processing.

2. HUAWEI ML Kit does not support the recognition of the object distance and colour.

3. Images in PNG, JPG, JPEG, and BMP formats are supported. GIF images are not supported.

Conclusion

In this article, we have learned how to integrate HMS ML Kit in Xamarin based Android application. User can easily search items online with the help of product visual search API in this application.

Thanks for reading this article. Be sure to like and comments to this article, if you found it helpful. It means a lot to me.

References

https://developer.huawei.com/consum...-Plugin-Guides/about-service-0000001052602130