[Huawei Developers] ProManojKumar:
Overview

In this article, I will create a demo app along with the integration of HMS ML Kit which based on Cross platform Technology Xamarin. User can easily scan any objects from this application with camera using Object Detection and Tracking API and choose best price and details of object. The following object categories are supported: household products, fashion goods, food, places, plants, faces, and others.

Service Introduction

HMS ML Kit allows your apps to easily leverage Huawei's long-term proven expertise in machine learning to support diverse artificial intelligence (AI) applications throughout a wide range of industries.

A user can take a photo of an Object through camera or gallery. Then the Object Detection and Tracking service searches for the same or similar objects in the pre-established object image library and returns the IDs of those object and related information.

We can capture or choose from gallery any kind object based image to buy or check the price of an object using Machine Learning. It will give the other options, so that you can improve your buying skills.

Prerequisite

1. Xamarin Framework

2. Huawei phone

3. Visual Studio 2019

App Gallery Integration process

1. Sign In and Create or Choose a project on AppGallery Connect portal.

2. Add SHA-256 key.

3. Navigate to Project settings and download the configuration file.

4. Navigate to General Information, and then provide Data Storage location.

5. Navigate to Manage APIs and enable APIs which require by application.

Xamarin ML Kit Setup Process

1. Download Xamarin Plugin all the aar and zip files from below url:

https://developer.huawei.com/consum...Library-V1/xamarin-plugin-0000001053510381-V1

2. Open the XHms-ML-Kit-Library-Project.sln solution in Visual Studio.

3. Navigate to Solution Explore and right-click on jar Add > Exsiting Item and choose aar file which download in Step 1.

4. Right click on added aar file then choose Properties > Build Action > LibraryProjectZip

Note: Repeat Step 3 & 4 for all aar file.

5. Build the Library and make dll files.

Xamarin App Development

1. Open Visual Studio 2019 and Create A New Project.

2. Navigate to Solution Explore > Project > Assets > Add Json file.

3. Navigate to Solution Explore > Project > Add > Add New Folder.

4. Navigate to Folder(created) > Add > Add Existing and add all DLL files.

5. Select all DLL files.

6. Right-click on Properties, choose Build Action > None.

7. Navigate to Solution Explore > Project > Reference > Right Click > Add References, then navigate to Browse and add all DLL files from recently added folder.

8. Added reference, then click OK.

ML Object Detection and Tracking API Integration

Camera stream detection

You can process camera streams, convert video frames into an MLFrame object, and detect objects using the static image detection method. If the synchronous detection API is called, you can also use the LensEngine class built in the SDK to detect objects in camera streams. The sample code is as follows:

1. Create an object analyzer.

1

2

3

4

5

6

7

8	// Create an object analyzer

// Use MLObjectAnalyzerSetting.TypeVideo for video stream detection.

// Use MLObjectAnalyzerSetting.TypePicture for static image detection.

MLObjectAnalyzerSetting setting = new MLObjectAnalyzerSetting.Factory().SetAnalyzerType(MLObjectAnalyzerSetting.TypeVideo)

.AllowMultiResults()

.AllowClassification()

.Create();

analyzer = MLAnalyzerFactory.Instance.GetLocalObjectAnalyzer(setting);

2. Create the ObjectAnalyzerTransactor class for processing detection results. This class implements the MLAnalyzer.IMLTransactor API and uses the TransactResult method in this API to obtain the detection results and implement specific services.

1

2

3

4

5

6

7

8

9

10

11

12	public class ObjectAnalyseMLTransactor : Java.Lang.Object, MLAnalyzer.IMLTransactor

{

public void Destroy()

{

}

public void TransactResult(MLAnalyzer.Result results)

{

SparseArray objectSparseArray = results.AnalyseList;

}

}

3. Set the detection result processor to bind the analyzer to the result processor.

1	analyzer.SetTransactor(new ObjectAnalyseMLTransactor());

4. Create an instance of the LensEngine class provided by the HMS Core ML SDK to capture dynamic camera streams and pass the streams to the analyzer.

1

2

3

4

5

6

7	Context context = this.ApplicationContext;

// Create LensEngine

LensEngine lensEngine = new LensEngine.Creator(context, this.analyzer).SetLensType(this.lensType)

.ApplyDisplayDimension(640, 480)

.ApplyFps(25.0f)

.EnableAutomaticFocus(true)

.Create();

5. Call the run method to start the camera and read camera streams for detection.

1

2

3

4

5

6

7

8

9

10

11

12	if (lensEngine != null)

{

try

{

preview.start(lensEngine , overlay);

}

catch (Exception e)

{

lensEngine .Release();

lensEngine = null;

}

}

6. After the detection is complete, stop the analyzer to release detection resources.

1

2

3

4

5

6	if (analyzer != null) {

analyzer.Stop();

}

if (lensEngine != null) {

lensEngine.Release();

}

LiveObjectAnalyseActivity.cs

This activity performs all the operation regarding object detecting and tracking with camera.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280	using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using Android;

using Android.App;

using Android.Content;

using Android.Content.PM;

using Android.OS;

using Android.Runtime;

using Android.Support.V4.App;

using Android.Support.V7.App;

using Android.Util;

using Android.Views;

using Android.Widget;

using Com.Huawei.Hms.Mlsdk;

using Com.Huawei.Hms.Mlsdk.Common;

using Com.Huawei.Hms.Mlsdk.Objects;

using HmsXamarinMLDemo.Camera;

namespace HmsXamarinMLDemo.MLKitActivities.ImageRelated.Object

{

[Activity(Label = "LiveObjectAnalyseActivity")]

public class LiveObjectAnalyseActivity : AppCompatActivity, View.IOnClickListener

{

private const string Tag = "LiveObjectAnalyseActivity";

private const int CameraPermissionCode = 1;

public const int StopPreview = 1;

public const int StartPreview = 2;

private MLObjectAnalyzer analyzer;

private LensEngine mLensEngine;

private bool isStarted = true;

private LensEnginePreview mPreview;

private GraphicOverlay mOverlay;

private int lensType = LensEngine.BackLens;

public bool mlsNeedToDetect = true;

public ObjectAnalysisHandler mHandler;

protected override void OnCreate(Bundle savedInstanceState)

{

base.OnCreate(savedInstanceState);

this.SetContentView(Resource.Layout.activity_live_object_analyse);

if (savedInstanceState != null)

{

this.lensType = savedInstanceState.GetInt("lensType");

}

this.mPreview = (LensEnginePreview)this.FindViewById(Resource.Id.object_preview);

this.mOverlay = (GraphicOverlay)this.FindViewById(Resource.Id.object_overlay);

this.CreateObjectAnalyzer();

this.FindViewById(Resource.Id.detect_start).SetOnClickListener(this);

mHandler = new ObjectAnalysisHandler(this);

// Checking Camera Permissions

if (ActivityCompat.CheckSelfPermission(this, Manifest.Permission.Camera) == Permission.Granted)

{

this.CreateLensEngine();

}

else

{

this.RequestCameraPermission();

}

}

//Request permission

private void RequestCameraPermission()

{

string[] permissions = new string[] { Manifest.Permission.Camera };

if (!ActivityCompat.ShouldShowRequestPermissionRationale(this, Manifest.Permission.Camera))

{

ActivityCompat.RequestPermissions(this, permissions, CameraPermissionCode);

return;

}

}

/// <summary>

/// Start Lens Engine on OnResume() event.

/// </summary>

protected override void OnResume()

{

base.OnResume();

this.StartLensEngine();

}

/// <summary>

/// Stop Lens Engine on OnPause() event.

/// </summary>

protected override void OnPause()

{

base.OnPause();

this.mPreview.stop();

}

/// <summary>

/// Stop analyzer on OnDestroy() event.

/// </summary>

protected override void OnDestroy()

{

base.OnDestroy();

if (this.mLensEngine != null)

{

this.mLensEngine.Release();

}

if (this.analyzer != null)

{

try

{

this.analyzer.Stop();

}

catch (Exception e)

{

Log.Info(LiveObjectAnalyseActivity.Tag, "Stop failed: " + e.Message);

}

}

}

public override void OnRequestPermissionsResult(int requestCode, string[] permissions, [GeneratedEnum] Permission[] grantResults)

{

if (requestCode != LiveObjectAnalyseActivity.CameraPermissionCode)

{

base.OnRequestPermissionsResult(requestCode, permissions, grantResults);

return;

}

if (grantResults.Length != 0 && grantResults[0] == Permission.Granted)

{

this.CreateLensEngine();

return;

}

}

protected override void OnSaveInstanceState(Bundle outState)

{

outState.PutInt("lensType", this.lensType);

base.OnSaveInstanceState(outState);

}

private void StopPreviewAction()

{

this.mlsNeedToDetect = false;

if (this.mLensEngine != null)

{

this.mLensEngine.Release();

}

if (this.analyzer != null)

{

try

{

this.analyzer.Stop();

}

catch (Exception e)

{

Log.Info("object", "Stop failed: " + e.Message);

}

}

this.isStarted = false;

}

private void StartPreviewAction()

{

if (this.isStarted)

{

return;

}

this.CreateObjectAnalyzer();

this.mPreview.release();

this.CreateLensEngine();

this.StartLensEngine();

this.isStarted = true;

}

private void CreateLensEngine()

{

Context context = this.ApplicationContext;

// Create LensEngine

this.mLensEngine = new LensEngine.Creator(context, this.analyzer).SetLensType(this.lensType)

.ApplyDisplayDimension(640, 480)

.ApplyFps(25.0f)

.EnableAutomaticFocus(true)

.Create();

}

private void StartLensEngine()

{

if (this.mLensEngine != null)

{

try

{

this.mPreview.start(this.mLensEngine, this.mOverlay);

}

catch (Exception e)

{

Log.Info(LiveObjectAnalyseActivity.Tag, "Failed to start lens engine.", e);

this.mLensEngine.Release();

this.mLensEngine = null;

}

}

}

public void OnClick(View v)

{

this.mHandler.SendEmptyMessage(LiveObjectAnalyseActivity.StartPreview);

}

private void CreateObjectAnalyzer()

{

// Create an object analyzer

// Use MLObjectAnalyzerSetting.TypeVideo for video stream detection.

// Use MLObjectAnalyzerSetting.TypePicture for static image detection.

MLObjectAnalyzerSetting setting =

new MLObjectAnalyzerSetting.Factory().SetAnalyzerType(MLObjectAnalyzerSetting.TypeVideo)

.AllowMultiResults()

.AllowClassification()

.Create();

this.analyzer = MLAnalyzerFactory.Instance.GetLocalObjectAnalyzer(setting);

this.analyzer.SetTransactor(new ObjectAnalyseMLTransactor(this));

}

public class ObjectAnalysisHandler : Android.OS.Handler

{

private LiveObjectAnalyseActivity liveObjectAnalyseActivity;

public ObjectAnalysisHandler(LiveObjectAnalyseActivity LiveObjectAnalyseActivity)

{

this.liveObjectAnalyseActivity = LiveObjectAnalyseActivity;

}

public override void HandleMessage(Message msg)

{

base.HandleMessage(msg);

switch (msg.What)

{

case LiveObjectAnalyseActivity.StartPreview:

this.liveObjectAnalyseActivity.mlsNeedToDetect = true;

//Log.d("object", "start to preview");

this.liveObjectAnalyseActivity.StartPreviewAction();

break;

case LiveObjectAnalyseActivity.StopPreview:

this.liveObjectAnalyseActivity.mlsNeedToDetect = false;

//Log.d("object", "stop to preview");

this.liveObjectAnalyseActivity.StopPreviewAction();

break;

default:

break;

}

}

}

public class ObjectAnalyseMLTransactor : Java.Lang.Object, MLAnalyzer.IMLTransactor

{

private LiveObjectAnalyseActivity liveObjectAnalyseActivity;

public ObjectAnalyseMLTransactor(LiveObjectAnalyseActivity LiveObjectAnalyseActivity)

{

this.liveObjectAnalyseActivity = LiveObjectAnalyseActivity;

}

public void Destroy()

{

}

public void TransactResult(MLAnalyzer.Result result)

{

if (!liveObjectAnalyseActivity.mlsNeedToDetect) {

return;

}

this.liveObjectAnalyseActivity.mOverlay.Clear();

SparseArray objectSparseArray = result.AnalyseList;

for (int i = 0; i < objectSparseArray.Size(); i++)

{

MLObjectGraphic graphic = new MLObjectGraphic(liveObjectAnalyseActivity.mOverlay, ((MLObject)(objectSparseArray.ValueAt(i))));

liveObjectAnalyseActivity.mOverlay.Add(graphic);

}

// When you need to implement a scene that stops after recognizing specific content

// and continues to recognize after finishing processing, refer to this code

for (int i = 0; i < objectSparseArray.Size(); i++)

{

if (((MLObject)(objectSparseArray.ValueAt(i))).TypeIdentity == MLObject.TypeFood)

{

liveObjectAnalyseActivity.mlsNeedToDetect = true;

liveObjectAnalyseActivity.mHandler.SendEmptyMessage(LiveObjectAnalyseActivity.StopPreview);

}

}

}

}

}

}

Xamarin App Build

1. Navigate to Solution Explore > Project > Right Click > Archive/View Archive to generate SHA-256 for build release and Click on Distribute.

2. Choose Distribution Channel > Ad Hoc to sign apk.

3. Choose Demo Keystore to release apk.

4. Finally here is the Result.

Tips and Tricks

1. HUAWEI ML Kit complies with GDPR requirements for data processing.

2. HUAWEI ML Kit does not support the recognition of the object distance and colour.

3. Images in PNG, JPG, JPEG, and BMP formats are supported. GIF images are not supported.

Conclusion

In this article, we have learned how to integrate HMS ML Kit in Xamarin based Android application. User can easily search objects online with the help of Object Detection and Tracking API in this application.

Thanks for reading this article.

Be sure to like and comments to this article, if you found it helpful. It means a lot to me.

References

https://developer.huawei.com/consum...n-Guides/object-detect-track-0000001052607676