[Huawei Developers] John Wo:
About This Document

The COVID-19 outbreak has changed the shopping habits of consumers. A great proportion of consumers are turning to live streaming e-commerce, a trend that is currently enjoying great success. To achieve a high transaction volume, a live streamer needs to spend a lot of time preparing information about products, discounts, and usage tips. The background used in the live streaming room is another deciding factor. A background that doesn't change is likely to bore the audience, especially when their interest in the products is weak, and cause them to leave the live stream. Unless the live streamer is incredibly charismatic, it's very difficult to keep the audience interested in all products being advertised. As a result, the audience size may decrease during the course of the live stream.

With the image segmentation service launched by HUAWEI ML Kit, various static and dynamic backgrounds are available and can be changed in real time according to product categories or custom requirements, making live streaming more lively and interesting. This service can crop out the portrait of the live streamer based on semantic segmentation. For example, when introducing a household product, the live streamer can change to a home background in real time. This helps give the audience a more immersive shopping experience.

Functional Description

This demo is developed based on the image segmentation and hand keypoint detection services launched by HUAWEI ML Kit, and demonstrates the function of allowing the user to change the background by waving his/her hand. To avoid misoperations, the background only changes when the hand waving motion reaches a certain threshold. Once custom backgrounds are loaded, the user can wave their hand over the phone from right to left to go back to the last background, or wave their hand from left to right to go to the next background. Dynamic backgrounds are also supported. If you want to allow users to change the background with custom hand gestures or implement other hand gesture effects, you can integrate HUAWEI ML Kit's hand keypoint detection service to develop such features.

Now, let's take a detailed look at the development process.

Development Process

1. Add the AppGallery Connect plug-in and the Maven repository.

buildscript {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
dependencies {
...
classpath 'com.huawei.agconnect:agcp:1.4.1.300'
}
}

allprojects {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
}

2. Perform integration in full SDK mode.

dependencies{
// Import the base SDK for image segmentation.
implementation 'com.huawei.hms:ml-computer-vision-segmentation:2.0.4.300'
// Import the multiclass segmentation model package.
implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-multiclass-model:2.0.4.300'
// Import the portrait segmentation model package.
implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-body-model:2.0.4.300'
// Import the base SDK for gesture detection.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.4.300'
// Import the hand keypoint detection model package.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.4.300'
}

3. Add configurations to the file header.

Add apply plugin: 'com.huawei.agconnect' after apply plugin: 'com.android.application'.

4. Add the function of automatically updating the machine learning model.

Add the following configurations to the AndroidManifest.xml file:
<manifest
...
<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value="imgseg,handkeypoint" />
...
</manifest>

5. Create an image segmentation analyzer.

MLImageSegmentationAnalyzer imageSegmentationAnalyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer(); // Image segmentation analyzer.
MLHandKeypointAnalyzer handKeypointAnalyzer = MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer(); // Gesture detection analyzer.

MLCompositeAnalyzer analyzer = new MLCompositeAnalyzer.Creator()
.add(imageSegmentationAnalyzer)
.add(handKeypointAnalyzer)
.create();

6. Create a detection result processing class.

public class ImageSegmentAnalyzerTransactor implements MLAnalyzer.MLTransactor<MLImageSegmentation> {
@Override
public void transactResult(MLAnalyzer.Result<MLImageSegmentation> results) {
SparseArray<MLImageSegmentation> items = results.getAnalyseList();
// Process the detection result as required.
// Other detection-related APIs provided by ML Kit cannot be called.
}
@Override
public void destroy() {
// Callback method used to release resources when detection completes.
}
}

public class HandKeypointTransactor implements MLAnalyzer.MLTransactor<List<MLHandKeypoints>> {
@Override
public void transactResult(MLAnalyzer.Result<List<MLHandKeypoints>> results) {
SparseArray<List<MLHandKeypoints>> analyseList = results.getAnalyseList();
// Process the detection result as required.
// Other detection-related APIs provided by ML Kit cannot be called.
}
@Override
public void destroy() {
// Callback method used to release resources when detection completes.
}
}

7. Set the detection result processor, and bind it to the analyzer.

imageSegmentationAnalyzer.setTransactor(new ImageSegmentAnalyzerTransactor());
handKeypointAnalyzer.setTransactor(new HandKeypointTransactor());

8. Create a LensEngine instance.

Context context = this.getApplicationContext();
LensEngine lensEngine = new LensEngine.Creator(context,analyzer)
// Use the front- or rear-facing camera. LensEngine.BACK_LENS indicates the rear-facing camera, and LensEngine.FRONT_LENS indicates the front-facing camera.
.setLensType(LensEngine.FRONT_LENS)
.applyDisplayDimension(1280, 720)
.applyFps(20.0f)
.enableAutomaticFocus(true)
.create();

9. Start the camera, read video streams, and perform detection.

// Implement other logic of the SurfaceView control by yourself.
SurfaceView mSurfaceView = new SurfaceView(this);
try {
lensEngine.run(mSurfaceView.getHolder());
} catch (IOException e) {
// Handle exceptions.
}

10. After the detection is complete, stop the analyzer, and release detection resources.

if (analyzer != null) {
try {
analyzer.stop();
} catch (IOException e) {
// Handle exceptions.
}
}
if (lensEngine != null) {
lensEngine.release();
}

Summary

ML Kit's background changing function can be quickly developed by importing packages, performing gesture detection, analyzing, and processing the detection result. Performing image segmentation allows you to do even more. For example, you can apply front-end rendering technologies to prevent bullet comments from covering the people on the screen or use graphical assets to create exquisite images of various sizes. With semantic segmentation, you can accurately segment any object from an image. In addition to portraits, food items, pets, buildings, landscapes, flowers, and grasses can also be segmented. With these functions integrated in your app, users no longer need to use professional photo editing software.

Reference

Official website of Huawei Developers

Development Guide

HMS Core official community on Reddit

Demo and sample code

Discussions on Stack Overflow