[Huawei Developers] John Wo:
Introduction:

ML Kit provides a hand keypoint detection capability, which can be used to detect sign language. Handle Key point detection currently provides 21 points in the hand to detect the signs. Here we will use the direction of each finger and compare that with ASL rules to find the alphabet.

Application scenarios:

Sign language can used by deaf and speech impaired people. Sign language is a collection of hand gestures involving motions and signs which are used in daily interaction.

Using ML kit you can create an intelligent signed alphabet recognizer can work as an aiding agent to translate the signs to words (and also sentences) and vice versa.

What I am trying here is American Sign Language (ASL) alphabets from hand gesture. The classification is based on the position of the joints, fingers and wrists. I have attempted to gather alphabets for the word “HELLO” from hand gestures.

[IMG="https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/745/308/2640091000001745308.20201110024427.33675809747260465062288565177882:50511125025920:2800:5D13DDFC95C842C1B920608206854A36370AB9BD671E18656F9A323EC6B59FA0.png"]https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/745/308/2640091000001745308.20201110024427.33675809747260465062288565177882:50511125025920:2800:5D13DDFC95C842C1B920608206854A36370AB9BD671E18656F9A323EC6B59FA0.png[/IMG]

Development Practice

1. Preparations

You can find detailed information about the preparations you need to make on the HUAWEI Developers-Development Process URL https://developer.huawei.com/consumer/en/doc/development/HMS-Guides/ml-process-4

Here, we will just look at the most important procedures.

Step 1 Enable ML Kit

In HUAWEI Developer AppGallery Connect, choose Develop > Manage APIs. Make sure ML Kit is activated

Step 2 Configure the Maven Repository Address in the Project-Level build.gradle File

buildscript {
repositories {
...
maven {url 'https://developer.huawei.com/repo/'}
}
}
dependencies {
...
classpath 'com.huawei.agconnect:agcp:1.3.1.301'
}
allprojects {
repositories {
...
maven {url 'https://developer.huawei.com/repo/'}
}
}

Step 3 Add SDK Dependencies to the App-Level build.gradle File

apply plugin: 'com.android.application'
apply plugin: 'com.huawei.agconnect'

dependencies{
// Import the base SDK.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.2.300'
// Import the hand keypoint detection model package.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.2.300'
}

Step 4 Add related meta-data tags into your AndroidManifest.xml.

<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value= "handkeypoint"/>

Step 5 Apply for Camera Permission and Local File Reading Permission

<!--Camera permission-->
<uses-permission android:name="android.permission.CAMERA" />
<!--Read permission-->
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

2. Coding Steps:

Step 1 Create SurfaceView for Camera Preview and also create surfaceView for the result.

At present we are showing theresult only in the UI, but you can extend and read the result using the TTS recognition.

mSurfaceHolderCamera.addCallback(surfaceHolderCallback)
private val surfaceHolderCallback = object : SurfaceHolder.Callback {
override fun surfaceCreated(holder: SurfaceHolder) {
createAnalyzer()
}
override fun surfaceChanged(holder: SurfaceHolder, format: Int, width: Int, height: Int) {
prepareLensEngine(width, height)
mLensEngine.run(holder)
}
override fun surfaceDestroyed(holder: SurfaceHolder) {
mLensEngine.release()
}
}

Step 2 Create a Hand Keypoint Analyzer

//Creates MLKeyPointAnalyzer with MLHandKeypointAnalyzerSetting.
val settings = MLHandKeypointAnalyzerSetting.Factory()
.setSceneType(MLHandKeypointAnalyzerSetting.TYPE_ALL)
.setMaxHandResults(2)
.create()
// Set the maximum number of hand regions that can be detected within an image. A maximum of 10 hand regions can be detected by default

mAnalyzer = MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer(settings)
mAnalyzer.setTransactor(mHandKeyPointTransactor)

Step 3 Create the HandKeypointTransactor class for processing detection results

Below class implements the MLAnalyzer.MLTransactor<T> API and uses the transactResult method in this class to obtain the detection results and implement specific services.

class HandKeyPointTransactor(surfaceHolder: SurfaceHolder? = null): MLAnalyzer.MLTransactor<MLHandKeypoints> {

override fun transactResult(result: MLAnalyzer.Result<MLHandKeypoints>?) {

var foundCharacter = findTheCharacterResult(result)

if (foundCharacter.isNotEmpty() && !foundCharacter.equals(lastCharacter)) {
lastCharacter = foundCharacter
displayText.append(lastCharacter)
}

canvas.drawText(displayText.toString(), paddingleft, paddingRight, Paint().also {
it.style = Paint.Style.FILL
it.color = Color.YELLOW
})

}

Step 4 Create an Instance of the Lens Engine

LensEngine lensEngine = new LensEngine.Creator(getApplicationContext(), analyzer)
setLensType(LensEngine.BACK_LENS)
applyDisplayDimension(width, height) // adjust width and height depending on the orientation
applyFps(5f)
enableAutomaticFocus(true)
create();

Step 5 Run the LensEngine

private val surfaceHolderCallback = object : SurfaceHolder.Callback {

// run the LensEngine in surfaceChanged()
override fun surfaceChanged(holder: SurfaceHolder, format: Int, width: Int, height: Int) {
createLensEngine(width, height)
mLensEngine.run(holder)
}

}

Step 6 Need the analyzer stop it and release resources

fun stopAnalyzer() {
mAnalyzer.stop()
}

Step 7. Process the transactResult() to detect character

You can use the transactResult method in HandKeypointTransactor class to obtain the detection results and implement specific services. In addition to the coordinate information of each hand keypoint, the detection result includes a confidence value of the palm and of each keypoint. The palm and hand keypoints that are incorrectly recognized can be filtered out based on the confidence values. In actual scenarios, a threshold can be flexibly set based on tolerance of misrecognition.

Step 7.1 Find the direction of the finger:

Let us start consider possible vector slope for finger is in two axis x-axis and y-axis

private const val X_COORDINATE = 0
private const val Y_COORDINATE = 1

We have Fingers possible in five vectros. The direction of any finger at any time can be categorised as UP, DOWN, DOWN_UP, UP_DOWN, NEUTRAL.

enum class FingerDirection {
VECTOR_UP, VECTOR_DOWN, VECTOR_UP_DOWN, VECTOR_DOWN_UP, VECTOR_UNDEFINED
}

enum class Finger {
THUMB, FIRST_FINGER, MIDDLE_FINGER, RING_FINGER, LITTLE_FINGER
}

First separate the corresponding keyupoints from the result to different fingers's key point array like this

var firstFinger = arrayListOf<MLHandKeypoint>()
var middleFinger = arrayListOf<MLHandKeypoint>()
var ringFinger = arrayListOf<MLHandKeypoint>()
var littleFinger = arrayListOf<MLHandKeypoint>()
var thumb = arrayListOf<MLHandKeypoint>()

Each keypoint in the finger corresponds to the joint in the finger. By finding the distance of the joints from from the average position value of the finger, you can find the slope. Check the x and y co-ordinates in the key point with respect to the x and y co-ordinates of the nearby keypoint.

For example :

Take two samples keypoints for H letter

int[] datapointSampleH1 = {623, 497, 377, 312, 348, 234, 162, 90, 377, 204, 126, 54, 383, 306, 413, 491, 455, 348, 419, 521 };
int [] datapointSampleH2 = {595, 463, 374, 343, 368, 223, 147, 78, 381, 217, 110, 40, 412, 311, 444, 526, 450, 406, 488, 532};

You can calculate the vector using average of the coordinates of the finger

//For ForeFinger - 623, 497, 377, 312

double avgFingerPosition = (datapoints[0].getX()+datapoints[1].getX()+datapoints[2].getX()+datapoints[3].getX())/4;
// find the average and subract it from the value of x
double diff = datapointSampleH1 [position] .getX() - avgFingerPosition ;
//vector either positive or negative representing the direction
int vector = (int)((diff *100)/avgFingerPosition ) ;

Resulting vector will be of negative or positive value, if it is positive it is in the direction of x-axis positive quad and if it negative it is viceversa. Using this approrach do the vector mapping for all alphabets. Once you have all the vectors we can use those for your programming.

Using above vector direction we can categorise into the vectors we defined first as FingerDirection enum .

private fun getSlope(keyPoints: MutableList<MLHandKeypoint>, coordinate: Int): FingerDirection {

when (coordinate) {
X_COORDINATE -> {
if (keyPoints[0].pointX > keyPoints[3].pointX && keyPoints[0].pointX > keyPoints[2].pointX)
return FingerDirection.VECTOR_DOWN
if (keyPoints[0].pointX > keyPoints[1].pointX && keyPoints[3].pointX > keyPoints[2].pointX)
return FingerDirection.VECTOR_DOWN_UP
if (keyPoints[0].pointX < keyPoints[1].pointX && keyPoints[3].pointX < keyPoints[2].pointX)
return FingerDirection.VECTOR_UP_DOWN
if (keyPoints[0].pointX < keyPoints[3].pointX && keyPoints[0].pointX < keyPoints[2].pointX)
return FingerDirection.VECTOR_UP
}
Y_COORDINATE -> {
if (keyPoints[0].pointY > keyPoints[1].pointY && keyPoints[2].pointY > keyPoints[1].pointY && keyPoints[3].pointY > keyPoints[2].pointY)
return FingerDirection.VECTOR_UP_DOWN
if (keyPoints[0].pointY > keyPoints[3].pointY && keyPoints[0].pointY > keyPoints[2].pointY)
return FingerDirection.VECTOR_UP
if (keyPoints[0].pointY < keyPoints[1].pointY && keyPoints[3].pointY < keyPoints[2].pointY)
return FingerDirection.VECTOR_DOWN_UP
if (keyPoints[0].pointY < keyPoints[3].pointY && keyPoints[0].pointY < keyPoints[2].pointY)
return FingerDirection.VECTOR_DOWN
}

}
return FingerDirection.VECTOR_UNDEFINED

Get the directions of each finger and store in an array

xDirections[Finger.FIRST_FINGER] = getSlope(firstFinger, X_COORDINATE)
yDirections[Finger.FIRST_FINGER] = getSlope(firstFinger, Y_COORDINATE )

More details, you can visit https://forums.developer.huawei.com/forumPortal/en/topic/0204403586406920123