[Huawei Developers] Basavaraj.navi:
View attachment 5376005

Introduction

In this article, I will explain what is Skeleton detection? How does Skeleton detection work in Android? At the end of this tutorial, we will create the Huawei Skeleton detection in an Android application using Huawei ML Kit.

What is Skeleton detection?

Huawei ML Kit Skeleton detection service detects the human body. So, represents the orientation of a person in a graphical format. Essentially, it’s a set of coordinates that can be connected to describe the position of the person. This service detects and locates key points of the human body such as the top of the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. Currently, full-body and half-body static image recognition and real-time camera stream recognition are supported.

What is the use of Skeleton detection?

Definitely, everyone will have the question like what is the use of it. For example, if you want to develop a fitness application, you can understand and help the user with coordinates from skeleton detection to see if the user has made the exact movements during exercises or you could develop a game about dance movements Using this service and ML kit can understand easily whether the user has done proper excise or not.

How does it work?

You can use skeleton detection over a static image or over a real-time camera stream. Either way, you can get the coordinates of the human body. Of course, when taking them, it’s looking out for critical areas like head, neck, shoulders, elbows, wrists, hips, knees, and ankles. At the same time, both methods will detect multiple human bodies.

There are two attributes to detect skeleton.

1. TYPE_NORMAL

2. TYPE_YOGA

TYPE_NORMAL: If you send the analyzer type as TYPE_NORMAL, perceives skeletal points for normal standing position.

TYPE_YOGA: If you send the analyzer type as TYPE_YOGA, it picks up skeletal points for yoga posture.

Note: The default mode is to detect skeleton points for normal postures.

Integration of Skeleton Detection

1. Configure the application on the AGC.

2. Client application development process.

Configure application on the AGC

This step involves a couple of steps, as follows.

Step 1: We need to register as a developer account in AppGallery Connect. If you are already a developer ignore this step.

Step 2: Create an app by referring to Creating a Project and Creating an App in the Project

Step 3: Set the data storage location based on the current location.

Step 4: Enabling ML. Open AppGallery connect, choose Manage API > ML Kit

Step 5: Generating a Signing Certificate Fingerprint.

Step 6: Configuring the Signing Certificate Fingerprint.

Step 7: Download your agconnect-services.json file, paste it into the app root directory.

Client application development process

This step involves a couple of steps, as follows.

Step 1: Create an Android application in the Android studio (Any IDE which is your favorite).

Step 2: Add the App level Gradle dependencies. Choose inside project Android > app > build.gradle.

1

2	apply plugin: 'com.android.application'

apply plugin: 'com.huawei.agconnect'

Root level gradle dependencies.

1

2	maven { url 'https://developer.huawei.com/repo/' }

classpath 'com.huawei.agconnect:agcp:1.4.1.300'

Step 3: Add the dependencies in build.gradle

1

2

3	implementation 'com.huawei.hms:ml-computer-vision-skeleton:2.0.4.300'

implementation 'com.huawei.hms:ml-computer-vision-skeleton-model:2.0.4.300'

implementation 'com.huawei.hms:ml-computer-vision-yoga-model:2.0.4.300'

To achieve the Skeleton detection example, follow the steps.

1. AGC Configuration

2. Build Android application

Step 1: AGC Configuration

1. Sign in to AppGallery Connect and select My apps.

2. Select the app in which you want to integrate the Huawei ML kit.

3. Navigate to Project Setting > Manage API > ML Kit

Step 2: Build Android application

In this example, I am getting image from the gallery or Camera and getting the skeleton detection and joints points from the ML kit skeleton detection.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147	private fun initAnalyzer(analyzerType: Int) {

val setting = MLSkeletonAnalyzerSetting.Factory()

.setAnalyzerType(analyzerType)

.create()

analyzer = MLSkeletonAnalyzerFactory.getInstance().getSkeletonAnalyzer(setting)

imageSkeletonDetectAsync()

}

private fun initFrame(type: Int) {

imageView.invalidate()

val drawable = imageView.drawable as BitmapDrawable

val originBitmap = drawable.bitmap

val maxHeight = (imageView.parent as View).height

val targetWidth = (imageView.parent as View).width

// Update bitmap size

val scaleFactor = (originBitmap.width.toFloat() / targetWidth.toFloat())

.coerceAtLeast(originBitmap.height.toFloat() / maxHeight.toFloat())

val resizedBitmap = Bitmap.createScaledBitmap(

originBitmap,

(originBitmap.width / scaleFactor).toInt(),

(originBitmap.height / scaleFactor).toInt(),

true

)

frame = MLFrame.fromBitmap(resizedBitmap)

initAnalyzer(type)

}

private fun imageSkeletonDetectAsync() {

val task: Task<List<MLSkeleton>>? = analyzer?.asyncAnalyseFrame(frame)

task?.addOnSuccessListener { results ->

// Detection success.

val skeletons: List<MLSkeleton>? = getValidSkeletons(results)

if (skeletons != null && skeletons.isNotEmpty()) {

graphicOverlay?.clear()

val skeletonGraphic = SkeletonGraphic(graphicOverlay, results)

graphicOverlay?.add(skeletonGraphic)

} else {

Log.e(TAG, "async analyzer result is null.")

}

}?.addOnFailureListener { /* Result failure. */ }

}

private fun stopAnalyzer() {

if (analyzer != null) {

try {

analyzer?.stop()

} catch (e: IOException) {

Log.e(TAG, "Failed for analyzer: " + e.message)

}

}

}

override fun onDestroy() {

super.onDestroy()

stopAnalyzer()

}

private fun showPictureDialog() {

val pictureDialog = AlertDialog.Builder(this)

pictureDialog.setTitle("Select Action")

val pictureDialogItems = arrayOf("Select image from gallery", "Capture photo from camera")

pictureDialog.setItems(pictureDialogItems

) { dialog, which ->

when (which) {

0 -> chooseImageFromGallery()

1 -> takePhotoFromCamera()

}

}

pictureDialog.show()

}

fun chooseImageFromGallery() {

val galleryIntent = Intent(Intent.ACTION_PICK, MediaStore.Images.Media.EXTERNAL_CONTENT_URI)

startActivityForResult(galleryIntent, GALLERY)

}

private fun takePhotoFromCamera() {

val cameraIntent = Intent(MediaStore.ACTION_IMAGE_CAPTURE)

startActivityForResult(cameraIntent, CAMERA)

}

public override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {

super.onActivityResult(requestCode, resultCode, data)

if (requestCode == GALLERY)

{

if (data != null)

{

val contentURI = data!!.data

try {

val bitmap = MediaStore.Images.Media.getBitmap(this.contentResolver, contentURI)

saveImage(bitmap)

Toast.makeText(this@MainActivity, "Image Show!", Toast.LENGTH_SHORT).show()

imageView!!.setImageBitmap(bitmap)

}

catch (e: IOException)

{

e.printStackTrace()

Toast.makeText(this@MainActivity, "Failed", Toast.LENGTH_SHORT).show()

}

}

}

else if (requestCode == CAMERA)

{

val thumbnail = data!!.extras!!.get("data") as Bitmap

imageView!!.setImageBitmap(thumbnail)

saveImage(thumbnail)

Toast.makeText(this@MainActivity, "Photo Show!", Toast.LENGTH_SHORT).show()

}

}

fun saveImage(myBitmap: Bitmap):String {

val bytes = ByteArrayOutputStream()

myBitmap.compress(Bitmap.CompressFormat.PNG, 90, bytes)

val wallpaperDirectory = File (

(Environment.getExternalStorageDirectory()).toString() + IMAGE_DIRECTORY)

Log.d("fee", wallpaperDirectory.toString())

if (!wallpaperDirectory.exists())

{

wallpaperDirectory.mkdirs()

}

try

{

Log.d("heel", wallpaperDirectory.toString())

val f = File(wallpaperDirectory, ((Calendar.getInstance()

.getTimeInMillis()).toString() + ".png"))

f.createNewFile()

val fo = FileOutputStream(f)

fo.write(bytes.toByteArray())

MediaScannerConnection.scanFile(this, arrayOf(f.getPath()), arrayOf("image/png"), null)

fo.close()

Log.d("TAG", "File Saved::--->" + f.getAbsolutePath())

return f.getAbsolutePath()

}

catch (e1: IOException){

e1.printStackTrace()

}

return ""

}

Result

Tips and Tricks

Check dependencies downloaded properly.

Latest HMS Core APK is required.

If you are taking an image from a camera or gallery make sure your app has camera and storage permission.

Conclusion

In this article, we have learned the integration of the Huawei ML kit, and what is skeleton detection, how it works, what is the use of it, how to get the Joints point from the skeleton detection, types of detections like TYPE_NORMAL and TYPE_YOGA.

Reference

Skeleton Detection

Happy coding