[Huawei Developers] Freemind R:
Allow your users the freedom to choose their Android platform providing the same feature

Some time ago I developed a Word Search game solver Android application using the services from Firebase ML Kit.

Solve WordSearch games with Android and ML Kit

A Kotlin ML Kit Data Structure & Algorithm Story

It was an interesting trip discovering the features of a framework that allows the developer to use AI capabilities without knowing all the rocket science behind.

In the specific, I’ve used the Document recognition feature to try to extract text from a word search game image.

After the text recognition phase, the output was cleaned and arranged into a matrix to be processed by the solver algorithm. This algo tried to look for all the words formed by grouping the letters respecting the rules of the games: contiguous letters in all the straight directions (vertical, horizontal, diagonal)

This app ran well on all the Android devices capable to run the Google Firebase SDK and the Google Mobile Services (GMS).

Since the second half of last year all new Huawei devices cannot run the GMS any more due to government restrictions, you can read more about this here:

[Update 14: Temporary License Extended Again]

Google has revoked Huawei's Android license

www.xda-developers.com

My app was not capable to run on the brand new Huawei devices :(

So I tried to look for solutions to make this case study app running on the new Huawei terminals.

Let’s follow my journey…

The Discovery of HMS ML Kit

I went throughout the Huawei documentation on HUAWEI Developer--The official site for HUAWEI developers. Provides HUAWEI appgallery service,HUAWEI Mobile Services,AI SDK,VR SDK

Here you can find many SDKs AKA Kits offering a set of smart features to the developers.

I’ve found one offering the features that I was looking for: HMS ML Kit. It is quite similar to the one from Firebase as it allows the developer to use Machine Learning capabilities like Image, Text, Face recognition and so on.

Huawei ML Kit

In particular, for my specific use case, I’ve used the text analyzer capable to run locally and taking advantage of the neural processing using NPU hardware.

Documentation HMS ML Kit Text recognition

Integrating HMS ML Kit was super easy. If you want to give it a try It’s just a matter of adding a dependency in your build.gradle file, enabling the service from the AppGallery web dashboard if you want to use the Cloud API and download the agconnect-services.json configuration file and use it in your app.

You can refer to the official guide here for the needed steps:

Documentation HMS ML Kit

Architectural Approach

My first desire was to maintain and deploy only one apk so I wanted to integrate both the Firebase ML Kit SDK and the HMS ML Kit one.

I thought about the main feature

Decode the image and getting back the text detected together with the bounding boxes surrounding each character to better display the spotted text to the user.

This was defined by this interface

package com.laquysoft.wordsearchai.textrecognizer

import android.graphics.Bitmap

interface DocumentTextRecognizer {

fun processImage(bitmap: Bitmap, success: (Document) -> Unit, error: (String?) -> Unit)
}

I’ve also defined my own data classes to have a common output format from both services

data class Symbol(
val text: String?,
val rect: Rect,
val idx: Int = 0,
val length: Int = 0
)

data class Document(val stringValue: String, val count: Int, val symbols: List<Symbol>)

Where Document represents the text result returned by the ML Kit services, it contains a list of Symbol (the character recognized) each one with its own char, the bounding box surrounding it (Rect), and the index in the string detected as both MLKit service will group some chars in a string with a unique bounding box.

Then I’ve created an object capable to instantiate the right service depending which service (HMS or GMS) is running on the device

object DocumentTextRecognizerService {

private fun getServiceType(context: Context) = when {
isGooglePlayServicesAvailable(
context
) -> ServiceType.GOOGLE
isHuaweiMobileServicesAvailable(
context
) -> ServiceType.HUAWEI
else -> ServiceType.GOOGLE
}

private fun isGooglePlayServicesAvailable(context: Context): Boolean {
return GoogleApiAvailability.getInstance()
.isGooglePlayServicesAvailable(context) == ConnectionResult.SUCCESS
}

private fun isHuaweiMobileServicesAvailable(context: Context): Boolean {
return HuaweiApiAvailability.getInstance()
.isHuaweiMobileServicesAvailable(context) == com.huawei.hms.api.ConnectionResult.SUCCESS

}

fun create(context: Context): DocumentTextRecognizer {
val type =
getServiceType(
context
)
if (type == ServiceType.HUAWEI)
return HMSDocumentTextRecognizer()
return GMSDocumentTextRecognizer()
}
}

This was pretty much all to make it works.

The ViewModel can use the service provided

class WordSearchAiViewModel(
private val resourceProvider: ResourceProvider,
private val recognizer: DocumentTextRecognizer
) : ViewModel() {

val resultList: MutableLiveData<List<String>> = MutableLiveData()
val resultBoundingBoxes: MutableLiveData<List<Symbol>> = MutableLiveData()

private lateinit var dictionary: List<String>

fun detectDocumentTextIn(bitmap: Bitmap) {

loadDictionary()

recognizer.processImage(bitmap, {
postWordsFound(it)
postBoundingBoxes(it)
},
{
Log.e("WordSearchAIViewModel", it)
})
}

by the right recognizer instantiated when the WordSearchAiViewModel is instantiated as well.

Running the app and choosing a word search game image on a Mate 30 Pro (an HMS device) shows this result

[IMG="https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/024/106/043/0890044000024106043.20200528150245.31107221049235275332842634657961:50510610030823:2800:7A67F4D78962BC8A36B90600823D858C747715C7E2F67F572BDB2B438D4078FB.png"]https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/024/106/043/0890044000024106043.20200528150245.31107221049235275332842634657961:50510610030823:2800:7A67F4D78962BC8A36B90600823D858C747715C7E2F67F572BDB2B438D4078FB.png[/IMG]

[IMG="https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/024/106/043/0890044000024106043.20200528150309.43714176236046313714384241591580:50510610030823:2800:934045D922F4DB654E5584CD7BEBD799F59790D9B11C1EF6F51096D48D867EFC.png"]https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/024/106/043/0890044000024106043.20200528150309.43714176236046313714384241591580:50510610030823:2800:934045D922F4DB654E5584CD7BEBD799F59790D9B11C1EF6F51096D48D867EFC.png[/IMG]

The Recognizer Brothers

You can check the code of the two recognizers below. What they are doing is to use the custom SDK implementation to get the result and adapt it to the interface, you can virtually use any other service capable to do the same.

package com.laquysoft.wordsearchai.textrecognizer

import android.graphics.Bitmap
import com.google.firebase.ml.vision.FirebaseVision
import com.google.firebase.ml.vision.common.FirebaseVisionImage

class GMSDocumentTextRecognizer : DocumentTextRecognizer {

private val detector = FirebaseVision.getInstance().onDeviceTextRecognizer

override fun processImage(
bitmap: Bitmap,
success: (Document) -> Unit,
error: (String?) -> Unit
) {
val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap)
detector.processImage(firebaseImage)
.addOnSuccessListener { firebaseVisionDocumentText ->
if (firebaseVisionDocumentText != null) {
val words = firebaseVisionDocumentText.textBlocks
.flatMap { it -> it.lines }
.flatMap { it.elements }

val symbols: MutableList<Symbol> = emptyList<Symbol>().toMutableList()

words.forEach {
val rect = it.boundingBox
if (rect != null) {
it.text.forEachIndexed { idx, value ->
symbols.add(
Symbol(
value.toString(),
rect,
idx,
it.text.length
)
)
}
}
}

val document =
Document(
firebaseVisionDocumentText.text,
firebaseVisionDocumentText.textBlocks.size,
symbols
)

success(document)
}
}
.addOnFailureListener { error(it.localizedMessage) }
}
}

package com.laquysoft.wordsearchai.textrecognizer

import android.graphics.Bitmap
import com.huawei.hms.mlsdk.MLAnalyzerFactory
import com.huawei.hms.mlsdk.common.MLFrame

class HMSDocumentTextRecognizer : DocumentTextRecognizer {

//private val detector = MLAnalyzerFactory.getInstance().remoteDocumentAnalyzer
private val detector = MLAnalyzerFactory.getInstance().localTextAnalyzer

override fun processImage(
bitmap: Bitmap,
success: (Document) -> Unit,
error: (String?) -> Unit
) {
val hmsFrame = MLFrame.fromBitmap(bitmap)
detector.asyncAnalyseFrame(hmsFrame)
.addOnSuccessListener { mlDocument ->
if (mlDocument != null) {
val words = mlDocument.blocks
.flatMap { it.contents }
.flatMap { it.contents }

val symbols: MutableList<Symbol> = emptyList<Symbol>().toMutableList()

words.forEach {
val rect = it.border
it.stringValue.forEachIndexed { idx, value ->
symbols.add(Symbol(
value.toString(),
rect,
idx,
it.stringValue.length
))
}
}

val document =
Document(
mlDocument.stringValue,
mlDocument.blocks.size,
symbols
)

success(document)
}
}
.addOnFailureListener { error(it.localizedMessage) }

}
}

Conclusion

As good Android developers we should develop and deploy our apps in all the platforms our user can reach, love and adopt, without excluding anyone.

We should spend some time trying to give the users the same experience. This is a small sample about it and others will comes in the future.