Capture the forms using Form Recognition feature by Huawei ML Kit in Android (Kotlin)
Tags: mlkit, kotlin, android, xml
Published: 2022-05-18 15:36:40


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nczuvwsf797az9jtp3bd.png)

**Introduction**

In this article, we can learn how to integrate **Form Recognition** feature using **Huawei ML Kit**.

**Form Recognition** service can recognize the information from Form and it will return table content such as **table count, rows, columns, cell coordinate, text Information** etc. form text in Chinese and English (including punctuation) from input images.

This service is majorly used in daily work scenarios. For example, suppose if you want to covert large number of paper questionnaires into electronic documents, this service reduces manual input costs and greatly improves work efficiency.

**Precautions**
1. Forms such as questionnaires can be recognized.
2. Currently images containing multiple forms cannot be recognized.
3. **Shooting Angle**: The horizontal tilt angle is less than 5 degrees.
4. **Form Integrity**: No missing corners and no bent or segment lines.
5. **Form Content**: Only printed content can be recognized, images, hand written content, seals and watermarks in the form cannot be recognized.
6. **Image Specification**: Image ratio should be less than or equal **3:1**, resolution must be greater than **960 x 960 px**.

**Requirements**
1. Any operating system (MacOS, Linux and Windows).
2. Must have a Huawei phone with HMS 4.0.0.300 or later.
3. Must have a laptop or desktop with Android Studio, Jdk 1.8, SDK platform 26 and Gradle 4.6 and above installed.
4. Minimum API Level 21 is required.
5. Required EMUI 9.0.0 and later version devices.

**How to integrate HMS Dependencies**

- First register as Huawei developer and complete identity verification in Huawei developers website, refer to register a [Huawei ID](https://developer.huawei.com/consumer/en/doc/start/registration-and-verification-0000001053628148).

- Create a project in android studio, refer [Creating an Android Studio Project](https://developer.huawei.com/consumer/en/codelab/HMSPreparation/index.html#1).

- Generate a **SHA-256 certificate fingerprint**.

- To generate **SHA-256 certificate fingerprint**. On right-upper corner of android project click **Gradle**, choose **Project Name > Tasks > android**, and then click **signingReport**, as follows.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a3oh52nwbt6pzvjecyho.png)

Note: Project Name depends on the user created name.

- [Create an App in AppGallery Connect](https://developer.huawei.com/consumer/en/codelab/HMSPreparation/index.html#0).

- Download the **agconnect-services.json** file from **App information**, copy and paste in android **Project** under **app** directory, as follows.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9y7y41rfehkd9gla93xk.png)

- Enter **SHA-256 certificate fingerprint** and click **Save** button, as follows.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7632ecsjlu0mc15z6fdu.png)

- Click **Manage APIs** tab and enable ML Kit.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3zp6u5y76p45c3jwz23t.png)

- Add the below maven URL in **build.gradle(Project)** file under the repositories of **buildscript, dependencies** and **allprojects**, refer [Add Configuration](https://developer.huawei.com/consumer/en/codelab/HMSPreparation/index.html#6).
```
maven { url 'http://developer.huawei.com/repo/' }
classpath 'com.huawei.agconnect:agcp:1.4.1.300'

```
- Add the below plugin and dependencies in **build.gradle(Module)** file.
```
apply plugin: 'com.huawei.agconnect'
// Huawei AGC
implementation 'com.huawei.agconnect:agconnect-core:1.5.0.300'
// ML Kit base SDK
implementation 'com.huawei.hms:ml-computer-vision-formrecognition:3.2.0.300'
// ML Kit Form Recognition model package
implementation 'com.huawei.hms:ml-computer-vision-formrecognition-model:3.2.0.300'
```

- Now Sync the gradle.
- Add the required permission to the **AndroidManifest.xml** file.
```
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
```

**Let us move to development**
I have created a project on Android studio with empty activity let us start coding.
In the **MainActivity.kt** we can find the business logic.
```
class MainActivity : AppCompatActivity() {

    private var setting: MLFormRecognitionAnalyzerSetting? = null
    private var analyzer: MLFormRecognitionAnalyzer? = null
    private var mImageView: ImageView? = null
    private var text: TextView? = null
    private var textTotal:TextView? = null
    private var mlFrame: MLFrame? = null
    private var imageUri: Uri? = null
    private var bitmap: Bitmap? = null
    private val camRequestCode = 100
    private val storageRequestCode = 200
    private val sum = 0

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        mImageView = findViewById<View>(R.id.image) as ImageView?
        text = findViewById(R.id.text)
        textTotal = findViewById(R.id.text_total)
        setting = MLFormRecognitionAnalyzerSetting.Factory().create()
        analyzer = MLFormRecognitionAnalyzerFactory.getInstance().getFormRecognitionAnalyzer(setting)

    }

    fun onLoadImage(view: View?) {
        val intent = Intent(Intent.ACTION_PICK, MediaStore.Images.Media.EXTERNAL_CONTENT_URI)
        startActivityForResult(intent, storageRequestCode)
    }

    fun onClikCam(view: View?) {
        if (checkSelfPermission(Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED) {
            requestPermissions(arrayOf(Manifest.permission.CAMERA), camRequestCode)
        } else {
            val intent = Intent(MediaStore.ACTION_IMAGE_CAPTURE)
            startActivityForResult(intent, camRequestCode)
        }
    }

    override fun onRequestPermissionsResult(requestCode: Int, permissions: Array<String?>, grantResults: IntArray) {
        super.onRequestPermissionsResult(requestCode, permissions, grantResults)
        if (requestCode == camRequestCode) {
            if (grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                val intent = Intent(MediaStore.ACTION_IMAGE_CAPTURE)
                startActivityForResult(intent, camRequestCode)
            } else {
                Toast.makeText(this, "Camera permission denied", Toast.LENGTH_LONG).show()
            }
        }
        if (requestCode == storageRequestCode) {
            if (grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                val intent = Intent(Intent.ACTION_PICK, MediaStore.Images.Media.EXTERNAL_CONTENT_URI)
                startActivityForResult(intent, storageRequestCode)
            } else {
                Toast.makeText(this, "Storage permission denied", Toast.LENGTH_LONG).show()
            }
        }
    }

    override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
        super.onActivityResult(requestCode, resultCode, data)
        if (resultCode === RESULT_OK && requestCode === storageRequestCode) {
            imageUri = data!!.data
            try {
                bitmap = MediaStore.Images.Media.getBitmap(this.contentResolver, imageUri)
                mImageView!!.setImageBitmap(bitmap)
                callFormService()
            } catch (e: IOException) {
                e.printStackTrace()
            }
        } else if (resultCode === RESULT_OK && requestCode === camRequestCode) {
            bitmap = data!!.extras!!.get("data") as Bitmap?
            mImageView!!.setImageBitmap(bitmap)
            callFormService()
        }
    }

    private fun callFormService() {
        mlFrame = MLFrame.fromBitmap(bitmap)
        analyzer = MLFormRecognitionAnalyzerFactory.getInstance().formRecognitionAnalyzer
        val task: Task<JsonObject> = analyzer!!.asyncAnalyseFrame(mlFrame)
        task.addOnSuccessListener(OnSuccessListener<JsonObject?> { jsonObject ->
            if (jsonObject != null && jsonObject["retCode"].asInt == MLFormRecognitionConstant.SUCCESS) {
                val gson = Gson()
                val result = jsonObject.toString()
                val mlObject = gson.fromJson(result, MLFormRecognitionTablesAttribute::class.java)
                val tableAttributeArrayList = mlObject.tablesContent.tableAttributes
                val tableCellAttributes = tableAttributeArrayList[0].tableCellAttributes
                for (attribute in tableCellAttributes) {
                    val info = attribute.textInfo
                    text!!.text = """
                    ${text!!.text}
                    $info
                    """.trimIndent()
                }
                Toast.makeText(this@MainActivity,"Successfully Form Recognized",Toast.LENGTH_LONG).show()
                Log.d("TAG", "result: $result")
            } else if (jsonObject != null && jsonObject["retCode"].asInt == MLFormRecognitionConstant.FAILED) {
                Toast.makeText(this@MainActivity, "Form Recognition Convertion Failed", Toast.LENGTH_LONG).show()
            }
            textTotal!!.text = "Total Cart Value : $sum Rs "
        }).addOnFailureListener(OnFailureListener {
            Toast.makeText(this@MainActivity,"Form Recognition API Failed", Toast.LENGTH_LONG).show()
        })
    }

}
```

In the **activity_main.xml** we can create the UI screen.

```
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical"
    tools:context=".MainActivity">

    <ImageView
        android:id="@+id/image"
        android:layout_width="match_parent"
        android:layout_height="250dp"
        android:layout_marginTop="?actionBarSize" />

    <LinearLayout
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:paddingTop="10dp"
        android:orientation="horizontal">
        <ImageButton
            android:id="@+id/btn_storage1"
            android:layout_width="85dp"
            android:layout_height="70dp"
            android:layout_marginLeft="30dp"
            android:layout_alignParentTop="true"
            android:layout_marginBottom="10dp"
            android:paddingTop="10dp"
            android:onClick="onLoadImage"
            android:layout_centerHorizontal="true"
            android:src="@drawable/gall" />
        <ImageButton
            android:id="@+id/btn_capture1"
            android:layout_width="85dp"
            android:layout_height="70dp"
            android:onClick="onClikCam"
            android:layout_marginLeft="100dp"
            android:layout_marginRight="30dp"
            android:layout_alignParentTop="true"
            android:layout_centerHorizontal="true"
            android:paddingBottom="10dp"
            android:src="@drawable/cam" />
    </LinearLayout>

    <TextView
        android:id="@+id/text_total"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_marginLeft="20dp"
        android:textSize="17sp"
        android:textColor="@android:color/holo_red_dark"
        android:textStyle="bold" />
    <ScrollView
        android:layout_marginTop="8dp"
        android:layout_width="match_parent"
        android:layout_height="wrap_content">
        <TextView
            android:id="@+id/text"
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:layout_marginLeft="20dp"
            android:textSize="17sp"
            android:textColor="@android:color/holo_red_dark"
            android:textStyle="bold" />
    </ScrollView>

</LinearLayout>
```

**Demo**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7f1z1tiq2gq087u9nbze.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b5jo83zgstxakg4zrw5o.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oxkkk7dzuiesldqmxgfd.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dao144ek4qgoia6yt645.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mf24ktt8zkhatyuiclo6.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j09cjq1sowsujw9gbo6b.png)

**Tips and Tricks**
1. Make sure you are already registered as Huawei developer.
2. Set minSDK version to **21** or later, otherwise you will get **AndriodManifest** merge issue.
3. Make sure you have added the **agconnect-services.json** file to app folder.
4. Make sure you have added **SHA-256 fingerprint** without fail.
5. Make sure all the dependencies are added properly.

**Conclusion**
In this article, we have learned how to integrate **Form Recognition** feature using **Huawei ML Kit**.
**Form Recognition** service can recognize the information from Form and it will return table content such as table count, rows, columns, cell coordinate, text Information etc. form text in Chinese and English (including punctuation) from input images.

I hope you have read this article. If you found it is helpful, please provide likes and comments.

**Reference**
ML Kit - [Form Recognition](https://developer.huawei.com/consumer/en/doc/development/hiai-Guides/form-recognition-0000001058920154)