[Huawei Developers] John Wo:
Prolusion

This era is revolutionary for the science and research as most of the innovation is for the consumer needs.

We all know that document scanning is routine errand for most of us and a dire need for today’s digital world.

In such needs, we often require a powerful mechanism which can correct the informalities and skew for our documents.

Document Skew Correction is a technique which helps in correcting the tilted images to the right facing angle which further improvise the visibility of the image.

Huawei ML Kit offers a robust API for skew correction which enables automatic position identification of a document in an image and corrects the shooting angle. It also allows users to customize the edge points.

[IMG="https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/834/235/2640091000001834235.20201030114032.83027910941452965530177849793080:50511111010452:2800:A5EFDD244F3174A001683B04025C6965ED3120439D541E549557E8E0E6C0A993.png"]https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/834/235/2640091000001834235.20201030114032.83027910941452965530177849793080:50511111010452:2800:A5EFDD244F3174A001683B04025C6965ED3120439D541E549557E8E0E6C0A993.png[/IMG]

Suggestions

It is recommended the shooting angle of the image should be within 30 degrees.

It is recommended that the image size be within the range of 320 x 320 px to 1920 x 1920 px.

Skew detection API supports JPG, JPEG, and PNG image formats.

Development Overview

Prerequisite

Must have a Huawei Developer Account

Must have Android Studio 3.0 or later

Must have a Huawei phone with HMS Core 4.0.2.300 or later

EMUI 3.0 or later

Software Requirements

Java SDK 1.7 or later

Android 5.0 or later

Preparation

Create an app or project in the Huawei app gallery connect.

Provide the SHA Key and App Package name of the project in App Information Section and enable the ML Kit API.

Download the agconnect-services.json file.

Create an Android project.

Integration

Add below to build.gradle (project)file, under buildscript/repositories and allprojects/repositories.

Maven {url 'http://developer.huawei.com/repo/'}

Add below to build.gradle (app) file, under dependencies.

To use the Base SDK of ML Kit-Document Skew Correction, add the following dependencies:

dependencies{

// Import the base SDK.

implementation 'com.huawei.hms:ml-computer-vision-documentskew:2.0.4.300'

}

To use the Full SDK of ML Kit- Document Skew Correction, add the following dependencies:

dependencies{

// Import the Model Package.

implementation 'com.huawei.hms:ml-computer-vision-documentskew-model:2.0.4.300'

}

Adding permissions

<uses-permission android:name="android.permission.CAMERA " />

<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

Automatically Updating the Machine Learning Model

Add the following statements to the AndroidManifest.xml file to automatically install the machine learning model on the user’s device.

<meta-data

android:name="com.huawei.hms.ml.DEPENDENCY"

android:value= "dsc"/>

Development Process

This article focuses on demonstrating the capabilities of Huawei’s ML Kit: Document Skew Correction API’s.

Here is the example of “SUPER DOC” application which allows user to capture and fetch the images from local memory of the device and let them correct using the document which explains how we can integrate this powerful API to leverage the benefits of correcting a skewed document image to provider the right angle to the document which eventually improves the readability of the document.

Skewdetect Activity

This activity is responsible to click and fetch the images and detect them for the skew correction and further align them and provide the output as aligned document image.

package com.mlkit.documentSkewCorrection;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import android.content.Intent;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.Point;
import android.net.Uri;
import android.os.Bundle;
import android.provider.MediaStore;
import android.util.Log;
import android.view.View;
import android.widget.ImageView;
import android.widget.Toast;
import androidx.appcompat.app.AppCompatActivity;
import com.google.android.material.floatingactionbutton.FloatingActionButton;
import com.huawei.hmf.tasks.OnFailureListener;
import com.huawei.hmf.tasks.OnSuccessListener;
import com.huawei.hmf.tasks.Task;
import com.huawei.hms.mlsdk.common.MLFrame;
import com.huawei.hms.mlsdk.dsc.MLDocumentSkewCorrectionConstant;
import com.huawei.hms.mlsdk.dsc.MLDocumentSkewCorrectionCoordinateInput;
import com.huawei.hms.mlsdk.dsc.MLDocumentSkewCorrectionResult;
import com.huawei.hms.mlsdk.dsc.MLDocumentSkewCorrectionAnalyzer;
import com.huawei.hms.mlsdk.dsc.MLDocumentSkewCorrectionAnalyzerFactory;
import com.huawei.hms.mlsdk.dsc.MLDocumentSkewCorrectionAnalyzerSetting;
import ccom.huawei.hms.mlsdk.dsc.MLDocumentSkewDetectResult;
import com.mlkit.documentSkewCorrection.R;

public class SkewDetect extends AppCompatActivity implements View.OnClickListener {
private static final String TAG = "SkewDetectActivity";

private MLDocumentSkewCorrectionAnalyzer analyzer;
private ImageView mImageView;
private Bitmap bitmap;
Uri imageUri;
private MLDocumentSkewCorrectionCoordinateInput input;
private MLFrame mlFrame;
Boolean FlagCameraClickDone = false;

@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
this.setContentView(R.layout.activity_document_skew_correction);
this.findViewById(R.id.image_refine).setOnClickListener(this);
this.mImageView = this.findViewById(R.id.image_refine_result);

if(FlagCameraClickDone)
{
this.findViewById(R.id.image_refine).setVisibility(View.VISIBLE);
}
else
{
this.findViewById(R.id.image_refine).setVisibility(View.GONE);
}
// Create the setting.
MLDocumentSkewCorrectionAnalyzerSetting setting = new MLDocumentSkewCorrectionAnalyzerSetting
.Factory()
.create();

// Get the analyzer.
this.analyzer = MLDocumentSkewCorrectionAnalyzerFactory.getInstance().getDocumentSkewCorrectionAnalyzer(setting);

FloatingActionButton fab = (FloatingActionButton) findViewById(R.id.fab);
fab.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View view) {

FlagCameraClickDone = false;
Intent gallery = new Intent(Intent.ACTION_PICK, MediaStore.Images.Media.EXTERNAL_CONTENT_URI);
startActivityForResult(gallery, 1);
}
});
}

@Override
protected void onActivityResult(int requestCode, int resultCode, Intent data){
super.onActivityResult(requestCode, resultCode, data);
if (resultCode == RESULT_OK && requestCode == 1){
imageUri = data.getData();
try {
bitmap = MediaStore.Images.Media.getBitmap(this.getContentResolver(), imageUri);

// Create a MLFrame by using the bitmap.
this.mlFrame = new MLFrame.Creator().setBitmap(this.bitmap).create();
} catch (IOException e) {
e.printStackTrace();
}
//BitmapFactory.decodeResource(getResources(), R.drawable.new1);
FlagCameraClickDone = true;
this.findViewById(R.id.image_refine).setVisibility(View.VISIBLE);
mImageView.setImageURI(imageUri);
}
}

@Override
public void onClick(View v) {
this.analyzer();
}

private void analyzer() {
// Call document skew detect interface to get coordinate data
Task<MLDocumentSkewDetectResult> detectTask = this.analyzer.asyncDocumentSkewDetect(this.mlFrame);
detectTask.addOnSuccessListener(new OnSuccessListener<MLDocumentSkewDetectResult>() {
@Override
public void onSuccess(MLDocumentSkewDetectResult detectResult) {
Log.e(TAG, detectResult.getResultCode() + ":");

if (detectResult != null) {
int resultCode = detectResult.getResultCode();
// Detect success.
if (resultCode == MLDocumentSkewCorrectionConstant.SUCCESS) {
Point leftTop = detectResult.getLeftTopPosition();
Point rightTop = detectResult.getRightTopPosition();
Point leftBottom = detectResult.getLeftBottomPosition();
Point rightBottom = detectResult.getRightBottomPosition();

List<Point> coordinates = new ArrayList<>();
coordinates.add(leftTop);
coordinates.add(rightTop);
coordinates.add(rightBottom);
coordinates.add(leftBottom);

SkewDetect .this.setDetectData(new MLDocumentSkewCorrectionCoordinateInput(coordinates));
SkewDetect .this.refineImg();
} else if (resultCode == MLDocumentSkewCorrectionConstant.IMAGE_DATA_ERROR) {
// Parameters error.
Log.e(TAG, "Parameters error！");
SkewDetect.this.displayFailure();
} else if (resultCode == MLDocumentSkewCorrectionConstant.DETECT_FAILD) {
// Detect failure.
Log.e(TAG, "Detect failed！");
SkewDetect .this.displayFailure();
}
} else {
// Detect exception.
Log.e(TAG, "Detect exception！");
SkewDetect .this.displayFailure();
}
}
}).addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(Exception e) {
// Processing logic for detect failure.
Log.e(TAG, e.getMessage() + "");
SkewDetect .this.displayFailure();
}
});
}

// Show result
private void displaySuccess(MLDocumentSkewCorrectionResult refineResult) {
if (this.bitmap == null) {
this.displayFailure();
return;
}
// Draw the portrait with a transparent background.
Bitmap corrected = refineResult.getCorrected();
if (corrected != null) {
this.mImageView.setImageBitmap(corrected);
} else {
this.displayFailure();
}
}

private void displayFailure() {
Toast.makeText(this.getApplicationContext(), "Fail", Toast.LENGTH_SHORT).show();
}

private void setDetectData(MLDocumentSkewCorrectionCoordinateInput input) {
this.input = input;
}

// Refine image
private void refineImg() {
// Call refine image interface
Task<MLDocumentSkewCorrectionResult> correctionTask = this.analyzer.asyncDocumentSkewCorrect(this.mlFrame, this.input);
correctionTask.addOnSuccessListener(new OnSuccessListener<MLDocumentSkewCorrectionResult>() {
@Override
public void onSuccess(MLDocumentSkewCorrectionResult refineResult) {
if (refineResult != null) {
int resultCode = refineResult.getResultCode();
if (resultCode == MLDocumentSkewCorrectionConstant.SUCCESS) {
SkewDetect .this.displaySuccess(refineResult);
} else if (resultCode == MLDocumentSkewCorrectionConstant.IMAGE_DATA_ERROR) {
// Parameters error.
Log.e(TAG, "Parameters error！");
SkewDetect .this.displayFailure();
} else if (resultCode == MLDocumentSkewCorrectionConstant.CORRECTION_FAILD) {
// Correct failure.
Log.e(TAG, "Correct failed！");
SkewDetect .this.displayFailure();
}
} else {
// Correct exception.
Log.e(TAG, "Correct exception！");
SkewDetect .this.displayFailure();
}
}
}).addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(Exception e) {
// Processing logic for refine failure.
SkewDetect .this.displayFailure();
}
});
}

@Override
protected void onDestroy() {
super.onDestroy();
if (this.analyzer != null) {
try {
this.analyzer.stop();
} catch (IOException e) {
Log.e(SkewDetect .TAG, "Stop failed: " + e.getMessage());
}
}
}
}

Skewdetect Activity View Class

This class is responsible for creating the UI definition of the application.

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:background="@drawable/shape"
tools:context="com.huawei.mlkit.example.face.StillFaceAnalyseActivity">

/**Create an image view to hold the bitmap**/
<ImageView
android:id="@+id/image_refine_result"
android:layout_width="500dp"
android:layout_height="300dp"
android:layout_below="@+id/image_foreground"
android:layout_marginTop="20dp"></ImageView>

<RelativeLayout
android:id="@+id/relativeLayout1"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:layout_margin="20dp">

/**Create a button to fetch the ML kit API for skew correction**/
<Button
android:id="@+id/imagecorrection"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_centerHorizontal="true"

android:layout_alignParentBottom="true"
android:layout_gravity="start|bottom"
android:background="@color/emui_color_gray_1"
android:text=" Skew Correction "
android:textAllCaps="false"
android:textColor="@color/emui_color_gray_7"></Button>

/**Create a button to capture the image for skew correction**/
<com.google.android.material.floatingactionbutton.FloatingActionButton
android:id="@+id/fab"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_alignParentRight="true"
android:layout_alignParentBottom="true"
android:layout_gravity="end|bottom"
android:contentDescription="@string/camera"
android:outlineProvider="none"
android:src="@drawable/gall"
app:backgroundTint="@color/emui_color_gray_1"
app:borderWidth="0dp"
app:elevation="2dp" />

/**Create a button to fetch the image for skew correction**/
<com.google.android.material.floatingactionbutton.FloatingActionButton
android:id="@+id/cam"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_alignParentBottom="true"
android:layout_gravity="bottom"
android:layout_alignParentLeft="true"
android:contentDescription="@string/camera"
android:outlineProvider="none"
android:src="@drawable/icon_cam"
app:backgroundTint="@color/emui_color_gray_1"
app:borderWidth="0dp"
app:elevation="2dp" />
</RelativeLayout>
</RelativeLayout>

Results

[IMG="https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/834/235/2640091000001834235.20201030121709.49813038912690023153832090930967:50511111024852:2800:E7AD90FA63859091A3EC5A1D4BB354A8189775BC2340B7E82CF8426DAE2DECC2.gif"]https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/834/235/2640091000001834235.20201030121709.49813038912690023153832090930967:50511111024852:2800:E7AD90FA63859091A3EC5A1D4BB354A8189775BC2340B7E82CF8426DAE2DECC2.gif[/IMG]

[IMG="https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/834/235/2640091000001834235.20201030114939.22974010924243853518700301331138:50511111024852:2800:C1B390ED6A0DEFBF0DF300ED39F0E1E6C212CAAE3A9D3941B3A592DAD1B3B30A.png"]https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/834/235/2640091000001834235.20201030114939.22974010924243853518700301331138:50511111024852:2800:C1B390ED6A0DEFBF0DF300ED39F0E1E6C212CAAE3A9D3941B3A592DAD1B3B30A.png[/IMG]

[IMG="https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/834/235/2640091000001834235.20201030114946.29320043258461272184254987145623:50511111024852:2800:CCA5D15F7823FD2FEFD15F53ED118428E74D24A69DAB939AC253ABE870B0181A.png"]https://communityfile-dre.op.hicloud.com/FileServer/getFile/cmtybbs/001/834/235/2640091000001834235.20201030114946.29320043258461272184254987145623:50511111024852:2800:CCA5D15F7823FD2FEFD15F53ED118428E74D24A69DAB939AC253ABE870B0181A.png[/IMG]

Conclusion

In this article we took a small step to create and demonstrate the integration of Document Skew Correction API’s from Huawei ML Kit for better document image readability.

Upcoming article will have the integration of multiple ML API’s under one powerful application.

Stay tuned!!

References

https://developer.huawei.com/consumer/en/doc/development/HMSCore-Guides/documentskewcorrection-0000001051703156