Android OCR Recognition for Passport MRZ
Tags: android, ocr, passport, mrz
Published: 2021-08-02 02:58:51

Previously, I wrote a [post](https://www.dynamsoft.com/codepool/passport-mrz-recognition-cpp-windows-linux.html) about how to recognize passport MRZ from an image file using C++. In this post, I will create a more productive Android app, which utilizes camera to recognize MRZ.

## Requirements
- [Dynamsoft OCR SDK](https://www.dynamsoft.com/label-recognition/overview/)
- [Dynamsoft Camera Enhancer](https://www.dynamsoft.com/camera-enhancer/overview/)

## Getting Started with Android Camera View
Android [camera2 API](https://developer.android.com/reference/android/hardware/camera2/package-summary) is a replacement for the older Camera API. Since it is more powerful and flexible, Android officially recommends it. However, it is still a bit complicated to use. To simplify the programming progress, I will use Dynamsoft Camera Enhancer which not only wraps the camera2 API, but also provides image processing functions to optimize camera frames.

We create a new project with an Empty Activity and add the following dependencies: 

```bash

# project/build.gradle
allprojects {
    repositories {
        maven {
            url "https://download.dynamsoft.com/maven/dce/aar"
        }
    }
}

# app/build.gradle
dependencies {
    implementation 'com.dynamsoft:dynamsoftcameraenhancer:1.0.3@aar'
}
```

After syncing the project, we can add the `CameraView` widget to `activity_main.xml`:

```xml
<com.dynamsoft.dce.CameraView
        android:id="@+id/cameraView"
        android:layout_width="match_parent"
        android:layout_height="match_parent" >
</com.dynamsoft.dce.CameraView>
```

In `MainActivity`, we add the following code to initialize the camera view:

```java
@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    initDCE();
}

private void initDCE() {
    cameraView = findViewById(R.id.cameraView);
    mCameraEnhancer = new CameraEnhancer(MainActivity.this);
    mCameraEnhancer.addCameraView(cameraView);
    com.dynamsoft.dce.DMDLSConnectionParameters info = new com.dynamsoft.dce.DMDLSConnectionParameters();
    info.organizationID = "200001"; // Get the organization ID from https://www.dynamsoft.com/customer/license/trialLicense?product=dce
    mCameraEnhancer.initLicenseFromDLS(info,new CameraDLSLicenseVerificationListener() {
        @Override
        public void DLSLicenseVerificationCallback(boolean isSuccess, Exception error) {
            if(!isSuccess){
                error.printStackTrace();
            }
        }
    });
    //Turn on the camera
    mCameraEnhancer.setCameraDesiredState(CameraState.CAMERA_STATE_ON);
}
```

You can register a Dynamsoft account and get the organization ID from [here](https://www.dynamsoft.com/customer/license/trialLicense?product=dce).

Finally, we put the code of camera status control in the Activity's lifecycle callbacks:

```java
@Override
protected void onPause() {
    super.onPause();
    mCameraEnhancer.pauseCamera();
}

@Override
protected void onResume() {
    super.onResume();
    mCameraEnhancer.resumeCamera();
    mProgressBar.setVisibility(View.GONE);
}

@Override
protected void onStart() {
    super.onStart();
    mCameraEnhancer.startScanning();
}

@Override
protected void onStop() {
    super.onStop();
    mCameraEnhancer.stopScanning();
}
``` 

Now, you can try the app by running it on an Android device.

## Recognizing Passport MRZ by Taking a Photo
Once the camera view is ready, we can start integrating OCR SDK into the app. Similar to the way of adding the dependent library of camera SDK, we add the dependent library of OCR SDK to the project: 

```bash

# project/build.gradle
allprojects {
    repositories {
        maven {
            url "http://download2.dynamsoft.com/maven/dlr/aar"
        }
    }
}

# app/build.gradle
dependencies {
    implementation "com.dynamsoft:dynamsoftlabelrecognition:1.2.1@aar"
}
```

The OCR SDK is implemented based on deep learning. Therefore, the first step is to import the [model files and a template file](https://github.com/yushulx/android-passport-mrz/releases/download/v1.0/model.zip) provided by Dynamsoft to the project. We extract all files to the assets folder.

![OCR MRZ model](https://www.dynamsoft.com/codepool/img/2021/07/ocr-mrz-model.png)

The following code is used to initialize the OCR SDK and load the model files:

```java
private void initDLR() {
    try {
        mRecognition = new LabelRecognition();
        DMLTSConnectionParameters parameters = new DMLTSConnectionParameters();
        // The organization id 200001 here will grant you a public trial license good for 7 days.
        // After that, please visit: https://www.dynamsoft.com/customer/license/trialLicense?product=dlr
        // to request for 30 days extension.
        parameters.organizationID = "200001";
        mRecognition.initLicenseFromLTS(parameters, new DLRLTSLicenseVerificationListener() {
            @Override
            public void LTSLicenseVerificationCallback(boolean b, final Exception e) {
                if (!b) {
                    e.printStackTrace();
                }
            }
        });
        loadModel();
    } catch (Exception e) {
        Toast.makeText(this, e.getMessage(), Toast.LENGTH_SHORT).show();
        e.printStackTrace();
    }
}

private void loadModel() {
    try {
        String[] fileNames = {"NumberUppercase","NumberUppercase_Assist_1lIJ","NumberUppercase_Assist_8B","NumberUppercase_Assist_8BHR","NumberUppercase_Assist_number","NumberUppercase_Assist_O0DQ","NumberUppercase_Assist_upcase"};
        for(int i = 0;i<fileNames.length;i++) {
            AssetManager manager = getAssets();
            InputStream isPrototxt = manager.open("CharacterModel/"+fileNames[i]+".prototxt");
            byte[] prototxt = new byte[isPrototxt.available()];
            isPrototxt.read(prototxt);
            isPrototxt.close();
            InputStream isCharacterModel = manager.open("CharacterModel/"+fileNames[i]+".caffemodel");
            byte[] characterModel = new byte[isCharacterModel.available()];
            isCharacterModel.read(characterModel);
            isCharacterModel.close();
            InputStream isTxt = manager.open("CharacterModel/"+fileNames[i]+".txt");
            byte[] txt = new byte[isTxt.available()];
            isTxt.read(txt);
            isTxt.close();
            mRecognition.appendCharacterModelBuffer(fileNames[i], prototxt, txt, characterModel);
        }

        StringBuilder stringBuilder = new StringBuilder();
        try {
            AssetManager manager = getAssets();
            BufferedReader bf = new BufferedReader(new InputStreamReader(
                    manager.open("wholeImgMRZTemplate.json")));
            String line;
            while ((line = bf.readLine()) != null) {
                stringBuilder.append(line);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }

        mRecognition.appendSettingsFromString(stringBuilder.toString());

    } catch (Exception e) {
        e.printStackTrace();
    }
}
```

OCR is orientation-sensitive. To get the correct orientation for rotating images, we add the Activity property `android:screenOrientation="fullSensor"` in `AndroidManifest.xml`. 

The frame acquired from camera enhancer is a gray-scale image. Referring to [StackOverflow](https://stackoverflow.com/questions/14167976/rotate-an-yuv-byte-array-on-android), a 90 degree rotation function for an image byte array could be written as follows:

```java
public static byte[] rotateGrayscale90(byte[] data, int width, int height)
{
    byte [] grayscale = new byte[width * height];
    int index = 0;
    for(int i = 0; i < width; i++)
    {
        for(int j = height - 1; j >= 0; j--)
        {
            grayscale[index] = data[j * width + i];
            index += 1;
        }
    }

    return grayscale;
}
```

The next step is to implement the logic of taking a picture and recognizing the passport MRZ in the button click event. Here we use `FloatingActionButton`:

```xml
<com.google.android.material.floatingactionbutton.FloatingActionButton
            android:id="@+id/fab"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_gravity="end|bottom"
            android:layout_margin="16dp"
            android:contentDescription="recognize"
            android:src="@android:drawable/ic_menu_camera"
            tools:ignore="MissingConstraints" />
```

```java
FloatingActionButton mButton = findViewById(R.id.fab);
mButton.setOnClickListener(new View.OnClickListener() {
    final Frame frame = mCameraEnhancer.AcquireListFrame(true);
    int orientation = getResources().getConfiguration().orientation;

    byte[] data = frame.data;
    int width = frame.width;
    int height = frame.height;
    int stride = frame.strides[0];
    if (orientation == Configuration.ORIENTATION_PORTRAIT) {
        // Rotate image
        data = Utils.rotateGrayscale90(data, width, height);
        width = frame.height;
        height = frame.width;
        stride = frame.height;
    }

    final DLRImageData dlrData = new DLRImageData();
    dlrData.bytes = data;
    dlrData.format = EnumDLRImagePixelFormat.DLR_IPF_GRAYSCALED;
    dlrData.stride = stride;
    dlrData.width = width;
    dlrData.height = height;

    try {
        results = mRecognition.recognizeByBuffer(dlrData, "locr");
    }
    catch (Exception e) {
        e.printStackTrace();
    }
});
```

When calling `recognizeByBuffer()`, we set the template name to `locr` which is defined in the template file `assets/wholeImgMRZTemplate.json`.

The last step is to parse the MRZ result and display it on the screen. According to the [standard of the passport MRZ](https://en.wikipedia.org/wiki/Machine-readable_passport), the parsing code is as follows:

```java
static String parse(String line1, String line2) {
    // https://en.wikipedia.org/wiki/Machine-readable_passport
    String result = "";
    // Type
    String tmp = "Type: ";
    tmp += line1.charAt(0);
    result += tmp + "\n\n";

    // Issuing country
    tmp = "Issuing country: ";
    tmp += line1.substring(2, 5);
    result += tmp + "\n\n";

    // Surname
    int index = 5;
    tmp = "Surname: ";
    for (; index < 44; index++) {
        if (line1.charAt(index) != '<') {
            tmp += line1.charAt(index);
        } else {
            break;
        }
    }
    result += tmp + "\n\n";

    // Given names
    tmp = "Given Names: ";
    index += 2;
    for (; index < 44; index++) {
        if (line1.charAt(index) != '<') {
            tmp += line1.charAt(index);
        } else {
            tmp += ' ';
        }
    }
    result += tmp + "\n\n";

    // Passport number
    tmp = "Passport number: ";
    index = 0;
    for (; index < 9; index++) {
        if (line2.charAt(index) != '<') {
            tmp += line2.charAt(index);
        } else {
            break;
        }
    }
    result += tmp + "\n\n";

    // Nationality
    tmp = "Nationality: ";
    tmp += line2.substring(10, 13);
    result += tmp + "\n\n";

    // Date of birth
    tmp = line2.substring(13, 19);
    tmp = tmp.substring(0, 2) +
            '/' +
            tmp.substring(2, 4) +
            '/' +
            tmp.substring(4, 6);
    tmp = "Date of birth (YYMMDD): " + tmp;
    result += tmp + "\n\n";

    // Sex
    tmp = "Sex: ";
    tmp += line2.charAt(20);
    result += tmp + "\n\n";

    // Expiration date of passport
    tmp = line2.substring(21, 27);
    tmp = tmp.substring(0, 2) +
            '/' +
            tmp.substring(2, 4) +
            '/' +
            tmp.substring(4, 6);
    tmp = "Expiration date of passport (YYMMDD): " + tmp;
    result += tmp + "\n\n";

    // Personal number
    if (line2.charAt(28) != '<') {
        tmp = "Personal number: ";
        for (index = 28; index < 42; index++) {
            if (line2.charAt(index) != '<') {
                tmp += line2.charAt(index);
            } else {
                break;
            }
        }
        result += tmp + "\n\n";
    }

    return result;
}
```

In addition, we save the gray-scale image to a PNG file:

```java
public static void saveFrame(byte[] data, int width, int height, String path) {
    Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ALPHA_8);
    bitmap.copyPixelsFromBuffer(ByteBuffer.wrap(data));
    try {
        FileOutputStream out = new FileOutputStream(path);
        bitmap.compress(Bitmap.CompressFormat.PNG, 100, out);
        out.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}
```

A new Activity is created to display the captured frame and passport MRZ result:

```java
public class ResultActivity extends AppCompatActivity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        Intent intent = getIntent();
        String message = intent.getStringExtra("path");
        String result = intent.getStringExtra("result");

        Bitmap bitmap = BitmapFactory.decodeFile(message);
        ImageView iv = new ImageView(this);
        iv.setImageBitmap(bitmap);
        setContentView(iv);

        Toast.makeText(this, result, Toast.LENGTH_LONG)
                .show();
    }
}
```

![Android OCR recognition for passport mrz](https://www.dynamsoft.com/codepool/img/2021/07/android-passport-mrz-ocr.jpg)

## Source Code
[https://github.com/yushulx/android-passport-mrz](https://github.com/yushulx/android-passport-mrz)