How to Enhance Passport MRZ Detection in Python by Correcting Image Orientation
Tags: python, mrz, passport, computervision
Published: 2023-09-13 02:03:46

**Passport Machine Readable Zone (MRZ)** detection is sensitive to the orientation of the passport. If the passport is not right side up, the MRZ detection rate will be low. In this article, we will discuss how to improve the MRZ detection rate from rotated images with Python. Edge detection, perspective transformation and face detection will be used to correct the orientation of the passport.

## Installation
```bash
pip install mrz-scanner-sdk document-scanner-sdk dlib mediapipe retina-face opencv-python
```

- [mrz-scanner-sdk](https://pypi.org/project/mrz-scanner-sdk/): Dynamsoft MRZ Scanner SDK for MRZ detection. A valid license key is required to use the SDK. You can get a free trial license from [here](https://www.dynamsoft.com/customer/license/trialLicense?product=dlr&package=c_cpp&source=codepool).
- [document-scanner-sdk](https://pypi.org/project/document-scanner-sdk/): Dynamsoft Document Scanner SDK for edge detection and perspective transformation.
- [dlib](https://pypi.org/project/dlib/): An open-source software library that provides highly accurate and efficient face detection algorithm.
- [mediapipe](https://pypi.org/project/mediapipe/): A Google-developed, open-source, cross-platform framework designed for rapid, real-time face detection.
- [retina-face](https://pypi.org/project/retina-face/): A deep learning based cutting-edge facial detector for Python coming with facial landmarks.
- [opencv-python](https://pypi.org/project/opencv-python/): Used to display images and draw lines.

## Passport Edge Detection and Perspective Transformation
Let's get started with a passport image taken in the correct orientation.

![passport image](https://www.dynamsoft.com/codepool/img/2023/09/passport.jpg)

Using the following Python code can successfully detect the MRZ area:

```python
import argparse
import mrzscanner
import sys
import numpy as np

def scanmrz():
    parser = argparse.ArgumentParser(description='Scan MRZ info from a given image')
    parser.add_argument('filename')
    parser.add_argument('-l', '--license', default='', type=str, help='Set a valid license key')
    args = parser.parse_args()
    try:
        filename = args.filename
        license = args.license
        
        # set license
        if  license == '':
            mrzscanner.initLicense("LICENSE-KEY")
        else:
            mrzscanner.initLicense(license)
            
        scanner = mrzscanner.createInstance()
        scanner.loadModel(mrzscanner.load_settings())
        
        import cv2
        image = cv2.imread(filename)
        results = scanner.decodeMat(image)
        for result in results:
            print(result.text)
            s += result.text + '\n'
            x1 = result.x1
            y1 = result.y1
            x2 = result.x2
            y2 = result.y2
            x3 = result.x3
            y3 = result.y3
            x4 = result.x4
            y4 = result.y4
            
            cv2.drawContours(image, [np.int0([(x1, y1), (x2, y2), (x3, y3), (x4, y4)])], 0, (0, 255, 0), 2)
            cv2.putText(image, result.text, (x1, y1), cv2.FONT_HERSHEY_SIMPLEX, 0.4, (0, 0, 255), 2)
        
        cv2.imshow("MRZ Detection", image)
        cv2.waitKey(0)
            
    except Exception as err:
        print(err)
        sys.exit(1)

if __name__ == "__main__":
    scanmrz()
```

![passport image](https://www.dynamsoft.com/codepool/img/2023/09/python-passport-mrz-detection.png)

If the image is rotated at a significant angle, MRZ detection may fail.

![rotated passport image](https://www.dynamsoft.com/codepool/img/2023/09/rotated-passport.jpg) 

To address this issue, we can use edge detection and perspective transformation to correct the orientation of the passport.

![rotated passport mrz detection](https://www.dynamsoft.com/codepool/img/2023/09/rotated-passport-mrz-detection.png)

Here are the steps:
1. Initialize the document scanner:
    
    ```python
    import docscanner
    doc_scanner = docscanner.createInstance()
    doc_scanner.setParameters(docscanner.Templates.color)
    ```
2. Detect the edges of the passport:
    
    ```python
    results = doc_scanner.detectMat(image)
    result = results[0]
    x1 = result.x1
    y1 = result.y1
    x2 = result.x2
    y2 = result.y2
    x3 = result.x3
    y3 = result.y3
    x4 = result.x4
    y4 = result.y4
    ```

3. Rectify the passport image:
    
    ```python
    rectified_document = doc_scanner.normalizeBuffer(image, x1, y1, x2, y2, x3, y3, x4, y4)
    rectified_document = docscanner.convertNormalizedImage2Mat(rectified_document)
    ```

4. Detect the MRZ area from the rectified passport image:
    
    ```python
    def detect_mrz(image, scanner):
        s = ""
        results = scanner.decodeMat(image)
        for result in results:
            # print(result.text)
            s += result.text + '\n'
            x1 = result.x1
            y1 = result.y1
            x2 = result.x2
            y2 = result.y2
            x3 = result.x3
            y3 = result.y3
            x4 = result.x4
            y4 = result.y4
            
            cv2.drawContours(image, [np.int0([(x1, y1), (x2, y2), (x3, y3), (x4, y4)])], 0, (0, 255, 0), 2)
            cv2.putText(image, result.text, (x1, y1), cv2.FONT_HERSHEY_SIMPLEX, 0.4, (0, 0, 255), 2)
            
        cv2.imshow("MRZ Detection", image)
    
    detect_mrz(rectified_document, mrz_scanner)
    ```

## Rotating Images Based on Facial Orientation
After perspective transformation, the image may be oriented in one of four directions: 0 degrees, 90 degrees, 180 degrees, or 270 degrees. 

![rotated passport](https://www.dynamsoft.com/codepool/img/2023/09/rotated-mrz-documents.png)

If you run the code above, you will find only the 0-degree orientation allows for normal MRZ detection. Thus, we aim to rotate the other three orientations to this correct angle. Considering that the orientation of the face on the passport is consistent with that of the Machine-Readable Zone, we can use face detection to rotate the image accordingly.

Numerous face detection algorithms exist, each with varying levels of performance. In this article, we will compare the effectiveness of three prominent algorithms: **Dlib**, **MediaPipe**, and **RetinaFace**.

### Dlib

1. Download the pre-trained model from [here](http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2).
2. Unzip the file and put it in the same folder as the Python script.
3. Create the Dlib face detector:
    
    ```python
    import dlib
    detector = dlib.get_frontal_face_detector()
    predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")
    ```
4. Detect the faces from the rectified passport image:
    
    ```python
    mg = cv2.imread(filename)  
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    start_time = time.time()
    faces = detector(gray)
    end_time = time.time()
    print("Elapsed Time:", end_time - start_time)
    ```

![dlib face detection](https://www.dynamsoft.com/codepool/img/2023/09/dlib-face-detection.png)

The dlib face detection algorithm is typically trained on datasets where faces are upright or near-upright. The features learned by the classifier assume that the faces in the images will be oriented in a specific way, usually right side up. When a face is rotated significantly (like upside-down or tilted at 90 degrees), the learned features may not match well, making it difficult for the algorithm to detect the face.

### Mediapipe
1. Download the pre-trained model from [here](https://developers.google.com/mediapipe/solutions/vision/face_detector/index#models). At present, only [BlazeFace (short-range)](https://storage.googleapis.com/mediapipe-models/face_detector/blaze_face_short_range/float16/latest/blaze_face_short_range.tflite) is available, which is a lightweight model for detecting single or multiple faces.

2. Put the model in the same folder as the Python script.
3. Create the MediaPipe face detector:
    
    ```python
    import mediapipe as mp
    from mediapipe.tasks import python
    from mediapipe.tasks.python import vision
    
    mp_face_detection = mp.solutions.face_detection
    mp_drawing = mp.solutions.drawing_utils
    
    base_options = python.BaseOptions(model_asset_path='blaze_face_short_range.tflite')
    options = vision.FaceDetectorOptions(base_options=base_options)
    detector = vision.FaceDetector.create_from_options(options)
    ```

4. Detect the faces from the rectified passport image:
    
    ```python
    img = cv2.imread(filename)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    image = mp.Image(image_format=mp.ImageFormat.SRGB, data=img)
    
    start_time = time.time()
    detection_result = detector.detect(image)
    end_time = time.time()
    print("Elapsed Time:", end_time - start_time)
    ```

![mediapipe face detection](https://www.dynamsoft.com/codepool/img/2023/09/mediapipe-face-detection.png)

Compared to Dlib, Mediapipe is faster and more accurate. However, it still falls short of our requirements because it fails to detect some facial landmarks correctly.

### Retinaface

RetinaFace is a deep learning-based face detection model aimed at identifying faces in images with high accuracy. Let's explore whether it meets our objectives.

```python
from retinaface import RetinaFace
img = cv2.imread(filename)
obj = RetinaFace.detect_faces(img_path=img)

if type(obj) == dict:
    for key in obj:
        identity = obj[key]

        facial_area = identity["facial_area"]
        facial_img = img[facial_area[1]: facial_area[3],
                            facial_area[0]: facial_area[2]]

        landmarks = identity["landmarks"]
        left_eye = landmarks["left_eye"]
        right_eye = landmarks["right_eye"]
        nose = landmarks["nose"]
        mouth_right = landmarks["mouth_right"]
        mouth_left = landmarks["mouth_left"]

        cv2.rectangle(img, (facial_area[0], facial_area[1]),
                        (facial_area[2], facial_area[3]), (0, 255, 0), 2)
        cv2.circle(img, (int(left_eye[0]), int(
            left_eye[1])), 2, (255, 0, 0), 2)
        cv2.circle(img, (int(right_eye[0]), int(
            right_eye[1])), 2, (0, 0, 255), 2)
        cv2.circle(img, (int(nose[0]), int(nose[1])), 2, (0, 255, 0), 2)
        cv2.circle(img, (int(mouth_left[0]), int(
            mouth_left[1])), 2, (0, 155, 255), 2)
        cv2.circle(img, (int(mouth_right[0]), int(
            mouth_right[1])), 2, (0, 155, 255), 2)

cv2.imshow(filename, img)
```

![retina face detection](https://www.dynamsoft.com/codepool/img/2023/09/retina-face-detection.png)

RetinaFace takes the longest time for face detection, but it is the most accurate. It correctly identifies facial landmarks in all four directions, which we can use to rotate the image.

```python
def rotate(img, left_eye, right_eye, nose):

    nose_x, nose_y = nose
    left_eye_x, left_eye_y = left_eye
    right_eye_x, right_eye_y = right_eye

    if (nose_y > left_eye_y) and (nose_y > right_eye_y):
        return img # no need to rotate
    elif (nose_y < left_eye_y) and (nose_y < right_eye_y):
        return cv2.flip(img, flipCode=-1) # 180 degrees
    elif (nose_x < left_eye_x) and (nose_x < right_eye_x):
        transposed = cv2.transpose(img)
        return cv2.flip(transposed, flipCode=0) # 90 degrees 
    else:
        transposed = cv2.transpose(img)
        return cv2.flip(transposed, flipCode=1) # 270 degrees 
```

## Combining Document Detection and Retina Face Detection for MRZ Detection
We can now combine the above steps to detect the MRZ area in rotated passport images.

```python
import argparse
import mrzscanner
import sys
import numpy as np
import cv2
import docscanner
import time
import face_retina

def detect_mrz(image, scanner):
    ...

def detect_doc(image, scanner):
    ...
        
    return mat

def scanmrz():
    parser = argparse.ArgumentParser(description='Scan MRZ info from a given image')
    parser.add_argument('filename')
    parser.add_argument('-l', '--license', default='', type=str, help='Set a valid license key')
    args = parser.parse_args()
    try:
        filename = args.filename
        license = args.license
        
        # set license
        if  license == '':
            defaultLicense = "LICENSE-KEY"
            mrzscanner.initLicense(defaultLicense)
            docscanner.initLicense(defaultLicense)
        else:
            mrzscanner.initLicense(license)
            docscanner.initLicense(license)
            
        mrz_scanner = mrzscanner.createInstance()
        mrz_scanner.loadModel(mrzscanner.load_settings())
        doc_scanner = docscanner.createInstance()
        doc_scanner.setParameters(docscanner.Templates.color)
        
        image = cv2.imread(filename)
        copy = image.copy()
        copy = detect_doc(copy, doc_scanner)
        copy = face_retina.detect(copy)
        detect_mrz(copy, mrz_scanner)
            
        cv2.imshow("Original", image)
            
            
    except Exception as err:
        print(err)
        sys.exit(1)

if __name__ == "__main__":
    scanmrz()
    cv2.waitKey(0)
```

![passport mrz detection in any orientation](https://www.dynamsoft.com/codepool/img/2023/09/passport-mrz-detection-any-orientation.png)

## Source Code
[https://github.com/yushulx/python-mrz-scanner-sdk/tree/main/examples/enhanced](https://github.com/yushulx/python-mrz-scanner-sdk/tree/main/examples/enhanced)