How to Build Desktop .NET MRZ Scanner for Passport and ID Card on Windows and Linux
Tags: dotnet, mr, ocr, winforms
Published: 2022-10-12 08:29:32

This article aims to help C# developers to build desktop .NET MRZ (Machine Readable Zone) scanner applications with [Dynamsoft C++ Label Recognizer SDK](https://www.dynamsoft.com/label-recognition/docs/introduction/?ver=latest). We firstly build a .NET class library for detecting MRZ zone and extracting MRZ text from passport, ID card, and travel documents. Then we demonstrate how to build console and GUI applications with the .NET class library on Windows and Linux.

## NuGet Package
[https://www.nuget.org/packages/MrzScannerSDK](https://www.nuget.org/packages/MrzScannerSDK)


## License Key
Get a [30-day FREE trial license](https://www.dynamsoft.com/customer/license/trialLicense/?product=dlr).

## Building .NET MRZ Scanner Library
In this section, we will create a .NET class library project containing MRZ detection and extraction functions.

### NuGet Package with .NET DLL, C++ Runtime Libraries and MRZ Detection model
We run the command `dotnet new classlib -o MrzScannerSDK` to create a .NET class library project, and then open the `MrzScannerSDK.csproj` file to configure the build.

To generate the NuGet package automatically when running `dotnet build`, add the following line to the `MrzScannerSDK.csproj` file:


```xml
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
```

The C++ OCR SDK contains some C++ runtime libraries (*.dll, *.so) and MRZ model files. We need to copy them to the output directory and package them into the NuGet package. 

```xml
<ItemGroup>
<None CopyToOutputDirectory="Always" Link="\%(Filename)%(Extension)" Include="lib/win/DynamicPdfx64.dll" Pack="true" PackagePath="runtimes/win-x64/native/DynamicPdfx64.dll" />
<None CopyToOutputDirectory="Always" Link="\%(Filename)%(Extension)" Include="lib/win/DynamsoftLabelRecognizerx64.dll" Pack="true" PackagePath="runtimes/win-x64/native/DynamsoftLabelRecognizerx64.dll"  />
<None CopyToOutputDirectory="Always" Link="\%(Filename)%(Extension)" Include="lib/win/DynamsoftLicenseClientx64.dll" Pack="true" PackagePath="runtimes/win-x64/native/DynamsoftLicenseClientx64.dll" />
<None CopyToOutputDirectory="Always" Link="\%(Filename)%(Extension)" Include="lib/win/vcomp140.dll" Pack="true" PackagePath="runtimes/win-x64/native/vcomp140.dll" />

<None CopyToOutputDirectory="Always" Link="\%(Filename)%(Extension)" Include="lib/linux/libDynamicPdf.so" Pack="true" PackagePath="runtimes/linux-x64/native/libDynamicPdf.so" />
<None CopyToOutputDirectory="Always" Link="\%(Filename)%(Extension)" Include="lib/linux/libDynamsoftLabelRecognizer.so" Pack="true" PackagePath="runtimes/linux-x64/native/libDynamsoftLabelRecognizer.so" />
<None CopyToOutputDirectory="Always" Link="\%(Filename)%(Extension)" Include="lib/linux/libDynamsoftLicenseClient.so" Pack="true" PackagePath="runtimes/linux-x64/native/libDynamsoftLicenseClient.so" />
</ItemGroup>

<ItemGroup>
<None Update="README.md">
    <Pack>True</Pack>
    <PackagePath>\</PackagePath>
</None>
<None Update="MRZ.json">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    <Pack>True</Pack>
    <PackagePath>\</PackagePath>
</None>
<None Update="model/**/*.*">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    <Pack>True</Pack>
    <PackagePath>\</PackagePath>
</None>
</ItemGroup>
```

You can try to build the project. The structure of the NuGet package should be like this:

![nuget package with *.dll, *.so and mrz model](https://www.dynamsoft.com/codepool/img/2022/10/nuget-package-dll-so-model.png)

To use the model, we need to modify the parameter `DirectoryPath` dynamically in the `MRZ.json` file. The detailed implementation will be introduced in the next paragraph.

### Encapsulating C++ MRZ Detection Functions in .NET Class Library
We use `DllImport` to import the primary C++ MRZ detection functions. The `DllImport` attribute specifies the name of the DLL, the name of the function, and the calling convention. 


- `DLR_InitLicense`: set the license key
    ```csharp
    [DllImport("DynamsoftLabelRecognizer")]
    static extern int DLR_InitLicense(string license, [Out] byte[] errorMsg, int errorMsgSize);
    ```
- `DLR_CreateInstance`: create an instance of the label recognizer
    ```csharp
    [DllImport("DynamsoftLabelRecognizer")]
    static extern IntPtr DLR_CreateInstance();
    ```
- `DLR_RecognizeByFile`: recognize the MRZ zone from the image file
    ```csharp
    [DllImport("DynamsoftLabelRecognizer")]
    static extern int DLR_RecognizeByFile(IntPtr handler, string sourceFilePath, string templateName);
    ```
- `DLR_RecognizeByBuffer`: recognize the MRZ zone from the image buffer
    ```csharp
    [DllImport("DynamsoftLabelRecognizer")]
    static extern int DLR_RecognizeByBuffer(IntPtr handler, IntPtr sourceImage, string templateName);
    ```
- `DLR_AppendSettingsFromFile`: load the MRZ detection model
    ```csharp
    [DllImport("DynamsoftLabelRecognizer")]
    static extern int DLR_AppendSettingsFromFile(IntPtr handler, string filename, [Out] byte[] errorMsg, int errorMsgSize);
    ```
- `DLR_GetAllResults`: get the MRZ text
    ```csharp
    [DllImport("DynamsoftLabelRecognizer")]
    static extern int DLR_GetAllResults(IntPtr hBarcode, ref IntPtr pDLR_ResultArray);
    ```
- `DLR_FreeResults`: free the memory of the MRZ text
    ```csharp
    [DllImport("DynamsoftLabelRecognizer")]
    static extern int DLR_FreeResults(ref IntPtr pDLR_ResultArray);
    ```

Meanwhile, we use `StructLayout` to define the C++ struct in C#. For example:
```csharp
[StructLayout(LayoutKind.Sequential, Pack = 1)]
internal struct DLR_LineResult
{
    public string lineSpecificationName;

    public string text;

    public string characterModelName;

    public Quadrilateral location;

    public int confidence;

    public int characterResultsCount;

    public IntPtr characterResults;

    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 64)]
    public char[] reserved;
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
internal struct DLR_Result
{
    public string referenceRegionName;

    public string textAreaName;

    public Quadrilateral location;

    public int confidence;

    public int lineResultsCount;

    public IntPtr lineResults;

    public int pageNumber;

    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 60)]
    public char[] reserved;
}
```

The next step is to encapsulate the C++ functions in C#. For example, a function for loading the MRZ detection model:
```csharp

public int LoadModel()
{
    if (handler == IntPtr.Zero) return -1;

    string dir = Directory.GetCurrentDirectory();
    string[] files = Directory.GetDirectories(dir, "model", SearchOption.AllDirectories);
    string modelPath = files[0];
    string config = Path.Join(modelPath.Split("model")[0], "MRZ.json");

    string contents = File.ReadAllText(config);

    JsonNode configNode = JsonNode.Parse(contents)!;

    if ((string)(configNode!["CharacterModelArray"]![0]!["DirectoryPath"]!) == "model")
    {
        configNode["CharacterModelArray"]![0]!["DirectoryPath"] = modelPath;

        var options = new JsonSerializerOptions { WriteIndented = true };

        contents = configNode.ToJsonString(options);

        File.WriteAllText(config, contents);
    }

    byte[] errorMsg = new byte[512];
    int ret = DLR_AppendSettingsFromFile(handler, config, errorMsg, 512);
    return ret;
}
```

As we mentioned above, the `MRZ.json` file is a JSON-formatted configuration file. The `DirectoryPath` parameter specifies the path of the MRZ detection model. We need to modify the `DirectoryPath` parameter dynamically to make it point to the absolute path of the MRZ detection model. `JsonNode` is a class used to deserialize string to JSON object and serialize JSON object to string. It simplifies the process of modifying the `DirectoryPath` parameter.

According to the C++ methods `DLR_RecognizeByFile()` and `DLR_RecognizeByBuffer()`, we create two corresponding C# methods `DetectFile()` and `DetectBuffer()`. The `DetectBuffer()` method is not as easy as `DetectFile()` because it requires the image buffer to be converted to an `ImageData` object before calling the `DLR_RecognizeByBuffer()` function:
```csharp
public Result[]? DetectBuffer(IntPtr pBufferBytes, int width, int height, int stride, ImagePixelFormat format)
{
    if (handler == IntPtr.Zero) return null;

    IntPtr pResultArray = IntPtr.Zero;

    ImageData imageData = new ImageData();
    imageData.width = width;
    imageData.height = height;
    imageData.stride = stride;
    imageData.format = format;
    imageData.bytesLength = stride * height;
    imageData.bytes = pBufferBytes;

    IntPtr pimageData = Marshal.AllocHGlobal(Marshal.SizeOf(imageData));
    Marshal.StructureToPtr(imageData, pimageData, false);
    int ret = DLR_RecognizeByBuffer(handler, pimageData, "locr");
    Marshal.FreeHGlobal(pimageData);

    return GetResults();
}
```

The `ImageData` object is defined as follows:


```csharp
[StructLayout(LayoutKind.Sequential, Pack = 1)]
internal struct ImageData
{
    public int bytesLength;

    public IntPtr bytes;

    public int width;

    public int height;

    public int stride;

    public ImagePixelFormat format;
}
```

After calling MRZ detection functions, we can get the MRZ text by calling the `GetResults()` method:
```csharp
private Result[]? GetResults()
{
    IntPtr pDLR_ResultArray = IntPtr.Zero;
    DLR_GetAllResults(handler, ref pDLR_ResultArray);

    if (pDLR_ResultArray != IntPtr.Zero)
    {
        List<Result> resultArray = new List<Result>();
        DLR_ResultArray? results = (DLR_ResultArray?)Marshal.PtrToStructure(pDLR_ResultArray, typeof(DLR_ResultArray));
        if (results != null)
        {
            int count = results.Value.resultsCount;

            if (count > 0)
            {
                IntPtr[] mrzResults = new IntPtr[count];
                Marshal.Copy(results.Value.results, mrzResults, 0, count);

                for (int i = 0; i < count; i++)
                {
                    DLR_Result result = (DLR_Result)Marshal.PtrToStructure(mrzResults[i], typeof(DLR_Result))!;
                    int lineResultsCount = result.lineResultsCount;
                    IntPtr lineResults = result.lineResults;
                    IntPtr[] lines = new IntPtr[lineResultsCount];
                    Marshal.Copy(lineResults, lines, 0, lineResultsCount);

                    for (int j = 0; j < lineResultsCount; j++)
                    {
                        Result mrzResult = new Result();
                        DLR_LineResult lineResult = (DLR_LineResult)Marshal.PtrToStructure(lines[j], typeof(DLR_LineResult))!;
                        mrzResult.Text = lineResult.text;
                        mrzResult.Confidence = lineResult.confidence;
                        DM_Point[] points = lineResult.location.points;
                        mrzResult.Points = new int[8];
                        for (int k = 0; k < 4; k++)
                        {
                            mrzResult.Points[k * 2] = points[k].coordinate[0];
                            mrzResult.Points[k * 2 + 1] = points[k].coordinate[1];
                        }

                        resultArray.Add(mrzResult);
                    }
                }
            }
        }

        DLR_FreeResults(ref pDLR_ResultArray);

        return resultArray.ToArray();
    }

    return null;
}
```

One more step is to extract the information from the MRZ text. We write the parsing logic based on [MRZ specification](https://en.wikipedia.org/wiki/Machine-readable_passport). The `Regex` class is helpful for matching and finding the information we need.

```csharp
public static JsonNode? Parse(string[] lines)
{
    JsonNode mrzInfo = new JsonObject();

    if (lines.Length == 0)
    {
        return null;
    }

    if (lines.Length == 2)
    {
        string line1 = lines[0];
        string line2 = lines[1];

        var type = line1.Substring(0, 1);
        if (!new Regex(@"[I|P|V]").IsMatch(type)) return null;
        if (type == "P")
        {
            mrzInfo["type"] = "PASSPORT (TD-3)";
        }
        else if (type == "V")
        {
            if (line1.Length == 44)
            {
                mrzInfo["type"] = "VISA (MRV-A)";
            }
            else if (line1.Length == 36)
            {
                mrzInfo["type"] = "VISA (MRV-B)";
            }
        }
        else if (type == "I")
        {
            mrzInfo["type"] = "ID CARD (TD-2)";
        }

        // Get issuing State infomation
        var nation = line1.Substring(2, 5);
        if (new Regex(@"[0-9]").IsMatch(nation)) return null;
        if (nation[nation.Length - 1] == '<') {
            nation = nation.Substring(0, 2);
        }
        mrzInfo["nationality"] = nation;
        ...
    }
    else if (lines.Length == 3)
    {
        string line1 = lines[0];
        string line2 = lines[1];
        string line3 = lines[2];
        var type = line1.Substring(0, 1);
        if (!new Regex(@"[I|P|V]").IsMatch(type)) return null;
        mrzInfo["type"] = "ID CARD (TD-1)";
        // Get nationality infomation
        var nation = line2.Substring(15, 3);
        if (new Regex(@"[0-9]").IsMatch(nation)) return null;
        nation = nation.Replace("<", "");
        mrzInfo["nationality"] = nation;
        // Get surname information
        ...
    }

    return mrzInfo;
}
```

As all C# methods are ready, we can build the package to `*.nupkg` file.

```bash
dotnet build --configuration Release
```

The package can be installed in Windows and Linux .NET project. 

## Desktop Console and GUI Applications for Scanning MRZ
The following paragraphs will introduce how to build a desktop console and GUI applications for scanning MRZ.

### Desktop Console Application
1. Create a new .NET console project in terminal.
    ```bash
    dotnet new console -o app
    ```
2. Set the license key acquired from [Dynamsoft customer portal](https://www.dynamsoft.com/customer/license/trialLicense?product=dlr).
    ```csharp
    MrzScanner.InitLicense("DLS2eyJoYW5kc2hha2VDb2RlIjoiMjAwMDAxLTE2NDk4Mjk3OTI2MzUiLCJvcmdhbml6YXRpb25JRCI6IjIwMDAwMSIsInNlc3Npb25QYXNzd29yZCI6IndTcGR6Vm05WDJrcEQ5YUoifQ==");
    ```
3. Create an instance of MRZ scanner.
    ```csharp
    MrzScanner scanner = MrzScanner.Create();
    ```
4. Load the MRZ detection model.
    ```csharp
    scanner.LoadModel();
    ```
5. Detect MRZ from an image file.
    ```csharp
    MrzScanner.Result[]? results = scanner.DetectFile("<image-file>");
    ```
6. Output the results.
    ```csharp
    if (results != null)
    {
        foreach (MrzScanner.Result result in results)
        {
            Console.WriteLine(result.Text);
            Console.WriteLine(result.Points[0] + ", " +result.Points[1] + ", " + result.Points[2] + ", " + result.Points[3] + ", " + result.Points[4] + ", " + result.Points[5] + ", " + result.Points[6] + ", " + result.Points[7]);
        }
    }
    ```

### WinForms Application
Using WinForms can create a fancy GUI application. Here is the UI designed with Visual Studio Toolbox.

![WinForm UI design](https://www.dynamsoft.com/codepool/img/2022/10/winform-ui-mrz-scanner.png)

- The menu allows users to enter a valid license key.
- The status bar shows the license activation status.
- The left picture box shows the original image and the right picture box shows the image with detected MRZ contours.
- The `Load File` button allows users to load an image file.
- The `Camera Scan` button allows users to scan MRZ from a camera.
- The first rich text box shows the parsed MRZ information and the second rich text box shows the image file list.

`OpenCvSharp4` is required for camera streaming and image decoding. Thus, add the following NuGet packages to the project.

```xml
<ItemGroup>
<PackageReference Include="OpenCvSharp4" Version="4.6.0.20220608" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="OpenCvSharp4.Extensions" Version="4.5.5.20211231" />
<PackageReference Include="OpenCvSharp4.runtime.win" Version="4.6.0.20220608" />
</ItemGroup>
```

To detect MRZ from an image file, we use `OpenFileDialog` to load images:

```csharp
private void button1_Click(object sender, EventArgs e)
{
    StopScan();
    using (OpenFileDialog dlg = new OpenFileDialog())
    {
        dlg.Title = "Open Image";
        dlg.Filter = "Image files (*.bmp, *.jpg, *.png) | *.bmp; *.jpg; *.png";

        if (dlg.ShowDialog() == DialogResult.OK)
        {
            listBox1.Items.Add(dlg.FileName);
            MrzDetection(dlg.FileName);
        }
    }
}
```

Use OpenCV to decode the image to `Mat` and then use `MrzScanner` to detect MRZ.

```csharp
private void MrzDetection(string filename)
{
    try
    {
        _mat = Cv2.ImRead(filename, ImreadModes.Color);
        Mat copy = new Mat(_mat.Rows, _mat.Cols, MatType.CV_8UC3);
        _mat.CopyTo(copy);
        pictureBox1.Image = BitmapConverter.ToBitmap(copy);
        pictureBox2.Image = DecodeMat(copy);
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.Message);
    }
}

private Bitmap DecodeMat(Mat mat)
{
    _results = scanner.DetectBuffer(mat.Data, mat.Cols, mat.Rows, (int)mat.Step(), MrzScanner.ImagePixelFormat.IPF_RGB_888);
    if (_results != null)
    {
        string[] lines = new string[_results.Length];
        var index = 0;
        foreach (Result result in _results)
        {
            lines[index++] = result.Text;
            richTextBox1.Text += result.Text + Environment.NewLine;
            if (result.Points != null)
            {
                Point[] points = new Point[4];
                for (int i = 0; i < 4; i++)
                {
                    points[i] = new Point(result.Points[i * 2], result.Points[i * 2 + 1]);
                }
                Cv2.DrawContours(mat, new Point[][] { points }, 0, Scalar.Red, 2);
            }
        }

        JsonNode? info = Parse(lines);
        if (info != null) richTextBox1.Text = info.ToString();
    }

    Bitmap bitmap = BitmapConverter.ToBitmap(mat);
    return bitmap;
}
```

To detect MRZ from a camera, we start a thread to keep capturing images from the camera and then use `MrzScanner` to detect MRZ.

```csharp
private void button2_Click(object sender, EventArgs e)
{
    if (!capture.IsOpened())
    {
        MessageBox.Show("Failed to open video stream or file");
        return;
    }

    if (button2.Text == "Camera Scan")
    {
        StartScan();
    }
    else
    {
        StopScan();
    }
}

private void StartScan()
{
    button2.Text = "Stop";
    isCapturing = true;
    thread = new Thread(new ThreadStart(FrameCallback));
    thread.Start();
}

private void StopScan()
{
    button2.Text = "Camera Scan";
    isCapturing = false;
    if (thread != null) thread.Join();
}

private void FrameCallback()
{
    while (isCapturing)
    {
        capture.Read(_mat);
        Mat copy = new Mat(_mat.Rows, _mat.Cols, MatType.CV_8UC3);
        _mat.CopyTo(copy);
        pictureBox1.Image = DecodeMat(copy);
    }
}
```

Finally, run the desktop .NET MRZ scanner:

```bash
dotnet run
```

![Winform .NET MRZ scanner](https://www.dynamsoft.com/codepool/img/2022/10/dotnet-mrz-scanner.png)

## Source Code
[https://github.com/yushulx/dotnet-mrz-sdk](https://github.com/yushulx/dotnet-mrz-sdk)