Integrating Document and MRZ Detection SDK into .NET MAUI for Windows
Tags: csharp, dotnet, maui, windows
Published: 2024-01-18 08:39:37

The [Capture.Vision.Maui](https://www.nuget.org/packages/Capture.Vision.Maui) NuGet package aims to equip developers with a .NET MAUI library, featuring a camera view and various image processing functions. This library streamlines the development of .NET MAUI applications across multiple platforms, including Windows, Android, and iOS. Previously, we introduced [barcode scanning capabilities](https://www.dynamsoft.com/codepool/dotnet-maui-camera-barcode-vision-plugin.html) within the .NET MAUI library. In this article, we will focus on integrating document and Machine-Readable Zone (MRZ) detection into .NET MAUI for Windows applications.


## Demo: Detecting Barcodes, Documents, and MRZ in .NET MAUI Windows Applications
<video src="https://github.com/yushulx/Capture-Vision-Maui/assets/2202306/63022269-a862-44a3-9217-bc6a9a2866be" controls="controls" muted="muted" style="max-height:640px; min-height: 200px; max-width: 100%;"></video>

## NuGet Package
[https://www.nuget.org/packages/Capture.Vision.Maui](https://www.nuget.org/packages/Capture.Vision.Maui)

## Integrating Document Detection SDK 
1. Add the [DocumentScannerSDK](https://www.nuget.org/packages/DocumentScannerSDK) NuGet package to your .NET MAUI project's `.csproj` file.
    
    ```xml

    <ItemGroup Condition="$([MSBuild]::IsOSPlatform('windows'))">
    	<PackageReference Include="DocumentScannerSDK " Version="1.1.0" />
    </ItemGroup>
    
    ```

    Note: When installing NuGet packages via the NuGet command line or the NuGet package explorer, the target platform is not specified. Since this package is only available for Windows, you need to manually add the platform condition.

2. Create a new class file named `DocumentResult.cs`. This class, `DocumentResult`, will contain the results of the document detection, including the confidence level, coordinates (points), and the rectified image data.

    ```csharp
    using static Dynamsoft.DocumentScanner;

    namespace Capture.Vision.Maui
    {
        public class DocumentResult
        {
            public int Confidence { get; set; }
    
            public int[] Points { get; set; }
    
            public int Width;
    
            public int Height;
    
            public int Stride;
    
            public ImagePixelFormat Format;
    
            public byte[] Data { get; set; }
        }
    }

    ```

3. In the `CameraView.cs` file, add new `BindableProperty` variables to enable switching the document detection mode on and off.

    ```csharp
    public static readonly BindableProperty EnableDocumentDetectProperty = BindableProperty.Create(nameof(EnableDocumentDetectProperty), typeof(bool), typeof(CameraView), false);
    public static readonly BindableProperty EnableDocumentRectifyProperty = BindableProperty.Create(nameof(EnableDocumentRectifyProperty), typeof(bool), typeof(CameraView), false);

    public bool EnableDocumentDetect
    {
        get { return (bool)GetValue(EnableDocumentDetectProperty); }
        set { SetValue(EnableDocumentDetectProperty, value); }
    }
    
    public bool EnableDocumentRectify
    {
        get { return (bool)GetValue(EnableDocumentRectifyProperty); }
        set { SetValue(EnableDocumentRectifyProperty, value); }
    }
    ```

4. Open the file `Platforms/Windows/NativeCameraView.cs`. Incorporate the document detection code into the `ProcessFrames()` function.
    
    ```csharp
    private void ProcessFrames() {
        ...
        
        if (cameraView.EnableDocumentDetect)
        {
            DocumentScanner.Result[] results = documentScanner.DetectBuffer(buffer, bitmap.PixelWidth, bitmap.PixelHeight, bitmap.PixelWidth, DocumentScanner.ImagePixelFormat.IPF_GRAYSCALED);
            DocumentResult documentResults = new DocumentResult();
            if (results != null && results.Length > 0)
            {
                documentResults = new DocumentResult
                {
                    Confidence = results[0].Confidence,
                    Points = results[0].Points
                };
        
                if (cameraView.EnableDocumentRectify)
                {
                    NormalizedImage image = documentScanner.NormalizeBuffer(buffer, bitmap.PixelWidth, bitmap.PixelHeight, bitmap.PixelWidth, DocumentScanner.ImagePixelFormat.IPF_GRAYSCALED, documentResults.Points);
                    documentResults.Width = image.Width;
                    documentResults.Height = image.Height;
                    documentResults.Stride = image.Stride;
                    documentResults.Format = image.Format;
                    documentResults.Data = image.Data;
                }
            }
        
            cameraView.NotifyResultReady(documentResults, bitmap.PixelWidth, bitmap.PixelHeight);
        
        }

        ...
    }
    
    ```

## Integrating MRZ Detection SDK 
1. Add the [MrzScannerSDK](https://www.nuget.org/packages/MrzScannerSDK) NuGet package to your .NET MAUI project's `.csproj` file.

    ```xml
    <ItemGroup Condition="$([MSBuild]::IsOSPlatform('windows'))">
    	<PackageReference Include="MrzScannerSDK" Version="1.3.3" />
    </ItemGroup>
    ```

2. Create a new class file named `MrzResult.cs`. This class, `MrzResult`, will store the machine-readable zone (MRZ) detection results, which include the detected text line and parsed MRZ data. This data encompasses various elements such as type, nationality, surname, given name, passport number, issuing country, date of birth, gender, and expiration date.
    ```csharp
    namespace Capture.Vision.Maui
    {
        public class Line
        {
            public int Confidence { get; set; }
    
            public string Text { get; set; }
    
            public int[] Points { get; set; }
        }
    
        public class MrzResult
        {
            public string Type { get; set; } = "N/A";
            public string Nationality { get; set; } = "N/A";
            public string Surname { get; set; } = "N/A";
            public string GivenName { get; set; } = "N/A";
            public string PassportNumber { get; set; } = "N/A";
            public string IssuingCountry { get; set; } = "N/A";
            public string BirthDate { get; set; } = "N/A";
            public string Gender { get; set; } = "N/A";
            public string Expiration { get; set; } = "N/A";
            public string Lines { get; set; } = "N/A";
    
            public Line[] RawData { get; set; }
        }
    }

    ```

3. In the `CameraView.cs` file, add a `BindableProperty` variable for toggling the MRZ detection mode.

    ```csharp
    public static readonly BindableProperty EnableMrzProperty = BindableProperty.Create(nameof(EnableMrzProperty), typeof(bool), typeof(CameraView), false);

    public bool EnableMrz
    {
        get { return (bool)GetValue(EnableMrzProperty); }
        set { SetValue(EnableMrzProperty, value); }
    }
    ```

4. Open the file `Platforms/Windows/NativeCameraView.cs` and integrate the MRZ detection code into the ProcessFrames() function.
    
    ```csharp
    private void ProcessFrames() {
        ...
        
        if (cameraView.EnableMrz)
        {
            MrzScanner.Result[] results = mrzScanner.DetectBuffer(buffer, bitmap.PixelWidth, bitmap.PixelHeight, bitmap.PixelWidth, MrzScanner.ImagePixelFormat.IPF_GRAYSCALED);
            MrzResult mrzResults = new MrzResult();
        
            if (results != null && results.Length > 0)
            {
                Line[] rawData = new Line[results.Length];
                string[] lines = new string[results.Length];
        
                for (int i = 0; i < results.Length; i++)
                {
                    rawData[i] = new Line()
                    {
                        Confidence = results[i].Confidence,
                        Text = results[i].Text,
                        Points = results[i].Points,
                    };
                    lines[i] = results[i].Text;
                }
        
                try
                {
                    Dynamsoft.MrzResult info = MrzParser.Parse(lines);
                    mrzResults = new MrzResult()
                    {
                        RawData = rawData,
                        Type = info.Type,
                        Nationality = info.Nationality,
                        Surname = info.Surname,
                        GivenName = info.GivenName,
                        PassportNumber = info.PassportNumber,
                        IssuingCountry = info.IssuingCountry,
                        BirthDate = info.BirthDate,
                        Gender = info.Gender,
                        Expiration = info.Expiration,
                        Lines = info.Lines
                    };
                }
                catch (Exception ex) { 
                    
                }
            }
        
            cameraView.NotifyResultReady(mrzResults, bitmap.PixelWidth, bitmap.PixelHeight);
        }
        ...
    }
    
    ```
    

## Building a .NET MAUI Windows App with Document and MRZ Detection
In the upcoming section, we're going to construct a .NET MAUI Windows application that is equipped with both document and MRZ (Machine-Readable Zone) detection features.

1. Create a new .NET MAUI project in Visual Studio and install the `Capture.Vision.Maui` and `SkiaSharp.Views.Maui.Controls` NuGet packages. The `SkiaSharp.Views.Maui.Controls` package is used for drawing the detection results.
    
    ```xml
    <ItemGroup>
    	<PackageReference Include="SkiaSharp.Views.Maui.Controls" Version="2.88.3" />
      <PackageReference Include="Capture.Vision.Maui" Version="1.2.1" />
    </ItemGroup>
    ```

2. In `MauiProgram.cs`, add the necessary code to register the `Capture.Vision.Maui` and `SkiaSharp.Views.Maui.Controls` packages.

    ```csharp

    using Capture.Vision.Maui;
    using SkiaSharp.Views.Maui.Controls.Hosting;
    ...
    var builder = MauiApp.CreateBuilder();
    builder.UseSkiaSharp().UseNativeCameraView()
    ```

3. Request a [trial license](https://www.dynamsoft.com/customer/license/trialLicense) for the Document Scanner SDK and MRZ Scanner SDK. Then, in the `MainPage.xaml.cs` file, set the license keys for Windows using directives.

    ```csharp
    using Dynamsoft;

    namespace Capture.Vision.Maui.Example
    {
        public partial class MainPage : ContentPage
        {
            public MainPage()
            {
                InitializeComponent();
                InitService();
            }
    
            private async void InitService()
            {
                await Task.Run(() =>
                {
                    BarcodeQRCodeReader.InitLicense("LICENSEK-KEY    ");
    #if WINDOWS
        DocumentScanner.InitLicense("LICENSEK-KEY"); 
        MrzScanner.InitLicense("LICENSEK-KEY"); 
    #elif ANDROID
    #elif IOS
    #endif
                    return Task.CompletedTask;
                });
            }
        }
    }
    ```
    
4. Create a new content page and incorporate the `CameraView` control. Ensure that the `EnableBarcode`, `EnableDocumentDetect`, and `EnableMrz` properties are set to `True`. Use `SKCanvasView` to render the detection results.

    ```xml
    <?xml version="1.0" encoding="utf-8" ?>
    <ContentPage xmlns="http://schemas.microsoft.com/dotnet/2021/maui"
                 xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
                 xmlns:skia="clr-namespace:SkiaSharp.Views.Maui.Controls;assembly=SkiaSharp.Views.Maui.Controls"
                 xmlns:cv="clr-namespace:Capture.Vision.Maui;assembly=Capture.Vision.Maui"
                 x:Class="Capture.Vision.Maui.Example.CameraPage"
                 Title="CameraPage">
        <ScrollView>
            <Grid>
                <cv:CameraView x:Name="cameraView" HorizontalOptions="FillAndExpand" "EnableBarcode="True" EnableDocumentDetect="True" EnableMrz="True"
    			VerticalOptions="FillAndExpand" ResultReady="cameraView_ResultReady" FrameReady="cameraView_FrameReady"/>
                <skia:SKCanvasView x:Name="canvasView" 
               Margin="0"
               HorizontalOptions="FillAndExpand" VerticalOptions="FillAndExpand"
               PaintSurface="OnCanvasViewPaintSurface" />
            </Grid>
        </ScrollView>
    </ContentPage>
    ```

5. In the `CameraPage.xaml.cs` file, add code to handle the `FrameReady` and `ResultReady` events. 

    ```csharp
    BarcodeQrData[] barcodeData = null;
    DocumentData documentData = null;
    MrzData mrzData = null;

    private void cameraView_FrameReady(object sender, FrameReadyEventArgs e)
    {
        MainThread.BeginInvokeOnMainThread(() =>
        {
            canvasView.InvalidateSurface();
        });
    }

    private void cameraView_ResultReady(object sender, ResultReadyEventArgs e)
    {
        lock (_lockObject)
        {
    
            if (e.Result is BarcodeResult[])
            {
                barcodeData = null;
                BarcodeResult[] barcodeResults = (BarcodeResult[])e.Result;
            }
            else if (e.Result is DocumentResult)
            {
                documentData = null;
                DocumentResult documentResult = (DocumentResult)e.Result;

                if (documentResult.Points != null)
                {
                    documentData = new DocumentData()
                    {
                        Reference = documentResult
                    };
                    int[] coordinates = documentData.Reference.Points;
                    if (coordinates != null && coordinates.Length == 8)
                    {
                        documentData.Points = new SKPoint[4];
                
                        for (int i = 0; i < 4; ++i)
                        {
                            SKPoint p = new SKPoint();
                            p.X = coordinates[i * 2];
                            p.Y = coordinates[i * 2 + 1];
                            documentData.Points[i] = p;
                
                            if (orientation == DisplayOrientation.Portrait)
                            {
                                documentData.Points[i] = rotateCW90(documentData.Points[i], imageHeight);
                            }
                
                            documentData.Points[i].X = (float)(documentData.Points[i].X / scale);
                            documentData.Points[i].Y = (float)(documentData.Points[i].Y / scale);
                        }
                    }
                }
    
            }
            else if (e.Result is MrzResult)
            {
                mrzData = null;
                MrzResult mrzResult = (MrzResult)e.Result;
                
                if (mrzResult.RawData != null)
                {
                    mrzData = new MrzData()
                    {
                        Reference = mrzResult
                    };
                
                    Line[] rawData = mrzData.Reference.RawData;
                    mrzData.Points = new SKPoint[rawData.Length][];
                    for (int index = 0; index < rawData.Length; index++)
                    {
                        Line line = rawData[index];
                        int[] coordinates = line.Points;
                        mrzData.Points[index] = new SKPoint[4];
                        for (int i = 0; i < 4; ++i)
                        {
                            SKPoint p = new SKPoint();
                            p.X = coordinates[i * 2];
                            p.Y = coordinates[i * 2 + 1];
                            mrzData.Points[index][i] = p;
                
                            if (orientation == DisplayOrientation.Portrait)
                            {
                                mrzData.Points[index][i] = rotateCW90(mrzData.Points[index][i], imageHeight);
                            }
                
                            mrzData.Points[index][i].X = (float)(mrzData.Points[index][i].X / scale);
                            mrzData.Points[index][i].Y = (float)(mrzData.Points[index][i].Y / scale);
                        }
                    }
                }
            }
        }
    }
    ```

    Invoke `canvasView.InvalidateSurface()` to trigger the `OnCanvasViewPaintSurface` function, which will draw the detection results.

    ```csharp
    void OnCanvasViewPaintSurface(object sender, SKPaintSurfaceEventArgs args)
    {    
        SKImageInfo info = args.Info;
        SKSurface surface = args.Surface;
        SKCanvas canvas = surface.Canvas;
    
        canvas.Clear();
        
        // Draw detection results
        lock (_lockObject)
        {
            if (documentData != null && cameraView.EnableDocumentDetect)
            {
                SKPaint skPaint = new SKPaint
                {
                    Style = SKPaintStyle.Stroke,
                    Color = SKColors.Red,
                    StrokeWidth = 10,
                };
        
                SKPaint textPaint = new SKPaint
                {
                    Style = SKPaintStyle.Stroke,
                    Color = SKColors.Red,
                    TextSize = (float)(16 * density),
                    StrokeWidth = 1,
                };
        
                canvas.DrawText("Detected Document", documentData.Points[0], textPaint);
                canvas.DrawLine(documentData.Points[0], documentData.Points[1], skPaint);
                canvas.DrawLine(documentData.Points[1], documentData.Points[2], skPaint);
                canvas.DrawLine(documentData.Points[2], documentData.Points[3], skPaint);
                canvas.DrawLine(documentData.Points[3], documentData.Points[0], skPaint);
            }
        
            if (mrzData != null && cameraView.EnableMrz)
            {
                SKPaint skPaint = new SKPaint
                {
                    Style = SKPaintStyle.Stroke,
                    Color = SKColors.Yellow,
                    StrokeWidth = 10,
                };
        
                SKPaint textPaint = new SKPaint
                {
                    Style = SKPaintStyle.Stroke,
                    Color = SKColors.Yellow,
                    TextSize = (float)(16 * density),
                    StrokeWidth = 1,
                };
        
                foreach (SKPoint[] line in mrzData.Points) {
                    canvas.DrawLine(line[0], line[1], skPaint);
                    canvas.DrawLine(line[1], line[2], skPaint);
                    canvas.DrawLine(line[2], line[3], skPaint);
                    canvas.DrawLine(line[3], line[0], skPaint);
                }
        
                SKPoint start = mrzData.Points[0][0];
                int delta = 20;
                start.X += 200;
                start.Y -= 200;
                canvas.DrawText(mrzData.Reference.Type, start, textPaint);
                start.Y += delta;
                canvas.DrawText(mrzData.Reference.Nationality, start, textPaint);
                start.Y += delta;
                canvas.DrawText(mrzData.Reference.Surname, start, textPaint);
                start.Y += delta;
                canvas.DrawText(mrzData.Reference.GivenName, start, textPaint);
                start.Y += delta;
                canvas.DrawText(mrzData.Reference.PassportNumber, start, textPaint);
                start.Y += delta;
                canvas.DrawText(mrzData.Reference.IssuingCountry, start, textPaint);
                start.Y += delta;
                canvas.DrawText(mrzData.Reference.BirthDate, start, textPaint);
                start.Y += delta;
                canvas.DrawText(mrzData.Reference.Gender, start, textPaint);
                start.Y += delta;
                canvas.DrawText(mrzData.Reference.Expiration, start, textPaint);
            }
        }
    }
    ```

6. Finally, run the .NET MAUI Windows application.
    
    ![.NET MAUI barcode document MRZ detection](https://www.dynamsoft.com/codepool/img/2024/01/dotnet-maui-barcode-document-mrz.png)

## Source Code
[https://github.com/yushulx/Capture-Vision-Maui](https://github.com/yushulx/Capture-Vision-Maui)