Empowering .NET MAUI Android Apps with Document and MRZ Detection
Tags: dotnet, maui, android, programming
Published: 2024-01-25 07:54:16

Last week, we enhanced the .NET MAUI library [Capture.Vision.Maui](https://www.nuget.org/packages/Capture.Vision.Maui) by adding document and MRZ (Machine-Readable Zone) detection capabilities for the Windows platform. We also enabled our .NET MAUI application to scan barcodes, documents, and MRZs in Windows. This week, we will further enhance the library by adding Android support. We will outline the entire process, from the Android AAR package to the .NET library package, and finally to the .NET MAUI library package. After upgrading the .NET MAUI library with integrated, platform-specific code for Android, your .NET MAUI application will be able to detect QR codes, documents, and MRZs from camera images without modifying any C# code.


## Demo: Scanning QR Codes, Documents, and MRZs in .NET MAUI Android Applications
<video src="https://github.com/yushulx/Capture-Vision-Maui/assets/2202306/cde6e832-36b4-4789-9f8f-2398fb9f28c2" controls="controls" muted="muted" style="max-height:640px; min-height: 200px; max-width: 100%;"></video>

## NuGet Package
[https://www.nuget.org/packages/Capture.Vision.Maui](https://www.nuget.org/packages/Capture.Vision.Maui)


## .NET MRZ SDK for Android

1. Download [https://github.com/yushulx/dotnet-mrz-sdk](https://github.com/yushulx/dotnet-mrz-sdk) and create a new `Android Java Library Binding` project in the project folder, using the same name as the desktop project.
2. Add `DynamsoftCore.aar`, `DynamsoftLabelRecognizer.aar` and `MRZLib-release.aar` to the project. Note: `MRZLib-release.aar` is generated by the [MRZLib](https://github.com/yushulx/dotnet-mrz-sdk/tree/main/android/aar/MRZLib) project.
3. Copy `MrzParser.cs`, `MrzResult.cs`, and `MrzScanner.cs` from the desktop project to the `Android Java Library Binding` project.
4. Refactor `MrzScanner.cs` to utilize the Dynamsoft Label Recognizer Android library.
    
    ```csharp
    using Com.Dynamsoft.Core;
    using Com.Dynamsoft.Dlr;
    
    namespace Dynamsoft
    {
        public class MrzScanner
        {
            private MRZRecognizer? recognizer;
            ...
    
            public class LicenseVerificationListener : Java.Lang.Object, ILicenseVerificationListener
            {
                public void LicenseVerificationCallback(bool isSuccess, CoreException ex)
                {
                    if (!isSuccess)
                    {
                        throw new Exception(ex.ToString());
                    }
                }
            }
    
            public static void InitLicense(string license, object? context = null)
            {
                if (context == null) { return; }
    
                LicenseManager.InitLicense(license, (Android.Content.Context)context, new LicenseVerificationListener());
            }
    
            private MrzScanner()
            {
                recognizer = new MRZRecognizer();
            }
    
            public static MrzScanner Create()
            {
                return new MrzScanner();
            }
    
            ~MrzScanner()
            {
                Destroy();
            }
    
            public void Destroy()
            {
                recognizer = null;
            }
    
            public static string? GetVersionInfo()
            {
                return MRZRecognizer.Version;
            }
    
            public Result[]? DetectFile(string filename)
            {
                if (recognizer == null) return null;
    
                DLRResult[]? mrzResult = recognizer.RecognizeFile(filename);
                return GetResults(mrzResult);
            }
    
            public Result[]? DetectBuffer(byte[] buffer, int width, int height, int stride, ImagePixelFormat format)
            {
                if (recognizer == null) return null;
    
                ImageData imageData = new ImageData()
                {
                    Bytes = buffer,
                    Width = width,
                    Height = height,
                    Stride = stride,
                    Format = (int)format,
                };
                DLRResult[]? mrzResult = recognizer.RecognizeBuffer(imageData);
                return GetResults(mrzResult);
            }
    
            private Result[]? GetResults(DLRResult[]? mrzResult)
            {
                if (mrzResult != null && mrzResult[0].LineResults != null)
                {
                    DLRLineResult[] lines = mrzResult[0].LineResults.ToArray();
                    Result[] result = new Result[lines.Length];
                    for (int i = 0; i < lines.Length; i++)
                    {
                        result[i] = new Result()
                        {
                            Confidence = lines[i].Confidence,
                            Text = lines[i].Text ?? "",
                            Points = new int[8]
                            {
                                lines[i].Location.Points[0].X,
                                lines[i].Location.Points[0].Y,
                                lines[i].Location.Points[1].X,
                                lines[i].Location.Points[1].Y,
                                lines[i].Location.Points[2].X,
                                lines[i].Location.Points[2].Y,
                                lines[i].Location.Points[3].X,
                                lines[i].Location.Points[3].Y
                            }
                        };
                    }
    
                    return result;
                }
    
                return null;
            }
        }
    }
    ```

5. Generate a `MrzScannerSDK.nuspec` file at the root directory.
    
    ```xml
    <?xml version="1.0" encoding="utf-8"?>
    <package>
      <metadata>
        <id>MrzScannerSDK</id>
        <version>1.3.5</version>
        <title>MRZ Scanner SDK</title>
        <authors>yushulx</authors>
        <requireLicenseAcceptance>false</requireLicenseAcceptance>
        <license type="expression">MIT</license>
        <readme>README.md</readme>
        <!-- <icon>icon.png</icon> -->
        <projectUrl>https://github.com/yushulx/dotnet-mrz-sdk</projectUrl>
        <description>The MRZ Scanner SDK is a .NET wrapper for Dynamsoft Label Recognizer,
          supporting x64 Windows, x64 Linux and Android.</description>
        <releaseNotes>Fixed build condition for Windows.</releaseNotes>
        <copyright>$copyright$</copyright>
        <tags>MRZ;mrz-scan;machine-readable-zone;mrz-detection;passport;visa;id-card;travel-document</tags>
      </metadata>
      <files>
        <file src="README.md" target="" />
        <file src="LICENSE.txt" target="" />
    
        <!-- Desktop -->
        <file src="desktop\lib\win\**\*.*" target="runtimes\win-x64\native" />
        <file src="desktop\lib\linux\**\*.*" target="runtimes\linux-x64\native" />
        <file src="desktop\bin\Release\net7.0\MrzScannerSDK.dll" target="lib\net7.0" />
        <file src="desktop\MrzScannerSDK.targets" target="build" />
        <file src="desktop\model\**\*.*" target="model" />
    
        <!-- Android -->
        <file src="android\sdk\bin\Release\net7.0-android\**\*.*" target="lib\net7.0-android33.0" />
    
      </files>
    </package>
    ```

6. Build the desktop and Android projects, then bundle them into a single NuGet package.
    
    ```bash
    cd desktop
    dotnet build --configuration Release

    cd android
    dotnet build --configuration Release
    
    nuget pack .\MrzScannerSDK.nuspec
    ```

    ![MRZ NuGet package](https://www.dynamsoft.com/codepool/img/2024/01/mrz-nuget-package.png)    

## .NET Document SDK for Android

1. Download [https://github.com/yushulx/dotnet-document-scanner-sdk](https://github.com/yushulx/dotnet-document-scanner-sdk) and create a new `Android Java Library Binding` project in the project folder, using the same name as the desktop project.
2. Add `DynamsoftCore.aar`, `DynamsoftDocumentNormalizer.aar`, `DynamsoftImageProcessing.aar` and `DynamsoftIntermediateResult.aar` to the project. 
3. Copy `DocumentScanner.cs` from the desktop project to the `Android Java Library Binding` project.
4. Refactor the `DocumentScanner.cs` to utilize the Dynamsoft Document Normalizer Android library.
    
     ```csharp
    using Com.Dynamsoft.Core;
    using Com.Dynamsoft.Ddn;
    
    namespace Dynamsoft
    {
        public class DocumentScanner
        {
            private DocumentNormalizer normalizer;
    
            public class NormalizedImage
            {
                public int Width;
                public int Height;
                public int Stride;
                public ImagePixelFormat Format;
                public byte[] Data = new byte[0];
            }
    
            public static string GetVersionInfo()
            {
                return DocumentNormalizer.Version;
            }
    
            ...
    
            public class LicenseVerificationListener : Java.Lang.Object, ILicenseVerificationListener
            {
                public void LicenseVerificationCallback(bool isSuccess, CoreException ex)
                {
                    if (!isSuccess)
                    {
                        throw new Exception(ex.ToString());
                    }
                }
            }
    
            public static void InitLicense(string license, object? context = null)
            {
                if (context == null) { return; }
    
                LicenseManager.InitLicense(license, (Android.Content.Context)context, new LicenseVerificationListener());
            }
    
            private DocumentScanner()
            {
                normalizer = new DocumentNormalizer();
            }
    
            public static DocumentScanner Create()
            {
                return new DocumentScanner();
            }
    
            public void SetParameters(string parameters)
            {
                try
                {
                    normalizer.InitRuntimeSettingsFromString(parameters);
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex.ToString());
                }
            }
    
            public Result[]? DetectFile(string filename)
            {
                DetectedQuadResult[]? results = normalizer.DetectQuad(filename);
                return GetResults(results);
            }
    
            public Result[]? DetectBuffer(byte[] buffer, int width, int height, int stride, ImagePixelFormat format)
            {
                ImageData imageData = new ImageData()
                {
                    Bytes = buffer,
                    Width = width,
                    Height = height,
                    Stride = stride,
                    Format = (int)format,
                };
                DetectedQuadResult[]? results = normalizer.DetectQuad(imageData);
                return GetResults(results);
            }
    
            private Result[]? GetResults(DetectedQuadResult[]? results)
            {
                if (results == null) return null;
    
                var result = new Result[results.Length];
                for (int i = 0; i < results.Length; i++)
                {
                    DetectedQuadResult tmp = results[i];
                    Quadrilateral quad = tmp.Location;
                    result[i] = new Result()
                    {
                        Confidence = tmp.ConfidenceAsDocumentBoundary,
                        Points = new int[8]
                        {
                            quad.Points[0].X, quad.Points[0].Y,
                            quad.Points[1].X, quad.Points[1].Y,
                            quad.Points[2].X, quad.Points[2].Y,
                            quad.Points[3].X, quad.Points[3].Y,
                        }
                    };
                }
    
                return result;
            }
    
            public NormalizedImage NormalizeFile(string filename, int[] points)
            {
                Quadrilateral quad = new Quadrilateral();
                quad.Points = new Android.Graphics.Point[4];
                quad.Points[0] = new Android.Graphics.Point(points[0], points[1]);
                quad.Points[1] = new Android.Graphics.Point(points[2], points[3]);
                quad.Points[2] = new Android.Graphics.Point(points[4], points[5]);
                quad.Points[3] = new Android.Graphics.Point(points[6], points[7]);
                NormalizedImageResult? result = normalizer.Normalize(filename, quad);
                return GetNormalizedImage(result);
            }
    
            public NormalizedImage NormalizeBuffer(byte[] buffer, int width, int height, int stride, ImagePixelFormat format, int[] points)
            {
                ImageData imageData = new ImageData()
                {
                    Bytes = buffer,
                    Width = width,
                    Height = height,
                    Stride = stride,
                    Format = (int)format,
                };
    
                Quadrilateral quad = new Quadrilateral();
                quad.Points = new Android.Graphics.Point[4];
                quad.Points[0] = new Android.Graphics.Point(points[0], points[1]);
                quad.Points[1] = new Android.Graphics.Point(points[2], points[3]);
                quad.Points[2] = new Android.Graphics.Point(points[4], points[5]);
                quad.Points[3] = new Android.Graphics.Point(points[6], points[7]);
                NormalizedImageResult? result = normalizer.Normalize(imageData, quad);
                return GetNormalizedImage(result);
            }
    
            private NormalizedImage GetNormalizedImage(NormalizedImageResult? result)
            {
                NormalizedImage normalizedImage = new NormalizedImage();
                if (result != null)
                {
                    ImageData imageData = result.Image;
                    normalizedImage.Width = imageData.Width;
                    normalizedImage.Height = imageData.Height;
                    normalizedImage.Stride = imageData.Stride;
                    normalizedImage.Format = (ImagePixelFormat)imageData.Format;
                    normalizedImage.Data = imageData.Bytes.ToArray();
                }
                return normalizedImage;
            }
        }
    }
    ```

5. Generate a `MrzScannerSDK.nuspec` file in the root directory.
    
    ```xml
    <?xml version="1.0" encoding="utf-8"?>
    <package>
        <metadata>
            <id>DocumentScannerSDK</id>
            <version>1.2.0</version>
            <title>Document Scanner SDK</title>
            <authors>yushulx</authors>
            <requireLicenseAcceptance>false</requireLicenseAcceptance>
            <license type="expression">MIT</license>
            <readme>README.md</readme>
            <!-- <icon>icon.png</icon> -->
            <projectUrl>https://github.com/yushulx/dotnet-document-scanner-sdk</projectUrl>
            <description>The Document Scanner SDK is a .NET wrapper for Dynamsoft Document Normalizer,
                supporting x64 Windows, x64 Linux and Android.</description>
            <releaseNotes>Added support for Android.</releaseNotes>
            <copyright>$copyright$</copyright>
            <tags>document;document-scan;edge-detection;document-detection</tags>
        </metadata>
        <files>
            <file src="README.md" target="" />
            <file src="LICENSE.txt" target="" />
    
            <!-- Desktop -->
            <file src="desktop\lib\win\**\*.*" target="runtimes\win-x64\native" />
            <file src="desktop\lib\linux\**\*.*" target="runtimes\linux-x64\native" />
            <file src="desktop\bin\Release\net7.0\DocumentScannerSDK.dll" target="lib\net7.0" />
    
            <!-- Android -->
            <file src="android\bin\Release\net7.0-android\**\*.*" target="lib\net7.0-android33.0" />
    
        </files>
    </package>
    ```

6. Build the desktop and Android projects, then bundle them into a single NuGet package.
    
    ```bash
    cd desktop
    dotnet build --configuration Release

    cd android
    dotnet build --configuration Release
    
    nuget pack .\DocumentScannerSDK.nuspec
    ```

    ![Document NuGet package](https://www.dynamsoft.com/codepool/img/2024/01/document-nuget-package.png)    

## Troubleshooting Error XA4215 When Building a .NET MAUI Library Project

When building a .NET MAUI library project that includes dependencies on the .NET MRZ SDK and the .NET Document SDK, you may encounter the following error:

```bash
Severity	Code	Description	Project	File	Line	Suppression State	Details
Error	XA4215	  `mono.com.dynamsoft.core.LicenseVerificationListenerImplementor` generated by: Com.Dynamsoft.Core.ILicenseVerificationListenerImplementor, MrzScannerSDK, Version=0.1.4.0, Culture=neutral, PublicKeyToken=null	Capture.Vision.Maui.Example	C:\Program Files\dotnet\packs\Microsoft.Android.Sdk.Windows\33.0.95\tools\Xamarin.Android.Common.targets	1476		
Error	XA4215	  `mono.com.dynamsoft.core.LicenseVerificationListenerImplementor` generated by: Com.Dynamsoft.Core.ILicenseVerificationListenerImplementor, DocumentScannerSDK, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null	Capture.Vision.Maui.Example	C:\Program Files\dotnet\packs\Microsoft.Android.Sdk.Windows\33.0.95\tools\Xamarin.Android.Common.targets	1476		
```

The error XA4215 you're encountering in your .NET MAUI project indicates a conflict between two assemblies trying to generate the same class, `mono.com.dynamsoft.core.LicenseVerificationListenerImplementor`. To resolve the issue, a feasible workaround is to merge the two packages into one.

## Merging .NET MRZ SDK and .NET Document SDK into One .NET Library
1. Create a `Class Library` project and an `Android Java Library Binding` project in the same folder, ensuring both projects share the same name, e.g., `CaptureVision`. The `Class Library` project will house .NET code and libraries for desktop platforms (Windows and Linux), whereas the `Android Java Library Binding` project will house .NET code and libraries for Android.
2. Copy all Android .aar packages and C# files from `MrzScannerSDK` and `DocumentScannerSDK` android folder to the `Android Java Library Binding` project. 
    
    ![.NET capture vision library](https://www.dynamsoft.com/codepool/img/2024/01/dotnet-capture-vision-library.png)
    
3. Copy MRZ model files, `*.so` files, `*.dll` files, and C# files from `MrzScannerSDK` and `DocumentScannerSDK` desktop folder to the `Class Library` project, maintaining the original folder structure.
    
    ```bash
    ├── model
    ├── lib
        ├── linux
        ├── win
    ├── CaptureVision.csproj
    ├── CaptureVision.targets
    ├── DocumentScaner.cs
    ├── MrzParser.cs
    ├── MrzResult.cs
    ├── MrzScanner.cs
    ```

4. Generate a `CaptureVision.nuspec` file in the root directory.
    
    ```xml
    <?xml version="1.0" encoding="utf-8"?>
    <package>
        <metadata>
            <id>CaptureVision</id>
            <version>1.0.1</version>
            <title>Capture Vision SDK</title>
            <authors>yushulx</authors>
            <requireLicenseAcceptance>false</requireLicenseAcceptance>
            <license type="expression">MIT</license>
            <readme>README.md</readme>
            <!-- <icon>icon.png</icon> -->
            <projectUrl>https://github.com/yushulx/Capture-Vision</projectUrl>
            <description>This is a package that is a compound of DocumentScannerSDK and MrzScannerSDK.</description>
            <releaseNotes>Merged DocumentScannerSDK and MrzScannerSDK into one package.</releaseNotes>
            <copyright>$copyright$</copyright>
            <tags>
                MRZ;Android;passport;id-card;visa;machine-readable-zone;document;document-scan;edge-detection;document-detection</tags>
        </metadata>
        <files>
            <file src="README.md" target="" />
            <file src="LICENSE.txt" target="" />
    
            <!-- Desktop -->
            <file src="desktop\lib\win\**\*.*" target="runtimes\win-x64\native" />
            <file src="desktop\lib\linux\**\*.*" target="runtimes\linux-x64\native" />
            <file src="desktop\bin\Release\net7.0\CaptureVision.dll" target="lib\net7.0" />
            <file src="desktop\CaptureVision.targets" target="build" />
            <file src="desktop\model\**\*.*" target="model" />
    
    
            <!-- Android -->
            <file src="android\bin\Release\net7.0-android\**\*.*" target="lib\net7.0-android33.0" />
    
        </files>
    </package>
    ```

5. Build the library projects respectively and pack them into a single NuGet package.

    ```bash
    cd desktop
    dotnet build --configuration Release
    
    cd android
    dotnet build --configuration Release
    
    nuget pack .\CaptureVision.nuspec
    ```

## Adding Android Platform-Specific Code to .NET MAUI Library
After generating the NuGet package, incorporate it as a dependency into your .NET MAUI library project.

```xml
<ItemGroup Condition="'$(TargetFramework)' == 'net7.0-android'">
	<PackageReference Include="CaptureVision " Version="1.0.1" />
</ItemGroup>

<ItemGroup Condition="$([MSBuild]::IsOSPlatform('windows'))">
	<PackageReference Include="CaptureVision " Version="1.0.1" />
</ItemGroup>
```

Next, navigate to `Platforms/Android/NativeCameraView.cs` within your project, and insert the following code snippet:

```csharp
private BarcodeQRCodeReader barcodeReader;
private MrzScanner mrzScanner;
private DocumentScanner documentScanner;

public NativeCameraView(Context context, CameraView cameraView) : base(context)
{
    ...

    barcodeReader = BarcodeQRCodeReader.Create();
    mrzScanner = MrzScanner.Create();
    documentScanner = DocumentScanner.Create();
}

private void StartPreview()
{
    ...
    backgroundHandler = new Handler(backgroundThread.Looper);
    frameListener = new ImageAvailableListener(cameraView, barcodeReader, mrzScanner, documentScanner);
    ...
}

class ImageAvailableListener : Java.Lang.Object, ImageReader.IOnImageAvailableListener
{
    private readonly CameraView cameraView;
    private BarcodeQRCodeReader barcodeReader;
    private MrzScanner mrzScanner;
    private DocumentScanner documentScanner;

    public ImageAvailableListener(CameraView camView, BarcodeQRCodeReader barcodeReader, MrzScanner mrzScanner, DocumentScanner documentScanner)
    {
        cameraView = camView;
        this.barcodeReader = barcodeReader;
        this.mrzScanner = mrzScanner;
        this.documentScanner = documentScanner;
    }

    public void OnImageAvailable(ImageReader reader)
    {
        try
        {
            var image = reader?.AcquireLatestImage();
            ...

            if (cameraView.EnableDocumentDetect)
            {
                DocumentScanner.Result[] results = documentScanner.DetectBuffer(bytes, width, height, nPixelStride * nRowStride, DocumentScanner.ImagePixelFormat.IPF_GRAYSCALED);
                DocumentResult documentResults = new DocumentResult();
                ...
                cameraView.NotifyResultReady(documentResults, width, height);
            }

            if (cameraView.EnableMrz)
            {
                MrzResult mrzResults = new MrzResult();
                try
                {
                    MrzScanner.Result[] results = mrzScanner.DetectBuffer(bytes, width, height, nPixelStride * nRowStride, MrzScanner.ImagePixelFormat.IPF_GRAYSCALED);
                    ...
                }

                catch (Exception ex)
                {
                    System.Diagnostics.Debug.WriteLine(ex.Message);
                }
                cameraView.NotifyResultReady(mrzResults, width, height);
            }
        }
        catch (Exception ex)
        {
            System.Diagnostics.Debug.WriteLine(ex.Message);
        }
    }
}
```

## Upgrading Capture.Vision.Maui to Scan QR Code, Document, and MRZ in .NET MAUI Android Applications
In the [.NET MAUI application project](https://github.com/yushulx/Capture-Vision-Maui/tree/main/Capture.Vision.Maui.Example), upgrade the `Capture.Vision.Maui` library to its latest version. After upgrading, rebuild the project and execute it on an Android device.

**QR Code Scan**

![.NET MAUI Android: scan QR code](https://www.dynamsoft.com/codepool/img/2024/01/dotnet-maui-android-qr-detection.jpg)

**Document Edge Detection**

![.NET MAUI Android: scan document](https://www.dynamsoft.com/codepool/img/2024/01/dotnet-maui-android-document-detection.jpg)

**MRZ Recognition**

![.NET MAUI barcode: scan MRZ](https://www.dynamsoft.com/codepool/img/2024/01/dotnet-maui-android-mrz-detection.jpg)

## Source Code
[https://github.com/yushulx/Capture-Vision-Maui](https://github.com/yushulx/Capture-Vision-Maui)