How to Create a .NET MAUI Plugin for Camera Barcode Qr Code Scanning
Tags: dotnet, maui, mobile, programming
Published: 2023-06-20 03:10:06

As .NET MAUI continues to gain traction, numerous .NET developers are exploring ways to transition their existing desktop and mobile applications to this new framework. Previously, I developed a .NET MAUI application capable of scanning barcodes and QR codes from camera frames across Windows, Android, and iOS platforms. For reuse, I decided to isolate the camera and barcode scanning functionality and encapsulate them into a .NET MAUI plugin. In this article, I will guide you through the creation of a .NET MAUI plugin, featuring a custom camera view and the integration of the Dynamsoft Barcode Reader SDK.

## Demo Video
<video src="https://user-images.githubusercontent.com/2202306/246987930-68e7588e-b46c-489f-b156-3cc3125a9aa7.mp4" controls="controls" muted="muted" style="max-height:640px; min-height: 200px; max-width: 100%;"></video>


## Try Capture.Vision.Maui
[https://www.nuget.org/packages/Capture.Vision.Maui/](https://www.nuget.org/packages/Capture.Vision.Maui/)


## The Shortcomings of My Current .NET MAUI Application
Check out the source code: [https://github.com/yushulx/Capture-Vision-Maui](https://github.com/yushulx/Capture-Vision-Maui). Before getting started with the plugin, let's take a look at the shortcomings of my current .NET MAUI application:
- **Windows**: The camera stream is captured using `OpenCVSharp`. You need to install [OpenCVSharp4](https://www.nuget.org/packages/OpenCvSharp4) and [OpenCV Windows runtime](https://www.nuget.org/packages/OpenCvSharp4.runtime.win) to make it work. Since the camera stream is displayed by rendering the OpenCVSharp Mat object to a SkiaSharp `SKCanvas`, the UI code cannot be shared across Android and iOS platforms. 
- **Android and iOS**: The code for Android and iOS is ported from my [Xamarin.Forms](https://github.com/yushulx/xamarin-forms-barcode-qrcode-scanner) Qr code scanner application. To make the code compatible with .NET MAUI, the `AddCompatibilityRenderer()` method is used. However, within the .NET MAUI environment, the `AddHandler()` method is recommended.

What's the difference between `AddCompatibilityRenderer()` and `AddHandler()`? 
- `AddCompatibilityRenderer`: This method is used to map Xamarin.Forms renderers to .NET MAUI handlers. It's a way of maintaining compatibility with existing Xamarin.Forms custom renderers as you transition your apps to .NET MAUI. 

- `AddHandler`: This is a method used in the context of the new architecture in .NET MAUI. Handlers are a more lightweight, efficient replacement for the old Xamarin.Forms renderers, and they allow greater control and easier customization of your controls. 

Our objective is to develop a unified camera view equipped with barcode scanning capabilities, compatible with Windows, Android, and iOS platforms. The following resources will be helpful:
- Camera: [https://github.com/hjam40/Camera.MAUI](https://github.com/hjam40/Camera.MAUI)
- Barcode: [https://github.com/yushulx/dotnet-barcode-qr-code-sdk](https://github.com/yushulx/dotnet-barcode-qr-code-sdk)

## Initiate a .NET MAUI Plugin Project
Let's set up a .NET MAUI plugin project and configure the dependencies:
1. Create a .NET MAUI class library project in Visual Studio 2022.

    ![.NET MAUI Class Library](https://www.dynamsoft.com/codepool/img/2023/06/dotnet-maui-class-library.png)

2. Double-click the project file and then add [BarcodeQRCodeSDK](https://www.nuget.org/packages/BarcodeQRCodeSDK) as a dependency. The package is a .NET wrapper for the Dynamsoft Barcode Reader SDK, supporting **Windows**, **Linux**, **macOS**, **Android**, and **iOS** platforms. A [valid license key](https://www.dynamsoft.com/customer/license/trialLicense) is required to use the package. Note that the **BarcodeQRCodeSDK** package does not support **Maccatalyst** framework.

    ```xml
    <ItemGroup>
		<PackageReference Include="BarcodeQRCodeSDK" Version="2.3.4" />		
	</ItemGroup>
    ```

3. In order to test the plugin, add a new .NET MAUI app project to the solution. 
    
    ![.NET MAUI APP](https://www.dynamsoft.com/codepool/img/2023/06/dotnet-maui-app.png)

    After that, add a reference to the class library project in the app project.

    ![Add Reference](https://www.dynamsoft.com/codepool/img/2023/06/dotnet-maui-reference.png)

## .NET MAUI Custom View with Platform-Specific Code
In .NET MAUI, a custom view is created by extending the `View` class. Simultaneously, an associated `ViewHandler` must be established to handle platform-specific rendering.


A `view` is responsible for defining what elements are present on the user interface and their properties. In our case, the camera view should be able to display the camera stream, retrieve camera frames and return barcode results.

```cs
using System.Collections.ObjectModel;
using static Capture.Vision.Maui.CameraInfo;

namespace Capture.Vision.Maui
{
    public class ResultReadyEventArgs : EventArgs
    {
        public ResultReadyEventArgs(object result, int previewWidth, int previewHeight)
        {
            Result = result;
            PreviewWidth = previewWidth;
            PreviewHeight = previewHeight;
        }

        public object Result { get; private set; }
        public int PreviewWidth { get; private set; }
        public int PreviewHeight { get; private set; }

    }

    public class FrameReadyEventArgs : EventArgs
    { 
        public enum PixelFormat
        {
            GRAYSCALE,
            RGB888,
            BGR888,
            RGBA8888,
            BGRA8888,
        }
        public FrameReadyEventArgs(byte[] buffer, int width, int height, int stride, PixelFormat pixelFormat)
        {
            Buffer = buffer;
            Width = width;
            Height = height;
            Stride = stride;
            Format = pixelFormat;
        }

        public byte[] Buffer { get; private set; }
        public int Width { get; private set; }
        public int Height { get; private set; }
        public int Stride { get; private set; }
        public PixelFormat Format { get; private set; }
    }

    public class CameraView : View
    {
        public static readonly BindableProperty CamerasProperty = BindableProperty.Create(nameof(Cameras), typeof(ObservableCollection<CameraInfo>), typeof(CameraView), new ObservableCollection<CameraInfo>());
        public static readonly BindableProperty CameraProperty = BindableProperty.Create(nameof(Camera), typeof(CameraInfo), typeof(CameraView), null);
        public static readonly BindableProperty EnableBarcodeProperty = BindableProperty.Create(nameof(EnableBarcode), typeof(bool), typeof(CameraView), false);
        public static readonly BindableProperty ShowCameraViewProperty = BindableProperty.Create(nameof(ShowCameraView), typeof(bool), typeof(CameraView), false, propertyChanged: ShowCameraViewChanged);
        public event EventHandler<ResultReadyEventArgs> ResultReady;
        public event EventHandler<FrameReadyEventArgs> FrameReady;
    }
}
``` 

A `handler` is responsible for taking the view's definitions and translating them into platform-specific code that can be rendered on the screen.

```cs
using Microsoft.Maui.Handlers;
using static Capture.Vision.Maui.CameraInfo;
#if IOS
using PlatformView = Capture.Vision.Maui.Platforms.iOS.NativeCameraView;
#elif ANDROID
using PlatformView = Capture.Vision.Maui.Platforms.Android.NativeCameraView;
#elif WINDOWS
using PlatformView = Capture.Vision.Maui.Platforms.Windows.NativeCameraView;
#else
using PlatformView = System.Object;
#endif

namespace Capture.Vision.Maui
{
    internal partial class CameraViewHandler : ViewHandler<CameraView, PlatformView>
    {
        public static IPropertyMapper<CameraView, CameraViewHandler> PropertyMapper = new PropertyMapper<CameraView, CameraViewHandler>(ViewMapper)
        {
        };
        public static CommandMapper<CameraView, CameraViewHandler> CommandMapper = new(ViewCommandMapper)
        {
        };
        public CameraViewHandler() : base(PropertyMapper, CommandMapper)
        {
        }

#if ANDROID
    protected override PlatformView CreatePlatformView() => new(Context, VirtualView);
#elif IOS || WINDOWS
        protected override PlatformView CreatePlatformView() => new(VirtualView);
#else
    protected override PlatformView CreatePlatformView() => new();
#endif
        protected override void ConnectHandler(PlatformView platformView)
        {
            base.ConnectHandler(platformView);
        }

        protected override void DisconnectHandler(PlatformView platformView)
        {
#if WINDOWS || IOS || ANDROID
            platformView.DisposeControl();
#endif
            base.DisconnectHandler(platformView);
        }

        public Task<Status> StartCameraAsync()
        {
            if (PlatformView != null)
            {
#if WINDOWS || ANDROID || IOS
                return PlatformView.StartCameraAsync();
#endif
            }
            return Task.Run(() => { return Status.Unavailable; });
        }

        public Task<Status> StopCameraAsync()
        {
            if (PlatformView != null)
            {
#if WINDOWS
            return PlatformView.StopCameraAsync();
#elif ANDROID || IOS
                var task = new Task<Status>(() => { return PlatformView.StopCamera(); });
                task.Start();
                return task;
#endif
            }
            return Task.Run(() => { return Status.Unavailable; });
        }
    }

}
```

The code snippet below demonstrates the platform-relevant classes used to display video across various platforms.

- Windows: `MediaPlayerElement`

    ```cs
    public sealed partial class NativeCameraView : UserControl, IDisposable
    {
        private readonly MediaPlayerElement mediaElement;
        private readonly CameraView cameraView;
        ...

        public NativeCameraView(CameraView cameraView)
        {
            this.cameraView = cameraView;
            mediaElement = new MediaPlayerElement
            {
                HorizontalAlignment = Microsoft.UI.Xaml.HorizontalAlignment.Stretch,
                VerticalAlignment = Microsoft.UI.Xaml.VerticalAlignment.Stretch
            };
            Content = mediaElement;
            ...
        }
        ...
    }

    ```

- Android: `TextureView`

    ```cs
    internal class NativeCameraView : FrameLayout
    {
        private readonly CameraView cameraView;
        private readonly Context context;
        private readonly TextureView textureView;
        ...

        public NativeCameraView(Context context, CameraView cameraView) : base(context)
        {
            this.context = context;
            this.cameraView = cameraView;

            textureView = new(context);

            AddView(textureView);

            ...
        }
        ...
    }
    ```

- iOS: `AVCaptureVideoPreviewLayer`

    ```cs
    internal class NativeCameraView : UIView, IAVCaptureVideoDataOutputSampleBufferDelegate, IAVCapturePhotoCaptureDelegate
    {
        private readonly CameraView cameraView;
        private readonly AVCaptureVideoPreviewLayer PreviewLayer;
        ...

        public NativeCameraView(CameraView cameraView)
        {
            this.cameraView = cameraView;

            captureSession = new AVCaptureSession
            {
                SessionPreset = AVCaptureSession.PresetPhoto
            };
            PreviewLayer = new(captureSession)
            {
                VideoGravity = AVLayerVideoGravity.ResizeAspectFill
            };
            Layer.AddSublayer(PreviewLayer);
            ...
        }
        ...
    }
    ```

Our primary concern is determining how to obtain camera frames for image processing across different platforms.

**Windows**
1. Create a `MediaFrameReader`:

    ```cs
    MediaFrameSource frameSource;
    MediaFrameReader frameReader = await mediaCapture.CreateFrameReaderAsync(frameSource);
    frameReader.AcquisitionMode = MediaFrameReaderAcquisitionMode.Realtime;
    if (frameReader != null)
    {
        frameReader.FrameArrived += OnFrameAvailable;
        var status = await frameReader.StartAsync();
    }

    
    ```

2. Retrieve the latest frame via the `OnFrameAvailable` event handler.
    ```cs
    private void OnFrameAvailable(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
        {
            var frame = sender.TryAcquireLatestFrame();
            if (frame == null) return;

            SoftwareBitmap bitmap = frame.VideoMediaFrame.SoftwareBitmap;
            // process image
            bitmap.Dispose();
        }

    ```

**Android**
1. Create an `ImageReader` and bind it to a background thread:

    ```cs
    private ImageReader imageReader;
    imageReader = ImageReader.NewInstance(videoSize.Width, videoSize.Height, ImageFormatType.Yuv420888, 1);
    backgroundThread = new HandlerThread("CameraBackground");
    backgroundThread.Start();
    backgroundHandler = new Handler(backgroundThread.Looper);
    frameListener = new ImageAvailableListener(cameraView, barcodeReader);
    imageReader.SetOnImageAvailableListener(frameListener, backgroundHandler);
    surfaces.Add(new OutputConfiguration(imageReader.Surface));
    previewBuilder.AddTarget(imageReader.Surface);
    ```

2. Fetch the latest frame via the `OnImageAvailable` event handler.
    ```cs
    class ImageAvailableListener : Java.Lang.Object, ImageReader.IOnImageAvailableListener
    {
        ...

        public void OnImageAvailable(ImageReader reader)
        {
            try
            {
                var image = reader?.AcquireLatestImage();
                if (image == null)
                    return;

                Image.Plane[] planes = image.GetPlanes();
                if (planes == null) return;

                int width = image.Width;
                int height = image.Height;
                ByteBuffer buffer = planes[0].Buffer;
                byte[] bytes = new byte[buffer.Remaining()];
                buffer.Get(bytes);
                int nRowStride = planes[0].RowStride;
                int nPixelStride = planes[0].PixelStride;
                image.Close();

                // process image
            }
            catch (Exception ex)
            {
            }
        }
    }
    ```

**iOS** 
1. Create an `AVCaptureVideoDataOutput` and set the `SampleBufferDelegate` to the current view.

    ```cs
    AVCaptureVideoDataOutput videoDataOutput = new AVCaptureVideoDataOutput();
            var videoSettings = NSDictionary.FromObjectAndKey(
                new NSNumber((int)CVPixelFormatType.CV32BGRA),
                CVPixelBuffer.PixelFormatTypeKey);
    videoDataOutput.WeakVideoSettings = videoSettings;
    videoDataOutput.AlwaysDiscardsLateVideoFrames = true;
    cameraDispacher = new DispatchQueue("CameraDispacher");

    videoDataOutput.SetSampleBufferDelegate(this, cameraDispacher);
    ```
2. Fetch the latest frame via the `DidOutputSampleBuffer` event handler.
    ```cs
    [Export("captureOutput:didOutputSampleBuffer:fromConnection:")]
    public void DidOutputSampleBuffer(AVCaptureOutput captureOutput, CMSampleBuffer sampleBuffer, AVCaptureConnection connection)
    {
        CVPixelBuffer cVPixelBuffer = (CVPixelBuffer)sampleBuffer.GetImageBuffer();
        cVPixelBuffer.Lock(CVPixelBufferLock.ReadOnly);
        nint dataSize = cVPixelBuffer.DataSize;
        width = cVPixelBuffer.Width;
        height = cVPixelBuffer.Height;
        IntPtr baseAddress = cVPixelBuffer.BaseAddress;
        bpr = cVPixelBuffer.BytesPerRow;
        cVPixelBuffer.Unlock(CVPixelBufferLock.ReadOnly);
        buffer = NSData.FromBytes(baseAddress, (nuint)dataSize);
        // process image
    }
    ```

## How to Scan Barcodes from Camera Frames
As the camera frames are retrieved, we can now use the Dynamsoft Barcode Reader SDK to scan barcodes from the frames. 
1. Create a `BarcodeReader` instance:

    ```cs
    BarcodeQRCodeReader barcodeReader = BarcodeQRCodeReader.Create();
    ```
2. To decode barcodes from the camera frame, you're required to retrieve the byte data, frame width, height, stride, and pixel format. 

    ```cs
    // Windows
    Result[] results = barcodeReader.DecodeBuffer(buffer, bitmap.PixelWidth, bitmap.PixelHeight, bitmap.PixelWidth, BarcodeQRCodeReader.ImagePixelFormat.IPF_GRAYSCALED);
    // Android
    Result[] results = barcodeReader.DecodeBuffer(bytes, width, height, nPixelStride * nRowStride, BarcodeQRCodeReader.ImagePixelFormat.IPF_GRAYSCALED);
    // iOS
    Result[] results = barcodeReader.DecodeBuffer(bytearray, (int)width, (int)height, (int)bpr, BarcodeQRCodeReader.ImagePixelFormat.IPF_ARGB_8888);
    ```

## How to Use the Plugin in a .NET MAUI Application
Now that we have a .NET MAUI plugin in place, let's see how to use it in a .NET MAUI application.
1. Add the plugin to the .NET MAUI app project via NuGet Manager.

    ![.NET MAUI camera barcode plugin](https://www.dynamsoft.com/codepool/img/2023/06/nuget-maui-camera-barcode-plugin.png)

2. In `MauiProgram.cs`, add the following code to register the plugin:

    ```cs
    using Microsoft.Extensions.Logging;

    namespace Capture.Vision.Maui.Example
    {
        public static class MauiProgram
        {
            public static MauiApp CreateMauiApp()
            {
                var builder = MauiApp.CreateBuilder();
                builder.UseNativeCameraView()
                    .UseMauiApp<App>()
                    .ConfigureFonts(fonts =>
                    {
                        fonts.AddFont("OpenSans-Regular.ttf", "OpenSansRegular");
                        fonts.AddFont("OpenSans-Semibold.ttf", "OpenSansSemibold");
                    });

    #if DEBUG
            builder.Logging.AddDebug();
    #endif

                return builder.Build();
            }
        }
    }
    ```

3. Apply for a [trial license](https://www.dynamsoft.com/customer/license/trialLicense?product=dbr) and then activate Dynamsoft Barcode Reader in `MainPage.xaml.cs`:

    ```cs
    using Dynamsoft;

    namespace Capture.Vision.Maui.Example
    {
        public partial class MainPage : ContentPage
        {
            public MainPage()
            {
                InitializeComponent();
                InitService();
            }

            private async void InitService()
            {
                await Task.Run(() =>
                {
                    BarcodeQRCodeReader.InitLicense("LICENSE-KEY");

                    return Task.CompletedTask;
                });
            }
        }
    }
    ```
4. Create a .NET MAUI content page and add a `CameraView` to it:

    ```xml
    <?xml version="1.0" encoding="utf-8" ?>
    <ContentPage xmlns="http://schemas.microsoft.com/dotnet/2021/maui"
                xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
                xmlns:skia="clr-namespace:SkiaSharp.Views.Maui.Controls;assembly=SkiaSharp.Views.Maui.Controls"
                xmlns:cv="clr-namespace:Capture.Vision.Maui;assembly=Capture.Vision.Maui"
                x:Class="Capture.Vision.Maui.Example.CameraPage"
                Title="CameraPage">
        <ScrollView>
            <Grid>
                <cv:CameraView x:Name="cameraView" HorizontalOptions="FillAndExpand"
                VerticalOptions="FillAndExpand" EnableBarcode="True" 
                                    ResultReady="cameraView_ResultReady" FrameReady="cameraView_FrameReady"
                                />
                <skia:SKCanvasView x:Name="canvasView" 
                            Margin="0"
                            HorizontalOptions="FillAndExpand" VerticalOptions="FillAndExpand"
                            PaintSurface="OnCanvasViewPaintSurface" />
            </Grid>
        </ScrollView>
    </ContentPage>
    ```

    The `cameraView_ResultReady` event handler is used to display the barcode result:

    ```cs
    private void cameraView_ResultReady(object sender, ResultReadyEventArgs e)
    {
        if (e.Result != null)
        {
            Result[] results = (Result[])e.Result;
            foreach (Result result in results)
            {
                System.Diagnostics.Debug.WriteLine(result.Text);
            }
        }
    }
    ```

    The `cameraView_FrameReady` event handler is used to retrieve the camera frame for image processing:
    ```cs
    private void cameraView_FrameReady(object sender, FrameReadyEventArgs e)
    {
        // process image
    }
    ```

    ![.NET MAUI Windows QR code scanner](https://www.dynamsoft.com/codepool/img/2023/06/dotnet-maui-windows-qr-code-scanner.png)

## Source Code
[https://github.com/yushulx/Capture-Vision-Maui](https://github.com/yushulx/Capture-Vision-Maui)