How to Get Camera Frames for Image Processing in .NET MAUI Windows App
Tags: dotnet, maui, windows, camera
Published: 2023-04-20 08:52:31

Last week, we demonstrated how to create a Windows .NET MAUI app that can access the USB camera using Blazor web view. Although this solution provides a way to access the camera, for a Windows desktop application that requires optimal performance, it is recommended to prioritize native camera APIs over web APIs. In this article, we will guide you through the process of creating a .NET MAUI Windows application in C# that can capture camera frames for barcode and QR code scanning.

![.NET MAUI Windows Camera QR Code Scanner](https://www.dynamsoft.com/codepool/img/2023/04/dotnet-maui-windows-camera-qr-scanner.gif)

## Required NuGet Packages
- [OpenCvSharp4](https://www.nuget.org/packages/OpenCvSharp4): A .NET wrapper for OpenCV 4.x. It is used to capture camera frames.
- [OpenCvSharp4.runtime.win](https://www.nuget.org/packages/OpenCvSharp4.runtime.win): Windows native binaries for OpenCvSharp4.
- [BarcodeQRCodeSDK](https://www.nuget.org/packages/BarcodeQRCodeSDK): A .NET wrapper for [Dynamsoft Barcode Reader SDK](https://www.dynamsoft.com/barcode-reader/overview/). It is used to detect 1D and 2D barcodes in camera frames.
- [SkiaSharp.Views.Maui.Controls](https://www.nuget.org/packages/SkiaSharp.Views.Maui.Controls): A set of views that can be used to draw on the screen.

Configure the above packages in your `csproj` file:
```xml
<ItemGroup>
  <PackageReference Include="BarcodeQRCodeSDK" Version="2.3.4" />
  <PackageReference Include="SkiaSharp.Views.Maui.Controls" Version="2.88.3" />
</ItemGroup>

<ItemGroup Condition="$([MSBuild]::IsOSPlatform('windows'))">
  <PackageReference Include="OpenCvSharp4" Version="4.6.0.20220608" />
  <PackageReference Include="OpenCvSharp4.runtime.win" Version="4.6.0.20220608" />
</ItemGroup>
```

The packages for Windows only are manually added to the `ItemGroup` with the condition `$([MSBuild]::IsOSPlatform('windows'))`. 

## Steps to Capture Camera Frames for Barcode Detection in .NET MAUI Windows 
We create a new MAUI content page and add the `SKCanvasView` to the page. The `SKCanvasView` is used to draw the camera frames on the screen.
```xml
<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://schemas.microsoft.com/dotnet/2021/maui"
            xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
            xmlns:skia="clr-namespace:SkiaSharp.Views.Maui.Controls;assembly=SkiaSharp.Views.Maui.Controls"
            x:Class="BarcodeQrScanner.DesktopCameraPage"
            Title="DesktopCameraPage">
    <Grid x:Name="scannerView" Margin="0">
        <skia:SKCanvasView x:Name="canvasView"
                          Margin="0"
                          HorizontalOptions="FillAndExpand" VerticalOptions="FillAndExpand"
                          PaintSurface="OnCanvasViewPaintSurface" />

    </Grid>
</ContentPage>
```

Afterwards, instantiate the barcode SDK and register the `OnDisappearing` and `OnAppearing` callback functions in the constructor:

```cs
public partial class DesktopCameraPage : ContentPage
{
    private BarcodeQRCodeReader reader;
    private Thread thread;
    private volatile bool isCapturing;
    private VideoCapture capture;

    private ConcurrentQueue<SKBitmap> _bitmapQueue = new ConcurrentQueue<SKBitmap>();
    private SKBitmap _bitmap;

    private static object lockObject = new object();

    public DesktopCameraPage()
    {
      InitializeComponent();
      this.Disappearing += OnDisappearing;
      this.Appearing += OnAppearing;
      
      reader = BarcodeQRCodeReader.Create();
    }
}
```

The two callback functions are used to start and stop the camera capture thread when the page is shown or hidden. For example, when you press the back button in the navigation bar, the `OnDisappearing` function will be called to release the camera resources. 

```csharp
private void OnAppearing(object sender, EventArgs e)
{
    Create();
}

private void OnDisappearing(object sender, EventArgs e)
{
    Destroy();
}

private void Create()
{
    lock (lockObject)
    {
        capture = new VideoCapture(0);

        if (capture.IsOpened())
        {
            isCapturing = true;
            thread = new Thread(new ThreadStart(FrameCallback));
            thread.Start();
        }
    }
}

private void Destroy()
{
    lock (lockObject)
    {
        if (thread != null)
        {
            isCapturing = false;

            thread.Join();
            thread = null;
        }

        if (capture != null && capture.IsOpened())
        {
            capture.Release();
            capture = null;
        }

        ClearQueue();

        if (_bitmap != null)
        {
            _bitmap.Dispose();
            _bitmap = null;
        }
    }
}
```

As the camera capture thread is started, it will call the `FrameCallback` function to capture camera frames in a loop.
```csharp
private void FrameCallback()
{
    while (isCapturing)
    {
        Decode();
    }
}
```

What does the `Decode()` function do:
1. Capture a camera frame to `Mat` using OpenCV `VideoCapture.Read()`:
    
    ```csharp
    Mat mat = new Mat();
    capture.Read(mat);
    ```

2. Convert the `Mat` to `byte[]`:
    
    ```csharp
    int length = mat.Cols * mat.Rows * mat.ElemSize();
    if (length == 0) return;
    byte[] bytes = new byte[length];
    Marshal.Copy(mat.Data, bytes, 0, length);
    ```

3. Decode barcodes from the `byte[]` using the barcode SDK:
    
    ```csharp
    BarcodeQRCodeReader.Result[] results = reader.DecodeBuffer(bytes, mat.Cols, mat.Rows, (int)mat.Step(), BarcodeQRCodeReader.ImagePixelFormat.IPF_RGB_888);
    ```
4. Draw the barcode contours and text results on the camera frame using OpenCV `Cv2.DrawContours` and `Cv2.PutText`:
    
    ```csharp
    if (results != null)
    {
        foreach (BarcodeQRCodeReader.Result result in results)
        {
            int[] points = result.Points;
            if (points != null)
            {
                OpenCvSharp.Point[] all = new OpenCvSharp.Point[4];
                int xMin = points[0], yMax = points[1];
                all[0] = new OpenCvSharp.Point(xMin, yMax);
                for (int i = 2; i < 7; i += 2)
                {
                    int x = points[i];
                    int y = points[i + 1];
                    OpenCvSharp.Point p = new OpenCvSharp.Point(x, y);
                    xMin = x < xMin ? x : xMin;
                    yMax = y > yMax ? y : yMax;
                    all[i / 2] = p;
                }
                OpenCvSharp.Point[][] contours = new OpenCvSharp.Point[][] { all };
                Cv2.DrawContours(mat, contours, 0, new Scalar(0, 0, 255), 2);
                if (result.Text != null) Cv2.PutText(mat, result.Text, new OpenCvSharp.Point(xMin, yMax), HersheyFonts.HersheySimplex, 1, new Scalar(0, 0, 255), 2);
            }
        }
    }
    ```
5. Convert the color space of the frame from `BGR` to `RGBA`:
    
    ```csharp
    Mat newFrame = new Mat();
    Cv2.CvtColor(mat, newFrame, ColorConversionCodes.BGR2RGBA);
    ```
6. Convert `Mat` to `SKBitmap`:
    ```csharp
    SKBitmap bitmap = new SKBitmap(mat.Cols, mat.Rows, SKColorType.Rgba8888, SKAlphaType.Premul);
    bitmap.SetPixels(newFrame.Data);
    ```
7. Queue the `SKBitmap` to the `_bitmapQueue` and invalidate the `SKCanvasView`:
    
    ```csharp
    if (_bitmapQueue.Count == 2) ClearQueue();
    _bitmapQueue.Enqueue(bitmap);
    canvasView.InvalidateSurface();
    ```

After the `InvalidateSurface()` is called, the `OnCanvasViewPaintSurface` function will be triggered to draw the `SKBitmap` on the screen:
```csharp
void OnCanvasViewPaintSurface(object sender, SKPaintSurfaceEventArgs args)
{
    lock (lockObject)
    {
        if (!isCapturing) return;

        SKSurface surface = args.Surface;
        SKCanvas canvas = surface.Canvas;

        canvas.Clear();

        _bitmapQueue.TryDequeue(out _bitmap);

        if (_bitmap != null)
        {
            try
            {
                canvas.DrawBitmap(_bitmap, new SKPoint(0, 0));
            }
            catch(Exception e)
            {
                Console.WriteLine(e.Message);
            }
            finally
            {
                _bitmap.Dispose();
                _bitmap = null;
            }
        }
    }
        
}
```

![.NET MAUI Windows Camera QR Code Scanner](https://www.dynamsoft.com/codepool/img/2023/04/dotnet-maui-windows-camera-qr-scan.png)

## Source Code
[https://github.com/yushulx/dotnet-barcode-qr-code-sdk/tree/main/example/maui](https://github.com/yushulx/dotnet-barcode-qr-code-sdk/tree/main/example/maui)