Labeling Multiple Barcodes with Augmented Reality and Dynamsoft Barcode Reader
Tags: android, ar, barcode, arcore
Published: 2023-09-01 06:17:55

The term "**augmented reality (AR)**" generally refers to the integration of digital information with the user's environment in real time. AR can make user interfaces more immersive and intuitive, overlaying real-time information onto the real world to make tasks more efficient and less error-prone. For example, in warehouse management, AR can be used to track the inventory of items on shelves. In hospitals, AR can help identify tubes and bottles, and in retail, it can display product information. In this article, we will demonstrate how to use [ARCore](https://developers.google.com/ar) and [Dynamsoft Barcode Reader](https://www.dynamsoft.com/barcode-reader/sdk-mobile/) to label multiple barcodes and avoid misreading.

<video src="https://user-images.githubusercontent.com/2202306/264591723-be665855-4720-4896-a838-f9105cd5f9c2.mp4" controls="controls" muted="muted" style="max-height:640px; min-height: 200px; max-width: 100%;"></video>

## Prerequisites
- An ARCore compatible device running [Google Play Services for AR](https://play.google.com/store/apps/details?id=com.google.ar.core) (ARCore) 1.24 or later
- Android Studio 4.1 or later

## 2D Barcode Overlay on Camera Preview
Usually, we display barcode results on the camera preview as follows:

![barcode 2D overlay](https://www.dynamsoft.com/codepool/img/2023/09/camera-overlay-simple-ar.png)

The overlay is rendered on a 2D canvas that sits above the camera preview, displaying both the barcode information and its bounding box in real time.

Drawing an overlay on a camera view can be considered a basic form of augmented reality. A more advanced AR system often involves understanding the geometry of the environment and placing digital objects in a way that makes them appear to exist within the real world, responding appropriately to changes in the camera's view or the environment itself.

In the following sections, we will combine ARCore and the Dynamsoft Barcode Reader SDK to enhance the barcode scanning experience.

## About ARCore and Dynamsoft Barcode Reader
- ARCore is Google's platform for building augmented reality (AR) experiences. By utilizing three key technologies — motion tracking, environmental understanding, and light estimation — ARCore enables your phone to sense its environment, understand the world, and interact with information in a more comprehensive and contextually aware manner.
- Dynamsoft Barcode Reader SDK is a commercial barcode scanning library for Windows, Linux, macOS, Android, iOS, and web. It supports various barcode types, including QR Code, DataMatrix, PDF417, Code 128, Code 39, EAN 13, UPC-A, etc. It also provides a free trial license for 30 days.

## Combining Google's ARCore with Dynamsoft Android Barcode SDK
Let's get started with the sample project of ARCore: [https://github.com/googlesamples/arcore-ml-sample](https://github.com/googlesamples/arcore-ml-sample).

The Android sample written in Kotlin demonstrates how to utilize ARCore and computer vision algorithms for object detection and tracking. For us, the primary goal is the detection and tracking of barcodes. Integrating the Dynamsoft Barcode Reader into the sample project involves a series of steps.

### Add Dynamsoft Barcode Reader SDK to the Project
1. Add the Dynamsoft repository source `https://download2.dynamsoft.com/maven/aar` to the project's `build.gradle` file:
    
    ```groovy
    allprojects {
        repositories {
            google()
            mavenLocal()
            mavenCentral()
            maven {
                url "https://download2.dynamsoft.com/maven/aar"
            }
        }
    }
    ```
2. Configure the dependency in the app's `build.gradle` file:
    
    ```groovy
    implementation 'com.dynamsoft:dynamsoftbarcodereader:9.6.20'
    ```

### Detect Barcode, QR Code and DataMatrix in Kotlin
1. In `MainActivity.kt`, set the license key. You can get a free trial license from [here](https://www.dynamsoft.com/customer/license/trialLicense/?product=dbr&source=codepool).
    
    ```kotlin
    import com.dynamsoft.dbr.BarcodeReader
    
    BarcodeReader.initLicense("LICENSE-KEY") { isSuccessful, e ->
      runOnUiThread {
        if (!isSuccessful) {
          e.printStackTrace()
          Log.e(TAG, "Failed to verify the license: $e")
        }
      }
    }
    ```

2. Create a barcode reader object in `AppRenderer.kt`:
    
    ```kotlin
    import com.dynamsoft.dbr.BarcodeReader
    import com.dynamsoft.dbr.BarcodeReaderException
    import com.dynamsoft.dbr.EnumImagePixelFormat
    import com.dynamsoft.dbr.Point

    var reader: BarcodeReader? = null

    fun bindView(view: MainActivityView) {
        try {
          reader = BarcodeReader()
          val settings = reader!!.runtimeSettings
          settings.expectedBarcodesCount = 999
          reader!!.updateRuntimeSettings(settings)
        } catch (e: BarcodeReaderException) {
          e.printStackTrace()
        }
    }
    ```   
3. Find the code block that starts with `launch(Dispatchers.IO)`. You can utilize the acquired camera image to detect barcodes:
    
    ```kotlin
    if (reader != null) {
        var bytes = ByteArray(cameraImage.planes[0].buffer.remaining())
        cameraImage.planes[0].buffer.get(bytes)

        var results = reader!!.decodeBuffer(bytes, cameraImage.width, cameraImage.height, cameraImage.planes[0].rowStride, EnumImagePixelFormat.IPF_NV21)
        objectResults = emptyList()
        if (results != null && results.isNotEmpty()) {
        val tmp: MutableList<DetectedObjectResult> = mutableListOf()
        for (result in results) {
            var points = result.localizationResult.resultPoints
            var confidence = 100

            val (x1, y1) = points[0].x to points[0].y
            val (x2, y2) = points[1].x to points[1].y
            val (x3, y3) = points[2].x to points[2].y
            val (x4, y4) = points[3].x to points[3].y
            val centerX = (x1 + x2 + x3 + x4) / 4
            val centerY = (y1 + y2 + y3 + y4) / 4
            val content = result.barcodeText
            val label = "✓"

            val detectedObjectResult = DetectedObjectResult(confidence.toFloat(), label, centerX.toInt() to centerY.toInt(), content)
            tmp.add(detectedObjectResult)
        }
        objectResults = tmp
        }
    }
    ```
    
A barcode detection result contains four points, which are the four corners of the barcode. We can calculate the center point of the barcode and use it as the anchor point for the label.

```kotlin
val (x1, y1) = points[0].x to points[0].y
val (x2, y2) = points[1].x to points[1].y
val (x3, y3) = points[2].x to points[2].y
val (x4, y4) = points[3].x to points[3].y
val centerX = (x1 + x2 + x3 + x4) / 4
val centerY = (y1 + y2 + y3 + y4) / 4
```

We convert the barcode results to a `DetectedObjectResult` list, which will be rendered on the screen later. 

If we set the `label` to `result.barcodeText`, the content of the barcode will be displayed on the screen. 

![barcode content](https://www.dynamsoft.com/codepool/img/2023/09/augmented-reality-barcode-text.jpg)

The effect works well for isolated barcodes. However, if multiple barcodes appear in the camera view, the display can become overcrowded. To address this issue, we can set the `label` to `✓` and present the barcode content in a list view.

![barcode detection results](https://www.dynamsoft.com/codepool/img/2023/09/barcode-detection-results.jpg)
    

### Enable Camera Auto Focus 
By default, the ARCore camera is set to fixed focus, which can make the barcode appear blurry. To enhance barcode detection accuracy, enabling auto-focus is necessary. 

A switch button is added to the UI to toggle the focus mode.

1. Create a switch button in the `activity_main.xml` layout file :
    
    ```xml
    <androidx.appcompat.widget.SwitchCompat
          android:id="@+id/switch_focus_mode"
          android:layout_width="wrap_content"
          android:layout_height="wrap_content"
          android:layout_alignParentEnd="true"
          android:checked="false"
          android:text="@string/switch_focus_mode"
          android:textColor="#ffffff" />
    ```

2. In `MainActivityView.kt`, obtain the reference to the switch button:
    
    ```kotlin
    var focusModeSwitch = root.findViewById<SwitchCompat>(R.id.switch_focus_mode)
    ```

3. Register the `OnCheckedChangeListener` of the switch button in `AppRenderer.kt`:
    
    ```kotlin
    view.focusModeSwitch.setOnCheckedChangeListener { _, isChecked ->
      val session = activity.arCoreSessionHelper.sessionCache ?: return@setOnCheckedChangeListener
      val config = session.config
      config.focusMode = if (isChecked) Config.FocusMode.AUTO else Config.FocusMode.FIXED
      session.configure(config)
    }
    ```

## Source Code
[https://github.com/yushulx/android-arcore-barcode-qr-detection](https://github.com/yushulx/android-arcore-barcode-qr-detection)