Android QR Code Detection with TensorFlow Lite
Tags: android, tensorflow, qrcode
Published: 2022-01-13 08:54:42

A week ago, I made an experiment to test the QR code detection performance with YOLOv4 on Android. The inference time of running the YOLOv4 model on CPU is more than 100ms, which does not reach my expectation. Since [Android Neural Networks API (NNAPI)](https://developer.android.com/ndk/guides/neuralnetworks) provides hardware acceleration for TensorFlow Lite models, I decided to train a TensorFlow Lite model to speed up the inference time. 

## Steps to Train QR Code Detection Model with TensorFlow Lite Model Maker
1. Install TensorFlow Lite Model Maker:
    
    ```bash
    pip install tflite-model-maker
    ```
2. Get the [GitHub repository](https://github.com/yushulx/barcode-qrcode-images) that contains the image set:
    
    ```bash
    git clone https://github.com/yushulx/barcode-qrcode-images.git
    ```

3. Partition the image set into training and validation sets: 
    
    ```bash
    cd tensorflow-lite
    python partition_dataset.py -x -i ../images -r 0.1 -o ./
    ```
4. According to the [TensorFlow tutorial](https://www.tensorflow.org/lite/tutorials/model_maker_object_detection), create the following Python script to train the model:
    
    ```python
    from tflite_model_maker.config import QuantizationConfig
    from tflite_model_maker.config import ExportFormat
    from tflite_model_maker import model_spec
    from tflite_model_maker import object_detector

    import tensorflow as tf
    assert tf.__version__.startswith('2')

    tf.get_logger().setLevel('ERROR')
    from absl import logging
    logging.set_verbosity(logging.ERROR)

    spec = model_spec.get('efficientdet_lite0')
    train_data = object_detector.DataLoader.from_pascal_voc(images_dir="train", annotations_dir="train", label_map={1: "QR_CODE"} )
    validation_data = object_detector.DataLoader.from_pascal_voc(images_dir="test", annotations_dir="test", label_map={1: "QR_CODE"} )
    model = object_detector.create(train_data, model_spec=spec, batch_size=8, train_whole_model=True, validation_data=validation_data)
    model.export(export_dir='.')
    ```

    [EfficientDet-D0](https://github.com/google/automl/tree/master/efficientdet) has comparable accuracy as YOLOv3.

    ![about EfficientDet](https://raw.githubusercontent.com/google/automl/master/efficientdet/g3doc/flops.png)

Once the model is saved, we can download the [TensorFlow Lite example](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android) and replace the model file with the one we just created.

To make the model work:
1. Drag the `model.tflite` file to assets folder.
2. Change `TF_OD_API_MODEL_FILE` to `model.tflite` in `DetectorActivity.java`:

    ```java
    private static final String TF_OD_API_MODEL_FILE = "model.tflite";
    ```
3. Clear `labelmap.txt` and add the following line:

    ```txt
    QR_CODE
    ```

So far, the QR code detection model is ready to be used. We can test it on the Android device.

![TensorFlow lite QR code detection](https://www.dynamsoft.com/codepool/img/2022/01/tensorflow-lite-qr-code-detection.jpg)

## QR Code Scanner with TensorFlow Lite
Now, let's integrate TensorFlow Lite API into our Android QR Code Scanner.
1. Copy `model.tflite` and `labelmap.txt` to assets folder.
2. Add the TensorFlow Lite library to `build.gradle`:

    ```gradle
    // Import the Task Vision Library dependency (NNAPI is included)
    implementation 'org.tensorflow:tensorflow-lite-task-vision:0.3.0'
    ```
3. Copy `Detector.java` and `TFLiteObjectDetectionAPIModel.java` from the TensorFlow Lite example project to our project. Use NNAPI delegate for TensorFlow Lite:
    
    ```java
    private TFLiteObjectDetectionAPIModel(Context context, String modelFilename) throws IOException {
      modelBuffer = FileUtil.loadMappedFile(context, modelFilename);
      optionsBuilder = ObjectDetectorOptions.builder();
      optionsBuilder.setBaseOptions(BaseOptions.builder().useNnapi().build());
      objectDetector = ObjectDetector.createFromBufferAndOptions(modelBuffer, optionsBuilder.build());
    }
    ```
4. Initialize the model detector in `CameraXActivity.java`:
    
    ```java
    // Configuration values for the prepackaged QR Code model.
    private static final int TF_OD_API_INPUT_SIZE = 416;
    private static final boolean TF_OD_API_IS_QUANTIZED = true;
    private static final String TF_OD_API_MODEL_FILE = "model.tflite";
    private static final String TF_OD_API_LABELS_FILE = "labelmap.txt";
    // Minimum detection confidence to track a detection.
    private static final float MINIMUM_CONFIDENCE_TF_OD_API = 0.5f;
    private Detector detector;
    int cropSize = TF_OD_API_INPUT_SIZE;

    private void initTFDetector() {
        try {
            detector =
                    TFLiteObjectDetectionAPIModel.create(
                            this,
                            TF_OD_API_MODEL_FILE,
                            TF_OD_API_LABELS_FILE,
                            TF_OD_API_INPUT_SIZE,
                            TF_OD_API_IS_QUANTIZED);
            cropSize = TF_OD_API_INPUT_SIZE;
        } catch (final IOException e) {
            e.printStackTrace();
            Toast toast =
                    Toast.makeText(
                            getApplicationContext(), "Detector could not be initialized", Toast.LENGTH_SHORT);
            toast.show();
            finish();
        }
    }
    ```

5. In the camera frame callback function, convert `ImageProxy` to `Bitmap`:
    
    ```java
    analysisUseCase.setAnalyzer(cameraExecutor,
                                    imageProxy -> {
        Bitmap bitmap = ImageUtils.getBitmap(imageProxy);
        imageProxy.close();
    });
    ```

    **Note: the bitmap has been rotated, so we don't need to rotate the coordinates of the QR code bounding box for rendering overlay.**

    To understand the implementation of `getBitmap()`, please refer to [BitmapUtils.java](https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/BitmapUtils.java).

6. Detect QR code and draw bounding boxes:
    
    ```java
    final List<Detector.Recognition> tfResults = detector.recognizeImage(bitmap);
    if (tfResults.size() > 0) {
        for (final Detector.Recognition tfResult : tfResults) {
            final RectF location = tfResult.getLocation();
            if (location != null && tfResult.getConfidence() >= MINIMUM_CONFIDENCE_TF_OD_API) {
                overlay.add(new BarcodeGraphic(overlay, location, null, isPortrait));
            }
        }
    }
    ```
7. Decode QR code with Dynamsoft Barcode Reader API:
    
    ```java
    TextResult[] results = null;
    try {
        PublicRuntimeSettings settings = reader.getRuntimeSettings();
        settings.barcodeFormatIds = EnumBarcodeFormat.BF_QR_CODE;
        settings.expectedBarcodesCount = tfResults.size();
        reader.updateRuntimeSettings(settings);
    } catch (BarcodeReaderException e) {
        e.printStackTrace();
    }
    ```
8. Finally, test the QR code detection model on the Android device.
    
    ![TensorFlow Lite Android QR code scanner](https://www.dynamsoft.com/codepool/img/2022/01/tensorflow-lite-android-qr-code-scanner.jpg)

    We can see running TensorFlow Lite model to scan QR code is faster than [running YOLOv4](https://www.dynamsoft.com/codepool/android-qr-code-recognition-yolo-dynamsoft-barcode.html), but still slower than purely running Dynamsoft Barcode Reader.

## Source Code
[https://github.com/yushulx/android-camera2-preview-qr-code-scanner](https://github.com/yushulx/android-camera2-preview-qr-code-scanner)