ML Kit in React Native (Part 1)
Published: 2021-09-18 20:23:59

Machine learning isn't a science problem anymore. It's a common part of mobile apps nowadays. I found [Google ML Kit](https://developers.google.com/ml-kit) is a quite interesting solution and decided to cover how we can use it in React Native. Since there is no dedicated library we will use native modules to wire up the JavaScript side with iOS and Android SDKs. 

This tutorial will cover two topics - native modules basics for iOS and Android and [Text Recognition](https://developers.google.com/ml-kit/vision/text-recognition) API from the Google ML Kit. I also have it in video format on my [YouTube channel](https://www.youtube.com/channel/UCReKeeIMZywvQoaZPZKzQbQ/)

[![ML Kit in React Native YouTube playlist](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5qmob8ogzal4ywnn66g4.png)](https://www.youtube.com/watch?v=-cB1nXlN2PU&list=PL97fL9DAn9QzKd4-exwa-vd1rNDov8wXZ)



---
## What are we building?

![Final look gif video](https://github.com/dimaportenko/react-native-mlkit-tutorial/blob/bc416886f3d88f47d693a3fd8b23c7afa1dd1795/design/demo.gif?raw=true)

---
## Boilerplate project

We are going to recognise text from the images. So I prepared project boilerplate which allow get image from Camera or Gallery. You can download it from [here](https://github.com/dimaportenko/react-native-mlkit-tutorial/tree/boilerplate). We're using `react-native-image-picker` to select image and get its uri. 

---
## Android Native Module

Let's start with Android. And first of all, we need to create a Native Module. To do so we need simply follow the steps in the official [docs](https://reactnative.dev/docs/next/native-modules-android). 

1. Open the `./android/` project in the Android Studio.

    ![Open the ./android/ project in Android Studio](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kmm45jacn96ok361zwy4.png)

2. Create a custom native module file `./android/app/src/main/java/com/rnmlkittutorial/mlkit/TextRecognitionModule.java`

    Then add the following content:

    ```java
    package com.rnmlkittutorial.mlkit;

    import android.util.Log;
    import com.facebook.react.bridge.ReactApplicationContext;
    import com.facebook.react.bridge.ReactContextBaseJavaModule;
    import com.facebook.react.bridge.ReactMethod;

    public class TextRecognitionModule extends ReactContextBaseJavaModule {
        TextRecognitionModule(ReactApplicationContext context) {
            super(context);
        }

        @Override
        public String getName() {
            return "TextRecognitionModule";
        }

        @ReactMethod
        public void recognizeImage(String url) {
            Log.d("TextRecognitionModule", "Url: " + url);
        }
    }
    ```

    Our class `TextRecognitionModule` extends `ReactContextBaseJavaModule` which has inner implementation required by React Native. 


    Method `getName` returns our module which we will use in  React JavaScript. 
    ```java
    public String getName() {
        return "TextRecognitionModule";
    }
    ```

    `@ReactMethod` annotation means this method will be accessible on the JavaScript side. And we will use `recognizeImage` method to pass image url to the native Java side. 
    ```java
    @ReactMethod
    public void recognizeImage(String url) {
        Log.d("TextRecognitionModule", "Url: " + url);
    }
    ```

    
3. To add your Native Module to `ReactPackage`, first create a new Java Class named `TextRecognitionPackage.java` that implements `ReactPackage` inside the `./android/app/src/main/java/com/rnmlkittutorial/mlkit/` folder. Then add the following content:
    ```java
    package com.rnmlkittutorial.mlkit;

    import com.facebook.react.ReactPackage;
    import com.facebook.react.bridge.NativeModule;
    import com.facebook.react.bridge.ReactApplicationContext;
    import com.facebook.react.uimanager.ViewManager;

    import java.util.ArrayList;
    import java.util.Collections;
    import java.util.List;

    public class TextRecognitionPackage implements ReactPackage {
        @Override
        public List<ViewManager> createViewManagers(ReactApplicationContext reactContext) {
            return Collections.emptyList();
        }

        @Override
        public List<NativeModule> createNativeModules(
            ReactApplicationContext reactContext) {
            List<NativeModule> modules = new ArrayList<>();

            modules.add(new TextRecognitionModule(reactContext));

            return modules;
        }
    }

    ```

    Inside `createNativeModules` method we create an instance of our module and add it to the `NativeModules` array which this package class contains. 
    `createViewManagers` is responsible for UI components that we don't use. So simply return the empty list here.    
4. To register the `TextRecognitionModule` package, we must add `TextRecognitionPackage` to the list of packages returned in ReactNativeHost's `getPackages()` method. Open up your `MainApplication.java` file, which can be found in the following path: `./android/app/src/main/java/com/rnmlkittutorial/MainApplication.java`     

    ```java
    @Override
    protected List<ReactPackage> getPackages() {
      @SuppressWarnings("UnnecessaryLocalVariable")
      List<ReactPackage> packages = new PackageList(this).getPackages();
      // Packages that cannot be autolinked yet can be added manually here, for example:
       packages.add(new TextRecognitionPackage());
      return packages;
    }
    ```
5. Here we can test what we have built. Let's back to our TypeScript side and create file `src/mlkit/index.ts`. And put the following code into it
    ```typescript
    import {NativeModules} from 'react-native';

    const {TextRecognitionModule} = NativeModules;

    export const recognizeImage = (url: string) => {
      return TextRecognitionModule.recognizeImage(url);
    }

    ```
    We take `NativeModules` from React Native which now contain our `TextRecognitionModule`. And we can invoke `recognizeImage` function we defined on the Java side. Let's go to the `src/screens/ProcessImageScreen.tsx` and use our module function like this
    ```typescript
    ...
    import {recognizeImage} from '../mlkit';
    ...
    const proccessImage = async (url: string) => {
      if (url) {
        recognizeImage(url);
      }
    };
    ...
    
    ```
6. Now run the android app `npx react-native run-android` or whatever you like. Select the image and tap `Process Image`.

    ![Select and Process Image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/684ct4na5uyl7h9tvfed.png)

    Then you can back to the Android Studio and press Logcat tab. To find our log put `TextRecognitionModule` in the search field. This is first argument of Android `Log.d` function `Log.d("TextRecognitionModule", "Url: " + url);`

    ![Android Studio Locat tab screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4gtzobjfmi5osqujkze6.png)

We prepared our React Native app to use Google ML Kit API on Android. Let's continue in `Part 2`. Source code is available [here](https://github.com/dimaportenko/react-native-mlkit-tutorial/tree/part-1).