Create a React Native Image Recognition App with Google Vision API
Tags: reactnative, firebase, javascript
Published: 2019-02-15 17:24:43

---
title: Create a React Native Image Recognition App with Google Vision API
published: true
description: In this tutorial, you'll learn how to use the Google Vision API to identify the content of images in a React Native mobile application.
tags: React Native, Firebase, JavaScript
canonical_url: https://blog.jscrambler.com/create-a-react-native-image-recognition-app-with-google-vision-api/
cover_image: https://thepracticaldev.s3.amazonaws.com/i/4uc9h5ypsxu6gtfjsdom.jpg
---


**This article was originally published on the [Jscrambler Blog](https://blog.jscrambler.com/create-a-react-native-image-recognition-app-with-google-vision-api/?utm_source=dev.to&utm_medium=referral) by Aman Mittal**.

Google Cloud Vision API is a machine learning tool that can classify details from an image provided as an input into thousands of different categories with pre-trained API models. It offers these pre-trained models through an API and the categories are detected as individual objects within the image.

In this tutorial, you are going to learn how to integrate Google Cloud Vision API in a React Native application and make use of real-time APIs. You can find the complete code inside [this GitHub repo](https://github.com/JscramblerBlog/google-vision-rn-demo).

## Installing Expo

If you are not familiar with Expo, this tutorial can be a good start. Basically, Expo provides a set of tools to create and publish React Native applications with minimal effort. Earlier, React Native had something called `create-react-native-app` which is now merged with Expo-Cli and is an official way to build a React Native app.

To create your React Native app, you need to install Expo as a global npm module.

```shell
npm install -g expo-cli
```

Once the command line interface for Expo is installed in your local development environment, you must run the following command in order to generate a project.

```shell
expo-cli init google-vision-rn-demo
```

It will ask you for which template to use; choose the option **blank template** rather than tabs template. We only need a single screen in our application for the demonstration purposes. In the last step, you will be prompted to write the name of the project — simply type it and hit enter. Then, it will start installing dependencies.

Once the project is created, traverse into the project directory. If you need any help with this setup, refer to the [Expo documentation](https://docs.expo.io/versions/v32.0.0/workflow/configuration/).

## Setting Up Firebase

In this section, we are going to set up a new Firebase project. It will provide us the database and backend service and we do not have to write our own backend for this tutorial, hence saving time and focusing on what we need to learn. For simplicity, we are going to make the Firebase project data public for demonstration purposes.

Visit [Firebase](https://console.firebase.google.com/) and sign-in with your Google ID. Once signed-in, click on a new project and enter a name. Lastly, hit the **Create Project** button.

<center><img src="https://thepracticaldev.s3.amazonaws.com/i/0azv7c3hkulaf24koo8z.jpg" alt="Firebase - Add Project" style="max-height: 600px;"></center>

The next step is to make sure we set up Firebase database rules to allow us to upload image files through the app. From the left-hand side menu in the Firebase console, open `Database` tab and then choose `Rules` and modify them as follows.

```js
service cloud.firestore {
 match /databases/{database}/documents {
   match /{document=**} {
     allow read, write;
   }
 }
}
```

We need to install the Firebase SDK in our React Native app. Run the following command from your terminal.

```shell
npm install -S firebase
```

Now, create a folder called `config` and inside it, create a new file called `environment.js`. This file will contain all keys needed to bootstrap and hook Firebase with our application.

```js
//environment.js
var environments = {
  staging: {
    FIREBASE_API_KEY: 'XXXX',
    FIREBASE_AUTH_DOMAIN: 'XXXX',
    FIREBASE_DATABASE_URL: 'XXXX',
    FIREBASE_PROJECT_ID: 'XXXX',
    FIREBASE_STORAGE_BUCKET: 'XXXX',
    FIREBASE_MESSAGING_SENDER_ID: 'XXXX',
    GOOGLE_CLOUD_VISION_API_KEY: 'XXXX'
  },
  production: {
    // Warning: This file still gets included in your native binary and is not a secure way to store secrets if you build for the app stores. Details: https://github.com/expo/expo/issues/83
  }
};

function getReleaseChannel() {
  let releaseChannel = Expo.Constants.manifest.releaseChannel;
  if (releaseChannel === undefined) {
    return 'staging';
  } else if (releaseChannel === 'staging') {
    return 'staging';
  } else {
    return 'staging';
  }
}
function getEnvironment(env) {
  console.log('Release Channel: ', getReleaseChannel());
  return environments[env];
}
var Environment = getEnvironment(getReleaseChannel());
export default Environment;
```

The `X`s are values of each key you have to fill in. Ignore the value for Key `GOOGLE_CLOUD_VISION_API_KEY` right now as we will get back to it in the next section. Other values for their corresponding keys can be attained at the Firebase console. You can get these values by visiting the Firebase console and then click the gear icon next to `Project Overview` in the left-hand side menu bar and lastly go to `Project settings` section. There are ways in Expo where you do not have to publish your secret keys when deploying the app or uploading the codebase on a site like GitHub. The initial step we recommend is to add this file inside `.gitignore`.

Then create another file called `firebase.js` inside the `config` directory. We will be using this file in the main application to send requests to upload an image to the Firebase storage. Also, note that we are importing `environment.js` in it to access Firebase keys.

```js
// firebase.js
import * as firebase from 'firebase';

firebase.initializeApp({
  apiKey: Environment['FIREBASE_API_KEY'],
  authDomain: Environment['FIREBASE_AUTH_DOMAIN'],
  databaseURL: Environment['FIREBASE_DATABASE_URL'],
  projectId: Environment['FIREBASE_PROJECT_ID'],
  storageBucket: Environment['FIREBASE_STORAGE_BUCKET'],
  messagingSenderId: Environment['FIREBASE_MESSAGING_SENDER_ID']
});

export default firebase;
```

## Getting Google Cloud Vision API Key

To use a Google Cloud Platform service, you need a Gmail account. Once you are signed-in from your Gmail ID, you can visit the [Google Cloud Console](https://console.cloud.google.com/). The next step is to create a new project.

![Google Cloud Platform](https://blog.jscrambler.com/content/images/2019/02/jscrambler-blog-create-react-native-image-recognition-app-google-vision-api-dashboard.jpg)

Click `select a project` from the drop-down menu and then `click new project`. Enter the name of your project and then click `Create`. Once you’ve created the project, we are placed back into the main console page again and then need to select our newly created project.

The next step in this process is to get your API key. This you can get by clicking on the console and moving over to `Dashboard` section and under that choose `Enable APIs and Services`.

![Enable API](https://blog.jscrambler.com/content/images/2019/02/jscrambler-blog-create-react-native-image-recognition-app-google-vision-api-enable-api.jpg)

Then type **vision** in the search as shown below and then click `Vision API`.

![Vision API](https://blog.jscrambler.com/content/images/2019/02/jscrambler-blog-create-react-native-image-recognition-app-google-vision-api-vision.jpg)

Lastly, click `Enable` like below

![Vision API Enable](https://blog.jscrambler.com/content/images/2019/02/jscrambler-blog-create-react-native-image-recognition-app-google-vision-api-enable-1.jpg)

In order to complete this process of enabling Vision API services, you are required to add billing information (if you haven't done already) to your Google Cloud Platform account.

Your URL in the dashboard will look like this: `https://console.cloud.google.com/apis/dashboard?project=FIREBASE-PROJECT-ID&folder&organizationId`. Once you are at the below screen, click on the `Credentials` section from the left-hand side menu and create a new API key if there isn't any by clicking on the button `Create Credentials` and then `API Key`.

![Create Credentials](https://blog.jscrambler.com/content/images/2019/02/jscrambler-blog-create-react-native-image-recognition-app-google-vision-api-credentials.jpg)

Once you have created your API key, it is time to add it in the file `environment.js` for the key `GOOGLE_CLOUD_VISION_API_KEY`.

That's it. Setting up the APIs is complete. We can now move on to work on the app itself.

## Building The App

To get started, we need to install an npm package called `uuid` to create a unique blob for the image that is going to upload on the Firebase storage service. Run the command `npm install --save uuid`. Next, open `App.js` and paste the code you can find [here](https://github.com/JscramblerBlog/google-vision-rn-demo/blob/master/App.js).

Note that most of the source code for accessing and uploading to Firebase is taken from [an example](https://github.com/expo/firebase-storage-upload-example) of using Expo with Firebase. We are going to explain below the bits that are essential to connect and run Firebase. First, let us start by understanding what `uploadImageAsync` is doing.

```js
async function uploadImageAsync(uri) {
  const blob = await new Promise((resolve, reject) => {
    const xhr = new XMLHttpRequest();
    xhr.onload = function() {
      resolve(xhr.response);
    };
    xhr.onerror = function(e) {
      console.log(e);
      reject(new TypeError('Network request failed'));
    };
    xhr.responseType = 'blob';
    xhr.open('GET', uri, true);
    xhr.send(null);
  });

  const ref = firebase
    .storage()
    .ref()
    .child(uuid.v4());
  const snapshot = await ref.put(blob);

  blob.close();

  return await snapshot.ref.getDownloadURL();
}
```

As shown in the above snippet, the `uploadImageAsync` function uploads the image by creating a unique image ID or blob with the help of `uuid`. It also uses `xhr` to send a request to the Firebase storage to upload the image. We are also defining a default state in the `App` component and asking for User Permissions for both using the camera roll or gallery or take a photo from the device's camera as shown in the code snippet below.

```js
state = {
    image: null,
    uploading: false,
    googleResponse: null
  };

  async componentDidMount() {
    await Permissions.askAsync(Permissions.CAMERA_ROLL);
    await Permissions.askAsync(Permissions.CAMERA);
 }
```

The `Button` in our `App component` publishes the image to Google's Cloud Vision API.

```js
<Button
  style={{ marginBottom: 10 }}
  onPress={() => this.submitToGoogle()}
  title="Analyze!"
/>
```

The `submitToGoogle` method is what sends requests and communicates with the API to fetch the result when the button `Analyze` is pressed by the user.

```js
submitToGoogle = async () => {
   try {
     this.setState({ uploading: true });
     let { image } = this.state;
     let body = JSON.stringify({
       requests: [
         {
           features: [
             { type: "LABEL_DETECTION", maxResults: 10 },
             { type: "LANDMARK_DETECTION", maxResults: 5 },
             { type: "FACE_DETECTION", maxResults: 5 },
             { type: "LOGO_DETECTION", maxResults: 5 },
             { type: "TEXT_DETECTION", maxResults: 5 },
             { type: "DOCUMENT_TEXT_DETECTION", maxResults: 5 },
             { type: "SAFE_SEARCH_DETECTION", maxResults: 5 },
             { type: "IMAGE_PROPERTIES", maxResults: 5 },
             { type: "CROP_HINTS", maxResults: 5 },
             { type: "WEB_DETECTION", maxResults: 5 }
           ],
           image: {
             source: {
               imageUri: image
             }
           }
         }
       ]
     });
     let response = await fetch(
       "https://vision.googleapis.com/v1/images:annotate?key=" +
         Environment["GOOGLE_CLOUD_VISION_API_KEY"],
       {
         headers: {
           Accept: "application/json",
           "Content-Type": "application/json"
         },
         method: "POST",
         body: body
       }
     );
     let responseJson = await response.json();
     console.log(responseJson);
     this.setState({
       googleResponse: responseJson,
       uploading: false
     });
   } catch (error) {
     console.log(error);
   }
 };
}
```

The Vision API uses HTTP Post request as a REST API endpoint to perform data analysis on images you send in the request. This is done via the URL `https://vision.googleapis.com/v1/images:annotate`. To authenticate each request, we need the API key. The body of this POST request is in JSON format. For example:

```json
{
  "requests": [
    {
      "image": {
        "content": "/9j/7QBEUGhvdG9...image contents...eYxxxzj/Coa6Bax//Z"
      },
      "features": [
        {
          "type": "LABEL_DETECTION",
          "maxResults": 1
        }
      ]
    }
  ]
}
```

You can change the value of `maxResults` for every category. The response from the Vision API is also in JSON format.

```json
"labelAnnotations": Array [
 Object {
   "description": "water",
   "mid": "/m/0838f",
   "score": 0.97380537,
   "topicality": 0.97380537,
 },
 Object {
   "description": "waterfall",
   "mid": "/m/0j2kx",
   "score": 0.97099465,
   "topicality": 0.97099465,
 },
 Object {
   "description": "nature",
   "mid": "/m/05h0n",
   "score": 0.9594912,
   "topicality": 0.9594912,
 }
]
```

The above result can be viewed in the terminal from Expo logs. You can see how the application works with a short demo done on iOS simulator below.

<center><img src="https://blog.jscrambler.com/content/images/2019/02/jscrambler-blog-create-react-native-image-recognition-app-google-vision-api-final.gif" alt="Demo" style="max-height: 600px;"></center>


If you visit the storage section in Firebase, you can notice that each image is stored with a name of base64 binary string.

![Stored Files](https://blog.jscrambler.com/content/images/2019/02/jscrambler-blog-create-react-native-image-recognition-app-google-vision-api-files.jpg)

If you have a real device, just download the Expo client, scan the QR code and then you can try the `Take a photo` feature inside the application.

## Conclusion

In this tutorial, we’ve shown you how to integrate Firebase storage services and use a machine learning API such as Google's Vision API with a React Native and Expo application.

If you have any questions, you can ask the author directly by tweeting to [@amanhimself](https://twitter.com/amanhimself).