Image Labeler App - AWS Amplify Auth, API (GraphQL), DataStore, Storage, Predictions + React Native
Tags: aws, awsamplify, reactnative, graphql
Published: 2021-04-09 20:15:11

Photo by <a href="https://unsplash.com/@alexb?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Alex Blăjan</a> on <a href="https://unsplash.com/s/photos/camera-phone?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>

## Intro
  
A while back I created a couple of apps to test drive some new (to me) frameworks: 
{% medium https://medium.com/google-cloud/flutter-vision-flutter-firebase-ml-vision-firebase-cloud-firestore-76f3794f5d38 %} 
{% medium https://medium.com/google-cloud/ionic-3-google-cloud-vision-firebase-78ea68cfbea8 %}

Continuing in that tradition I decided to write a similar app using AWS Amplify and React Native. Thanks to AWS Amplify I ended up with a much more secure and private version of my previous apps with virtually no additional work from me.

The app we will build allows a user to take a photo which is uploaded and stored securely and privately. The photo is then analyzed using Amplify Predictions which returns a list of labels. The labels are then stored along with the image key so that the user can view a list of all of the images and the associated labels. 

Here is a little preview of what we are building: {% youtube j_7j3BzBHlo %}

The finished app is available here:
{% github https://github.com/cbetz/react-native-amplify-image-labeler %}

Here is a list of frameworks and libraries we will use:
- React Native
- AWS Amplify Authentication
- AWS Amplify API (GraphQL)
- AWS Amplify DataStore
- AWS Amplify Storage
- AWS Amplify Predictions
- react-native-camera
- react-native-paper
- react-native-vector-icons

Ready? Let's get to it. 

## Environment and AWS Setup

First, let's get your environment and the Amplify CLI setup. Follow the instructions [here](https://docs.amplify.aws/start/getting-started/installation/q/integration/react-native). At this point your environment is setup and you've linked your AWS account to the Amplify CLI.

## React Native + Amplify Project Setup

Second, we need to initialize our fullstack project including React Native and Amplify. First create a new React Native project following the instructions [here](https://docs.amplify.aws/start/getting-started/setup/q/integration/react-native#initialize-a-new-backend). You can change `RNAmplify` to whatever name you'd like; I used `RNAmplifyImageLabeler`. Stop when you reach the bottom of the page, don't proceed to **Connect API and database to the app** as we are going to do something different next.

## Amplify API and Datastore

Next we are going to add our API and Datastore. Under the covers these steps will create and setup both AWS App Sync and DynamoDB. We are going to use [GraphQL](https://graphql.org/) for our API. Amplify makes this easy for us:

```
amplify add api
```

Answer the prompts as follows:
```
? Please select from one of the below mentioned services: GraphQL
? Provide API name: rnimagelabeler
? Choose the default authorization type for the API API key
? Enter a description for the API key: demo
? After how many days from now the API key should expire (1-365): 365
? Do you want to configure advanced settings for the GraphQL API No, I am done.
? Do you have an annotated GraphQL schema? No
? Choose a schema template: Single object with fields (e.g., “Todo” with ID, name, description)
? Do you want to edit the schema now? Yes
```
This should open a file, `schema.graphql`, in your editor. If not, you can find it in your project at `amplify/backend/api/rnimagelabeler/schema.graphql`

You should see a Todo model:
```
type Todo @model {
  id: ID!
  name: String!
  description: String
}
``` 
Change this to:
```
type Image @model {
  id: ID!
  key: String!
  labels: [String]
}
```
An image object will have an id, a key that corresponds to the file key, and a list of labels that will store the labels found by Amplify Predictions. 

Let's push this backend to AWS:

```
amplify push
``` 
You'll see another series of prompts:
```
? Are you sure you want to continue? Yes
? Do you want to generate code for your newly created GraphQL API Yes
? Choose the code generation language target javascript
? Enter the file name pattern of graphql queries, mutations and subscriptions src/graphql/**/*.js
? Do you want to generate/update all possible GraphQL operations - queries, mutations and subscriptions Yes
? Enter maximum statement depth [increase from default if your schema is deeply nested] 2
```
When complete, you should see 
```
✔ Generated GraphQL operations successfully and saved at graphql
✔ All resources are updated in the cloud
```
Awesome! Let's test it. Open up `App.js`. 
{% collapsible If you are using VS Code and run into issues after opening **App.js** check this out %} I ran into some compatibility issues with [Flow](https://flow.org/) and VS Code. If you run into issues make sure you follow the instructions [here](https://github.com/flowtype/flow-for-vscode). I also had to change the version in `.flowconfig` to `>=0.107.0`. {% endcollapsible %}

Let's add a list to display some of the image info and a button to add a new image. We will build upon this to actually deal with images in a later step. Change `App.js` to this:

{% gist https://gist.github.com/cbetz/70261dcbb3a028f7168b15289eb4daea/7f2b8facda828a93a41cdf44b444248ed7b8a20f %}

`fetchImages()` uses the auto-generated listImages to get a list of images from our API. `addImage()` is a temporary function to test adding a new image with a `key` value of the timestamp and `labels` set to a list of test labels. 

Now let's run it on an iOS simulator:
```
npx react-native run-ios
```
Or, you can run on an Android emulator:
```
npx react-native run-android
```

You should see something like this:
![Test adding an image](https://firebasestorage.googleapis.com/v0/b/cbetzdotcom.appspot.com/o/rn-amplify-imagelabeler-1.gif?alt=media&token=dc6bbdb8-2e73-4a69-be10-f9e29b24743e)

We just successfully tested creating and fetching image objects using Amplify API and DataStore! 

## Amplify Authentication

Let's add authentication to our app. We want to make sure that only the owner of the images is able to view their images. We need to make sure we secure the image objects in the database as well as the images we upload (covered in the next step). We also need a way for users to register for our app, login, reset their password, etc. Lucky for us Amplify makes this *ridiculously* easy! 

First let's add auth to our backend:
```
amplify add auth
```
And follow the prompts (you can choose a different sign in method if you'd like):
```
Do you want to use the default authentication and security configuration? Default configuration
How do you want users to be able to sign in? Email
Do you want to configure advanced settings? No, I am done.
```
Let's push our changes:
```
amplify push
```
If you're asked if you'd like to continue just hit Enter.

Now let's add to our app. This is the really cool part. Add an import to App.js:
```
import { withAuthenticator } from 'aws-amplify-react-native';
```
And change the last line of App.js...
```
export default withAuthenticator(App);
```
...and save. You should see your simulator or emulator refresh and voilà! We now have a complete auth flow with 2 lines of code. 🤯 

![Authentication](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tjfciwu1rri16gg4qo59.png)

Let's update our Image model to use [Owner Based Authorization](https://docs.amplify.aws/lib/datastore/setup-auth-rules/q/platform/ios#owner-based-authorization). This specifies create, read, update, and delete operations are private to the owner. Change  `amplify/backend/api/rnimagelabeler/schema.graphql` to
```
type Image @model @auth(rules: [{ allow: owner }]) {
  id: ID!
  key: String!
  labels: [String]
}
``` 
We now have to update our API to use Amazon Cognito instead of the API Key we initially setup:
```
amplify update api
```

Walk through the series of prompts:
```
? Please select from one of the below mentioned services: GraphQL
? Select from the options below Update auth settings
? Choose the default authorization type for the API Amazon Cognito User Pool
Use a Cognito user pool configured as a part of this project.
? Configure additional auth types? No
```
Push the changes we just made to the backend:
```
amplify push
```
Once that is complete go back to your simulator (or emulator), tap Sign Up, create an account, then sign in. Where did our test Image object go? Now that we're using owner based auth we don't see any as we don't own any. Before we create any more image objects let's add the ability to upload and display actual images.

## Amplify Storage

We need a way to use the device's camera and upload to Storage (S3). There are many react native camera libraries out there but we are going to use `react-native-camera`. Let's add it now:
```
yarn add react-native-camera
```
Then update pods if you're building an iOS app: 
```
npx pod-install
```
There are some additional steps around adding permissions that we need to do. Follow instructions for [iOS](https://github.com/react-native-camera/react-native-camera/blob/master/docs/installation.md#ios---other-required-steps) and/or [Android](https://github.com/react-native-camera/react-native-camera/blob/master/docs/installation.md#android---other-required-steps).

Next, add storage:
```
amplify add storage
```
And follow the prompts:
```
? Please select from one of the below mentioned services: Content (Images, audio, video, etc.)
? Please provide a friendly name for your resource that will be used to label this category in the project: rnamplifyimagelabelerstorage
? Please provide bucket name: rnimagelabelerdceb74205c75476884aef48a88a40517
? Who should have access: Auth users only
? What kind of access do you want for Authenticated users? create/update, read, delete
? Do you want to add a Lambda Trigger for your S3 Bucket? No
```
Push to the cloud:
```
amplify push
```

Before we go any further we have to change `index.js` to get around this issue {% github https://github.com/aws-amplify/amplify-js/issues/5918 %} 

Change
```
Amplify.configure(config);
```
to 
```
Amplify.configure({
    ...config,
    Analytics: {
        disabled: true,
    },
});
```

Now, let's update `App.js` to take a picture with the device camera, upload it to S3, and save the info to the database:

{% gist https://gist.github.com/cbetz/70261dcbb3a028f7168b15289eb4daea/2f1b3653b580f94dbae66ef54b891c92cc9c6642 %}

We've added a few things. `onShowCameraPressed()` sets the newly created `showCamera` state, which allows us to present the camera view to the user. Inside the camera view we have a button, **Label Image**, that is bound to `takePicture()`. `takePicture()` sets some camera options and takes a photo. In order to get the image in a format we can upload to S3, we fetch the image file and take the [blob]
(https://developer.mozilla.org/en-US/docs/Web/API/Body/blob) from the response. We then grab the filename to use as the key and use `Storage.put()` to upload the image. We specify `{ level: 'private' }` to ensure the image is only viewable by the owner of the image. We then add the image to the database with the S3 key and dismiss the camera view. 

We need a way to display the image back to the user using the S3 key. Lucky for us Amplify has `S3Image` that we can pass the key and `level="private"` along with some basic styling. That's all we need!

Now let's test this on a real device since we want to use the camera. For iOS, run:
```
npx react-native run-ios --device
```

You will probably see an error similar to this:
```
error Failed to build iOS project. We ran "xcodebuild" command but it exited with error code 65. To debug build logs further, consider building your app with Xcode.app, by opening RNAmplifyImageLabeler.xcworkspace.
```

Since we are testing on a device we need to add a team to our project. Open up XCode (change the path to whatever you named your project):
```
open ios/RNAmplifyImageLabeler.xcworkspace/
```
Make sure your app project is selected on the left:
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1t2v7t18i9tf9nsw6l36.png)

Then click the **Signing & Capabilities** tab and choose a Team:

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o9re7jekw38it6s3zbnf.png)

Back to the terminal. You should be able to build and run the app now:
```
npx react-native run-ios --device
``` 
Go ahead and take a picture. You should see something like the following:
![Saving image to storage](https://firebasestorage.googleapis.com/v0/b/cbetzdotcom.appspot.com/o/rn-amplify-imagelabeler-2.gif?alt=media&token=e03d80ac-c31c-458d-b8dc-da77e2b48067)

Sweet! We are still adding the test labels to each image. Let's add image labeling next.

## Amplify Predictions

[Amplify Predictions](https://docs.amplify.aws/lib/predictions/intro/q/platform/js#iam-policy) offers language translations, text to speech, audio transcriptions, and much more. We are going to use it to label objects in our images. Under the covers, this utilizes [Amazon Rekognition](https://docs.aws.amazon.com/rekognition/latest/dg/what-is.html) 

First, add it:
```
amplify add predictions
```
Then follow the prompts:
```
? Please select from one of the categories below Identify
? What would you like to identify? Identify Labels
? Provide a friendly name for your resource rnamplifyimagelabeler
? Would you like use the default configuration? Default Configuration
? Who should have access? Auth users only
```
Next run:
```
amplify push
```
Now we need to [Configure our app](https://docs.amplify.aws/lib/predictions/getting-started/q/platform/js#configure-the-frontend) by changing `index.js` to 
```
import { AppRegistry } from 'react-native';
import App from './App';
import { name as appName } from './app.json';
import Amplify from 'aws-amplify';
import Predictions, { AmazonAIPredictionsProvider } from '@aws-amplify/predictions'
import config from './aws-exports';

Amplify.configure({
    ...config,
    Analytics: {
        disabled: true,
    },
});
Amplify.addPluggable(new AmazonAIPredictionsProvider());

AppRegistry.registerComponent(appName, () => App);
```

Next update `App.js` to label the images:
{% gist https://gist.github.com/cbetz/70261dcbb3a028f7168b15289eb4daea/0b8782f23645f41e76db43dcfadf19068855ef6c %}

Now after we upload the image to S3, we call `Predictions.identify()` with the image key, the level (private), and `type: "ALL"` (you can change type as you see fit). We then map the image label names and pass them into our `addImage()` to save them to our database. Let's fire up our app again:

![Labeling Images](https://firebasestorage.googleapis.com/v0/b/cbetzdotcom.appspot.com/o/rn-amplify-imagelabeler-3.gif?alt=media&token=af783abd-38f3-4725-967b-a0ac2c006171)

Awesome! We allow our users to register for our app, save images privately to the cloud, identify labels, and display a list back to them. From here we can add all sorts of functionality: removing images, displaying labels based on confidence (we are showing all so far), searching the list of images based on labels, adding bounding boxes to the image to outline the objects that are labeled, etc, etc. If anyone ends up using this as the basis for an app, I'd love to hear about it!

Before we go, let's clean up our app a bit.

## React Native Paper

[Paper](https://callstack.github.io/react-native-paper/index.html) is "Paper is a collection of customizable and production-ready components for React Native, following Google’s Material Design guidelines." 

In our case we are going to use [Cards](https://callstack.github.io/react-native-paper/card.html) and [AppBar](https://callstack.github.io/react-native-paper/appbar.html) to spruce up our app. 

Run
```yarn add react-native-paper```
Then 
```yarn add react-native-vector-icons```

On iOS linking isn't working (at least for me), so I followed [these instructions](https://github.com/oblador/react-native-vector-icons#option-with-cocoapods). Add the following line to `ios/Podfile`:
```
pod 'RNVectorIcons', :path => '../node_modules/react-native-vector-icons'
```
Then 
```
npx pod-install
```
Finally, add the following to the end of `ios/<project name>/Info.plist`, inside the closing `</dict>`:
```
<key>UIAppFonts</key>
<array>
     <string>MaterialCommunityIcons.ttf</string>
</array>
```

Now let's change `App.js` a bit:

{% gist https://gist.github.com/cbetz/70261dcbb3a028f7168b15289eb4daea/840d91dd3870bcc27714f9d2392aea91df4a4f5f %}

Let's do a fresh build and test:
```
npx react-native run-ios --device
``` 

![Finished App](https://firebasestorage.googleapis.com/v0/b/cbetzdotcom.appspot.com/o/rn-amplify-imagelabeler-4.gif?alt=media&token=6565b324-1c4d-4f40-8d35-25369272e256)

Beautiful! 

Let me know if you run into any issues and I'll do my best to help. I'm thinking about re-creating the above as either a live coding session or a prerecorded video. Let me know if you're interested!