AWS Textract and React Native
Tags: aws, reactnative, nodej, npm
Published: 2022-06-14 12:35:36


Amazon Textract is a machine learning (ML) service presented by Amazon which extracts text, handwriting, and data from scanned documents, PDFs and images into text documents which then can be stored in any kind of storage service such as DynamoDB, s3, etc....

Today I will go through the process of capturing or selecting an image from react native mobile application and uploading these images to S3, then the data extraction from those images will occur once we trigger a lambda function using API Gateway, then after processing the data we will insert this data as DynamoDB records.

![Textract LucidChart](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kwgxzyl23v6s00n21dtc.png)

**Prerequisites**
- npm or yarn
- react-native > 0.59
- aws-amplify
- nodejs
- aws-sdk

> In this blog I will assume you have some knowledge in developing and deploying lambda functions and API Gateway.

Let us split our blog into 2 main parts:

1. Frontend Part
2. Backend Part

**1. Frontend Part**
In this section we will handle the upload of images we captured in our mobile application into S3 in order for our backend to extract the data from these images.

First of all we will start by installing:

- [aws-amplify](https://docs.amplify.aws/start/q/integration/react-native/) for react native.
`npm install aws-amplify` or using `npm install @aws-amplify/api @aws-amplify/core @aws-amplify/storage` since we don't need all the aws-amplify libraries
- [react-native-image-picker](https://www.npmjs.com/package/react-native-image-picker) to select a photo from the device library or camera.
`npm install react-native-image-picker`

We will start by implementing two functions, one if user choose image from library and one from camera :

```javascript
import {launchCamera, launchImageLibrary} from 'react-native-image-picker';


//Inside your component

const options = {
  mediaType: 'photo',
  quality: 0.5,
  includeBase64: true,
};

const libraryPickerHandler = () => {
    launchImageLibrary(options, async (response) => {
      if (response.didCancel) {
        console.log('User cancelled image picker');
      } else if (response.errorMessage) {
        console.log('ImagePicker Error: ', response.errorMessage);
      } else {
        await onImageSelect(response?.assets[0].uri);
      }
    });
  };

  const cameraPickerHandler = async () => {
    launchCamera(options, async (response) => {
      if (response.didCancel) {
        console.log('User cancelled image picker');
      } else if (response.errorMessage) {
        console.log('ImagePicker Error: ', response.errorMessage);
      } else {
        await onImageSelect(response?.assets[0].uri);
      }
    });
  };
 ```

The **onImageSelect** function will handle the upload of the image to S3, and will send the S3 key to our API endpoint **/textract-scan** that we will develop in our backend part:


```javascript
import Storage from '@aws-amplify/storage';
import API from '@aws-amplify/api';
// or
import { Storage, API } from 'aws-amplify';

const onImageSelect = async (uri: string) => {
  let imageResponse = await fetch(uri);
  const blob = await imageResponse.blob();

  // timestamp for random image names
  let naming = `{new Date().getTime()}.jpeg`;

  const s3Response = await Storage.put(naming, blob, {
    contentType: 'image/jpeg',
    level: 'protected',
  });
    
 await API.post('your-endpoint-name', '/main/textract-scan', {
    body: {
      imageKey: s3Response.key,
    },
  });
};
```

**2. Backend Part**

In this section we will handle the extraction of data from the images which will be written in nodejs.

We will start by installing:

- [aws-sdk](https://github.com/aws/aws-sdk-js-v3#getting-started) for javascript which enables you to easily work with Amazon Web Services.

`npm install aws-sdk`
or
`yarn add aws-sdk`

We will create a file called `textract.ts` which will include the lambda function called **textractScan**.
**textractScan** will be our main function which will be called from the frontend through the specified endpoint.

The function will be a post method which will take in the body a `imageKey` attribute. This `imageKey` represent the S3 object key in the specified Bucket.

you will need to add this in your serverless.yml file inside the function block :

```javascript
TextractScanLambda:
  handler: path-to-your-file/textract.textractScan
  events:
    - http:
        method: post
        path: main/textract-scan
        authorizer: aws_iam
```
now inside the `textract.ts` file we start implementing the lambda function.

lets first write our Texract service function **analyzeText** which we will be using in our lambda function:


```javascript
import { Textract } from 'aws-sdk';

const analyzeText = async(key: string) => {
  const payload = {
      Document: {
        S3Object: {
          //the bucket where you uploaded your images
          Bucket: 'BUCKET_NAME',
          Name: key,
        },
      },
    };

    return new Textract().detectDocumentText(payload);
}
```

now we start developing our lambda function **textractScan**:

```javascript
const textractScan = async (event: AWSLambda.APIGatewayProxyEvent) => {
  try {
    console.log(event);
    const body = JSON.parse(event.body);

    const { imageKey } = body;

    const analyzeTextResult = await analyzeText(imageKey);
  } catch (e) {
    console.log(e);
    return {
      statusCode: 500,
      body: JSON.stringify({ message: 'ERROR_ANALYZING_DOCUMENT' }),
    };
  }
};
```

Now that we completed the function, we can use it to extract text from images. The result in **analyzeTextResult** will contain an array of block objects that contain the text that's detected in the document, but it will be time consuming to extract the actual data that we need from this object.
That's why [aws-textract-json-parser](https://www.npmjs.com/package/aws-textract-json-parser) is made for, this library parses the json response from AWS Textract into a more usable format, which you can then insert to DynamoDB:

```javascript
import { DynamoDB } from 'aws-sdk';

const textractScan = async (event: AWSLambda.APIGatewayProxyEvent) => {
  try {
    console.log(event);
    const body = JSON.parse(event.body);

    const { imageKey } = body;

    const analyzeTextResult = await analyzeText(imageKey);
    const parsedData = await AWSJsonParser(analyzeTextResult);
    console.info(parsedData);
    const rawData = parsedData.getRawData();
    console.info(data);
      if (data.length === 0) {
        console.error('no text detected');
 return {
      statusCode: 400,
      body: JSON.stringify({ message: 'INVALID_DOCUMENT' }),
    };
      }
    const payload = {
     ...someData,
     textractData: rawData
    }
    new DynamoDB.DocumentClient(payload).put;
    
    ....

  } catch (e) {
    console.log(e);
    return {
      statusCode: 500,
      body: JSON.stringify({ message: 'ERROR_ANALYZING_DOCUMENT' }),
    };
  }
};
``` 
So now, you can achieve many scenarios that requires the user to take photos, and extract the data and associate it with his profile in simple steps.