Building an Intelligent Reimbursement Tracking App using OCR with Gemini API + ToolJet 🚀
Tags: gemini, ai, lowcode, tooljet
Published: 2024-06-06 13:25:06


## Introduction
This tutorial will guide you through building an intelligent reimbursement tracking app with OCR using [ToolJet](https://github.com/ToolJet/ToolJet) and the Gemini API. The app will allow users to upload images of receipts, extract text from the images using OCR, and store the extracted information in a database. We'll also add an AWS S3 integration to store the receipt images.

-------------------------------------------------------------

## Prerequisites:

- **ToolJet** (https://github.com/ToolJet/ToolJet) : An open-source, low-code business application builder. [Sign up](https://www.tooljet.com/signup) for a free ToolJet cloud account or [run ToolJet on your local machine](https://docs.tooljet.com/docs/setup/try-tooljet/) using Docker.
- **Gemini API Key** : Log into [Google AI Studio](https://aistudio.google.com/app/apikey) using your existing Google credentials. Within the AI Studio interface, you'll be able to locate and copy your API key.

Here is a quick preview of our final application:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qim5qd042q11qq1pxtx3.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/98cdkxof9nkg2opompzb.png)


-------------------------------------------------------------


## Building our UI

- Login to your [ToolJet account](https://app.tooljet.com/). Navigate to ToolJet dashboard and click on the **Create new app** button on the top left corner. ToolJet comes with 45+ built-in components. This will let us set up our UI in no time.
- Drag and drop the **Container** component onto the canvas from the component library on the right side. Adjust the height and width of the **Container** component appropriately.
- Similarly, drag-and-drop the **Icon** and two **Text** components inside your container. We'll use these two **Text** components for our header and instructional text.
- Select the **Icon** component, navigate to its properties panel on the right and select the _ZoomMoney_ icon under its **Icon** property.
- Change the font size and content of the **Text** component appropriately.
- Drag and drop the **File Picker** and the **Button** component inside your container. We'll use the **File Picker** component to allow users to upload images of their receipts. The **Button** component will be used to trigger the **OCR** process.
- Rename the **File Picker** component to _fileUploader_.
- Adjust the width of the **File Picker** component according to your preference.
- Change the colour and text of the **Button** component according to your preference.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/slg0w97ly9httwd5py1r.png)

We have implemented the user interface for uploading receipts. The next step is to create the UI for administrators to approve or reject expense submissions.

- Click on the **Pages** icon on the left side of the screen. On the header of the Pages Panel, click on the **+** button to create a new page.
- Drag and drop the **Container** component onto the canvas. Adjust the height and width of the **Container** component appropriately.
- Drag and drop the **Icon** and a **Text** component inside your container. We'll use these two components for our logo and header text.
- Drag and drop the **Table** component inside the container. We'll use the **Table** component to display the list of expense submissions. We'll later also add **Action** Buttons to approve, reject, and view the receipt image for each submission.
- Drag and drop the **Modal** component inside the container. Open the **Modal** component and drag and drop the **HTML** component inside the Modal. We'll use the **HTML** component to display the receipt image.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bj5z6mg5jmeegfutpkgi.png)


-------------------------------------------------------------


## Creating Queries

**ToolJet** allows connecting to third-party APIs using its REST API query feature. We'll use this to connect to the **Gemini** API to extract the text from the uploaded receipt images. ToolJet comes with built-in integration with **AWS S3**, we'll use this to store the receipt images.

- Using ToolJet's [Workspace Constants](https://docs.tooljet.com/docs/org-management/workspaces/workspace_constants/) feature, create a new constant named **GEMINI_API_KEY** with your Gemini API key.
- In the query panel, click the **+ Add** button and choose the **REST API** option.
- Rename the query to _extractTextFromImage_.
In the Request parameter, choose **POST** as the Method from the drop-down and paste the following URL.
```
https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro:generateContent?key={{constants.GEMINI_API_KEY}}
```
- Navigate to the Body section of the extractTextFromImage. Toggle on **Raw JSON** and enter the following code:

```
{
  "contents": [
    {
      "parts": [
        {
          "text": "In this image of a receipt, analyze the receipt image and return the following information in JSON format without any formatting or syntax highlighting: total_amount, date."
        },
        {
          "inline_data": {
            "mime_type":"image/jpeg",
              "data": "{{components.fileUploader.file[0].base64Data}}"
          }
        }
      ]
    }
  ]
}

```

- Next, we'll create a query to save extracted text to the ToolJet database.

- Click on the **ToolJet** logo on the top left corner and select the Database option.

- Click on the **Create new table** button and name the table reimbursement_requests. Add the following columns to the table: _id_, _name_, _email_, _total_amount_, _status_ and _receipt_date_.



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/56abdgwm05sti36sct6d.png)

- Navigate back to the Query panel and click on the **+ Add** button. Choose the **ToolJet Database** option.

- Rename the query to _addReceiptData_.

- Select the _reimbursement_requests_ table name from the drop-down.

- In the Operation drop-down, choose **Create row** and enter the following data for the columns.

| Column Name | Key |
| --- | --- |
| email | `{{globals.currentUser.email}}` |
| name | `{{globals.currentUser.name}}` |
| total_amount | `{{JSON.parse(queries.extractTextFromImage.data.candidates[0].content.parts[0].text).total_amount}}` |
| receipt_date | `{{JSON.parse(queries.extractTextFromImage.data.candidates[0].content.parts[0].text).date}}` |

- Next, we'll create a query to store the receipt image in the AWS S3 bucket.

- Create a new data source to connect to the AWS S3 using ToolJet's built-in [AWS S3 integration](https://docs.tooljet.com/docs/data-sources/s3/).

- Click on the **+ Add** button in the Query panel and choose the newly created AWS S3 data source.

- Rename the query to _addToS3_ and select the operation as **Upload object**.

- Add the following data for the columns:

| Column Name | Key | 
| --- | --- |
| Bucket | bucket name (should already exist) |
| Key | `{{"reimbursement_id" + "_" + queries.addReceiptData.data[0].id}}` |
| Content Type | `{{components.fileUploader.file[0].type}}` |
| Upload Data | `{{JSON.parse(queries.extractTextFromImage.data.candidates[0].content.parts[0].text).date}}` |
| Encoding | base64 |

Next, we'll create the query to fetch the list of expense submissions from the ToolJet database.

- Click on the **+ Add** button in the Query panel and choose the ToolJet Database option and rename the query to _getReimbursementRequests_.

- Select the _reimbursement_requests_ table from the drop-down.

- In the Operation drop-down, choose **List rows**.

- To ensure that the query runs every time the application loads, enable the **Run this query on application load?** toggle.

- Next, create two more ToolJet database queries to approve and reject the expense submissions, named _approveRequest_ and _rejectRequest_.

- For both of these queries, select the _reimbursement_requests_ table and choose the **Update row** operation and to use the filter field to match the id of the row to be updated using the `{{components.reimbursementRequestsTable.selectedRow.id}}` variable.

- In the columns field, update the Status column to approved for the _approveRequest_ query and rejected for the _rejectRequest_ query.

Let's add our last query to fetch the receipt image from the AWS S3 bucket.

- Click on the **+ Add** button in the Query panel and choose the AWS S3 data source and rename the query to _getReceiptImage_ and enter the following data.

| Column Name | Key |
| --- | --- |
| Operation | Signed url for download |
| Bucket | bucket name |
| Key | `{{"reimbursement_id" + "_" + components.table1.selectedRow.id}}` |
| Expiries in | 3600 |

-------------------------------------------------------------


## Binding Queries to the UI Components

Now that we have successfully built our UI and queries, the next step is to integrate them.

- Select the **Button** component and navigate to the properties panel on the right. Click on the **+ New event handler** button. Change the **Action** to **Run query** and select the _extractTextFromImage_ query.

- Next, navigate to the _extractTextFromImage_ query and click on the **+ New event handler** button. Change the **Action** to **Run query** and select the _addReceiptData_ query.

- Navigate to the _addReceiptData_ query and click on the **+ New event handler** button. Change the **Action** to **Run query** and select the _addToS3_ query.

- Our upload receipts feature is now complete. You can test this out by uploading a receipt image and verify the extracted data in your ToolJet database.

Next, we'll implement the admin approval feature.

- Navigate to the Admin page and select the **Table** component. In the properties panel on the right, enter `{{queries.getReimbursementRequests.data}}` in the Data field.

- Click on the **+ New action button** in the properties panel and create three new actions buttons: **Approve**, **Reject**, and **View Receipt**. Bind the _approveRequest_ and _rejectRequest_ to the respective action buttons using Event Handlers.

- Change the background colour and text of the action buttons according to your preference.

- Click on the **View Receipt** action button and Add a new event handler. Change the Action to Open Modal and select the _displayReceiptImage_ modal.

- Navigate to the _displayReceiptImage_ modal and in the properties panel on the right, uncheck the Use **default trigger button** toggle.

- Inside the modal, click on the **HTML** component and enter `<img src={{queries.getReceiptFromS3.data.url}}>` in the Raw HTML field. 

- Our admin approval feature is now complete. You can test this out by uploading a receipt image and verifying the extracted data in the Table component. You can click on the **Approve** or **Reject** buttons to approve or reject the expense submission. You can also view the receipt image by clicking on the **View Receipt** button.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7chbcr4tutudc0iie5rs.png)



-------------------------------------------------------------


## Conclusion

Congratulations! You've successfully built a powerful reimbursement tracking app with OCR capabilities using ToolJet and the Gemini API. You can now track and manage expense submissions with ease. Feel free to customize the app further by adding more features and functionalities.
To learn and explore more about ToolJet, check out the [ToolJet docs](https://docs.tooljet.com/docs/) or connect with us and post your queries on [Slack](https://join.slack.com/t/tooljet/shared_invite/zt-2ij7t3rzo-qV7WTUTyDVQkwVxTlpxQqw).