Expenses made easier with Cloudflare AI
Tags: cloudflarechallenge, devchallenge, ai
Published: 2024-04-11 14:50:45

*This is a submission for the [Cloudflare AI Challenge](https://dev.to/devteam/join-us-for-the-cloudflare-ai-challenge-3000-in-prizes-5f99).*

## What I Built

This project is a **Cloudflare serverless worker** with an AI service binding deployed for processing either audio, image or text submitted by users into semantic useful information, especially in the context for _submitting expenses_.


For users that prefer having a UI to interact with, a simple **NextJS web app** is created to demonstrate how to upload files and utilizes API routes to make a POST request to the Cloudflare worker endpoint.

## How it works

> The diagram below looks a bit blurry on this platform, please refer to the raw image [here](https://github.com/lyqht/cf-journal/blob/main/demo/How%20it%20works.png?raw=true)

![Illustrates the user journey below](https://github.com/lyqht/cf-journal/blob/main/demo/How%20it%20works.png?raw=true)

User can pass any of the following types of content - audio, image or text. Based on the input type given, we will process them eventually to form JSON like this that **can be used by any app** to display useful expense information for them.

```ts
const SAMPLE_RECORD: Expense = {
	date: "2024-04-11",
	time: "14:00",
	type: "product",
	item: "Rubber Ducky",
	expenditure: 100.0,
};
```

### Models used in this project:

- Image-to-text: [@cf/unum/uform-gen2-qwen-500m](https://developers.cloudflare.com/workers-ai/models/uform-gen2-qwen-500m/)
- Automatic Speech Recognition: [@cf/openai/whisper](https://developers.cloudflare.com/workers-ai/models/whisper/)
- Text Generation: [@hf/thebloke/mistral-7b-instruct-v0.1-awq](https://developers.cloudflare.com/workers-ai/models/mistral-7b-instruct-v0.1/)


## Demo

### Try out the worker!

The worker is deployed at https://cf-journal.senchatea.workers.dev.

Text input

```bash
curl --location 'https://cf-journal.senchatea.workers.dev?type=text' \
--header 'Content-Type: text/plain' \
--data '2 weeks ago, I went to eat a buffet at Swensens Unlimited at the T2 airport, it'\''s really nice but it costs like $36 per person after GST, and there were 2 of us.'
```

Audio input

```bash
curl --location 'https://cf-journal.senchatea.workers.dev?type=audio' \
--header 'Content-Type: application/octet-stream' \
--header 'Authorization: Bearer pYrzMvsyURxsCUeaQDsa3lSO_tBDQEuiPB3iLEQt' \
--data '@postman-cloud:///1eef4b5b-a64c-4c10-ad4b-5f22c528880b'
```

Image input

```bash
curl --location 'https://cf-journal.senchatea.workers.dev?type=image' \
--header 'Content-Type: application/octet-stream' \
--header 'Authorization: Bearer pYrzMvsyURxsCUeaQDsa3lSO_tBDQEuiPB3iLEQt' \
--data '@postman-cloud:///1eef4b68-e5d7-49e0-a28d-f60b3bd7d70e'
```

### Try it in the web app!

https://cf-journal.vercel.app/

![On load of the app, user sees a sample record of a Rubber Ducky costing $100, but not for sale. User then tries to upload a .mp3 voice recording file and sees "Uploading" text. After upload is completed, they can see a new expense being added on top of the sample record, with the expenditure of $100, date of "2024-04-11", time of "14:00" and type of expense "Activity".](https://github.com/lyqht/cf-journal/blob/cc8bc6d1aae0a638214d5ab197c5d7c1e8425237/demo/web_demo.gif?raw=true)

## My Code

The project has 2 folders:
- `serverless`: Code that is deployed to a Cloudflare worker. You can find the text generation prompt & code to use the AI models at `utils.ts`. The Cloudflare worker request handler is found at `main.ts`
- `app`: Nextjs app code that is deployed to Vercel

{% embed https://github.com/lyqht/cf-journal %}

## Journey

### How the idea came about

Before OpenAI introduced ChatGPT and revitalized the AI scene, I once created [Billy](https://github.com/lyqht/Billy), a cute a little expense tracker app to help my mom to track her expenses easier. It actually received good reception with my friends both online & offline because of the clean and straight forward UI. 

However, the user retention wasn't very good because **the act of logging expenses is very tedious** because of the multitude of fields to fill. It was very difficult to encourage the habit of logging expenses for the users.

For many users, it is much direct and easier to talk or take a picture with a phone as compared to typing. But it was rather difficult to interpret information from the data of these input types back then, so I dropped that project eventually.

Now, by utilizing different AI models, we can interpret and organize important information for them more easily, simplifying the expense submission process.

This project is meant to be a MVP to demonstrate that.

### Room for improvements

I'm very new to AI so there will definitely a lot of things to improve on!

The webapp in this submission is a very early MVP of what I envision the actual expense tracker app to be. 

- For now, it only create expenses based on the information in the file input (audio/mp3/text).
- Ideally after submission, you can still modify the individual fields of the expense form. The AI portion is just there to assist the auto-filling process.

I'm also not very familiar with prompt engineering, but I would also like to improve the text generation prompt in the future.

### Resources

These are some helpful resources from Cloudflare that I have used to work on this project since I wasn't familiar with the AI models available for text generation/inference.

- [Workers AI LLM Playground](https://playground.ai.cloudflare.com/)
- [Guide on choosing the right
  text generation model](https://developers.cloudflare.com/workers-ai/tutorials/how-to-choose-the-right-text-generation-model/)

---

My project is eligible for the **Triple Task** type category.