Xpenser - AI Expense Manger using Twilio & Gemini AI with WhatsApp Integration
Tags: devchallenge, twiliochallenge, ai, twilio
Published: 2024-06-24 05:31:19

*This is a submission for the [Twilio Challenge ](https://dev.to/challenges/twilio)*

## What I Built
**Xpenser** is not just a simple expense manager. It is an advanced expense management tool designed to simplify logging and analyzing your spending habits. With robust features, including seamless **WhatsApp integration** powered by **Twilio**, Xpenser stands out as a unique solution for personal finance management.

## Demo

Checkout the Demo on YouTube at:

{% embed https://youtu.be/r4B9uW-3aeQ %}

Experience Xpenser live [https://xpenser.amitwani.dev](https://xpenser.amitwani.dev). 

Explore the project on GitHub: {% embed https://github.com/mtwn105/xpenser-whatsapp-bot %}

To test the WhatsApp integration, join the Twilio Sandbox WhatsApp Number: [+14155238886](https://wa.me/+14155238886) by sending **`join per-mud`**

### Workflow

### User Registration

Users register on the web portal, verifying their phone numbers via OTP (Powered by Twilio Verify).

![User Registration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ctisw6tmbqvdw25n4mxa.png)

![User Registration OTP](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qvwyja8ag0phud3vf98t.png)

### Dashboard

The dashboard offers detailed analytics, including **AI-driven insights**, **category-wise distribution**, and **average spending**.

![Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nv197f1m3jx8zigb4kcr.png)

### Expenses

Users can easily **add**, **manage**, and **delete** expenses. The **AI** automatically categorizes expenses upon saving.

![Add Expense](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6pjvv9ns0w14ck167sia.png)

![Expense List](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i07vsr7abupddbc4a748.png)

### WhatsApp Integration

Join the Twilio Sandbox WhatsApp Number: [+14155238886](https://wa.me/+14155238886) by sending **`join per-mud`**

![WhatsApp Join](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x7gfgmz5yodgw51wakav.png)

Log new expenses effortlessly via WhatsApp. The **AI** interprets user messages to gather details for saving the expense.

![WhatsApp Log Expense](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qmtvcrhzffom52r14jpz.png)

Request insights on spending, such as top categories or average spending. The AI provides answers based on the last 100 expenses.

![WhatsApp Expense Analysis](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3mgvenhhzy7yp4j5tunl.png)

## Tech Stack
**Backend:** NodeJS, Express
**Frontend:** Angular, Typescript
**Database:** MongoDB
**Deployment:** Docker, Northflank
**AI:** Gemini 1.5 Flash
**Twilio:** Verify, Functions, WhatsApp 


### Architecture


### WhatsApp Integration

![WhatsApp Integration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g2kwfuz0xisu96uhinl9.png)


### Portal 

![Xpenser Portal](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ftxcofyz1hawirhjeq0k.png)


## Twilio and AI

### Twilio

Xpenser leverages three Twilio services: **Verify**, **Functions** and **WhatsApp Messaging**

**Verify**

Twilio's Verify service sends OTPs for user authentication.

**Send OTP**

```typescript
const verification = await client.verify.v2.services(otpServiceId)
      .verifications
      .create({ to: req.body.phonenumber, channel: 'sms' });
```

**Verify OTP**

```typescript
 const verification = await client.verify.v2.services(otpServiceId)
      .verificationChecks
      .create({ to: req.body.phonenumber, code: req.body.code })
```

**Functions**

Twilio Functions handles incoming WhatsApp messages, interfacing with the **backend** and **Gemini AI**.

Generated functions using 

```bash
 twilio serverless:init twilio --typescript
```

```typescript
export const handler: ServerlessFunctionSignature = async function (
  context: Context<MyContext>,
  event: MyEvent,
  callback: ServerlessCallback
) {
  // Handle Incoming Message 
}
```

**WhatsApp Messaging**

Twilio's Programmable WhatsApp Messaging facilitates user interaction through WhatsApp.

**Send WhatsApp Message using TwiML**

```typescript
const twiml = new Twilio.twiml.MessagingResponse();
twiml.message("Please ask about spends you have done or spend you want to log.");
callback(null, twiml);
```

### AI

**Gemini 1.5 Flash**, a Google AI Model, enhances Xpenser with advanced analytics and predictions.

**Expense Analytics**

Analyzing user expense history to provide insights and respond to user queries.

```typescript
const analyticsModel = genAI.getGenerativeModel({
	model: "gemini-1.5-flash",
	systemInstruction: "Based on below expenses done by user give answer to user. Optimize as WhatsApp message reply and add emojis.\n\nExpense History:\n" + expenseCsvText,
  });

const generationConfig = {
  temperature: 0.6,
  topP: 0.95,
  topK: 64,
  maxOutputTokens: 8192,
  responseMimeType: "application/json",
};

const chatSession = model.startChat({
  generationConfig
});

const result = await chatSession.sendMessage(event.Body);
const aiResponse = result.response.text()

```

**Expense Categorization**

Categorizing expenses based on user-provided descriptions.

```typescript
const analyticsModel = genAI.getGenerativeModel({
	model: "gemini-1.5-flash",
	systemInstruction: "detect the category and subcategory from below\n1. Housing - Rent, Taxes, Insurance, Utilities, Repairs, Improvement, Fees\n2. Transportation - Payments, Fuel, Insurance, Repairs, Public, Parking, Tolls, Licensing\n3. Food - Groceries, Dining, Coffee, Delivery, Snacks, etc.",
  });

const generationConfig = {
  temperature: 0.6,
  topP: 0.95,
  topK: 64,
  maxOutputTokens: 8192,
  responseMimeType: "application/json",
};

const chatSession = model.startChat({
  generationConfig
});

const result = await chatSession.sendMessage(event.Body);
const aiResponse = result.response.text()

```

## Additional Prize Categories
**Xpenser** qualifies for the **Twilio Times Two** prize category by utilizing multiple Twilio features: **Verify**, **Functions**, and **WhatsApp Messaging**.

## Future Scope
- Spending Habits: Use AI to identify spending patterns and suggest ways to optimize expenses based on historical data.
- Mobile Applications: Develop native iOS and Android applications for enhanced accessibility and a more seamless user experience.
- Custom Reports: Allow users to generate customizable reports based on various parameters such as date range, categories, or tags.
- Investment Tracking: Add features to track investments and provide insights into portfolio performance.
- Manage Categories: Create and manage custom categories and sub-categories for a more personalized experience.

## Conclusion
It was a great experience to work on this project while exploring **Twilio** services and integrating AI to enhance the user experience. Thanks to **Dev.to** team for this amazing hackathon.