Integrating TinyML, GenAI, and Twilio API for Smart Solutions
Tags: devchallenge, twiliochallenge, ai, twilio
Published: 2024-06-24 06:58:40

*This is a submission for the [Twilio Challenge](https://dev.to/challenges/twilio)*

## What I Built
<!-- Share an overview about your project. -->

The goal of this project is to demonstrate the integration of Generative AI (GenAI) in embedded systems and to highlight the ease of accessing AI capabilities via Twilio's API.

I have integrated a continuous motion recognition system with generative AI that recognizes and tracks motion and uses Twilio's WhatsApp API to alert users and respond to their questions.

![Project Skeleton](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/553w1xgal3fss10uxgfh.png)

## How I Built

### Continuous Motion Recognition

Unlike rule-based programming, which struggles with the variability in human gestures, machine learning easily adapts to these differences. 

The Continuous Motion Recognition project using [Edge Impulse](https://edgeimpulse.com/) leverages advanced ML techniques to enable real-time, on-device motion detection and classification. Edge Impulse, a leading platform for embedded machine learning, provides a comprehensive toolset for collecting data, designing, and deploying custom models directly to edge devices. This project typically involves gathering motion data from sensors such as accelerometers and gyroscopes, then using Edge Impulse to preprocess the data, train models, and optimize them for efficient, low-power inference on microcontrollers or other edge hardware.

You can follow the [Continuous motion recognition | Edge Impulse Documentation](https://docs.edgeimpulse.com/docs/tutorials/end-to-end-tutorials/continuous-motion-recognition) tutorial to get started with it. You can view and clone the project [here](https://studio.edgeimpulse.com/public/432186/live).


So, if you have gone through it successfully and deployed it with your firmware and when you run the inference, you will be able to see the real-time recognition of the motion.

![Edge Impulse Inference](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bgzsh84vv1obvmx3lkfa.png)

### Storing the result

For this project, I used [Xata](https://xata.io/) as the database. I have filtered out the result with the maximum value from each prediction and stored it in the `predictions` table. If you are new to Xata, checkout [Getting started with Xata](https://aadarshkannan.vercel.app/posts/Getting-Started-With-Xata/) guide.

![Xata Table](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d0ay7ybf5nqiuba7t170.png)

### Twilio and AI
<!-- Tell us how you leveraged Twilio’s capabilities with AI -->

#### Twilio's WhatsApp Messaging Service

Twilio provides access to the WhatsApp Business Platform through its APIs. This allows developers to integrate WhatsApp messaging into their applications. We can leverage familiar Twilio APIs like Programmable Messaging to send and receive messages with text, media, and even certain pre-built templates.

I have used Python-Flask for creating the server and responding to the incoming messages. You can check out the Twilio documentation on [Receive and Reply to Incoming Messages - Python](https://www.twilio.com/docs/messaging/tutorials/how-to-receive-and-reply/python) for a better understanding of how it needs to be implemented.

#### Delivering the message

By using the Twilio sandbox, we are able to send and receive messages, create automated responses, and integrate interactive messaging features within a controlled setting.

When someone replies to one of our messages, we will receive a webhook request from Twilio. So, we need to configure the webhooks by connecting to our server/app. I ran my server using [Replit](https://replit.com/). Search for the Flask template and integrate the Twilio API.

{% embed https://replit.com/@dotaadarsh/EdgeAIxTwilioxXata?v=1 %}

![Replit Server](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/opyzl31rjxtului2539z.png)

#### Google Gemini AI

I used Google's 1.5 Flash Model to generate responses based on predicted data. When a user replies on WhatsApp, the endpoint URL is configured to our server on Replit, which listens for these messages. We then retrieve all records from the database and pass them, along with the user's message, to the Gemini AI for generating a response. The generated response is then sent back to the user on WhatsApp.

## Demo
<!-- Share a link to your app and include some screenshots here. -->

![Twilio Whatsapp Response](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7cz934uq3pav68e6auay.png)

![Twilio Whatsapp Response](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ocmwjk0neiu712lxe2q6.png)

{% embed https://github.com/dotAadarsh/EdgeAIxTwilioxGenAI %}

### References

- [Twilio Documentation](https://www.twilio.com/docs)
- [Edge Impulse Documentation](https://docs.edgeimpulse.com/docs)
- [Xata Documentation](https://xata.io/docs)
- [Gemini API Documentation](https://ai.google.dev/gemini-api/docs/api-overview)

---

The primary objective of this project is to demonstrate the use of GenAI in EdgeAI/ML devices. By integrating Twilio's AI, users can now easily analyze ML-predicted data simply by asking questions through WhatsApp. While this project may not directly solve specific problems, it showcases the potential of applying GenAI to predicted data on a large scale.

Feedback is welcome :)
Thank you for reading and have a great day!